Hierarchical Linear Disentanglement of Data-Driven Conceptual Spaces

1 Introduction

Vector space representations of entities, i.e. entity embeddings, play a central role in information retrieval [Deerwester et al., 1990], natural language processing [Mikolov et al., 2013] and machine learning [Norouzi et al., 2014], among others. Accordingly, a wide variety of approaches have already been proposed for learning such representations, most of which essentially try to learn vectors that represent the similarity structure of the given domain. The use of geometric representations is also common in cognitive science [Shepard, 1957; Nosofsky, 1986; G¨ardenfors, 2000], with the conceptual spaces framework by G¨ardenfors being a particularly prominent example [G¨ardenfors, 2000]. Compared to entity embeddings, conceptual spaces have a much richer structure, allowing them to act as an interface between symbolic and sub-symbolic representations. Essentially, we can think of a conceptual space as being defined by a set of domains, such as e.g. colour, shape or emotion, where each of them is associated with a set of primitive semantic features, called quality dimensions. For instance, the colour domain can be described using the features hue, saturation and intensity. The vector representation of a given domain is then given by the Cartesian product of these quality dimensions. When representing a particular entity in a conceptual space, we need to specify which domains it belongs to, and for each of these domains we need to provide a corresponding vector. We note that conceptual spaces have been used for two different purposes in the literature. On the one hand, they are commonly used in perceptual domains, e.g. for music cognition [Forth et al., 2010; Chella, 2015], where quality dimensions are carefully chosen to maximize how well the resulting conceptual spaces can predict human similarity judgements. On the other hand, they have also been considered as a more general intermediate representation in between neural representations and symbolic ones [G¨ardenfors, 1997]. We consider this more general view in this paper, where our aim is add structure to embeddings in the form of semantically meaningful dimensions.

∗Contact Author

In contrast to conceptual spaces, most methods for learning entity embeddings only aim to capture similarity, and the dimensions of the resulting vector spaces do not typically have any particular meaning. However, despite the similaritycentric nature of most embedding methods, learned vector spaces often exhibit remarkable linear regularities. For example, a well-known property of word embeddings is that many syntactic and semantic relationships can be captured in terms of word vector differences [Mikolov et al., 2013]. We are particularly interested in the fact that important semantic features from a given domain can often be modelled as directions in the corresponding entity embedding. More precisely, for a salient semantic feature f, there often exists a vector df such that df · a < df · b tends to hold if b has that feature to a higher extent than a, where we write a and b for the vector representations of a and b. Note that only the direction of df matters in this case, which is why we say that the feature is modelled as a direction rather than as a vector. For example, studying the word vector representations of countries and cities, [Gupta et al., 2015] found directions in the word embedding that highly correlate with (the rankings induced by) properties such as the GDP, life expectancy and military expenditure of a country. In another study, [Kim and de Marneffe, 2013] identified directions that signify adjective scales such as hot-warm-cool-cold in the word embedding. In addition to these supervised approaches, [Derrac and Schockaert, 2015] proposed an unsupervised method which uses text descriptions of the considered entities to identify semantic features that can be characterized as directions. Their core assumption is that words describing semantically meaningful features can be identified by learning for each candidate word w a linear classifier which separates the embeddings of entities that have w in their description from the others. The performance of the classifier for w then tells us to what extent w describes a semantically meaningful feature 1 . This allows learning a linear transformation from the given embedding space to a "disentangled" representation, where entities are represented as vectors whose coordinates correspond to coherent semantic features. The dimensions of this disentangled representation can intuitively be interpreted as the quality dimensions of a conceptual space.

However, many semantic features do not make sense for all entities. For instance, in an embedding of movies, we may consider a feature that captures how closely a movie adheres to the book it is based on. While meaningful for book adaptations, this feature would be non-sensical for other movies. As an important practical implication, if quality dimensions are learned from the full set of entities, while only being sensical for a subset of these entities, we may expect them to be sub-optimal. This problem is illustrated in Fig. 1, which displays a projection of an embedding of organisations. In Fig. 1a, the green dots correspond to those organisations whose associated description contains words such as political, politic, party, parties, politicians. Because organisations whose description contains such words are more or less linearly separable from other organisations, the method from [Derrac and Schockaert, 2015] discovered this cluster as a semantic feature. Now consider Fig. 1b, where the yellow dots correspond to organisations whose descriptions contain words such as democratic and left-wings. While this cluster describes a feature that is intuitively clear (i.e. organisations associated with left-wing political ideas), this feature is only relevant for a subset of organisations (i.e. political ones). A key, and perhaps surprising, observation is that this is reflected in the vector space. In particular, as can be seen in the figure, this feature cannot be characterized well using a single hyperplane.

Decomposing the embedding into different domains could solve this issue, but finding a suitable decomposition is a highly non-trivial problem, especially in unsupervised settings [Locatello et al., 2018; Alshaikh et al., 2019]. Instead of trying to find a hard decomposition of the entity embedding into separate domains, we propose a simple but effective method which is based on applying the method from [Derrac and Schockaert, 2015] in a hierarchical fashion. In the example from Fig. 1, for instance, we can view the feature political as defining a domain. To obtain a suitable characterization of the feature democratic, it then suffices to apply the method from [Derrac and Schockaert, 2015] to that domain instead of to the full space. In this way, we obtain a set of primary features, and for each of these primary features we obtain a set of sub-features. As confirmed by the experimental results, by learning the features in such a hierarchical way, we obtain semantically more meaningful representations than when directly applying the method from [Derrac and Schockaert, 2015]. This shows that the intuitive hierarchical relationship that exists between features (e.g. the fact that democratic is a sub-feature of political) is effectively reflected in the structure of the entity embedding. While in general it is well-known that learned embeddings exhibit linear regularities, to the best of our knowledge, this is the first paper to show that these linear regularities have an inherent hierarchical structure.

1 It may seem counter-intuitive to use binary classifiers to learn representations of ordinal features. However, the occurrence or nonoccurrence of a word in the description is binary, and this is the most important available signal. We experimented with statistics such as pointwise mutual information, which did not lead to better results.

2 RelatedWork

An improvement to the method from [Derrac and Schockaert, 2015] was recently proposed in [Ager et al., 2018], where a (non-linear) fine-tuning step was introduced. In [Alshaikh et al., 2019] the problem of clustering the learned dimensions into different facets was studied, finding that this only appears feasible with an external supervision signal, in that case coming from pre-trained word embeddings. Some methods for learning entity embeddings directly incorporate the idea that semantic features should correspond to vector directions. For instance, [Jameel et al., 2017] learn entity embeddings from Wikipedia based on a method that associates an ordinal linear regression model with each word from the vocabulary.

The idea of disentangled representation learning has been widely studied within the context of images. For example, the seminal the InfoGANmodel [Chen et al., 2016] uses a variant of generative adversarial networks [Goodfellow et al., 2014] to learn interpretable embeddings of images. Their main idea is to use mutual information to force the individual dimensions of the learned latent representation to correspond to informative properties. When it comes to representations that are learned from text, however, disentangled representation learning has received far less attention. One notable exception is [Jain et al., 2018], where a supervised approach is proposed for learning disentangled document embeddings. As training data, they use triples of the form (s, d, o)a, meaning that relative to aspect a, d is more similar to s than to o. In the context of sentiment analysis, some work has been done on learning representations that disentangle different aspects mentioned in reviews [Ruder et al., 2016]. In [Esmaeili et al., 2019], a general unsupervised disentangled representation learning method is proposed, which is also applied to text. However, this method is only evaluated intrinsically based on statistical measures of disentanglement which do not reveal how semantically meaningful the learned features are. The aforementioned methods focus on representations which are relatively low-dimensional (e.g. typically involving 100 to 300 dimensions). Besides such methods, several approaches have been proposed for learning disentangled text representations which are much higher-dimensional. For example, [Gabrilovich and Markovitch, 2007] learn document vectors with one coordinate for each Wikipedia page, encoding how related the document is to the corresponding Wikipedia entity. Similarly, [Camacho-Collados et al., 2016] proposed entity representations in which dimensions correspond to BabelNet synsets.

3 Identifying Feature Directions

The problem we consider is to derive a feature-based (i.e. disentangled) description of a given set of entities. In particular, we want to represent each entity e as a vector (f e 1 , ..., f e k ) ∈ Rk such that each of its components corresponds to a semantically meaningful feature. This amounts to identifying vectors d1, ..., dk, in the given embedding, which represent the most salient semantic features. The learned vectors will be referred to as feature directions to emphasize the fact that only the ordering induced by the dot product di · e matters. The featurebased representation of e is simply given by (e·d1, ..., e·dk).

Learning primary features. We first recall the method from [Derrac and Schockaert, 2015] and propose two modifications. This method trains for each word w in the vocabulary a linear classifier which predicts from the embedding of an entity whether w occurs in its description. The words w1, ..., wn for which this classifier performs sufficiently well are then used as basic features. To assess classifier performance, Cohen's Kappa score, which can be seen as a correction of classification accuracy to deal with class imbalance, is used. Each of the basic features w is associated with a corresponding vector dw (i.e. the normal vector of the separating hyperplane learned by the classifier). These directions are subsequently clustered, which serves to reduce the total number of features. This is useful to keep the representation lowdimensional (e.g. to avoid overfitting when training a classifier on the resulting feature-based representation). By associating features with clusters of words, rather than individual words, features are more likely to correspond to salient properties of the domain. We noticed that irrelevant words tend to be clustered together, resulting in a small number of non-informative clusters, with most of the other clusters corresponding to semantically meaningful features. While [Derrac and Schockaert, 2015] used a variant of k-means, we use affinity propagation [Frey and Dueck, 2007], which gives better results and does not require us to specify the number of clusters, which is crucial in our hierarchical setting.

Once the clusters C1, ...,Ck are obtained, the final step is to associate with each of them a corresponding feature direction. In [Derrac and Schockaert, 2015], the direction dC for a cluster {u1, .., um} is defined as the average of du1 , ..., dum . We instead learn a new linear classifier, which tries to separate entities whose description contains at least one of the words u1, ..., um from the other entities. We then define dC as the normal vector of the corresponding hyperplane. This was found to perform better, and it gives us a natural criterion to select entities that have a given feature to a sufficient extent, i.e. those that are classified as positive by this new classifier. For a cluster C, we write posC and negC for the set of positively and negatively classified entities. We will also refer to the clusters C1, ...,Ck as the primary feature clusters, and to the associated directions dC as the primary feature directions.

Learning sub-features. To find sub-features of a given primary feature f with associated cluster Cf and direction df , we essentially re-apply the same method used for learning the primary features, but now only considering the set of entities in posCf . In fact, we look for words that linearly separate those entities that are deemed to have the corresponding primary feature 2 . Specifically, for each word w from Cf , we train a linear classifier using only the entities in posCf . As before, we use the Kappa score to select those words for which the associated classifier performs sufficiently well. The directions modelling the selected words are then again clustered using affinity propagation. We crucially rely on the fact that this does not require us to specify the number of clusters, as it would not be feasible to tune the number of sub-clusters for each primary feature. Note that we could recursively repeat the whole process to further refine the sub-features. In this paper, however, we only consider two-level hierarchies, as these were found to be sufficient for the considered datasets.

2 We also experimented with a variant in which the entities were weighted by the probabilities predicted by a logistic regression classifier. We then trained the sub-feature classifier using a weighted logistic loss, but noticed no consistent improvements.

Let D1, ...,Dkf be the sub-feature clusters that were found for the primary feature f, and let dfD1 , ..., dfDkf be the corresponding sub-feature directions. To define the feature-based representation of a given entity, we now associate one component with each primary feature and one component with each sub-feature. To compute the value corresponding to a given sub-feature for an entity e, we simply take the dot product e · dfDi , i.e. the feature values are computed in exactly the same way as for primary features. Note that while the features are discovered in a hierarchical way, the resulting feature-based representation is thus essentially flat. An alternative would be to combine e · dfDi with e · df to define the value of the sub-feature. In initial experiments, however, this was found to perform poorly. One reason is that the classification of entities into posCf and negCf is not perfect. For instance, an organisation could be a borderline instance of the category of political organisations, but still be a representative example of a left-wing organisation (e.g. a newspaper with a strong left-wing bias), even if in general we tend to think of left-wing as a sub-feature of political. Another reason is that by simply using the value e · dfDi , we minimize the redundancy between the information captured by the primary feature f and the information captured by this sub-feature.

Inspired by this latter view, we also consider a variant in which we require each sub-feature direction dfDi to be orthogonal to the corresponding primary feature direction df , as a way to directly impose the idea that primary features and sub-features should provide complementary information. To this end, we first obtain a sub-feature direction dfDi as before, and then compute the orthogonal decomposition of this vector w.r.t. the vector df . In particular, as the final sub-feature direction, we then use the following vector:

4 Evaluation

To evaluate whether the discovered features are semantically meaningful, we test how similar they are to natural categories, by training depth-1 decision trees (meaning that only a single feature can be used for prediction) on our feature-based representations. For instance, in the movie domain, we should expect to see common movie genres among the features. Depth1 decision trees should thus be able to predict these genres well. Following [Ager et al., 2018], we also evaluate how well natural categories can be characterized using a small set of features, based on the performance of depth-3 decision trees.

Methods. We compare two versions of our method: the standard version (Sub) and the version where the orthogonal decomposition (1) is used (Ortho). We also consider three baselines. First, we use a model that only uses primary features (Primary). Second, we use a model in which feature directions are randomly chosen (Random), by sampling each coordinate from a standard normal distribution (which after normalization is equivalent to sampling from a uniform distribution on the hypersphere). Note that while using random directions may seem naive, related methods such as using random projections for dimensionality reduction often perform surprisingly well [Bingham and Mannila, 2001]. As the third baseline, we use average-link Agglomerative Hierarchical Clustering (AHC) to cluster word directions instead of affinity propagation. To obtain a two-level clustering from the dendrogram, we tune distance cut-offs d1 and d2 to determine the set of primary clusters and their corresponding subclusters. Once the clusters are determined, we learn a corresponding cluster direction in the same way as how the cluster directions for primary features are learned with our method. We also experimented with Hierarchical LDA, but found it too slow to be used on our datasets.

Apart from how the feature directions are constructed, the overall number of features also has a strong impact on the result, where increasing the number of features increases the chance that one of them reflects a natural category (even if directions are chosen by chance), but at the risk of overfitting. For the random baseline, we directly tune the number of directions on a held-out development set, considering values from {100, 500, 1000, 1500, 2000, 2500}. We also verified that no further improvements were possible by choosing more than 2500 or fewer than 100 directions. For the methods which use affinity propagation, we can only influence the number of clusters indirectly, by changing the so-called preference parameter of this clustering algorithm. As is usual, this parameter is chosen relative to the median µ of the affinity scores. For the methods Sub and Ortho, we considered values from {0.7µ, 0.9µ, µ, 1.1µ, 1.3µ}. For the Primary method, we considered a larger set of values to verify that no further improvements were possible: {0.5µ, 0.7µ, 0.9µ, µ, 1.1µ, 1.3µ, 1.5µ}. In the case of AHC, to make the results as comparable as possible to those of our model, we tune the cut-offs d1 and d2 such that the number of selected clusters is as close as possible to the optimal numbers obtained with affinity propagation. We also tried tuning the cut-off values directly, but this did not give better results.

Datasets. Learning disentangled entity representations only makes sense in domain-specific contexts: while it is natural to think about meaningful features for comparing different movies (e.g. genres), there are few features that would be meaningful in an open-domain setting. To test our method, we focus on five different domains, where for each domain a number of classification problems are considered. The first two domains are obtained from [Derrac and Schockaert, 2015]: a movies dataset, where the text descriptions correspond to movie reviews, and a place types dataset, where the text descriptions correspond to bags of Flickr tags. In addition, we considered the organisations and buildings datasets that were introduced in [Alshaikh et al., 2019]. In both cases, the text descriptions correspond to Wikipedia articles and the entities were selected based on Wikidata semantic types. The associated classification tasks similarly correspond to predicting Wikidata attributes of the entities. We used the same process to construct a fifth dataset, focusing on the domain of music bands 3 . Table 1 provides statistics about the datasets.

Entity embeddings. For the movies and place type domains, we used the embeddings that were shared by [Derrac and Schockaert, 2015]. These are 100-dimensional vector spaces obtained using multi-dimensional scaling (MDS), which is a common approach for learning semantic spaces in cognitive science. For the other domains, we generated 100dimensional embeddings using both MDS and Doc2Vec [Le and Mikolov, 2014], the latter being a popular neural network model for document embedding. In initial experiments, we also considered document embeddings learned by the neural variational document model [Miao et al., 2016]. However, we found the resulting embeddings to be of much lower quality.

Methodology. The datasets are divided into 70% training and 30% testing splits. To tune the parameters, we used 5fold cross-validation on the training split. Since the movies dataset is substantially larger, in that case we instead used a fixed 60% training, 20% testing and 20% tuning split. We report the results in terms of F1 score. To obtain the feature directions, we used logistic regression and only considered words for which the corresponding Kappa score is at least 0.3. To reduce the computation time, for datasets where this led to more than 5000 features, only the 5000 top-scoring words are retained. When learning directions for the sub-features, we use a lower Kappa score of 0.1, as the corresponding classification problems are often harder (given that sub-features are often about subtle nuances of the primary feature) and the set of candidate words is smaller.

3 The datasets and source code are available online at https: //github.com/rana-alshaikh/Hierarchical Linear Disentanglement.

Experimental results. The results are summarized in Tables 2 and 3. We can see that our Sub method clearly outperforms Primary. In fact, there are several cases where the performance of Primary is comparable with that of Random, yet where our Sub method performs substantially better (e.g. the depth-1 results for the genres in the movies domain). While the sub-features never perform clearly worse than the primary features, there are some cases where both approaches perform similarly (e.g. the genre attribute in the bands domain). This can be expected when the considered attribute is sufficiently dominant, in which case most or all of the attribute values might correspond to primary features. Interestingly, however, for the movies domain, many of the considered genres were only modelled well as sub-features. The Ortho method generally performs well, but slightly worse than Sub. Interestingly, however, for the place types, the Ortho method performs best overall. This dataset is rather noisy, and taking the orthogonal complement in this case seems to help with preventing overfitting. AHC fails to consistently outperform the other baselines, and is clearly worse than our method. This shows that the improvements obtained by our model are not due to the fact that we impose a hierarchical structure on the features as such, but rather due to specific way in which we learn the directions of the sub-features. Comparing the MDS and Doc2Vec embeddings, we generally see the same trends, although the results for Doc2Vec are consistently worse. To better understand the impact of the number of features, Fig. 2 shows the performance of the different methods in function of the total number of features. As can be seen, while the number of features clearly matters, the improvement that we observed for the sub-features is remarkably robust.

Qualitative analysis. Table 4 shows some illustrative examples of features that were found for the movies domain. As can be seen, sub-features can play a number of different roles. For the primary feature [delightful,cute], the sub-features correspond to different kinds of movies that are often described as delightful or cute. These comprise a rather diverse set of movies, encompassing romantic, musicals and children's movies, among others. Without considering sub-features, movies from these various genres would have a very similar representation, which is clearly undesirable. By identifying sub-features, we are able to differentiate between these genres. The primary feature [gore,gory], on the other hand, corresponds to a more coherent movie genre. The sub-features in this case essentially correspond to sub-genres, such as zombie and serial killers movies. For [sci,science] as primary feature, we see the sub-feature [cgi,technology,computer,graphics] as an example of a quality dimension that mostly makes sense within the scope of scifi movies. A similar example is [educational] which is found as a sub-feature of [documentary,interviews]. The examples in Tab. 4 also illustrate the hybrid nature of the found features, where some of them intuitively correspond to more or less well-defined domains while others intuitively correspond to quality dimensions. As far as the primary features are concerned, we would expect to see mostly features of the former kind, but as the example [delightful,cute] illustrates, that is not always the case. One immediate advantage of learning quality dimensions is that we can use them to learn interpretable classifiers. This is illustrated in Fig. 3 that shows depth-3 decision trees for the movie genres family and music. For most features, we show a subset of the words from the corresponding cluster. It is clear what kind of properties about the considered genres the learned models have uncovered. Such examples show that data-driven conceptual spaces, while clearly being messier than the idealised representations considered by [G¨ardenfors, 2000], can indeed bridge between vector space and symbolic representations in a useful way. While these learned representations are not a substitute for the carefully constructed spaces that are often used in perceptual domains, it would be interesting to study in future work to what extent they can be used, for instance, to implement methods for computational creativity based on conceptual spaces [Agres et al., 2015].

5 Conclusions

Learning feature-based entity representations is complicated by the fact that many features only make sense for particular subsets of entities. We showed that this issue can be mitigated by learning feature representations, characterized as directions in the embedding, in a hierarchical way. Essentially, some of the identified features serve to split up the given entity embedding into different sub-domains, while other features are more similar in spirit to quality dimensions. Compared to strategies that aim to explicitly decompose the embedding into separate domains, our method has the advantage that no hard decisions have to be made about which directions define sub-domains and which directions correspond to quality dimensions. equivalently be seen as semantic features).

Acknowledgments

S. Schockaert was funded by ERC Starting Grant 637277. Z. Bouraoui was funded by ANR CHAIRE IA BE4musIA.