Learning Conceptual Space Representations of Interrelated Concepts

1 Introduction

Conceptual spaces are geometric representations of knowledge, in which the objects from some domain of interest are represented as points in a metric space, and concepts are modelled as (possibly vague) convex regions [G¨ardenfors, 2000]. The theory of conceptual spaces has been extensively used in philosophy, e.g. to study metaphors and vagueness [Douven et al., 2013], and in psychology, e.g. to study perception in domains such as color [J¨ager, 2009] and music [Forth et al., 2010], and in other areas such as Robotics [Chella et al., 2003] and Machine Vision [Chella et al., 1997]. However, the lack of automated methods for learning conceptual spaces from data has held back its adoption in the field of artificial intelligence. While a number of such methods have recently been proposed [Jameel et al., 2017], an important remaining problem is that these methods typically do not explicitly model concepts, i.e. they only learn the representations of the objects, while it is the concept representations that are mostly needed in applications such as knowledge base completion. The problem we study in this paper is to induce these missing concept representations from these object representations.

Steven Schockaert

Cardiff University, UK SchockaertS1@Cardiff.ac.uk

To illustrate the considered problem, Figure 1 shows two dimensions from a higher-dimensional conceptual space of places. The red dots correspond to places which are asserted to be train stations in the SUMO ontology or on WikiData. From this red point cloud, we can learn a soft boundary for the concept train station, which is illustrated by the ellipsoidal contours in the figure. We can then plausible conclude that points which are within these boundaries are likely to be train stations. In accordance with prototype theory, this model essentially assumes that the likelihood that an object is considered to be an instance of a train station depends on its distance to a prototype. Note that by considering ellipsoidal rather than spherical contours, we can take into consideration that different dimensions may have different levels of importance in any given context. In principle, there are several strategies that could be used to find suitable ellipsoids for a given concept, e.g. we could train a support vector machine with a quadratic kernel or we could fit a Gaussian distribution. The key problem with these methods, however, is that conceptual spaces usually have hundreds of dimensions, whereas only a few instances of each concept may be available, making learned concept representations potentially unreliable.

While this cannot be avoided in general, in many applications we have some background knowledge about how the considered concepts are interrelated. In this paper, we propose a Bayesian model which exploits such background knowledge while jointly learning a representations for all concepts. In particular, we assume that concepts can be modelled using Gaussians, and we use the available background knowledge to construct informative priors on the parameters of these Gaussians. We will consider two kinds of such background knowledge. First, we will consider logical dependencies between the concepts, which can be encoded using description logic (DL). For instance, SUMO encodes the knowledge that each instance of TrainStation is an instance of terminalBuilding or of transitBuilding, which we should intuitively be able to exploit when learning representations for these concepts. Second, we will use the fact that many concepts themselves also correspond to objects in some conceptual space. For example, while train station is a concept in a conceptual space of places, it is an object in a conceptual space of place types. As we will see below, these representations of concepts-as-points can be used to implement a form of analogical reasoning. We experimentally demonstrate the effectiveness of our proposed model in a knowledge base completion task. Specifically, we consider the problem of identifying missing instances of the concepts from a given ontology. We show that our method is able to find such instances, even for concepts for which initially only very instances were known, or even none at all.

2 RelatedWork

Learning Conceptual Spaces One common strategy to obtain conceptual spaces is to learn them from human similarity judgments using multidimensional scaling. Clearly, however, such a strategy is only feasible in small domains. To enable larger-scale application, a number of approaches have recently been proposed for learning Euclidean 1 conceptual space representations in a purely data-driven way. In our experiments, we will in particular rely on the MEmbER model from [Jameel et al., 2017], which learns vector space representations that can be seen as approximate conceptual spaces. For instance, in contrast to most other vector space models, objects of the same semantic type are grouped in lowerdimensional subspaces within which dimensions corresponding to salient features (i.e. quality dimensions) can be found. In other words, this approach can be seen as learning a set of conceptual spaces, one for each considered semantic type, which are themselves embedded in a higher-dimensional vector space. Most importantly for this paper, the objective of the MEmbERmodel directly imposes the requirement that all entities which are strongly related to a given word (i.e. whose textual descriptions contain sufficiently many occurrences of the word) should be located within some ellipsoidal region of the space. It thus aims to learn a representation in which concepts can be faithfully modelled as densities with ellipsoidal contours, such as Gaussians.

1 While the use of Euclidean spaces is quite natural, another common choice is to use a two-level representation, where a concept at the top level is a weighted set of properties, each of which corresponds to a convex region in a different Euclidean space. Such representations open up interesting possibilities, but there are currently no established methods for learning them in an automated way.

Throughout this paper, we will assume that natural concepts can be modelled as (scaled) Gaussians. This corresponds to a common implementation of prototype theory [Rosch, 1973], in which the prototype of a concept is represented as a point and the membership degree of an object is proportional to its exponential squared distance to the prototype. Note that, in general, prototypes do not have to be modelled as points, e.g. a more general approach is to model prototypes as regions [Douven et al., 2013]. However, the restriction to prototype points is a useful simplifying assumption if we want to learn reasonable concept representations from small numbers of instances. Similarly, while in principle it would be useful to model concepts as Gaussian mixture models, a strategy which was proposed in [Rosseel, 2002] to generalize both prototype and exemplar models, this would only be feasible if a large number of instances of each concept were known.

Knowledge Graph Completion The main application task considered in this paper is knowledge base completion, i.e. identifying plausible facts which are missing from a given knowledge base. Broadly speaking, three types of highly complementary methods have been considered for this task. First, some methods focus on identifying and exploiting statistical regularities in the given knowledge base, e.g. by learning predictive latent clusters of predicates [Kok and Domingos, 2007; Rockt¨aschel and Riedel, 2016; Sourek et al., 2016] or by embedding predicates and entities in a lowdimensional vector space [Bordes et al., 2013]. The second class consists of approaches which extract facts that are asserted in a text corpus. For example, starting with [Hearst, 1992], a large number of methods for learning taxonomies from text have been proposed [Kozareva and Hovy, 2010; Alfarone and Davis, 2015]. Several authors have proposed methods that use a given incomplete knowledge base as a form of distant supervision, to learn how to extract specific types of fine-grained semantic relationships from a text corpus [Mintz et al., 2009; Riedel et al., 2010]. Thirdly, some methods, including ours, aim to explicitly represent concepts in some underlying feature space. For example, [Neelakantan and Chang, 2015] represents each Freebase entity using a combination of features derived from Freebase itself and fromWikipedia, and then uses a max-margin model to identify missing types. In [Bouraoui et al., 2017], description logic concepts were modelled as Gaussians in a vector space embedding. Crucially, these existing works consider each concept in isolation, which requires that large numbers of instances are known, and this is often not the case.

Few Shot Learning Considerable attention has also been paid to the problem of learning categories for which no, or only few training examples are available, especially within the area of image recognition. For example, in one common setting, each category is defined w.r.t. a set of features, and the assumption is that we have training examples for some of the categories, but not for all of them. Broadly speaking, the aim is then to learn a model of the individual features, rather than the categories, which then makes it possible to make predictions about previously unseen categories [Palatucci et al., 2009; Romera-Paredes and Torr, 2015]. Other approaches instead exploit the representation of the category names in a word embedding [Socher et al., 2013]. We will similarly exploit vector space representations of concept names.

3 Background

We will rely on description logic encoding of how different concepts are related, and we will use a Bayesian approach for estimating Gaussians modelling these concepts. In this section, we briefly recall the required technical background on these two topics. For a more comprehensive discussion, we refer to [Baader et al., 2003] and [Murphy, 2007] respectively.

Description Logics Description logics are a family of logics which are aimed at formalizing ontological knowledge about the concepts from a given domain of interest. The basic ingredients are individuals, concepts and roles, which at the semantic level respectively correspond to objects, sets of objects, and binary relations between objects. A knowledge base in this context consists of two parts: a TBox, which encodes how the different concepts and roles from the ontology are related, and an ABox, which enumerates some of the instances of the considered concepts and roles. The TBox is encoded as a set of concept inclusion axioms of the form C ⊑ D, which intuitively encodes that every instance of the concept C is also an instance of the concept D. Here, C and D are so-called concept expressions. These concept expressions are either atomic concepts or complex concepts. In this paper we will consider complex concepts that are constructed in the following ways:

• If C and D are concept expressions, then C ⊓ D and C ⊔D are also concept expressions, modelling the intersection and union of the concepts C and D respectively.
• If C is a concept expression and R is a role, then ∃R.C and ∀R.C are also concept expressions. Intuitively, an individual belongs to the concept ∃R.C if it is related (w.r.t. the role R) to some instance fromC; an individual belongs to ∀R.C if it can only be related (w.r.t. R) to instances of C.

From a given knowledge base, we can typically infer further concept inclusion axioms and ABox assertions, although the complexity of such reasoning tasks crucially depends on the specific description logic variant that is considered (e.g. which types of constructs are allowed and what restrictions are imposed on concept inclusion axioms). Note that the methods we propose in this paper are independent of any particular description logic variant; we will simply assume that an external reasoner is available to infer such axioms.

Bayesian Estimation of Gaussians Suppose a set of data points x1, ..., xn ∈ R have been generated from a univariate Gaussian distribution G with a known variance σ and an unknown mean. Rather than estimating a single value µ∗ of this mean, in the Bayesian setting we estimate a probability distribution M over possible means. Suppose our prior beliefs about the mean are modelled by the Gaussian P = N(µP , σ2P ). After observing the data points x1, ..., xn our beliefs about µ are then modelled by the distribution M defined by:

It can be shown that this distribution M is a Gaussian N(µM, σ2M ), where:

Now consider a setting where the variance of the Gaussian is unknown, but the mean is known to be µ. For computational reasons, prior beliefs on the variance are usually modelled using an inverse χ2 distribution (or a related distribution such as inverse Gamma); let us write this as Q = χ-2 (νQ, σ2Q). Intuitively, this means that we a priori believe the variance is approximately σ2Q, with νQ expressing the strength of this belief. After observing the data, our beliefs about the possible values of σ2 are modelled by the distribution S, defined by:

It can be shown that S = χ-2 (νS, σ2S ) where:

4 Learning Concept Representations

We assume that a description logic ontology is given, and that for each individual a mentioned in the ABox of this ontology, a vector representation va ∈ Rn is available. For our experiments, these representations will be obtained using the MEmbER model, although in principle other vector space models could also be used. The task we consider is to learn a Gaussian GC = (µC,ΣC) for each concept which is mentioned in the TBox or ABox, as well as for all constituents of these concepts (e.g. if the concept C1 ⊔ ... ⊔ Ck is mentioned then we also learn Gaussians for C1, ..., Ck), along with a scaling factor λC > 0, such that the probability that the individual a belongs to concept C is given by:

Intuitively, the variance of GC encodes how much the instances of C are dispersed across the space, while λC allows us to control how common such instances are. Formally, if we assume that the prior on va is uniform, λC is proportional to the prior probability P(C) that an individual belongs to C.

Given that the number of known instances of the concept C might be far lower than the number of dimensions n, it is impossible to reliably learn the covariance matrix ΣC without imposing some drastic regularity assumptions. To this end, we will make the common simplifying assumption that ΣC is a diagonal matrix. The problem of estimating the multivariate Gaussian GC then simplifies to the problem of estimating n univariate Gaussians. In the following, for a multivariate Gaussian G = N(µG, ΣG), we write µG,i for the ith component of µG and σ2G,i for the ith diagonal element of ΣG.

To find the parameters of these Gaussians, we will exploit background knowledge about the logical relationships between the concepts. However, this means that the parameters of the Gaussian corresponding to some concept C may depend on the parameters of the Gaussians corresponding to other concepts. To cope with the fact that this may result in cyclic dependencies, we will rely on Gibbs sampling, which is explained next. This process will crucially rely on the construction of informative priors on the parameters of the Gaussians, which is discussed in Sections 4.2 and 4.3. Finally, Section 4.4 explains how the scaling factors λC are estimated.

4.1 Gibbs Sampling

The purpose of Gibbs sampling is to generate sequences of parameters µ0C , µ1C , ... and Σ0C ,Σ1C , ... for each concept. To make predictions, we will then average over the samples in these sequences. We will write µC,i,j for the ith component of µjC and σ2C,i,j for the ith diagonal element of ΣjC .

The initial parameters µ0C and Σ0C are chosen as follows. If v1, ..., vk are the vector representations of the known instances of C and k ≥ 2, we choose:

where we write vl,i for the ith coordinate of vl. If k ≤ 1 the parameters µ0C and σ2C,i,0 are estimated based on the superconcepts of C in the ontology; more details about these corner cases are provided in an online appendix [Bouraoui and Schockaert, 2018].

After the initial parameters have been chosen, we repeatedly iterate over all concepts. In the ith iteration (i > 0), we choose the next samples µiC and ΣiC for each concept C, according to (1) and (2) respectively. To do this, however, we first need to define prior probabilities on ΣiC and µiC . These prior probabilities will be constructed by taking into account the available background knowledge about how the different concepts are interrelated, as we explain in detail in Sections 4.2 and 4.3. In particular, the prior probabilities on ΣiC and µiC will be defined in function of the parameters of the Gaussians of the other concepts. When using Gibbs sampling, we always use the most recent samples of the parameters of these other concepts.

For the ease of presentation, we will write µ∗B and Σ∗B for the most recent samples of µB and ΣB. In other words, µ∗B = µi-1 B or µ∗B = µiB holds, depending on whether µB was already updated in the current iteration of the Gibbs sampler, and similar for Σ∗B . We also write G∗B for N(µ∗B , Σ∗B ), i.e. the most recent estimation of the Gaussian GB. We also use the notations µB,i,∗ and σ2B,i,∗ to refer to the ith component of µ∗B and the ith diagonal element of Σ∗B respectively.

4.2 Priors on the Mean

The type of information that is available to construct a prior on the mean µiC is different for atomic and for complex concepts, which is why we discuss these cases separately.

Atomic Concepts

For an atomic concept A, we use two types of information to construct the prior PA = N(µPA , ΣPA ) that is used for sampling µiA. First, the TBox may contain a number of axioms of the form A ⊑ C1, ...,A ⊑ Ck. If A ⊑ Cl holds then µA should correspond to a plausible instance of Cl. In particular, we would expect the probability G∗Cl (µA) to be high.

Second, if a vector representation vA of the concept A itself is available, it can also provide us with useful information about the likely values of µA. Suppose B1, ...,Br are atomic concepts such that the TBox contains or implies the axioms B1 ⊑ Cl, ..., Br ⊑ Cl (in addition to A ⊑ Cl). We will refer to B1, ...,Br as the siblings of A w.r.t. Cl. The information that we want to encode in the prior PA is that the vector differences vB1 -µ∗B1 , ..., vBr -µ∗Br should all be similar to the vector difference vA -µA. This is motivated by the fact that, in the context of word embeddings, analogical word pairs typically have similar vector differences [Mikolov et al., 2013; Vylomova et al., 2016]. In particular, it corresponds to the intuitive assumption that the relation between the prototype of a concept and the vector space embedding of the concept name should be analogous for all concepts, and in particular for all siblings of A. This intuition can be encoded by estimating a Gaussian ECl = N(µECl , ΣECl ) from the vector differences vB1 -µ∗B1 , ..., vBr -µ∗Br and the representations vB1 , ..., vBr themselves, as follows (assuming r ≥ 2):

where vA,j is the j th coordinate of vA, and similar for vBu,j. Using this Gaussian, we can encode our intuition by requiring that ECl (µA) should be high. If r = 1, then σ2ECl ,j is estimated based on the superconcepts of Cl; details about this corner case can be found in the online appendix.

Combining both types of background knowledge, we choose a prior PA which encodes that PA(µA) is proportional to G∗C1 (µA) · ...G∗Cl (µA) · EC1 (µA) · ... · ECl (µA), as follows:

For the ease of presentation, here we have assumed that A has at least one sibling w.r.t. each Cl. In practice, if this is not the case, the corresponding Gaussian ECl (µA) is simply omitted. Note that when the translation assumption underlying the Gaussians ECl is not satisfied, the associated variances σ2ECu ,j will be large, and accordingly the information from the vector space embeddings will be mostly ignored.

Complex Concepts

To construct a prior on µiC for a complex concept C, we can again use concept inclusion axioms of the form C ⊑ C1, ...,C ⊑ Ck entailed from the ontology, but for complex concepts we do not have access to a vector space representations. However, additional prior information can be derived from the atomic concepts and roles from which C is constructed. For example, let C ≡ D1 ⊓ ... ⊓ Ds. Then we can make the assumption that P(C|v) is proportional to G∗D1 (c) · ... · G∗Ds (c). The product of these Gaussians is proportional to a Gaussian H∗C with the following parameters:

This leads to the following choice for the prior:

For complex concepts of the form D1 ⊔ ... ⊔ Ds, ∃R . C and ∀R.C, a similar strategy can be used. Details about these cases can be found in the online appendix.

4.3 Priors on the Variance

For a concept C, we write QC,j = χ-2 (νQC,j , σ2QC,j ) for the prior on σ2C,j , the j th diagonal element of ΣC. We now discuss how the parameters νQC,j and σ2QC,j are chosen.

Atomic Concepts

Let A be an atomic concept. We will exploit two types of information about the variance σ2A,j . First, if A ⊑ C then σ2A,j ≤ σ2C,j should hold. Second, we might expect that the covariance matrix of GA is similar to that of the most closely related concepts. Specifically, let B1, ...,Bk be all the atomic siblings of A (i.e. each Bl is an atomic concept, and there is some C such that both A ⊑ C and Bl ⊑ C appear in the TBox). Then one possibility is to choose σ2QC,j as the average of σ2B1,j , ..., σ2Bk,j . If we have a vector space representation for A and for some of its (atomic) siblings, then we can improve on this estimation by only considering the most similar siblings, i.e. the siblings whose vector space representation is closest in terms of Euclidean distance. In particular, let B ⊆ {B1, ...,Bk} be the set of the κ most similar siblings of A, and let C1, ...,Cl be the set of concepts for which the TBox contains the axiom A ⊑ Cl, then we choose:

The parameter νQC,j intuitively reflects howstronglywewant to impose the prior on σ2A,j . Given that even closely related concepts could have a considerably different variance, we set νQA,j as a small constant η for each A and j.

Complex Concepts

For a complex concept C, the covariance matrix of H∗C , with H∗C the Gaussian constructed for complex concepts in Section 4.2, can be used to define the prior on σ2C,j . Let C1, ...,Ck be the concepts for which the TBox contains or implies the axiom C ⊑ Cl then we choose:

Furthermore, we again set νQC,j = η.

4.4 Making Predictions

For a given individual with vector representation v, we can estimate the probability P(C|v) that this individual is an instance of concept C as follows:

Note that compared to (3), in (4) the parameters of GC are averaged over the Gibbs samples. As usual with Gibbs sampling, the first few samples are discarded, so here µ1C and Σ1C refer to the first samples after the burn-in period.

The number of Gibbs sample that we used is equal to 1000 where each sample is generated after 25 every 25 iterations. The burn-in period that we use is fixed to 200 samples.

We estimate the scaling factor λC by maximizing the likelihood of the training data. In particular, let v1, ..., vs be the vector representations of the known instances of C, and let u1, ..., ur be the vector representations of the individuals which are not asserted to belong to C. Then we choose the value of λC that maximizes the following expression:

which is equivalent to maximizing:

Note that for concepts without any known instances, we would obtain λC = 0, which is too drastic. To avoid this issue, we replace s by s + 1 in (5), which is similar in spirit to the use of Laplace smoothing when estimating probabilities from sparse frequency counts. Furthermore note that the estimation of λC relies on a closed world assumption, i.e. we implicitly assume that individuals do not belong to C if they are not asserted to belong to C. Since this assumption may not be correct (i.e. some of the individuals u1, ..., ur may actually be instances of C, even if they are not asserted to be so), the value of λC we may end up with could be too low. This is typically not a problem, however, since it simply means that the predictions we make might be more cautious then they need to be.

5 Experimental Results

In this section, we experimentally evaluate our method 2 against a number of baseline methods. In our experiments, we have used SUMO 3 , which is a large open-domain ontology.

2 Implementation and data available at https://github.com/ flexilogalgo

3 http://www.adampease.org/OP/

An important advantage of using SUMO is that several of its concepts and individuals are explicitly mapped to WordNet, which itself is linked to WikiData. This means that we can straightforwardly align this ontology with our entity embedding. For concepts and individuals for which we do not have such a mapping, we use BabelNet 4 to suggest likely matches. We split the set of individuals into a training set Itrain containing 2/3 of all individuals, and a test set Itest containing the remaining 1/3. All ABox assertions involving individuals from Itrain are used as training data.

The considered evaluation task is to decide for a given assertion A(a) (meaning "a is an instance of A") whether it is correct or not. As positive examples, we use all assertions from the ABox involving individuals from Itest. To generate negative test examples, we use the following strategies. First, for each positive example A(a) and each concept B ̸= A such that the TBox implies A ⊑ B, we add a negative example by randomly selecting an individual x such that B(x) can be deduced from SUMO while A(x) cannot. Second, for each positive example A(a), we also add 10 negative examples by randomly selecting individuals among all those that are not known to be instances of x. Note that even if A(x) is not asserted by SUMO, it may be the case that x is an instance of A. This means that in a very small number of cases, the selected negative examples might actually be positive examples. The reported results are thus a lower bound on the actual performance of the different methods. Importantly, the relative performance of the different methods should not be affected by these false negatives.

The performance is reported in terms of average precision (AP), and micro-averaged precision (Pr), recall (Rec) and F1 score. To compute the AP scores, we rank the assertions from the test data (i.e. the correct ABox assertions as well as the constructed negative examples), across all considered concepts, according to how strongly we believe them to be correct, and then we compute the average precision of that ranking. To give a clearer picture of the performance of the different methods, however, we will break up the results according to the number of training examples we have for each concept (see below). Note that AP only evaluates our ability to rank individuals, and hence does not depend on the scaling factors λA. The precision, recall and F1 scores, however, do require us to make a hard choice.

4 BabelNet Java API available at http://babelnet.org

As baselines, we have considered a linear and quadratic support vector machine (SVM). We will refer to our model as Gibbs. We also consider three variants of our method: Gibbsflat, in which flat priors are used (i.e. no dependencies between concepts are taken into account), Gibbs-emb, in which the priors on the mean and variance are only obtained from the embedding (i.e. no axioms from the TBox are taken into account), and Gibbs-DL, in which the priors on the mean and variance are only obtained from the TBox axioms (i.e. the embedding is not taken into account).

The results are summarized in Table 2, where |X| refers to the number of training examples for concept X. Overall, our model consistently outperforms the baselines in both F1 and MAP score. For concepts with few known instances, the gains are substantial. Somewhat surprisingly, however, we even see clear gains for larger concepts. Regarding the variants of our model, it can be observed that using TBox axioms to estimate the priors on the mean and variance (GibbsDL) leads to better results than when a flat prior is used. The model Gibbs-emb, however, does not outperform Gibbs-flat. This means that the usefulness of the embedding, on its own, is limited. However, the full model, where the embedding is combined with the TBox axioms, does perform better than Gibbs-DL.

6 Conclusions and FutureWork

We have proposed a method for learning conceptual space representations of concepts. In particular, we associate with each concept a Gaussian distribution over a learned vector space embedding, which is in accordance with some implementations of prototype theory. In contrast to previous work, we explicitly take into account known dependencies between the different concepts when estimating these Gaussians. To this end, we take advantage of description logic axioms and information derived from the vector space representation of the concept names. This means that we can often make faithful predictions even for concepts for which only a few known instances are specified.

Acknowledgments

This work was supported by ERC Starting Grant 637277.