Learning Multi-faceted Knowledge Graph Embeddings for Natural Language Processing

1 Introduction

Knowledge graph (KG) embeddings are the essential tools that transfer the important concepts of the complicated human languages into machine-understandable representations. The past half decade of research has paid much attention to translation-based methods, which provide simple techniques to encode entities in low-dimensional spaces and capture relations as means of translations among entity vectors.

Although efforts have been made to improve such methods by introducing various forms of relation-specific entity projections (see [Chen et al., 2017] for summary), almost all works focus on characterizing monolingual knowledge defined on simple or multi-mapping relations. This is far from enough for completely representing the KGs, inasmuch as KGs usually contain more complicated forms of knowledge, including multilingual knowledge that synchronizes multiple versions of KGs [Mahdisoltani et al., 2015], as well as monolingual knowledge that enforces relational properties such as transitivity and symmetry, and forms hierarchies [McGuinness et al., 2004]. We propose some embedding approaches that pursue to joint learning of monolingual and multilingual knowledge, and better preserving relational properties and hierarchies of monolingual knowledge in learning process. So far as related works largely limit the use of KG embeddings to tasks directly associated with KGs [Ji et al., 2016], like relation extraction and triple classification, we propose the use of our approaches in a wider spectrum of NLP tasks.

2 Proposed Approaches

We give the formalization of the corpora, with the highlight on the multi-faceted knowledge we consider.

2.1 Formalization of Corpora

In a knowledge base KB, we use L to denote the set of languages, and L2 to denote the unordered language pairs. For L ∈ L, GL denotes the language-specific KG of L, and EL and RL respectively denote the corresponding vocabularies of entities and relations. T = (h, r, t) denotes a triple in GL such that h, t ∈ EL and r ∈ RL. Boldfaced h, r, t represent the embedding vectors of head h, relation r, and tail t. For a language pair (L1,L2) ∈ L2 , δ(L1,L2) denotes the alignment set which contains the aligned pairs of triples.

Within a GL, we extend the relations by RL = Rtr ∪ Rs∪Rh∪Ro, which respectively denote the sets of transitive, symmetric, hierarchical, and other simple relations. Thereof, Rh = Rr ∪Rc where Rr denotes refinement relations that divide entities to finer ones, and Rc denotes coercion relations that merge entities to coarser ones [Chen et al., 2016].

2.2 LearningMultilingual Knowledge

Leveraging KG embeddings to multilingual knowledge no doubt provides more generic representations that benefits cross-lingual NLP. But it is non-trivial for several reasons: (i) multilingual knowledge has far larger domains than monolingual relations; (ii) it applies to both entities and relations with incoherent vocabularies in every language; (iii) usually only a small portion of KB has been aligned with such knowledge.

We have proposed the MTransE model [Chen et al., 2017] that employs two model components to learn on the two facets of KB: knowledge model that encodes the entities and relations from each language-specific graph structure, and alignment model that learns cross-lingual transitions. It is noteworthy that, MTransE is defined on a pair of language. A set of model for each (Li, Lj) ∈ L2 composes the solution for more than two languages w.l.o.g.

Knowledge Model. For each language L ∈ L, a dedicated embedding space RkL is assigned for vectors of EL and RL, where R is the field of real numbers. We employ TransE [Bordes et al., 2013] for each language by adopting the objective function SK = ∑L∈{Li,Lj} ∑(h,r,t)∈GL ‖h+ r - t‖, which benefits the cross-lingual NLP tasks with the uniform representation of entities under different contexts of relations.

Alignment Model. Alignment model constructs the transitions between the vector spaces of Li and Lj according to the objective function SA = ∑(T,T ′ )∈δ(Li,Lj) Sa(T, T ′ ). The alignment score Sa(T, T ′ ) thereof iterates through all pairs of aligned triples. Three different techniques to represent the alignment scores are considered, i.e., distance-based axis calibration, translation vectors, and linear transforms. These techniques lead to five forms of Sa.

MTransE minimizes the objective function J = SK +αSA via jointly training on triples and alignment sets, where α is a hyperparameter that weights between two model components. Several variants of MTransE are implemented based on different techniques applied in the alignment model.

2.3 Learning with Properties and Hierarchies

In many KGs, monolingual knowledge commonly enforces relational properties, and forms hierarchies. For example, 85% of triples in ConceptNet [Speer and others, 2013] and 96% in Yago3 [Mahdisoltani et al., 2015] are with relational properties, while 60% and 38% of their triples are with hierarchical relations respectively. Thus, we investigate another model On2Vec that forthputs the Component-specific Model and Hierarchy Model to handle such knowledge.

Component-specific Model. Existing approaches mainly apply the same relation-specific projection on both head and tail entities [Ji et al., 2016]. This inevitably causes conflicts on triples with relational properties:

• Consider r ∈ Rtr and (e1, r, e2), (e2, r, e3), (e1, r, e3) ∈ GL, where e1, e2, and e3 are projected to e1r, e2r, and e3r in the relation space of r. Since e1r + r ≈ e2r and e2r + r ≈ e3r hold for the first and second triples, it is impossible for e1r + r ≈ e3r to hold as r is nonzero.
• Consider r ∈ Rs and (e1, r, e2), (e2, r, e1) ∈ GL. It is impossible for both e1r + r ≈ e2r and e2r + r ≈ e1r to hold, since r is nonzero.

To eliminate such conflicts, the component-specific model applies two different relation-specific projections f1,r and f2,r on head and tail entities respectively to the same relation space. f1,r and f2,r can be implemented as hyperplane mappings [Wang et al., 2014] or linear transforms.

Hierarchy Model. Given r ∈ Rh, we define σ(e, r) as the refine operator to fetch the finer entities that apply r with the coarser entity e. Hierarchy model serves as an auxiliary learning process for Rh towards the goal of converging the projected embeddings of e′ ∈ σ(e, r) within a tight neighborhood given each e and r, while penalizing other unrelated entities that fall inside the margin.

Training of On2Vec jointly optimizes the objective of both model components. Bernoulli negative sampling [Wang et al., 2014] is adopted for more efficient learning. Experimental results on relation extraction show that On2Vec outperforms the related approaches [Chen, 2017], which verifies the effectiveness of the two On2Vec model components.

3 NLP Applications

We propose to apply these approaches to several NLP tasks. Knowledge Alignment. We have applied MTransE to both entity and triple-wise cross-lingual alignment tasks on large KGs derived from Wikipedia and ConceptNet, which received promising results and outperformed approaches based on multilingual word embeddings [Chen et al., 2017]. Results of monolingual tasks also show that MTransE better preserves monolingual knowledge than its monolingual counterpart.

Semantics Relatedness Analysis (SRA). Unsupervised SRA tasks such as LP50 [Chen, 2017] request machine-estimated document semantic distances to receive a high Pearson's correlation with human judgement. A straightforward solution is via vector representations of documents by aggregating the embeddings of Wikipedia entities that are recognized in each document. Currently, as we use Annotated Skip-gram trained on Wikipedia articles to conceive the vectors, this simple solution is effective enough to outperform many approaches [Chen, 2017]. As next step, we will represent the documents by training On2Vec on the Wikipedia-derived Yago3. Similarly, we will use MTransE to enable cross-lingual SRA.

Open Information Extraction (OpenIE). Neural openIE [Lin et al., 2016] relies on word embeddings to obtain the tensor representations of sentences. A practical alternative is to use word-knowledge joint embeddings [Chen, 2017] based on On2Vec and Annotated Skip-gram, which stands on the assumption that KG embeddings better highlight the important concepts in sentences [Ji et al., 2016], amongst which the neural classier aims at predicting the relations.

Sentiment Analysis. Deep neural networks similar to the one for openIE will be used to tackle sentiment analysis tasks summarized in [Kim, 2014]. We will evaluate the following sentence representation techniques: (i) joint embeddingbased tensor representations; (ii) aggregated embeddings of highlighted KG entities; (iii) combination of (i) and (ii).