Neighborhood-Aware Attentional Representation for Multilingual Knowledge Graphs

1 Introduction

The multilingual knowledge graphs (KGs) like YAGO [Suchanek et al., 2008] and DBpedia [Bizer et al., 2009] increasingly play an significant role in supporting various knowledge-driven tasks. Those multilingual knowledge graphs consist of monolingual knowledge, in forms of directed graphs, where entities are represented as nodes and relations as edges. The monolingual knowledge are stored as triplets (eh, r, et), representing that the head entity eh and tail entity et are linked by relation r. Besides monolingual knowledge, the multilingual KGs also embody cross-lingual knowledge (eh, align(), e′h ) that matches the same real-world entities eh and e′h among different human languages L and L′ by alignment operation align(), see Figure 1. A great deal of methods focus on exploiting monolingual knowledge graphs in recent years. Particularly, the embedding-based methods that encode entities and relations into continue lowdimensional vector spaces, have achieved promising performance. Exemplarily, given a triplet (eh, r, et), TransE [Bordes et al., 2013] regards the relation embedding r as the translation vector between the head and tail entity embedding eh and et, and expects eh +r ≈ et when (eh, r, et) holds. Other extended works such as TransH [Wang et al., 2014], TransR [Lin et al., 2015a] and TransD [Ji et al., 2015] also emerged with different translation forms in characterizing relation r. However, a few methods have been done for modeling multilingual knowledge graphs.

∗Corresponding Author

Entity alignment is an effective way to integrate the multilingual KGs, which is the task of finding the same realworld entities in different KGs. The traditional multilingual entity alignment methods mainly based on machine translation, have low accuracy due to the poor performance in translation between multiple languages. Most recently, following above popular embedding-based models, MTransE [Chen et al., 2017] provides the cross-lingual transitions for both entities and relations across different knowledge graph embeddings. IPTransE [Zhu et al., 2017] jointly encodes both entities and relations of various KGs into an unified low-dimensional semantic space via sharing parameters on a seed set of aligned entities, JAPE [Sun et al., 2017] further incorporates attribute triplets as additional information for learning KGs' embeddings in an unified space. BootEA [Sun et al., 2018] adopts bootstrapping [Yarowsky, 1995; Abney, 2004] approach to iteratively label likely entity alignment as training data and leverage it for learning alignmentoriented embeddings. Existing entity alignment methods only use the triplet-based information, but ignore the inherent neighborhood information of entities for aligning entities.

Compared with the triplet-based information, the neighborlevel information factually implies more richer alignment properties because arbitrary two aligned (equivalent) entities in different KGs usually contain equivalent neighborhood information.

This paper proposes a neighborhood-aware attentional representation method NAEA for multilingual knowledge graph. As Figure 2 shown, NAEA are composed of knowledge embedding component KE and entity alignment component EA. Both KE and EA components incorporate neighbor-level and relation-level information with different weights for learning KGs' embedding representation and aligning entities respectively. KE component devises an attention mechanism to obtain neighbor-level representation by aggregating neighbors with a weighted combination, and uses the triplet-based characteristic information to learn relation-level representation. EA component determines alignments between entities across multilingual KGs by measuring the similarity of their integrated representations, which are also fused from neighbor-level and relation-level representations with different weights.

Our contributions. We propose a neighborhood-aware attentional representation model NAEA for multilingual knowledge graphs, which can better capture the entity and relation features for targeting aligned entities. Two components, knowledge embedding component KE and entity alignment component EA, are included in NAEA.

• NAEA incorporates neighbor-level with relation-level feature information in knowledge graphs to perform multilingual entity alignment task.
• KE component devises a neighborhood-aware attention mechanism to learn neighbor-level representation by aggregating neighbors' embeddings with different impacts.
• EA component performs entity alignment by the similarity of entity integrated representations, which integrates neighbor-level and relation-level feature representation.

We evaluate our NAEA model on two real-world datasets DBP15K and DWY100K. Experimental results show that NAEA significantly achieves state-of-the-art performance on entity alignment task.

2 ProblemFormulation

We describe an entity set as E, a relation set as R, a knowledge graph as G = {τ|τ = (eh, r, et)} with eh, et ∈ E, r ∈ R. A KG pair specified by languages L1 and L2 is G′ = (GL1 ,GL2 ), and its known aligned entities set is A = {(eI, eJ)|eI ∈ EL1 , eJ ∈ EL2 }, where EL1 and EL2 are respectively the entity set of GL1 and GL2 . Each aligned entity pair (eI , eJ) represents an entity eI in GL1 has its synonymous counterpart eJ in GL2 with language-specific surface names.

The entity alignment task in this paper is as follows: given the language-specific knowledge graph pair G′ = (GL1 , GL2 ), and the known alignment seeds A = {(eI , eJ )}, entity alignment aims to automatically find and align more unaligned entities.

3 Model

This paper proposes a neighborhood-aware attentional representation method NAEA for multilingual knowledge graphs. As illustrated in Figure 2, NAEA consists of knowledge embedding component KE and entity alignment component EA: (1) KE in section 3.1 aims to learn alignment-oriented knowledge embedding representation, and (2) EA in section 3.2 aims to perform alignment in various KGs.

3.1 Neighborhood-aware Knowledge Embedding

KE fuses neighbor-level and relation-level information, and introduces an attention mechanism to learn neighbor-level representation of heterogeneous KGs.

Neighbor-level Attentional Representation

For obtaining neighbor-level entity representation, we introduce an attention mechanism to aggregate entities' neighbors with different importance. As shown in Figure 2, the attention mechanism takes as input an entity eh with its neighborhood set {(rh1, eh1), (rh2, eh2), ..., (rhn, ehn)}, and outputs its corresponding neighbor-level feature representation Ne(eh). Here rhj with j = 1, ..., n is the relation that links from entity eh to ehj or vice versa, and n is the number of neighbors of entity eh.

Specifically, for each entity-neighbor pair, we first use two shared linear transformations, parameterized by two weight matrices W1 ∈ Rm×m and W2 ∈ Rm×2m, to transform the input features into higher-level features, and then adopt self-attention on entities to compute attention coefficients between entity eh and its neighborhood entity ehj with relation rhj, i.e.,

where ch,j represents the importance of entity ehj's features to entity eh with relation rhj , ah ∈ R2m is an entity-specific weight vector, eh,ehj,rhj ∈ Rm are the vector embeddings of eh,ehj,rhj, and [, ] indicates the concatenation operation. m is the dimension of entity and relation embedding space.

Obviously the above self-attention mechanism allows to calculate the importance of arbitrary two linked entities. To make coefficients easily comparable across different entities, we normalize them across all choices of j = 1, 2, ..., n using the softmax function:

To obtain the final output neighbor-level feature representation Ne(eh) of input entity eh, we compute a linear combination of its neighbors' high-level features with different weight coefficients αh,j , and adopt multi-head attention [Vaswani et al., 2017] to stabilize the process of self-attention, i.e.,

where φ = sigmoid is the activation function, αkh,j are the normalized attention coefficients computed by the k-th attention mechanism, and K is the number of head. Although the multi-head attention mechanism expands the parameter requirements by a factor of K, the individual heads' computations are fully independent and can be parallelized.

We mainly perform such attention mechanism to get neighbor-level entity representations. For the sake of fairness, we use Nr(r) = Wr to represent the neighbor-level relation representation of relation r with W ∈ Rm×m.

Relation-level Representation

Besides utilizing the neighbor-level information, we also consider the triplet-based information for our model NAEA. In order to fully preserve the inherent structural property of triplets, we use (eh, r, et) to describe the relation-level embedding representation of a triplet (eh, r, et).

Loss Function

For capturing neighbor-level information of knowledge graphs, similar to the widely-used embedding based model TransE [Bordes et al., 2013], we define the neighbor-level score function f1(τ) = ‖Ne(eh) + Nr(r) - Ne(et)‖22 for a triplet τ = (eh, r, et). Different from the margin-based ranking loss function used in TransE, we adopt following limitbased scoring loss function [Zhou et al., 2017] since it not only ensures the discrimination between the scores of positive and negative triplets, but also ensures the lower scores for positive triplets, i.e.,

where [·]+ means that max(·, 0), τ and τ ′ are respectively the positive and negative triplet from positive and negative triplet set T and T ′ , γ > 0 is a balance hyper-parameter, and µ1, µ2 are two hyper-parameters with the constraint µ1, µ2 > 0.

For capturing relation-level information of knowledge graphs, we also define the relation-level score function f2(τ) = ‖eh + r - et‖22 for measuring the plausibility of the triplet τ = (eh, r, et), and use the same loss framework as L1 to define relation-level scoring loss function L2.

We aim to learn embedding representations of KGs by fusing neighbor-level and relation-level information of knowledge graphs with different weights. Therefore, we propose following combined loss function for optimization:

where β is the weight hyper-parameter that balances the importance of neighbor-level of information.

3.2 Entity Alignment

Entity alignment aims to find A′ = {(ei, ej) ∈ EL1 × EL2 | ei ≡r ej}, where ei ≡r ej means that an equivalence relation ≡r holds between ei and ej. Usually the already known aligned pairs subset A = {(eI , eJ )|eI ∈ EL1 , eJ ∈ EL2 } of A′ is used as training data.

In this paper, we consider the alignment task as the classification problem that labels entity ei in GL1 using entity ej in GL2 . As said in [Lacoste-Julien et al., 2013; Zhang et al., 2015; Sun et al., 2018], this entity alignment task has the oneto-one alignment constraint: an entity can be aligned with at most one label, and a label can be assigned to at most one entity. This constraint makes difference between our problem and the common classification problem [Liu et al., 2019; Liu et al., 2017]. We define the alignment probability of using entity ej to label ei as

where sim(·) is the cosine similarity. Based on neighbor-level and relation-level knowledge information, we minimize the following negative log-likelihood function to obtain the optimal parameters with the highest alignment likelihood:

where 1 is the indicator function that equals to 1 when ej is the true label of entity ei, otherwise 0.

3.3 Training

To enable parameters of our model not only to capture the alignment likelihood information, but also model the inherent semantic knowledge of KGs. We minimize following joint objective function to performmodel training:

where λ is the positive hyper-parameter that measures the importance of LA.

In training stage, for regularization and avoiding overfitting, we apply dropout to the weight parameters W,W1,W2 and L2 regularization to embeddings of KGs. For addressing the inadequate prior alignment, we use bootstrapping strategy used in [Sun et al., 2018] that iteratively label likely alignment as training data to train our model.

Construct T and T ′ . For constructing the positive triplets set T , we deliberately exchange the entity with its counterpart in triplets to calibrate different KGs in the unified embedding space. Given an aligned entity pair (eI, eJ) ∈ A, following positive triplets are generated:

where TL1 and TL2 are respectively the positive triplets set of knowledge graphs GL1 and GL2 . Thus we have the general positive triplets set in Eq. (1) as T = TL1 ∪ TL2 ∪ TA with TA = ⋃(eI ,eJ )∈A Ta(eI , eJ ). For constructing the negative triplets set T ′ , instead of the traditional uniform negative sampling in [Bordes et al., 2013] that samples the replacer of entity from entire entity set, we adopt Nearest Neighbor Sampling strategy to limit the scope of sampling candidates. Specifically, for an entity e that will be replaced, we choose the top-s nearest neighbors of entity e in the embedding space as candidates. In this way, those entities that have low embedding similarities with entity e, are truncated and would not be sampled. Here, we use cosine similarity between integrated representation of entities to determine the top-s nearest neighbors of entities. The integrated representation of entity e is defined as β Ne(e) + (1 - β)e.

4 Experiment

4.1 Experiment Setup

Datasets. We conduct experiments on two real-world datasets DBP15K and DWY100K. DBP15K [Sun et al., 2017] is selected from the multilingual versions of DBpedia that includes entity alignment links from entities of English version to those in other languages. Three multilingual datasets DBPZH-EN (Chinese to English), DBPJA-EN (Japanese to English) and DBPFR-EN (French to English) are built in DBP15K. Each dataset contains 15 thousand reference alignment links with popular entities from English to Chinese, Japanese and French respectively. DWY100K [Sun et al., 2018] is built from three large-scale multi-lingual knowledge graph DBpedia, Wikidata and YAGO3. Two large-scale datasets, DBP-WD and DBP-YG, are respectively extracted with 100 thousand reference alignment links from the English version of DBpedia to that of Wikidata and YAGO3. Table 1 illustrates the statistics of those data sets.

Parameter settings. In our model, we set the maximum number of neighbors n as 200, and select the dimension of entity(relation) embeddings m from {50, 75, 100, 150, 200}, the learning rate η from {0.001, 0.01, 0.1}, β from {0, 0.2, 0.4, 0.6, 0.8, 1}, λ from {0.1, 0.5, 1, 1.5, 2}, µ1 from {0.5, 1, 2, 3, 4}, µ2 from {0.01, 0.1, 0.5, 0.8, 1, 1.5, 2}, γ from {0.1, 0.5, 1, 1.5, 2, 2.5}, the number of head K from {1, 2, 4, 6, 8}. For our model, the best optimal parameter configurations are m = 75, β = 0.8, λ = 1, µ1 = 1, µ2 = 0.1, γ = 2, K = 4, η = 0.01. For each positive triplet, we select 10 negative triples for training, and set the training epochs as 1000. Following BootEA [Sun et al., 2018], we used 30% of the gold standards as seed alignment while left the remaining as testing data, i.e., the latent aligned entities to discover.

4.2 Baselines

For comparative models, we select current four state-of-theart methods as baselines in our experiments.

• MTransE [Chen et al., 2017] provides the transformation for entities and relations in different languagespecific knowledge graphs.
• IPTransE [Zhu et al., 2017] incorporates relational paths and learns joint embeddings by sharing parameters based on iteratively aligning entities according to their semantic distance.
• JAPE [Sun et al., 2017] is an attribute-preserving embedding model that incorporates the relation and attribute embeddings for entity alignment.
• BootEA [Sun et al., 2018] adopts the bootstrapping process to label likely alignment as training data and edit alignment during iterations.

In this experiment, we directly copy the experiment results of baseline models reported in their papers since the same datasets are used.

4.3 Multilingual Entity Alignment

The objective of this task is to find the same semantic entities from different languages in knowledge graphs. Comparing with baseline models, we consider two measures as evaluation metrics: (1)Hits@k: the proportion of correct alignment ranked in top-k. Here Hits@1 and Hits@10 are adopted, (2) Mean Reciprocal Rank(MRR): the average of the reciprocal ranks of results. Higher Hits@k and MRR are expected, which indicates better alignment performance.

Table 2 gives the convinced results of entity alignment on DBP15K and DWY100K. From Table 2, we can observe that our model NAEA consistently outperforms all baselines on five data sets. More specifically, NAEA achieves at least 1.76% on Hits@1, 1.81% on Hits@10 and 1.3% on MRR higher performance than other models. We attribute the superiority of our model to its three advantages: (1) Our model devises knowledge embedding component for learning knowledge embeddings and entity alignment component for aligning entities ingeniously. (2) Our model incorporates the neighbor-level and relation-level information of KGs with different weights for multilingual entity alignment. (3) Our model introduces a multi-head attention mechanism to learn the neighbor-level representations by integrating neighbors' embeddings with a weighted combination.

MTransE [Chen et al., 2017] obtains the worst alignment performance because it learns the embeddings of KGs in different vector spaces, and losses information when modeling the translation between different embedding spaces. IPTransE [Zhu et al., 2017] and JAPE [Sun et al., 2017] achieve better performance than MTransE due to relational path and entity attribute information of KGs are respectively leveraged. Expect for our model NAEA, BootEA has the highest results among baselines since it uses bootstrapping process to accurately label likely alignment as training data, and address the problem caused by the small proportion of prior alignment.

Link prediction in monolingual KG. Effective link prediction can help to improve the completeness of a knowledge graph, and further helps to improve the alignment performance. Link prediction task is to predict the missing head/tail entity given a triplet (h, r, ?)/(?, r, t). Usually two evaluation metrics are used: (1) mean rank of correct entities (Mean Rank), (2) proportion of correct answers ranked in top-10

(Hit@10). In above settings, lower Mean and higher Hit@10 are expected. To testify the effectiveness of neighbor-level information for improving link prediction performance, we compare NAEA with TransE, BootEA on a monolingual KG with the representations obtained by only using individual embedding objective, such as Eq. (2). In this experiment, we randomly select monolingual triplets from DBP15KZH-EN and DBP-WD to organize the training, valid and test set according to ratio 8 : 1 : 1. The results of link prediction on datasets are shown in Table 3, which are that NAEA achieves best performance among baselines on all metrics since NAEA successfully alleviates the neighbor-level information of entities in knowledge graphs. It suggests that the neighbor-level information in KGs can provide indispensable characteristic information of knowledge graphs, and is indeed beneficial to enhance link prediction performance.

4.4 Discussion

In this section, we evaluate how different choices of parameters affect our model's performance. In the following experiments, except for the parameter being tested, the rest parameters are set as the optimal configurations.

Performance on different K. The above experimental results show that the usage of neighbor-level information through the multi-head attention mechanism is beneficial for improving alignment performance. Thus this subsection mainly explores how the performance of our model changes with different number of head K. In this experiment, we test our model with selecting K from {1, 2, 3, 4, 5, 6}. Figure 3 gives the convinced results. From this figure, we can see that:(1) NAEA has the best performance when K = 4, indicating that K = 4 setting best expresses the neighbor-level information of entities, and delivers alignment characteristics of entities in multilingual knowledge graphs. (2) The performance on all datasets begins to gradually rise to the highest point and then declines as the number of head K grows. This mainly because that too small K can not capture the richer neighbor-level information, and too large K may introduce noisy and lead to over-fitting problem.

Performance on different β. In our model, β is to weight the importance of neighbor-level information for multilingual entity alignment. To evaluate the influences of different β on alignment performance, we test our model with different β selected from {0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7. 0.8, 0.9, 1} on Hit@10 metric. We report the experimental results in Figure 4. From Figure 4, we find that:(1) Under β = 0.8 setting, NAEA can achieve the best alignment performance on all datasets, which indicates that such setting can sufficiently reflect the significance of neighbor-level information in aligning entities. (2) The performance of our model first is increasing with the growth of β and then drops with β further increases, and has the lowest accuracy when β = 0 or β = 1. It suggests that the neighbor-level and relation-level information in KGs are both necessary.

5 RelatedWork

5.1 Knowledge Representation

In recent years, a series of embedding-based methods such as [Bordes et al., 2013; Wang et al., 2014; Zhu et al., 2019] have been quickly developed for knowledge representation learning. Those methods attempt to encode entities and relations into low-dimension vector spaces while preserving KGs' properties. Among those methods, the most widely used translation-based method TransE [Bordes et al., 2013] projects entities and relations into a continuous lowdimensional vector space, and treats relation vector as the translation between head and tail entity vectors, i.e., expects eh + r ≈ et for a triplet (eh, r, et). TransE is effective and promising for knowledge graph completion, while it has issues in modeling complex relations. Therefore, later works such as [Wang et al., 2014; Lin et al., 2015a; Ji et al., 2015; Yang et al., 2015] are proposed to address the issues of TransE. Also, there exists some non-translation based models, such as [Socher et al., 2013; Nickel et al., 2016; Shi and Weninger, 2017; Dettmers et al., 2018], for learning knowledge graph embedding representations.

Besides above models that use one-step relational path to complete knowledge graphs, many models such as [Garc´ıaDur´an et al., 2015; Lin et al., 2015b] based on multi-step relational paths are also presented and achieve better performance in knowledge graph embedding.

5.2 Knowledge Alignment

Traditional knowledge alignment methods heavily rely on human efforts [Vrandecic, 2012] or well-designed hand-crafted features [Mahdisoltani et al., 2015]. Though achieving better alignment performance, they are time-consuming, laborexpensive and usually suffer from inflexible extension.

Automated knowledge alignment mainly leverages various heterogeneous information in different KGs for knowledge alignment. Some models such as [Wang et al., 2013; Suchanek et al., 2011] make use of external lexicons or Wikipedia links to address the heterogeneity between different KGs. Differing from these models, many embeddingbased models only using internal triplet-based information, were proposed for entity alignment. MTransE [Chen et al., 2017] utilizes TransE to embed language-specific KGs into separate embedding spaces, and learns the transition between different spaces. IPTransE [Zhu et al., 2017] uses PTransE [Lin et al., 2015b] and parameter sharing module to jointly encode entities and relations into a unified semantic spaces. JAPE [Sun et al., 2017] employs semantic structures and attribute correlations of KBs to embed entity and relation embedding representations into an unified embedding spaces. BootEA [Sun et al., 2018] leverages bootstrapping idea to label likely alignment as training data and edit alignment during iterations, which is state-of-the-art entity alignment model.

6 Conclusion

This paper presents a neighborhood-aware attentional representation method NAEA for multilingual knowledge graphs, which incorporates the neighbor-level and relation-level information of KGs. NAEA devises knowledge embedding component KE for learning knowledge embeddings and entity alignment component EA for aligning entities. KE introduces an attention mechanism to obtain neighbor-level representation by assigning different importance to entities' neighbors. EA discovers aligned entities based on the integration of their neighbor-level and relation-level representation with different weights. We empirically conduct experiments on multilingual entity alignment task and monolingual link prediction task with two data sets DBP15K and DB100K. The experimental results show that NAEA significantly and consistently achieves state-of-the-art performance.

Acknowledgments

This work is supported by National Key R&D Program No.2017YFB0803003, the National Natural Science Foundation of China (No.61202226), MQEPS (No 96804590) and MQRSG (No. 95109718). We thank all anonymous reviewers for their constructive comments.