Knowledge Graphs Enhanced Neural Machine Translation

1 Introduction

Neural machine translation (NMT) based on the encoderdecoder architecture becomes a new state-of-the-art approach due to its distributed representation and end-to-end learning [Luong et al., 2015; Vaswani et al., 2017].

During translation, entities in a sentence play an important role, and their correct translation can heavily affect the whole translation quality of this sentence. Therefore, due to the importance of the entities, various methods are proposed to improve their translation [Zhang and Zong, 2016; Dinu et al., 2019; Ugawa et al., 2018; Wang et al., 2019]. Among them, a kind of methods aim to incorporate the knowledge graphs (KGs) to improve the entity translation.

In many languages and domains, people construct various large-scale KGs to organize structured knowledge on entities. Meanwhile, some studies incorporate KGs into NMT to enhance the semantic representation of the entities in sentence pairs and improve the translation [Shi et al., 2016; Lu et al., 2018; Moussallem et al., 2019]. However, these studies have a drawback that they only focus on the entities that both appear in KGs and training sentence pair dataset

(We denote these entities as K+D entities 1 ). Actually, besides these K+D entities, KGs also contain many entities which do not appear in the training sentence pair dataset (We denote these entities as K-D entities, whose formal definition can be found in Section 3). While these K-D entities have been ignored in previous studies.

In this paper we think that these K-D entities seriously harm the translation quality while KGs could alleviate this problem. Fig. 1 shows an example that assuming two translation pairs can be extracted from Chinese-to-English parallel sentence pairs, i.e., "asipilin-aspirin" and "yaopindrug". Meanwhile, the source entity "yixianshuiyangsuan" is a K-D entity and does not appear in the parallel sentence pairs. While we can induce that this entity may be translated into "aspirin", since the source triple "(asipilin, alias, yixianshuiyangsuan)" indicates that "yixianshuiyangsuan" is another name for "asipilin".

Therefore, in this paper we propose an effective method incorporating non-parallel source and target KGs into the NMT system. With the help of KGs, the proposed method could enable the NMT to learn new entity translation pairs containing the K-D entities. More specifically, the proposed method contains three steps: 1) Bilingual K-D entities induction: in this step we first extract the seed pairs from the phrase translation table. We then transform the source and target KGs into a unified semantic space by minimizing the distance between source and target entities in the seed pairs. We finally induce the translation results of the K-D entities under this semantic space. 2) Pseudo parallel sentence pairs generation: we generate adequate pseudo parallel sentence pairs containing the induced entity pairs. 3) Joint training: in this step we jointly train the NMT model by the original and pseudo senescent pairs, enabling NMT to learn the mapping between source and target entities in induced translation pairs. The extensive experiments on Chinese-to-English and English-toJapanese translation tasks demonstrate that our method significantly outperforms the strong baseline models in translation quality, especially in handling the induced K-D entities. We make the following contributions in this paper:

1 K denotes KGs and D denotes the sentence pair dataset.

• We propose a method to incorporate the non-parallel KGs into NMT model.
• We design a novel approach to induce the translation results of the K-D entities with KGs, generate the pseudo parallel sentence pairs and promote NMT to make better predictions of K-D entities.

2 Background Knowledge

2.1 Neural Machine Translation

To date there are various NMT frameworks [Luong et al., 2015; Vaswani et al., 2017]. Among them, self-attention based framework (called as Transformer) achieves the stateof-the-art translation performance.

Transformer follows the encoder-decoder architecture, where the encoder transforms a source sentence X into a set of context vectors C. The decoder generates the target sentence Y from the context vectors C. Given a parallel sentence pair dataset D = {(X, Y )}, where X is the source sentence and Y is the target sentence, the loss function can be defined as:

More details can be found in [Vaswani et al., 2017].

2.2 Knowledge Embedding

The current KGs are always organized in the form of triples (h, r, t), where h and t indicate head and tail entities, and r denotes the relation between h and t, e.g., (aspirin, type, drug). Recently, various approaches are proposed to embed both entities and relations into a continuous low-dimensional space, such as TransE [Bordes et al., 2013], TransH [Wang et al., 2014] and TransR [Lin et al., 2015]. Here we take TransE as an example to introduce the embedding methods.

TransE projects both relations and entities into the same continuous low-dimensional vector space E. The goal of TransE is to make E(h) + E(r) ≈ E(t). To achieve this, the score function is defined as:

where E(h), E(r) and E(t) are the embeddings for h, r and t, respectively. ||.|| means l1 or l2 norm. More details can be found in [Bordes et al., 2013].

3 ProblemDefinition

In this paper we use the following three data resources to train a NMT model θ:

1. Parallel Sentence Pairs D = {(X, Y )}, where X denotes the source sentence. Y denotes the target sentence.
2. Source KG KGs = {(hs, rs, ts)}, where hs, ts and rs denote the head entity, tail entity and relation in source language, respectively.
3. Target KG KGt = {(ht, rt, tt)}, where ht, tt and rt denote the head entity, tail entity and relation in target language, respectively.

Since the parallel KGs are difficult to obtain, in this paper KGs and KGt are not parallel, Meanwhile, we assume that KGs and KGt contain many entities which do not appear in the parallel sentence pairs D. We called these entities as K-D entities. Formally, K-D entities set O can defined by

where Oes and Oes denote the K-D source entity and target entity, respectively.

We think that although sentence pairs D may contain little translation knowledge on these K-D entities, the KGs could help to induce their translation results. Therefore, our goal in this paper is to improve the translation quality of these K-D entities with the help of KGs and KGt.

4 Method Descriptions

Fig. 2 shows the framework of our proposed method, which contains three steps: i) bilingual K-D entities induction, 2) pseudo sentence pairs generation and 3) joint training. Next we will introduce each step in the following each subsection.

4.1 Bilingual K-D Entities Induction

In this step we hope to induce the translation results of K-D entities. To achieve this goal, our main idea is to transform the source and target KGs into a unified semantic space, and then induce the translation results of these entities under this semantic space.

Specifically, Algorithm 1 shows our bilingual K-D entities induction method, where the method first needs four preparations (line 1-4). We first represent KGs and KGt into the entity embedding Es ∈ Rn×d and Et ∈ Rm×d , respectively (line 1-2). We then extract the phrase translation pairs P = {(s, t, p(s,t))} from parallel sentence pairs D by statistical method 2 , where s is the source phrase, t is the source phrase, p(s,t) is the translation probability (line 3). The last preparation is extracting K-D entity set O by Eq. (3). In the example of Fig. 2, there are three K-D entities "yixianshuiyangsuan", "purexitong" and "paracetamol", where the first two are K-D source entities and the last one is K-D target entity.

With above preparations, we now need to construct the seed pair set S (line 5-8). If there is a phrase translation pair

2 http://www.statmt.org/moses/

(s, t, p(s,t)) whose source phrase s belongs to KGs and target phrase t belongs to KGt, we add this phrase pair into seed pair set S. In the example of Fig. 2, two phrase pairs "(yaopin, drug, 0.4)" and "(asipilin, aspirin, 0.9)" are selected into the seed pairs.

The derived Es ∈ Rn×d and Et ∈ Rm×d are learned separately, making them be in different semantic spaces. Now our task is to transform Es and Et into a unified semantic space. Inspired by [Zhu et al., 2017], we conduct a linear transformation and make the source entities and target entities in seed pairs as close as possible. Specifically, given a seed pair (s, t, p(s,t)), we define a transformation matrix W, so that WEs(s) ≈ Et(t). Futhur more, we take the translation probability p(s,t) into consideration. If a seed pair with a larger probability, this seed pair has a larger weigh in the loss function. Therefore, the loss function can be defined as Eq. (4) (line 9).

The final task is to induce the translation results of K-D entities (line 10-17). Given a K-D source entity Oes ∈ O (line 10), we traverse each target entity et in KGt (line 11). If the distance between Oes and et is lower than the pre-defined threshold δ (line 12), we treat pair (Oes, et) as a new induced translation pair and add it into induction set I (line 13). Similarly, given a K-D target entity Oet ∈ O (line 14), we traverse each source entity es in KGs (line 15). We also add the

Algorithm 1 Bilingual K-D Entities Induction Method

Input:

Parallel sentence pairs D; source KGKGs; target KGKGt; pre-defined hyper-parameter δ

Output:

Bilingual K-D entities induction set I Algorithm:

• 1: represent KGs into embedding Es ∈ Rm×d
• 2: represent KGt into embedding Et ∈ Rn×d
• 3: extract the phrase translation pairs P = {(s, t, p(s,t))}, where s is the source phrase, t is the source phrase, p(s,t) is the translation probability.
• 4: extract K-D entity set O by Eq. (3)
• 5: initialize the seed set S = {}
• 6: for each phrase pair (s, t, p(s,t)) ∈ P do
• 7: if s ∈ KGs and t ∈ KGt then
• 8: add the phrase pair (s, t, p(s,t)) into S
• 9: learning the transform matrix W to represent Es and Et into a unified semantic space with seed set S by minimizing the following loss function

where (s, t, p(s,t)) is the seed pair in S. Es(s) is the embedding for s and Et(t) is the embedding for t.

10: for each K-D source entity Oes ∈ O do 11: for each target entity et ∈ KGt do 12: if ||WEs(Oes) -Et(et)|| < δ then 13: adding the induced pair (Oes, et) into I 14: for each target K-D entity Oet ∈ O do 15: for each source entities es ∈ KGs do 16: if ||WEs(es) -Et(Oet)|| < δ then 17: add the induced pair (es,Oet) into I return I

pair (es,Oet) into induction set I, if the distance between es and Oet is lower than the pre-define threshold δ (line 16-17). In the example of Fig. 2, we induced two new pairs: "(yixianshuiyangsuan, aspirin)" and "(purexitong, paracetamol)". Now the set I contains all new induced translation pairs.

4.2 Pseudo Sentence Pairs Generation

Now our goal is to generate the sentence pairs containing the induced entities pairs. The main idea is to transfer the context of seed pairs to the induced pairs which are close to this seed pairs. Specifically, if the distance between an induced pair (is, it) ∈ I and a seed pair (ss, st) ∈ S is lower than a predefined hyper-parameter λ as follows:

we hope to transfer the context of seed pair (ss, st) to that of induced pair (is, it). To achieve this goal, we first retrieve D and get all sentence pairs {(Xs, Ys)} containing the seed pair (ss, st). Then we replace (ss, ts) in (Xs, Ys) by the induced pair (is, it) and get the pseudo sentence pair (Xp, Yp). Now the pseudo sentence pair (Xp, Yp) contains the induced pair (is, it).

In the example of Fig. 2, assuming that both induced pairs "(yixianshuiyangsuan, aspirin)" and "(purexitong, paracetamol)" are close to the seed pair "(asipilin, aspirin)", we replace "(asipilin, aspirin)" by these two induced pairs and get the pseudo sentence pairs as shown in middle part of Fig. 2.

4.3 Joint Training

The final task is to train the NMT model θ with the original parallel sentence pairs D and pseudo parallel sentence pairs Dp. Our experiments (Section 6) show that the number of pseudo sentence pairs Dp is significantly less than that of original sentence pairs D. To overcome this imbalance problem, we first over-sample the pseudo sentence pairs Dp by n times and design the loss function by

where the former one is the loss from the original data D, and the later one shows the loss from the over-sampled pseudo data Dp.

5 Experimental Setting

We test the proposed method on Chinese-to-English (CN⇒EN) and English-to-Japanese (EN⇒JA) translation tasks. The CN⇒EN parallel sentence pairs are extracted from LDC corpus, which contains 2.01M sentence pairs. On CN⇒EN task, we utilize three different KGs: i) Medical KG, where the source KG contains 0.38M triples 3 and the target KG contains 0.23M triples, which are filtered from YAGO [Suchanek et al., 2007]. We construct 2000 medical sentence pairs as development set and 2000 medical sentence pairs as test set. ii) TourismKG, where the source KG contains 0.16M triples. The target KG contains 0.28M triples, which are also filtered from YAGO 4 . We also construct 2000 sentence pairs on tourism as development set, and 2000 other sentence pairs as test set. iii) General KG, where the source KG is randomly selected from CN-DBpedia 5 and the target KG is randomly selected from YAGO. We choose the NIST 03 as development set and NIST 04-06 as test set. We use KFTT dataset as EN⇒JA parallel sentence pairs. The source and target KGs are DBP15K from [Sun et al., 2017]. The statistics of training pairs and KGs are shown in Table 1.

We implement the NMT model based on the THUMT toolkit 6 and the knowledge embedding method based on the openKE toolkit 7 . We use the "base" version parameters of the Transformer model. On all translation tasks, we use the BPE [Sennrich et al., 2016] method to merge 30K steps. We evaluate the final translation quality with case-insensitive BLEU for all translation tasks.

In this method, we compare the following NMT systems:

3 http://www.openkg.cn/dataset/symptom-in-chinese

4 The target KGs in Medical KG and Tourist KG are filtered by retaining the triples which contain the pre-defined key words.

5 http://www.openkg.cn/dataset/cndbpedia

6 https://github.com/THUNLP-MT/THUMT

7 https://github.com/thunlp/OpenKE

1. RNMT: The baseline NMT system using two LSTM layers as encoder and decoder [Luong et al., 2015].
2. Transformer: The state-of-the-art NMT system with self-attention mechanism.
3. Transformer+RC: This is the method which incorporates KGs by adding the Relation Constraint between the entities in the sentences [Lu et al., 2018], whose goal is to get a better representation of K+D entities in sentence pairs.
4. Transformer/RNMT+KG: This is our proposed KGs enhanced NMT model on the basis of Transformer and RNMT, where we set the hyper-parameter δ (Algorithm 1) by 0.45 (Medical), 0.47 (Tourism), 0.39 (General) and 0.43 (DBP15K) and λ (Section 4.2) by 0.86 (Medical), 0.82 (Tourism), 0.73 (General) and 0.82 (DBP15K). The oversample time n (Section 4.3) is set to 4 (Medical), 3 (Tourism), 2 (General) and 3 (DBP15K), respectively. All these hyperparameters are fine-tuned in development set.

6 Experimental Results

6.1 Translation Results

Results on RNMT model. Table 2 lists the main translation results of CN⇒EN and EN⇒JA translation tasks. We first compare our method with RNMT. Comparing the row 1 and row 4-6, the proposed RNMT+KG can improve over RNMT on all test sets. Specifically, when utilizing the medical, tourism and general KG, the proposed method can exceed RNMT by 1.29 (12.54 vs. 11.25), 0.88 (12.77 vs. 11.89) and 0.55 (41.89 vs. 41.34) BLEU points, respectively. Meanwhile, on EN⇒JA translation task, the improvement can reach 0.48 BLEU points (27.91 vs. 27.43).

Results on Transformer model. We conduct experiments to evaluate proposed method on the basis of Transformer. As shown in row 2 and row 7-9, our method can also improve the translation quality on Transformer, where with the help of these three KGs, the improvements can reach 1.12 (15.69 vs. 14.57), 0.90 (14.88 vs. 13.98) and 0.51 (44.91 vs. 44.40) BLEU points, respectively. Besides, on EN⇒JA translation task, the proposed Transformer+KG can outperform Transformer by 0.60 BLEU points (30.10 vs. 29.50).

Results on different embedding methods. We are also interested the results when we utilize different knowledge embedding methods. Here, we test the following three knowledge embedding methods: TransE, TransD and TransR. From the results (row 4-9), we can see that on all tasks, these three knowledge embedding methods can achieve similar BLEU scores.

Transformer+RC vs. Our method. We also compare the proposed method with Transformer+RC (row 3). The results show that our proposed method (row 7-9) can outperform Transformer+RC by 0.90 (15.69 vs. 14.79), 0.77 (14.88 vs. 14.11), 0.08 (44.91 vs. 44.83) and 0.27 (30.10 vs. 29.83) BLEU points, respectively. The results show the advantage of proposed methods. More importantly, on the basis of Transformer+RC, our method (row 10) can further improve the translation quality, indicating that Transformer+RC still faces the problem of K-D entities, and our method can alleviate this problem.

6.2 Effect of Hyper-paramters

In Algorithm 1, we set a pre-defined hyper-parameter δ to determine the bilingual induced pairs. Table 3 shows the BLEU scores with different δ (medical KG). We can see that the BLEU score is the largest when δ = 0.45. When δ exceeds 0.45, the BLEU score (dev) decreases from 15.96 to 14.94 BLEU points.

Meanwhile we are also curious about the precision of induced bilingual K-D entities. Thus we randomly select 300 induced bilingual entity translation pairs under different δ and analyze the correct ratio manually. The results are also reported in Table 3 (Column Precison). From the results, we can see that with the increasing of hyper-parameter δ, more K-D entity translation pairs can be induced, while the precision decreases from 43.1% to 13.7%. The results show that it is necessary to strike a balance between the number of K-D entity translation pairs and the precision of that.

In section 4.2, we set a pre-defined hyper-parameter λ to generate pseudo sentence pairs. Fig. 3 shows the results (medical KG), where x axis denotes hyper-parameter λ, y axis denotes the BLEU score of development and test set. The number in the figure denotes the number of pseudo sentence pairs. We can see from the results that with the increasing of hyper-parameter λ, more pseudo sentence pairs can be generated. The BLEU score (dev) becomes the largest when λ = 0.86. We think the reason is that when λ becomes too large, pseudo sentence pairs may contain more noises, which consequently harm the final translation quality.

6.3 Analysis of K-D Entities

In this paper we aim at enhancing the K-D entities of NMT with KGs. Thus, we also analyze the results on K-D entities of proposed methods. Specifically, the analysis is conducted on sentence level and entity level.

On sentence level analysis, we divide the test sentences into two different parts: i) the sentence with K-D entities (sent w K-D) and ii) the sentences without K-D entities (sent w/o K-D). Table 4 reports results. From the results, we can see that our proposed method has little effect on the sentences without K-D entities. While it can significantly improve the sentence with K-D entities from 9.96 to 11.75 BLEU points (RNMT) and from 13.46 to 15.15 BLEU points (Transformer), respectively. The results indicate that our proposed method can produce better translation results on the sentences containing the K-D entities.

We also analyze the results of induced K-D entities on word level. Specifically, we randomly selected 300 sentences (medical KG), which contains 162 K-D entities (267 times) and 72 K+D entities (96 times). And we count the following three numbers: 1) correct ratio (times) of induced K-D entities; 2) correct ratio (times) of un-induced K-D entities; and 3) correct ratio (times) of K+D entities. The statistics are reported in Table 5. From the result, we can see that Transformer+RC could improve the translation correct ratio (times) of K+D entities from 36.5% (35) to 43.8% (42). And our method is most effective to induced K-D entities, which improves the translation correct ratio (times) from 21.5% (31) to 31.3% (45). More importantly, when combining Transformer+RC and our method, RC+Ours can both improve the induced K-D and K+D to 31.9% (46) and 46.9% (45), respectively, which shows that our method and Transformer+RC are complementary.

Fig. 4 shows the example that the proposed method could improve the translation of induced K-D entities. In the example, the mentioned K-D entity "yixianshuiyangsuan" is totally translated into a wrong target phrase by Transformer. While the proposed Transformer+KG could overcome this mistake and produce the correct translation result "asipirin".

7 RelatedWork

In this paper we aim at incorporating the KGs into NMT to improve the K-D entities. The related work can be divided into the following three categories :

Knowledge Graph in NMT. The early studies using the knowledge graph or semantic web are conducted in statisti-

Source: 不 建议 使用 乙酰水杨酸 和 布洛芬 。

Pinyin: bu jianyi shiyong yixianshuiyangsuan he buluofen 。

Target： aspirin and ibuprofen are not recommended .

NMT:

it is not recommended to use the book of agriculture ,

yaoyang and braufen .

NMT+KG: it is not recommended to use aspirin and braufen .

Figure 4: An example to show that the proposed method could improve the induced K-D entities.

cal machine translation framework [Moussallem et al., 2018]. Recently, several studies incorporate the KGs into NMT, where [Shi et al., 2016] proposes a knowledge-based semantic embedding for NMT by extracting the important semantic vectors with KGs. [Lu et al., 2018] incorporates KGs by adding the relation constraint between the entities in the sentences. [Moussallem et al., 2019] exploits the entity linking to disambiguate the entities found in a sentence. The biggest difference between our method and previous methods is that previous studies tend to enhance the semantic representation of K+D entities in sentence pairs. While the goal of our method is to improve the translation of K-D entities.

Cross-lingual Knowledge Alignment. Our induced method is inspired by the work of knowledge alignment [Hao et al., 2016; Zhu et al., 2017; Chen et al., 2017; Wang et al., 2018; Cao et al., 2019; Wu et al., 2019]. The goal of knowledge alignment is to find entities in different KGs that refer to the same meaning. Different from these studies, our method aims to improve the translation quality of K-D entities of NMT with KGs.

Incorporating Bilingual Lexicons or Phrases. Our method is also inspired by the studies of incorporating bilingual lexicons or Phrases into NMT [Zhang and Zong, 2016; Hasler et al., 2018; Ugawa et al., 2018; Zhao et al., 2018a; Zhao et al., 2018b; Dinu et al., 2019; Huck et al., 2019]. The difference between our method and these studies that is they utilize the external bilingual lexicons to improve the lexical translation, while we incorporate the KGs to improve the K-D entities.

8 Conclusion

To address K-D entities in NMT, we propose a knowledge graph enhanced NMT method. We first induce the translation results of the K-D entities by utilizing non-parallel KGs, then generate pseudo parallel sentence pairs, and finally jointly train the NMT model. The extensive experiments on Chinese-to-English and English-to-Japanese tasks demonstrate that our method significantly outperforms the baseline models in translation quality, especially in handling the induced K-D entities.

Acknowledgments

The research work described in this paper has been supported by the Natural Science Foundation of China under grant No. U1836221 and 61673380. The research work in this paper has also been Sponsored by CCF-Tencent Open Fund.