Maximum Expected Likelihood Estimation for Zero-resource Neural Machine Translation

1 Introduction

Recently, neural machine translation (NMT) has achieved state-of-the-art performance on language pairs with abundant parallel corpora available [Sutskever et al., 2014; Bahdanau et al., 2014]. Nevertheless, large-scale, high-quality parallel corpora are non-existent for most language pairs [Utiyama and Isahara, 2007]. Studies reveal that it is challenging for NMT to yield satisfactory results for resource-scarce language pairs [Zoph et al., 2016; Firat et al., 2016]. Zoph et al. [2016] indicate that NMT tends to learn degenerate estimates from low-count events, which deteriorates the translation performance of NMT under small-data conditions.

As a result, developing methods to achieve zero-resource neural machine translation, in which no direct source-target parallel corpora are available, has attracted increasing attention in the community recently [Johnson et al., 2016; Firat et al., 2016]. Existing work can be roughly divided into two broad categories: multilingual and pivot-based approaches. The multilingual approaches focus on leverging multilingual parallel corpora to achieve zero-resource NMT. Johnson et al. [2016] introduce a simple solution to use a single NMT model to translate between multiple languages. Requiring no change to the model architecture, their universal model takes advantage of multilingual data to improve NMT for all languages pairs involved. Firat et al. [2016] propose a finetuning algorithm for multiway, multilingual neural machine translation that enables zero-resource machine translation. Although these multilingual methods achieve direct source-to-target translation without parallel corpora available, it is fairly difficult to learn and analyze the universal representation for multiple languages.

∗ Corresponding author: Yang Liu.

Another important direction is to introduce a third language called pivot to bridge the source and target languages, which has been widely used in zero-resource statistical machine translation [Cohn and Lapata, 2007; Wu and Wang, 2007; Utiyama and Isahara, 2007; Bertoldi et al., 2008; Habash and Hu, 2009]. Bertoldi et al. [2008] present an approach to constructing a pseudo source-target parallel corpus by translating the pivot sentences in the pivot-target corpus into the source language with the pivot-to-source model. On the other hand, one of the most representative pivot-based methods is the pivot-based translation [Utiyama and Isahara, 2007], which achieves source-to-target translation indirectly using the source-to-pivot and pivot-to-target models: a source sentence is first translated into a pivot sentence, which is then translated into a target sentence. Recently, Johnson et al. [2016] show the pivot-based translation for NMT significantly outperforms their universal NMT without incremental training. Although the pivot-based translation is a simple and effective approach to zero-resource NMT, they often suffer from the error propagation problem due to indirect modeling: mistakes made in source-to-pivot translation will be propagated to pivot-to-target translation.

In this work, we achieve an extension of the pivot-based method using pseudo parallel corpus [Bertoldi et al., 2008] for zero-resource neural machine translation. We propose an approach to directly training the source-to-target translation model via maximum expected likelihood estimation. Our training objective is to maximize the expectation with respect to a pivot-to-source translation model for the intended sourceto-target model on a pivot-target parallel corpus. The assumption underlying our idea is that if a pivot sentence z and a target sentence y constitute a parallel sentence pair, the source translation x of the pivot sentence should also be translationally equivalent to y. Therefore, our approach involves two models: the pivot-to-source model that provides an artificial parallel corpus and the source-to-target model that is learned on the artificial data. As it is intractable to enumerate all sentences in the source language, we also propose two strategies to connect the pivot-to-source and source-to-target models. Experiments on two zero-resource translation tasks demonstrate that the proposed approach yields substantial gains over the baseline methods.

2 Background

2.1 MaximumLikelihood Estimation

Given a source-language sentence x and a target-language sentence y, we use P (y|x; θx→y) to denote a source-to-target NMT model [Bahdanau et al., 2014].

For resource-rich language pairs, a source-target parallel corpus Dx,y = {〈x(k) , y(k) 〉}Kk=1 is available to train the source-to-target NMT model. The standard training objective in this scenario is to maximize the log-likelihood of the training data:ˆ

where the log-likelihood is defined as

2.2 Pivot-based Translation

For zero-resource language pairs, parallel corpora are usually not readily available. As a result, previous work has endeavored to leverage a third language called pivot to bridge the source and target languages. Let z be a pivot-language sentence. As shown in Figure 1, source-to-target translation can be modeled indirectly by cascading two sub-models:

The pivot-based translation assumes that a source-pivot parallel corpus Dx,z = {〈x(m) , z(m) 〉}Mm=1 and a pivot-target parallel corpus Dz,y = {〈z(n) , y(n) 〉}Nn=1 are available. As a result, the source-to-pivot and pivot-to-target models can be trained separately:

Then, the two models can be used to perform source-totarget translation in two steps:

One drawback of this approach is that the translation quality of source-to-target translation heavily depends on the selection of ˆz, which often fails to retain exactly the same information with the original source sentence and thus leads to severe cascaded translation errors.

3 Approach

3.1 MaximumExpected Likelihood Estimation

As shown in Figure 1, our idea is to maximize the expectation with respect to a pivot-to-source translation model for the intended source-to-target model on a pivot-target parallel corpus:

where the training objective is defined as

Note that the pivot-to-source model is pre-trained on the source-pivot corpus:

where the log-likelihood is defined as

We refer to Eq. (8) and (9) as maximum expected likelihood estimation.

Maximum expected likelihood estimation is an extension of standard maximum likelihood estimation for learning the intended distribution P(y|x; θx→y) by introducing a data generating distribution P(x|z; θz→x). This new training criterion for zero-resource NMT is based on the following assumption: since 〈z(n) , y(n) 〉 is a bilingual sentence pair, the source translation of z(n) (i.e., x) should also be translationally equivalent to y(n) .

However, as calculating the expectation in Eq. (9) needs to enumerate all possible source sentences, it is intractable to optimize source-to-target model parameters due to the exponential search space. As a result, we propose two approximation methods to address this problem.

3.2 Approximation

Using Single Word Embeddings

Let X(z) be the set of all possible source translations of a pivot sentence z. A standard solution is to approximate the full search space with a sampled subset:

where S(z) ⊂ X(z) is a sampled subset.

Note that the intended distribution P(y(n) |x(s) ; θx→y) actually takes the concatenation of word embeddings of x(s) as input. Let x(s) = x1, . . . , xt, . . . , xT be a sampled source translation with T words for z(n) . We use e(xt) ∈ Rd×1 to represent the word embedding of xt. Therefore, the vector representation of x(s) is given by

Therefore, Eq. (12) can be equivalently written as

Although approximation with sampling proves to achieve a reasonable balance between effectiveness and efficiency [Shen et al., 2016], the sampled space is still restricted to a limited number of candidate translations. In addition, only a small fraction of words in the entire vocabulary are included in the samples. As a result, our approach also potentially faces the error propagation problem: sampling mistakes will affect the estimation of intended model parameters.

Using Expected Word Embeddings

Inspired by [Koˇcisk´y et al., 2016], we propose to use expected word embeddings rather than single word embeddings to circumvent this drawback. Given a sampled source translation x(s) ∈ S(z(n) ), at each time step in the decoder of the pivotto-source model, an expected word embedding for the t-th source word xt is calculated as

where Vx is the vocabulary of the source language.

As a result, provided with a sampled source sentence x(s) , the expected vector representation of the source sentence can be approximated with the concatenation of expected word embeddings, which is defined as

Note that E(x(s) , z(n) , Vx, ˆ θz→x) depends on the selection of x(s) .

As the expected word embeddings consider the entire vocabulary, we can leverage the expected word embeddings to implicitly represent the full search space X(z(n) ) approximately:

Besides taking the full vocabulary into consideration, another important advantage of using expected word embeddings over the single embeddings counterpart is that pivotto-source and source-to-target models are connected more closely. Consider Eq. (14). It is clear that the partial derivatives with respect to the parameters of the two models are independent of each other. In contrast, using expected word embeddings explicitly makes the calculation of partial derivatives of one model dependent on another model since it allows the error from the source-to-target model to be backpropagated to the pivot-to-source model through the expected word embeddings.

Therefore, we propose to use joint training to enable the interaction between pivot-to-source and source-to-target models:ˆ

where the new training objective is given by

4 Experiments

4.1 Setup

Data Preparation

We evaluate our approach on two zero-resource translation tasks:

1. Spanish-English-French: Spanish as the source language, English as the pivot language, and French as the target language;
2. German-English-French: German as the source language, English as the pivot language, and French as the target language.

We use Spanish-English, German-English and EnglishFrench parallel corpora from the Europarl dataset. For each language pair, we retain sentence pairs with no more than 50 words. We also split the overlapping part of pivot sentences in the source-pivot and pivot-target corpora into two separate parts with equal size, which are then merged with the non-overlapping parts of the source-pivot and pivot-target corpora, respectively. Removing overlapped data ensures that none of the source-target parallel sentence pairs is present in our training data. All sentences are tokenized by the tokenize.perl script [Koehn et al., 2007].

For Spanish-English, the training data contains 850K sentence pairs with 23.23M Spanish words and 21.44M English words. For German-English, the training data consists of 840K sentence pairs with 20.88M German words and 21.91M English words. They share the same English-French corpus which has 900K sentence pairs with 22.56M English words and 25.00M French words. The shared task 2006 datasets are used as development and test sets. The evaluation metric is case-insensitive BLEU [Papineni et al., 2002], calculated by the multi-bleu.perl script.

Baseline Methods

We compare our approach with the following baseline methods:

1. PSEUDO [Bertoldi et al., 2008]: a method for building pseudo parallel corpora adapted for NMT.
2. PIVOT [Utiyama and Isahara, 2007]: a pivot-based translation method adapted for NMT.

For the PSEUDO method, we first train a pivot-to-source NMT model P(x|z; ˆ θz→x) on the source-pivot parallel corpus. Then, the pivot sentences {z(n) }Nn=1 in the pivot-target parallel corpus Dz,y = {〈z(n) , y(n) 〉}Nn=1 are translated into source sentences {˜x(n) }Nn=1 using the pivot-to-source NMT model P(x|z; ˆ θz→x). These source sentences {˜x(n) }Nn=1 and the original target sentences {y(n) }Nn=1 in the pivot-target parallel corpus constitute a pseudo source-target parallel corpus D˜x,y = {〈˜x(n) , y(n) 〉}Nn=1 , which can be used to train the source-to-target model P(y|x; θx→y). The PSEUDO method can be treated as a special case of maximum expected likelihood estimation since only the candidate translation with the highest probability in the full space is considered.

For the PIVOT method, we first train source-to-pivot and pivot-to-target NMT models separately on source-pivot and pivot-target parallel corpora. In decoding, a source sentence is first translated into a pivot sentence using the source-topivot model, which is then translated to a target sentence using the pivot-to-target model. Please refer to Section 2.2 for more details. This approach serves as the baseline in previous work on zero-source neural machine translation task [Johnson et al., 2016; Firat et al., 2016]. It is worth emphasizing that the PIVOT method is a very strong baseline as Johnson et al. [2016] show that it yields much higher BLEU scores than their universal NMT model without incremental training and the direct source-to-target translation model proposed in [Firat et al., 2016] alone also performs worse than the PIVOT method.

We implement all methods on top of the state-of-the-art open-source NMT system GROUNDHOG [Bahdanau et al., 2014]. All neural translation models use the default setting of network hyper-parameters of GROUNDHOG.

Training Details

Our joint training approach requires the parameters of multiple translation models to be updated jointly. In practice, we find that it is extremely slow for the models to converge from random initialization. Inspired by previous work on training multi-layered perceptrons [Bengio et al., 2007], we use the weights of the pivot-to-source model trained independently and the source-to-target model trained with the PSEUDO method to initialize joint training. For fair comparison, we also initialize the source-to-target model in our independent training approach with the model obtained by the PSEUDO method. Although each mini-batch takes more time to update than standard MLE training, our approaches converge very fast thanks to the aforementioned initialization techniques.

Each time the sentence z(n) is selected in a mini-batch, we only randomly sample one sentence x in consideration of the limited GPU memory. Note that a set of different sentences will still be sampled for each z(n) as z(n) is usually selected in mini-batches multiple times. This is important since it can effectively lower the variance of approximation. For the single embedding approach, we find the probability weight P(x|z; ˆ θz→x) is usually very small, making the training extremely slow. Therefore, in practice we instead take its q-th root q√P(x|z; ˆ θz→x) and set q = 10 for speed-up.

4.2 Comparison with PSEUDO and PIVOT

Table 1 shows the comparison of our approach with PSEUDO and PIVOT on the Europarl corpus. PSEUDO achieves a BLEU score of 28.27 on the Spanish-French task and 19.92 on German-French, which are much lower than those of other approaches. This is because PSEUDO only uses the source translation with the highest probability to build the pseudo parallel corpus, which may also cause severe error propagation problem in training: mistakes made in the pivot-to-source translation affect the quality of pseudo parallel corpus.

The PIVOT approach significantly outperforms PSEUDO for both translation tasks. Although PIVOT also faces the error propagation problem in decoding, the source-pivot and pivot-target parallel corpora used to train source-to-pivot and pivot-to-target NMT models are supposed to be clean. In contrast, the source part of the pseudo parallel corpus inevitably contains much noise due to translation errors.

Our approach significantly outperforms both PSEUDO and PIVOT on both tasks. Note that PSEUDO can be considered as a special case of maximum expected likelihood estimation: only the candidate with the highest probability in the full space is used. This finding suggests that it is important to take multiple candidates into consideration in maximum expected likelihood estimation. As using expected embeddings is capable of exploiting the full vocabulary, it outperforms using single word embeddings significantly.

Another important finding is that joint training leads to significant improvements over independent training thanks to the interaction between the pivot-to-source and source-to-target models during training. Table 2 shows that the interaction not only improves source-to-target translation, but also enhances pivot-to-source translation. The BLEU scores increase by over 2% for both tasks.

4.3 Effect of Bidirectional Training

From the perspective of pivot-based approaches, our approach can be seen as indirectly modeling pivot-to-target translation via pivot-to-source and source-to-target translation models on the pivot-target parallel corpus Dz,y = {〈z(n) , y(n) 〉}Mm=1 :

An alternative is to make use of the source-pivot parallel corpus Dx,z = {〈x(n) , z(n) 〉}Nn=1 :

For convenience, we refer to Eq. (19) as the z → x → y direction and Eq. (20) as the x → y → z direction.

The joint training objective for the x → y → z direction is given by

As bidirectional information has proven to be complementary for the learning of mapping between two languages [Och and Ney, 2002; Li and Jurafsky, 2016], the two directions can be combined to perform bidirectional training:

where λ is a hyper-parameter that balance the preference between two directions. We set λ = 0.1 in our experiments.

Table 3 shows the effect of bidirectional training. "unidirectional" denotes the z → x → y direction and "bidirectional" denotes combining the the z → x → y and x → y → z directions. Combining the two directions leads to significant improvements (p < 0.01). Note that training alone in the x → y → z direction results in very low BLEU scores. One possible reason is that the training objective Jjoint(θx→y, θy→z) serves as more like a regularization item, in the sense that it can provide some useful information to further improve the model's performance, but it alone lacks the capability to provide complete and effective supervision.

4.4 Comparison with ORACLE

Table 4 compares our best approach with ORACLE that uses direct source-target parallel corpora. We observe that our approach outperforms ORACLE with 100K and 1M parallel sentences. This is very encouraging since our approach only uses three source-pivot and pivot-target corpora with around 850K parallel sentences.

5 RelatedWork

Recently, zero-resource neural machine translation has received much attention in the community. Firat et al. [2016] pre-train multi-way multilingual model and then fine-tune it with pseudo parallel data generated by the model. However, the use of pseudo parallel data as supervision unavoidably suffers from the error propagation problem. This is probably why their one-to-one direct translation model performs worse than the pivot-based translation strategy. Google's multilingual neural machine translation system uses a single NMT

model to translate between multiple languages, naturally enabling zero-resource translation [Johnson et al., 2016].

A wide variety of approaches have been proposed for zeroresource or low-resource translation in conventional SMT systems. Wu et al. [2007] and Cohn et al. [2007] use source-to-pivot and pivot-to-target translation models to induce a new source-to-target phrase table, on which a sourceto-target translation model is built. Bertoldi et al. [2008] exploit existing models to build a pseudo source-target parallel corpus, from which a source-to-target model can be trained. These methods fall into the broad category of directly constructing a source-to-target translation model. Utiyama and Isahara [2007] compare two translation strategies, namely phrase translation and sentence translation. The former is similar to the work in [Wu and Wang, 2007], while the latter is the pivot-based translation which first translates the source sentence to the pivot sentence and then to the target sentence. Cheng et al. [2016] improve the pivot-based translation by jointly training the source-to-pivot and pivot-to-target translation models. Nakayama and Nishida [2016] perform zeroresource machine translation using multimedia as the pivot.

Our work is in spirit close to Bertoldi et al. [2008] since both assume that if a pivot sentence z and a target sentence y constitute a parallel sentence pair, the source translation x of the pivot sentence z should also be translationally equivalent to y. However, we formally define the expected training objective and adapt it to NMT, thus enabling the novel expected embedding and joint training strategies.

6 Conclusion

We have presented a method for training neural machine translation models for zero-resource language pairs using maximum expected likelihood estimation. The central idea is to leverage a data generating distribution to provide pseudo parallel corpora for training the intended distribution. Experiments on the Europarl corpus show that our approach achieves significant improvements over the pivot-based translation approach.

Acknowledgments

This work was done while Hao Zheng was visiting Tsinghua University. This work is supported by the National Natural Science Foundation of China (No.61522204), the 863 Program (2015AA015407), and the National Natural Science Foundation of China (No.61432013). This research is also supported by the Singapore National Research Foundation under its International Research Centre@Singapore Funding Initiative and administered by the IDM Programme.