ME-MD: An Effective Framework for Neural Machine Translation with Multiple Encoders and Decoders

1 Introduction

The encoder-decoder framework [Kalchbrenner and Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014] is widely exploited for Neural Machine Translation. In this framework, an encoder compresses the source sentence to a distribution representation and a decoder generates target words one by one regarding the source representation. Compared with the Statistical Machine Translation (SMT), NMT models the translation knowledge through training a single network in the end-to-end style and gets ride of constructing several sub-components separately.

Plenty of approaches are proposed to enhance the NMT performance, such as attention mechanisms [Bahdanau et al., 2015; Luong et al., 2015a; Meng et al., 2016], effective connections [Zhou et al., 2016; Wu et al., 2016], coverage models [Tu et al., 2016], addressing rare words [Jean et al., 2015; Luong et al., 2015b; Sennrich et al., 2016; Chung et al., 2016], joint training [Dong et al., 2015; Luong et al., 2016;

Firat et al., 2016; Zoph and Knight, 2016], external memory [Wang et al., 2016] and sentence-level training [Shen et al., 2016].

The translation performance heavily relies on the source sentence representation ability of the encoder and the target sentence generation ability of the decoder. To further enhance NMT, we propose a novel framework named as "ME-MD (multiple encoders and multiple decoders)", which exploits multiple encoders to represent the source sequence and multiple decoders to generate target words. These encoders and decoders are allowed to possess different depths or multiple types. The basic idea is that multiple encoders provide more comprehensive source representation and multiple decoders capture more complicated translation knowledge. We implement several ME-MD systems and carry experiments on the Chinese-English translation task. Experimental results show that ME-MD systems outperform the encoder-decoder baseline by large margins. Our best system surpasses the state-of-the-art NMT system by 2.1 BLEU points and exceeds phrase-based Moses by 7.38 BLEU points. We also validate that our approach benefits more from the architecture change rather than making the network wider and deeper. Although, we conduct the experiment on machine translation task, our framework is general and can be applied to other sequence to sequence tasks.

2 Neural Machine Translation

We briefly introduce the NMT architecture [Bahdanau et al., 2015] that our systems build on. Formally, given a source sentence x = x1, ..., xm and a target sentence y = y1 , ..., yn , NMT models the translation probability as

where y<t = y1 , ..., yt-1 .

The NMT system primarily consists of two parts: the encoder and the decoder. For convenient explanation, we attribute the attention mechanism as a sub-component of the decoder. The encoder compresses the source sentence into the distribution representation and the decoder generates target words one by one regarding the source representation, as

where h is the source representation. The generation probability of yt is computed as

where q is the context to predict the target word, g(·) is a linear function and st is the hidden state of decoder which represents the translation status. The attention ct denotes the related source words for generating yt and is computed as the weighted-sum of source representation h upon an alignment vector αt shown in Eq.(4) where the align(·) function is a feed-forward network with the softmax normalization.

The hidden state st is updated as

where f(·) is a gated hidden unit.

In recent, a varietal attention mechanism 1 is implemented as

where f1(·) and f2(·) are recurrent functions. We adopt this varietal attention mechanism in our NMT systems.

3 ME-MD Framework

We aim to enhance NMT through incorporating multiple encoders and decoders. Our intuition is that multiple encoders provide comprehensive source representations and multiple decoders capture complicated translation knowledge.

1 https://github.com/nyu-dl/dl4mt-tutorial/tree/master/session2

3.1 General Architecture

The proposed ME-MD framework consists of two parts: MEncoder and M-Decoder, as shown in Figure 1. The MEncoder compresses the source sentence into the distribution representation as the source representation and the MDecoder generates target sentence word by word upon the source representation.

Compared with the single encoder in the encoder-decoder framework, M-Encoder allows multiple encoders to represent the source sentence, respectively. All the source representation are combined to construct the ultimate source representation. Through incorporating diverse encoders, we expect to obtain a more comprehensive representation of the source sentence. In the M-Decoder, more than one decoders are incorporated to capture more complicated translation knowledge. The outputs of the decoders are combined before the softmax layer for predicting target words. The proposed ME-MD framework is flexible to integrate variable encoders and decoders and can be applied to other sequence to sequence tasks.

3.2 M-Encoder

Encoders in M-Encoder module can posses multi-depths and multi-types. The multi-depth M-Encoder combines a couple of encoders with different depths and the multi-type MEncoder exploits encoders with different types.

We consider that multi-depth encoders can provide multiple level abstraction of the source sentence. Figure 2 (a) shows a multi-depth M-Encoder with three encoders, which depths are 2, 4 and 6, respectively. Without loss of generality, we take the "encoder 2" for detailed illustration. We exploit the left-to-right gated recurrent unit (LGRU) [Cho et al., 2014] to forwardly compress the source sequence and the right-to-left gated recurrent unit (RGRU) to reversely compress the source sequence. Layers with different directions are alternately stacked with direct connections. After the input sequence is compressed by stacked GRU layers to the vector o2 = {o21, ..., o2m}, a gated unit is employed to combine original word embedding e(xi) and o2i as

where Wxz,Woz,Wxh and Woh ∈ Rd×d are the weight matrix parameters , bz and bh ∈ Rd are the bias parameters. For conveniently builds the network, we set the word dimension and hidden unit numbers to the identical value d. Three encoders produce three source representations as

We combine the three representations with a feed-forward network as

in which Wh1,Wh2, and Wh3 ∈ Rd×d are the weight matrix parameters, bz ∈ Rd is the bias parameter.

Figure 2 (b) shows a multi-type M-Encoder with two types of encoders to compress the source sentence. One encoder is the GRU-based network and another is the CNN-based network. The CNN encoder only possesses a convolutional layer with a fixed window size. The GRU encoder captures the global source representation and the CNN encoder focus on the local representation. The output of the CNN-encoder is computed as

where Wf ∈ Rd×p×d is the weight matrix parameters, b is the bias parameter and p is the convolutional window size. The gate computation and the ultimate source representation computation are identical to the multi-depth M-Encoder.

3.3 M-Decoder

The M-Decoder aims to enhance the generation ability of the decoder through integrating multiple decoders. Similar to the M-Encoder, the M-Decoder can also have multidepth and multi-type. The multi-depth M-Decoder consists of a couple of decoders with different depths. The multi-type M-Decoder allows to exploit the variable attention mechanisms [Bahdanau et al., 2015; Luong et al., 2015a; Wu et al., 2016] and multiple recurrent networks.

Figure 3 presents a multi-depth M-Decoder that contains three decoders with different depths. We take the "decoder2" for detailed description without loss of generality. We adopt the varietal decoder implementation in our NMT systems. Formally, the output q2,t of the "decoder2" at time t is computed as following:

where ˜s21,t and ˜s22,t are outputs of the GRU layers, c2,t is the related source context for generating target word yt, function attention(·) is computed as in Eq.(4) and function g(·) is a linear one. The gate computation DGate2(˜s22,t, c2,t) is

in which z is the update gate, r is the reset gate, Wcz, Wsz, Wcr, Wsr, Wss and Wcz are weight matrix parameters and bgz, bgr and bss are bias parameters. The outputs of three decoders are combined by a feed-forward network and feeded into softmax function to predict the target word as

where Wq1, Wq2 and Wq3 are weight matrix parameters and bq is the bias parameter.

Although we present several categories of M-Encoder and M-Decoder in this section, a large variety of encoders and decoders can be incorporated into our framework for its flexibility.

4 Experiment

We validate the effectiveness of the proposed framework on the Chinese-English translation task.

4.1 Data andMetrics

Our Chinese-English training corpus consists of 1.25M sentence pairs extracted from LDC corpora 2 with 27.9M Chinese words and 34.5M English words respectively. The 30K vocabularies cover approximately 97.7% and 99.3% words in Chinese and English respectively. We choose NIST 2002 dataset as the validation set. NIST 2003-2006 are used as test sets. The translation quality evaluation metric is the caseinsensitive 4-gram BLEU 3 [Papineni et al., 2002].

4.2 Systems

We implements 4 ME-MD systems and compare them with two baseline systems. The systems are listed as following:

1. Moses [Koehn et al., 2007] is an open source phrasebased SMT baseline system with default settings. Words are aligned with GIZA++ [Och and Ney, 2003]. The 4-gram language model with modified Kneser-Ney smoothing is trained on the target portion of training data by SRILM [Stolcke and others, 2002].
2. RNNsearch∗ is our in-house implementation of RNNsearch [Bahdanau et al., 2015] baseline system

2 The corpora includes LDC2002E18, LDC2003E07, LDC2003E14, Hansards portion of LDC2004T07, LDC2004T08 and LDC2005T06.

3 ftp://jaguar.ncsl.nist.gov/mt/resources/ mteval-v11b.pl

with the varietal attention mechanism. Different from the original model , we stack a forward GRU layer and a backward GRU layer with direct connection as a two layers encoder. The system can be regarded as the "1Encoders-1Decoder" ME-MD system and is the basis of other ME-MD systems.

1. 2Encoders-1Decoder gets two GRU-based encoders and one GRU-based decoder. The depths of encoders are 2 and 4, respectively. The depth of the decoder is 1.
2. 3Encoders-1Decoder has three GRU-based encoders and one GRU-based decoder. The depths of encoders are 2, 4 and 6, respectively. The depth of the decoder is 1.
3. 3Encoders-3Decoders consists of three GRU-based encoders and three GRU-based decoders. The depths of encoders and decoders are 2, 4 and 6, respectively.
4. GCEncoders-1Decoders contains one GRU-based encoder and one CNN-based encoder. The depth of the GRU-based encoder is 2 and the convolutional window size of the CNN-based encoder is 3. The depth of the decoder is 1.

4.3 NMT Training

The sentence length for training NMT models is up to 50, while SMT model exploits whole training data without any restrictions. The word embedding dimension and the hidden unit numbers are set to 512. Square matrices are initialized in a random orthogonal way. Non-square matrices are initialized by sampling each element from the Gaussian distribution with mean 0 and variance 0.012 . All bias are initialized to 0. Parameters are updated by Mini-batch Gradient Descent and learning rate is controlled by AdaDelta [Zeiler, 2012] with decay constant ρ = 0.95 and denominator constant ϵ = 1e - 6. The batch size is 80. Dropout strategy [Srivastava et al., 2014] is applied to the output layer with the dropout rate=0.5 to avoid over-fitting. The gradients of the cost function which have L2 norm larger than a predefined threshold 1.0 is normalized to the threshold to avoid gradients explosion [Pascanu et al., 2013]. We exploit length normalization on candidate translations and the beam size for decoding is 12. The systems are implemented on the Theano library and trained with Tesla K40 GPUs.

4.4 Experiment Result

Table 1 shows the performance of each system. The 2Encoder-1Decoder system and the 3Encoder-1Decoder system surpass the RNNsearch∗ baseline by 0.52 and 1.90 BLEU points, fromwhich we conclude that incorporating additional encoders can effectively improve the NMT performance. Through extending the number of decoders to three, we obtain further more 0.43 BLEU points that proves the effectiveness of the M-Decoder module. The GCEncoders-1Decoders system outperforms the RNNsearch∗ baseline by 1.19 BLEU points shows the CNN-based encoder improves the source representation ability of the M-Encoder. The GCEncoders1Decoders system surpasses 2Encoders-1Decoder system by 0.67 BLEU points shows that CNN-based encoder provides the diverse source representation.

We present the performance of previous works that employ identical training corpora in Table 2. Although we restrict the maximum length of sentence to 50, our model achieves the state-of-the-art performance on all test sets. Our ME-MD system outperforms previous work by at least 2.1 BLEU points.

4.5 Comparison with Deeper andWider Networks

We carry more experiments to investigate whether our approach achieves improvements through just making the neural network seems deeper and wider or not. Table 3 shows the performance comparison between wider, deeper networks and the ME-MD systems and Figure 4 presents the training speeds of each system.

• Wider Networks. We enlarge the word embedding dimension and the hidden unit numbers to make networks wider. We achieve improvements as 0.79 BLEU points by extending the width from 512 to 1024 and obtain further more 0.32 BLEU points through set the width to 2048. However, the approach leads the rapid increase of the parameters and the dramatic decrease on the training speed. Compared with the wider networks, our approach provides larger improvements with less parameters and saves significant computation overhead.
• Deeper Networks. With the increasing of the depth, RNNsearch∗ achieves slight improvements or even obtains inferior performance. The reason is that it is difficult to train a very deep networks for the gradient propagation problem. Although our encoders and decoders are also very deep, we still achieve significant improvements for the shallow encoders is able to alleviate the gradient propagation problem. From the speed experiment, we observer that the speed of a ME-MD system is

mainly depend on the deepest encoder and is close to the RNNsearch∗ that possess the same depth.

The experimental results show that the improvements of our approach benefit more from the effective architecture than just introducing more parameters and ME-MD systems achieves significant improvements with modest increase on the training overhead.

5 RelatedWork

Our proposed ME-MD framework possesses multiple encoders and decoders, which is similar to the multi-task learning architectures that based on the encoder-decoder framework. Dong et al. [2015] proposed a unified network with one encoder and multiple decoders to simultaneously train couples of translation systems. These translation systems share the source sentence representation and generate target translations in different languages. Luong et al. [2016] presented a framework with multiple encoders and decoder for multiple-task sequence to sequence learning. The encoders and decoders are designed for multiple specific tasks, such as translation, parsing and image caption. Firat et al. [2016] proposed to share the attention mechanism for jointly training multi-lingual translation systems, in which encoders and decoders are employed for certain languages. The above mentioned works just activate one encoder and one decoder when deal with a certain task or translation direction. In our framework, all the encoders and decoders are utilized simultaneously, through which the translation quality is improved. The multi-source translation model with multiple encoders and attention mechanisms was proposed by Zoph and Knight [2016]. One encoder is applied to compress one kinds of source languages and all encoders outputs are combined to generate target translation. Compared with our work, their approach requires multi-way parallel corpus that is difficult to obtain.

6 Conclusion

We proposed an effective framework named "ME-MD" to enhance NMT performance with multiple encoders and decoders. Compared with the encoder-decoder framework, our approach enables to exploit multiple encoders and decoders with variable depths and types. The basic idea is that multiple encoders provide the more comprehensive representation of the source sentence and multiple decoders captures more complicated translation knowledge. To validate the effectiveness of our approach, we carry the experiment on Chinese-English translation task. We trained various networks architecture with M-Encoder and M-Decoder modules. Experiments show that ME-MD systems achieve significant improvements on translation quality over the basic encoderdecoder system and the phrase-based system by large margins. Through increasing the number and category of encoders and decoders, we acquire continuous improvements.

The improvements benefit from the structural change of the original architecture. The comparison with previous works on identical training corpora shows our best model achieves the state-of-the-art performance. We also implemented the wider networks and found that enlarging the word embedding and hidden size can bring further improvements on translation quality. While, wider networks require enormous computing overhead, which needs longer training time and larger GPU memory space. The networks with deeper architecture do not produce considerable improvements. with the increase of depth,the experiment show that translation quality is decreased on the contrary. Compared with the wider and deeper networks, our model enables diverse encoders and decoders which leads to the translation quality enhancement with less computation overhead.

Although, we carry the experiment on the machine translation task, the ME-MD framework is general and can be applied to other sequence to sequence tasks. The framework is a kind of novel approaches to enhance the neural network performance. Except the implementations that shown in this paper, more categories of encoders and decoders can be introduced into the framework. In the future, we will validate our approach on more language pairs and explore more effective methods to improve the model capacity.

Acknowledgments

Qun Liu's work is partially supported by Science Foundation Ireland in the ADAPT Centre for Digital Content Technology (www.adaptcentre.ie) at Dublin City University funded under the SFI Research Centres Programme (Grant 13/RC/2106) co-funded under the European Regional Development Fund. We thank the anonymous reviewers for their insightful comments.