Improving Context-Aware Neural Machine Translation with Source-side Monolingual Documents

1 Introduction

Document context-aware machine translation aims at translating each sentence in a document under the guidance of the global context, with expectations to obtain more coherent and less ambiguous translations. Due to the availability of document-level parallel datasets, recent years have witnessed great progress in context-aware neural machine translation (NMT) with extensive attempts at leveraging document-level context, from sentence concatenation [Tiedemann and Scherrer, 2017], and mechanisms with multiple encoders [Jean et al., 2017; Wang et al., 2017; Zhang et al., 2018; Bawden et al., 2018; Voita et al., 2018; Miculicich et al., 2018; Maruf et al., 2019; Yang et al., 2019], to cache and memorybased NMT [Tu et al., 2018; Kuang et al., 2018; Maruf and Haffari, 2018]. However, large-scale document parallel corpora are costly to build as they require specialized expertise. Conversely, monolingual document data is much easier to find. In this paper, our goal is to leverage large-scale sourceside monolingual documents to improve context-aware NMT performance.

∗Corresponding Author

There have been several attempts to boost context-aware translation performance in the scenarios where the documentlevel parallel corpora are small/middle-scale, or even not available. On the one hand, sentence-level parallel data is a natural resource to use. For example, [Zhang et al., 2018] propose a two-stage training strategy for context-aware NMT by pre-training the model on a sentence-level parallel dataset. Such or similar training strategies are widely applied in related studies [Miculicich et al., 2018; Tan et al., 2019; Voita et al., 2019b; Miculicich et al., 2018; Maruf et al., 2019]. On the other hand, monolingual target language document data could be used to increase the coherence of document translation. For example, [Voita et al., 2019a] propose DocRepair trained on target-side monolingual documents to correct the inconsistencies in sentence-level translation. [Yu et al., 2020] train a document-level language model to re-rank N-best translation outputs. To the best of our knowledge, [Junczys-Dowmunt, 2019] is the only work that leverages large-scale source-side monolingual documents, in which they simply concatenate sentences within a document into a long sequence and explore multi-task training via the BERT-objective [Devlin et al., 2019] on the encoder.

As typical context-aware NMT models usually contain a component of capturing global context, in this paper we aim at greatly enhancing the capability of capturing useful global context by exploring large-scale source-side documents. Specifically, to achieve this goal, we propose a novel self-supervised pre-training task on monolingual documents, Pre-training with Global Context, i.e., PGC. We find that it is beneficial to mask entire selected sentences from a document and generate these gap-sentences from their neighbouring sentences as a pre-training objective.

As shown in Figure 1, our PGC model consists of a context encoder, a sentence encoder and a decoder, similar to various context-aware NMT models [Zhang et al., 2018; Miculicich et al., 2018; Maruf et al., 2019; Zheng et al., 2020]. Therefore, the PGC model could easily and effectively adapt to downstream context-aware NMT models. We initialize the downstream NMT models' two encoders with the counterparts of the pre-trained PGC model and then finetune the NMT models. Extensive experimental studies on four different translation tasks suggesting that our approach has strong generality to various downstream models.

Overall, this paper makes the following contributions:

• To leverage document-level monolingual corpora, we first propose a novel self-supervised pre-training task, in which sentence generation can benefit from documentlevel context.
• We evaluate the effectiveness and generality of our approach by adapting it to various downstream contextaware NMT models on four translation tasks.

2 Pre-training with Global Context (PGC) on Monolingual Documents

In this section, we first propose our pre-training task with global context (PGC). Then we present the details of our PGC model which leverages global context.

2.1 PGC Task

To be consistent with context-aware NMT, our pre-training task is sentence generation augmented by the document-level context. Our pre-training objectives are inspired by both gap sentence objective [Zhang et al., 2020] and masked language model objective [Devlin et al., 2019].

Context-Aware Gap Sentence Generation (CA-GSG)

Given a document with N sentences, we randomly select M sentences as gap sentences and replace them with a mask token [MASK1] to inform the model. For each selected gap sentence, we use its left and right neighbours as input while the gap sentence serves as output.

Context-Aware Denoisying Auto-Encoder (CA-DAE)

Given a sentence X, we follow BERT and randomly select 15% tokens in it. The selected tokens are (1) 80% of time replaced by a mask token [MASK2], or (2) 10% of time replaced by a random token, or (3) 10% of time unchanged. For a sentence, we use its masked ̂X as input while the original X serves as output.

Combination of CA-GSG and CA-DAE

Both CA-GSG and CA-DAE are applied simultaneously in our pre-training task. For convenience of presentation, we use a concrete example to illustrate the input and output of our pre-training task. As shown in Figure 2, let assume that the original document X contains 6 sentences and the third and fifth sentences (i.e., X3 and X5) are selected as gap sentences while the others are not. On the one hand, for a sentence which is not selected as gap sentence, e.g., X1, we use its masked version (e.g., ̂ X1) as input while try to predict its original sentence (e.g., X1). On the other hand, for a gap sentence, e.g., X3, we concatenate its left and right neighbouring sentences with separator [MASK1] and try to predict the gap sentence (e.g., X3). As shown in Figure 2, sentences from S1 to S6 constitute document-level input S while sentences from T1 to T6 make up output T . Note that we do not include either gap sentences themselves or their masked version in S, in case the document context contains obvious hints for generating gap sentences.

Overall, the pre-training task is to predict target output T by giving source input S, which is the same as the task of context-aware translation, except that in our pre-training task S and T are in the same language while in the latter the two are in different languages.

2.2 PGCModel

We define some notations and describe our model of pretraining. Given a document-level source input S = (S1, · · · , SN) and target output T = (T1, · · · , TN) with N sentence pairs, we assume each source sentence Si = (si,1, · · · , si,n) consists of n words. We use dm as the size of embedding and hidden state through the entire model.

We now design model to generate T by given S. To be consistent with our purpose that our model of pre-training could easily adapt to various downstream context-aware NMT models, we propose a universal context-aware model, rather than a sophisticated one.

Figure 3 shows our model for the pre-training task. It contains two parts, namely global context encoder and a seq2seq model augmented by context representation. Note that in our pre-training, we take documents as input units.

Global Context Encoder

For the i-th input sentence Si in a document, the global context encoder aims to extract useful global context for every word si,j in it. As shown in Figure 3(a), the encoder consists of a stack of Ng identical encoder layers. Each encoder layer consists of three major sub-layers: a self-attention sub-layer, a global context attention sub-layer and a feed-forward neural sub-layer.

In the k-th encoder layer, the self-attention sub-layer takes A(k) i ∈ Rn×dm as input and computes a new sequence B(k) i with the same length via multi-head attention function:

where the output B(k) i is in the shape of Rn×dm . 1 For the first encoder layer, A(1) i is the addition of Si's word embedding and its position embedding while for other layers, A(k) i is the output of the proceeding encoder layer.

We denote B(k) ∈ R(N·n)×dm as the stacking result of (B(k) 1 , · · · ,B(k) N ). Note that B(k) is at document-level and represents the global context. The global context attention sub-layer extracts useful global context for si,j in Si. This is also done via multi-head attention function:

where the output C (k) i is in the shape of Rn×dm .

Finally, the feed-forward sub-layer is applied to each position separately and identically by two linear transformations with a ReLU activation in between.

where W F1 ,W F2 ∈ Rdm×dm , and bF1 , bF2 ∈ Rdm are model parameters.

We denote Gi ∈ Rn×dm as the final output of the global context encoder, i.e., Gi = D(Ng) i . That is to say, Gi represents the context representation for sentence Si.

1 The actual output of this sub-layer is LayerNorm(B(k) i + A(k) i ), where LayerNorm is the layer normalization function. For simplicity, we do not include the residual addition and layer normalization functions in our sub-layers.

Sentence-Level Seq2Seq Model Augmented by Context Representation

As shown in Figure 3 (b), the sentence-level seq2seq model is almost same as the standard Transformer, except that it is now equipped with context representation obtained by the global context encoder. For sentence Si, we denote the sentence encoder output as Hi ∈ Rn×dm . To leverage its context representation Gi, we define a gate to linearly combine the two kinds of representation via:

where the gating weight is computed by

where W G ∈ R2dm×dm are model parameters.

Then we use H′ i to replace Hi as the input to the decoder.

3 Applying Pre-trainedModel to Downstream Context-Aware NMT models

To test if our pre-trained model is helpful for context-aware NMT, we select the following four typical NMT models:

• DocT [Zhang et al., 2018], takes two previous sentences as context. As a Document-aware Transformer, the context representations are fed into both sentence encoder and decoder.
• HAN [Miculicich et al., 2018], leverages all previous source and target sentences as context and proposes a Hierarchical Attention Network to capture the context in a structured and dynamic manner. The context representations are then fed into the decoder.
• SAN [Maruf et al., 2019], further uses the whole document as context. It uses Sparse Attention Network to selectively focus on relevant sentences and then attends to key words in those sentences.
• MCN [Zheng et al., 2020], use a encoder builds local and global context from the entire document to understand the inter-sentential dependencies towards making the Most of Context.

Meanwhile, we also provide the following contrastive NMT model to test if it is necessary for the downstreamNMTmodels to have same structure as the pre-trained model.

• PGC4NMT, is similar to our model of pre-training. When use PGC for context-aware NMT, the only difference lies in that for NMT we use two different vocabularies for the source and target sides.

All the above context-aware NMT models consist of a context encoder, a sentence encoder and a decoder. Moreover, all the context and sentence encoders are Transformer-based with similar or same structure to the standard Transformer encoder [Vaswani et al., 2017].

Applying pre-trained model to these context-aware NMT models. We load generic parameters from the context encoder and the source encoder of the pre-trained model to initialize the downstream context-aware NMT models. That is to say, if a sub-layer that does not exist in the pre-trained model, it will be randomly initialized, otherwise it will be initialized by the pre-trained model.

Fine-Tuning Strategy. We use a two-step fine-tuning strategy to fine-tune the downstream context-aware translation models. In the first step, as shown in Figure 1, generic parameters from the context encoder and the sentence encoder of the pre-trained model are loaded to initialize the counterparts in the translation model. Then we freeze the generic parameters and update the remaining parameters of the translation model. In the second step, we train all model parameters with a small number of iterations.

4 Experimentation

To test the effect of our approach in leveraging source-side monolingual documents, we conduct experiments on various tasks, including Chinese-English (ZH-EN), English-Spanish (EN-ES), and English-German (EN-DE) translation.

4.1 Experimental Settings

Pre-Training Data Settings. We pre-train two models, one for Chinese and the other for English based on the following two corpora, respectively:

• Chinese Gigaword (LDC2009T27), consists of eight distinct international sources of Chinese newswire. We convert traditional Chinese into simplified ones. Then all sentences are segmented by Jieba. 2
• English Gigaword (LDC2012T21), consists of six distinct international sources of English newswire. We perform sentence segmentation, and then tokenize and truecase all sentences by Moses scripts. 3

For both Chinese and English, we segment words into subwords by a BPE model with 30K operations. For efficient training, we split long documents into sub-documents with at most 30 sentences. We have 2.6M (7.3M) sub-documents with 24M (102M) sentences in total for Chinese (English). On the monolingual documents, we prepare training instances for the pre-training task and set gap sentence ratio to 20%.

2 https://github.com/messense/jieba-rs

3 http://www.statmt.org/moses/

Context-Aware NMT Data Settings. For ZH-EN, the document-level parallel corpus of training set include 41K documents with 780K sentence pairs. 4 We use the NIST MT 2006 dataset as the development set, and the NIST MT 02, 03, 04, 05, 08 datasets as test sets. The Chinese sentences are segmented by Jieba while the English sentences are tokenized and lowercased by Moses scripts. For EN-ES, the training set is from IWSLT 2014 and 2015 while the development set is dev2010 and the test set is test2010, test2011, and test2012. 5 For EN-DE (TED), the training set is from IWSLT 2017. We use test2016 and test2017 as our test set while the other as the development set. For EN-DE (News), the training set is News Commentary v11 corpus, 6 while the development set is news-test2015 and the test set is newstest2016. We tokenize and truecase all the datasets by Moses scripts. For all the translation datasets, we segment the source sentences with the corresponding BPE model learned from the pre-training data while all the target sentences are segmented by the BPE model with 25K operations learned on the corresponding target-side data. Meanwhile, we split long documents into sub-documents with at most 30 sentences.

Model Settings. We use OpenNMT 7 as the implementation of Transformer and extend it to capture context. We also reimplement DocT, HAN, and SAN as to better evaluate our pre-trained PGC models. 8 For all pre-trained and translation models, the numbers of layers in the context encoder, sentence encoder and decoder (i.e., Ng, Ne, and Nd in Figure 3) to 4, 6, 6, respectively. In inferring, we set beam size to 5.

Evaluation We report BLEU score as calculated by the multi-bleu.perl script and Meteor score. To be consistent with related studies [Zhang et al., 2018; Miculicich et al., 2018; Maruf et al., 2019; Yang et al., 2019], here we report caseinsensitive BLEU score for ZH-EN and case-sensitive BLEU score for other translation tasks.

4.2 Experimental Results

Table 1 shows the BLEU and Meteor scores on the four translation tasks. It shows that on the one hand, even without pre-training, all context-aware NMT models outperform sentence-level Transformer. On the other hand, by leveraging source-side monolingual documents, our approach significantly improves the translation performance for all the four context-aware NMT models, with 0.92 to 1.96 improvement of BLEU scores and 1.04 to 1.85 improvement of Meteor scores, suggesting the effectiveness and generality of our approaching in leveraging source-side documents.

Taking SAN as representative, we compare its performance to that of PGC4NMT, and observe that although the structure of SAN is different from PGC4NMT and our pre-trained model, the degree to which SAN can benefit from pre-trained model is similar to or even more than PGC4NMT. This further demonstrates the generality of our pre-trained model.

4 It consists of LDC2002T01, LDC2004T07, LDC2005T06, LDC2005T10, LDC2009T02, LDC2009T15, LDC2010T03.

5 https://wit3.fbk.eu

6 http://www.casmacat.eu/corpus/news-commentary.html

7 https://github.com/OpenNMT/OpenNMT-py

8 Although our experimental settings have subtle differences from theirs, e.g., not sharing vocabulary, the performance of our reimplemented systems is comparable to theirs.

4.3 Result Analysis

Next we analyze how the pre-trained PGC models affect downstream translation performance.

Statistics on Our Pre-trained PGC Models

Table 2 presents statistics on our two pre-trained PGC models. With 500K training steps, we complete 3.0 and 1.2 passes over the pre-training data within 70 and 75 hours for Chinese and English, respectively.

In total, this batch size and number of steps corresponds to pre-training on 500K × 8196 ≈ 4.1B tokens. To better leverage pre-training dataset, it is a common practice to eke out additional performance by using an ensemble of models. We leave this in our future work.

Effect of Different Pre-Training Objectives

As shown in Figure 2, in this paper we combine both CAGSG and CA-DAE objectives in our pre-training. To investigate the effect of CA-GSG, we use CA-DAE as the only objective in our pre-training task. In this way, the S3 and S5 in Figure 2 (b), for example, will be ̂ X3 and ̂ X5, respectively. Figure 4 compares the performance when the pre-training task is of CA-DAE objective or combination of CA-GSG and CA-DAE. For better illustration, for each translation task we scale the BLEU scores by viewing the performance of DocT·DAE as 1.0 and the performance of other systems as the ratio of their BLEU scores over DocT·DAE score.

From the figure, we see that in 19 out of 20 cases, combining CA-GSG and CA-DAE achieves better performance than CA-DAE. This result suggests pre-training benefits from the CA-GSG, which is more challenging than CA-DAE.

Effect of Gap Sentence Ratios

A significant hyper-parameter in designing the pre-training task is the gap sentence ratio. A low ratio makes the pretraining less challenging while choosing gap sentences at a high ratio makes the global context have more overlapped. For example, X4 and its masked version ̂ X4 in Figure 2(a) appear three times on the source input S due to that both its left and right neighbouring sentences X3 and X5 are gap sentences. We compare three variants of gap sentence ratio (10%, 20%, and 30%). As shown in Figure 5, we see that the best performance always appears at the ratio of 20%.

Effect of Pre-Trained Global Context Encoder

Next we examine whether the proposed global context encoder actually learns to effectively extract global context from the entire document. To this end, we change our per-training task to be context-agnostic with a standard seq2seq framework. As a result, only the pre-trained sentence encoders are applied to downstream translation models. From Table 3 we observe that pre-trained global context encoder helps contextaware translation, suggesting that our pre-trained context encoder is indeed capable of extracting useful global context from the entire document.

1.04

Discourse Phenomena

Following [Voita et al., 2019a; Zheng et al., 2020], we use contrastive test sets for the evaluation of discourse phenomena for English-Russian. There are four test sets in the suite regarding deixis, lexicon consistency, ellipsis (inflection), and ellipsis (verb phrase). Each test set contains groups of contrastive examples consisting of a positive translation and negative translations. The goal is to figure out if a model is more likely to generate a correct translation compared to the incorrect variation. We summarize the results in Table 3, which shows that our approach is better at resolving discourse consistencies compared to both the context-agnostic baseline and our proposed context-aware approach without pre-training.

5 RelatedWork

5.1 Context-Aware NMT

According to the scope of context being modeled, we group the related studies into two categories: (1) those which use partial context and (2) those which employ global context.

Among the first line, great efforts have been made in modeling local context. Concatenation method proposed by [Tiedemann and Scherrer, 2017] is an early attempt in RNN-based NMT. Then mechanisms with multiple encoders become promising in both RNNSearch and Transformer NMT[Jean et al., 2017; Wang et al., 2017; Zhang et al., 2018; Bawden et al., 2018; Voita et al., 2018; Voita et al., 2019b; Yang et al., 2019]. Cache/Memory-based approaches [Tu et al., 2018; Kuang et al., 2018; Maruf and Haffari, 2018] also fall in this line because the cache stores word/translation in previous sentences.

Another strand of context-aware NMT takes the whole document as a translation unit and dynamically extracts useful global knowledge for every sentence in the document. The global context can be either source-side [Maruf and Haffari, 2018; Mace and Servan, 2019; Maruf et al., 2019; Tan et al., 2019; Zheng et al., 2020; Kang et al., 2020] or target-side [Xiong et al., 2019].

To make translations within a document more coherent, [Voita et al., 2019a] propose DocRepair trained on monolingual target language document corpora to correct the inconsistencies in sentence-level translation while [Yu et al., 2020] train a context-aware language model to re-rank sentencelevel translations. Finally, [Junczys-Dowmunt, 2019] use source-side monolingual documents to explore multi-task training via the BERT-objective on the encoder. They simply concatenate sentences within a document into a long sequence, which is different from our approach.

5.2 Pre-training on Document-Level Corpora

Recently, pre-trained models with large corpora built with Transformer have made great success. Among them, BART, mBART and PEGASUS are built on document-level corpora.

BART [Lewis et al., 2020] and mBART [Liu et al., 2020] use a (multilingual) denoising auto-encoder which learns to reconstruct the original document from a corrupted version. BART is effective for machine translation by achieving significant improvement over a back-translation system while mBART produces significant performance gains across a wide variety of translation tasks.

PEGASUS [Zhang et al., 2020] is a Transformer-based encoder-decoder model with a self-supervised objective tailored for abstractive text summarization. In PEGASUS, important sentences are removed/masked from a document then generated together as one output sequence from the remaining sentences, similar to an extractive summary.

BART, mBART and PEGASUS all fall into a standard seq2seq framework by viewing a document (or masked sentences) as a sequence. This is very different from ours, where we use an additional global context extractor for contextaware sentence generation.

6 Conclusion

To leverage monolingual document data, in this paper we have proposed a PGC model with two different training objectives. PGC model consists of a global context encoder, a sentence encoder and a decoder, which is similar to various context-aware NMT models. We have evaluated the effectiveness and generality of our pre-trained PGC model by adapting it to various downstream context-aware NMT models. Experimental results on various translation tasks demonstrate the effectiveness and generality of our approach.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Grant No. 61976148, 61876120 and 62076175).