Correct-and-Memorize: Learning to Translate from Interactive Revisions

1 Introduction

Recently, sequence-to-sequence models [Sutskever et al.,

2014; Cho et al., 2014] have gain superior performance in machine translation [Bahdanau et al., 2014; Vaswani et al., 2017]. Yet these state-of-the-art neural machine translation (NMT) models still fail to generation target sentences with comparable quality as human translators.

To obtain a high-quality translation, there are a few recent works attempt to incorporate human revision instructions into the algorithmic translation process. For example, previous interactive NMT [Sanchis-Trilles et al., 2014; ´Alvaro Peris et al., 2016; Knowles and Koehn, 2016] proposes to ask human to revise the translation output from the beginning of a sentence to the end (i.e. from the left to right), and regenerates the partial translation on the right side of the Source：

∗Equal contribution, part of this work was done while Rongxiang Weng was a research intern at ByteDance AI Lab.

†Corresponding author.

1 Source code is available at: https://github.com/wengrx/CAMIT

� (they) �� (discuss) �� (prevent) � (nuclear) ��

(proliferate) � (and) �� (free) ��� (trade zone) �� (issue)

Target：

Human Step 1

Source：

Translate： they to discuss the issue of nuclear UNK and trade area

Target：

they discuss nuclear non-proliferation and free trade zone issues � (they) �� (discuss) �� (prevent) � (nuclear) �� (proliferate) � (and) �� (free) ��� (trade zone) �� (issue)

they discuss nuclear non-proliferation and free trade zone issues

they to discuss the issue of nuclear UNK and trade area

Translate： they to discuss the issue of nuclear UNK and trade area

they to discuss the issue of nuclear UNK and trade area

vising "to discuss" → " discuss".

Human Step 1

(b) Our proposed CAMIT takes the most critical revision "UNK" → "non-proliferation" and automatically fixes the rest mistakes across the whole sentence.

Figure 1: Illustration of the previous and our proposed interactive NMT methods. Words with bold and orange fonts are revised by human. Those with underlines and blue fonts are automatically fixed by models. Note our proposed CAMIT only needs one revision while the previous method requires two.

revised token. We refer to these models as uni-directional interactive models. The interactive steps can repeat multiple times until a satisfactory sentence is obtained. In this case, given human revisions for model-translation mistakes, the process could potentially fix some minor mistakes automatically, exploiting the base translation models' ability to save human efforts (See Figure 1a).

In this paper, we make a few observations about neural machine translation and note two issues in previous interactive NMT approaches. First, the previous uni-directional interactive NMT method could only possibly fix those errors to the right of human-revised words. Cheng et al. [2016] show that revising critical mistakes first could significantly reduce the number of revisions. However, their intuition could not be directly applied to current uni-directional NMT models, because after revising critical mistakes, uni-directional interactive models cannot correct minor mistakes to the left of the revision, even with advanced decoding algorithm [Hokamp and Liu, 2017; Post and Vilar, 2018; Hasler et al., 2018]. Since the uni-directional interactive model only regenerates the partial sentence to the right of revisions, instead of updating the whole sentence. This leads to the constraint that each human revision should be proposed at the left most errors. Thus, human annotators have to revise these remaining mistakes manually, which is not efficient enough.

Second, neural machine translation models often keep making the same mistakes with similar discourse and domain context. Previous interactive NMT models rarely exploit the revision history to avoid similar mistakes. Our basic intuition is that previous sentences which have been corrected by human may be of greatly help to the translation of the following sentences when they are in the same discourse or domain. Learning from past revisions could preventing NMT models from making similar mistakes.

To address the above issues, we propose a correctand-memorize framework for interactive machine translation (CAMIT), a novel framework to perform efficient corrections and then softly memorize those corrections for enhancing translation results.

First of all, CAMIT contains an improved decoder given a sentence and one revised word. CAMIT performs the interactive NMT from both ends of the sentence, updating the whole sentence after getting a revision. Different from previous work about bi-directional decoder [Liu et al., 2016; Mou et al., 2016; Liu et al., 2018], our method is designed to take human revisions for interactive NMT. In such case, human can revise the most critical mistake first in an arbitrary position of the sentence; and after that the model will update the whole sentence, fixing minor mistakes automatically. Thus the efficiency of interaction could be significantly improved (Figure 1b).

Secondly, we propose to learn from and memorize interactive revisions to enhance the translation results. The revision history could be learned in the word level and the sentence level, respectively. At the word level, we propose a key-value memory mechanism [Weston et al., 2014] called revision memory, to remember prior revisions in the interactive process. At the sentence level, we use the online learning approach to fine-tune our base translation model on prior revised sentences, adapting our model parameters to the specific discourse or domain. When translating a new sentence, our model avoids generating similar mistakes in the past by looking up the revision memory, which is especially helpful to the translation of rare words.

Experiments show that our proposed CAMIT outperforms the previous interactive methods in both ideal and real machine translation settings. Our method's decoder obtains the improvement of 17 BLEU points with merely two revisions in the ideal and 8 BLEU points with 1.81 revisions in the real environments, respectively. Using interaction history further helps improve interactive efficiency and translation quality.

2 Background

Neural machine translation (NMT) is based on a standard Seq2Seq model, which adopts an encoder-decoder architecture for sentence modeling and generation [Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2014; Vaswani et al., 2017]. The encoder summarizes the source sentence into an intermediate representation, and the decoder generates the target sentence from left to right.

Formally, let x = {x1, · · · , xi, · · · } be a given input sentence and y = {y1, · · · , yj, · · · } be the output sentence. The encoder transforms the sentence to a sequence of annotations H, with each hi being the annotation of input word xi.

Based on the source annotations, the decoder generates the translation by predicting a target word yj at each time step j:

where g is a non-linear activation function, and sj is the decoding state for time step j, computed by

Here -→f is a recurrent unit [Cho et al., 2014] or self-attention network [Vaswani et al., 2017]. cj is a vector summarizing relevant source information. It is computed by the attention mechanism [Luong et al., 2015b; Vaswani et al., 2017].

where the r is a state from last time step or previous layer when the -→f is a recurrent unit or a self-attention network, respectively.

In the training stage, y is the gold sentence provided by training set. NMT models optimize the networks by maximizing the likelihood denoted by LL.

where |y| is the length of y and P(yj|y<j, x) is defined in Equation 1.

3 The Proposed CAMIT

Practically, there may be multiple mistakes in the translation output. Prior work propose to incorporate human efforts to boost the translation performance from left to right, both in the statistical machine translation (SMT) [Barrachina et al., 2009; Gonz´alez-Rubio et al., 2013] and the neural machine translation (NMT) [Knowles and Koehn, 2016; ´Alvaro Peris et al., 2016; Wuebker et al., 2016]. However, current interactive model is not efficient enough in both decoding and learning, as previously discussed.

In this section, we propose CAMIT, to improve the interactive efficiency in these two aspects. As shown in Figure 2, after each interaction from human, we propose a sequential bi-directional decoder for updating the whole sentence. All revisions are written into a revision memory, and the base Seq2Seq model will read from it latter for avoiding making same mistakes. Additionally, previously revised sentence will be learned by an online learning approach.

3.1 Preliminary

The interactive process always contains many sentences; and each sentence may need to be revised for many times. One revision is the process that human translator manually corrects one mistake in a sentence.

Commonly, the revision can include several operations: replacement, insertion and deletion. Following previous work [ ´Alvaro Peris et al., 2016; Knowles and Koehn, 2016; Cheng et al., 2016], we focus on the replacement and others can be implemented with several replacement operations. For example, deletion could be converted to replace incorrect words with spacing, and the insertion could be converted to replace spacing with inserted words. Furthermore, we only apply the word level revision in this paper, other types of revision, such as phrase, subword, etc., can also be used in our interactive model.

Because human may need to give multiple revisions to one sentence, we refer to the multiple revisions for one sentence as one round of interaction. Commonly, adjacent sentences in one interactive process, which are usually in the same discourse or domain, are related. We model the interactive process of many related sentences as one session. One session may contain many rounds, which could be accomplished by one or more human translators.

3.2 Sequential Bi-Directional Decoding

In the interactive process, let y = {y1, · · · , yj, · · · } be the initial sequence of words, output by the base Seq2Seq model. If human translator revises the word yj, the output is divided into 2 parts:

Uni-directional interactive model only updates the right part [ ´Alvaro Peris et al., 2016; Knowles and Koehn, 2016], ignoring potential mistakes in the left part, which means the revised word should always be the left most error, otherwise the errors in the left part will never be corrected.

We purpose a sequential bi-directional decoder for interactive NMT, which can update both parts of the sentence. The proposed model includes two decoders: a forward decoder -→f and a backward decoder ←- f . -→f and ←- f share the same encoder, but working sequentially.

Formally, in the interactive process, the forward decoder -→f first acts as the basic model to generate the basic translation: y = {y1, · · · , yj, · · · }. Given a revision {yj → yr j }, i.e. revising yj to yr j , -→f then generates the new right part for the translation, the same as in the uni-directional model. The new decoding state s′j+1 for -→f at the position j + 1 is computed as:

and the new sentence is

Here words before yr j are kept the same, the probabilities of words in the new right part is computed the same as in Equation 1.

Different from previous approaches, we then feed the inverted new right part {· · · , y′j+2 , y′j+1 , yr j } into ←- f , obtaining a new decoding state at the current revising position:

Then starting from s′j-1 , ←- f generates the new left part {y′j-1 , y′j-2 , · · · } according to a reversed decoding probability:

Finally, the output is

Note that, the length of the new left part may be not equal to the original left part.

In this way, the whole sentence is updated jointly by the two decoders. With the human revision, the new right part is expected to be better than the right part. Based on this better right part, the new left part is expected to become better as well.

The training stage is similar with multi-task models [Dong et al., 2015; Luong et al., 2015a], both decoders could be trained using cross-entropy as the objective:

where LR is computed by:

The P (yj |y>j , x) and LL are defined in Equation 1 and Equation 4, respectively. |y| is the length of y.

We have shown how to update the sentence after a single revision. However, another challenge for interactive NMT

End yc 2 yc 4 happens when human translator performs several revisions in one round, because the interactive process should regenerate the translation with all the revisions considered.

Matching Matching End y j-1 To solve the problem, we propose to combine the grid beam search [Hokamp and Liu, 2017] with our bi-directional decoder. Intuitively, the grid beam search adopts a grid to store the partial translations that containing the previous revisions. After the gird search, all previous revisions will be included in the final translation output. We propose to perform the grid beam search twice in our bi-directional decoding framework.

Specifically, given a current revision yrk+1 and several previous revisions {yr 1 , yr 2 , yr 3 , yr 4 }, in which {yr 1 , yr 3 } are in the right part of the current revision and {yr 2 , yr 4 } are in the left part of the current revision, respectively. We will first perform the gird beam search starting from yrk+1 with the forward decoder -→f , obtaining the new right part containing previous revisions ({yr 1 , yr 3 }) right to yrk+1 . Then given the new right part, we will perform the grid beam search the second time from right to left using the backward decoder ←- f , obtaining the new right part containing previous revisions ({yr 2 , yr 4 }) left to yrk+1 . In such case, all previous revisions ({yr 2 , yr 4 , yr 1 , yr 3 }) are included in the final output, keeping their sequential order. Figure 3 gives an illustration of the bi-directional grid search process.

3.3 Learning from Interaction History

The interaction history is very helpful to improve the efficiency of interactive models. In this section, we propose methods to learn from the interactive history by online learning and adopting revision memory.

Revision Memory

We first propose to exploit the interaction history in the word level, using a key-value memory [Weston et al., 2014], referred to as the revision memory, to memorize previously performed revisions of the session. This makes sense because some specific rare words, e.g. mostly out-of-vocabulary words (OOV), occur multiple times in a discourse. The proposed mechanism can help avoid making the same mistakes by memorizing and copying past revisions of the current session.

Specifically, as shown in Figure 4, the revision memory

consists of many items, each of which has a key (revision context) and a value (revised word). The revised word is used to Decoder Matching Matching denote the revision itself. The revision context is used to determine whether the memorized revision should be performed again at the current decoding step. We define the revision context to include two parts: the decoding state s′ (Equation 2) and the context vector c′ (Equation 3) of the past revision, which represent the revision context in aspects of the language model and the translation model, respectively.

Every revision of the current session will be written into the memory. When generating a word in decoding, the model will read the revision memory, trying to automatically fix mistakes with a copy mechanism [Gu et al., 2016]. The final output distribution of word w is computed by distributions from the decoder and from the revision memory:

where rj,t is the probability of copying word yr t in current position j. It is calculated by the revision context < sj, cj > of current decoding state and revision contexts of items in the memory. θj is a weight computed at each decoding step:

If the output word w is an unknown word (UNK), but has been corrected by human, our method could successfully generate w in future translation by copying it from the revision memory, which partially alleviate the problem of UNK. To keep the translation process going, the embedding of UNK is fed into the decoder, as the input for the next time step.

Online Learning

Adjacent sentences processed in one session are highly related, because they are always in the same discourse or domain. In this sense, former revised sentences may greatly help the translation for the following sentences. We use an online learning approach [Peris et al., 2017; Peris and Casacuberta, 2018] to learn from previously corrected sentences in the same session.

The process starts with the basic model trained on the parallel training data. At the beginning of each session, the basic model is used for translation. The parameters of the basic model is fine-tuned after each round to maximize the generation probability of the revised sentence at current round. Formally, every time we obtain a correct translation pair {x, y′ } after interaction, we update the whole translation model for one step, according to Equation 4. By learning from the sentence level interaction history, our Seq2Seq model better fits sentences of current session. Especially for sessions containing large amounts of sentences, this continuous learning approach may significantly improve the interactive efficiency.

4 Experiment

4.1 Setup

Data-Set

We conduct experiments on Chinese-English (ZH-EN) and English-Chinese (EN-ZH) translation tasks. For both ZH-EN and EN-ZH, we use NIST data-set to evaulate the proposed framework. The training data consists of about 1.6 million sentence pairs. 2 We use NIST03 as our validation set, and NIST04 and NIST05 as our test sets. These sets have 919, 1597 and 1082 source sentences, respectively, with 4 references. In EN-ZH, we use ref0 of each data set as source sentences. We extract about 0.2 million sentence pairs 3 in our training set, which retains the discourse information for training parameters of the revision memory. Furthermore, we also use IWSLT2015 data-set [Cettolo et al., 2012] on the ZH-EN translation task. We use the dev2010 as our validation set and the tst2010-13 as our test set. We also sample 100 sentences randomly from test sets for the human evaluation.

Implementation Details

We test our proposed approach on both of the RNNSearch [Bahdanau et al., 2014] and Transformer [Vaswani et al., 2017] baselines. We train the bi-directional NMT model with the sentences of length up to 50 words. For the RNNSearch, vocabularies of both Chinese and English includes the most frequent 30K words for both Chinese and English. We map all out-of-vocabulary words to the special token UNK. The dimension of word embedding is 512, and the size of hidden layers is 1024. We use the gradient descent approach to update the parameters, with a batch size of 80. The learning rate is controlled by Adam [Kingma and Ba, 2014]. For the Transformer, we apply byte pair encoding (BPE) [Sennrich et al., 2016] to all languages and limit the vocabulary size to 32K. We set the dimension of input and output of all layers as 512, and that of feed-forward layer to 2048. We employ 8 parallel attention heads. The number of layers for the encoder and decoder are 6. Other settings is same as Vaswani et al. [2017] . We use beam search for heuristic decoding, and the size is set as 4.

2 includes LDC2002E18, LDC2003E14, LDC2004T08,

LDC2005T06

We measure the translation quality with the IBM-BLEU score [Papineni et al., 2002]. We implement our interactive NMT model upon NJUNMT-pytorch 4 . The forward and backward decoders of the bi-directional model are trained together with a shared encoder. The learning rate of online learning is 10-5 . Following previous work [Tu et al., 2017], we train the revision memory by randomly sampling words and contexts in the same discourse. The size of revision memory is 100 and it will be used when revision number is more than 20. When the revision number is more than 100, the first part will be discarded until revision number is less than 100.

Ideal and Real Interactive Environments

Following previous work [Cheng et al., 2016; ´Alvaro Peris et al., 2016; Hokamp and Liu, 2017], we experiment on both the ideal and real environments. Because real-world human interactions are expensive and time-consuming to obtain, we first report results on the ideal environment, which generates simulated human interactions by identifying critical mistakes. The simulated critical mistake is those lead to the most significant BLEU score improvement after being corrected.

We also validate the interactive efficiency on the real environment, in which three human annotators are asked to revise the translation, until obtaining correct translations. The three human annotators are asked to work with all different systems with no idea of which system they are working with.

4.2 Results on the Ideal Environment

We first evaluate CAMIT for ZH-EN on the ideal interactive environment. Tan et al. [2017] demonstrate that two revisions already can lead to very good translation outputs, which is also verified in our experiments on the real interactive environment. We also report the results of our model after multiple interactions as a complement.

4 https://github.com/whr94621/NJUNMT-pytorch

Sequential Bi-directional Decoding

First, we valid our proposed bi-directional interactive model (BiDiR) in CAMIT on the RNN based NMT model (RNNSearch). As shown in Table 1, BiDiR significantly outperforms the uni-directional (UniDiR) counterpart, obtaining more than 10 absolute BLEU score improvements.

For fair comparison, we also report results of Hokamp and Liu [2017] (UniDiRG), which works in a uni-directional fashion, but using a grid beam search. This enables the unidirectional decoder to revise the critical mistake first as in our bi-directional decoder, but the generation process is also unidirection. The results for their model still fall behind ours in performance, and the gap may increase on the real environment. Because mistakes left to the revision still exist, and could not be fixed automatically. This harms their interactive efficiency.

We also apply BiDiR decoder on Transformer with the same setting of RNNSearch. By revision once, BiDiR achieves an absolute BLEU improvement of 6.93, upon a strong Transformer baseline (43.65 to 50.58), significantly outperforming UniDiR (+2.04) and UniDiRG (+4.57). We also compare RNNSearch and Transformer with our proposed bidirectional decoder in Table 2. Results show that although the baseline of Transformer is significantly better than RNNSearch, both of them give pretty high BLEU scores (around 60) after 4 revisions. RNNSearch even obtains higher BLEU scores. For convenience, we will only report results based on RNNSearch in followed experiments.

Learning from Interaction History

Table 3 shows that our baseline RNNSearch model obtains a significant BLEU score improvement of 4.1 on the ideal environment by learning from interaction history with both online learning (+OL) and revision memory (+RM). Additionally, we find that incorporating history learning into BiDiR can still give about 1 absolute BLEU score improvement (+0.95), which is achieved on a very strong baseline with over 50 BLEU score points. This experiment shows that learning from the interaction history is helpful to boost the translation performance. In such case, human will take less revision number for obtaining satisfactory translations.

Effectiveness in Different Domains

In order to verify the effectiveness of the CAMIT in different domains, we also employ the proposed method on IWSLT ZH-EN translation task. Results are shown in Table 4, compared with baseline, the proposed CAMIT model could achieve XX BLEU score improvement. Moreover, compared with UniDiR model, the CAMIT also gets a significantly gain.

4.3 Results on the Real Environment

We conduct experiments on the real environment for both EN-ZH and ZH-EN on the RNNSearch based system. We report the average results of 3 volunteers, who could revise arbitrary times until they feel satisfied for the translation. For better comparison, we include the results of Hokamp and Liu [2017] in Table 5. Our proposed model CAMIT gives 8.03 and 5.49 BLEU score improvements on ZH-EN and ENZH, respectively, with significantly 5 lower revision numbers than the uni-directional one, and taking less times.

We also compare the number of UNK in final translation outputs, between adopting revision memory or not. Table 6 shows that, the revision memory helps relatively decrease the UNK rates by 16.1%, reducing the number of UNK from 2052 to 1723. This indicates that the revision memory enables our model to remember the past revisions, which helps avoid making same mistakes, e.g., UNK.

5 P-values are below 0.01 using pairwise t-test.

4.4 Case Study

We list two real cases of the interactive translation from UniDiRG [Hokamp and Liu, 2017] and our propsed model in Table 7. The two cases are related, and the second case is the next sentence of the first one in a discourse. For the first one, the baseline NMT system makes several mistakes. Our proposed bi-directional interactive model corrects these errors by two revisions, while the uni-directional method takes three revisions. After correcting the word "filmmakers", our bi-directional model automatically fixes the mistake of "independent" left to the revision position while the uni-directional model cannot do this.

The second case shows the translation output after using online learning and revision memory, which could reduce the revision number in practice. After using the first revised sentence to tune the model by online training, and remembering past revisions by the revision memory, our model have more prior knowledge that the session is related to the topic of movie. So the revised word "redford" and "filmmakers" can be fixed automatically.

5 RelatedWork

Interactive machine translation has been widely exploited to improve the translation by using interaction feedback from human users [Langlais et al., 2000; Simard et al., 2007; Barrachina et al., 2009; Gonz´alez-Rubio et al., 2013; Cheng et al., 2016] in statistic machine translation (SMT) [Yamada and Knight, 2001; Koehn et al., 2003; Chiang, 2007].

Recently, researchers employ it in neural machine translation (NMT). Barrachina et al. [2009] , Gonz´alez-Rubio et al. [2013] and Knowles and Koehn [2016] present an interactive NMT model with the uni-directional interaction protocol (UniDiR), in which users can only interact with the model from left to right.

´Alvaro Peris et al. [2016] purpose a new protocol for NMT, which is different to UniDiR, using the average of word probability to make use of the feedback. Hokamp and Liu [2017] and Post and Vilar [2018] propose a grid beam search algorithm, in which users can be allowed to make any interaction on arbitrary position with the UniDiR model. But their interactive model also regenerate translations from left to right, which omits errors left to the revision. In our proposed model, human can revise the most critical mistake first in arbitrary position of the sentence; and after that the whole sentence will updated, fixing minor mistakes automatically.

Liu et al. [2016] propose the bi-directional decoding model to improve translation quality. However, in their approach, two decoders individually generate outputs. The translation output of our model is obtained by coupling the forward and backward decoders sequentially. Peris et al. [2017] propose online learning method to improve the performance of NMT, however, their model is independent of the interactive process. External information, such as word or char, has been verified to be effective in promoting translation [Chen et al., 2018; Zheng et al., 2018], but these methods can not employ on the interactive process directly. Our proposed revision memory is inspired by the CopyNet [Gu et al., 2016] and history cache [Tu et al., 2017], which only focus on better using source or global context in supervised learning, and is different from our model.

6 Conclusion

In this paper, we propose CAMIT, to improve the interactive efficiency of NMT models in both decoding and learning aspects. For decoding, we propose an sequential bi-directional decoder, updating the whole sentence after each revision. For learning, we exploit the interaction history for the sentence level and word level. Experiments show that our model significantly improve the efficiency of interactive NMT.

Acknowledgements

We would like to thank the anonymous reviewers for their insightful comments. Thanks to Zewei Sun and Zaixiang Zheng for their insightful comments. Thanks to the volunteers for their hard work on the human evaluation experiment. This work is supported by the National Science Foundation of China (No. U1836221, 61672277), the Jiangsu Provincial Research Foundation for Basic Research (No. BK20170074).