Improving Text Generation with Dynamic Masking and Recovering

1 Introduction

Recent years have seen growing interest in various text generation tasks which aim to generate mostly-grammatical natural language text from diverse input forms. Representative text generation tasks include machine translation [Vaswani et al., 2017] which generates text from an input word sequence, AMR-to-text generation [Zhu et al., 2019] which generates text from a semantic graph, concept-to-text [Mei et al., 2016] which generates text from structured data records, and image captioning [Ren et al., 2016] which generates text description from an image.

Most existing approaches to text generation adopt the widely used encoder-decoder framework: the encoder takes structured or unstructured data as input and returns a sequence of distributed representations, from which the decoder generates text as output. Previous studies on a variety of text generation tasks have shown the effectiveness of the framework. However, training a robust text generation model usually requires large-scale training data. To alleviate such requirement, one feasible way is to introduce more uncertainty to overcome overfitting during training [Dhillon et al., 2018], especially when the training data is limited in size. Techniques in this line include drop-out and randomness. Another possible way is to enlarge the training data through techniques like back translation, data augmentation. In this paper we propose a simple yet effective approach that can achieve these two goals simultaneously.

∗Corresponding Author

The proposed approach is inspired by the idea of masked language modeling [Devlin et al., 2018]. To train the models of text generation tasks, we introduce randomness by randomly masking some percentage of tokens on the decoder side. In this way, different from a standard decoder which uses ground truth history context, we use its masked (corrupted) version to predict the next token. The advantage of using corrupted history context is two-fold. On the one hand, with dynamic masking strategy which generates a new random masking pattern every time we feed a sequence to the decoder, we enable the decoder to see more training instances during training. On the other hand, the discrepancy of history context between training and inference is a classic issue in text generation [Zhang et al., 2019]. Using non-ground truth words as history context narrows the gap between training and inference. Upon the masked sequence, we further propose an auxiliary task to properly recover those masked tokens, just as masked language model does. Therefore, our approach jointly maximizes both the likelihoods of both sentence generation and prediction of masked tokens.

We verify the effectiveness and generality of our approach on three types of text generation tasks which use various forms of input data including text, graph, and image. For sequence-to-sequence (seq2seq) generation task (specifically, machine translation), our model obtains significant improvement of 1.01 and 0.90 BLEU scores over competitive baseline on IWSLT14 German→English and WMT14 English→German, respectively. For graph-to-text generation task (specifically, AMR-to-text), our approach achieves a significant improvement of 1.39 BLEU score over the stateof-the-art on AMR 2.0. Finally, for image-to-text generation task (specifically, image captioning), our results on the MSCOCO test set show improvement of 1.5, 0.5, and 0.9 in CIDEr, SPICE, and BLEU-4 scores, respectively. Our main contributions can be summarized as follows:

• We add a novel objective to force the models of text generation not only predict the target sentence, but also resume the original tokens, which could enhance the language representation for text generation.
• Our approach is applicable to arbitrary types of input and easily transparent to other encoder-decoder architectures. We do not manipulate the encoder and only mask a number of tokens from target sentence.
• Experimental results show that our method outperforms all baselines on a variety of text generation tasks.

2 Methodology

Since our goal is to improve robustness of encoder-decoder models by focusing on the decoder part, we keep the encoders unchanged.

2.1 Model Architecture

A text generation model basically consists of an encoder and an decoder, as shown in Figure 1(a). Depending on the format of input, the encoders of text generation tasks may be very different. We use X to indicate the input.

We use Y = (y1, · · · , yT ) to indicate the output, while T is the length of sequence Y . For simplification, we uniform their decoders as Transformer-based, which consist of a stack of identical decoder layers. Each decoder layer has three sublayers. The first is a masked multi-head self-attention mechanism, the second performs multi-head attention over the output of the encoder stack while the third is a simple, positionwise fully connected feed-forward network. As shown in Figure 1(a), at each decoding time t, the predictions for position yt can depend on source-side input X and target-side ground truth history context y<t.

where Θ is model parameters. The model is trained by maximum likelihood estimate, i.e., minimizing the negative log likelihood loss:

Figure 1(b) illustrates our proposed decoder, which falls into a multi-task learning framework. Compared to the baseline decoder, our proposed decoder have two differences: 1) rather than using the gold target-side history context to predict Y , we use a corrupted version; 2) we introduce a masked language model on the target side. Next, we present the proposed decoder in details.

2.2 Dynamic Masking

In order to obtain more training samples efficiently and cheaply, we corrupt a given word sentence Y = (y1, · · · , yTy ,<eos>) and obtain a noise version Y ′ = (y′1 , · · · , y′Ty ,<eos>). To this end, we follow BERT [Devlin et al., 2018] and randomly sample 15% of tokens in Y . The selected tokens are (1) 80% of time replaced by a mask token [MASK], or (2) 10% of time replaced by a random token, or (3) 10% of time unchanged. Consequently, we update Eq. 1 and Eq. 2 as:

That is to say, rather than using ground truth history context y<t, we now use its corrupted version y′<t to predict yt at the t-th time step.

Note that BERT performs random masking and replacement once in data pre-processing. To avoid using the same mask for each training instance in every epoch, we generate the masking pattern every time we feed a sequence to the model. For example, if the training process finishes after K epochs, each training sequence would be seen with the K different corrupted versions.

2.3 Masked Language Model

Now the history context may contain noise tokens. Besides the text generation task, which predicts the next word based on history context, we propose another task, masked language model, which predict those noised tokens. Unlike masked language model in BERT which uses bidirectional context, here we use the left context and allow the model to see itself, i.e., the [MASK] token. Figure 2 gives a concrete example where the first target word (i.e., the) is not masked, i.e., y′1 = y1 while the second y2 (i.e., boy) is masked, i.e., y′2 = [MASK]. At time steps t = 1 and t = 2, the text generation task predicts y1 and y2, respectively while the masked language model predicts nothing. At time step t = 3, the former task predicts y3 while the latter task tries to predict y2 itself since y2 is masked in the history context. Formally, the loss function for the masked language model task is defined as:

where 1 (y′t ̸= yt) returns 1 if y′t is different from yt, otherwise 0.

2.4 Jointly Learning

As shown in Figure 1(b), the tasks of text generation and masked language model are jointly learned via a hard parameter sharing multi-task learning framework. Specifically, they share N-M decoder layers while have M task-specific decoder layers each. The joint loss is shown as Eq. 6.

where α is a hyperparameter used in controlling the weight of masked language model task.

Also note that we introduce dynamic masking and masked language model task only in the training stage.

3 Experiments

To evaluate the effect and generality of our proposed approach, we conduct experiments on several representative text generation tasks, including machine translation, AMR-to-text generation, and image captioning. Note that those tasks have different types of inputs, thus employ different encoders, i.e., sequential, graph-based, and CNN-based, respectively. For each task, we evaluate the effect of dynamic masking (DM), and its combination with the masked language model (MLM) task. For the decoder layers, we set N to 6 and set M to 1 for all experiments, as shown in Figure 1. The M task-specific decoder layers do not share parameters. That is to say, we share the N -M decoder layers in the bottom while the two prediction tasks have their respective M decoder layers in the upper. For the two tasks, we use the same final linear transformation layers to convert output logits to token probabilities. Moreover, we report results of single models that are trained at most 300K iterations and tuned on the respective development sets.

3.1 Machine Translation

Dataset and evaluation. For machine translation, we evaluate our approach on two widely used benchmarks: WMT14 English→German (WMT14 EN-DE) 1 and IWSLT14 German→English (IWSLT14 DE-EN). 2 WMT14 EN-DE consists of 3.9M training sentence pairs after filtering out long and imbalanced pairs. We use newstest2013 and newstest2014 as the validation and test set respectively. For IWSLT14 DE-EN, we conduct the same data cleanup and train/dev splitting as [Ott et al., 2019], resulting 160K parallel sentence pairs for training and 7,284 sentence pairs for development. We combine tst2010, tst2011, tst2012, dev2010, and dev2012 as our testing data. We tokenize all sentences with Moses scripts. 3 Then we segment words into subwords by using byte-pair encoding (BPE) [Sennrich et al., 2016b]. The sizes of resulting vocabularies shared by the source and target language are 32K and 10K for respectively WMT14 EN-DE and IWSLT14 DE-EN. For evaluation, we utilize multi-bleu.perl to report BLEU scores, as in [Papineni et al., 2002].

Experimental settings. We use OpenNMT [Klein et al., 2017] as the implementation of the Transformer model. 4

Results. Table 1 shows BLEU scores on WMT14 EN-DE and IWSLT14 DE-EN. From the results we can see that without introducing any parameters, dynamic masking achieves significant improvement of 0.61 and 0.66 BLEU scores for the two translation tasks, respectively. This suggests that our approach is effective even when the training data is big, e.g., 3.9M for WMT14 EN-DE. One particularly nice property of dynamic masking is that it is incredibly easy to implement. By introducing a decoder layer with 4.3M parameters, the masked language model task further improves translation performance. Our approach achieves final improvement of 0.90 and 1.01 BLEU scores over the Transformer baseline.

1 https://wit3.fbk.eu

2 http://statmt.org/wmt14/translation-task.html

3 http://www.statmt.org/moses/

4 https://github.com/OpenNMT/OpenNMT-py

3.2 AMR-to-text Generation

Dataset and evaluation. Following previous studies on AMR-to-text, we use the benchmark dataset AMR2.0 (LDC2017T10), which contains 36,521/1,368/1,371 training/development/testing sentences with corresponding AMR graphs. Following Zhu et al. [2019] and Ge et al. [2019], we segment tokens in AMR graphs and words in sentences into subwords by BPE with 10K operations and form a shared vocabulary for the source and target side. For evaluation, we follow related studies and adopt three different metrics: BLEU, Meteor [Banerjee and Lavie, 2005], and chrF++ [Popovi´c, 2017].

Experimental settings. We use self-attention-based graph Transformer [Zhu et al., 2019] as our baseline system, 5 which achieves the state-of-the-art performance for AMR-to-text generation. We follow the parameter settings in [Zhu et al., 2019].

Results. Table 2 presents the detailed results of AMR-totext generation on AMR 2.0. From the results we can see that both dynamic masking and masked language model task succeed to achieve improvement according to the three evaluation metrics. For example, dynamic masking achieves a significant improvement of 0.87 BLEU scores over the baseline and adding masked language model task gives a further improvement of 0.52 BLEU scores.

3.3 Image Caption Generation

Dataset and evaluation. For image caption generation, we experiment with the widely used dataset MSCOCO 2014 [Lin et al., 2014]. The popular Karpathy splitting [Karpathy and Fei-Fei, 2015] is adopted, which results in 113,287 images for training, 5K images for validation, and 5K images for testing. Each of the images has five corresponding captions. We follow standard practice and perform only minimal text preprocessing that converts all sentences to lower case, tokenizes on white spaces, and filters words that occur less than five times. This way, we obtain a vocabulary of 9,487 words for this task. All sentences are truncated to contain at most 16 words during training. For evaluation, we employ several standard metrics, including SPICE [Anderson et al., 2016], CIDEr [Vedantam et al., 2015], METEOR, ROUGE-L [Lin, 2004], and BLEU.

5 https://github.com/Amazing-J/structural-transformer

Experimental settings. Following previous studies, we first train Faster R-CNN [Ren et al., 2016] on Visual Genome dataset [Krishna et al., 2017] to identify and localize instances of objects. For each image, we take the global average pooling of the final convolutional layer output, which results in a vector of 2,048 dimensions. In practice, these spatial features are extracted before the training of captioning models and are fixed during training. For captioning models, the dimension sizes of Transformer hidden states, image feature embeddings, and word embeddings are all set to 512.

Results. Table 3 presents the results of various single models on MSCOCO. From the results we can see that dynamic masking achieves improvement over the baseline and adding the task of masked language model obtains further improvement. The conclusion comes true regarding all the evaluation metrics.

4 Analysis

Next, we provide a series of analyses on a number of key factors that may affect the system performance. Such analyses help to better understand relative importance of the factors.

4.1 Effect of Different Noisy Strategies

We compare the dynamic masking strategy with two other noisy strategies.

Static masking (SM). Following BERT, we perform random masking during data preprocessing. Therefore, each training example is seen with fixed masks no matter how many times it is used in the training stage.

Gaussian noise (GN). Following Cheng et al. [2018], we modify the target sentence at feature level. Specifically, given a target sentence, we add Gaussian noises to word embeddings according to the following distribution:

where E[yi] is the word embedding for yi, vector ϵ is a sample from a Gaussian distribution with variance σ2 . σ is a hyperparameter which we set to 0.01. We simply inject Gaussian noises to all of word embeddings when we feed a target sentence to decoder. Note that we perform Gaussian noise injection every time we feed a sequence to the model. Thus, each training example would be seen with K different corrupted versions, where K is the number of epochs in training stage.

Results. Table 4 compares the performance of different noisy strategies. From the results, we observe that:

• Static masking achieves an improvement of 0.6 BLEU score on IWSLT14 DE-EN. However, it has very limited effect in the other three text generation tasks.
• Based on static masking, the masked language model also fails to bring further improvement in the text generation tasks, except AMR-to-text generation.
• Although adding Gaussian noises to word embedding has positive impact across all text generation tasks, the effect is quite limited (improvement of 0.1 ∼ 0.3 BLEU score).
• Finally, our approach outperforms both static masking and Gaussian noise across all the text generation tasks.

4.2 Effect of Hyperparameter α

We explore the effect of hyperparameter α in Eq. 6, which controls the weight of the proposed task of masked language model. Table 5 presents the BLEU scores on the respective development set of the text generation tasks when α ranges from 0.0 to 1.0. From the results in Table 5, we find that for different text generation tasks, best performance is achieved with different values of α smaller than 0.7, which suggests that the optimal value of α is irrelevant to the task or the size of training data. Moreover, it shows that the task of masked language model brings more improvement on AMR-to-text than others. This is probably due to the smaller training data of AMR-to-text generation.

4.3 Effect of Sampled Token Ratio

An important hyperparameter in dynamic masking is the sampled token ratio. A low ratio makes the history context less different from gold one while sampling tokens at a high ratio makes the history context have more noise. Taking AMRto-text generation as example, we compare four variants of sampled token ratio (10%, 15%, 20%, and 25%). As shown in Table 6, we see that the best performance of different metrics always appears at the ratio of 15%.

4.4 Convergence Analysis

Introducing the task of masking language model to predict the masked token can not only achieve better performance, but also help the model converge faster. One reason is that compared to standard text generation tasks, our approach roughly makes prediction on additional 15% tokens in each batch.

Figure 3 shows the BLEU scores on the development set of AMR-to-text generation over different training iterations. Comparing the baseline (e.g., Graph Transformer) to the system with dynamic masking (e.g., + DM), we observe that the performance of + DM is lower than that of the baseline in the first 100K iterations. Then it starts to surpass the baseline after 100K iterations. With the task of masking language model, the system of + DM + MLM consistently outperforms both the baseline and + DM from the beginning. Moreover, it converges at about 200K iterations while the other two at about 300K iterations, suggesting fewer training iterations may be required for our approach to converge.

5 RelatedWork

We describe related work from the following two perspectives: robust text generation and masked language modeling.

5.1 Robust Text Generation

Using neural networks to generate natural language texts has been widely studied, ranging from machine translation to dialogue generation, AMR-to-text and image captioning. However, neural networks often suffer from vulnerability in the sense that small perturbations in various parameters or inputs can nevertheless result in different and often incorrect output.

To alleviate the effect of noisy perturbations and enhance the robustness of machine translation systems, Belinkov and Bisk [2017] design structure-invariant representations and resort to robust adversarial training. Cheng et al. [2018] introduce adversarial stability training to improve the robustness on arbitrary noise type. Sennrich et al. [2016a] and He et al. [2016] improve the robustness of machine translation models with monolingual corpora with the aid of an inverse model. Different from prior works, we do not resort to methods like adversarial training or monolingual data to improve robustness. Instead, we introduce randomness by masking tokens in the target side of training data. The idea is similar to Zhong et al. [2020] in spirit which improves the robustness of image processing by randomly erasing a rectangle region in an image. Such a method is simple and universal enough to be applied to arbitrary encoder-decoder architectures.

5.2 Masked Language Modeling

Our approach is inspired by masked language modeling (MLM), which is first adopted in a novel pre-training model to learn deep bidirectional language representations [Devlin et al., 2018]. Subsequently, the same technique has become an essential component in many state-of-the-art pre-training methods [Lample and Conneau, 2019; Chen et al., 2021a; Chen et al., 2021b]. Briefly speaking, MLM randomly masks out a subset of tokens in the input sentences and is optimized to predict the masked tokens according to residual tokens.

To adapt MLM to the seq2seq framework, Song et al. [2019] (MASS) andWang et al. [2019] (PoDA) propose to feed the encoder with a masked sequence and the decoder sequentially generates the masked tokens word-by-word. This seq2seq MLM can benefit the seq2seq-style downstream tasks, such as dialogue generation, machine translation, and summarization. However, for text generation tasks, the input to the encoder may be beyond token sequence. Considering the diversity of text generation tasks, the input may be table, graph, or even image. As a result, it is difficult to mask the different types of input. This is the reason that we propose to mask the output sentence of decoder, based on which we add a simple network to predict the masked tokens.

6 Conclusion

In this paper, we have proposed a general approach to improve the performance of text generation tasks. This is done from two levels. To train the models, we first have introduced randomness by randomly masking some percentage of tokens on the decoder side. This strengths the decoder by asking it to predict next tokens with rather than ground truth history context, but corrupted one. Meanwhile, we have proposed an auxiliary task to recover those masked tokens via a multi-task learning framework. Experimental results on various text generation tasks have demonstrated the effectiveness and generality of our proposed approach.

Acknowledgments

This work was supported by the National Key R&D Program of China under Grant No. 2020AAA0108600 and by the National Natural Science Foundation of China under Grant No. 61876120.