Automatically Paraphrasing via Sentence Reconstruction and Round-trip Translation

1 Introduction

The paraphrase of a sentence retains its meaning but makes different choices of words and expressions than the original form. Paraphrase generation plays an important role in many downstream tasks, such as question answering, machine translation, and information retrieval [Hu et al., 2019a].

Most existing parallel datasets for paraphrase generation are domain-specific. Quora and WikiAnswers [Fader et al., 2013] datasets only contain questions; sentences in MSCOCO [Lin et al., 2014] dataset are mostly descriptions for physical objects since they are the captions of images; and PPDB [Ganitkevitch et al., 2013] contains phrases rather than sentences. The performance of a model trained with these domain-specific parallel data declines seriously when it is used in another domain [Li et al., 2019].

Many efforts were made to solve this domain adaptation problem. These efforts are roughly divided into three directions: unsupervised fine-tuning for supervised model, unsupervised methods based on word/phrase replacement, and distantly-supervised methods based on bilingual data. Li et al. [2019] chose to fine-tune the supervised model with nonparallel in-domain data, but the performance of their model decreases a lot when the domain span is large. Liu et al. [2019] and Miao et al. [2019] used unsupervised methods to generate paraphrases, but their models are mostly based on the variation of words and phrases and can hardly change the structure of the whole sentence. Wieting and Gimpel [2017] generated paraphrases with a round-trip translation model, but the existing translation models are sometimes not very accurate, which also affects the performance of their method. Liu et al. [2020] use bilingual data to train an unsupervised model, but their improvement is mainly brought about by the follow-up supervised fine-tuning.

∗The corresponding author, supported by Alibaba Visiting Scholar Program and SJTU-CMB Joint Research Scheme.

In this paper, we propose a novel paraphrase generation framework that does not require any parallel paraphrase data and can be applied in any domain. 1 In our framework, two kinds of underlying semantics are extracted from the original sentence and are recombined into a new sentence through a hybrid decoder.

The first kind of underlying semantics is represented by a word set, which is inspired by the Denoising Auto-Encoder (DAE) [Vincent et al., 2008]. A bag of words is a great carrier of information, as it communicates the central idea without syntactic constraints. People can produce different sentences with similar meaning from the same set of words. Table 1 shows an example of such paraphrase sentences. We construct a word set from the original sentence and extend the word set into a complete sentence with a set-to-sequence (set2seq) model, which is adapted from the well-known sequence-tosequence (seq2seq) model by ignoring the sequential information from the input sequence.

The second carrier of semantics is the translation of the original sentence into another language. Semantics is preserved but syntactic perturbations are added when the translation is then translated back to the original language. This is known as round-trip translation [Wieting and Gimpel, 2017].

1 Code is available: https://github.com/Karlguo/paraphrase

The above two types of semantics are complementary. The round-trip translation makes up for the missing information in the set2seq model, such as sequential information. The set2seq model gives the round-trip translation model some lexical hints and makes the translation result more accurate. We thus integrate the decoding parts of the set2seq model and the round-trip translation model to jointly generate paraphrases.

We evaluate our framework on four paraphrasing datasets, namely Quora, WikiAnswers, MSCOCO, and Twitter [Lan et al., 2017], and achieve the state-of-the-art accuracies compared to existing models trained with non-parallel data.

We also train the set2seq model on a big common-domain dataset and test it on these four datasets, and still obtain decent results. We call the set2seq model trained from the big common-domain dataset "set2seq-common", and can apply it to any domain when there is no in-domain data to train a set2seq model.

Finally, we propose an application of our paraphrase generator: to augment the training data of a neural machine translation (NMT) model between low-resource languages and English. We paraphrase the English sentences in the parallel training pairs with set2seq-common and improve the BLEU score of X-to-English translation by 1.53 to 2.17, where X is a low-resource language.

In summary, the main contributions of this work are:

• We are the first to apply the set2seq model to the task of paraphrase generation by combining it with a round-trip translation model through a hybrid decoder.
• The framework proposed by us achieve state-of-the-art accuracies on four benchmark datasets compared with existing methods.
• We apply our method to augment the training data of low-resource translation tasks and obtain significant improvement in translation quality.

2 Approach

We first give an overview and then describe the detailed components of the framework.

2.1 Overview

The set2seq model and the two translation models used in round-trip translation are trained separately, and our framework is designed for use during inference time only. Figure 1 shows the architecture for our framework, which is divided into two major components and two major phases. The two components are sentence reconstruction based on word set, and round-trip translation. The two phases are information extraction and paraphrase generation.

Suppose the original sentence is in language L1 and the round-trip translation is via language L2. During information extraction phase, given an input sequence of tokens X = [x1, x2, · · · ], we process it in two different approaches to extract two different representations of the underlying semantics: a word set and a translation in language L2. For the former, we construct a word set WS = {w1,w2, · · · }. For the latter, we use a L1-L2 translation model to get a sequence of translated tokens Z = [z1, z2, · · · ] in L2.

In the paraphrase generation phase, we employ a hybrid decoder which takes inputs from two separate encoders, one from the set2seq model and the other from the L2-L1 translation model. We encode the word set WS and the L2 token sequence Z respectively to obtain two hidden states Hws and Hbt. The hybrid decoder maintains a single output sequence, generating one token at each step based on Hws, Hbt, and the previously generated tokens.

2.2 Word Set Constructor

We use the word set constructor to extract a word set from the original sentence. To ensure accuracy and diversity of sentences generated from the word set, the word set constructor tries to strike a balance between both content preservation and lexical variation.

For content preservation, we could select informative words from the original sentence by either removing stopwords or retaining high-IDF words to build the keywords set KWS, which will be passed to the next stage. Here, we choose to remove stopwords, the reason for which will be explained in the result analysis in Section 3.5

To increase the lexical diversity of the generated paraphrase, each word in KWS is randomly replaced with one of its synonyms usingWordNet [Miller, 1995], and optionally, itself. This process is known as "random replacement". We obtain WS after this step. BERT based methods, instead of WordNet, can also be used to generate synonyms. They are not used because: i) we have to generate synonyms for every single word in the training set, and it is too computational expensive if we use BERT; and ii) WordNet is good enough for generating high-quality word sets.

2.3 Set-to-Sequence

A set2seq model consists of an encoder and a decoder, similar to a seq2seq model. However, instead of taking a sequence as the input, the input of a set2seq model is a set of tokens with no sequential information.

To train a set2seq model, we prevent the encoder to do serial processing for the input set. RNN-based models are inappropriate for this purpose due to their recurrent nature. Therefore we use a transformer-based model. In transformer, the sequential information of the input sequence is captured in the position encoding. We use a transformer but omit the position encoding in the encoder as the set2seq model.

We train set2seq with word set WS as the input and original sentence X as the output. This training data is automatically created and thus the training process is considered self-supervised. Specifically, given a set of words WS = {w1, w2, · · · }, the set2seq model does the following steps in a single layer while encoding:

where hi+1 is the output of layer i and h0 is the embedding of tokens inWS.

et al. [2019], we randomly choose 20K parallel paraphrase pairs as the test set and 3K parallel paraphrase pairs as the validation set for Quora, WikiAnswers, and MSCOCO.

2.4 Hybrid Decoding

A hybrid decoder can take the hidden states of multiple encoders as input and generate a single output sequence based on the information from all hidden states.

As we mentioned before, we divide the framework into two components, the set2seq model and the round-trip translation model, and obtain two hidden states Hws and Hbt.

Assume that our vocabulary is V = {v1, v2, · · · , vD} with D different tokens. In decoding step t, the decoder of the set2seq and the L2-L1 translation model can give the probability of v being the next token individually. Supposing we already generated t - 1 tokens y1, y2, ...yt-1, the next token yt to be generated is given by the following equation:

Here Pws and Pbt are the probabilities of vi being the next token calculated by the decoder of the set2seq model and the L2-L1 translation model respectively, and λ is the hyperparameter to balance the weight between the two probabilities.

3 Experimental Results

In this section, we first introduce the experimental setup, including datasets, baselines, evaluation metrics, and implementation details. Then, we show the results the competing methods. Finally, we analyze the results from different aspects.

3.1 Datasets

We evaluate our framework on four different datasets, namely Quora, WikiAnswers, MSCOCO, and Twitter. Following Liu Training with In-domain Data. We randomly sample the remaining parallel paraphrase pairs and pick one sentence from each pair to construct the non-parallel training data. The number of selected sentences is the same as the work by Liu et al. [2019], which is 400K for Quora 2 , 500K for WikiAnswers, 320K for MSCOCO and 110K for Twitter.

Training with Common-Domain Data. When there is no sufficient available target-domain non-parallel data , it is hard to train unsupervised models or fine-tune supervised models in the target-domain. Our solution is to train the set2seq model with a big common-domain dataset and apply it to the targetdomain. We name the model "set2seq-common". We test the performance of our framework with set2seq-common on four datasets to show the generality of our framework. Further, we apply set2seq-common in the Application section.

3.2 Baselines and EvaluationMetrics

We compare our framework with five unsupervised/distantlysupervised methods and four supervised methods with domain adaptation. We re-produce ParaNMT [Wieting and Gimpel, 2017] and ParaBank [Hu et al., 2019b] using our translation models, and take the results from Liu et al. [2019] and Liu et al. [2020] for other baselines. For a fair comparison, we keep the scripts for data pre-processing and evaluation from UPSA. On the Quora dataset, we even use the same train-test split as UPSA. 3

2 https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs

3 https://github.com/anonymity-person/UPSA

Unsupervised and distantly-supervised methods. The current state-of-the-art unsupervised method is Unsupervised Paraphrasing by Simulated Annealing (UPSA) [Liu et al., 2019], which is also our main target of comparison. The other unsupervised methods is CGMH [Miao et al., 2019]. Distantly-supervised baselines are the unsupervised part by Liu et al. [Liu et al., 2020], ParaNMT and ParaBank(-3rd IDF). Note that ParaNMT used round-trip translation to generate paraphrases, so it can be viewed as "round-trip translation only".

Supervised methods with domain adaptation. Decomposable Neural Paraphrase Generation (DNPG) [Li et al., 2019] is the current state-of-the-art method for supervised paraphrase generation. Other baselines are Pointer-generator [See et al., 2017], Transformer [Vaswani et al., 2017] with copy mechanism, and MTL[Domhan and Hieber, 2017] with copy mechanism.

Evaluation metrics. For fair comparisons, we take the same evaluation metrics as in UPSA and DNPG, which are iBLEU [Sun and Zhou, 2012], BLEU [Papineni et al., 2002] and ROUGE [Lin, 2004] scores. BLEU and ROUGE scores are common evaluation metrics for NLG tasks while iBLEU is especially designed for paraphrase generation tasks. It penalizes the similarity between paraphrase and the original sentence. Suppose the input sentence is src, the output paraphrase is out, and the ground truth paraphrase is trg, we calculate iBLEU as follows:

BLEU and ROUGE only consider the accuracy but ignore the diversity of generated paraphrases, while iBLEU considers both. So we use iBLEU as our main evaluation metric. We set α = 0.9, same as other baselines.

3.3 Implementation and Training Details

To be consistent with the pre-processing of UPSA and DNPG, we convert the input words into lower-case and truncate all sentences to up to 20 words. For the convenience of hybrid decoding, we learn a shared byte-pair encoding (BPE, [Sennrich et al., 2016]) with size 50k from the training data for translation models, and use a 30K vocabulary for all models.

For the translation models in round-trip translation, we train them with the WMT17 4 zh-en dataset [Ziemski et al., 2016] with a standard transformer for 3 days on two GTX2080 GPUs. We reuse these translation models for ParaNMT and ParaBank. For the set2seq-common model, we use the news-crawl-2016 English monolingual data from WMT17 and train 1.5 days with a standard transformer. For the domainspecific set2seq models, we use a 2-layer transformer with 300 embedding size, 256 units, 1024 feed-forward dimensions for all layers to train them. The training lasts 3 hours on a single GTX-2080 GPU. Set2seq is a lightweight model with 31M parameters, 3.7M parameters for multi-head attention layers, only one-third of a standard transformer.

To calculate iBLEU and BLEU, four references are used for MSCOCO, five for WikiAnswers, and one for other datasets.

4 http://statmt.org/wmt17/translation-task.html

Some test cases in WikiAnswers may have fewer than 5 references. For ROUGE scores, we take the average score against all references.

3.4 Results

Table 2 presents our experimental results. We compare three different models with the previous methods, namely set2seq, set2seq-common+RTT, and set2seq+RTT, where RTT stands for round-trip translation. We show the set2seq alone here to demonstrate that the useful information comes not only from the translation, since the set2seq model alone can already outperform almost all competitors. Our framework outperforms all existing unsupervised methods, distantly-supervised methods, and supervised methods with domain adaptation.

For the hyper-parameter λ, when it is close to 0, the result is similar to the round-trip translation. When λ is between 0.4-0.8, the result is stable, and iBLEU is above 14. As λ goes to infinity, the result is slowly approaching that of set2seq. We set the value to 0.5 for all datasets after experimenting with difference choices.

3.5 Analysis

Datasets

Due to the domain-specific differences between four datasets, it is understandable that scores on all metrics vary a lot across different datasets. Paraphrases fromMSCOCO are descriptions of images, the set2seq model fits this dataset quite well since the process of generating paraphrases are similar: one extends information from a static picture; the other extends from a word set. Lack of training data for Twitter leads to insufficient training of most models. Models using round-trip translation perform extraordinary well since they have adequate information. Besides, set2seq-common+RTT achieves an excellent result, which shows the advantages of the set2seqcommon model compared with the set2seq model trained with insufficient in-domain data.

Ablation Study

Table 3 shows the result of the ablation study on the Quora dataset, where BLEUref is the BLEU between reference and output, the higher the better and BLEUsrc is the BLEU between source sentence and output, the lower the better. We demonstrate that removing stopwords is better than keeping high-IDF words. For high-IDF words, we keep the top k% high-IDF words in the original sentence. We set k = 50, the best from {30, 40, 50, 60, 70} by empirics. We also tried TextRank [Mihalcea and Tarau, 2004] to score words and get similar results with IDF scores. Removing random replacement and adding position encoding can both give high BLEUs between reference sentences and output paraphrases, but substantially reduce the diversity of the generated sentences.

Human Evaluation

We choose 100 sentences from Quora and ask 3 human annotators to score the results from different methods blindly on a scale of 1 to 5 according to fluency and accuracy (the higher the better). Fluency measures whether the paraphrase conforms to grammar and common sense; accuracy measures whether the paraphrase has the same meaning as the original sentence though in a different expression.

We can see that word/phrase based methods have bad performances on fluency since their language model is trained on a small dataset. Paraphrases generated by round-trip translation are not very accurate since they are not trained by in-domain data. By both fluency and accuracy, our method performs the overall best.

4 Application on Translation Tasks

We apply our paraphrase generator to augment the training data of X-English translation task, where X is a low-resource language. Since it is difficult to find high-quality test sets for low-resource languages, we use three commonly-studied languages and reduce their parallel training pairs to 150k and 300k to simulate low-resource languages.

4.1 Data Augmentation

The size of training pairs for NMT tasks is of great importance to the final result. However, it is quite expensive to enlarge the dataset manually due to the high labor price and the huge workload, so data augmentation for NMT is a popular research topic in recent years. To this end, in the task of translating lowresource languages to English, we paraphrase the sentence in English to augment training pairs automatically.

For each language, we carry on two experiments with 150k data and 300k data respectively. For each experiment, we train the model with original data as the baseline. For each experiment, we train the model with the original data as the baseline.

Regarding augmentation, we make 10 copies of the original sentences, construct 10 word sets with different seeds in random replacement from the 10 copies and generate 10 paraphrases with set2seq-common+RTT. To increase the diversity of the results, we use random sampling [Edunov et al., 2018] during decoding. We take the 10 copies and 10 paraphrases as the augmented data.

For the set2seq-common model, since the sentences in the NMT training set is longer, we truncate all sentences to 50 words instead of 20 during the training stage and do not truncate any sentences during the inference stage.

4.2 Experimental Setup and Results

We experiment on German-English (de-en), Chinese-English (zh-en), and Russian-English (ru-en) translation pairs. For the training data, we obtain the de-en data from WMT17europarl 5 [Koehn, 2005], and the ru-en data from WMT17 news-commentary and zh-en data from LDC [Liberman, 2002; Huang et al., 2002]. The reason for not using zh-en data from WMT17 is that we are already using the zh-en pairs from WMT17 to train the translation models. For test sets, there are 3004 pairs for de-en, 2000 pairs for zh-en and 3000 pairs for ru-en from the WMT17 news-test.

For each language, we learn a shared BPE of size 50,000 and extract vocabulary of up to 50,000 from the training set.

We train translation models with a standard transformerbase model [Vaswani et al., 2017]. We take the average of test results from 5 checkpoints after convergence.

Table 5 shows the result. Paraphrase augmentation improves the model trained with original data pairs by anywhere from 1.53 to 2.17 BLEU.

When producing paraphrases, our methods do use additional data, such as monolingual English data and Chinese English translation data. It is conceivable that there exists other advanced NMT methods that use these data in different ways. However, the purpose of this section is to show the effectiveness of our long-sentence paraphrase generation methods, since only accurate and diverse paraphrases can be used as good translation pairs and subsequently train good translators.

5 http://www.statmt.org/europarl/

5 RelatedWork

We show the relevant work of paraphrase generation from the aspects of supervised, distance-supervised, and unsupervised methods.

For supervised methods, Prakash et al. [2016] proposed "stacked residual LSTM" as the earliest deep-learning method in this topic, seq2seq models like transformer [Vaswani et al., 2017] and MTL [Domhan and Hieber, 2017] outperformed many methods due to the advantages of their model structures. We include these well-known methods in our baseline. Li et al. [2019] proposed the current state-of-the-art method DNPG and revealed the disadvantage of supervised methods when it comes to domain adaptation. Other methods include VAESVG [Gupta et al., 2018] and transformer-pb [Wang et al., 2019], but these methods perform worse than DNPG and have no discussion about domain adaptation, so we do not include them in our baselines.

For distance-supervised methods, Wieting and Gimpel [2017] created a 50M parallel dataset for paraphrases with round-trip translation, Hu et al. [2019b] used lexicallyconstrained to improve the diversity of generated paraphrase, and their work is proved to be useful for many downstream tasks like Natural Language Inference [Hu et al., 2019a]. Liu et al. [2020] also use bilingual data to generate paraphrase without parallel data. However, their focus is on the supervised fine-tuning part. Their method do not performs well without the fine-tuning part.

For unsupervised methods, Miao et al. [2019] used Metropolis-Hastings Sampling to generate paraphrases, Liu et al. [2019] generated paraphrases with Simulated Annealing, both of them were the best at their times. We compare our framework with these two methods to show changes on the sentential level are more reliable than changes on the lexical level. Siddique et al. [2020] proposed a method for paraphrasing with deep reinforcement learning. However, we do not regard it as a baseline since their results are not convincing enough for the following two reasons:

• By iBLEU in (4), their BLEUsrc is 28.04 on Quora, and 91.98 on WikiAnswers, which shows a very large and abnormal disparity.
• The authors provided us with their test set on Quora, where the BLEU score between source sentences and references is 69.75. However, the score should be around 25 if the test samples are randomly selected fromQuora. With their test set, it is easier to generate paraphrases similar to references.

6 Conclusion

In this paper, we proposed a novel framework for automatical paraphrase generation without parallel training data. It outperforms most existing unsupervised and distantly supervised methods. While the results are positive, some questions remain. Can we find more underlying semantics to represent the input sentence? Can we replace the round-trip translation model with a lighter-weight model? We plan to look into these questions in the future and generate better paraphrases.