Exploring Bilingual Parallel Corpora for Syntactically Controllable Paraphrase Generation

1 Introduction

Paraphrase generation (PG) creates different expressions that share the same meaning (e.g., "how far is Earth from Sun" and "what is the distance between Sun and Earth"). It is a crucial technology in many downstream natural language processing (NLP) applications such as question answering x: we need to further strengthen the agency's capacities.

∗Contribution during internship at Lenovo Research AI Lab.

†Corresponding author.

• z: the damage in this area seems to be quite minimal.
• y: the capacity of this office needs to be reinforced even further.

x: his teammates' eyes got an ugly, hostile expression.

• z: the smell of flowers was thick and sweet.
• y: the eyes of his teammates had turned ugly and hostile.

Figure 1: Illustration of generating syntactically-controllable paraphrases with sentential exemplars, which transforms a given input x to a new sentence y that is semantically similar to x but in a different syntactic style like z.

[Dong et al., 2017], machine translation [Zhou et al., 2018], and text summarization [Zhao et al., 2018].

Most recent state-of-the-art approaches to PG employ neural architectures [Prakash et al., 2016; Hasan et al., 2016; Gupta et al., 2018], which normally depend on a large amount of manually annotated paraphrase corpus for training. As constructing a large paraphrase corpus is inevitably not cheap and time-consuming, quickly developing a high-quality PG system on a small corpus mount a formidable practical challenge in many languages and domains. An effective solution to this problem is transferring knowledge from a highresource task with abundant annotated data to a low-resource task with limited or even no annotated data.

Recent progress has also witnessed that learning controllable paraphrase generation (CPG) with desirable styles is emerging as an area of intense focus in the literature, e.g., satisfying particular sentiment, template or syntactic structure [Ficler and Goldberg, 2017; John et al., 2018; Iyyer et al., 2018; Chen et al., 2019]. This technique has benefited several NLP tasks, such as generating diverse and adversarial samples to improve model generalization capability and robustness [Iyyer et al., 2018]. However, CPG aggravates the requirement of data annotation as guidance signals such as syntactic templates are needed [Chen et al., 2019].

In this work, following recent efforts, we focus on using a sentential exemplar to control the syntactic realization of generated sentences [Wang et al., 2019; Chen et al., 2019], as shown in Figure 1. But unlike existing CPG models that rely on large annotated paraphrase corpora, we are interested in developing a high-quality CPG model even without annotated paraphrase data.

As there are plenty of large-scale bilingual parallel corpora which can be regarded as paraphrases written in different languages, we propose to explore large-scale off-the-shelf bilingual corpora for CPG. We find a way to transfer knowledge in bilingual paraphrases into monolingual paraphrase generation and use sentences from bilingual corpora as syntactical exemplars, avoiding annotation work in manually creating paraphrases and guiding exemplars. Our method is therefore able to reduce the reliance of CPG on large paraphrase data.

Specifically, we propose to extend the widely-used encoder-decoder model [Bahdanau et al., 2014] to include a content encoder for meaning modeling, a style encoder for style extraction from syntactical exemplars and two syntaxguided variational decoders. By projecting input sentences into the same space via cross-lingual embeddings, we share both of the content and style encoder across different languages. This sharing mechanism enables the model to be trained on bilingual parallel data and monolingual paraphrase data simultaneously, allowing knowledge transfer from bilingual sentence pairs to PG. To disentangle the content from style space, we employ an adversarial loss over the learned content and style representations. To provide global syntactic guiding signal for the two decoders, we introduce a variational latent variable to model style representation. Additionally, we introduce a masking learning scheme to reduce the dependence of the model on token representations of the syntactic exemplar, and therefore to encourage the style encoder to extract more syntax-related information.

In order to enable our model to generate paraphrases by only learning from bilingual parallel data without using any annotated monolingual paraphrases, we further propose a cycle learning scheme that uses the learned model to generate pseudo paraphrase and translation data, and updates the learned model by training it on the generated data. The strategy allows paraphrase generation and machine translation to benefit each other in a cycle, which forms the basis for our proposed multi-task learning framework for the two tasks.

In summary, our contributions are threefold as follows:

• We propose a new controllable paraphrase generation framework with syntactical exemplars, which benefits PG from bilingual parallel corpora.
• In order to enable the model to explore bilingual corpora for paraphrase generation, we propose to equip the model with several vital components and learning strategies in a new way: (1) two shared encoders for disentangled content and style representations, enhanced by crosslingual embeddings to create the same space for different languages and an adversarial discriminator to distinguish content and syntactic style; (2) syntax-guided variational decoders that generate a sentence from both the encoded content representation and a latent syntactic variable learned by variational autoencoder; and (3) a translation-generation cycle learning scheme that enhances PG from translation and vice versa.
• Experiment results show that our model can generate diverse paraphrases with desirable semantic content and syntactic style. When the model is fine-tuned on a small annotated paraphrase data, it substantially outperforms previous state-of-the-art approaches.

2 RelatedWork

Neural paraphrase generation (PG) is often formalized as a sequence-to-sequence (Seq2Seq) learning formalism. Prakash et al. [2016] employ a stacked residual LSTM network in the Seq2Seq model to enlarge the model capacity. Hasan et al. [2016] incorporate the attention mechanism [Bahdanau et al., 2014] to generate paraphrases. Gupta et al. [2018] use a variational autoencoder framework to generate diverse paraphrases. Despite their success, the performance of their models suffers from small training data. Different from their work, we substantially benefit PG from large-scale bilingual corpora, and therefore alleviate the heavy reliance of PG on annotated paraphrase data.

Our approach also relates to recent works on style transfer and controllable text generation. Many previous methods attempt to control the attributes of generated texts such as sentiment and formality [Ficler and Goldberg, 2017; John et al., 2018; Lample et al., 2018], and also to control the structural aspects of generated sentences [Wiseman et al., 2018; Iyyer et al., 2018]. More recent works use sentences as exemplars to graft their syntax patterns to other sentences [Wang et al., 2019; Chen et al., 2019]. Similar to them, we also use a sentential exemplar to provide a syntax form for the generation. However, significantly different from them, we extend the model to use bilingual corpora so that our model can generate paraphrases with the desirable syntactic styles even without using any annotated paraphrase training data.

3 The ProposedModel

In this section, we elaborate our proposed model, including its essential components and their working mechanisms.

3.1 Overall Architecture

Our model is illustrated in Figure 2. Its backbone is built on the encoder-decoder architecture [Bahdanau et al., 2014].

The model has two encoders: a content encoder Encc for encoding the content of a sentence and a style encoder Encs for modeling the syntactic style of the given exemplar. The two encoders are shared across the two different languages in the bilingual training corpora. We use cross-lingual word embeddings to bridge the two different languages, and introduce a discriminator with adversarial loss on the two encoders to enforce the separation of the style and content spaces. We also have two syntax-guided variational decoders Decx and Decy , each of which is to generate sentences in one language with the guidance of syntactic exemplars. In this work, we use a two-layer bidirectional RNN in each encoder, and two-layer unidirectional RNN in each decoder. All RNNs use LSTM cells [Hochreiter and Schmidhuber, 1997].

We formulate the problem of syntactic-controlled paraphrase generation as follows. Given two sentences x and z as input, we want the generated sentence y from our CPG model to inherit the meaning of sentence x and be in the syntactic form of sentence z. We refer to x and z as the semantic template and syntactic exemplar, respectively.

We can use bilingual parallel sentence pairs (x, y) to train our model. z can be any valid sentence from the same language of y, but we set z = y at training procedure to avoid constructing additional syntactical exemplars. In this way, we want our model to learn for y the meaning of x from the content encoder and the syntactic style of y itself from the disentangled style encoder.

Concretely, for the two languages X and Y in bilingual training data, the model is trained in two directions X→Y and Y→X. We use (x, z; y) to train X→Y, where x and z are input to the content and style encoder and y is the ground truth for the Y language decoder. Similarly, we use (y, z; x) to train Y→X. To save space, we use the X→Y direction to introduce our model thereafter and all computations in the other direction can be derived accordingly.

3.2 Content and Style Encoders

As we use the two encoders Encc and Encs for both X and Y language, we project sentences from the two languages into the same content and style spaces. For this, we use the pre-trained cross-lingual embeddings that are kept fixed during training as the input of the two encoders. Concretely, we use word2vec to train word embeddings for each language on monolingual corpus. After that, we employ the unsupervised self-learning method [Artetxe et al., 2017] to obtain crosslingual embeddings in the same space.

3.3 Adversarial Discriminator

Inspired by previous work [John et al., 2018], we design a discriminator to enforce the separation of the content from the style space. In particular, the discriminator predicts a content label when the input from the content space, while a style label when the input from the style space.

Formally, for each sentence, let s be the vector representation learned by either the content or style encoder, we first use the mean pooling operation to average across each position, which is followed by a feed-forward neural network (FFN). After that, a two-way softmax layer is applied to s, given by:

where θdis = [Wdis, bdis] are parameters, and ys indicates the probability that the input is from the content or style space.

The discriminator is trained with a cross-entropy loss against the ground-truth distributions ts(·):

where θEncc and θEncs are parameters of the two encoders, l ∈ {content, style} and ys(l) is the predicted distribution.

3.4 Syntax-Guided Variational Decoders

We use two non-shared decoders (Decx ) and (Decy ), one decoder per language. Each decoder generates sentences in the corresponding language based on the semantic and syntactic representations learned by the encoders. To bridge the decoders with the content encoder, we use the attention mechanism with bilinear product [Bahdanau et al., 2014] to compute semantic representation hc. To incorporate information from the style encoder into the decoders, we introduce a latent variable z based on variational autoencoder (VAE) [Kingma et al., 2014] to model the underlying syntactic style as a global signal for generation.

In particular, because of the nature of bilinguality in the style encoder shared by the two languages, we make the latent variable z have the same distribution across the two different languages in shared space. Following the setting of VAE, we use KL-divergence to encourage the posterior distribution q(z|y) to be close to the prior p(z|x, y), where the prior p(z|x, y) is modeled from two languages.

Formally, the joint training objective for a training instance (x, y) is defined as follows:

where KL stands for KL divergence between the posterior q(z|y) and the prior p(z|x, y). The p(yt|y1:t-1, z, x) is the decoder with the guidance from z, where yt is the t-th word of y and p(yt|y1:t-1, z, x) is given by a softmax over a vocabulary V. The details of the syntax-guided variational decoder are given below.

Latent Syntactic Variable with VAE. We model the posterior and prior with the multivariate Gaussian distribution, which allows the latent syntactic variable to be in a continuous space [Bowman et al., 2015]. We use two sentences x and y to compute the posterior q(z|y) and prior p(z|x, y):

where the mean µ and s.d. σ of the approximate posterior and prior are the outputs of the style encoder.

Concretely, for the output of the style encoder, we use the mean pooling operation to average across each position to get hxs and hys . Then we use two FFNs to project the hxs and hys into a latent space for computing the posterior and prior.

where [·; ·] refers to the concatenation operation.

In this latent syntactic space, we obtain the prior Gaussian parameters µ(x, y) and σ(x, y) through a linear mapping:

where θvae=[Wµ,bµ,Wσ,bσ] are parameters. We use the same method as Eqs (8) and (9) to compute the posterior Gaussian parameters µ(y) and σ(y), except that the input is zy.

Then we employ the reparameteried technique [Kingma et al., 2014] for VAE training, setting hz = µ(x, y)+σ(x, y)⊙ϵ, ϵ ∼ N(0, I). This facilitates its optimization since we can apply the standard backpropagation to compute the gradient in an end-to-end manner.

Decoding Stage. At each decoding step t, we use the latent syntactic variable hz as a global signal to control the decoder prediction, which enforces the generated sentence to satisfy the given syntactic style. Specifically, we concatenate the latent syntactic variable hz, the attentional semantic representation hc and the previous word's embedding as the input to the decoder, for predicting the word at next time step.

4 Cycle Learning

In order to train our model in a true paraphrase generation setting, we exploit a "cycle" learning scheme to further improve the model, by creating pseudo paraphrase training data.

Our model is also designed to be able to perform paraphrase generation if we use the decoder that generates sentences in the same language as that of the input of the content encoder. Therefore, we can adopt the following two methods to generate pseudo training data. We use the learned model in an inference way to perform generation via greedy decoding.

Given a bilingual parallel sentence pair (x, y), we use the learned model to translate sentence x from one language to the other language. The generated translation y′ should be very close to the original sentence y in terms of meanings.

where we set z = x. Specially, in order to generate diverse sentences, we inject random noise into the latent variable in the continuous syntactic space, where hz = µ(z) + ϵ, ϵ ∼ N(0, I). Thus we obtain pseudo paraphrase pairs (y′ , y).

We also enhance translation by using the learned paraphrase generation model. We use the learned model to paraphrase a sentence x to another x′ in the same language. The sentence x′ can be translated back to the original sentence y in the other language.

where we set z = x. We also use noise in the syntactic variable. In this way, we generate pseudo bilingual pairs (x′ , y).

5 Training and Inference

Training Procedure. Given two bilingually aligned sentences x in X language and y in Y language, we use (x, y) to train the parameters related to the direction from X to Y (i.e., the two encoders, discriminator and the Y decoder), and (y, x)

for the other direction from Y to X. Meanwhile, we also use the generated pseudo parallel data (as described in Sect. 4) to train the model. We take (x′ , y) as (x, y) to train the translation from X to Y, and (y′ , x) as (y, x) to train the translation from Y to X. Specially, we take (x′ , x) to train the paraphrase generation model for X language, and (y′ , y) to train the paraphrase generation model for Y language. We alternately use the six types of sentence pairs to train the model.

Training Objectives. The training loss for a training instance (x, z; y) (where z = y) is a combination of the generation loss, the KL-divergence penalty (Eq. 3), and the adversarial loss (Eq. 2).

where loss(y|z, x) = - ∑Tt=1 log p(yt|y1:t-1, z, x) is the cross-entropy loss for generating sentence y given x and z, and λ∗ are hyperparameters that balance these losses.

Masking on Syntactic Exemplars. During training, in order to avoid additionally collecting syntactic exemplars, we set z = y as the input of the style encoder. In order to encourage the model to capture more syntax-related information through context and reduce the dependence on token representations, we use the [MASK] token as introduced in Devlin et al. [2019] to mask the syntactic input of each training example at the token level. We replace a chosen token with the [MASK] 30% of the time, a random token 10% of the time and do not change the chosen token 60% of the time.

Inference. Given a semantic template and syntactic exemplar from the same language, we input the semantic template to the content encoder and the syntactic exemplar to the style encoder, and then use the decoder corresponding to the language of the inputs to generate a syntactically controllable paraphrase.

6 Experiments

We conducted two groups of experiments, one for controllable paraphrase generation and the other for general paraphrase generation without syntactic guidance. We used the Chinese-English bilingual dataset CWMT (neu2017) to train the proposed model, and the ParaNMT dataset for evaluating controllable paraphrase generation, the Quora and MSCOCO datasets for evaluating general paraphrase generation.

6.1 Datasets

CWMT Chinese-English Dataset. This dataset contains 2 million bilingual parallel pairs from news domain. It has been originally released publicly for Chinese-English machine translation evaluation. 1

ParaNMT Dataset. Following previous work [Chen et al., 2019], we used this paraphrase dataset [Wieting and Gimpel, 2018] to evaluate model performance for controllable paraphrase generation. For fair comparison, we used the same training data (500K sentence pairs) as Chen et al. [2019]. The manually annotated 800 instances created by Chen et al. [2019] were used as our test set, and 500 for development set.

1 http://nlp.nju.edu.cn/cwmt-wmt/

Quora Dataset. This dataset is a paired paraphrase dataset in question domain. It consists of 150K paraphrase pairs. Following previous work [Hasan et al., 2016; Gupta et al., 2018], we used 100K and 4K pairs for training and testing, and the remaining pairs as development set, respectively.

MSCOCODataset. This dataset [Lin et al., 2014] contains human-annotated captions of over 120K images. Each image contains five captions from five different annotators. Following Prakash et al. [2016], we obtained a collection of 330K instances for training and 20K instances for testing.

6.2 Model Configuration

We used Adam [Kingma and Ba, 2014] for optimization. We set the initial learning rate to 5e-4. We set the mini-batch size to 50 for each training corpus. The size of cross-lingual word embeddings and the hidden states of the encoders and decoders were set 512. The size of the latent syntactic variable were 256. We trained the cross-lingual embeddings on the CWMT dataset. For the VAE training, following Bowman et al. [2015], we set the weight λ2 to zero at the start of training, and gradually increased this weight to 1 as training progressed. We set λ1 to 1 and λ3 to 1e-5.

6.3 Automatic EvaluationMetrics

Semantic Accuracy. Following previous work [Prakash et al., 2016; Hasan et al., 2016], we used well-known automatic evaluation metrics: BLEU (BL), ROUGE (R) and METEOR (MET). Pervious studies have shown that these metrics perform well in evaluating generated paraphrases.

Syntactic Similarity. To measure the syntactic similarity, we reported the syntactic tree (ST) edit distance. We computed the tree edit distance between constituency parse trees after removing word tokens following Chen et al. [2019].

6.4 Controllable Paraphrase Generation

Experiment Settings. We trained the proposed model in the setting that the input and output is the same language. The model trained in this way was used as the baseline model. We then trained our model on the bilingual corpus (CWMT Chinese-English Dataset) to compare against the baseline model. We also analyzed the effect of using different amount of training data. Additionally, we investigated the performance of fine-tuning our bilingually trained model on the condition that a small paraphrase dataset is available. Finally, we compared the proposed method with previous work.

Results. The results are shown in Table 1. It can be seen that our model achieves competitive performance on the CPG task, even without using any annotated paraphrase training data. For the baseline model, please notice that we used the same training data as the model VGVAE + WN [Chen et al., 2019], but we did not use external POS tags to preprocess syntactic inputs, and only used masking scheme.

Compared to the baseline model trained on paraphrase data, the performance of our model trained on bilingual data is surprisingly promising, which is quite comparable to the baseline model in terms of the six evaluation metrics. The results with different amount of bilingual training data show that our method can obtain a good performance even if the training data is relatively small, and suggest that more training data can further improve the quality of generated paraphrases. This validates that our proposed model is able to learn to generate syntactically-controllable paraphrases from bilingual parallel data.

Fine-tuning our model on a small amount of available paraphrase data, we can gain further improvements. When compared to the baseline trained on 500K paraphrase data, finetuning our model on only 100K (20%) paraphrase data can obtain the same performance. With more paraphrase data, the performance of our model can be further steadily improved on all evaluation metrics. Finally, when all 500k paraphrase pairs are used to fine tune our model, it reaches the performance level that is substantially better than the previous work [Chen et al., 2019] on all evaluation metrics. For the challenging syntactically-controllable paraphrase generation task, our initial success of using off-the-shelf bilingual data shows a promising direction for paraphrase generation that does not need a large scale of annotated paraphrase data.

Comparison to Model Trained on MT-Generated Paraphrase Data. Following Wieting and Gimpel [2018], we generated paraphrase data via neural machine translation (NMT). We spilt the bilingual data into two parts, 1M for training a Transformer NMT system [Vaswani et al., 2017], and 1M for using the trained NMT model to construct paraphrase data. We translated Chinese sentences into English, and paired the translations and original English sentences into paraphrases. The paraphrase data generated in this way were used to train our model. As shown in Table 2, the model trained on the bilingual corpus is better than the model trained on MT-generated paraphrase data.

Ablation Study. To better understand the impact of each component of our model on learning controllable paraphrase generation from bilingual data, we conducted ablation study. The results are shown in Table 3. In this experiment, we used 1M bilingual corpus as training data.

When the KL loss for building the syntactic space is removed, the performance is obviously degraded. We also observe that disentangling the content and style space via the proposed adversarial discriminator is important for the model to learn syntactically-controllable generation. Without the cycle learning, the performance also substantially drops. The masking technique has the largest impact on performance as it successfully prevents the model from just copying.

Generated Examples. To take a deep look into the proposed model only relying on bilingual data, we manually analyzed some examples, as shown in Table 4.

We can see that the proposed model produces fairly good samples in terms of both closeness in meaning and diversity in expressions. Meanwhile, our model trained only on bilingual data can effectively generate meaning-preserved paraphrases in the syntactic form similar to syntactic inputs.

6.5 General Paraphrase Generation

In order to study our model in learning PG from bilingual data, we conducted the second group of experiments without the controllable nature by omitting the part of the style encoder.

Results. The experiment results on the Quora dataset are shown in Table 5. We compared our model with previous methods VAE [Gupta et al., 2018] and EDD-LG [Patro et al., 2018] that are built on large annotated paraphrase data (100K). As we can see, the proposed model obtains very strong results that are comparable to the previous state-ofthe-art models trained on large paraphrase data. The finetuning results on smaller paraphrase data show that our model with only 30K (30%) paraphrase data outperforms the stateof-the-art models trained on 100K paraphrase data. With the increasing of the amount of paraphrase data, the performance is further improved.

The experiment results on the MSCOCO dataset are displayed in Table 6. The MSCOCO is an image caption dataset. Since different annotators may focus on different objects in the same image, the descriptions created by them may be quite different in meaning. Thus, the model trained on bilingual data cannot capture these differences. However, our model still obtains promising performance when it is finetuned with a small annotated data.

As both the MSCOCO and Quora dataset are smaller than the ParaNMT dataset, the results here further demonstrate the effectiveness of our method in low-resource scenarios.

7 Conclusion

In this work, we have presented a novel method to learn controllable paraphrase generation from bilingual data. We model semantic content and syntactic style via two shared encoders and one adversarial discriminator. Experiments and analyses disclose that the proposed method can effectively learn to generate high-quality meaning-preserved paraphrases in the syntactic forms controlled by given exemplars. This allows us to significantly reduce the amount of needed manually annotated paraphrase data, and makes it easier to develop a high-quality paraphrase generation system with large-scale available bilingual corpora.

Acknowledgments

The present research was supported by the National Nature Science Foundation of China (No. 61876198, 61976015, 61976016, 61370130 and 61473294), and also supported by the Beijing Municipal Natural Science Foundation (No. 4172047), and the International Science and Technology Cooperation Program of China (No. K11F100010).