Utilizing Non-Parallel Text for Style Transfer by Making Partial Comparisons

1 Introduction

The style of a text conveys important information beyond its literal meaning [Hovy, 1987]. The ability to take control over some style attributes (e.g. sentiment, formality) of the generated text is essential to make language generation systems more intelligent, and is potentially useful in many applications, such as dialogue systems [Niu and Bansal, 2018] and image captioning [Mathews et al., 2018]. More specifically, text style transfer aims to rephrase a given sentence into a different style (e.g. transform the sentiment from negative to positive) without changing the main content of the original sentence (e.g. the aspects be discussed) (Figure 1).

Similar to neural machine translation [Sutskever et al., 2014; Bahdanau et al., 2015], one possible solution to text style transfer is to train sequence-to-sequence models with

∗ Shujian Huang is the corresponding author

1 https://github.com/yd1996/PartialComparison

Figure 1: Examples of text style transfer (e.g. changing the sentiment attribute of a review from negative to positive). The dashed line indicates the transformation of style-related words from the source style to the target style. The solid line indicates the preservation of content-related words. In this figure, the first example is a successful case of style transfer, and the second one includes some wrong transformations.

parallel corpora [Xu et al., 2012; Rao and Tetreault, 2018]. However, since parallel corpora are usually unavailable for most scenarios, some researchers have proposed approaches for building style transfer systems using non-parallel corpora only [Hu et al., 2017; Shen et al., 2017; Fu et al., 2018; Yang et al., 2018; Li et al., 2018a]. Most of them try to disentangle style attributes and style-independent content using Conditional Generative Adversarial Nets (ConditionalGANs) [Goodfellow et al., 2014; Mirza and Osindero, 2014]. Ideally, the adversarial training includes a generator to generate the transferred sentence, and a discriminator to decide whether this transferred sentence is correct, i.e. whether it has the same content and different style compared to the original sentence.

Again, due to the lack of parallel data, training such a discriminator is unfeasible. As a compromise, previous work guide their training with style-related distributional information, e.g. style-related binary classifiers [Shen et al., 2017] or language models [Yang et al., 2018]. However, even with such an extra process to reconstruct the original sentence, the transfer performance of their model is still weak, especially for content preservation [Li et al., 2018a], because the distributional information is not enough for the adversarial discriminator to decide whether two sentences have the same content.

In this paper, we analyze the Conditional GAN framework

(Section 2), and emphasize that the instance-level comparison between original and transferred sentences is necessary during the training process. Without parallel data to make a complete comparison of both content and style, we propose partial comparators to guide the adversarial training process by making partial comparisons (Section 3). Each partial comparator aims to model only one kind of correspondence, either content or style. To train these comparators, we propose a simple but effective method to automatically extract initial training instances with high quality from the non-parallel data. To take advantage of all the non-parallel training data, we propose to further enhance the training process by data augmentation.

To demonstrate the effectiveness of our method, we perform experiments on two review datasets to compare our method to other existing approaches (Section 4). Results of automatic and human evaluation show that our approach can significantly improve the performance of existing adversarial methods, and outperforms most state-of-the-art models. We also provide analysis about how the proposed method utilizes the non-parallel data (Section 5).

2 Non-parallel Text Style Transfer

2.1 ProblemFormulation

Given two text datasets X = {x(1) , x(2) , . . . , x(m) } and Y = {y(1) , y(2) , . . . , y(n) } from different styles vx and vy, respectively, we call X, Y non-parallel datasets where no pairs of (x(i) , y(i) ) with the same content can be directly fetched. Our task is to learn a style transfer model on this kind of datasets, which can generate a new sentence with a different style attribute conditioned on a given sentence, without changing the original content.

Most previous work [Hu et al., 2017; Shen et al., 2017; Yang et al., 2018] assume all texts are generated conditioned on two disentangled representations, the style v and the content z. The transfer process can be formulated as follows: first, an encoder E encodes x into the latent representation z = E(x, vx), from which the information about the original style vx has been removed; then, conditioned on z and the target style vy, the generator G produces a new sentence y = G(z, vy). The same process could go in the other direction, and these dual processes can be formulated as follows.

2.2 Adversarial Training

As is shown in the left part of Figure 2, the training of previous GAN-based models [Shen et al., 2017; Yang et al., 2018] often relies on the following two separate processes:

The Reconstruction Process

Previous methods try to disentangle style attributes and styleindependent content using an auto-encoder model. In order to preserve the main content, the style transfer model should have the ability to reconstruct the original sentence from the disentangled representations of content and the original style.

Formally, the encoder E firstly encodes the sentence x, given style vx, into a style-independent representation. Then, the generator G reconstructs x conditioned on zx and vx. Same for the other direction. The corresponding reconstruction loss is as follows.

Note that although the reconstruction process aims at preserving the main content, the process is only trained with the input of the original style, while the content preservation when transferring to a different style is still not under control.

The Style Transfer Process

The transfer process performs a real style transfer. It uses the same process to get zx and zy as in the reconstruction process, and generates ˜y and ˜x, respectively.

To guide the style transfer process, Shen et al. [2017] and Yang et al. [2018] use adversarial training to align several distribution pairs. The discriminator Dz, which aims to distinguish between zx and zy, is introduced to align the distributions p(zx) and p(zy) in an adversarial way.

The discriminators Dx and Dy are introduced to align distribution of real and fake sentences via adversarial training: Dx distinguishes between x and ˜x, and Dy distinguishes between y and ˜y. These discriminators can be binary classifiers [Shen et al., 2017] or language models [Yang et al., 2018]. The adversarial objectives of them are as follows.

The overall training objective is a min-max game played among the encoder E, the generator G and the discriminators Dz, Dx, Dy, and it can be formulated as follows.

Distributions v.s. Instances

Although aligning the distribution in a adversary way is an effective approach to building the distributional correspondence, this distributional correspondence is still not enough for content preservation, because there is no guarantee that two sentences have the same content even if they come from the same distribution. The only solution is to explicitly compare the content and style of two sentences, in order to build correspondences between instances instead of distributions.

3 Making Partial Comparisons

As discussed before, without parallel data, it's hard to train a discriminator to directly make complete comparison between two sentences to decide whether they not only have the same content, but also belong to different styles. Therefore, we propose to make partial comparisons, which means that the comparison between two sentences is made in only one aspect, either content or style.

To make these two kinds of partial comparisons during the training process, we introduce two partial comparators, i.e. the content comparator and the style comparator (denoted as Dc and Ds, respectively). Given two sentences, the content comparator Dc judges whether they share the same content, and the style comparator Ds judges whether they have different styles.

3.1 Adversarial Training with Partial Comparators

Jointly with Dc and Ds, the transfer process could be guided via adversarial training. Taking the content comparator as an example, the adversarial objective is as follows,

where the first term is the likelihood of sentence pairs that have the same content; the second term is the likelihood of the fake sentence pairs, consisting of the original sentence x and the generated sentence ˜y; similar for the third term.

The adversarial objective for the style comparator is similar as follows.

Working together, the two comparators could accomplish the comparison between two sentences, while each one of them is easy to be built and trained. We will introduce the modeling of the partial comparators and the training of them in the following subsections.

Note that Dc and Ds could model instance-level correspondences between two sentences in content and style, but cannot ensure the distributional correspondence and their fluency. Therefore, we also introduce two language models pretrained by the sentences from each style into our framework, inspired by Yang et al. [2018] .

The overall framework of our proposed method is illustrated in the right part of Figure 2. The min-max game is formulated as follows.

3.2 Partial Comparators as Text MatchingModels

Partial comparison is similar to text matching, which is to decide whether two texts are relevant or not. Therefore, we borrow some techniques from the area of sentence pair modeling [Hu et al., 2014] to implement two kinds of partial comparator. In most tasks of text style transfer, such as sentiment modification, we judge whether two sentences share the same content by observing whether there exists some correspondences of keywords (e.g. the food name in restaurant reviews) between them, so we employ the sentence interaction model ARC-II [Hu et al., 2014] as the content comparator. At the same time, since the style of a sentence is a global attribute, we employ a sentence encoding model like ARC-I in Hu et al. [2014] as the style comparator Ds. Please refer to their original paper for details.

3.3 Training Data for Partial Comparators

Training instances are needed to train the two partial comparators. The partial-parallel training instances for the style comparator Ds is sentences pairs that have different style, which is easy to get by simply sampling from X and Y.

The partial-parallel training instances for the content comparator Dc are sentence-pairs that have similar content. Because the content of a sentence are usually carried by words, we mine the instances from the union of X and Y using lexical clues.

Inspired by Li et al. [2018a] , we extract noun words as keywords of each sentence according to the automatic POS tags and group sentences with the same keywords. Within each group, we calculate the edit distance between every two sentences and collect those sentence pairs with an edit distance lower than a given threshold. These sentence pairs are considered to have same content.

Although the union of X and Y could be mined for Dc, the coverage of the above mining process is still low 2 .

To take full advantage of these uncovered sentences, we perform data augmentation (DA) to further improve the adversarial training. During the training procedure, we use the same procedure as we extract the initial partial-parallel training instances on the automatically generated sentences. In other words, if the generated sentence ˜y has the same keywords with the original sentence x, and their edit distance is within a threshold, we add the pair (x, ˜y) into the partialparallel training data. Similar for (˜x, y).

3.4 Training

Due to the discreteness of texts, gradients cannot be directly propagated from discriminators to the style transfer model. One possible solution to this problem is to use the REINFORCE [Sutton et al., 2000] algorithm. However, previous work [Yu et al., 2017] shows that this way suffers from high variance. We choose to use a Gumbel-Softmax [Jang et al., 2016] distribution as input to the generator and the discriminators, instead of a single sampled word, which makes the training process differentiable.

where the gi's are independent samples from Gumbel(0,1). During training, we use an annealing strategy to update the

2 Our empirical study shows that more than 70% of the original sentences do not have a proper content matching sentence in our training data.

Algorithm 1 Making partial comparisons

Input:

Two non-parallel corpora X, Y of different styles vx, vy ; Partial-parallel training data for content comparator (X, Y, lc);

Pretrained language model LMx and LMy;

Lagrange multiplier λz, λc, λs, λLM, temperature τ .

20: Calculate Ltotal according to (11), and update θE, θG. 21: until convergence

Output: A text style transfer model, which consists of the encoder E and the generator G.

value of τ . The initial value of τ is set to 1.0 and it decays by half every epoch until reaching the minimum value of 0.001.

We follow the training procedure proposed in WGAN [Arjovsky et al., 2017]; as is shown in Algorithm 1, we train the partial comparators ncritic steps, then one step on the style transfer model. We use the Adam [Kingma and Ba, 2015] optimization algorithm to train the style transfer model and RMSprop [Tieleman and Hinton, 2012] for partial comparators.

4 Experiment

4.1 Experiment Setup

We perform experiments two review datasets to evaluate our model. For convenient comparison, we follow the previous setting and focus only on generating short texts(shorter than 20 words).

Datasets

We conduct experiments on the Yelp review dataset and Amazon review dataset (Table 1) released by [Li et al., 2018a], with the same pre-processing steps.

Parameter Setting

The encoder E and the generator G are single-layer LSTMRNNs with input dimension of 300 and hidden dimensions of 350. The dimension of style embedding is 50. The word embeddings are pretrained using Word2Vec 3 . The discriminators Ds and Dc both have two layers of convolution and max-pooling. We use a batch size of 160, which contains 80 samples from X and Y respectively. Hyper-parameters are selected based on the validation set, and we use grid search to pick the best parameters. The learning rate is selected from [1e-4, 2e-4, 5e-4, 1e-3], and the weights of each term in the training objective (λz, λc, λs, λlm) are all selected from [0.1, 0.5, 1.0, 2.0, 5.0]. We implement our model based on PyTorch 4 and use four NVIDIA GTX1080Ti graphic cards for learning. Our source code and data will be released 5 .

4.2 Automatic Evaluation

Two key aspects need to be evaluated: attribute transfer and content preservation. We follow previous work [Hu et al., 2017; Shen et al., 2017; Yang et al., 2018], and use a pretrained CNN-based sentence classifier to measure whether transferred sentences have the correct style (sentiment in the task of sentiment modification). The results are reported in accuracy (ACC). To evaluate the degree of content preservation, we calculate BLEU scores using the human annotated sentences provided by Li et al. [2018a] as the ground truth of transferred sentences.

The result are in Table 2. We can see that our model has a better overall performance than most state-of-the-art models in automatic evaluation and achieves an large improvement over previous GAN-based methods [Shen et al., 2017; Yang et al., 2018] in content preservation.

4.3 Ablation Study

We conduct ablation study to evaluate the contribution of each component in our training framework (Table 3). The baseline is equipped with two separate binary style classifiers, similar to Shen et al. [2017] , but with higher performance.

Firstly, it is interesting to see that replace the two style classifiers with our style comparator already leads to an improvement, indicating that even for style the instance level correspondence is more helpful. Secondly, using our content comparator improves the content preservation of model by a considerable margin. Thirdly, adding the style language model still improves our model, suggesting that our improvement is orthogonal to Yang et al. [2018] . Finally, using data augmentation further improves the results again, and achieves the best results on both the two datasets.

3 https://radimrehurek.com/gensim/models/word2vec.html

4 https://pytorch.org/

5 https://github.com/yd1996/PartialComparison

4.4 Manual Evaluation

We also conduct manual evaluation on the generated result. For each test set, we randomly sample 100 sentences and collect the transfer results. Annotators are asked to score the generated sentences in three aspects: attribute transfer, content preservation and language fluency. For each aspect, the score ranges from 0 to 2, where 0 means failure, 1 means partial success and 2 means success. The result of manual evaluation is shown in Table 4, and we can see that our model outperforms other models in all aspects.

4.5 Case Study

We further analyze the results generated by different models. Table 5 shows some typical outputs of each system on the Yelp dataset. [Hu et al., 2017] may change the sentiment correctly, but for most cases it also changes the original content. [Shen et al., 2017] often changes the original content as well. Sometimes it generates a complete different sentence (e.g. 'they are worth five stars!'). MultiDecoder [Fu et al., 2018] does not succeed in the transferring for the given cases. DeleteAndRetrieval, a simple method based on replacement [Li et al., 2018a], can effectively preserve the subject of the review, but changed the meaning in some case (e.g. from 'overcooked' to 'beat feet out of there'). [Yang et al., 2018] improves the content preservation, but tends to generate shorter sentences (e.g. 'the escargot was mediocre at best.'), which will change the syntactic structure of original sentences. On the contrary, our model produce reasonable results for both the two examples.

5 Discussion

The main problem of non-parallel text style transfer is the lack of parallel data, we further discuss how our mined partial-parallel instances and the data augmentation works.

5.1 Mining Parallel Instances

A simple question may rise that why not directly construct parallel instances and use them to guide the learning process. According to our experiments, the mined partial-parallel instances for the content comparator only covers 30% of the sentences in the dataset; using the same threshold to mining parallel instances could only obtain about 10% of instances comparing to the partial parallel case, which seems to be too small to be useful.

5.2 MiningMore Partial-parallel Instances

The threshold of edit distance between two sentences directly affects the number and quality of instances.

For a quick study, we set the threshold to different values to get 50K, 150K and 300K positive partial-parallel instances from the Yelp dataset, and use them to train a style transfering model ('+Ds+Dc+DA' in Table 3). The accuracy and BLEU of the three systems are 90.8/18.6, 91.9/22.3 and 89.5/19.7, respectively, showing a clear trade-off between the scale and the quality of the partial-parallel data.

As a result, using a larger threshold to obtain more instances but with lower content similarity may hurt the training performance. To increase the coverage of the data, the data augmentation method may be a better choise.

5.3 Data Augmentation

It is reasonable that the style transfer results would be better for the mined partial-parallel instances, because they have close content-related sentences in the dataset. In Table 6, we present an example which is not in the mined partial-parallel training data. We list the results generated by different models for comparison. Without data augmentation, adding the content comparator Dc cannot bring much improvement to the content preservation in these cases, because they may not be covered in the training data of the comparator. With data augmentation, after exploring uncovered instances in the training data, both the two models learn better correspondence between the content words, which improve the performance of the model on these uncovered cases.

6 RelatedWork

For the task of text style transfer, Li et al. [2018a] uses simple lexical operation such as 'delete' to remove the style related information based on resources previously extract from data. They also use the 'retrieve' operation to find sentence with the same or similar content. Policies need to be design for using these operations. In contrast, we start with training instances mined by lexical evidences, but the transferring process is still in the framework of sequence-to-sequence, which could be trained in an end-to-end process.

The practice in sentence pair modeling [Lan and Xu, 2018; Hu et al., 2014] inspires us for the design of comparators. Li et al.; Zhang et al. [2018b; 2018] also incorporate content discriminators for the task of paraphrase, which aims at generating sentences with the same meaning. The task of style transfer is more challenging because there are two aspects to be considered. Our contribution is to design separate comparators for each aspect, which is different from the practice in paraphrase tasks.

7 Conclusion

In this paper, we propose an effective method to make instance-level comparisons with only non-parallel corpora. The proposed partial-comparison strategy enhanced the performance of adversarial training for style transfer models. Our work explore possibilities for text generation without parallel data, which may be useful for other scenarios. For future work, we will explore the possibility to improve the instance mining and data augmentation process with a component which maybe automatically learned during the training. It may also be interesting to apply the proposed method to other similar text generation tasks.

Acknowledgements

We would like to thank the anonymous reviewers for their insightful comments. This work is supported by the National Science Foundation of China (No. U1836221), the Jiangsu Provincial Research Foundation for Basic Research (No. BK20170074), "13th Five-Year" All-Army Common Information System Equipment Pre-Research Project (No. 31510040201).