Toward Diverse Text Generation with Inverse Reinforcement Learning
Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence. Main track. Pages 4361-4367.
https://doi.org/10.24963/ijcai.2018/606
Keywords
- Machine Learning Applications: Applications of Reinforcement Learning
- Natural Language Processing: Natural Language Generation
Abstract
Text generation is a crucial task in NLP. Recently, several adversarial generative models have been proposed to improve the exposure bias problem in text generation. Though these models gain great success, they still suffer from the problems of reward sparsity and mode collapse. In order to address these two problems, in this paper, we employ inverse reinforcement learning (IRL) for text generation. Specifically, the IRL framework learns a reward function on training data, and then an optimal policy to maximum the expected total reward. Similar to the adversarial models, the reward and policy function in IRL are optimized alternately. Our method has two advantages: (1) the reward function can produce more dense reward signals. (2) the generation policy, trained by ``entropy regularized'' policy gradient, encourages to generate more diversified texts. Experiment results demonstrate that our proposed method can generate higher quality texts than the previous methods.
- [Bayer and Osendorfer, 2014] Justin Bayer and Christian Osendorfer. Learning stochastic recurrent networks. arXiv, 2014.
- [Bengio et al., 2015] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for se- quence prediction with recurrent neural networks. In NIPS, pages 1171-1179, 2015.
- [Che et al., 2017] Tong Che, Yanran Li, Ruixiang Zhang, R Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua Bengio. Maximum-likelihood augmented discrete genera- tive adversarial networks. arXiv, 2017.
- [Chen et al., 2015] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data col- lection and evaluation server. arXiv, 2015.
- [Chung et al., 2015] Junyoung Chung, Kyle Kastner, Lau- rent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In NIPS, pages 2980-2988, 2015.
- [Diao et al., 2014] Qiming Diao, Minghui Qiu, Chao-Yuan Wu, Alexander J Smola, Jing Jiang, and Chong Wang. Jointly modeling aspects, ratings and sentiments for movie recommendation (jmars). In SIGKDD, pages 193-202. ACM, 2014.
- [Finn et al., 2016] Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal con- trol via policy optimization. In ICML, pages 49-58, 2016.
- [Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In NIPS, pages 2672-2680, 2014.
- [Graves, 2013] Alex Graves. Generating sequences with re- current neural networks. arXiv, 2013.
- [Guo et al., 2017] Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information. arXiv, 2017.
- [Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.
- [Joji Toyama, 2018] Kotaro Nakayama Yutaka Matsuo Joji Toyama, Yusuke Iwasawa. Toward learning better metrics for sequence generation training with policy gradient. ICLR submission, 2018.
- [Kusner and Hernández-Lobato, 2016] Matt J Kusner and José Miguel Hernández-Lobato. GANs for sequences of discrete elements with the gumbel-softmax distribution. arXiv, 2016.
- [Lin et al., 2017] Kevin Lin, Dianqi Li, Xiaodong He, Ming- ting Sun, and Zhengyou Zhang. Adversarial ranking for language generation. In NIPS, pages 3158-3168, 2017.
- [Metz et al., 2016] Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. arXiv, 2016.
- [Nachum et al., 2017] Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. arXiv, 2017.
- [Salimans et al., 2016] Tim Salimans, Ian Goodfellow, Woj- ciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NIPS, pages 2234-2242, 2016.
- [Serban et al., 2017] Iulian Vlad Serban, Alessandro Sor- doni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C Courville, and Yoshua Bengio. A hierarchical latent vari- able encoder-decoder model for generating dialogues. In AAAI, pages 3295-3301, 2017.
- [Su et al., 2018] Jinyue Su, Jiacheng Xu, Xipeng Qiu, and Xuanjing Huang. Incorporating discriminator in sentence generation: a gibbs sampling method. In AAAI, 2018.
- [Wang et al., 2017] Heng Wang, Zengchang Qin, and Tao Wan. Text generation based on generative adversarial nets with latent variable. arXiv, 2017.
- [Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4):229-256, 1992.
- [Yu et al., 2017] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. SeqGAN: Sequence generative adversarial nets with policy gradient. In AAAI, pages 2852-2858, 2017.
- [Zhang et al., 2017] Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. Adversarial feature matching for text generation. arXiv, 2017.
- [Ziebart et al., 2008] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.
References were extracted automatically from the PDF and may contain errors.
Full text extracted automatically from the PDF; figures, tables and equations may differ. The PDF is the version of record.
1 Introduction
Text generation is one of the most attractive problems in NLP community. It has been widely used in machine translation, image captioning, text summarization and dialogue systems.
Currently, most of the existing methods [Graves, 2013] adopt auto-regressive models to predict the next words based on the historical predictions. Benefiting from the strong ability of deep neural models, such as long short-term memory (LSTM) [Hochreiter and Schmidhuber, 1997], these autoregressive models can achieve excellent performance. However, they suffer from the so-called exposure bias issue [Bengio et al., 2015] due to the discrepancy distribution of histories between the training and inference stage. In training stage, the model predicts the next word according to groundtruth histories from the data distribution rather than its own historical predictions from the model distribution.
Recently, some methods have been proposed to alleviate this problem, such as scheduled sampling [Bengio et al.,
∗Corresponding Author, xpqiu@fudan.edu.cn
2015], Gibbs sampling [Su et al., 2018] and adversarial models, including SeqGAN [Yu et al., 2017], RankGAN [Lin et al., 2017], MaliGAN [Che et al., 2017] and LeakGAN [Guo et al., 2017]. Following the framework of generative adversarial networks (GAN) [Goodfellow et al., 2014], the adversarial text generation models use a discriminator to judge whether a given text is real or not. Then a generator is learned to maximize the reward signal provided by the discriminator via reinforcement learning (RL). Since the generator always generates a entire text sequence, these adversarial models can avoid the problem of exposure bias.
Inspired of their success, there are still two challenges in the adversarial model.
The first problem is reward sparsity. The adversarial model depends on the ability of the discriminator, therefore we wish the discriminator always correctly discriminates the real texts from the "generated" ones. Instead, a perfect discriminator increases the training difficulty due to the sparsity of the reward signals. There are two kinds of work to address this issue. The first one is to improve the signal from the discriminator. RankGAN [Lin et al., 2017] uses a ranker to take place of the discriminator, which can learn the relative ranking information between the generated and the real texts in the adversarial framework. MaliGAN [Che et al., 2017] develops normalized maximum likelihood optimization target to alleviate the reward instability problem. The second one is to decompose the discrete reward signal into various sub-signals. LeakGAN [Guo et al., 2017] takes a hierarchical generator, and in each step, generates a word using leaked information from the discriminator.
The second problem is the mode collapse. The adversarial model tends to learn limited patterns because of mode collapse. One kind of methods, such as TextGAN [Zhang et al., 2017], uses feature matching [Salimans et al., 2016; Metz et al., 2016] to alleviate this problem, it is still hard to train due to the intrinsic nature of GAN. Another kind of methods [Bayer and Osendorfer, 2014; Chung et al., 2015; Serban et al., 2017; Wang et al., 2017] introduces latent random variables to model the variability of the generated sequences.
To tackle these two challenges, we propose a new method to generate diverse text via inverse reinforcement learning (IRL) [Ziebart et al., 2008]. Typically, the text generation can be regarded as an IRL problem. Each text in the training data is generated by some experts with an unknown reward function. There are two alternately steps in IRL framework. Firstly, a reward function is learned to explain the expert behavior. Secondly, a generation policy is learned to maximize the expected total rewards. The reward function aims to increase the rewards of the real texts in training set and decrease the rewards of the generated texts. Intuitively, the reward function plays the similar role as the discriminator in SeqGAN. Unlike SeqGAN, the reward function is an instant reward of each step and action, thereby providing more dense reward signals. The generation policy generates text sequence by sampling one word at a time. The optimized policy be learned by "entropy regularized" policy gradient [Finn et al., 2016], which intrinsically leads to a more diversified text generator.
The contributions of this paper are summarized as follows.
- We regard text generation as an IRL problem, which is a new perspective on this task.
- Following the maximum entropy IRL [Ziebart et al., 2008], our method can improve the problems of reward sparsity and mode collapse.
- To better evaluate the quality of the generated texts, we propose three new metrics based on BLEU score, which is very similar to precision, recall and F1 in traditional machine learning task.
2 Text Generation via Inverse Reinforcement Learning
Text generation is to generate a text sequence x1:T = x1, x2, · · · , xT with a parameterized auto-regressive probabilistic model qθ(x), where xt is a word in a given vocabulary V. The generation model qθ(x) is learned from a given dataset {x(n) }Nn=1 with an underlying generating distribution pdata.
In this paper, we formulate text generation as inverse reinforcement learning (IRL) problem. Firstly, the process of text generation can be regarded as Markov decision process (MDP). In each timestep t, the model generates xt according a policy πθ(at|st), where st is the current state of the previous prediction x1:t and at is the action to select the next word xt+1. A text sequence x1:T = x1, x2, · · · , xT can be formulated by a trajectory of MDP τ = {s1, a1, s2, a2..., sT , aT }.
Text Generator
Therefore, the probability of x1:T is
where the state transition p(st+1 = x1:t+1|st = x1:t, at = xt+1) = 1 is deterministic and can be ignored.
Secondly, the reward function is not explicitly given for text generation. Each text sequence x1:T = x1, x2, · · · , xT in the training dataset is formulated by a trajectory τ by experts from the distribution p(τ ), and we have to learn a reward function that explains the expert behavior.
Concretely, IRL consists of two phases: (1) estimate the underlying reward function of experts from the training dataset; (2) learn an optimal policy to generate texts, which aims to maximize the expected rewards. These two phases are executed alternately. The framework of our method is as shown in Figure 1.
2.1 Reward Approximator
Following the framework of maximum entropy IRL [Ziebart et al., 2008], we assume that the texts in training set are sampled from the distribution pφ(τ ),
where Rφ(τ) an unknown reward function parameterized by φ, Z = ∫τ exp(Rφ(τ))dτ is the partition function.
The reward of trajectory Rφ(τ) is a parameterized reward function and assumed to be summation of the rewards of each steps rφ(st, at):
where rφ(st, at) is modeled a simple feed-forward neural network as shown in Figure 2.
Objective of Reward Approximator
The objective of the reward approximator is to maximize the log-likehood of the samples in the training set:
where τn denotes the nth sample in the training set Dtrain.
Thus, the derivative of Jr(φ) is:
Intuitively, the reward approximator aims to increase the rewards of the real texts and decrease the trajectories drawn from the distribution pφ(τ). As a result, pφ(τ) will be an approximation of pdata.
Importance Sampling Though it is quite straightforward to sample τ ∼ pφ(τ) in Eq. (5), it is actually inefficient in practice. Instead, we directly use trajectories sampled by text generator qθ(τ) with importance sampling. Concretely, Eq. (5) is now formalized as:
where wj ∝ exp(Rφ(τj)) qθ(τj) . For each batch, we sample N texts from the train set and M texts drawn from qθ.
2.2 Text Generator
The text generator uses a policy πθ(a|s) to predict the next word one by one. The current state st can be modeled by LSTM neural network as shown in Figure 2. For τ = {s1, a1, s2, a2..., sT , aT },
where st is the vector representation of state st; at is distribution over the vocabulary; eat-1 is the word embedding of at-1; θ denotes learnable parameters includingW, b and all the parameters of LSTM.
Objective of Text Generator
Following "entropy regularized" policy gradient [Williams, 1992; Nachum et al., 2017], the objective of text generator is to maximize the expected reward plus an entropy regularization.
where H(qθ(τ)) = -Eqθ(τ)[log qθ(τ)] is an entropy term, which can prevent premature entropy collapse and encourage the policy to generate more diverse texts.
Intuitively, the "entropy regularized" expected reward can be rewrite as
where Z = ∫τ exp(Rφ(τ))dτ is the partition function and can be regarded as a constant unrelated to θ. Therefore, the objective is also to minimize the KL divergence between the text generator qθ(τ) and the underlying distribution pφ(τ).
Thus, the derivative of Jg(θ) is
where Rφ(τt:T ) denotes the reward of partial trajectory τt, · · · , τT . For obtaining lower variance, R(τt:T ) can be approximately computed by
where V (st+1) denotes the expected total reward at state st+1 and can be approximately computed by MCMC. Figure 3 gives an illustration.
2.3 Why Can IRL Alleviate Mode Collapse?
GANs often suffer from mode collapse, which is partially caused by the use of Jensen-Shannon (JS) divergence. There is a reverse KL divergence KL(qθ(τ)‖pdata) in JS divergence. Since the pdata is approximated by training data, the reverse KL divergence encourages qθ(τ) to generate safe samples and avoid generating samples where the training data does not occur. In our method, the objective is KL(qθ(τ)||pφ(τ)). Different from GANs, we use pφ(τ) in IRL framework instead of pdata. Since pφ(τ) never equals to zero due to its assumption, IRL can alleviate the model collapse problem in GANs.
3 Training
The training procedure consists of two steps: (I) reward approximator update step (r-step) and (II) text generator update step (g-step). These two steps are applied iteratively as described in Algorithm (1).
Initially, we have rφ with random parameters and πθ with pre-trained parameters by maximum log-likelihood estimation on Dtrain. The r-step aims to update rφ with πθ fixed. The g-step aims to update πθ with rφ fixed.
4 Experiment
To evaluate the proposed model, we experiment on three corpora: the synthetic oracle dataset [Yu et al., 2017], the COCO image caption dataset [Chen et al., 2015] and the IMDB movie review dataset [Diao et al., 2014]. Furthermore, we also evaluate the performance by human on the image caption dataset and the IMDB corpus. Experimental results show that Our method outperforms the previous methods. Table 1 gives the experimental settings on the three corpora.
4.1 Synthetic Oracle
The synthetic oracle dataset is a set of sequential tokens which are regraded as simulated data comparing to the real-
Algorithm 1 IRL for Text Generation
| 1: | repeat |
| 2: | Pretrain πθ on Dtrain with MLE |
| 3: | for nr epochs in r-step do |
| 4: | Drawn τ (1) , τ (2) , · · · , τ (i) , · · · , τ (N) ∼ |
| 5: | pdata Drawn τ ′(1) , τ ′(2) , · · · , τ ′(j) , · · · , τ ′(M) ∼ qθ |
| 6: | Update φ←φ+α∇φJr(φ) |
| 7: | end for |
| 8: | for ng batches in g-step do |
| 9: | Drawn τ (1) , τ (2) , · · · , τ (i) , · · · , τ (N) ∼ qθ |
| 10: | Calculate expected reward Rφ(τt:T ) by MCMC |
| 11: | Update θ ← θ +β∇θJg(θ) |
| 12: | end for |
| 13: | until Convergence |
| Hyper-Parameters | Synthetic Oracle COCO & IMDB L = 20 L = 40 | ||
|---|---|---|---|
| Text Generator | |||
| - Embedding dimension | 32 | 64 | 128 |
| - Hidden layer dimension | 32 | 64 | 128 |
| - Batch size | 64 | 128 | |
| - Optimizer & lr rate | Adam, 0.005 | Adam, 0.005 | |
| Reward Approximator | |||
| - Drop out | 0.75 | 0.45 | 0.75 |
| - Batch size | 64 | 1024 | |
| - Optimizer & lr rate | Adam, 0.0004 | Adam, 0.0004 | |
world language data. It uses a randomly initialized LSTM 1 as the oracle model to generate 10000 samples of length 20 and 40 respectively as the training set for the following experiments.
The oracle model, which has an intrinsic data distribution Poracle, can be used to evaluate the sentences generated by the generative models. The average negative loglikelihood(NLL) is usually conducted to score the quality of the generated sequences [Yu et al., 2017; Guo et al., 2017; Lin et al., 2017]. The lower the NLL score is, the better token sequences we have generated.
Training Strategy In experiments, we find that the stability and performance of our framework depend on the training strategy. Figure 4 shows the effects of pretraining epochs. It works best in generating texts of length 20 with 50 epochs of MLE pretraining, and in generating texts of length 40 with 10 epochs of pretraining.
Figure 5 shows that the proportion of nr : ng in Algorithm 1 affects the convergence and final performance. It implies that sufficient training on the approximator in each iteration will lead to better results and convergence. Therefore, we take nr : ng = 10 : 1 as our final training configuration.
1 The synthetic data and the oracle LSTM are publicly available at https://github.com/LantaoYu/SeqGAN and https://github.com/CRGjx/LeakGAN
| Length | MLE | SeqGAN | RankGAN | LeakGAN | IRL | Ground Truth |
|---|---|---|---|---|---|---|
| 20 | 9.038∗ | 8.736∗ | 8.247∗ | 7.038∗ | 6.913 | 5.750 |
| 40 | 10.411∗ | 10.310∗ | 9.958∗ | 7.197∗ | 7.083 | 4.071 |
Results Table 2 gives the results. We compare our method with other previous state-of-the-art methods: maximum likelihood estimation (MLE), SeqGAN, RankGAN and LeakGAN. The listed ground truth values are the average NLL of the training set. Our method outperforms the previous stateof-the-art results (6.913 and 7.083 on length of 20 and 40
respectively). Figure 6 shows that Our method convergences faster and obtains better performance than other state-of-art methods.
Analysis Our method performs better due to the instant rewards approximated at each step of generation. It addresses the reward sparsity issue occurred in previous methods. Thus, the dense learning signals guide the generative policy to capture the underlying distribution of the training data more efficiently.
4.2 COCO Image Captions
The image caption dataset [Chen et al., 2015] consists of image-description pairs. The length of captions is between 8 and 20. Following LeakGAN [Guo et al., 2017], for preprocessing, we remove low frequency words (less than 10 times) as well as the sentences containing them. We randomly choose 80,000 texts as training set, and another 5,000 as test set. The vocabulary size of the dataset is 4,939. The average sentence length is 12.8.
NewEvaluationMeasures onBLEU To evaluate different methods, we employ BLEU score to evaluate the qualities of the generated texts.
- Forward BLEU (BLEUF) uses the testset as reference, and evaluates each generated text with BLEU score.
- Backward BLEU (BLEUB) uses the generated texts as reference, and evaluates each text in testset with BLEU score.
- BLEUHA is the harmonic average value of BLEUF and BLEUB.
Intuitively, BLEUF aims to measure the precision (quality) of the generator, while BLEUB aims to measure the recall (diversity) of the generator.
The configurations of three proposed valuation measures are shown in Table 3.
| Metrics | Evaluated Texts | Reference Texts |
| BLEUF BLEUB | Generated Texts Test Set | Test Set Generated Texts |
| BLEUHA | 2×BLEUF×BLEUB BLEUF+BLEUB | |
BLEUF For BLEUF, we sample 1000 texts for eachmethod as evaluated texts. The reference texts are the whole test set. We list the BLEUF scores of different frameworks and ground truth as shown in first subtable of Table 4. Surprisingly, it shows that results of LeakGAN beat the rest, even the ground truth (LeakGAN has averagely 10 points higher than the ground truth). It may due to the mode collapse which frequently occurs in GAN. The text generator is prone to generate safe text patterns but misses many other patterns. Therefore, BLEUF is failing to measure the diversity of the generated sentences.
| Metrics | MLE | SeqGAN | RankGAN | LeakGAN | IRL | Ground Truth |
|---|---|---|---|---|---|---|
| BLEUF-2 | 0.798 | 0.821 | 0.850∗ | 0.914 | 0.829 | 0.836 |
| BLEUF-3 | 0.631 | 0.632 | 0.672∗ | 0.816 | 0.662 | 0.672 |
| BLEUF-4 | 0.498 | 0.511 | 0.557∗ | 0.699 | 0.586 | 0.598 |
| BLEUF-5 | 0.434 | 0.439 | 0.544∗ | 0.632 | 0.542 | 0.557 |
| BLEUB-2 | 0.801 | 0.682 | - | 0.790 | 0.868 | 0.869 |
| BLEUB-3 | 0.622 | 0.542 | - | 0.605 | 0.718 | 0.710 |
| BLEUB-4 | 0.551 | 0.513 | - | 0.549 | 0.660 | 0.649 |
| BLEUB-5 | 0.508 | 0.469 | - | 0.506 | 0.609 | 0.601 |
| BLEUHA-2 | 0.799 | 0.745 | - | 0.847 | 0.848 | 0.852 |
| BLEUHA-3 | 0.626 | 0.584 | - | 0.695 | 0.689 | 0.690 |
| BLEUHA-4 | 0.523 | 0.512 | - | 0.615 | 0.621 | 0.622 |
| BLEUHA-5 | 0.468 | 0.454 | - | 0.562 | 0.574 | 0.578 |
| Metrics | MLE | SeqGAN | LeakGAN | IRL | Ground Truth |
|---|---|---|---|---|---|
| BLEUF-2 | 0.652 | 0.683 | 0.809 | 0.788 | 0.791 |
| BLEUF-3 | 0.405 | 0.418 | 0.554 | 0.534 | 0.539 |
| BLEUF-4 | 0.304 | 0.315 | 0.358 | 0.352 | 0.355 |
| BLEUF-5 | 0.202 | 0.221 | 0.252 | 0.262 | 0.258 |
| BLEUB-2 | 0.672 | 0.615 | 0.730 | 0.755 | 0.785 |
| BLEUB-3 | 0.495 | 0.451 | 0.483 | 0.531 | 0.534 |
| BLEUB-4 | 0.316 | 0.299 | 0.318 | 0.347 | 0.357 |
| BLEUB-5 | 0.226 | 0.209 | 0.232 | 0.254 | 0.258 |
| BLEUHA-2 | 0.662 | 0.647 | 0.767 | 0.771 | 0.788 |
| BLEUHA-3 | 0.445 | 0.434 | 0.516 | 0.533 | 0.537 |
| BLEUHA-4 | 0.310 | 0.307 | 0.337 | 0.350 | 0.356 |
| BLEUHA-5 | 0.213 | 0.215 | 0.242 | 0.258 | 0.258 |
BLEUB For BLEUB, we sample 5000 texts for each method as reference texts. The evaluated texts consist 1000 texts sampled from the test set. The BLEUB of each method is listed in the second block of Table 4. Intuitively, the higher the BLEUB score is, the more diversity the generator gets. From Table 4, our method outperforms the other methods, which implies that our method generates more diversified texts than the other methods. As we have analyzed before, the diversity of our method may be derived from "entropy regularization" policy gradient.
BLEUHA Finally, BLEUHA takes both generation quality and diversity into account and the results are shown in the last block of Table 4. The BLEUHA reveals that our work gains better performance than other methods.
| Models | COCO | IMDB |
|---|---|---|
| MLE | (1) A girl sitting at a table in front of medical chair. (2) The person looks at a bus stop while talking on a phone. | (1) If somebody that goes into a films and all the film cuts throughout the movie. (2) Overall, it is what to expect to be she made the point where she came later. |
| SeqGAN | (1) Aman holding a tennis racket on a tennis court. (2) A woman standing on a beach next to the ocean. | (1) The story is modeled after the old classic "B" science fiction movies we hate to love, but do. (2) This does not star Kurt Russell, but rather allows him what amounts to an extended cameo. |
| LeakGAN | (1) A bathroomwith a toilet , window , and white sink. (2) A man in a cowboy hat is milking a black cow. | (1) I was surprised to hear that he put up his own money to make this movie for the first time. (2) It was nice to see a sci-fi movie with a story in which you didn't know what was going to happen next. |
| IRL (This work) | (1) A woman is standing underneath a kite on the sand. (2) A dog owner walks on the beach holding surfboards. | (1) Need for Speed is a great movie with a very enjoyable storyline and a very talented cast. (2) The effects are nothing spectacular, but are still above what you would expect, all things considered. |
| Corpora | MLE | SeqGAN | LeakGAN | IRL | Ground Truth |
|---|---|---|---|---|---|
| COCO | 0.205 | 0.450 | 0.543 | 0.550 | 0.725 |
| IMDB | 0.138 | 0.205 | 0.385 | 0.463 | 0.698 |
4.3 IMDBMovie Reviews
We use a large IMDB text corpus [Diao et al., 2014] for training the generative models as long-length text generation. The dataset is a collection of 350K movie reviews. We select sentences with the length between 17 and 25, set word frequency at 180 as the threshold of frequently occurred words and remove sentences with low frequency words. Finally we randomly choose 80000 sentences for training and 3000 sentences for testing with the vocabulary size at 4979 and the average sentence length is 19.6.
IMDB is a more challenging corpus. Unlike sentences in COCO Image captions dataset, which mainly contains simple sentences, e.g., sentences only with the subject-predicate structure, IMDB movie reviews are comprised of various kinds of compound sentences. Besides, the sentence length of IMDB is much longer than that of COCO.
We also use the same metrics (BLEUF, BLEUB, BLEUHA) to evaluate our method. The results in Table 5 show our method outperforms other models.
4.4 Turing Test and Case Study
The evaluation metrics mentioned above are still not sufficient for evaluating the quality of the sentences because they just focus on the local statistics, ignoring the long-term dependency characteristic of language. So we have to conduct a Turing Test based on scores by a group of people. Each sentence will get 1 point when it is viewed as a real one, otherwise 0 point. We perform the test on frameworks of MLE, SeqGAN, LeakGAN and our method on COCO Image captions dataset and IMDB movie review dataset.
Practically, we sample 20 sentences by each generator from different methods, and for each sentence, we ask 20 different people to score it. Finally, we compute the average score for each sentence, and then calculate the average score for each method according to the sentences it generate.
Table 6 shows some generated samples of our and the baseline methods. These samples are what we have collected for people to score.
The results in Table 7 indicate that the generated sentences of our method have better quality than those generated by MLE, SeqGAN and LeakGAN, especially for long texts.
5 RelatedWork
Text generation is a crucial task in NLP which is widely used in a bunch of NLP applications. Text generation is more difficult than image generation since texts consist of sequential discrete decisions. Therefore, GAN fails to back propagate the gradients to update the generator. Recently, several methods have been proposed to alleviate this problem, such as Gumbel-softmax GAN [Kusner and Hern´andez-Lobato, 2016], RankGAN [Lin et al., 2017], TextGAN [Zhang et al., 2017], LeakGAN [Guo et al., 2017], etc.
SeqGAN [Yu et al., 2017] addresses the differentiation problem by introducing RL methods, but still suffers from the problem of reward sparsity. LeakGAN [Guo et al., 2017] manages the reward sparsity problem via Hierarchical RL methods. Joji Toyama [2018] designs several reward functions for partial sequence to solve the issue. However, the generated texts of these methods still lack diversity due to the mode collapse issue. In this paper, we employ IRL framework [Finn et al., 2016] for text generation. Benefiting from its inherent instant reward learning and entropy regularization, our method can generate more diverse texts.
6 Conclusions &FutureWork
In this paper, we propose a new method for text generation by using inverse reinforcement learning (IRL). This method alleviates the problems of reward sparsity and mode collapse in the adversarial generation models. In addition, we propose three new evaluation measures based on BLEU score to better evaluate the generated texts.
In the future, we would like to generalize the IRL framework to the other NLP tasks, such as machine translation, summarization, question answering, etc.
Acknowledgements
We would like to thank the anonymous reviewers for their valuable comments. The research work is supported by the National Key Research and Development Program of China (No. 2017YFB1002104), Shanghai Municipal Science and Technology Commission (No. 17JC1404100 and 16JC1420401), and National Natural Science Foundation of China (No. 61672162).
@inproceedings{ijcai2018p606,
title = {Toward Diverse Text Generation with Inverse Reinforcement Learning},
author = {Shi, Zhan and Chen, Xinchi and Qiu, Xipeng and Huang, Xuanjing},
booktitle = {Proceedings of the Twenty-Seventh International Joint Conference on
Artificial Intelligence, {IJCAI-18}},
publisher = {International Joint Conferences on Artificial Intelligence Organization},
editor = {Jérôme Lang},
pages = {4361--4367},
year = {2018},
month = {7},
note = {Main track},
doi = {10.24963/ijcai.2018/606},
url = {https://doi.org/10.24963/ijcai.2018/606},
}
TY - CPAPER TI - Toward Diverse Text Generation with Inverse Reinforcement Learning AU - Shi, Zhan AU - Chen, Xinchi AU - Qiu, Xipeng AU - Huang, Xuanjing T2 - Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18 PB - International Joint Conferences on Artificial Intelligence Organization PY - 2018 SP - 4361 EP - 4367 DO - 10.24963/ijcai.2018/606 UR - https://doi.org/10.24963/ijcai.2018/606 ER -Copied
Machine-readable versions of this paper, for tools and AI agents:
