{"type":"ScholarlyArticle","title":"Toward Diverse Text Generation with Inverse Reinforcement Learning","authors":[{"name":"Zhan Shi","slug":"zhan-shi","given":"Zhan","family":"Shi","affiliation":"Shanghai Key Laboratory of Intelligent Information Processing , Fudan University School of Computer Science , Fudan University"},{"name":"Xinchi Chen","slug":"xinchi-chen","given":"Xinchi","family":"Chen","affiliation":"Shanghai Key Laboratory of Intelligent Information Processing , Fudan University School of Computer Science , Fudan University"},{"name":"Xipeng Qiu","slug":"xipeng-qiu","given":"Xipeng","family":"Qiu","affiliation":"Shanghai Key Laboratory of Intelligent Information Processing , Fudan University School of Computer Science , Fudan University"},{"name":"Xuanjing Huang","slug":"xuanjing-huang","given":"Xuanjing","family":"Huang","affiliation":"Shanghai Key Laboratory of Intelligent Information Processing , Fudan University School of Computer Science , Fudan University"}],"abstract":"Text generation is a crucial task in NLP. Recently, several adversarial generative models have been proposed to improve the exposure bias problem in text generation. Though these models gain great success, they still suffer from the problems of reward sparsity and mode collapse. In order to address these two problems, in this paper, we employ inverse reinforcement learning (IRL) for text generation. Specifically, the IRL framework learns a reward function on training data, and then an optimal policy to maximum the expected total reward. Similar to the adversarial models, the reward and policy function in IRL are optimized alternately. Our method has two advantages: (1)  the reward function can produce more dense reward signals. (2) the generation policy, trained by ``entropy regularized'' policy gradient,  encourages to generate more diversified texts. Experiment results demonstrate that our proposed method can generate higher quality texts than the previous methods.","doi":"10.24963/ijcai.2018/606","url":"https://ijcai.org/proceedings/2018/606","pdf":"https://ijcai.org/proceedings/2018/0606.pdf","markdown":"https://ijcai.org/proceedings/2018/606.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/606","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":606,"pages":{"first":4361,"last":4367},"keywords":["Machine Learning Applications: Applications of Reinforcement Learning","Natural Language Processing: Natural Language Generation"],"videos":[],"references":[{"position":1,"raw":"[Bayer and Osendorfer, 2014] Justin Bayer and Christian Osendorfer. Learning stochastic recurrent networks. arXiv, 2014.","title":"Justin Bayer and Christian Osendorfer. Learning stochastic recurrent networks","authors":"Osendorfer Bayer","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2015] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for se- quence prediction with recurrent neural networks. In NIPS, pages 1171-1179, 2015.","title":"Scheduled sampling for sequence prediction with recurrent neural networks","authors":"Samy Bengio, Oriol Vinyals, Navdeep Jaitly, Noam Shazeer","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Che et al., 2017] Tong Che, Yanran Li, Ruixiang Zhang, R Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua Bengio. Maximum-likelihood augmented discrete genera- tive adversarial networks. arXiv, 2017.","title":"Maximum-likelihood augmented discrete generative adversarial networks","authors":"Yanran Tong Che, Ruixiang Li, Devon Zhang, Wenjie Hjelm, Yangqiu Li, Yoshua Song, Bengio","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2015] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data col- lection and evaluation server. arXiv, 2015.","title":"Microsoft coco captions: Data collection and evaluation server","authors":"Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, C Lawrence, Zitnick","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chung et al., 2015] Junyoung Chung, Kyle Kastner, Lau- rent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In NIPS, pages 2980-2988, 2015.","title":"A recurrent latent variable model for sequential data","authors":"Chung","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Diao et al., 2014] Qiming Diao, Minghui Qiu, Chao-Yuan Wu, Alexander J Smola, Jing Jiang, and Chong Wang. Jointly modeling aspects, ratings and sentiments for movie recommendation (jmars). In SIGKDD, pages 193-202. ACM, 2014.","title":"Jointly modeling aspects, ratings and sentiments for movie recommendation (jmars)","authors":"Qiming Diao, Minghui Qiu, Chao-Yuan, Alexander J Wu, Jing Smola, Chong Jiang, Wang","year":"2014","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Finn et al., 2016] Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal con- trol via policy optimization. In ICML, pages 49-58, 2016.","title":"Guided cost learning: Deep inverse optimal control via policy optimization","authors":"Finn","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In NIPS, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Graves, 2013] Alex Graves. Generating sequences with re- current neural networks. arXiv, 2013.","title":"Generating sequences with recurrent neural networks","authors":"Alex Graves","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guo et al., 2017] Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information. arXiv, 2017.","title":"Long text generation via adversarial training with leaked information","authors":"Guo","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Joji Toyama, 2018] Kotaro Nakayama Yutaka Matsuo Joji Toyama, Yusuke Iwasawa. Toward learning better metrics for sequence generation training with policy gradient. ICLR submission, 2018.","title":"Toward learning better metrics for sequence generation training with policy gradient","authors":"Kotaro Nakayama, Yutaka Matsuo, Joji Toyama, Yusuke Iwasawa","year":"2018","venue":"ICLR submission","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kusner and Hernández-Lobato, 2016] Matt J Kusner and José Miguel Hernández-Lobato. GANs for sequences of discrete elements with the gumbel-softmax distribution. arXiv, 2016.","title":"GANs for sequences of discrete elements with the gumbel-softmax distribution","authors":"Hernández-Lobato ; Kusner, J Matt, José Kusner, Miguel Hernández-Lobato","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin et al., 2017] Kevin Lin, Dianqi Li, Xiaodong He, Ming- ting Sun, and Zhengyou Zhang. Adversarial ranking for language generation. In NIPS, pages 3158-3168, 2017.","title":"Adversarial ranking for language generation","authors":"Kevin Lin, Dianqi Li, Xiaodong He, Mingting Sun, Zhengyou Zhang","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Metz et al., 2016] Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. arXiv, 2016.","title":"Unrolled generative adversarial networks","authors":"Metz","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nachum et al., 2017] Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. arXiv, 2017.","title":"Bridging the gap between value and policy based reinforcement learning","authors":"Ofir Nachum, Mohammad Norouzi, Kelvin Xu, Dale Schuurmans","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Salimans et al., 2016] Tim Salimans, Ian Goodfellow, Woj- ciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NIPS, pages 2234-2242, 2016.","title":"Improved techniques for training gans","authors":"Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Serban et al., 2017] Iulian Vlad Serban, Alessandro Sor- doni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C Courville, and Yoshua Bengio. A hierarchical latent vari- able encoder-decoder model for generating dialogues. In AAAI, pages 3295-3301, 2017.","title":"A hierarchical latent variable encoder-decoder model for generating dialogues","authors":"Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C Courville, Yoshua Bengio","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Su et al., 2018] Jinyue Su, Jiacheng Xu, Xipeng Qiu, and Xuanjing Huang. Incorporating discriminator in sentence generation: a gibbs sampling method. In AAAI, 2018.","title":"Incorporating discriminator in sentence generation: a gibbs sampling method","authors":"Su","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2017] Heng Wang, Zengchang Qin, and Tao Wan. Text generation based on generative adversarial nets with latent variable. arXiv, 2017.","title":"Text generation based on generative adversarial nets with latent variable","authors":"Heng Wang, Zengchang Qin, Tao Wan","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4):229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yu et al., 2017] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. SeqGAN: Sequence generative adversarial nets with policy gradient. In AAAI, pages 2852-2858, 2017.","title":"SeqGAN: Sequence generative adversarial nets with policy gradient","authors":"Lantao Yu, Weinan Zhang, Jun Wang, Yong Yu","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2017] Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. Adversarial feature matching for text generation. arXiv, 2017.","title":"Adversarial feature matching for text generation","authors":"Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ziebart et al., 2008] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.","title":"Maximum entropy inverse reinforcement learning","authors":"Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey","year":"2008","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}