{"type":"ScholarlyArticle","title":"Reinforcing Coherence for Sequence to Sequence Model in Dialogue Generation","authors":[{"name":"Hainan Zhang","slug":"hainan-zhang","given":"Hainan","family":"Zhang","affiliation":"University of Chinese Academy of Sciences , Beijing , China CAS Key Lab of Network Data Science and Technology , Institute of Computing Technology ,"},{"name":"Yanyan Lan","slug":"yanyan-lan","given":"Yanyan","family":"Lan","affiliation":"University of Chinese Academy of Sciences , Beijing , China CAS Key Lab of Network Data Science and Technology , Institute of Computing Technology ,"},{"name":"Jiafeng Guo","slug":"jiafeng-guo","given":"Jiafeng","family":"Guo","affiliation":"University of Chinese Academy of Sciences , Beijing , China CAS Key Lab of Network Data Science and Technology , Institute of Computing Technology ,"},{"name":"Jun Xu","slug":"jun-xu","given":"Jun","family":"Xu","affiliation":"University of Chinese Academy of Sciences , Beijing , China CAS Key Lab of Network Data Science and Technology , Institute of Computing Technology ,; Chinese Academy of Sciences"},{"name":"Xueqi Cheng","slug":"xueqi-cheng","given":"Xueqi","family":"Cheng","affiliation":"University of Chinese Academy of Sciences , Beijing , China CAS Key Lab of Network Data Science and Technology , Institute of Computing Technology ,"}],"abstract":"Sequence to sequence (Seq2Seq) approach has gained great attention in the field of single-turn dialogue generation. However, one serious problem is that most existing Seq2Seq based models tend to generate common responses lacking specific meanings. Our analysis show that the underlying reason is that Seq2Seq is equivalent to optimizing Kullback–Leibler (KL) divergence, thus does not penalize the case whose generated probability is high while the true probability is low. However, the true probability is unknown, which poses challenges for tackling this problem. Inspired by the fact that the coherence (i.e. similarity) between post and response is consistent with human evaluation, we hypothesize that the true probability of a response is proportional to the coherence degree. The coherence scores are then used as the reward function in a reinforcement learning framework to penalize the case whose generated probability is high while the true probability is low. Three different types of coherence models, including an unlearned similarity function, a pretrained semantic matching function, and an end-to-end dual learning architecture, are proposed in this paper. Experimental results on both Chinese Weibo dataset and English Subtitle dataset show that the proposed models produce more specific and meaningful responses, yielding better performances against Seq2Seq models in terms of both metric-based and human evaluations.","doi":"10.24963/ijcai.2018/635","url":"https://ijcai.org/proceedings/2018/635","pdf":"https://ijcai.org/proceedings/2018/0635.pdf","markdown":"https://ijcai.org/proceedings/2018/635.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/635","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":635,"pages":{"first":4567,"last":4573},"keywords":["Machine Learning: Learning Generative Models","Natural Language Processing: Natural Language Generation","Machine Learning: Neural Networks","Machine Learning: Deep Learning","Natural Language Processing: Dialogue"],"videos":[],"references":[{"position":1,"raw":"[Arjovsky et al., 2017] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. ICML, 2017.","title":"Wasserstein gan. ICML","authors":"Martin Arjovsky, Soumith Chintala, Léon Bottou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. Computer Science, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fleiss, 1971] Joseph L Fleiss. Measuring nominal scale agreement among many raters. American Psychological Association, 1971.","title":"Measuring nominal scale agreement among many raters","authors":"L Joseph, Fleiss","year":"1971","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Guo et al., 2016] Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. A deep relevance matching model for ad- hoc retrieval. In CIKM, pages 55-64. ACM, 2016.","title":"A deep relevance matching model for adhoc retrieval","authors":"Guo","year":"2016","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hu et al., 2015] Baotian Hu, Qingcai Chen, and Fangze Zhu. Lcsts: A large scale chinese short text summariza- tion dataset. EMNLP, 2015.","title":"Lcsts: A large scale chinese short text summarization dataset","authors":"Hu","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Li et al., 2016a] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting ob- jective function for neural conversation models. NAACL, 2016.","title":"A diversity-promoting objective function for neural conversation models","authors":"Li","year":"2016","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2016b] Jiwei Li, Will Monroe, Alan Ritter, and Galley et al. Deep reinforcement learning for dialogue generation. EMNLP, 2016.","title":"Deep reinforcement learning for dialogue generation","authors":"Li","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2017] Jiwei Li, Will Monroe, Tianlin Shi, Alan Ritter, and Dan Jurafsky. Adversarial learning for neural dialogue generation. EMNLP, 2017.","title":"Adversarial learning for neural dialogue generation","authors":"Li","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mou et al., 2017] Lili Mou, Yiping Song, Rui Yan, Ge Li, Lu Zhang, and Zhi Jin. Sequence to backward and forward sequences: A content-introducing approach to generative short-text conversation. ACL, 2017.","title":"Sequence to backward and forward sequences: A content-introducing approach to generative short-text conversation","authors":"Lili Mou, Yiping Song, Rui Yan, Ge Li, Lu Zhang, Zhi Jin","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Pang et al., 2016] Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. Text matching as image recognition. In AAAI, pages 2793-2799, 2016.","title":"Text matching as image recognition","authors":"Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, Xueqi Cheng","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Shang et al., 2015] Lifeng Shang, Zhengdong Lu, and Hang Li. Neural responding machine for short-text conversation. ACL, 2015.","title":"Neural responding machine for short-text conversation","authors":"Lifeng Shang, Zhengdong Lu, Hang Li","year":"2015","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Shen et al., 2017] Xiaoyu Shen, Hui Su, Yanran Li, Wenjie Li, Shuzi Niu, and Zhao et al. A conditional variational framework for dialog generation. ACL, 2017. [Socher et al., 2013]","title":"A conditional variational framework for dialog generation","authors":"Xiaoyu Shen, Hui Su, Yanran Li, Wenjie Li, Shuzi Niu, Zhao","year":"2013","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"Richard Socher, Danqi Chen, Christo- pher D Manning, and Andrew Ng. Reasoning with neural tensor networks for knowledge base completion. In NIPS, pages 926-934, 2013.","title":"Reasoning with neural tensor networks for knowledge base completion","authors":"Richard Socher, Danqi Chen, Christopher D Manning, Andrew Ng","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NIPS, pages 3104-3112, 2014.","title":"Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sutton et al., 2000] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approx- imation. In NIPS, pages 1057-1063, 2000.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David A Richard S Sutton, Mcallester, P Satinder, Yishay Singh, Mansour","year":"2000","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Vinyals and Le, 2015] Oriol Vinyals and Quoc Le. A neural conversational model. Computer Science, 2015.","title":"Oriol Vinyals and Quoc Le. A neural conversational model","authors":"Le Vinyals","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wan et al., 2016a] Shengxian Wan, Yanyan Lan, Jiafeng Guo, Jun Xu, Liang Pang, and Xueqi Cheng. A deep archi- tecture for semantic matching with multiple positional sen- tence representations. In AAAI, pages 2835-2841, 2016.","title":"A deep architecture for semantic matching with multiple positional sentence representations","authors":"Wan","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wan et al., 2016b] Shengxian Wan, Yanyan Lan, Jun Xu, Ji- afeng Guo, Liang Pang, and Xueqi Cheng. Match-srnn: Modeling the recursive matching structure with spatial rnn. IJCAI, 2016.","title":"Match-srnn: Modeling the recursive matching structure with spatial rnn","authors":"Wan","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/415"},{"position":20,"raw":"[Xia et al., 2016] Yingce Xia, Di He, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. Dual learn- ing for machine translation. NIPS, 2016.","title":"Dual learning for machine translation","authors":"Yingce Xia, Di He, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, Wei-Ying Ma","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xing et al., 2017] Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma. Topic aware neural response generation. In AAAI, pages 3351-3357, 2017.","title":"Topic aware neural response generation","authors":"Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, Wei-Ying Ma","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yu et al., 2017] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In AAAI, pages 2852-2858, 2017.","title":"Seqgan: Sequence generative adversarial nets with policy gradient","authors":"Lantao Yu, Weinan Zhang, Jun Wang, Yong Yu","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}