{"type":"ScholarlyArticle","title":"Get The Point of My Utterance! Learning Towards Effective Responses with Multi-Head Attention Mechanism","authors":[{"name":"Chongyang Tao","slug":"chongyang-tao","given":"Chongyang","family":"Tao","affiliation":"† Institute of Computer Science and Technology , Peking University , Beijing , China"},{"name":"Shen Gao","slug":"shen-gao","given":"Shen","family":"Gao","affiliation":"† Institute of Computer Science and Technology , Peking University , Beijing , China; ♦"},{"name":"Mingyue Shang","slug":"mingyue-shang","given":"Mingyue","family":"Shang","affiliation":"† Institute of Computer Science and Technology , Peking University , Beijing , China"},{"name":"Wei Wu","slug":"wei-wu","given":"Wei","family":"Wu","affiliation":"Beijing Institute of Big Data Research , Beijing , China"},{"name":"Dongyan Zhao","slug":"dongyan-zhao","given":"Dongyan","family":"Zhao","affiliation":"† Institute of Computer Science and Technology , Peking University , Beijing , China"},{"name":"Rui Yan","slug":"rui-yan","given":"Rui","family":"Yan","affiliation":"‡ Microsoft Corporation , Beijing , China; † Institute of Computer Science and Technology , Peking University , Beijing , China"}],"abstract":"Attention mechanism has become a popular and widely used component in sequence-to-sequence models. However, previous research on neural generative dialogue systems always generates universal responses, and the attention distribution learned by the model always attends to the same semantic aspect. To solve this problem, in this paper, we propose a novel Multi-Head Attention Mechanism (MHAM) for generative dialog systems, which aims at capturing multiple semantic aspects from the user utterance. Further, a regularizer is formulated to force different attention heads to concentrate on certain aspects. The proposed mechanism leads to more informative, diverse, and relevant response generated. Experimental results show that our proposed model outperforms several strong baselines.","doi":"10.24963/ijcai.2018/614","url":"https://ijcai.org/proceedings/2018/614","pdf":"https://ijcai.org/proceedings/2018/0614.pdf","markdown":"https://ijcai.org/proceedings/2018/614.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/614","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":614,"pages":{"first":4418,"last":4424},"keywords":["Natural Language Processing: Dialogue","Natural Language Processing: Natural Language Generation"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bousmalis et al., 2016] Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Du- mitru Erhan. Domain separation networks. In NIPS, pages 343-351, 2016.","title":"Domain separation networks","authors":"Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, Dumitru Erhan","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the proper- ties of neural machine translation: Encoder-decoder ap- proaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103-111, 2014.","title":"On the properties of neural machine translation: Encoder-decoder approaches","authors":"Cho","year":"2014","venue":"Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chung et al., 2014] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evalua- tion of gated recurrent neural networks on sequence mod- eling. arXiv preprint arXiv:1412.3555, 2014.","title":"Empirical evaluation of gated recurrent neural networks on sequence modeling","authors":"Chung","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Duchi et al., 2011] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 12:2121-2159, 2011.","title":"Adaptive subgradient methods for online learning and stochastic optimization","authors":"John Duchi, Elad Hazan, Yoram Singer","year":"2011","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2016a] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models. In NAACL, pages 110-119, 2016.","title":"A diversity-promoting objective function for neural conversation models","authors":"Li","year":"2016","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2016b] Jiwei Li, Will Monroe, and Dan Jurafsky. A simple, fast diverse decoding algorithm for neural gen- eration. arXiv preprint arXiv:1611.08562, 2016.","title":"A simple, fast diverse decoding algorithm for neural generation","authors":"Li","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin et al., 2017] Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. In ICLR, 2017.","title":"A structured self-attentive sentence embedding","authors":"Lin","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2016] Chia-Wei Liu, Ryan Lowe, Iulian V Ser- ban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for di- alogue response generation. In EMNLP, pages 2122-2132, 2016.","title":"How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation","authors":"Chia-Wei Liu, Ryan Lowe, Michael Iulian V Serban, Laurent Noseworthy, Joelle Charlin, Pineau","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mou et al., 2016] Lili Mou, Yiping Song, Rui Yan, Ge Li, Lu Zhang, and Zhi Jin. Sequence to backward and forward sequences: A content-introducing approach to generative short-text conversation. In COLING, pages 3349-3358, 2016.","title":"Sequence to backward and forward sequences: A content-introducing approach to generative short-text conversation","authors":"Lili Mou, Yiping Song, Rui Yan, Ge Li, Lu Zhang, Zhi Jin","year":"2016","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for au- tomatic evaluation of machine translation. In ACL, pages 311-318, 2002.","title":"BLEU: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rush et al., 2015] Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In EMNLP, pages 379-389, 2015.","title":"A neural attention model for abstractive sentence summarization","authors":"Sumit Alexander M Rush, Jason Chopra, Weston","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Serban et al., 2017] Iulian Vlad Serban, Alessandro Sor- doni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C Courville, and Yoshua Bengio. A hierarchical latent vari- able encoder-decoder model for generating dialogues. In AAAI, pages 3295-3301, 2017.","title":"A hierarchical latent variable encoder-decoder model for generating dialogues","authors":"Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C Courville, Yoshua Bengio","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shang et al., 2015] Lifeng Shang, Zhengdong Lu, and Hang Li. Neural responding machine for short-text conversation. In ACL, pages 1577-1586, 2015.","title":"Neural responding machine for short-text conversation","authors":"Lifeng Shang, Zhengdong Lu, Hang Li","year":"2015","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Stent et al., 2005] Amanda Stent, Matthew Marge, and Mo- hit Singhai. Evaluating evaluation methods for generation in the presence of variation. In International Conference on Intelligent Text Processing and Computational Linguis- tics, pages 341-351, 2005.","title":"Evaluating evaluation methods for generation in the presence of variation","authors":"Amanda Stent, Matthew Marge, Mohit Singhai","year":"2005","venue":"International Conference on Intelligent Text Processing and Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NIPS, pages 3104-3112, 2014.","title":"Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tao et al., 2018] Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. RUBER: An unsupervised method for auto- matic evaluation of open-domain dialog systems. In AAAI, pages 722-729, 2018.","title":"RUBER: An unsupervised method for automatic evaluation of open-domain dialog systems","authors":"Chongyang Tao, Lili Mou, Dongyan Zhao, Rui Yan","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tian et al., 2017] Zhiliang Tian, Rui Yan, Lili Mou, Yiping Song, Yansong Feng, and Dongyan Zhao. How to make context more useful? An empirical study on context-aware neural conversational models. In ACL, volume 2, pages 231-236, 2017.","title":"How to make context more useful? An empirical study on context-aware neural conversational models","authors":"Zhiliang Tian, Rui Yan, Lili Mou, Yiping Song, Yansong Feng, Dongyan Zhao","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 6000-6010, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xing et al., 2017] Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma. Topic aware neural response generation. In AAAI, pages 3351-3357, 2017.","title":"Topic aware neural response generation","authors":"Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, Wei-Ying Ma","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu et al., 2017] Zhen Xu, Bingquan Liu, Baoxun Wang, SUN Chengjie, Xiaolong Wang, Zhuoran Wang, and Chao Qi. Neural response generation via gan with an approxi- mate embedding layer. In EMNLP, pages 628-637, 2017.","title":"Neural response generation via gan with an approximate embedding layer","authors":"Zhen Xu, Bingquan Liu, Baoxun Wang, Xiaolong Sun Chengjie, Zhuoran Wang, Chao Wang, Qi","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yan et al., 2016] Rui Yan, Yiping Song, Xiangyang Zhou, and Hua Wu. Shall i be your chat companion?: Towards an online human-computer conversation system. In CIKM, pages 649-658, 2016.","title":"Shall i be your chat companion?: Towards an online human-computer conversation system","authors":"Rui Yan, Yiping Song, Xiangyang Zhou, Hua Wu","year":"2016","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yao et al., 2017] Lili Yao, Yaoyuan Zhang, Yansong Feng, Dongyan Zhao, and Rui Yan. Towards implicit content- introducing for generative short-text conversation systems. In EMNLP, pages 2180-2189, 2017.","title":"Towards implicit contentintroducing for generative short-text conversation systems","authors":"Lili Yao, Yaoyuan Zhang, Yansong Feng, Dongyan Zhao, Rui Yan","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhao et al., 2017] Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. Learning discourse-level diversity for neural di- alog models using conditional variational autoencoders. In ACL, pages 654-664, 2017.","title":"Learning discourse-level diversity for neural dialog models using conditional variational autoencoders","authors":"Zhao","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhou et al., 2017] Ganbin Zhou, Ping Luo, Rongyu Cao, Fen Lin, Bo Chen, and Qing He. Mechanism-aware neural machine for dialogue response generation. In AAAI, pages 3400-3407, 2017.","title":"Mechanism-aware neural machine for dialogue response generation","authors":"Zhou","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}