{"type":"ScholarlyArticle","title":"One \"Ruler\" for All Languages: Multi-Lingual Dialogue Evaluation with Adversarial Multi-Task Learning","authors":[{"name":"Xiaowei Tong","slug":"xiaowei-tong","given":"Xiaowei","family":"Tong","affiliation":"Institute of Computer Science and Technology , Peking University , China; Beijing Institute of Big Data Research , China"},{"name":"Zhenxin Fu","slug":"zhenxin-fu","given":"Zhenxin","family":"Fu","affiliation":"Institute of Computer Science and Technology , Peking University , China"},{"name":"Mingyue Shang","slug":"mingyue-shang","given":"Mingyue","family":"Shang","affiliation":"Institute of Computer Science and Technology , Peking University , China"},{"name":"Dongyan Zhao","slug":"dongyan-zhao","given":"Dongyan","family":"Zhao","affiliation":"Institute of Computer Science and Technology , Peking University , China; Beijing Institute of Big Data Research , China"},{"name":"Rui Yan","slug":"rui-yan","given":"Rui","family":"Yan","affiliation":"Institute of Computer Science and Technology , Peking University , China; Beijing Institute of Big Data Research , China"}],"abstract":"Automatic evaluating the performance of Open-domain dialogue system is a challenging problem. Recent work in neural network-based metrics has shown promising opportunities for automatic dialogue evaluation. However, existing methods mainly focus on monolingual evaluation, in which the trained metric is not flexible enough to transfer across different languages. To address this issue, we propose an adversarial multi-task neural metric (ADVMT) for multi-lingual dialogue evaluation, with shared feature extraction across languages. We evaluate the proposed model in two different languages. Experiments show that the adversarial multi-task neural metric achieves a high correlation with human annotation, which yields better performance than monolingual ones and various existing metrics.","doi":"10.24963/ijcai.2018/616","url":"https://ijcai.org/proceedings/2018/616","pdf":"https://ijcai.org/proceedings/2018/0616.pdf","markdown":"https://ijcai.org/proceedings/2018/616.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/616","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":616,"pages":{"first":4432,"last":4438},"keywords":["Natural Language Processing: Natural Language Processing","Humans and AI: Cognitive Systems"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Banerjee and Lavie, 2005] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In A- CL workshop, volume 29, pages 65-72, 2005.","title":"Meteor: An automatic metric for mt evaluation with improved correlation with human judgments","authors":"Lavie Banerjee, Satanjeev Banerjee, Alon Lavie","year":"2005","venue":"A-CL workshop","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ben-David et al., 2003] Shai Ben-David, Reba Schuller, et al. Exploiting task relatedness for multiple task learn- ing. Lecture notes in computer science, pages 567-580, 2003.","title":"Exploiting task relatedness for multiple task learning","authors":"Shai Ben-David, Reba Schuller","year":"2003","venue":"Lecture notes in computer science","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bousmalis et al., 2016] Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Du- mitru Erhan. Domain separation networks. In Advances in Neural Information Processing Systems, pages 343-351, 2016.","title":"Domain separation networks","authors":"Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, Dumitru Erhan","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Caruana, 1998] Rich Caruana. Multitask learning. In Learning to learn, pages 95-133. Springer, 1998.","title":"Multitask learning","authors":"Rich Caruana","year":"1998","venue":"Learning to learn","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2016] Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. Adversarial deep averaging networks for cross-lingual sentiment classifica- tion. arXiv preprint arXiv:1606.01614, 2016.","title":"Adversarial deep averaging networks for cross-lingual sentiment classification","authors":"Chen","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2017] Xinchi Chen, Zhan Shi, Xipeng Qiu, and Xuanjing Huang. Adversarial multi-criteria learning for chinese word segmentation. In ACL (Volume 1: Long Pa- pers), volume 1, pages 1193-1203, 2017.","title":"Adversarial multi-criteria learning for chinese word segmentation","authors":"Chen","year":"2017","venue":"Long Papers)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Collobert and Weston, 2008] Ronan Collobert and Jason Weston. A unified architecture for natural language pro- cessing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Ma- chine learning, pages 160-167. ACM, 2008.","title":"A unified architecture for natural language processing: Deep neural networks with multitask learning","authors":"Weston ; Collobert, Ronan Collobert, Jason Weston","year":"2008","venue":"Proceedings of the 25th international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ganin et al., 2016] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59):1-35, 2016.","title":"Domain-adversarial training of neural networks","authors":"Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Mario Franc ¸ois Laviolette, Victor Marchand, Lempitsky","year":"2016","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hashimoto et al., 2017] Kazuma Hashimoto, Yoshimasa T- suruoka, Richard Socher, et al. A joint many-task mod- el: Growing a neural network for multiple nlp tasks. In EMNLP, pages 1923-1933, 2017.","title":"A joint many-task model: Growing a neural network for multiple nlp tasks","authors":"Kazuma Hashimoto, Yoshimasa Tsuruoka, Richard Socher","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2016] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting ob- jective function for neural conversation models. In ACL, pages 110-119, 2016.","title":"A diversity-promoting objective function for neural conversation models","authors":"Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, Bill Dolan","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin, 2004] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In ACL workshop, volume 8. Barcelona, Spain, 2004.","title":"Rouge: A package for automatic evaluation of summaries","authors":"Chin-Yew Lin","year":"2004","venue":"ACL workshop","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2016a] Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue re- sponse generation. In EMNLP, pages 2122-2132, 2016.","title":"How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation","authors":"Liu","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2016b] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Deep multi-task learning with shared memory for text classification. In EMNLP, pages 118-127, 2016.","title":"Deep multi-task learning with shared memory for text classification","authors":"Liu","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2017] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Adversarial multi-task learning for text classifi- cation. In ACL (Volume 1: Long Papers), volume 1, pages 1-10, 2017.","title":"Adversarial multi-task learning for text classification","authors":"Pengfei Liu, Xipeng Qiu, Xuanjing Huang","year":"2017","venue":"Long Papers)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lowe et al., 2017] Ryan Lowe, Michael Noseworthy, Iu- lian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Ben- gio, and Joelle Pineau. Towards an automatic turing test: Learning to evaluate dialogue responses. In ACL (Volume 1: Long Papers), volume 1, pages 1116-1126, 2017.","title":"Towards an automatic turing test: Learning to evaluate dialogue responses","authors":"Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, Joelle Pineau","year":"2017","venue":"Long Papers)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311-318. Association for Computational Linguis- tics, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rus and Lintean, 2012] Vasile Rus and Mihai Lintean. A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity met- rics. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 157-162. As- sociation for Computational Linguistics, 2012.","title":"A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metrics","authors":"Vasile Rus, Mihai Lintean","year":"2012","venue":"Proceedings of the Seventh Workshop on Building Educational Applications Using NLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Serban et al., 2016] Iulian Vlad Serban, Alessandro Sor- doni, Yoshua Bengio, Aaron C Courville, and Joelle Pineau. Building end-to-end dialogue systems using gen- erative hierarchical neural network models. In AAAI, pages 3776-3784, 2016.","title":"Building end-to-end dialogue systems using generative hierarchical neural network models","authors":"Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C Courville, Joelle Pineau","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Song et al., 2018] Yiping Song, Rui Yan, Yansong Feng, Yaoyun Zhang, Dongyan Zhao, and Ming Zhang. Towards a neural conversation model with diversity net using deter- minantal point processes. In AAAI, 2018.","title":"Towards a neural conversation model with diversity net using determinantal point processes","authors":"Yiping Song, Rui Yan, Yansong Feng, Yaoyun Zhang, Dongyan Zhao, Ming Zhang","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tao et al., 2018] Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. Ruber: An unsupervised method for auto- matic evaluation of open-domain dialog systems. In AAAI, 2018.","title":"Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems","authors":"Chongyang Tao, Lili Mou, Dongyan Zhao, Rui Yan","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tian et al., 2017] Zhiliang Tian, Rui Yan, Lili Mou, Yiping Song, Yansong Feng, and Dongyan Zhao. How to make context more useful? an empirical study on context-aware neural conversational models. In ACL (Volume 2: Short Papers), volume 2, pages 231-236, 2017.","title":"How to make context more useful? an empirical study on context-aware neural conversational models","authors":"Zhiliang Tian, Rui Yan, Lili Mou, Yiping Song, Yansong Feng, Dongyan Zhao","year":"2017","venue":"Short Papers)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yan et al., 2016] Rui Yan, Yiping Song, and Hua Wu. Learning to respond with deep neural networks for retrieval-based human-computer conversation system. In SIGIR, pages 55-64. ACM, 2016.","title":"Learning to respond with deep neural networks for retrieval-based human-computer conversation system","authors":"Rui Yan, Yiping Song, Hua Wu","year":"2016","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yan et al., 2017] Rui Yan, Dongyan Zhao, et al. Joint learn- ing of response ranking and next utterance suggestion in human-computer conversation system. In SIGIR, pages 685-694. ACM, 2017.","title":"Joint learning of response ranking and next utterance suggestion in human-computer conversation system","authors":"Rui Yan, Dongyan Zhao","year":"2017","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yao et al., 2017] Lili Yao, Yaoyuan Zhang, Yansong Feng, Dongyan Zhao, and Rui Yan. Towards implicit content- introducing for generative short-text conversation systems. In EMNLP, pages 2190-2199, 2017.","title":"Towards implicit contentintroducing for generative short-text conversation systems","authors":"Lili Yao, Yaoyuan Zhang, Yansong Feng, Dongyan Zhao, Rui Yan","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}