{"type":"ScholarlyArticle","title":"Evaluating Natural Language Generation via Unbalanced Optimal Transport","authors":[{"name":"Yimeng Chen","slug":"yimeng-chen","given":"Yimeng","family":"Chen","affiliation":"University of Chinese Academy of Sciences; Academy of Mathematics and Systems Science , CAS"},{"name":"Yanyan Lan","slug":"yanyan-lan","given":"Yanyan","family":"Lan","affiliation":"University of Chinese Academy of Sciences; CAS Key Lab of Network Data Science and Technology , Institute of Computing Technology , CAS"},{"name":"Ruinbin Xiong","slug":"ruinbin-xiong","given":"Ruinbin","family":"Xiong","affiliation":"University of Chinese Academy of Sciences; CAS Key Lab of Network Data Science and Technology , Institute of Computing Technology , CAS"},{"name":"Liang Pang","slug":"liang-pang","given":"Liang","family":"Pang","affiliation":"University of Chinese Academy of Sciences; CAS Key Lab of Network Data Science and Technology , Institute of Computing Technology , CAS"},{"name":"Zhiming Ma","slug":"zhiming-ma","given":"Zhiming","family":"Ma","affiliation":"University of Chinese Academy of Sciences; Academy of Mathematics and Systems Science , CAS"},{"name":"Xueqi Cheng","slug":"xueqi-cheng","given":"Xueqi","family":"Cheng","affiliation":"University of Chinese Academy of Sciences; CAS Key Lab of Network Data Science and Technology , Institute of Computing Technology , CAS"}],"abstract":"Embedding-based evaluation measures have shown promising improvements on the correlation with human judgments in natural language generation. In these measures, various intrinsic metrics are used in the computation, including generalized precision, recall, F-score and the earth mover's distance. However, the relations between these metrics are unclear, making it difficult to determine which measure to use in real applications. In this paper, we provide an in-depth study on the relations between these metrics. Inspired by the optimal transportation theory, we prove that these metrics correspond to the optimal transport problem with different hard marginal constraints. However, these hard marginal constraints may cause the problem of incomplete and noisy matching in the evaluation process. Therefore we propose a family of new evaluation metrics, namely Lazy Earth Mover's Distances, based on the more general unbalanced optimal transport problem. Experimental results on WMT18 and WMT19 show that our proposed metrics have the ability to produce more consistent evaluation results with human judgements, as compared with existing intrinsic metrics.","doi":"10.24963/ijcai.2020/516","url":"https://ijcai.org/proceedings/2020/516","pdf":"https://ijcai.org/proceedings/2020/0516.pdf","markdown":"https://ijcai.org/proceedings/2020/516.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/516","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":516,"pages":{"first":3730,"last":3736},"keywords":["Natural Language Processing: Natural Language Generation","Natural Language Processing: Machine Translation","Natural Language Processing: Dialogue"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25165/file_25165.m3u8","length_sec":291,"provider":"hls"}],"references":[{"position":1,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Banerjee and Lavie, 2005] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceed- ings of the acl workshop on intrinsic and extrinsic evalua- tion measures for machine translation and/or summariza- tion, pages 65-72, 2005.","title":"Meteor: An automatic metric for mt evaluation with improved correlation with human judgments","authors":"Lavie Banerjee, Satanjeev Banerjee, Alon Lavie","year":"2005","venue":"Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chow et al., 2019] Julian Chow, Lucia Specia, and Pranava Swaroop Madhyastha. Wmdo: Fluency-based word mover's distance for machine translation evaluation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 494-500, 2019.","title":"Wmdo: Fluency-based word mover's distance for machine translation evaluation","authors":"Julian Chow, Lucia Specia, Pranava Swaroop, Madhyastha","year":"2019","venue":"Proceedings of the Fourth Conference on Machine Translation","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Flamary and Courty, 2017] R'emi Flamary and Nicolas Courty. Pot python optimal transport library, 2017.","title":"R'emi Flamary and Nicolas Courty. Pot python optimal transport library","authors":"Courty Flamary","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kilickaya et al., 2016] Mert Kilickaya, Aykut Erdem, Na- zli Ikizler-Cinbis, and Erkut Erdem. Re-evaluating au- tomatic metrics for image captioning. arXiv preprint arXiv:1612.07600, 2016.","title":"Re-evaluating automatic metrics for image captioning","authors":"Mert Kilickaya, Aykut Erdem, Nazli Ikizler-Cinbis, Erkut Erdem","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kullback and Leibler, 1951] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79-86, 1951.","title":"On information and sufficiency","authors":"Leibler ; A Kullback, Leibler","year":"1951","venue":"The annals of mathematical statistics","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kusner et al., 2015] Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to docu- ment distances. In International conference on machine learning, pages 957-966, 2015.","title":"From word embeddings to document distances","authors":"Matt Kusner, Yu Sun, Nicholas Kolkin, Kilian Weinberger","year":"2015","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2016] Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. Deep reinforce- ment learning for dialogue generation. arXiv preprint arXiv:1606.01541, 2016.","title":"Deep reinforcement learning for dialogue generation","authors":"Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, Dan Jurafsky","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin, 2004] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.","title":"Rouge: A package for automatic evaluation of summaries","authors":"Chin-Yew Lin","year":"2004","venue":"Text summarization branches out","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lo, 2019] Chi-kiu Lo. Yisi-a unified semantic mt quality evaluation and estimation metric for languages with dif- ferent levels of available resources. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 507-513, 2019.","title":"Chi-kiu Lo. Yisi-a unified semantic mt quality evaluation and estimation metric for languages with different levels of available resources","authors":"Lo","year":"2019","venue":"Proceedings of the Fourth Conference on Machine Translation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ma et al., 2018] Qingsong Ma, Ondřej Bojar, and Yvette Graham. Results of the wmt18 metrics shared task: Both characters and embeddings achieve good performance. In Proceedings of the third conference on machine transla- tion: shared task papers, pages 671-688, 2018.","title":"Results of the wmt18 metrics shared task: Both characters and embeddings achieve good performance","authors":"Ma","year":"2018","venue":"Proceedings of the third conference on machine translation: shared task papers","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ma et al., 2019] Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. Results of the wmt19 metrics shared task: Segment-level and strong mt systems pose big chal- lenges. In Proceedings of the Fourth Conference on Ma- chine Translation (Volume 2: Shared Task Papers, Day 1), pages 62-90, 2019.","title":"Results of the wmt19 metrics shared task: Segment-level and strong mt systems pose big challenges","authors":"Qingsong Ma, Johnny Wei, Ondřej Bojar, Yvette Graham","year":"2019","venue":"Proceedings of the Fourth Conference on Machine Translation","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Novikova et al., 2017] Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. Why we need new evaluation metrics for nlg. arXiv preprint arXiv:1707.06875, 2017.","title":"Why we need new evaluation metrics for nlg","authors":"Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, Verena Rieser","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for auto- matic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"Proceedings of the 40th annual meeting on association for computational linguistics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Peters et al., 2018] Matthew E Peters, Mark Neumann, Mo- hit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word repre- sentations. arXiv preprint arXiv:1802.05365, 2018.","title":"Deep contextualized word representations","authors":"Mark Matthew E Peters, Mohit Neumann, Matt Iyyer, Christopher Gardner, Kenton Clark, Luke Lee, Zettlemoyer","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Peyré et al., 2019] Gabriel Peyré, Marco Cuturi, et al. Com- putational optimal transport. Foundations and Trends R in Machine Learning, 11(5-6):355-607, 2019.","title":"Computational optimal transport","authors":"Gabriel Peyré, Marco Cuturi","year":"2019","venue":"Foundations and Trends R in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rubner et al., 1998] Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. A metric for distributions with ap- plications to image databases. In Sixth International Con- ference on Computer Vision (IEEE Cat. No. 98CH36271), pages 59-66. IEEE, 1998.","title":"A metric for distributions with applications to image databases","authors":"Yossi Rubner, Carlo Tomasi, Leonidas J Guibas","year":"1998","venue":"Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rus and Lintean, 2012] Vasile Rus and Mihai Lintean. A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity met- rics. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 157-162. As- sociation for Computational Linguistics, 2012.","title":"A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metrics","authors":"Vasile Rus, Mihai Lintean","year":"2012","venue":"Proceedings of the Seventh Workshop on Building Educational Applications Using NLP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048-2057, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, Yoshua Bengio","year":"2015","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2020] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Con- ference on Learning Representations, 2020.","title":"Bertscore: Evaluating text generation with bert","authors":"Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, Yoav Artzi","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhao et al., 2019] Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. Mover- score: Text generation evaluating with contextualized em- beddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP), pages 563-578, 2019.","title":"Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance","authors":"Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, Steffen Eger","year":"2019","venue":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}