{"type":"ScholarlyArticle","title":"Bridging the Gap between Training and Inference for Neural Machine Translation (Extended Abstract)","authors":[{"name":"Wen Zhang","slug":"wen-zhang","given":"Wen","family":"Zhang","affiliation":"Smart Platform Product Department of Tencent Inc ., China"},{"name":"Yang Feng","slug":"yang-feng","given":"Yang","family":"Feng","affiliation":"Key Laboratory of Intelligent Information Processing Institute of Computing Technology , Chinese Academy of Sciences (ICT/CAS); University of Chinese Academy of Sciences , Beijing , China"},{"name":"Qun Liu","slug":"qun-liu","given":"Qun","family":"Liu","affiliation":"Huawei Noah's Ark Lab , Hong Kong , China"}],"abstract":"Neural Machine Translation (NMT) generates target words sequentially in the way of predicting the next word conditioned on the context words. At training time, it predicts with the ground truth words as context while at inference it has to generate the entire sequence from scratch. This discrepancy of the fed context leads to error accumulation among the translation. Furthermore, word-level training requires strict matching between the generated sequence and the ground truth sequence which leads to overcorrection over different but reasonable translations. In this paper, we address these issues by sampling context words not only from the ground truth sequence but also from the predicted sequence during training. Experimental results on NIST Chinese->English and WMT2014 English->German translation tasks demonstrate that our method can achieve significant improvements on multiple data sets compared to strong baselines.","doi":"10.24963/ijcai.2020/667","url":"https://ijcai.org/proceedings/2020/667","pdf":"https://ijcai.org/proceedings/2020/0667.pdf","markdown":"https://ijcai.org/proceedings/2020/667.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/667","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":667,"pages":{"first":4790,"last":4794},"keywords":["Natural Language Processing: Natural Language Processing","Natural Language Processing: Machine Translation"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25470/file_25470.m3u8","length_sec":268,"provider":"hls"}],"references":[{"position":1,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. ICLR 2015, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2015] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for se- quence prediction with recurrent neural networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Pro- cessing Systems 28, pages 1171-1179.","title":"Scheduled sampling for sequence prediction with recurrent neural networks","authors":"Samy Bengio, Oriol Vinyals, Navdeep Jaitly, Noam Shazeer","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"Curran Associates, Inc., 2015.","title":null,"authors":"Curran Associates, Inc","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Collins et al., 2005] Michael Collins, Philipp Koehn, and Ivona Kucerova. Clause restructuring for statistical ma- chine translation. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguis- tics (ACL'05), pages 531-540, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics.","title":"Clause restructuring for statistical machine translation","authors":"Michael Collins, Philipp Koehn, Ivona Kucerova","year":"2005","venue":"Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gehring et al., 2017] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convo- lutional sequence to sequence learning. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th Inter- national Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1243- 1252, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR.","title":"Convolutional sequence to sequence learning","authors":"Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N Dauphin","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gumbel, 1954] Emil Julius Gumbel. Statistical theory of extreme valuse and some practical applications. Nat. Bur. Standards Appl. Math. Ser. 33, 1954.","title":"Statistical theory of extreme valuse and some practical applications","authors":"Emil Julius Gumbel, Gumbel","year":"1954","venue":"Nat. Bur. Standards Appl. Math. Ser","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lin, 2004] Chin-Yew Lin. ROUGE: A package for auto- matic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.","title":"ROUGE: A package for automatic evaluation of summaries","authors":"Chin-Yew Lin","year":"2004","venue":"Text Summarization Branches Out","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Maddison et al., 2014] Chris J Maddison, Daniel Tarlow, and Tom Minka. A* sampling. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Wein- berger, editors, Advances in Neural Information Process- ing Systems 27, pages 3086-3094.","title":"A* sampling","authors":"Chris J Maddison, Daniel Tarlow, Tom Minka","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Curran Associates, Inc., 2014.","title":null,"authors":"Curran Associates, Inc","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Meng and Zhang, 2019] Fandong Meng and Jinchao Zhang. Dtmt: A novel deep transition architecture for neural ma- chine translation. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence, AAAI'19. AAAI Press, 2019.","title":"Dtmt: A novel deep transition architecture for neural machine translation","authors":"Zhang Meng, Fandong Meng, Jinchao Zhang","year":"2019","venue":"Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence, AAAI'19","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for auto- matic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"Proceedings of the 40th annual meeting on association for computational linguistics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ranzato et al., 2015] Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732, 2015.","title":"Sequence level training with recurrent neural networks","authors":"Ranzato","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Sennrich et al., 2016] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Ger- many, August 2016. Association for Computational Lin- guistics. Shen et al., 2016]","title":"Neural machine translation of rare words with subword units","authors":"Sennrich","year":"2016","venue":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Mini- mum risk training for neural machine translation. In Pro- ceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol- ume 1, pages 1683-1692, 2016.","title":"Minimum risk training for neural machine translation","authors":"Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu","year":"2016","venue":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3104- 3112.","title":"Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"Curran Associates, Inc., 2014.","title":null,"authors":"Curran Associates, Inc","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tu et al., 2016] Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. Modeling coverage for neu- ral machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 76-85, Berlin, Germany, August 2016. Association for Computational Linguistics.","title":"Modeling coverage for neural machine translation","authors":"Tu","year":"2016","venue":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wal- lach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł Ukasz Kaiser, Illia Polosukhin","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"Curran Associates, Inc., 2017.","title":null,"authors":"Curran Associates, Inc","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Venkatraman et al., 2015] Arun Venkatraman, Martial Hebert, and J. Andrew Bagnell. Improving multi-step prediction of learned time series models. In Proceed- ings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI'15, pages 3024-3030. AAAI Press, 2015.","title":"Improving multi-step prediction of learned time series models","authors":"Arun Venkatraman, Martial Hebert, J Andrew Bagnell","year":"2015","venue":"Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI'15","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wu et al., 2016] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation sys- tem: Bridging the gap between human and machine trans- lation. arXiv preprint arXiv:1609.08144, 2016.","title":"Google's neural machine translation system: Bridging the gap between human and machine translation","authors":"Yonghui Wu, Mike Schuster, Zhifeng Chen, V Quoc, Mohammad Le, Wolfgang Norouzi, Maxim Macherey, Yuan Krikun, Qin Cao, Klaus Gao, Macherey","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2017] Jinchao Zhang, Mingxuan Wang, Qun Liu, and Jie Zhou. Incorporating word reordering knowl- edge into attention-based neural machine translation. In Proceedings of the 55th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Pa- pers), pages 1524-1534, Vancouver, Canada, July 2017. Association for Computational Linguistics.","title":"Incorporating word reordering knowledge into attention-based neural machine translation","authors":"Zhang","year":"2017","venue":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2019] Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. Bridging the gap between training and inference for neural machine translation. In Proceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguistics, pages 4334-4343, Florence, Italy, July 2019. Association for Computational Linguistics.","title":"Bridging the gap between training and inference for neural machine translation","authors":"Wen Zhang, Yang Feng, Fandong Meng, Di You, Qun Liu","year":"2019","venue":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}