{"type":"ScholarlyArticle","title":"Neural Machine Translation with Error Correction","authors":[{"name":"Kaitao Song","slug":"kaitao-song","given":"Kaitao","family":"Song","affiliation":"Nanjing University of Science and Technology"},{"name":"Xu Tan","slug":"xu-tan","given":"Xu","family":"Tan","affiliation":"Microsoft Research Asia"},{"name":"Jianfeng Lu","slug":"jianfeng-lu","given":"Jianfeng","family":"Lu","affiliation":"Nanjing University of Science and Technology"}],"abstract":"Neural machine translation (NMT) generates the next target token given as input the previous ground truth target tokens during training while the previous generated target tokens during inference, which causes discrepancy between training and inference as well as error propagation, and affects the translation accuracy. In this paper, we introduce an error correction mechanism into NMT, which corrects the error information in the previous generated tokens to better predict the next token. Specifically, we introduce two-stream self-attention from XLNet into NMT decoder, where the query stream is used to predict the next token, and meanwhile the content stream is used to correct the error information from the previous predicted tokens. We leverage scheduled sampling to simulate the prediction errors during training. Experiments on three IWSLT translation datasets and two WMT translation datasets demonstrate that our method achieves improvements over Transformer baseline and scheduled sampling. Further experimental analyses also verify the effectiveness of our proposed error correction mechanism to improve the translation quality.","doi":"10.24963/ijcai.2020/538","url":"https://ijcai.org/proceedings/2020/538","pdf":"https://ijcai.org/proceedings/2020/0538.pdf","markdown":"https://ijcai.org/proceedings/2020/538.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/538","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":538,"pages":{"first":3891,"last":3897},"keywords":["Natural Language Processing: Natural Language Generation","Natural Language Processing: Machine Translation"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24809/file_24809.m3u8","length_sec":272,"provider":"hls"}],"references":[{"position":1,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv e-prints, abs/1409.0473, September 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2015] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for se- quence prediction with recurrent neural networks. In NIPS, pages 1171-1179. 2015.","title":"Scheduled sampling for sequence prediction with recurrent neural networks","authors":"Samy Bengio, Oriol Vinyals, Navdeep Jaitly, Noam Shazeer","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, oct 2014.","title":"Learning phrase representations using RNN encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL, pages 4171-4186, 2019.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hassan et al., 2018] Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Fe- dermann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, et al. Achieving human parity on automatic chinese to english news translation. arXiv preprint arXiv:1803.05567, 2018.","title":"Achieving human parity on automatic chinese to english news translation","authors":"Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2018] Tianyu He, Xu Tan, Yingce Xia, Di He, Tao Qin, Zhibo Chen, and Tie-Yan Liu. Layer-wise coordi- nation between encoder and decoder for neural machine translation. In NIPS, pages 7944-7954. 2018.","title":"Layer-wise coordination between encoder and decoder for neural machine translation","authors":"Tianyu He, Xu Tan, Yingce Xia, Di He, Tao Qin, Zhibo Chen, Tie-Yan Liu","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kingma and Ba, 2014] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference for Learning Representations, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":"International Conference for Learning Representations","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ott et al., 2019] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for se- quence modeling. In NAACL, June 2019.","title":"fairseq: A fast, extensible toolkit for sequence modeling","authors":"Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, Michael Auli","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":"OpenAI Blog","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ranzato et al., 2016] Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In ICLR, 2016.","title":"Sequence level training with recurrent neural networks","authors":"Ranzato","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Sennrich et al., 2016] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, August 2016.","title":"Neural machine translation of rare words with subword units","authors":"Sennrich","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Song et al., 2018] Kaitao Song, Xu Tan, Di He, Jianfeng Lu, Tao Qin, and Tie-Yan Liu. Double path networks for sequence to sequence learning. In COLING, pages 3064- 3074, August 2018.","title":"Double path networks for sequence to sequence learning","authors":"Song","year":"2018","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Song et al., 2019] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mass: Masked sequence to se- quence pre-training for language generation. In ICML, pages 5926-5936, 2019.","title":"Mass: Masked sequence to sequence pre-training for language generation","authors":"Song","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27, pages 3104-3112. 2014.","title":"Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Tan et al., 2019] Xu Tan, Yingce Xia, Lijun Wu, and Tao Qin. Efficient bidirectional neural machine translation. arXiv preprint arXiv:1908.09329, 2019.","title":"Efficient bidirectional neural machine translation","authors":"Yingce Xu Tan, Lijun Xia, Tao Wu, Qin","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł Ukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Venkatraman et al., 2015] Arun Venkatraman, Martial Hebert, and J. Andrew Bagnell. Improving multi-step prediction of learned time series models. In AAAI, 2015.","title":"Improving multi-step prediction of learned time series models","authors":"Arun Venkatraman, Martial Hebert, J Andrew Bagnell","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wu et al., 2018] Lijun Wu, Xu Tan, Di He, Fei Tian, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. Beyond error prop- agation in neural machine translation: Characteristics of language also matter. arXiv preprint arXiv:1809.00120, 2018.","title":"Beyond error propagation in neural machine translation: Characteristics of language also matter","authors":"Lijun Wu, Xu Tan, Di He, Fei Tian, Tao Qin, Jianhuang Lai, Tie-Yan Liu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wu et al., 2019] Lijun Wu, Xu Tan, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. Beyond error propagation: Language branching also affects the accuracy of sequence genera- tion. IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, 27(12):1868-1879, 2019.","title":"Beyond error propagation: Language branching also affects the accuracy of sequence generation","authors":"Lijun Wu, Xu Tan, Tao Qin, Jianhuang Lai, Tie-Yan Liu","year":"2019","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xia et al., 2019] Yingce Xia, Tianyu He, Xu Tan, Fei Tian, Di He, and Tao Qin. Tied transformers: Neural machine translation with shared encoder and decoder. In AAAI, pages 5466-5473, 2019.","title":"Tied transformers: Neural machine translation with shared encoder and decoder","authors":"Yingce Xia, Tianyu He, Xu Tan, Fei Tian, Di He, Tao Qin","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2019] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. In NIPS, pages 5754-5764, 2019.","title":"Xlnet: Generalized autoregressive pretraining for language understanding","authors":"Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, Quoc V Le","year":"2019","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2019] Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. Bridging the gap between training and inference for neural machine translation. In ACL, pages 4334-4343, July 2019.","title":"Bridging the gap between training and inference for neural machine translation","authors":"Wen Zhang, Yang Feng, Fandong Meng, Di You, Qun Liu","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}