{"type":"ScholarlyArticle","title":"Explicit Alignment Learning for Neural Machine Translation","authors":[{"name":"Zuchao Li","slug":"zuchao-li","given":"Zuchao","family":"Li","affiliation":"Department of Computer Science and Engineering , Shanghai Jiao Tong University; MoE Key Lab of Artificial Intelligence , AI Institute , Shanghai Jiao Tong University"},{"name":"Hai Zhao","slug":"hai-zhao","given":"Hai","family":"Zhao","affiliation":"Department of Computer Science and Engineering , Shanghai Jiao Tong University; MoE Key Lab of Artificial Intelligence , AI Institute , Shanghai Jiao Tong University"},{"name":"Fengshun Xiao","slug":"fengshun-xiao","given":"Fengshun","family":"Xiao","affiliation":"Department of Computer Science and Engineering , Shanghai Jiao Tong University; MoE Key Lab of Artificial Intelligence , AI Institute , Shanghai Jiao Tong University"},{"name":"Masao Utiyama","slug":"masao-utiyama","given":"Masao","family":"Utiyama","affiliation":"National Institute of Information and Communications Technology (NICT) , Kyoto , Japan"},{"name":"Eiichiro Sumita","slug":"eiichiro-sumita","given":"Eiichiro","family":"Sumita","affiliation":"National Institute of Information and Communications Technology (NICT) , Kyoto , Japan"}],"abstract":"Even though neural machine translation (NMT) has become the state-of-the-art solution for end-to-end translation, it still suffers from a lack of translation interpretability, which may be conveniently enhanced by explicit alignment learning (EAL), as performed in traditional statistical machine translation (SMT). To provide the benefits of both NMT and SMT, this paper presents a novel model design that enhances NMT with an additional training process for EAL, in addition to the end-to-end translation training. \r\nThus, we propose two approaches an explicit alignment learning approach, in which we further remove the need for the additional alignment model, and perform embedding mixup with the alignment based on encoder--decoder attention weights in the NMT model. We conducted experiments on both small-scale (IWSLT14 De->En and IWSLT13 Fr->En) and large-scale (WMT14 En->De, En->Fr, WMT17 Zh->En) benchmarks. Evaluation results show that our EAL methods significantly outperformed strong baseline methods, which shows the effectiveness of EAL. Further explorations show that the translation improvements are due to a better spatial alignment of the source and target language embeddings. Our method improves translation performance without the need to increase model parameters and training data, which verifies that the idea of incorporating techniques of SMT into NMT is worthwhile.","doi":"10.24963/ijcai.2022/587","url":"https://ijcai.org/proceedings/2022/587","pdf":"https://ijcai.org/proceedings/2022/0587.pdf","markdown":"https://ijcai.org/proceedings/2022/587.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/587","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":587,"pages":{"first":4230,"last":4237},"keywords":["Natural Language Processing: Machine Translation and Multilinguality","Natural Language Processing: Language Generation"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986792","length_sec":92,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984935","length_sec":323,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Al-Onaizan and Papineni, 2006] Yaser Al-Onaizan and Kishore Papineni. Distortion models for statistical machine translation. In COLING, 2006.","title":"Distortion models for statistical machine translation","authors":"Yaser Al-Onaizan, Kishore Papineni","year":"2006","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alkhouli and Ney, 2017] Tamer Alkhouli and Hermann Ney. Biasing attention-based recurrent neural networks us- ing external alignment information. In WMT, 2017.","title":"Tamer Alkhouli and Hermann Ney. Biasing attention-based recurrent neural networks using external alignment information","authors":"Ney Alkhouli","year":"2017","venue":"WMT","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Alkhouli et al., 2018] Tamer Alkhouli, Gabriel Bretschner, et al. On the alignment problem in multi-head attention- based neural machine translation. In WMT, 2018.","title":"Tamer Alkhouli, Gabriel Bretschner, et al. On the alignment problem in multi-head attentionbased neural machine translation","authors":"Alkhouli","year":"2018","venue":"WMT","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, et al. Neural machine translation by jointly learning to align and translate. In ICLR, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brown et al., 1993] Peter F. Brown, Stephen Della Pietra, et al. The mathematics of statistical machine translation: Parameter estimation. CL, 1993.","title":"The mathematics of statistical machine translation: Parameter estimation","authors":"F Peter, Stephen Brown, Della Pietra","year":"1993","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chatterjee et al., 2017] Rajen Chatterjee, Matteo Negri, et al. Guiding neural machine translation decoding with external knowledge. In WMT, 2017.","title":"Guiding neural machine translation decoding with external knowledge","authors":"Rajen Chatterjee, Matteo Negri","year":"2017","venue":"WMT","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Duan et al., 2021] Chaoqun Duan, Kehai Chen, and othrs. Modeling future cost for neural machine translation. TASLP, 2021.","title":"Modeling future cost for neural machine translation","authors":"Duan","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Garg et al., 2019a] Sarthak Garg, Stephan Peitz, et al. Jointly learning to align and translate with transformer models. In EMNLP, 2019.","title":"Jointly learning to align and translate with transformer models","authors":"Garg","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Garg et al., 2019b] Sarthak Garg, Stephan Peitz, et al. Jointly learning to align and translate with transformer models. In EMNLP, 2019.","title":"Jointly learning to align and translate with transformer models","authors":"Garg","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gehring et al., 2017a] Jonas Gehring, Michael Auli, et al. A convolutional encoder model for neural machine trans- lation. In ACL, 2017.","title":"A convolutional encoder model for neural machine translation","authors":"Gehring","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gehring et al., 2017b] Jonas Gehring, Michael Auli, et al. Convolutional sequence to sequence learning. In ICML, 2017.","title":"Convolutional sequence to sequence learning","authors":"Gehring","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2016] Wei He, Zhongjun He, et al. Improved neu- ral machine translation with smt features. In AAAI, 2016.","title":"Improved neural machine translation with smt features","authors":"Wei He, Zhongjun He","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Koehn et al., 2003a] Philipp Koehn, Franz J. Och, et al. Sta- tistical phrase-based translation. In NAACL, 2003.","title":"Statistical phrase-based translation","authors":"Koehn","year":"2003","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Koehn et al., 2003b] Philipp Koehn, Franz Josef Och, et al. Statistical phrase-based translation. In NAACL, 2003.","title":"Statistical phrase-based translation","authors":"Koehn","year":"2003","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2016] Lemao Liu, Masao Utiyama, et al. Neural machine translation with supervised attention. In COL- ING, 2016.","title":"Neural machine translation with supervised attention","authors":"Liu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2019] Xuebo Liu, Derek F Wong, et al. Shared- private bilingual word embeddings for neural machine translation. In ACL, 2019.","title":"Sharedprivate bilingual word embeddings for neural machine translation","authors":"Liu","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Luong et al., 2015] Minh-Thang Luong, Hieu Pham, et al. Effective approaches to attention-based neural machine translation. In EMNLP, 2015.","title":"Effective approaches to attention-based neural machine translation","authors":"Minh-Thang Luong, Hieu Pham","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mi et al., 2016] Haitao Mi, Zhiguo Wang, et al. Supervised attentions for neural machine translation. In EMNLP, 2016.","title":"Supervised attentions for neural machine translation","authors":"Haitao Mi, Zhiguo Wang","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, et al. Ef- ficient estimation of word representations in vector space. In ICLR, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Mikolov","year":"2013","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Och and Ney, 2003] Franz Josef Och and Hermann Ney. A systematic comparison of various statistical alignment models. CL, 2003.","title":"A systematic comparison of various statistical alignment models","authors":"Ney ; Och, Josef Franz, Hermann Och, Ney","year":"2003","venue":"CL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Och et al., 2004] Franz Josef Och, Daniel Gildea, et al. A smorgasbord of features for statistical machine translation. In NAACL, 2004.","title":"A smorgasbord of features for statistical machine translation","authors":"Och","year":"2004","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sennrich et al., 2016] Rico Sennrich, Barry Haddow, et al. Neural machine translation of rare words with subword units. In NAACL, 2016.","title":"Neural machine translation of rare words with subword units","authors":"Sennrich","year":"2016","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shaw et al., 2018] Peter Shaw, Jakob Uszkoreit, et al. Self- attention with relative position representations. In NAACL, 2018.","title":"Selfattention with relative position representations","authors":"Peter Shaw, Jakob Uszkoreit","year":"2018","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Song et al., 2020] Kai Song, Kun Wang, et al. Alignment- enhanced transformer for constraining NMT with pre- specified translations. In AAAI, 2020.","title":"Alignmentenhanced transformer for constraining NMT with prespecified translations","authors":"Kai Song, Kun Wang","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Stahlberg et al., 2016] Felix Stahlberg, Eva Hasler, et al. Syntactically guided neural machine translation. In ACL, 2016.","title":"Syntactically guided neural machine translation","authors":"Felix Stahlberg, Eva Hasler","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, et al. Sequence to sequence learning with neural networks. In NIPS, 2014.","title":"Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tillmann, 2004] Christoph Tillmann. A unigram orientation model for statistical machine translation. In NAACL, 2004.","title":"A unigram orientation model for statistical machine translation","authors":"Christoph Tillmann","year":"2004","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, et al. Attention is all you need. In NIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Vilar et al., 2006] David Vilar, Maja Popović, et al. Aer: Do we need to \"improve\" our alignments? In IWSLT, 2006.","title":"Aer: Do we need to \"improve\" our alignments?","authors":"David Vilar, Maja Popović","year":"2006","venue":"IWSLT","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2017] Xing Wang, Zhengdong Lu, et al. Neural machine translation advised by statistical machine transla- tion. In AAAI, 2017.","title":"Neural machine translation advised by statistical machine translation","authors":"Xing Wang, Zhengdong Lu","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2018] Xinyi Wang, Hieu Pham, et al. SwitchOut: an efficient data augmentation algorithm for neural machine translation. In EMNLP, 2018.","title":"SwitchOut: an efficient data augmentation algorithm for neural machine translation","authors":"Wang","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2017] Jinchao Zhang, Mingxuan Wang, et al. Incorporating word reordering knowledge into attention- based neural machine translation. In ACL, 2017.","title":"Incorporating word reordering knowledge into attentionbased neural machine translation","authors":"Zhang","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2021] Jiacheng Zhang, Huanbo Luan, et al. Neural machine translation with explicit phrase alignment. TASLP, 2021.","title":"Neural machine translation with explicit phrase alignment","authors":"Zhang","year":"2021","venue":"TASLP","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhao et al., 2018] Yang Zhao, Yining Wang, et al. Phrase table as recommendation memory for neural machine translation. In IJCAI, 2018.","title":"Phrase table as recommendation memory for neural machine translation","authors":"Yang Zhao, Yining Wang","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/641"},{"position":35,"raw":"[Zheng et al., 2019] Zaixiang Zheng, Shujian Huang, Zhaopeng Tu, Xin-Yu Dai, and Jiajun Chen. Dynamic past and future for neural machine translation. In EMNLP, 2019.","title":"Dynamic past and future for neural machine translation","authors":"Zheng","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhou et al., 2017] Long Zhou, Wenpeng Hu, et al. Neu- ral system combination for machine translation. In ACL, 2017.","title":"Neural system combination for machine translation","authors":"Long Zhou, Wenpeng Hu","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}