{"type":"ScholarlyArticle","title":"Joint Training for Pivot-based Neural Machine Translation","authors":[{"name":"Yong Cheng","slug":"yong-cheng","given":"Yong","family":"Cheng","affiliation":"# Institute for Interdisciplinary Information Sciences , Tsinghua University , Beijing , China"},{"name":"Qian Yang","slug":"qian-yang","given":"Qian","family":"Yang","affiliation":""},{"name":"Yang Liu","slug":"yang-liu","given":"Yang","family":"Liu","affiliation":"† State Key Laboratory of Intelligent Technology and Systems Tsinghua National Laboratory for Information Science and Technology Department of Computer Science and Technology , Tsinghua University , Beijing , China"},{"name":"Maosong Sun","slug":"maosong-sun","given":"Maosong","family":"Sun","affiliation":"† State Key Laboratory of Intelligent Technology and Systems Tsinghua National Laboratory for Information Science and Technology Department of Computer Science and Technology , Tsinghua University , Beijing , China"},{"name":"Wei Xu","slug":"wei-xu","given":"Wei","family":"Xu","affiliation":"# Institute for Interdisciplinary Information Sciences , Tsinghua University , Beijing , China"}],"abstract":"While recent neural machine translation approaches have delivered state-of-the-art performance for resource-rich language pairs, they suffer from the data scarcity problem for resource-scarce language pairs. Although this problem can be alleviated by exploiting a pivot language to bridge the source and target languages, the source-to-pivot and pivot-to-target translation models are usually independently trained. In this work, we introduce a joint training algorithm for pivot-based neural machine translation. We propose three methods to connect the two models and enable them to interact with each other during training. Experiments on Europarl and WMT corpora show that joint training of source-to-pivot and pivot-to-target models leads to significant improvements over independent training across various languages.","doi":"10.24963/ijcai.2017/555","url":"https://ijcai.org/proceedings/2017/555","pdf":"https://ijcai.org/proceedings/2017/0555.pdf","markdown":"https://ijcai.org/proceedings/2017/555.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/555","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Natural Language Processing","track":"Main track","number":555,"pages":{"first":3974,"last":3980},"keywords":["Natural Language Processing: Machine Translation","Natural Language Processing: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"Proceedings of ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bertoldi et al., 2008] Nicola Bertoldi, Madalina Barbaiani, Marcello Federico, and Roldano Cattoni. Phrase-based statistical machine translation with pivot languages. In IWSLT, 2008.","title":"Phrase-based statistical machine translation with pivot languages","authors":"Nicola Bertoldi, Madalina Barbaiani, Marcello Federico, Roldano Cattoni","year":"2008","venue":"IWSLT","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cheng et al., 2016] Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Semi- supervised learning for neural machine translation. In Pro- ceedings of ACL, 2016.","title":"Semisupervised learning for neural machine translation","authors":"Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu","year":"2016","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cohn and Lapata, 2007] Trevor Cohn and Mirella Lapata. Machine translation by triangulation: Making effective use of multi-parallel corpora. In Proceedings of ACL, 2007.","title":"Machine translation by triangulation: Making effective use of multi-parallel corpora","authors":"Trevor Cohn, Mirella Lapata","year":"2007","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[El Kholy et al., 2013] Ahmed El Kholy, Nizar Habash, Gregor Leusch, Evgeny Matusov, and Hassan Sawaf. Language independent connectivity strength features for phrase pivot statistical machine translation. In Proceed- ings of ACL, 2013.","title":"Language independent connectivity strength features for phrase pivot statistical machine translation","authors":"Ahmed El Kholy, Nizar Habash, Gregor Leusch, Evgeny Matusov, Hassan Sawaf","year":"2013","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Firat et al., 2016] Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T Yarman Vural, and Kyunghyun Cho. Zero-resource translation with multi-lingual neural ma- chine translation. In Proceedings of EMNLP, 2016.","title":"Zero-resource translation with multi-lingual neural machine translation","authors":"Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T Yarman, Kyunghyun Vural, Cho","year":"2016","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gulccehre et al., 2015] Caglar Gulccehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loïc Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. On using monolingual corpora in neural machine translation. arXiv:1503.03535 [cs.CL], 2015.","title":"On using monolingual corpora in neural machine translation","authors":"Caglar Gulccehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loïc Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, Yoshua Bengio","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jean et al., 2015] Sebastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. On using very large target vocabulary for neural machine translation. In Proceedings of ACL, 2015.","title":"On using very large target vocabulary for neural machine translation","authors":"Sebastien Jean, Kyunghyun Cho, Roland Memisevic, Yoshua Bengio","year":"2015","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Johnson et al., 2016] Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. Google's multilingual neural machine translation system: Enabling zero-shot translation. arXiv preprint arXiv:1611.04558, 2016.","title":"Google's multilingual neural machine translation system: Enabling zero-shot translation","authors":"Melvin Johnson, Mike Schuster, V Quoc, Maxim Le, Yonghui Krikun, Zhifeng Wu, Nikhil Chen, Fernanda Thorat, Martin Viégas, Greg Wattenberg, Corrado","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Junczys-Dowmunt et al., 2016] Marcin Junczys-Dowmunt, Tomasz Dwojak, and Hieu Hoang. Is neural machine translation ready for deployment? a case study on 30 trans- lation directions. arXiv:1610.01108v2, 2016.","title":"Is neural machine translation ready for deployment? a case study on 30 translation directions","authors":"Marcin Junczys-Dowmunt, Tomasz Dwojak, Hieu Hoang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Koehn, 2004] Philipp Koehn. Statistical significance tests for machine translation evaluation. In Proceedings of EMNLP, 2004.","title":"Statistical significance tests for machine translation evaluation","authors":"Philipp Koehn, Koehn","year":"2004","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a methof for au- tomatic evaluation of machine translation. In Proceedings of ACL, 2002.","title":"Bleu: a methof for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Sennrich et al., 2016a] Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving nerual machine translation models with monolingual data. In Proceedings of ACL, 2016.","title":"Improving nerual machine translation models with monolingual data","authors":"Sennrich","year":"2016","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sennrich et al., 2016b] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of ACL, 2016.","title":"Neural machine translation of rare words with subword units","authors":"Sennrich","year":"2016","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Shen et al., 2016] Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Minimum risk training for neural machine translation. In Proceed- ings of ACL, 2016.","title":"Minimum risk training for neural machine translation","authors":"Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu","year":"2016","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In Proceedings of NIPS, 2014.","title":"Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"Proceedings of NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Utiyama and Isahara, 2007] Masao Utiyama and Hitoshi Isahara. A comparison of pivot methods for phrase-based statistical machine translation. In HLT-NAACL, 2007.","title":"A comparison of pivot methods for phrase-based statistical machine translation","authors":"Masao Utiyama, Hitoshi Isahara","year":"2007","venue":"HLT-NAACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wu and Wang, 2007] Hua Wu and Haifeng Wang. Pivot language approach for phrase-based statistical machine translation. Machine Translation, 2007.","title":"Pivot language approach for phrase-based statistical machine translation","authors":"Hua Wu, Haifeng Wang","year":"2007","venue":"Machine Translation","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zahabi et al., 2013] Samira Tofighi Zahabi, Somayeh Bakhshaei, and Shahram Khadivi. Using context vectors in improving a machine translation system with bridge language. In Proceedings of ACL, 2013.","title":"Using context vectors in improving a machine translation system with bridge language","authors":"Samira Tofighi Zahabi, Somayeh Bakhshaei, Shahram Khadivi","year":"2013","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhang and Zong, 2016] Jiajun Zhang and Chengqing Zong. Exploiting source-side monolingual data in neural ma- chine translation. In Proceedings of EMNLP, 2016.","title":"Exploiting source-side monolingual data in neural machine translation","authors":"Jiajun Zhang, Chengqing Zong","year":"2016","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zoph and Knight, 2016] Barret Zoph and Kevin Knight. Multi-source neural translation. In Proceedings of NAACL, 2016.","title":"Multi-source neural translation","authors":"Knight Zoph, Barret Zoph, Kevin Knight","year":"2016","venue":"Proceedings of NAACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zoph et al., 2016] Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. Transfer learning for low-resource neu- ral machine translation. In Proceedings of EMNLP, 2016.","title":"Transfer learning for low-resource neural machine translation","authors":"Barret Zoph, Deniz Yuret, Jonathan May, Kevin Knight","year":"2016","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}