{"type":"ScholarlyArticle","title":"UM4: Unified Multilingual Multiple Teacher-Student Model for Zero-Resource Neural Machine Translation","authors":[{"name":"Jian Yang","slug":"jian-yang","given":"Jian","family":"Yang","affiliation":"State Key Lab of Software Development Environment , Beihang University"},{"name":"Yuwei Yin","slug":"yuwei-yin","given":"Yuwei","family":"Yin","affiliation":"Microsoft Research"},{"name":"Shuming Ma","slug":"shuming-ma","given":"Shuming","family":"Ma","affiliation":"Microsoft Research"},{"name":"Dongdong Zhang","slug":"dongdong-zhang","given":"Dongdong","family":"Zhang","affiliation":"Microsoft Research"},{"name":"Shuangzhi Wu","slug":"shuangzhi-wu","given":"Shuangzhi","family":"Wu","affiliation":"Tencent Cloud Xiaowei"},{"name":"Hongcheng Guo","slug":"hongcheng-guo","given":"Hongcheng","family":"Guo","affiliation":"Microsoft Research"},{"name":"Zhoujun Li","slug":"zhoujun-li","given":"Zhoujun","family":"Li","affiliation":"State Key Lab of Software Development Environment , Beihang University"},{"name":"Furu Wei","slug":"furu-wei","given":"Furu","family":"Wei","affiliation":"Microsoft Research"}],"abstract":"Most translation tasks among languages belong to the zero-resource translation problem where parallel corpora are unavailable. Multilingual neural machine translation (MNMT) enables one-pass translation using shared semantic space for all languages compared to the two-pass pivot translation but often underperforms the pivot-based method. In this paper, we propose a novel method, named as Unified Multilingual Multiple teacher-student Model for NMT (UM4). Our method unifies source-teacher, target-teacher, and pivot-teacher models to guide the student model for the zero-resource translation. The source teacher and target teacher force the student to learn the direct source-target translation by the distilled knowledge on both source and target sides. The monolingual corpus is further leveraged by the pivot-teacher model to enhance the student model. Experimental results demonstrate that our model of 72 directions significantly outperforms previous methods on the WMT benchmark.","doi":"10.24963/ijcai.2022/618","url":"https://ijcai.org/proceedings/2022/618","pdf":"https://ijcai.org/proceedings/2022/0618.pdf","markdown":"https://ijcai.org/proceedings/2022/618.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/618","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":618,"pages":{"first":4454,"last":4460},"keywords":["Natural Language Processing: Machine Translation and Multilinguality","Natural Language Processing: Language Generation"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986462","length_sec":80,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984605","length_sec":300,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bapna and Firat, 2019] Ankur Bapna and Orhan Firat. Sim- ple, scalable adaptation for neural machine translation. In EMNLP 2019, pages 1538-1548, 2019.","title":"Simple, scalable adaptation for neural machine translation","authors":"Ankur Bapna, Orhan Firat","year":"2019","venue":"EMNLP 2019","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bertoldi et al., 2008] Nicola Bertoldi, Madalina Barbaiani, Marcello Federico, and Roldano Cattoni. Phrase-based statistical machine translation with pivot languages. In IWSLT 2008, pages 143-149, 2008.","title":"Phrase-based statistical machine translation with pivot languages","authors":"Nicola Bertoldi, Madalina Barbaiani, Marcello Federico, Roldano Cattoni","year":"2008","venue":"IWSLT 2008","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2017] Yun Chen, Yang Liu, Yong Cheng, and Victor O. K. Li. A teacher-student framework for zero- resource neural machine translation. In ACL 2017, pages 1925-1935, 2017.","title":"A teacher-student framework for zeroresource neural machine translation","authors":"Yun Chen, Yang Liu, Yong Cheng, O K Victor, Li","year":"2017","venue":"ACL 2017","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cheng et al., 2017] Yong Cheng, Qian Yang, Yang Liu, Maosong Sun, and Wei Xu. Joint training for pivot-based neural machine translation. In IJCAI 2017, pages 3974- 3980, 2017.","title":"Joint training for pivot-based neural machine translation","authors":"Yong Cheng, Qian Yang, Yang Liu, Maosong Sun, Wei Xu","year":"2017","venue":"IJCAI 2017","doi":null,"url":null,"internal_url":"/proceedings/2017/555"},{"position":5,"raw":"[Currey and Heafield, 2019] Anna Currey and Kenneth Heafield. Zero-resource neural machine translation with monolingual pivot data. In EMNLP 2019, pages 99-107, 2019.","title":"Zero-resource neural machine translation with monolingual pivot data","authors":"Anna Currey, Kenneth Heafield","year":"2019","venue":"EMNLP 2019","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Firat et al., 2016] Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T. Yarman-Vural, and Kyunghyun Cho. Zero-resource translation with multi-lingual neural ma- chine translation. In EMNLP 2016, pages 268-277, 2016.","title":"Zero-resource translation with multi-lingual neural machine translation","authors":"Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, T Fatos, Kyunghyun Yarman-Vural, Cho","year":"2016","venue":"EMNLP 2016","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Garcia et al., 2020] Xavier Garcia, Pierre Foret, Thibault Sellam, and Ankur P. Parikh. A multilingual view of un- supervised machine translation. In EMNLP 2020, pages 3160-3170, 2020.","title":"A multilingual view of unsupervised machine translation","authors":"Xavier Garcia, Pierre Foret, Thibault Sellam, Ankur P Parikh","year":"2020","venue":"EMNLP 2020","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gheini et al., 2021] Mozhdeh Gheini, Xiang Ren, and Jonathan May. Cross-attention is all you need: Adapt- ing pretrained transformers for machine translation. In EMNLP 2021, pages 1754-1765, 2021.","title":"Jonathan May. Cross-attention is all you need: Adapting pretrained transformers for machine translation","authors":"Gheini","year":"2021","venue":"EMNLP 2021","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Johnson et al., 2017] Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. Google's multilingual neural machine translation system: Enabling zero-shot translation. TACL 2017, 5, 2017.","title":"Google's multilingual neural machine translation system: Enabling zero-shot translation","authors":"Melvin Johnson, Mike Schuster, V Quoc, Maxim Le, Yonghui Krikun, Zhifeng Wu, Nikhil Chen, Fernanda Thorat, Martin Viégas, Greg Wattenberg, Corrado","year":"2017","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kim and Rush, 2016] Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In EMNLP 2016, pages 1317-1327, 2016.","title":"Sequence-level knowledge distillation","authors":"Yoon Kim, Alexander M Rush","year":"2016","venue":"EMNLP 2016","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim et al., 2019a] Yunsu Kim, Petre Petrov, Pavel Petrushkov, Shahram Khadivi, and Hermann Ney. Pivot-based transfer learning for neural machine transla- tion between non-english languages. In EMNLP 2019, pages 866-876, 2019.","title":"Pivot-based transfer learning for neural machine translation between non-english languages","authors":"Kim","year":"2019","venue":"EMNLP 2019","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim et al., 2019b] Yunsu Kim, Petre Petrov, Pavel Petrushkov, Shahram Khadivi, and Hermann Ney. Pivot- based transfer learning for neural machine translation between non-english languages. In EMNLP 2019, pages 866-876, 2019.","title":"Pivotbased transfer learning for neural machine translation between non-english languages","authors":"Kim","year":"2019","venue":"EMNLP 2019","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Ba, 2014] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Koehn et al., 2019] Philipp Koehn, Francisco Guzmán, Vishrav Chaudhary, and Juan Pino. Findings of the WMT 2019 shared task on parallel corpus filtering for low- resource conditions. In WMT 2019, pages 54-72, 2019.","title":"Findings of the WMT 2019 shared task on parallel corpus filtering for lowresource conditions","authors":"Koehn","year":"2019","venue":"WMT 2019","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lakew et al., 2019] Surafel Melaku Lakew, Marcello Fed- erico, Matteo Negri, and Marco Turchi. Multilingual neural machine translation for zero-resource languages. CoRR, abs/1909.07342, 2019.","title":"Multilingual neural machine translation for zero-resource languages","authors":"Lakew","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Philip et al., 2020] Jerin Philip, Alexandre Berard, Matthias Gallé, and Laurent Besacier. Monolingual adapters for zero-shot neural machine translation. In EMNLP 2020, pages 4465-4470, 2020.","title":"Monolingual adapters for zero-shot neural machine translation","authors":"Jerin Philip, Alexandre Berard, Matthias Gallé, Laurent Besacier","year":"2020","venue":"EMNLP 2020","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tan et al., 2019] Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Tie-Yan Liu. Multilingual neural machine trans- lation with knowledge distillation. In ICLR 2019, 2019.","title":"Multilingual neural machine translation with knowledge distillation","authors":"Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, Tie-Yan Liu","year":"2019","venue":"ICLR 2019","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS 2017, pages 6000-6010, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS 2017","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2020] Yiren Wang, ChengXiang Zhai, and Hany Hassan. Multi-task learning for multilingual neural machine translation. In EMNLP 2020, pages 1022-1034, 2020.","title":"Multi-task learning for multilingual neural machine translation","authors":"Yiren Wang, Chengxiang Zhai, Hany Hassan","year":"2020","venue":"EMNLP 2020","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wu and Wang, 2009] Hua Wu and Haifeng Wang. Revisit- ing pivot language approach for machine translation. In ACL 2009, pages 154-162, 2009.","title":"Revisiting pivot language approach for machine translation","authors":"Hua Wu, Haifeng Wang","year":"2009","venue":"ACL 2009","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2021] Yilin Yang, Akiko Eriguchi, Alexandre Muzio, Prasad Tadepalli, Stefan Lee, and Hany Hassan. Improving multilingual translation by representation and gradient regularization. In EMNLP 2021, pages 7266- 7279, 2021.","title":"Improving multilingual translation by representation and gradient regularization","authors":"Yilin Yang, Akiko Eriguchi, Alexandre Muzio, Prasad Tadepalli, Stefan Lee, Hany Hassan","year":"2021","venue":"EMNLP 2021","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zahabi et al., 2013] Samira Tofighi Zahabi, Somayeh Bakhshaei, and Shahram Khadivi. Using context vectors in improving a machine translation system with bridge language. In ACL 2013, pages 318-322, 2013.","title":"Using context vectors in improving a machine translation system with bridge language","authors":"Samira Tofighi Zahabi, Somayeh Bakhshaei, Shahram Khadivi","year":"2013","venue":"ACL 2013","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2020] Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. Improving massively multilin- gual neural machine translation and zero-shot translation. In ACL 2020, pages 1628-1639, 2020.","title":"Improving massively multilingual neural machine translation and zero-shot translation","authors":"Biao Zhang, Philip Williams, Ivan Titov, Rico Sennrich","year":"2020","venue":"ACL 2020","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zheng et al., 2017] Hao Zheng, Yong Cheng, and Yang Liu. Maximum expected likelihood estimation for zero- resource neural machine translation. In IJCAI 2017, pages 4251-4257, 2017.","title":"Maximum expected likelihood estimation for zeroresource neural machine translation","authors":"Yong Hao Zheng, Yang Cheng, Liu","year":"2017","venue":"IJCAI 2017","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou et al., 2021] Chunting Zhou, Daniel Levy, Xian Li, Marjan Ghazvininejad, and Graham Neubig. Distribution- ally robust multilingual machine translation. In EMNLP 2021, pages 5664-5674, 2021.","title":"Distributionally robust multilingual machine translation","authors":"Zhou","year":"2021","venue":"EMNLP 2021","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhu et al., 2013] Xiaoning Zhu, Zhongjun He, Hua Wu, Haifeng Wang, Conghui Zhu, and Tiejun Zhao. Improving pivot-based statistical machine translation using random walk. In EMNLP 2013, pages 524-534, 2013.","title":"Improving pivot-based statistical machine translation using random walk","authors":"Xiaoning Zhu, Zhongjun He, Hua Wu, Haifeng Wang, Conghui Zhu, Tiejun Zhao","year":"2013","venue":"EMNLP 2013","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}