{"type":"ScholarlyArticle","title":"High-resource Language-specific Training for Multilingual Neural Machine Translation","authors":[{"name":"Jian Yang","slug":"jian-yang","given":"Jian","family":"Yang","affiliation":"State Key Lab of Software Development Environment , Beihang University"},{"name":"Yuwei Yin","slug":"yuwei-yin","given":"Yuwei","family":"Yin","affiliation":"Microsoft Research"},{"name":"Shuming Ma","slug":"shuming-ma","given":"Shuming","family":"Ma","affiliation":"Microsoft Research"},{"name":"Dongdong Zhang","slug":"dongdong-zhang","given":"Dongdong","family":"Zhang","affiliation":"Microsoft Research"},{"name":"Zhoujun Li","slug":"zhoujun-li","given":"Zhoujun","family":"Li","affiliation":"State Key Lab of Software Development Environment , Beihang University"},{"name":"Furu Wei","slug":"furu-wei","given":"Furu","family":"Wei","affiliation":"Microsoft Research"}],"abstract":"Multilingual neural machine translation (MNMT) trained in multiple language pairs has attracted considerable attention due to fewer model parameters and lower training costs by sharing knowledge among multiple languages. Nonetheless, multilingual training is plagued by language interference degeneration in shared parameters because of the negative interference among different translation directions, especially on high-resource languages. In this paper, we propose the multilingual translation model with the high-resource language-specific training (HLT-MT) to alleviate the negative interference, which adopts the two-stage training with the language-specific selection mechanism. Specifically, we first train the multilingual model only with the high-resource pairs and select the language-specific modules at the top of the decoder to enhance the translation quality of high-resource directions. Next, the model is further trained on all available corpora to transfer knowledge from high-resource languages (HRLs) to low-resource languages (LRLs). Experimental results show that HLT-MT outperforms various strong baselines on WMT-10 and OPUS-100 benchmarks. Furthermore, the analytic experiments validate the effectiveness of our method in mitigating the negative interference in multilingual training.","doi":"10.24963/ijcai.2022/619","url":"https://ijcai.org/proceedings/2022/619","pdf":"https://ijcai.org/proceedings/2022/0619.pdf","markdown":"https://ijcai.org/proceedings/2022/619.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/619","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":619,"pages":{"first":4461,"last":4467},"keywords":["Natural Language Processing: Machine Translation and Multilinguality","Natural Language Processing: Language Generation"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986463","length_sec":79,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984606","length_sec":300,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Aharoni et al., 2019] Roee Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine trans- lation. In NAACL 2019, pages 3874-3884, 2019.","title":"Massively multilingual neural machine translation","authors":"Roee Aharoni, Melvin Johnson, Orhan Firat","year":"2019","venue":"NAACL 2019","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Conneau et al., 2020] Alexis Conneau, Kartikay Khandel- wal, Naman Goyal, Vishrav Chaudhary, Guillaume Wen- zek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross- lingual representation learning at scale. In ACL 2020, pages 8440-8451, 2020.","title":"Unsupervised crosslingual representation learning at scale","authors":"Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov","year":"2020","venue":"ACL 2020","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Escolano et al., 2021] Carlos Escolano, Marta R. Costa- jussà, José A. R. Fonollosa, and Mikel Artetxe. Multilin- gual machine translation: Closing the gap between shared and language-specific encoder-decoders. In EACL 2021, pages 944-948, 2021.","title":"Multilingual machine translation: Closing the gap between shared and language-specific encoder-decoders","authors":"Carlos Escolano, Marta R Costajussà, A R José, Mikel Fonollosa, Artetxe","year":"2021","venue":"EACL 2021","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fan et al., 2020] Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. Beyond english-centric multi- lingual machine translation. CoRR, abs/2010.11125, 2020.","title":"Beyond english-centric multilingual machine translation","authors":"Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, Armand Joulin","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Firat et al., 2016] Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. Multi-way, multilingual neural machine translation with a shared attention mechanism. In NAACL 2016, pages 866-875, 2016.","title":"Multi-way, multilingual neural machine translation with a shared attention mechanism","authors":"Orhan Firat, Kyunghyun Cho, Yoshua Bengio","year":"2016","venue":"NAACL 2016","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gong et al., 2021] Hongyu Gong, Xian Li, and Dmitriy Genzel. Adaptive sparse transformer for multilingual translation. CoRR, abs/2104.07358, 2021.","title":"Adaptive sparse transformer for multilingual translation","authors":"Hongyu Gong, Xian Li, Dmitriy Genzel","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Johnson et al., 2017] Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation. TACL 2017, 5:339-351, 2017.","title":"Google's multilingual neural machine translation system: Enabling zero-shot translation","authors":"Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, Jeffrey Dean","year":"2017","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kong et al., 2021] Xiang Kong, Adithya Renduchintala, James Cross, Yuqing Tang, Jiatao Gu, and Xian Li. Multi- lingual neural machine translation with deep encoder and multiple shallow decoders. In EACL 2021, pages 1613- 1624, 2021.","title":"Multilingual neural machine translation with deep encoder and multiple shallow decoders","authors":"Xiang Kong, Adithya Renduchintala, James Cross, Yuqing Tang, Jiatao Gu, Xian Li","year":"2021","venue":"EACL 2021","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lin et al., 2020] Zehui Lin, Xiao Pan, Mingxuan Wang, Xipeng Qiu, Jiangtao Feng, Hao Zhou, and Lei Li. Pre- training multilingual neural machine translation by lever- aging alignment information. In EMNLP 2020, pages 2649-2663, 2020.","title":"Pretraining multilingual neural machine translation by leveraging alignment information","authors":"Lin","year":"2020","venue":"EMNLP 2020","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2020] Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. TACL, 8:726-742, 2020.","title":"Multilingual denoising pre-training for neural machine translation","authors":"Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer","year":"2020","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ma et al., 2020] Shuming Ma, Jian Yang, Haoyang Huang, Zewen Chi, Li Dong, Dongdong Zhang, Hany Has- san Awadalla, Alexandre Muzio, Akiko Eriguchi, Sak- sham Singhal, Xia Song, Arul Menezes, and Furu Wei. XLM-T: scaling up multilingual machine translation with pretrained cross-lingual transformer encoders. CoRR, abs/2012.15547, 2020.","title":"XLM-T: scaling up multilingual machine translation with pretrained cross-lingual transformer encoders","authors":"Shuming Ma, Jian Yang, Haoyang Huang, Zewen Chi, Li Dong, Dongdong Zhang, Hany Hassan Awadalla, Alexandre Muzio, Akiko Eriguchi, Saksham Singhal, Xia Song, Arul Menezes, Furu Wei","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Maaten and Hinton, 2008] Laurens van der Maaten and Ge- offrey Hinton. Visualizing data using t-sne. JMLR, 9(Nov):2579-2605, 2008.","title":"Laurens van der Maaten and Geoffrey Hinton","authors":"Hinton Maaten","year":"2008","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pan et al., 2021] Xiao Pan, Mingxuan Wang, Liwei Wu, and Lei Li. Contrastive learning for many-to-many multilin- gual neural machine translation. In ACL 2021, pages 244- 258, 2021.","title":"Contrastive learning for many-to-many multilingual neural machine translation","authors":"Xiao Pan, Mingxuan Wang, Liwei Wu, Lei Li","year":"2021","venue":"ACL 2021","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Philip et al., 2020] Jerin Philip, Alexandre Berard, Matthias Gallé, and Laurent Besacier. Monolingual adapters for zero-shot neural machine translation. In EMNLP 2020, pages 4465-4470, 2020.","title":"Monolingual adapters for zero-shot neural machine translation","authors":"Jerin Philip, Alexandre Berard, Matthias Gallé, Laurent Besacier","year":"2020","venue":"EMNLP 2020","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Post, 2018] Matt Post. A call for clarity in reporting BLEU scores. In WMT 2018, pages 186-191, 2018.","title":"A call for clarity in reporting BLEU scores","authors":"Matt Post","year":"2018","venue":"WMT 2018","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tan et al., 2019] Xu Tan, Jiale Chen, Di He, Yingce Xia, Tao Qin, and Tie-Yan Liu. Multilingual neural machine translation with language clustering. In EMNLP 2019, pages 963-973, 2019.","title":"Multilingual neural machine translation with language clustering","authors":"Jiale Xu Tan, Di Chen, Yingce He, Tao Xia, Tie-Yan Qin, Liu","year":"2019","venue":"EMNLP 2019","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tang et al., 2021] Yuqing Tang, Chau Tran, Xian Li, Peng- Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. Multilingual translation from denoising pre-training. In ACL 2021, pages 3450-3466, 2021.","title":"Multilingual translation from denoising pre-training","authors":"Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, Angela Fan","year":"2021","venue":"ACL 2021","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS 2017, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS 2017","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vázquez et al., 2019] Raúl Vázquez, Alessandro Raganato, Jörg Tiedemann, and Mathias Creutz. Multilingual NMT with a language-independent attention bridge. In ACL 2019, pages 33-39, 2019.","title":"Multilingual NMT with a language-independent attention bridge","authors":"Raúl Vázquez, Alessandro Raganato, Jörg Tiedemann, Mathias Creutz","year":"2019","venue":"ACL 2019","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2018] Yining Wang, Jiajun Zhang, Feifei Zhai, Jingfang Xu, and Chengqing Zong. Three strategies to improve one-to-many multilingual translation. In EMNLP 2018, pages 2955-2960, 2018.","title":"Three strategies to improve one-to-many multilingual translation","authors":"Yining Wang, Jiajun Zhang, Feifei Zhai, Jingfang Xu, Chengqing Zong","year":"2018","venue":"EMNLP 2018","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2019] Yining Wang, Long Zhou, Jiajun Zhang, Feifei Zhai, Jingfang Xu, and Chengqing Zong. A compact and language-sensitive multilingual translation method. In ACL 2019, pages 1213-1223, 2019.","title":"A compact and language-sensitive multilingual translation method","authors":"Yining Wang, Long Zhou, Jiajun Zhang, Feifei Zhai, Jingfang Xu, Chengqing Zong","year":"2019","venue":"ACL 2019","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2020a] Yiren Wang, ChengXiang Zhai, and Hany Hassan. Multi-task learning for multilingual neural machine translation. In EMNLP 2020, pages 1022-1034, 2020.","title":"Multi-task learning for multilingual neural machine translation","authors":"Wang","year":"2020","venue":"EMNLP 2020","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2020b] Zirui Wang, Zachary C. Lipton, and Yulia Tsvetkov. On negative interference in multilin- gual models: Findings and A meta-learning treatment. In EMNLP 2020, pages 4438-4450, 2020.","title":"On negative interference in multilingual models: Findings and A meta-learning treatment","authors":"Wang","year":"2020","venue":"EMNLP 2020","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yu et al., 2020] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In NeurIPS 2020, 2020.","title":"Gradient surgery for multi-task learning","authors":"Yu","year":"2020","venue":"NeurIPS 2020","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2020] Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. Improving massively multilin- gual neural machine translation and zero-shot translation. In ACL 2020, pages 1628-1639, 2020.","title":"Improving massively multilingual neural machine translation and zero-shot translation","authors":"Biao Zhang, Philip Williams, Ivan Titov, Rico Sennrich","year":"2020","venue":"ACL 2020","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}