{"type":"ScholarlyArticle","title":"CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP","authors":[{"name":"Libo Qin","slug":"libo-qin","given":"Libo","family":"Qin","affiliation":"Research Center for Social Computing and Information Retrieval Harbin Institute of Technology , China"},{"name":"Minheng Ni","slug":"minheng-ni","given":"Minheng","family":"Ni","affiliation":"Research Center for Social Computing and Information Retrieval Harbin Institute of Technology , China"},{"name":"Yue Zhang","slug":"yue-zhang","given":"Yue","family":"Zhang","affiliation":"School of Engineering , Westlake University , China; Institute of Advanced Technology , Westlake Institute for Advanced Study"},{"name":"Wanxiang Che","slug":"wanxiang-che","given":"Wanxiang","family":"Che","affiliation":"Research Center for Social Computing and Information Retrieval Harbin Institute of Technology , China"}],"abstract":"Multi-lingual contextualized embeddings, such as multilingual-BERT (mBERT), have\r\n\tshown success in a variety of zero-shot cross-lingual tasks.\r\n\tHowever, these models are limited by having inconsistent contextualized representations of subwords across different languages.\r\n\tExisting work addresses this issue by bilingual projection and fine-tuning technique.\r\n\tWe propose a data augmentation framework to generate multi-lingual code-switching data to fine-tune mBERT, which encourages model to align representations from source and multiple target languages once by mixing their context information.\r\n\tCompared with the existing work, our method does not rely on bilingual sentences for training, and requires only one training process for multiple target languages.\r\n \tExperimental results on five tasks with 19 languages show that our method leads to significantly improved performances for all the tasks compared with mBERT.","doi":"10.24963/ijcai.2020/533","url":"https://ijcai.org/proceedings/2020/533","pdf":"https://ijcai.org/proceedings/2020/0533.pdf","markdown":"https://ijcai.org/proceedings/2020/533.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/533","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":533,"pages":{"first":3853,"last":3860},"keywords":["Natural Language Processing: Natural Language Processing"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24464/file_24464.m3u8","length_sec":268,"provider":"hls"}],"references":[{"position":1,"raw":"[Artetxe and Schwenk, 2018] Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. arXiv preprint arXiv:1812.10464, 2018.","title":"Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond","authors":"Artetxe, Schwenk, Mikel Artetxe, Holger Schwenk","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Artetxe et al., 2019] Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferabil- ity of monolingual representations. arXiv preprint arXiv:1910.11856, 2019.","title":"On the cross-lingual transferability of monolingual representations","authors":"Mikel Artetxe, Sebastian Ruder, Dani Yogatama","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Barnes et al., 2018] Jeremy Barnes, Roman Klinger, and Sabine Schulte im Walde. Bilingual sentiment embed- dings: Joint projection of sentiment across languages. In Proc. of ACL, pages 2483-2493, Melbourne, Australia, July 2018. Association for Computational Linguistics.","title":"Bilingual sentiment embeddings: Joint projection of sentiment across languages","authors":"Barnes","year":"2018","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen and Cardie, 2018] Xilun Chen and Claire Cardie. Un- supervised multilingual word embeddings. arXiv preprint arXiv:1808.08933, 2018.","title":"Xilun Chen and Claire Cardie. Unsupervised multilingual word embeddings","authors":"Cardie Chen","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2018] Wenhu Chen, Jianshu Chen, Yu Su, Xin Wang, Dong Yu, Xifeng Yan, and William Yang Wang. XL-NBT: A cross-lingual neural belief tracking frame- work. In Proc. of EMNLP, October-November 2018.","title":"XL-NBT: A cross-lingual neural belief tracking framework","authors":"Chen","year":"2018","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Conneau and Lample, 2019] Alexis Conneau and Guil- laume Lample. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems, pages 7057-7067, 2019.","title":"Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining","authors":"Lample Conneau","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Conneau et al., 2018] Alexis Conneau, Ruty Rinott, Guil- laume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross- lingual sentence representations. In Proc. of EMNLP, 2018.","title":"XNLI: Evaluating crosslingual sentence representations","authors":"Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, Veselin Stoyanov","year":"2018","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understand- ing. In Proc. of NAACL, 2019.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proc. of NAACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guo et al., 2016] Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. A representation learning framework for multi-source transfer parsing. In Proc. of AAAI, 2016.","title":"A representation learning framework for multi-source transfer parsing","authors":"Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, Ting Liu","year":"2016","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2019] Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. Unicoder: A universal language encoder by pre- training with multiple cross-lingual tasks. In Proc. of EMNLP, November 2019.","title":"Unicoder: A universal language encoder by pretraining with multiple cross-lingual tasks","authors":"Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, Ming Zhou","year":"2019","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Klementiev et al., 2012] Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. Inducing crosslingual distributed representations of words. In Proc. of COLING, 2012.","title":"Inducing crosslingual distributed representations of words","authors":"Alexandre Klementiev, Ivan Titov, Binod Bhattarai","year":"2012","venue":"Proc. of COLING","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kočiský et al., 2014] Tomáš Kočiský, Karl Moritz Her- mann, and Phil Blunsom. Learning bilingual word repre- sentations by marginalizing alignments. In Proc. of ACL, June 2014.","title":"Learning bilingual word representations by marginalizing alignments","authors":"Kočiský","year":"2014","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lample et al., 2018] Guillaume Lample, Alexis Conneau, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In Inter- national Conference on Learning Representations, 2018.","title":"Word translation without parallel data","authors":"Guillaume Lample, Alexis Conneau, Marc'aurelio Ranzato, Ludovic Denoyer, Hervé Jégou","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2019a] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.","title":"Roberta: A robustly optimized bert pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2019b] Zihan Liu, Genta Indra Winata, Zhao- jiang Lin, Peng Xu, and Pascale Fung. Attention-informed mixed-language training for zero-shot cross-lingual task- oriented dialogue systems, 2019.","title":"Attention-informed mixed-language training for zero-shot cross-lingual taskoriented dialogue systems","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mrkšić et al., 2017] Nikola Mrkšić, Ivan Vulić, Diarmuid Ó Séaghdha, Ira Leviant, Roi Reichart, Milica Gašić, Anna Korhonen, and Steve Young. Semantic specialization of distributional word vector spaces using monolingual and cross-lingual constraints. Transactions of the Association for Computational Linguistics, 5:309-324, 2017.","title":"Semantic specialization of distributional word vector spaces using monolingual and cross-lingual constraints","authors":"Nikola Mrkšić, Ivan Vulić, Ó Diarmuid, Ira Séaghdha, Roi Leviant, Milica Reichart, Anna Gašić, Steve Korhonen, Young","year":"2017","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ruder et al., 2017] Sebastian Ruder, Ivan Vulić, and Anders Søgaard. A survey of cross-lingual word embedding mod- els. arXiv preprint arXiv:1706.04902, 2017.","title":"A survey of cross-lingual word embedding models","authors":"Sebastian Ruder, Ivan Vulić, Anders Søgaard","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schuster et al., 2019a] Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. Cross-lingual transfer learning for multilingual task oriented dialog. In Proc. of NAACL, June 2019.","title":"Cross-lingual transfer learning for multilingual task oriented dialog","authors":"Schuster","year":"2019","venue":"Proc. of NAACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schuster et al., 2019b] Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. Cross-lingual alignment of contextual word embeddings, with applications to zero- shot dependency parsing. In Proc. of NAACL, June 2019.","title":"Cross-lingual alignment of contextual word embeddings, with applications to zeroshot dependency parsing","authors":"Schuster","year":"2019","venue":"Proc. of NAACL","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schwenk and Li, 2018] Holger Schwenk and Xian Li. A corpus for multilingual document classification in eight languages. In Proceedings of the 11th Language Re- sources and Evaluation Conference, May 2018.","title":"A corpus for multilingual document classification in eight languages","authors":"Li ; Schwenk, Holger Schwenk, Xian Li","year":"2018","venue":"Proceedings of the 11th Language Resources and Evaluation Conference","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2019] Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu. Cross-lingual BERT trans- formation for zero-shot dependency parsing. In Proc. of EMNLP, November 2019.","title":"Cross-lingual BERT transformation for zero-shot dependency parsing","authors":"Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, Ting Liu","year":"2019","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu and Dredze, 2019] Shijie Wu and Mark Dredze. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proc. of EMNLP, 2019.","title":"becas: The surprising cross-lingual effectiveness of BERT","authors":"Shijie Wu, Mark Dredze, Beto, Bentz","year":"2019","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xing et al., 2015] Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. Normalized word embedding and orthogo- nal transform for bilingual word translation. In Proc. of NAACL, 2015.","title":"Normalized word embedding and orthogonal transform for bilingual word translation","authors":"Chao Xing, Dong Wang, Chao Liu, Yiye Lin","year":"2015","venue":"Proc. of NAACL","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yin et al., 2019] Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dialog state tracking with reinforced data augmentation. arXiv preprint arXiv:1908.07795, 2019.","title":"Dialog state tracking with reinforced data augmentation","authors":"Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Qun Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu et al., 2018] Katherine Yu, Haoran Li, and Barlas Oguz. Multilingual seq2seq training with similarity loss for cross-lingual document classification. In Proceedings of The Third Workshop on Representation Learning for NLP, July 2018.","title":"Multilingual seq2seq training with similarity loss for cross-lingual document classification","authors":"Katherine Yu, Haoran Li, Barlas Oguz","year":"2018","venue":"Proceedings of The Third Workshop on Representation Learning for NLP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2019] Meishan Zhang, Yue Zhang, and Guo- hong Fu. Cross-lingual dependency parsing using code- mixed TreeBank. In Proc. of EMNLP, pages 997-1006, Hong Kong, China, November 2019. Association for Computational Linguistics.","title":"Cross-lingual dependency parsing using codemixed TreeBank","authors":"Zhang","year":"2019","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}