{"type":"ScholarlyArticle","title":"Unsupervised Domain Adaptation of a Pretrained Cross-Lingual Language Model","authors":[{"name":"Juntao Li","slug":"juntao-li","given":"Juntao","family":"Li","affiliation":"Center for Data Science , Academy for Advanced Interdisciplinary Studies , Peking University; Department of Computer Science , National University of Singapore"},{"name":"Ruidan He","slug":"ruidan-he","given":"Ruidan","family":"He","affiliation":"DAMO Academy , Alibaba Group"},{"name":"Hai Ye","slug":"hai-ye","given":"Hai","family":"Ye","affiliation":"Department of Computer Science , National University of Singapore"},{"name":"Hwee Tou Ng","slug":"hwee-tou-ng","given":"Hwee Tou","family":"Ng","affiliation":"Department of Computer Science , National University of Singapore"},{"name":"Lidong Bing","slug":"lidong-bing","given":"Lidong","family":"Bing","affiliation":"DAMO Academy , Alibaba Group"},{"name":"Rui Yan","slug":"rui-yan","given":"Rui","family":"Yan","affiliation":"Center for Data Science , Academy for Advanced Interdisciplinary Studies , Peking University"}],"abstract":"Recent research indicates that pretraining cross-lingual language models on large-scale unlabeled texts yields significant performance improvements over various cross-lingual and low-resource tasks. Through training on one hundred languages and terabytes of texts, cross-lingual language models have proven to be effective in leveraging high-resource languages to enhance low-resource language processing and outperform monolingual models. In this paper, we further investigate the cross-lingual and cross-domain (CLCD) setting when a pretrained cross-lingual language model needs to adapt to new domains. Specifically, we propose a novel unsupervised feature decomposition method that can automatically extract domain-specific features and domain-invariant features from the entangled pretrained cross-lingual representations, given unlabeled raw texts in the source language. Our proposed model leverages mutual information estimation to decompose the representations computed by a cross-lingual model into domain-invariant and domain-specific parts. Experimental results show that our proposed method achieves significant performance improvements over the state-of-the-art pretrained cross-lingual language model in the CLCD setting.","doi":"10.24963/ijcai.2020/508","url":"https://ijcai.org/proceedings/2020/508","pdf":"https://ijcai.org/proceedings/2020/0508.pdf","markdown":"https://ijcai.org/proceedings/2020/508.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/508","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":508,"pages":{"first":3672,"last":3678},"keywords":["Natural Language Processing: Sentiment Analysis and Text Mining"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24533/file_24533.m3u8","length_sec":299,"provider":"hls"}],"references":[{"position":1,"raw":"[Artetxe et al., 2018] Mikel Artetxe, Gorka Labaka, and Eneko Agirre. A robust self-learning method for fully un- supervised cross-lingual mappings of word embeddings. In ACL, pages 789-798, 2018.","title":"A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings","authors":"Mikel Artetxe, Gorka Labaka, Eneko Agirre","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Belghazi et al., 2018] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, De- von Hjelm, and Aaron Courville. Mutual information neu- ral estimation. In ICML, pages 530-539, 2018.","title":"Mutual information neural estimation","authors":"Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Devon Hjelm, Aaron Courville","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018] Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. Adversarial deep averaging networks for cross-lingual sentiment classifica- tion. TACL, 6:557-570, 2018.","title":"Adversarial deep averaging networks for cross-lingual sentiment classification","authors":"Chen","year":"2018","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2019] Xilun Chen, Ahmed Hassan Awadallah, Hany Hassan, Wei Wang, and Claire Cardie. Multi-source cross-lingual model transfer: Learning what to share. In ACL, 2019.","title":"Multi-source cross-lingual model transfer: Learning what to share","authors":"Chen","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Conneau and Lample, 2019] Alexis Conneau and Guil- laume Lample. Cross-lingual language model pretraining. In NeurIPS, 2019.","title":"Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining","authors":"Lample Conneau","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Conneau et al., 2018a] Alexis Conneau, Guillaume Lam- ple, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In ICLR, 2018.","title":"Word translation without parallel data","authors":"Conneau","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Conneau et al., 2018b] Alexis Conneau, Ruty Rinott, Guil- laume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross- lingual sentence representations. In EMNLP, pages 2475- 2485, 2018.","title":"Xnli: Evaluating crosslingual sentence representations","authors":"Conneau","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Conneau et al., 2019] Alexis Conneau, Kartikay Khandel- wal, Naman Goyal, Vishrav Chaudhary, Guillaume Wen- zek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross- lingual representation learning at scale. arXiv preprint arXiv:1911.02116, 2019.","title":"Unsupervised crosslingual representation learning at scale","authors":"Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In ACL, pages 4171-4186, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Donsker and Varadhan, 1983] Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of cer- tain markov process expectations for large time. IV. Communications on Pure and Applied Mathematics, 36(2):183-212, 1983.","title":"Asymptotic evaluation of certain markov process expectations for large time","authors":"D Monroe, Donsker, Sr Srinivasa Varadhan","year":"1983","venue":"IV. Communications on Pure and Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He and McAuley, 2016] Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fash- ion trends with one-class collaborative filtering. In WWW, pages 507-517, 2016.","title":"Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering","authors":"Mcauley He","year":"2016","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2019] Junxian He, Zhisong Zhang, Taylor Berg- Kiripatrick, and Graham Neubig. Cross-lingual syntactic transfer through unsupervised adaptation of invertible pro- jections. arXiv preprint arXiv:1906.02656, 2019.","title":"Cross-lingual syntactic transfer through unsupervised adaptation of invertible projections","authors":"He","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Heinzerling and Strube, 2018] Benjamin Heinzerling and Michael Strube. Bpemb: Tokenization-free pre-trained subword embeddings in 275 languages. In LREC, 2018.","title":"Bpemb: Tokenization-free pre-trained subword embeddings in 275 languages","authors":"Strube Heinzerling, Benjamin Heinzerling, Michael Strube","year":"2018","venue":"LREC","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hjelm et al., 2019] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representa- tions by mutual information estimation and maximization. In ICLR, 2019.","title":"Learning deep representations by mutual information estimation and maximization","authors":"Hjelm","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim et al., 2019] Yunsu Kim, Yingbo Gao, and Hermann Ney. Effective cross-lingual transfer of neural machine translation models without shared vocabularies. arXiv preprint arXiv:1905.05475, 2019.","title":"Effective cross-lingual transfer of neural machine translation models without shared vocabularies","authors":"Kim","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2019] Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, et al. Choos- ing transfer languages for cross-lingual learning. arXiv preprint arXiv:1905.12688, 2019.","title":"Choosing transfer languages for cross-lingual learning","authors":"Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Maaten and Hinton, 2008] Laurens van der Maaten and Ge- offrey Hinton. Visualizing data using t-sne. JMLR, 9(Nov):2579-2605, 2008.","title":"Laurens van der Maaten and Geoffrey Hinton","authors":"Hinton Maaten","year":"2008","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pham et al., 2015] Hieu Pham, Thang Luong, and Christo- pher Manning. Learning distributed representations for multilingual text sequences. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Lan- guage Processing, pages 88-94, 2015.","title":"Learning distributed representations for multilingual text sequences","authors":"Hieu Pham, Thang Luong, Christopher Manning","year":"2015","venue":"Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Prettenhofer and Stein, 2010] Peter Prettenhofer and Benno Stein. Cross-language text classification using structural correspondence learning. In ACL, pages 1118-1127, 2010.","title":"Cross-language text classification using structural correspondence learning","authors":"Stein Prettenhofer, Peter Prettenhofer, Benno Stein","year":"2010","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vulić et al., 2019] Ivan Vulić, Simone Paolo Ponzetto, and Goran Glavaš. Multilingual and cross-lingual graded lexi- cal entailment. In ACL, pages 4963-4974, 2019.","title":"Multilingual and cross-lingual graded lexical entailment","authors":"Ivan Vulić, Simone Paolo Ponzetto, Goran Glavaš","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xiao and Guo, 2013] Min Xiao and Yuhong Guo. Semi- supervised representation learning for cross-lingual text classification. In EMNLP, pages 1465-1475, 2013.","title":"Semisupervised representation learning for cross-lingual text classification","authors":"Xiao, Guo ; ] Min, Xiao, Yuhong Guo","year":"2013","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu and Yang, 2017] Ruochen Xu and Yiming Yang. Cross- lingual distillation for text classification. In ACL, pages 1415-1425, 2017.","title":"Crosslingual distillation for text classification","authors":"Yang ; Xu, Ruochen Xu, Yiming Yang","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yarowsky et al., 2001] David Yarowsky, Grace Ngai, and Richard Wicentowski. Inducing multilingual text analy- sis tools via robust projection across aligned corpora. In HLT, pages 1-8, 2001.","title":"Inducing multilingual text analysis tools via robust projection across aligned corpora","authors":"David Yarowsky, Grace Ngai, Richard Wicentowski","year":"2001","venue":"HLT","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}