{"type":"ScholarlyArticle","title":"GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost","authors":[{"name":"Qingcheng Zeng","slug":"qingcheng-zeng","given":"Qingcheng","family":"Zeng","affiliation":"Department of Linguistics , Northwestern University"},{"name":"Lucas Garay","slug":"lucas-garay","given":"Lucas","family":"Garay","affiliation":"School of Public Health and the Second Affiliated Hospital , Zhejiang University"},{"name":"Peilin Zhou","slug":"peilin-zhou","given":"Peilin","family":"Zhou","affiliation":"Data Science and Analytics Thrust , Hong Kong University of Science and Technology (Guangzhou )"},{"name":"Dading Chong","slug":"dading-chong","given":"Dading","family":"Chong","affiliation":"School of Electronic and Computer Engineering , Peking University"},{"name":"Yining Hua","slug":"yining-hua","given":"Yining","family":"Hua","affiliation":"Department of Biomedical Informatics , Harvard University"},{"name":"Jiageng Wu","slug":"jiageng-wu","given":"Jiageng","family":"Wu","affiliation":"School of Public Health and the Second Affiliated Hospital , Zhejiang University"},{"name":"Yikang Pan","slug":"yikang-pan","given":"Yikang","family":"Pan","affiliation":"School of Public Health and the Second Affiliated Hospital , Zhejiang University"},{"name":"Han Zhou","slug":"han-zhou","given":"Han","family":"Zhou","affiliation":"Language Technology Lab , University of Cambridge"},{"name":"Rob Voigt","slug":"rob-voigt","given":"Rob","family":"Voigt","affiliation":"Department of Linguistics , Northwestern University"},{"name":"Jie Yang","slug":"jie-yang","given":"Jie","family":"Yang","affiliation":"School of Public Health and the Second Affiliated Hospital , Zhejiang University"}],"abstract":"Large pre-trained models have revolutionized natural language processing (NLP) research and applications, but high training costs and limited data resources have prevented their benefits from being shared equally amongst speakers of all the world's languages. To address issues of cross-linguistic access to such models and reduce energy consumption for sustainability during large-scale model training, this study proposes an effective and energy-efficient framework called GreenPLM that uses bilingual lexicons to directly ``translate'' pre-trained language models of one language into another at almost no additional cost. We validate this approach in 18 languages' BERT models and show that this framework is comparable to, if not better than, other heuristics with high training costs. In addition, given lightweight continued pre-training on limited data where available, this framework outperforms the original monolingual language models in six out of seven tested languages with up to 200x less pre-training efforts. Aiming at the Leave No One Behind Principle (LNOB), our approach manages to reduce inequalities between languages and energy consumption greatly. We make our codes and models publicly available at https://github.com/qcznlp/GreenPLMs.","doi":"10.24963/ijcai.2023/698","url":"https://ijcai.org/proceedings/2023/698","pdf":"https://ijcai.org/proceedings/2023/0698.pdf","markdown":"https://ijcai.org/proceedings/2023/698.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/698","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"AI for Good","track":"AI for Good","number":698,"pages":{"first":6290,"last":6298},"keywords":["AI for Good: AI Ethics, Trust, Fairness","AI for Good: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Adelani et al., 2021] David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D'souza, Julia Kreutzer, Constan- tine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, Jesujoba Alabi, Seid Muhie Yimam, Tajuddeen Rabiu Gwadabe, Ig- natius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiibi, Verrah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chiamaka Chukwuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akinode, Deborah Nabagereka, Maurice Katusiime, Ayodele Awokoya, Mouhamadane MBOUP, Dibora Gebreyohannes, Henok Tilaye, Kelechi Nwaike, Degaga Wolde, Abdoulaye Faye, Blessing Sibanda, Orevaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thierno Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, and Salomey Osei. MasakhaNER: Named entity recognition for African languages. Transactions of the Association for Computational Linguistics, 9:1116-1131, 2021.","title":"MasakhaNER: Named entity recognition for African languages","authors":"David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, D' Daniel, Julia Souza, Constantine Kreutzer, Chester Lignos, Happy Palen-Michel, Shruti Buzaaba, Sebastian Rijhwani, Stephen Ruder, Israel Mayhew, Shamsuddeen H Abebe Azime, Chris Chinenye Muhammad, Joyce Emezue, Perez Nakatumba-Nabende, Aremu Ogayo, Catherine Anuoluwapo, Derguene Gitau, Jesujoba Mbaye, Seid Alabi, Tajuddeen Muhie Yimam, Ignatius Rabiu Gwadabe, Ezeani, Andre Rubungo, Jonathan Niyongabo, Verrah Mukiibi, Iroro Otiende, Davis Orife, Samba David, Tosin Ngom, Paul Adewumi, Mofetoluwa Rayson, Gerald Adeyemi, Emmanuel Muriuki, Chiamaka Anebi, Nkiruka Chukwuneke, Eric Odu, Samuel Peter Wairagala, Clemencia Oyerinde, Tobius Siro, Temilola Saul Bateesa, Yvonne Oloyede, Victor Wambui, Deborah Akinode, Maurice Nabagereka, Ayodele Katusiime, Awokoya, Mboup Mouhamadane, Dibora Gebreyohannes, Henok Tilaye, Kelechi Nwaike, Degaga Wolde, Abdoulaye Faye, Blessing Sibanda, Orevaoghene Ahia, F P Bonaventure, Kelechi Dossou, Thierno Ogueji, Diop Ibrahima, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, Salomey Osei","year":"2021","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Artetxe et al., 2016] Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Learning principled bilingual mappings of word embed- dings while preserving monolingual invariance. In Proceedings of the 2016 EMNLP, pages 2289-2294, Austin, Texas, November 2016.","title":"Learning principled bilingual mappings of word embeddings while preserving monolingual invariance","authors":"Mikel Artetxe, Gorka Labaka, Eneko Agirre","year":"2016","venue":"Proceedings of the 2016 EMNLP","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Artetxe et al., 2020] Mikel Artetxe, Sebastian Ruder, and Dani Yo- gatama. On the cross-lingual transferability of monolingual rep- resentations. In Proceedings of the 58th ACL, pages 4623-4637, Online, July 2020.","title":"On the cross-lingual transferability of monolingual representations","authors":"Mikel Artetxe, Sebastian Ruder, Dani Yogatama","year":"2020","venue":"Proceedings of the 58th ACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Assembly, 2015] UN General Assembly. Transforming our world: the 2030 agenda for sustainable development. https://www. refworld.org/docid/57b6e3e44.html, 2015. Accessed: 2023-06- 13.","title":"UN General Assembly. Transforming our world: the 2030 agenda for sustainable development","authors":null,"year":"2015","venue":null,"doi":null,"url":"https://www.refworld.org/docid/57b6e3e44.html,2015","internal_url":null},{"position":5,"raw":"[Bird, 2020] Steven Bird. Decolonising speech and language tech- nology. In Proceedings of the 28th COLING, pages 3504-3519, Barcelona, Spain (Online), December 2020.","title":"Decolonising speech and language technology","authors":"Steven Bird","year":"2020","venue":"Proceedings of the 28th COLING","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Blevins and Zettlemoyer, 2022] Terra Blevins and Luke Zettle- moyer. Language contamination helps explains the cross-lingual capabilities of English pretrained models. In Proceedings of the 2022 EMNLP, pages 3563-3574, Abu Dhabi, United Arab Emi- rates, December 2022.","title":"Terra Blevins and Luke Zettlemoyer. Language contamination helps explains the cross-lingual capabilities of English pretrained models","authors":"Zettlemoyer Blevins","year":"2022","venue":"Proceedings of the 2022 EMNLP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christo- pher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, NIPS, volume 33, pages 1877-1901.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mccandlish, Alec Radford, Ilya Sutskever, Dario Amodei","year":null,"venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"Curran Associates, Inc., 2020.","title":null,"authors":"Curran Associates, Inc","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cañete et al., 2020] José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, and Jorge Pérez. Spanish pre- trained bert model and evaluation data. In PML4DC at ICLR 2020, 2020.","title":"Spanish pretrained bert model and evaluation data","authors":"José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, Jorge Pérez","year":"2020","venue":"PML4DC at ICLR 2020","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen et al., 2022] Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, and Qun Liu. bert2BERT: Towards reusable pretrained lan- guage models. In Proceedings of the 60th ACL, pages 2134- 2148, Dublin, Ireland, May 2022. [Conneau and Lample, 2019] Alexis Conneau and Guillaume Lam- ple. Cross-Lingual Language Model Pretraining. Curran Asso- ciates Inc., Red Hook, NY, USA, 2019.","title":"bert2BERT: Towards reusable pretrained language models","authors":"Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, Qun Liu","year":"2019","venue":"Alexis Conneau and Guillaume Lample. Cross-Lingual Language Model Pretraining","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Conneau et al., 2017] Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. arXiv preprint arXiv:1710.04087, 2017.","title":"Word translation without parallel data","authors":"Alexis Conneau, Guillaume Lample, Marc'aurelio Ranzato, Ludovic Denoyer, Hervé Jégou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Conneau et al., 2020a] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Fran- cisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th ACL, pages 8440- 8451, Online, July 2020.","title":"Unsupervised cross-lingual representation learning at scale","authors":"Conneau","year":"2020","venue":"Proceedings of the 58th ACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Conneau et al., 2020b] Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. Emerging cross-lingual structure in pretrained language models. In Proceedings of the 58th ACL, pages 6022-6034, Online, July 2020.","title":"Emerging cross-lingual structure in pretrained language models","authors":"Conneau","year":"2020","venue":"Proceedings of the 58th ACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Cruz and Cheng, 2019] Jan Christian Blaise Cruz and Charibeth Cheng. Evaluating language model finetuning techniques for low-resource languages. arXiv preprint arXiv:1907.00409, 2019.","title":"Evaluating language model finetuning techniques for low-resource languages","authors":"Cheng Cruz","year":"2019","venue":"Jan Christian Blaise Cruz and Charibeth Cheng","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Cruz and Cheng, 2020] Jan Christian Blaise Cruz and Charibeth Cheng. Establishing baselines for text classification in low- resource languages. arXiv preprint arXiv:2005.02068, 2020.","title":"Establishing baselines for text classification in lowresource languages","authors":"Cheng Cruz","year":"2020","venue":"Jan Christian Blaise Cruz and Charibeth Cheng","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Cruz et al., 2020] Jan Christian Blaise Cruz, Jose Kristian Resa- bal, James Lin, Dan John Velasco, and Charibeth Cheng. Investi- gating the true performance of transformers in low-resource lan- guages: A case study in automatic corpus creation. arXiv preprint arXiv:2010.11574, 2020.","title":"Investigating the true performance of transformers in low-resource languages: A case study in automatic corpus creation","authors":"Cruz","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[de Marneffe et al., 2021] Marie-Catherine de Marneffe, Christo- pher D. Manning, Joakim Nivre, and Daniel Zeman. Univer- sal Dependencies. Computational Linguistics, 47(2):255-308, 07 2021.","title":"Universal Dependencies","authors":"Marie-Catherine De Marneffe, Christopher D Manning, Joakim Nivre, Daniel Zeman","year":"2021","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[de Vries and Nissim, 2021] Wietse de Vries and Malvina Nissim. As good as new. how to successfully recycle English GPT-2 to make models for other languages. In Findings of the ACL: ACL- IJCNLP 2021, pages 836-846, Online, August 2021.","title":"Wietse de Vries and Malvina Nissim. As good as new. how to successfully recycle English GPT-2 to make models for other languages","authors":"De Vries, Nissim","year":"2021","venue":"Findings of the ACL: ACL-IJCNLP 2021","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 NAACL-HLT, pages 4171-4186, Minneapolis, Minnesota, June 2019.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proceedings of the 2019 NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Dumitrescu et al., 2020] Stefan Dumitrescu, Andrei-Marius Avram, and Sampo Pyysalo. The birth of Romanian BERT. In Findings of the ACL: EMNLP 2020, pages 4324-4328, Online, November 2020.","title":"The birth of Romanian BERT","authors":"Stefan Dumitrescu, Andrei-Marius Avram, Sampo Pyysalo","year":"2020","venue":"Findings of the ACL: EMNLP 2020","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ebrahimi and Kann, 2021] Abteen Ebrahimi and Katharina Kann. How to adapt your pretrained multilingual model to 1600 lan- guages. In Proceedings of the 59th ACL, pages 4555-4567, On- line, August 2021.","title":"How to adapt your pretrained multilingual model to 1600 languages","authors":"Kann Ebrahimi, Abteen Ebrahimi, Katharina Kann","year":"2021","venue":"Proceedings of the 59th ACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Farahani et al., 2021] Mehrdad Farahani, Mohammad Gharachor- loo, Marzieh Farahani, and Mohammad Manthouri. ParsBERT: Transformer-based model for persian language understanding. Neural Processing Letters, 53(6):3831-3847, oct 2021.","title":"ParsBERT: Transformer-based model for persian language understanding","authors":"Mehrdad Farahani, Mohammad Gharachorloo, Marzieh Farahani, Mohammad Manthouri","year":"2021","venue":"Neural Processing Letters","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Joshi et al., 2020] Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of lin- guistic diversity and inclusion in the NLP world. In Proceedings of the 58th ACL, pages 6282-6293, Online, July 2020.","title":"The state and fate of linguistic diversity and inclusion in the NLP world","authors":"Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, Monojit Choudhury","year":"2020","venue":"Proceedings of the 58th ACL","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Kamholz et al., 2014] David Kamholz, Jonathan Pool, and Susan Colowick. PanLex: Building a resource for panlingual lexical translation. In Proceedings of the Ninth LREC, pages 3145-3150, Reykjavik, Iceland, May 2014.","title":"PanLex: Building a resource for panlingual lexical translation","authors":"David Kamholz, Jonathan Pool, Susan Colowick","year":"2014","venue":"Proceedings of the Ninth LREC","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Koto et al., 2020] Fajri Koto, Afshin Rahimi, Jey Han Lau, and Timothy Baldwin. IndoLEM and IndoBERT: A benchmark dataset and pre-trained language model for Indonesian NLP. In Proceedings of the 28th COLING, pages 757-770, Barcelona, Spain (Online), December 2020.","title":"IndoLEM and IndoBERT: A benchmark dataset and pre-trained language model for Indonesian NLP","authors":"Koto","year":"2020","venue":"Proceedings of the 28th COLING","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Lewis et al., 2020] Mike Lewis, Yinhan Liu, Naman Goyal, Mar- jan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence- to-sequence pre-training for natural language generation, transla- tion, and comprehension. In Proceedings of the 58th ACL, pages 7871-7880, Online, July 2020.","title":"BART: Denoising sequenceto-sequence pre-training for natural language generation, translation, and comprehension","authors":"Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer","year":"2020","venue":"Proceedings of the 58th ACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Lin et al., 2019] Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. Choosing transfer languages for cross-lingual learning. In Proceedings of the 57th ACL, pages 3125-3135, Florence, Italy, July 2019.","title":"Choosing transfer languages for cross-lingual learning","authors":"Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, Graham Neubig","year":"2019","venue":"Proceedings of the 57th ACL","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Liu et al., 2022] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv., sep 2022.","title":"Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing","authors":"Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, Graham Neubig","year":"2022","venue":"ACM Comput. Surv","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Minixhofer et al., 2022] Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. WECHSEL: Effective initialization of sub- word embeddings for cross-lingual transfer of monolingual lan- guage models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3992-4006, Seattle, United States, July 2022. Association for Computational Linguistics.","title":"WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models","authors":"Benjamin Minixhofer, Fabian Paischer, Navid Rekabsaz","year":"2022","venue":"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Muller et al., 2021] Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. When being unseen from mBERT is just the beginning: Handling new languages with mul- tilingual language models. In Proceedings of the 2021 NAACL- HLT, pages 448-462, Online, June 2021. [OpenAI, 2023] OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.","title":"When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models","authors":"Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, Djamé Seddah","year":"2021","venue":"Proceedings of the 2021 NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Ortiz Suárez et al., 2019] Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. Pro- ceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 -16, Mannheim, 2019. Leibniz-Institut für Deutsche Sprache.","title":"Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures","authors":"Ortiz Suárez","year":"2019","venue":"Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Peters et al., 2018] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Pro- ceedings of the 2018 NAACL-HLT, pages 2227-2237, New Or- leans, Louisiana, June 2018.","title":"Deep contextualized word representations","authors":"Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer","year":"2018","venue":"Proceedings of the 2018 NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Pires et al., 2019] Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual BERT? In Proceedings of the 57th ACL, pages 4996-5001, Florence, Italy, July 2019.","title":"How multilingual is multilingual BERT?","authors":"Pires","year":"2019","venue":"Proceedings of the 57th ACL","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Qin et al., 2022] Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Knowledge inheritance for pre- trained language models. In Proceedings of the 2022 NAACL- HLT, pages 3921-3937, Seattle, United States, July 2022.","title":"Knowledge inheritance for pretrained language models","authors":"Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou","year":"2022","venue":"Proceedings of the 2022 NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Raffel et al., 2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learn- ing with a unified text-to-text transformer. JMLR, 21(140):1-67, 2020.","title":"Exploring the limits of transfer learning with a unified text-to-text transformer","authors":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu","year":"2020","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Schwartz et al., 2020] Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM, 63(12):54-63, 2020.","title":"Green ai","authors":"Roy Schwartz, Jesse Dodge, Noah A Smith, Oren Etzioni","year":"2020","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Schweter, 2020a] Stefan Schweter. Berturk -bert models for turk- ish. https://doi.org/10.5281/zenodo.3770924, 2020. Accessed: 2023-06-13.","title":"Berturk -bert models for turkish","authors":"Schweter","year":"2020","venue":null,"doi":"10.5281/zenodo.3770924","url":"https://doi.org/10.5281/zenodo.3770924,2020","internal_url":null},{"position":38,"raw":"[Schweter, 2020b] Stefan Schweter. Italian bert and electra models. https://doi.org/10.5281/zenodo.4263142, 2020. Accessed: 2023- 06-13.","title":"Stefan Schweter. Italian bert and electra models","authors":"Schweter","year":"2020","venue":null,"doi":"10.5281/zenodo.4263142","url":"https://doi.org/10.5281/zenodo.4263142,2020","internal_url":null},{"position":39,"raw":"[Strubell et al., 2019] Emma Strubell, Ananya Ganesh, and An- drew McCallum. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th ACL, pages 3645- 3650, Florence, Italy, July 2019.","title":"Energy and policy considerations for deep learning in NLP","authors":"Emma Strubell, Ananya Ganesh, Andrew Mccallum","year":"2019","venue":"Proceedings of the 57th ACL","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Strubell et al., 2020] Emma Strubell, Ananya Ganesh, and An- drew McCallum. Energy and policy considerations for modern deep learning research. Proceedings of the AAAI, 34(09):13693- 13696, Apr. 2020.","title":"Energy and policy considerations for modern deep learning research","authors":"Emma Strubell, Ananya Ganesh, Andrew Mccallum","year":"2020","venue":"Proceedings of the AAAI","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Tiedemann and Nygaard, 2004] Jörg Tiedemann and Lars Ny- gaard. The OPUS corpus -parallel and free: http://logos.uio. no/opus. In Proceedings of the Fourth LREC, Lisbon, Portugal, May 2004.","title":"The OPUS corpus -parallel and free","authors":"Nygaard Tiedemann, Jörg Tiedemann, Lars Nygaard","year":"2004","venue":null,"doi":null,"url":"http://logos.uio.no/opus","internal_url":null},{"position":42,"raw":"[Tran, 2020] Ke Tran. From english to foreign languages: Transferring pre-trained language models. arXiv preprint arXiv:2002.07306, 2020.","title":"From english to foreign languages: Transferring pre-trained language models","authors":"Tran ;, Ke Tran","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Par- mar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Pro- ceedings of the 31st International Conference on Neural Informa- tion Processing Systems, NIPS'17, page 6000-6010, Red Hook, NY, USA, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Curran Associates Inc. Wang et al., 2022] Xinyi Wang, Sebastian Ruder, and Graham Neubig. Expanding pretrained models to thousands more lan- guages via lexicon-based adaptation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 863-877, Dublin, Ireland, May 2022. Association for Computational Linguistics.","title":"Expanding pretrained models to thousands more languages via lexicon-based adaptation","authors":"Curran Associates, Inc Wang","year":"2022","venue":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Warstadt et al., 2023] Alex Warstadt, Leshem Choshen, Aaron Mueller, Adina Williams, Ethan Wilcox, and Chengxu Zhuang. Call for papers-the babylm challenge: Sample-efficient pre- training on a developmentally plausible corpus. arXiv preprint arXiv:2301.11796, 2023.","title":"Call for papers-the babylm challenge: Sample-efficient pretraining on a developmentally plausible corpus","authors":"Alex Warstadt, Leshem Choshen, Aaron Mueller, Adina Williams, Ethan Wilcox, Chengxu Zhuang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Wu and Dredze, 2020] Shijie Wu and Mark Dredze. Are all lan- guages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120-130, Online, July 2020.","title":"Are all languages created equal in multilingual BERT?","authors":"Dredze ; ] Shijie Wu, Mark Wu, Dredze","year":"2020","venue":"Proceedings of the 5th Workshop on Representation Learning for NLP","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Yang and Zhang, 2018] Jie Yang and Yue Zhang. NCRF++: An open-source neural sequence labeling toolkit. In Proceedings of ACL 2018, System Demonstrations, pages 74-79, Melbourne, Australia, July 2018.","title":"NCRF++: An open-source neural sequence labeling toolkit","authors":"Yang, Zhang ; ] Jie Yang, Yue Zhang","year":"2018","venue":"Proceedings of ACL 2018, System Demonstrations","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Yao et al., 2022] Xingcheng Yao, Yanan Zheng, Xiaocong Yang, and Zhilin Yang. NLP from scratch without large-scale pretrain- ing: A simple and efficient framework. In Kamalika Chaud- huri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th ICML, vol- ume 162 of PMLR, pages 25438-25451. PMLR, 17-23 Jul 2022.","title":"NLP from scratch without large-scale pretraining: A simple and efficient framework","authors":"Xingcheng Yao, Yanan Zheng, Xiaocong Yang, Zhilin Yang","year":"2022","venue":"Proceedings of the 39th ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}