{"type":"ScholarlyArticle","title":"Prompt Learning with Extended Kalman Filter for Pre-trained Language Models","authors":[{"name":"Quan Li","slug":"quan-li","given":"Quan","family":"Li","affiliation":"School of Computer Science and Technology , University of Science and Technology of China , China; Data Darkness Lab , MIRACLE Center , Suzhou Institute for Advanced Research , USTC , China"},{"name":"Xike Xie","slug":"xike-xie","given":"Xike","family":"Xie","affiliation":"School of Biomedical Engineering , University of Science and Technology of China (USTC) , China; Data Darkness Lab , MIRACLE Center , Suzhou Institute for Advanced Research , USTC , China"},{"name":"Chao Wang","slug":"chao-wang","given":"Chao","family":"Wang","affiliation":"School of Computer Science and Technology , University of Science and Technology of China , China; School of Software Engineering , USTC , China"},{"name":"S. Kevin Zhou","slug":"s-kevin-zhou","given":"S. Kevin","family":"Zhou","affiliation":"School of Biomedical Engineering , University of Science and Technology of China (USTC) , China; Data Darkness Lab , MIRACLE Center , Suzhou Institute for Advanced Research , USTC , China; Key Laboratory of Precision and Intelligent Chemistry , USTC , Hefei Anhui , 230026 , China"}],"abstract":"Prompt learning has gained popularity as a means to leverage the knowledge embedded in pre-trained language models (PLMs) for NLP tasks while using a limited number of trainable parameters. While it has shown promise in tasks like sentiment classification and natural language inference, generating suitable prompts for PLMs, as opposed to human prompts, remains a challenge. In this paper, we introduce an abstraction of the prompt learning process using an extended Kalman filter. Our approach, called Conditional Extended Kalman Filter based on Neural Networks (CEKFNN), effectively infers more appropriate prompt tokens by enhancing the classic extended Kalman filter with PLM's contextual representation power. Specifically, CEKFNN learns transition and emission functions from PLM embeddings of input sentences to infer latent prompt tokens. We refine CEKFNN using an alternate-training approach, retraining a PLM's emission function with prompt tokens inferred by prompt models (PMs), as well as the initial and transition functions. PLM's output labels assist in PMs' training. When updating the pre-trained language model (PLM), we use an adapter approach with few trainable parameters, leaving PLM parameters frozen. We evaluate CEKFNN across open-source PLMs, demonstrating performance improvements over state-of-the-art methods while using a limited number of trainable parameters. It shows that CEKFNN performs on-par or better than fine-tuning, which requires updating all parameters in the PLM.","doi":"10.24963/ijcai.2024/492","url":"https://ijcai.org/proceedings/2024/492","pdf":"https://ijcai.org/proceedings/2024/0492.pdf","markdown":"https://ijcai.org/proceedings/2024/492.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/492","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":492,"pages":{"first":4452,"last":4460},"keywords":["Natural Language Processing: NLP: Language models","Machine Learning: ML: Deep learning architectures","Machine Learning: ML: Bayesian learning","Planning and Scheduling: PS: Markov decisions processes"],"videos":[],"references":[{"position":1,"raw":"[Alishahi et al., 2019] Afra Alishahi, Grzegorz Chrupała, and Tal Linzen. Analyzing and interpreting neural net- works for nlp: A report on the first blackboxnlp workshop. Natural Language Engineering, 25(4):543-557, 2019.","title":"Analyzing and interpreting neural networks for nlp: A report on the first blackboxnlp workshop","authors":"Alishahi","year":"2019","venue":"Natural Language Engineering","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arora et al., 2023] Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel J. Orr, Neel Guha, Kush Bhatia, Ines Chami, and Christopher Ré. Ask me anything: A simple strategy for prompting language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.","title":"Ask me anything: A simple strategy for prompting language models","authors":"Simran Arora, Avanika Narayan, Mayee F Chen, Laurel J Orr, Neel Guha, Kush Bhatia, Ines Chami, Christopher Ré","year":"2023","venue":"The Eleventh International Conference on Learning Representations, ICLR 2023","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bentivogli et al., 2009] Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recog- nizing textual entailment challenge. TAC, 7:8, 2009.","title":"The fifth pascal recognizing textual entailment challenge","authors":"Luisa Bentivogli, Peter Clark, Ido Dagan, Danilo Giampiccolo","year":"2009","venue":"TAC","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing sys- tems, 33:1877-1901, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cer et al., 2017] Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval- 2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055, 2017.","title":"Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation","authors":"Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, Lucia Specia","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Clark et al., 2019a] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019.","title":"Boolq: Exploring the surprising difficulty of natural yes/no questions","authors":"Clark","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Clark et al., 2019b] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, 2019.","title":"What does bert look at? an analysis of bert's attention","authors":"Clark","year":"2019","venue":"Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Davison et al., 2019] Joe Davison, Joshua Feldman, and Alexander M Rush. Commonsense knowledge mining from pretrained models. In Proceedings of the 2019 con- ference on empirical methods in natural language process- ing and the 9th international joint conference on natu- ral language processing (EMNLP-IJCNLP), pages 1173- 1178, 2019. [","title":"Commonsense knowledge mining from pretrained models","authors":"Joe Davison, Joshua Feldman, Alexander M Rush","year":"2019","venue":"Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dettmers et al., 2023] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetun- ing of quantized llms. arXiv preprint arXiv:2305.14314, 2023.","title":"Qlora: Efficient finetuning of quantized llms","authors":"Dettmers","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics, 2019.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dolan and Brockett, 2005] Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential para- phrases. In Third International Workshop on Paraphrasing (IWP2005), 2005.","title":"Automatically constructing a corpus of sentential paraphrases","authors":"Brockett Dolan, Bill Dolan, Chris Brockett","year":"2005","venue":"Third International Workshop on Paraphrasing","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Dong et al., 2019] Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre- training for natural language understanding and gener- ation. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Informa- tion Processing Systems 32: Annual Conference on Neu- ral Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13042-13054, 2019.","title":"Unified language model pretraining for natural language understanding and generation","authors":"Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon","year":"2019","venue":"Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hewitt and Manning, 2019] John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129-4138, 2019.","title":"A structural probe for finding syntax in word representations","authors":"John Hewitt, Christopher D Manning","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hu et al., 2022] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.","title":"Lora: Low-rank adaptation of large language models","authors":"Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen","year":"2022","venue":"The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim et al., 2018] Yoon Kim, Sam Wiseman, and Alexan- der M Rush. A tutorial on deep latent variable models of natural language. arXiv preprint arXiv:1812.06834, 2018.","title":"A tutorial on deep latent variable models of natural language","authors":"Yoon Kim, Sam Wiseman, Alexander M Rush","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lampinen et al., 2022] Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill. Can language models learn from explanations in context? In Findings of the Association for Computational Linguis- tics: EMNLP 2022, pages 537-563, 2022.","title":"Can language models learn from explanations in context?","authors":"Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James Mcclelland, Jane Wang, Felix Hill","year":"2022","venue":"Findings of the Association for Computational Linguistics: EMNLP 2022","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Levesque et al., 2012a] Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012.","title":"The winograd schema challenge","authors":"Levesque","year":"2012","venue":"Thirteenth international conference on the principles of knowledge representation and reasoning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Levesque et al., 2012b] Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012.","title":"The winograd schema challenge","authors":"Levesque","year":"2012","venue":"Thirteenth international conference on the principles of knowledge representation and reasoning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2021] Yinghao Li, Pranav Shetty, Lucas Liu, Chao Zhang, and Le Song. Bertifying the hidden markov model for multi-source weakly supervised named entity recogni- tion. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pages 6178-6190, 2021.","title":"Bertifying the hidden markov model for multi-source weakly supervised named entity recognition","authors":"Yinghao Li, Pranav Shetty, Lucas Liu, Chao Zhang, Le Song","year":"2021","venue":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2023a] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.","title":"Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing","authors":"Liu","year":"2023","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2023b] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. AI Open, 2023.","title":"Gpt understands, too","authors":"Liu","year":"2023","venue":"AI Open","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ouyang et al., 2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instruc- tions with human feedback. Advances in Neural Informa- tion Processing Systems, 35:27730-27744, 2022.","title":"Training language models to follow instructions with human feedback","authors":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pilehvar and Camacho-Collados, 2019] Mohammad Taher Pilehvar and Jose Camacho-Collados. Wic: the word- in-context dataset for evaluating context-sensitive mean- ing representations. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 1267- 1273, 2019.","title":"Wic: the wordin-context dataset for evaluating context-sensitive meaning representations","authors":"Camacho-Collados ; Pilehvar, Taher Mohammad, Jose Pilehvar, Camacho-Collados","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"Radford et al., ] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Lan- guage models are unsupervised multitask learners.","title":"Language models are unsupervised multitask learners","authors":"Radford","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Raffel et al., 2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67, 2020.","title":"Exploring the limits of transfer learning with a unified text-to-text transformer","authors":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu","year":"2020","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Rajpurkar et al., 2016] Pranav Rajpurkar, Jian Zhang, Kon- stantin Lopyrev, and Percy Liang. Squad: 100,000+ ques- tions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, 2016.","title":"Squad: 100,000+ questions for machine comprehension of text","authors":"Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang","year":"2016","venue":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Revach et al., 2022] Guy Revach, Nir Shlezinger, Xiaoyong Ni, Adria Lopez Escoriza, Ruud JG Van Sloun, and Yon- ina C Eldar. Kalmannet: Neural network aided kalman filtering for partially known dynamics. IEEE Transactions on Signal Processing, 70:1532-1547, 2022.","title":"Kalmannet: Neural network aided kalman filtering for partially known dynamics","authors":"Guy Revach, Nir Shlezinger, Xiaoyong Ni, Adria Lopez Escoriza, Ruud Jg Van Sloun, Yonina C Eldar","year":"2022","venue":"IEEE Transactions on Signal Processing","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Roemmele et al., 2011] Melissa Roemmele, Cos- min Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011.","title":"Choice of plausible alternatives: An evaluation of commonsense causal reasoning","authors":"Melissa Roemmele, Cosmin, Adrian Bejan, Andrew S Gordon","year":"2011","venue":"AAAI Spring Symposium Series","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Shin et al., 2020] Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automati- cally generated prompts. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 4222-4235, 2020.","title":"Autoprompt: Eliciting knowledge from language models with automatically generated prompts","authors":"Shin","year":"2020","venue":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Socher et al., 2013] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013.","title":"Recursive deep models for semantic compositionality over a sentiment treebank","authors":"Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, Christopher Potts","year":"2013","venue":"Proceedings of the 2013 conference on empirical methods in natural language processing","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2019a] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language un- derstanding systems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3261-3275, 2019.","title":"Superglue: A stickier benchmark for general-purpose language understanding systems","authors":"Wang","year":"2019","venue":"Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2019b] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Con- ference on Learning Representations, ICLR 2019, New Or- leans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.","title":"GLUE: A multi-task benchmark and analysis platform for natural language understanding","authors":"Wang","year":"2019","venue":"7th International Conference on Learning Representations, ICLR 2019","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Warstadt et al., 2019] Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Compu- tational Linguistics, 7:625-641, 2019.","title":"Neural network acceptability judgments","authors":"Alex Warstadt, Amanpreet Singh, Samuel R Bowman","year":"2019","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Williams et al., 2018] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, 2018.","title":"A broad-coverage challenge corpus for sentence understanding through inference","authors":"Adina Williams, Nikita Nangia, Samuel Bowman","year":"2018","venue":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wolf et al., 2020] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Fun- towicz, et al. Transformers: State-of-the-art natural lan- guage processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: sys- tem demonstrations, pages 38-45, 2020.","title":"Transformers: State-of-the-art natural language processing","authors":"Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz","year":"2020","venue":"Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Xie et al., 2022] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In The Tenth Inter- national Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.","title":"An explanation of in-context learning as implicit bayesian inference","authors":"Sang Michael Xie, Aditi Raghunathan, Percy Liang, Tengyu Ma","year":"2022","venue":"The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhang et al., 2023] Qingru Zhang, Minshuo Chen, Alexan- der Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter- efficient fine-tuning. In The Eleventh International Con- ference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.","title":"Adaptive budget allocation for parameterefficient fine-tuning","authors":"Zhang","year":"2023","venue":"The Eleventh International Conference on Learning Representations, ICLR 2023","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}