{"type":"ScholarlyArticle","title":"In-Context Meta LoRA Generation","authors":[{"name":"Yihua Shao","slug":"yihua-shao","given":"Yihua","family":"Shao","affiliation":""},{"name":"Minxi Yan","slug":"minxi-yan","given":"Minxi","family":"Yan","affiliation":""},{"name":"Yang Liu","slug":"yang-liu","given":"Yang","family":"Liu","affiliation":""},{"name":"Siyu Chen","slug":"siyu-chen","given":"Siyu","family":"Chen","affiliation":""},{"name":"Wenjie Chen","slug":"wenjie-chen","given":"Wenjie","family":"Chen","affiliation":""},{"name":"Xinwei Long","slug":"xinwei-long","given":"Xinwei","family":"Long","affiliation":""},{"name":"Ziyang Yan","slug":"ziyang-yan","given":"Ziyang","family":"Yan","affiliation":""},{"name":"Lei Li","slug":"lei-li","given":"Lei","family":"Li","affiliation":""},{"name":"Chenyu Zhang","slug":"chenyu-zhang","given":"Chenyu","family":"Zhang","affiliation":""},{"name":"Nicu Sebe","slug":"nicu-sebe","given":"Nicu","family":"Sebe","affiliation":""},{"name":"Hao Tang","slug":"hao-tang","given":"Hao","family":"Tang","affiliation":""},{"name":"Yan Wang","slug":"yan-wang","given":"Yan","family":"Wang","affiliation":""},{"name":"Hao Zhao","slug":"hao-zhao","given":"Hao","family":"Zhao","affiliation":""},{"name":"Mengzhu Wang","slug":"mengzhu-wang","given":"Mengzhu","family":"Wang","affiliation":""},{"name":"Jingcai Guo","slug":"jingcai-guo","given":"Jingcai","family":"Guo","affiliation":""}],"abstract":"Low-rank Adaptation (LoRA) has demonstrated remarkable capabilities for task specific fine-tuning. However, in scenarios that involve multiple tasks, training a separate LoRA model for each one results in considerable inefficiency in terms of storage and inference. Moreover, existing parameter generation methods fail to capture the correlations among these tasks, making multi-task LoRA parameter generation challenging. To address these limitations, we propose In-Context Meta LoRA (ICM-LoRA), a novel approach that efficiently achieves task-specific customization of large language models (LLMs). Specifically, we use training data from all tasks to train a tailored generator, Conditional Variational Autoencoder (CVAE). CVAE takes task descriptions as inputs and produces task-aware LoRA weights as outputs. These LoRA weights are then merged with LLMs to create task-specialized models without the need for additional fine-tuning. Furthermore, we utilize in-context meta-learning for knowledge enhancement and task mapping, to capture the relationship between tasks and parameter distributions. As a result, our method achieves more accurate LoRA parameter generation for diverse tasks using CVAE. ICM-LoRA enables more accurate LoRA parameter reconstruction than current parameter reconstruction methods and is useful for implementing task-specific enhancements of LoRA parameters. At the same time, our method occupies 283MB, only 1% storage compared with the original LoRA. The code is available at https://github.com/YihuaJerry/ICM-LoRA.","doi":"10.24963/ijcai.2025/683","url":"https://ijcai.org/proceedings/2025/683","pdf":"https://ijcai.org/proceedings/2025/0683.pdf","markdown":"https://ijcai.org/proceedings/2025/683.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/683","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Machine Learning","track":"Main Track","number":683,"pages":{"first":6138,"last":6146},"keywords":["Machine Learning: ML: Multi-task and transfer learning","Machine Learning: ML: Cost-sensitive learning","Machine Learning: ML: Foundation models","Computer Vision: CV: Transfer, low-shot, semi- and un- supervised learning"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}