M-LoRA: Efficient Serving for Concurrent LoRA Adapters with Memory-Aware Speculative Scheduler on Single GPU

Li, Shaolong; Yang, Xiang; Qi, Qi; Sun, Haifeng; Zhuang, Zirui; He, Bo; Ning, Wanyi; Wang, Jingyu

doi:10.24963/ijcai.2026/502

M-LoRA: Efficient Serving for Concurrent LoRA Adapters with Memory-Aware Speculative Scheduler on Single GPU

Shaolong Li, Xiang Yang, Qi Qi, Haifeng Sun, Zirui Zhuang, Bo He, Wanyi Ning, Jingyu Wang

Download BibTeX Cite

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. Main Track. Pages 4509-4517.

https://doi.org/10.24963/ijcai.2026/502

Keywords

Machine Learning: Deep learning architectures
Planning and Scheduling: Scheduling

Abstract

Low-Rank Adaptation (LoRA) is a popular approach that enables large language models (LLMs) to quickly adapt to domain-specific tasks by adding lightweight trainable adapters. Existing multi-LoRA serving systems typically exploit parameter sharing to serve hundreds of LoRA models with a single base model. However, most systems rely on first-come-first-serve (FCFS) scheduling, which can incur severe queuing delay and lead to excessive adapter memory usage, squeezing KV cache space and reducing concurrency and throughput. To address these challenges, we propose M-LoRA, a memory-aware multi-LoRA serving system that reduces queuing delay and improves throughput through efficient request scheduling guided by fine-grained memory modeling. M-LoRA consists of three components: (1) Multi-LoRA Length Predictor, which estimates the output length and KV-cache demand of each request; (2) Memory-aware Speculative Scheduler, which dispatches requests based on predicted KV-cache and adapter memory requirements to maximize throughput under a fixed GPU memory budget; and (3) Recovery Mechanism, which corrects mispredictions while preserving generation quality. Compared to state-of-the-art LoRA serving systems, M-LoRA reduces average per-token latency by 87% and improves throughput by up to 1.7x.

Shaolong Li, Xiang Yang, Qi Qi, Haifeng Sun, Zirui Zhuang, Bo He, Wanyi Ning, and Jingyu Wang. 2026. M-LoRA: Efficient Serving for Concurrent LoRA Adapters with Memory-Aware Speculative Scheduler on Single GPU. In Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26. IJCAI, 4509–4517. https://doi.org/10.24963/ijcai.2026/502

@inproceedings{ijcai2026p502,
  title     = {M-LoRA: Efficient Serving for Concurrent LoRA Adapters with Memory-Aware Speculative Scheduler on Single GPU},
  author    = {Li, Shaolong and Yang, Xiang and Qi, Qi and Sun, Haifeng and Zhuang, Zirui and He, Bo and Ning, Wanyi and Wang, Jingyu},
  booktitle = {Proceedings of the Thirty-Fifth International Joint Conference on
               Artificial Intelligence, {IJCAI-26}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Diego Calvanese},
  pages     = {4509--4517},
  year      = {2026},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2026/502},
  url       = {https://doi.org/10.24963/ijcai.2026/502},
}

TY  - CPAPER
TI  - M-LoRA: Efficient Serving for Concurrent LoRA Adapters with Memory-Aware Speculative Scheduler on Single GPU
AU  - Li, Shaolong
AU  - Yang, Xiang
AU  - Qi, Qi
AU  - Sun, Haifeng
AU  - Zhuang, Zirui
AU  - He, Bo
AU  - Ning, Wanyi
AU  - Wang, Jingyu
T2  - Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2026
SP  - 4509
EP  - 4517
DO  - 10.24963/ijcai.2026/502
UR  - https://doi.org/10.24963/ijcai.2026/502
ER  -