M-LoRA: Efficient Serving for Concurrent LoRA Adapters with Memory-Aware Speculative Scheduler on Single GPU

1 Introduction

Large language models (LLMs) have demonstrated strong capabilities in text generation tasks. Under the pretrainingand-fine-tuning paradigm, LLMs can be efficiently adapted to domain-specific tasks. LoRA (Low-Rank Adaptation) is a widely used parameter-efficient fine-tuning method. It freezes the base model parameters and introduces small additional trainable low-rank matrices (called LoRA adapters)

∗Corresponding authors.

HBM on GPU

(b) Queuing delay when serving 20 LoRA models with different methods.

Figure 1: Existing multi-LoRA serving methods and their bottleneck.

into each transformer layer. This design allows multiple LoRA adapters to be derived from a single base model. As the number and scale of LoRA models continue to grow, it becomes crucial for cloud platforms to serve them efficiently, which directly impacts user experience and economic profits.

Recent works [Sheng et al., 2023; Chen et al., 2024b] have explored online multi-LoRA serving. The key to improving deployment efficiency is parameter sharing (Figure 1a). Multiple LoRA models share one copy of the base-model weights in GPU memory, while their LoRA adapters are deployed separately. Adapters are initially stored in CPU memory and loaded onto the GPU on demand. This design enables a single GPU to host hundreds of LoRA models.

Existing systems propose optimizations in memory management [Zhang et al., 2025], request migration [Wu et al., 2024], and heterogeneous inference [Li et al., 2025a]. However, these systems use simple request-scheduling strategies. For example, vLLM [Kwon et al., 2023] and Punica [Chen et al., 2024b] use a first-come-first-serve (FCFS) strategy, while S-LoRA [Sheng et al., 2023] employs an adapter-first strategy that prioritizes requests whose target adapter is already resident on the GPU. These strategies do not account for request length, leading to severe head-of-line blocking. As shown in Figure 1b, queuing delay can exceed 69s and account for over 90% of end-to-end latency. In such a scenario, it is well established that Shortest Job First (SJF) and Shortest Remaining Time First (SRTF) minimize the average waiting time.

However, in LLM inference, the output length is unknown at scheduling time, making such length-aware policies difficult to apply directly to multi-LoRA serving.

Recent studies [Jin et al., 2023; Qiu et al., 2024; Fu et al., 2024] have explored output-length prediction for online scheduling. Before inference, they use a lightweight predictor to estimate the generation length from the input prompt and schedule requests accordingly, approximating SJF and substantially reducing mean completion time. However, applying such length-aware scheduling to multi-LoRA serving faces two key challenges:

Challenge I: A single predictor cannot capture the outputlength behaviors of diverse LoRA models. Prior work typically uses a single predictor to estimate the output length for a single LLM, whereas multi-LoRA serving deploys many LoRA models simultaneously. Even when LoRA models are fine-tuned from the same pretrained base model, their outputlength distributions can differ substantially. We analyze the output-length distributions of 25 models in the LMSYSChat-1M dataset (Figure 2) and observe apparent distribution shifts. For example, claude-instant-1 (M6) tends to generate longer responses (peaking around 410 tokens), whereas oasst-pythia-12b (M7) is more concentrated in shorter ranges (mostly 10-200 tokens). As a result, a single predictor trained across all models cannot reliably capture the output-length distributions across them.

Challenge II: KV cache growth and adapter residency jointly create GPU memory contention. To reduce computation in autoregressive decoding, LLM systems cache intermediate attention key-value states in GPU memory as the KV cache. As generation proceeds, the KV cache grows and gradually consumes more GPU memory. A larger KV cache budget enables the system to serve more concurrent requests. Meanwhile, in multi-LoRA serving, adapters are loaded on demand and remain resident in GPU memory during execution, with their footprint varying as requests are scheduled and completed. Under a fixed GPU memory budget, KV caches and LoRA adapters directly compete. An inappropriate scheduling policy can result in LoRA adapters occupying excessive GPU memory, shrinking KV-cache space and reducing system concurrency and throughput. When using an FCFS scheduler, we observe that LoRA adapters can occupy nearly 40% of the cache space (Figure 3). Therefore, an efficient scheduling policy must jointly account for request length and LoRA adapters' memory usage.

To this end, we propose M-LoRA, a memory-aware multi-LoRA serving system that reduces queuing delay and improves throughput through efficient request scheduling guided by fine-grained memory modeling. M-LoRA consists of three components. First, theMulti-LoRALength Predictor (MLLP) predicts request output lengths for multiple LoRA models, using a shared backbone with model-specific LoRA modules to capture model-specific length patterns. Second, we design MeSS to derive a scheduling policy that balances KV cache space and LoRA adapters' memory usage, maximizing system concurrency and throughput. Finally, MLoRA provides an automatic recovery mechanism that handles mispredictions while preserving generation quality.

We evaluated LoRA models adapted from LLaMA [Touvron et al., 2023b] models on 4090 and A100 GPUs. Evaluation results show that M-LoRA can significantly improve throughput and queuing delay with only modest overhead. Compared to the state-of-the-art multi-LoRA serving systems, M-LoRA reduces average per-token latency by 87% and improves throughput by up to 1.7×.

In summary, our contributions are as follows:

• We identify a coupled scheduling problem in multi-LoRA serving. Beyond the unknown output length of each request, request scheduling must also address GPU memory contention between growing KV caches and on-demand LoRA adapters under a fixed memory budget.
• We design M-LoRA, a memory-aware multi-LoRA serving system. Through model-specific length prediction and fine-grained memory modeling, M-LoRA derives scheduling decisions that mitigate contention between KV caches and on-demand LoRA adapters, enabling more effective GPU memory allocation under a fixed memory budget.
• We implement and evaluate M-LoRA on 4090 and A100 GPUs. Evaluation results show that M-LoRA outperforms state-of-the-art LoRA serving systems, reducing latency per token by 87% and improving throughput by up to 1.7×.

2 RelatedWorks

Multi-LoRA serving systems. Advanced multi-LoRA serving systems adopt parameter sharing, avoiding duplicate base-model weights and reducing GPU memory demand. Punica [Chen et al., 2024b] proposes the BGMV kernel to enable batched inference with rank-heterogeneous LoRA adapters. S-LoRA [Sheng et al., 2023] introduces Unified Memory to manage the KV cache and LoRA adapters jointly, reducing memory fragmentation. dLoRA [Wu et al., 2024] targets large-scale multi-LoRA serving in clusters, supporting dynamic adapter merging and cross-node request migration. FastLibra [Zhang et al., 2025] models the dependencies between LoRA adapters and the KV cache using a tree structure, and builds a cost model to guide efficient memory swap-in and swap-out. Toppings [Li et al., 2025a] uses CPUassisted prefill to hide adapter loading overhead and applies rank-aware scheduling for cross-node dispatch.

LLM Inference Acceleration. Advanced systems optimize LLM inference through batching [Chen et al., 2024a; Yu et al., 2022], memory optimization [Kwon et al., 2023], GPU kernel optimization [Shah et al., 2024; Chen et al., 2018], and parallelization strategies [Ye et al., 2024; Li et al., 2023]. Speculative decoding [Li et al., 2024; Li et al., 2025b; Chen et al., 2025] improves LLM throughput by using a lightweight draft model to generate multiple tokens in a single forward pass. KV cache compression [He et al., 2025; Su et al., 2025; Wei et al., 2025] further reduces GPU memory footprint via low-bit quantization or compressed representations, thereby improving concurrency and throughput.

Sequence Length Prediction. Zheng et al. [Zheng et al., 2024] modify prompts to let LLMs predict their own output lengths. Other studies [Jin et al., 2023; Qiu et al., 2024; Fu et al., 2024] use BERT-based models to estimate sequence lengths. TetriInfer [Hu et al., 2024] uses OPT-125M to estimate the output length of requests. However, these approaches focus on predicting output lengths for a single LLM.

3 Methodology

3.1 Overview of M-LoRA

This section presents M-LoRA, a memory-aware multiLoRA serving system. M-LoRA aims to reduce queuing delay and improve throughput through efficient request scheduling guided by fine-grained memory modeling. M-LoRA consists of three components:

• Multi-LoRA Length Predictor (MLLP). MLLP is a LoRA-based predictor that estimates the output length of each request for multiple LoRA models, enabling lengthaware scheduling and more accurate memory provisioning.
• Memory-aware Speculative Scheduler (MeSS). MeSS performs fine-grained GPU memory modeling based on predicted lengths and LoRA adapter sizes. It formulates request scheduling as an optimization problem and solves it to derive a scheduling decision that reduces queuing delay and maximizes throughput under a fixed memory budget.
• Recovery mechanism. To account for inevitable mispredictions, M-LoRA employs an online recovery mechanism that identifies mispredictions and adjusts execution accordingly, preserving generation correctness and completeness.

Figure 4 illustrates the workflow of M-LoRA. In the offline phase, M-LoRA trains model-specific predictors by LoRA fine-tuning on top of a shared pretrained backbone (Figure 5a), and unifies them into MLLP with shared backbone weights and model-specific LoRA modules (Figure 5b). In the online phase, MLLP predicts the output length of each incoming request with the corresponding LoRA module and inserts it into the request pool. MeSS then selects requests from the pool using Algorithm 1 to maximize throughput under the GPU memory budget. If a request exceeds its predicted length, M-LoRA re-estimates its predicted length and reschedules the request through the recovery mechanism to ensure complete generation.

To jointly manage KV caches and LoRA adapters, we adopt the Unified Memory from S-LoRA [Sheng et al., 2023]. Under Unified Memory, both KV cache and adapter weights are stored in pages: a sequence of length L consumes L pages, and an adapter with rank R consumes R pages.

3.2 Multi-LoRA Length Predictor (MLLP)

In multi-LoRA serving, each request consumes GPU memory for both its KV cache and target LoRA adapter. Since the KV cache grows with each generated token in autoregressive decoding, the memory demand of a request is difficult to determine at arrival time. To support memory-aware scheduling, we estimate the output length of each request before execution to approximate its KV-cache demand.

We develop a Multi-LoRA Length Predictor (MLLP) to predict request output lengths in the multi-LoRA scenario. Figure 5 shows the training and inference structures of MLLP. Prior works [Jin et al., 2023; Qiu et al., 2024] have shown that lightweight predictors can achieve a practical accuracyoverhead trade-off for output-length estimation. Following this insight, MLLP adopts a small language model (LLaMA3.2-1B [Touvron et al., 2023b]) as its backbone. MLLP further improves accuracy and handles output length distribution shifts across LoRA models through two key designs: bucketed output length and LoRA-based fine-tuning.

Bucketed output length. As shown in Figure 2, output lengths in the LMSYS-Chat-1M dataset [Zheng et al., 2023a] exhibit pronounced skew and long-tail distributions. This imbalance makes fixed-width length binning highly skewed, thereby degrading classifier training accuracy. To address this issue, we adopt a quantile-based bucketing strategy that partitions output lengths using empirical percentiles computed from the training data, resulting in more balanced buckets. In M-LoRA, we set the number of buckets to K=5, yielding the ranges [0, p25), [p25, p50), [p50, p75), [p75, p99), and [p99,∞).

LoRA-based fine-tuning. Beyond the within-model skewness above, output length distributions also vary widely across LoRA models. As a result, a fully shared predictor struggles to fit all target models simultaneously, leading to degraded classification accuracy. To address this issue, MLLP adopts a shared-backbone, model-specific design.

• (a) Training structure offline.
• (b) Inference structure online.

Figure 5: The structure of the Multi-LoRA length predictor.

Specifically, we use a lightweight pretrained language model (LLaMA-3.2-1B) as the shared backbone to extract general textual representations, and train a lightweight LoRA adapter together with a classification head for each target model. Compared with fully shared predictors used in prior works [Jin et al., 2023; Qiu et al., 2024; Stojkovic et al., 2024], our LoRA-based predictor offers two key benefits:

• Specialization with low overhead. Model-specific LoRA module captures per-model length characteristics while adding only a small number of parameters.
• Scalability. Supporting a new target model only requires training its LoRA adapter and classification head, without retraining the entire predictor.

Training and inference. MLLP consists of a training phase and an inference phase. Figure 5a shows the training structure. Since MLLP predicts length buckets, we train it as a multi-class classifier with cross-entropy loss. For each target LoRA model, we adopt a two-stage training procedure. In Stage 1, we freeze the shared backbone and optimize the corresponding LoRA adapter and classification head. In Stage 2, we freeze the LoRA adapter and fine-tune only the classification head to refine the decision boundary.

During inference (Figure 5b), we keep one shared backbone and store all model-specific LoRA modules and classification heads in GPU memory. For each request, MLLP activates only the LoRA module and heads for the requested LoRA model to predict the output-length bucket. Each model-specific adapter and classification head occupies less than 10MB, resulting in tolerable overhead.

3.3 Memory-aware Speculative Scheduler

To maximize concurrency and throughput under a fixed GPU memory budget, we propose Memory-aware Speculative Scheduler (MeSS), which formulates request scheduling as an optimization problem. We next define the problem.

Problem definition. Given a set of requests to be scheduled, each with an input length, a predicted output length, and the corresponding LoRA model and adapter size, our goal is to select the subset of requests that maximizes the batch size while satisfying GPU memory constraints.

Algorithm 1 Computing the Peak KV Cache for a Batch.

Input: Requests list: R; whether the request has been selected: X; prompt length, generated length and predicted

length of requests: Lin, Lgen, Lpred; Output: the peak pages required for the batch: Pmin 1: C ← [] 2: for r ∈ R do 3: C.append((r, Lpred[r] -Lgen[r])) 4: end for 5: Sort C in ascending order of the second element. 6: l ← the length of C 7: Pmin ← 0 8: for k = 0, ..., l -1 do 9: cur sum x ← 0, cur pages ← 0 10: for i = 0, ..., k do 11: cur sum x += X[Ci] 12: cur pages += (Lin[Ci] + Lgen[Ci]) × X[Ci] 13: end for 14: cur pages += (Lpred[Ck]-Lgen[Ck])×cur sum x 15: if cur pages > Pmin then 16: Pmin ← cur pages 17: end if 18: end for

ILP Formulation. We formulate request scheduling as an Integer Linear Programming (ILP) problem and solve it with an off-the-shelf solver. We adopt a simple but effective objective: maximizing the decoding batch size. Under a fixed GPU memory budget, a larger decoding batch typically increases concurrency and improves GPU utilization. We now describe the ILP formulation. Given the waiting requests Rwaiting and running requests in current batch Rrunning, with R = Rwaiting ∪ Rrunning, we formulate the problem as follows:

subject to:

We introduce a binary variable xi for each request i, where xi = 1 means request i is selected for scheduling; otherwise, it continues to wait. Constraint (2) represents the memory constraint, where P denotes the total number of pages of Unified Memory. Unified Memory can be decomposed into two parts: Pkv pages for KV caches and Padapters pages for adapters. The sum of Pkv and Padapters must be less than P to satisfy the memory constraints. The key challenge in modeling the memory constraint is to accurately estimate the memory required for the selected batch to complete decoding, as both the KV cache and LoRA adapters can change over time. We next describe how GPU memory is modeled.

KV cache modeling. Since we perform iteration-level scheduling, the memory allocated to shorter requests is released as soon as they finish decoding. As a result, simply summing request lengths within a batch can substantially overestimate the required KV cache space. Therefore, we design an algorithm to compute the peak KV cache memory

required for the batch to complete decoding. Equation (3) formalizes this minimum requirement.

where Lin, Lgen, and Lpred denote the input lengths, generated lengths, and predicted output lengths of the requests, respectively. C is the list of request indices sorted by the remaining output length, calculated as Lpred - Lgen.

Algorithm 1 implements Equation (3) by simulating KVcache usage as shorter requests complete and release their cache. The algorithm returns the peak KV cache usage during the simulation, which corresponds to the minimum KV cache capacity required for the batch to complete the decoding phase. Lines 1-5 construct and sort the list C in ascending order of remaining output length. Lines 7-18 compute KVcache usage after each request in C completes, recording the number of pages occupied by the batch at each point. Pmin denotes the minimum number of pages required for batched inference. In addition, Constraint (5) prevents preemption by enforcing xi = 1 for all currently running requests.

Adapters modeling. In batched inference with multiple LoRA models, the BGMV [Chen et al., 2024b] kernel groups requests by target adapters. Thus, the parameters of each adapter need to be stored only once in GPU memory. We introduce a binary variable yj for each adapter j ∈ {1, . . . ,M}, where yj = 1 if adapter j is required by the selected batch and yj = 0 otherwise. Let M denote the number of adapters. The adapter memory cost is modeled as follows:

where S lists adapter sizes in pages under Unified Memory. Let A be a matrix {0, 1}N×M that describes the relationship between requests and adapters. Aij = 1 if request i uses adapter j, and Aij = 0 otherwise. Constraint (7) ensures that if request i is selected, adapter j is either already in HBM or will be loaded. Constraint (8) ensures that each adapter is stored at most once in HBM. Equation (6) can compute the total number of pages required by the selected adapters.

The complexity of the solver. We use an off-the-shelf solver [Google, 2024] to solve the ILP. Table 1 reports the solver runtime under different queue lengths. We run MeSS asynchronously on the CPU to overlap scheduling with the GPU decoding. To fully hide the scheduling latency, we restrict each solving step to the top 64 requests in the queue, which keeps the solver time below 30ms, while decoding typically takes 35-40ms per iteration. Based on this constraint, we prioritize requests in the request pool as follows:

where prompt len is the prompt length and predicted len is the length predicted by MLLP. Intuitively, shorter requests are prioritized to admit larger batches under the memory budget, while dividing by waitingtime mitigates starvation. After ranking by p, we select the top 64 requests as scheduling candidates to reduce solver time while maintaining fairness.

3.4 RecoveryMechanism

In this section, we design a recovery mechanism to handle output-length mispredictions. Mispredictions fall into two cases: (1) When the predicted length is longer than the actual output, M-LoRA over-reserves KV-cache memory, but the request can still complete decoding normally; (2) When the predicted length is shorter than the actual output, the request may be interrupted due to insufficient memory reservation, potentially degrading generation quality. Therefore, our recovery mechanism focuses on the second case to avoid affecting text generation quality.

RecoveryMechanism. During decoding, M-LoRA continuously tracks the execution state of each in-flight request. If a request reaches its predicted output length but has not generated the EOS token, M-LoRA triggers recovery: it removes the request from the current batch, re-estimates its remaining output length based on the generated tokens, and re-enqueues it into the request pool for re-scheduling by MeSS. We provide two recovery strategies: recomputation and CPU-load:

• Recomputation. We append the generated tokens to the input and discard the KV cache. When the request is rescheduled, we recompute the KV cache via a prefill pass, which is faster than performing multiple decoding iterations.
• CPU-load. We first offload the KV cache of a mispredicted request from HBM to CPU memory. When the request is rescheduled, it reallocates HBM space and copies the KV cache back to the GPU, avoiding recomputation.

Automatic recovery with a cost model. Since the two recovery strategies rely on different data paths, they exhibit different performance characteristics. Recomputation rebuilds KV caches using GPU compute and writes to HBM via the GPU internal bus, whereas CPU-load transfers KV caches between CPU memory and GPU HBM over PCIe. We evaluate both strategies by measuring their recovery overhead across different sequence lengths. We find that the overhead of both strategies grows approximately linearly with sequence length. Accordingly, we fit the measured overhead of each strategy with a linear model. Given the sequence length of request i, we estimate the recovery overhead as follows:

where li represents the sequence length of request i, and frecomp and fcpu load are the linear functions for the two methods, used to predict the recovery overhead based on the sequence length. We obtain the cost model by offline profiling. During online serving, M-LoRA selects the strategy with the lower estimated cost to recover the request. The fitted functions are reported in Section 4.5.

4 Evaluation

4.1 Experiment Setup

Models. As shown in Table 2, we evaluated the LLaMA [Touvron et al., 2023a] series models, including LLaMA-27B and LLaMA-2-13B. We use several LoRA adapters for each LoRA model, with LoRA ranks ranging from 8 to 64.

Workloads. We generate workloads using LMSYS-Chat1M dataset [Zheng et al., 2023a], which contains 1M conversations from 25 real-world LLMs. To capture skewed LoRA popularity, we map LoRA adapters to these 25 models in a round-robin manner. We sample the request arrival pattern from the Chatbot Arena [Zheng et al., 2023b] dataset, which records user dialogues from 20 LLMs. For each experiment, we set the request rate R and the test duration D. We sample R×D requests and linearly scale the arrival times into (0, D].

Testbed. We evaluate LLaMA-2-7B on an NVIDIA RTX 4090 24GB GPU and LLaMA-2-13B on an NVIDIA A100 40GB GPU, using PyTorch 2.1 and CUDA 11.8.

Baselines. We compareM-LoRAwith three state-of-the-art LoRA serving systems.

• vLLM [Kwon et al., 2023]: vLLM uses PagedAttention to reduce memory fragmentation and statically partitions HBM for adapters and KV caches. It also supports prefix caching [Zheng et al., 2023c] and chunked prefill [Agrawal et al., 2023], and employs an FCFS scheduling strategy.
• Punica [Chen et al., 2024b]: Punica is a LoRAmodel serving system, which implements a custom CUDA kernel to support batch inference of multiple LoRA models. Punica uses the FCFS strategy to schedule user requests.
• S-LoRA [Sheng et al., 2023]: S-LoRA stores KV caches and LoRA adapters in Unified Memory and uses an adapter-first policy that prioritizes requests whose adapters are already resident in HBM.

4.2 End-to-end Performance

We first compare the end-to-end performance of M-LoRA with baselines on LLaMA-2-7B and LLaMA-2-13B models. In each experiment, we concurrently serve 20 LoRA models. The results are shown in Figure 6 and Figure 7. We use latency per token, throughput, and time-to-first-token (TTFT) as evaluation metrics. Following prior works [Yu et al., 2022; Kwon et al., 2023; Sheng et al., 2023], we define average latency per token as the sum of the end-to-end latency of each request divided by the total number of output tokens.

Across all test cases, M-LoRA consistently outperforms the three baselines in latency per token, throughput, and TTFT. Specifically, it reduces latency per token by 74.6%, 87.5%, and 85.6% over vLLM, Punica, and S-LoRA, respectively. In terms of throughput, M-LoRA achieves 1.4×, 1.6×, and 1.7× improvements over the same baselines. For the TTFT, M-LoRA reduces latency by 57.4%, 81.1%, and 79.2%, respectively. These results demonstrate the effectiveness of MeSS in improving latency per token and throughput. The significant TTFT reduction further shows that M-LoRA effectively reduces queuing delay.

4.3 Predictor Evaluation

We evaluate three predictor designs for multi-model prediction on LMSYS-Chat-1M and ShareGPT [ShareGPT, 2024]

• (a) Recomputation overhead.

(b) CPU-load overhead.

Figure 9: Overhead of two recovery strategies when serving LLaMA-2-13B on A100. The fitted lines show the cost functions of the two strategies, which intersect at token length 113.7.

datasets: a shared BERT predictor (BERT), MLLP with a BERT backbone (MLLP+BERT), and MLLP with a LLaMA3.2-1B backbone (MLLP+LLaMA). As shown in Table 3, MLLP+LLaMA achieves the best overall performance in terms of accuracy, latency, and throughput. These results demonstrate that the shared-backbone, model-specific design of MLLP better captures the output-length distributions of different LoRA models, leading to higher prediction accuracy and improved system performance.

4.4 Scheduling Strategies

We compare four scheduling strategies: (1) First-Come-FirstServe (FCFS). (2) The S-LoRA scheduling strategy. It prioritizes requests whose adapters are already in GPU memory. (3) Shortest Job First (SJF), which sorts requests by the predicted output length. (4) The MeSS scheduler described in Section 3.3. We evaluated the throughput of different scheduling strategies under varying request rates. As shown in Figure 8, MeSS demonstrated the best performance, improving throughput by up to 1.64×, 1.89×, and 1.12× over FCFS, S-LoRA, and SJF, respectively.

4.5 Efficiency Analysis

Automatic Recovery Mechanism. We evaluate the overheads of Recomputation and CPU-load across token lengths on an A100 GPU. As shown in Figure 9, both overheads grow approximately linearly with token length. Therefore, we fit them using linear functions frecomp(x) and fcpu load(x). The latter includes swap-in and swap-out costs for KV-cache

• (a) Reduction in queuing delay.
• (b) Scalability of M-LoRA

Figure 10: Efficiency of M-LoRA.

transfers between GPU and CPU. The automatic recovery mechanism selects the lower-cost strategy based on these functions and only needs reevaluation when the hardware environment changes. Among mispredicted requests, 81.4% use recomputation and 18.6% use CPU-load recovery.

Queuing delay. We compare the queuing delay of vLLM and S-LoRA. vLLM uses FCFS, while S-LoRA adopts an adapter-first policy that prioritizes requests whose target adapter is in HBM. As shown in Figure 10a, M-LoRA reduces queuing delay by 60.5% and 80.2% on average over vLLM and S-LoRA, respectively. Even with imperfect predictions, MeSS can still leverage estimated lengths to schedule requests effectively and reduce queuing delay.

Scalability. We evaluate the scalability of M-LoRA under varying numbers of LoRA adapters at 24 reqs/s. As shown in Figure 10b, S-LoRA and M-LoRA maintain stable throughput as the adapter count increases, whereas vLLM degrades significantly. This is because vLLM statically partitions memory between KV caches and LoRA adapters. As the number of adapters increases, requests spread across more adapters, increasing the number of adapters required during inference. The fixed adapter memory becomes insufficient, leading to degraded performance.

5 Conclusion

We present M-LoRA, a memory-aware multi-LoRA serving system that reduces queuing delay and improves throughput through efficient request scheduling. To handle unknown output lengths across diverse LoRA models, M-LoRA uses a model-specific length predictor to estimate the output length and KV-cache demand of each request, and applies online recovery to handle length mispredictions. To mitigate memory contention between growing KV caches and on-demand LoRA adapters, M-LoRA employs a memory-aware scheduling strategy for efficient GPU memory allocation under a fixed budget. Evaluation results show that M-LoRA reduces average per-token latency by 87% and improves throughput by up to 1.7× over state-of-the-art LoRA serving systems.

Acknowledgments

This work was supported in part by the National Key R&D Program of China 2024YFE0200800, the National Natural Science Foundation of China under Grants (62471055, 62406039, 62401080, U23B2001, 62321001, 62201072, 62101064, 62171057), the Ministry of Education and China Mobile Joint Fund (MCM20200202, MCM20180101), the Fundamental Research Funds for the Central Universities (2024PTB-004).