Budget-Aware LLM Quantization and Low-Rank Correction via Information-Guided Subspace Matrices

1 Introduction

Current large language models (LLMs) demonstrate exceptional performance in language and various tasks, but their inference process typically requires ample memory bandwidth, sufficient storage space, and optimized kernels. In contrast, edge devices and some general-purpose hardware have very strict limitations on memory and resources, making model compression after training a necessary step for practical deployment [Jaiswal et al., 2024]. Research on system deployment further emphasizes that memory and scheduling are major bottlenecks in the deployment of current large models [Dao et al., 2022], leading researchers to investigate whether high-accuracy compression methods can be achieved under specific resource constraints [Clements and Lao, 2024b].

At the moment, there has been a lot of advancement in model compression methods, including traditional ones like pruning, quantization, and low-rank decomposition. Pruning can remove redundancy [Frantar and Alistarh, 2023]; quantization can achieve relatively low-precision inference through calibration and optimization [Frantar et al., 2022; Dettmers et al., 2022b; Lin et al., 2024; Esser et al., 2020]; and low-rank methods reduce the number of parameters in the matrix structures of neural networks and Transformer models to lower hardware computation requirements [Aghajanyan et al., 2021; Trockman and Kolter, 2021; Hu et al., 2022]. But when there are billions of parameters, compression is naturally heterogeneous: some layers and subspaces stay strong even at low precision, while others lose a lot of performance when compressed [Frantar and Alistarh, 2023; Dettmers et al., 2022a; Xiao et al., 2023]. This lack of uniformity makes it impossible to compress layers and models evenly when there are stringent hardware budget limits, especially when trying to keep the quality of the text (e.g., perplexity) at a certain compression or quantization bit rate.

This paper adopts a global planning perspective: with a fixed compression space, where can adding one bit (or one additional low-rank capacity unit) bring the greatest improvement in model quality? To make this perspective workable, the following two points need to be addressed: (i) the importance needs to be expressed in a way that clearly conveys "what needs to be retained"; and (ii) the rating representation needs to be consistent with the quantification and low-rank correction methods to be applied. For matrix weights, singular value vectors provide a natural coordinate system for model weights, since degradation or recovery is usually concentrated in a small set of vulnerable directions in the singular subspace of the current matrix [Meng et al., 2025].

In this paper, we introduce BALANCER 1 , a post-training compression framework that treats compression as a global allocation problem under an explicit storage budget. Figure 1 demonstrates the high-level overview. Starting from transformer projection matrices, BALANCER collects calibration signals from activation statistics, reference gradients, and a diagonal curvature surrogate. These signals are then converted into information-guided subspace matrices that define a consistent coordinate-wise weighting for analyzing each projection in its singular subspace. In this subspace, BALANCER converts the signal matrix into gain curves over singular directions, which quantify the marginal quality recovery expected from preserving additional directions. A principled greedy allocator then distributes bits and ranks globally across the model under an explicit bits-per-parameter budget, accounting for both weights and associated metadata. Finally, BALANCER exports a single set of reconstructed weights, enabling standard inference without fine-tuning or custom kernels.

1 Code and experiment scripts are available at https://github.com/ sinuofan/IJCAI2026-BALANCER.

Our main contributions are summarized below:

• Budget-aware post-training formulation. We cast compression as a global allocation problem under a unified storage accounting that includes quantized integers and metadata, enabling direct control of average bits-perparameter.
• Information-guided subspace matrices. We construct a signal matrix from activation statistics, gradient significance, and a Hessian diagonal curvature surrogate, and transform it into information-guided subspace matrices for direction ranking in the matrix singular subspace.
• Global allocation of bits and ranks. We introduce a greedy allocator that distributes compression bits and low-rank capacity across the entire model to maximize marginal quality recovery under a target budget.
• Evaluation and edge deployment. We evaluate BALANCER on standard benchmarks and demonstrate deployability on Jetson Orin Nano (8 GB), achieving 0.780.92 s first-token latency and 1.1-1.3 tok/s steady-state throughput with 3.5-3.8 GB resident memory.

2 RelatedWork

Post-training quantization (PTQ) and calibration strategies. Transformer PTQ has progressed from calibration heuristics to structure-aware, low-bit regimes. Representative advances include INT8 with outlier handling [Dettmers et al., 2022a], second-order layerwise optimization [Frantar et al., 2022], activation-aware channel protection [Lin et al., 2024], activation smoothing for joint W+A quantization [Xiao et al., 2023], and efficient group-wise pipelines [Yao et al., 2022]. Recent works further push sub-4-bit operation via codebooks, orthogonalization, or lattice formulations [Tseng et al., 2024; Egiazarian et al., 2024; Eliseev and Mazur, 2023; Ashkboos et al., 2024; Zhang et al., 2024]. Beyond improving a single fixed bit-width, several concurrent directions target budget-adaptive or variable-memory deployment: BitStack enables any-size compression under changing memory envelopes [Wang et al., 2025], Any-Precision LLM targets low-cost deployment of multiple precision variants [Park et al., 2024], and Matryoshka Quantization trains nested quantizers that support multiple bit-widths from shared parameters [Ahemad, 2025]. These works motivate treating PTQ as a budgeted allocation problem rather than a per-layer independent choice [Chu et al., 2025; Hoang et al., 2026; Yao et al., 2024].

Hybrid quantization with structured residuals. To tighten the quality-compression tradeoff under stringent budgets, hybrid schemes keep a quantized backbone while adding small structured corrections where they are beneficial. A representative example is CALDERA, which couples lowbit quantization with low-rank residual structure for posttraining compression [Saha et al., 2024; Lee et al., 2025]. Related lines also study the co-design of quantization and adapter/low-rank components [Li et al., 2024; Lawton et al., 2024], serving-oriented low-bit designs [Zhao et al., 2024], and elastic precision scheduling [Lee et al., 2023; Kwon et al., 2022]. Compared with pure PTQ, these methods explicitly reserve extra capacity for structured residuals; compared with full mixed-precision, they preserve a simple backbone and concentrate extra parameters into compact correction paths. In particular, LoftQ also leverages lowrank structure around quantization, but is primarily designed to be LoRA-fine-tuning-aware, whereas our setting focuses on training-free post-hoc allocation under an explicit storage budget [Li et al., 2024].

Direction-aware protection and global allocation. A growing body of evidence indicates that compression sensitivity is anisotropic within layers. Hessian-guided sparsification [Frantar and Alistarh, 2023], magnitude/activation proxies [Sun et al., 2024], and block-wise second-order reconstruction [Li et al., 2021] all reveal direction-dependent sensitivity. At a coarser granularity, mixed-precision schedulers formalize bit-width assignment as a budgeted optimization so that precision is redistributed toward components with higher predicted gain per additional bit [Lee et al., 2023]. Hybrid schemes further combine a low-bit backbone with structured residuals, using spectral cues to decide where limited corrective capacity is applied [Saha et al., 2024]. Recent budgetadaptive methods (e.g., BitStack [Wang et al., 2025], AnyPrecision LLM [Park et al., 2024], Matryoshka Quantization [Ahemad, 2025]) reinforce the same theme: global allocation under an explicit memory/bit budget is central for practical deployment.

3 Method

Algorithm 1 summarizes the end-to-end pipeline. All decisions are computed deterministically on a small calibration set, the subspace matrices, and a single bpp constraint (Eq. 3), with no hyperparameter tuning.

3.1 Setup

Notation. We compress a pretrained LLM with linear projection weights {Wℓ}Lℓ=1 . For module ℓ, Wℓ ∈ Rmℓ×nℓ maps an input activation x ∈ Rnℓ to an output y ∈ Rmℓ via y = Wℓx. Given a chosen bit-width bℓ ∈ B and rank rℓ ∈ {0, 1, . . . , rmax}, the exported weight is

where ̂ Wℓ,bℓ is the row-wise quantized reconstruction defined in Sec. 3.2.

Low-rank. We store the low-rank term in factored form

Budget accounting. Let N ≜ ∑Lℓ=1 mℓnℓ be the total number of weight entries across all compressed matrices. Let B be the target bits-per-parameter (bpp), so the total budget is BN bits. For module ℓ, the integer storage is mℓnℓbℓ. We store one FP16 scale per row, so Cmeta(mℓ, bℓ) = mℓbs with bs = 16. The low-rank factors are stored at blr bits, so Algorithm 1 BALANCER: global allocation with information-guided subspace matrices Require: {Wℓ}Lℓ=1 , calibration set D, budget B, bit set B, rank limit rmax; optional {Gℓ}Lℓ=1 , optional {hℓ}Lℓ=1

Ensure: {˜ Wℓ}Lℓ=1

1: for ℓ = 1 to L do

2: (S(x)

3:

for each b ∈ B do

• 5: Rℓ,b ←Wℓ -̂ Wℓ,b

• 8: end for
• 9: end for

11: for ℓ = 1 to L do

• 13: ˜ Wℓ ← ̂ Wℓ,bℓ +∆ℓ
• 14: end for
• 15: return {˜ Wℓ}

Clr(mℓ, nℓ, rℓ) = (mℓ + nℓ)rℓblr. In all main experiments, blr = 4. We enforce

(3)

Thus Cmeta accounts for per-row scales, and Clr accounts for the stored factors Lℓ and Rℓ.

3.2 Row-wise symmetric quantization

For W ∈ Rm×n and bit-width b ∈ B, define the signed integer range

For each row i ∈ {1, . . . ,m}, define the scale

with a fixed εq > 0 to avoid division by zero for all-zero rows. The quantized integers and reconstructed weights are

where ⌊·⌉ denotes rounding to the nearest integer and clip(·) clips elementwise.

3.3 Information-guided subspace matrices

Let D be a calibration set and x ∈ Rn be the input activation observed on D. We construct two nonnegative diagonal matrices Dx and Dy (equivalently, their diagonal entries S(x) and S(y) ), referred to as information-guided subspace matrices. When available, we denote by Gℓ the reference-gradient

ℓ ) ← SUBSPACE(Wℓ,D;Gℓ, hℓ) (Alg. 2)

ℓ , S(y)

matrix associated with module Wℓ, and by hℓ the corresponding nonnegative diagonal curvature surrogate. We construct information-guided subspace matrices by aggregating three complementary subspace signals that act on the input/output coordinates: (i) an activation subspace view from the second moment of x, (ii) a reference-gradient subspace view from G, and (iii) a curvature subspace view from the diagonal surrogate h.

1. Activation RMS (root second moment).

(2) Reference-gradient magnitude. When a reference model is available, we collect a gradient matrix G ∈ Rm×n for the same projection w.r.t. a reference objective (e.g., a distillation-style loss), and define

(3) Diagonal curvature surrogate. For projection Wℓ, we use hℓ,i = E(x,y)∼D ∑nℓ j=1 (∂Lref/∂Wℓ,ij)2 as the outputside curvature surrogate.

Information-guided subspace matrices. For any nonnegative vector u ∈ Rd≥0 , define min-max normalization

where ε > 0 is a fixed numerical constant. We combine available components by summation followed by Norm(·), hence no hyperparameters are introduced.

Let IG ∈ {0, 1} indicate whether G is available, and Ih ∈ {0, 1} indicate whether h is available. We define

Min-max normalization aligns signal scales, summation avoids tunable fusion weights, and the 1+ shift makes S(x) and S(y) positive bounded weights. Algorithm 2 summarizes the construction of (S(x) , S(y) ).

3.4 Subspace gain curves

For any residual R ∈ Rm×n , define

This weighting emphasizes residual components aligned with informative input/output directions encoded by (S(x) , S(y) ). Intuitively, it penalizes distortion more heavily along directions that are deemed important by calibration-time signals, making the subsequent gain curves direction-aware under a unified criterion. Equivalently, E(R) measures reconstruction error after re-scaling columns and rows according to calibration signal strengths, so directions deemed more important incur a larger penalty when distorted.

Algorithm 2 SUBSPACE: information-guided subspace matrices construction

Require: W ∈ Rm×n , calibration set D; optional G ∈ Rm×n ; optional h ∈ Rm

Ensure: S(x) ∈ Rn , S(y) ∈ Rm

• 2: IG ← 1 if G exists else 0; Ih ← 1 if h exists else 0

3: if IG = 1 then

• 5: end if

8: return S(x) , S(y)

Scaled SVD. Fix b ∈ B and let Rb ≜W -̂ Wb. Define the weighted residual

Let Mb = UΣV ⊤ denote the (standard) singular value decomposition (SVD), where U = [u1, . . . , um] and V = [v1, . . . , vn]. Let the singular values satisfy σb(1) ≥ σb(2) ≥ · · · . For rank r, define the truncated approximation

We map it back to a low-rank correction for the original residual:

In practice we store ∆(r) b in factored form by letting

where Ur = [u1, . . . , ur], Vr = [v1, . . . , vr], and Σr = Diag(σb(1), . . . , σb(r)). Then ∆(r) b = L(r) b (R(r) b )⊤, matching Eq. 2. Allocating the t-th singular component reduces the error in Eq. 13 by

which yields the gain curve used by the allocator. No additional coefficients are introduced beyond (S(x) , S(y) ).

3.5 Budget-aware global greedy allocation

We choose {(bℓ, rℓ)}Lℓ=1 by a global greedy allocator that maximizes reconstruction value per additional bit.

Coupling bit-width and rank. The correction is always computed on the quantization residual Rℓ,b = Wℓ - ̂ Wℓ,b, hence the gain curve gℓ,b(t) is specific to the current bit-width b. To keep allocation well-defined and training-free, we restrict bit-width changes to be applied at rℓ = 0 and reset rℓ ← 0 whenever bℓ changes. After a bit-width update, subsequent rank actions for that module are taken from the precomputed gain curve of the new bit-width. A rank-one correction costs (mℓ + nℓ)blr bits, whereas increasing the backbone by one bit costs mℓnℓ bits, so the allocator compares these actions by the same density ρ = ∆e/∆c.

Value terms. For module ℓ and bit-width b, let

For a rank increment at fixed b, the marginal value is gℓ,b(t) from Eq. 18.

Rank-r weighted residual error. For module ℓ at bit-width b and rank r, let

With the weighted SVD in Sec. 3.4, this admits the closed form

where eℓ(b) is Eq. 19 and gℓ,b(t) = σℓ,b(t)2 is Eq. 18.

Bit costs. For a bit-width change b → b′ on module ℓ, we use

For a rank increment r → r + 1 on module ℓ, we use

These costs are consistent with the single constraint in Eq. 3. Actions and densities. Wemaintain a state (bℓ, rℓ) for each module and consider two action types.

Total module cost. For module ℓ, define the total bit cost under choice (b, r) as

For brevity, write cb,r ℓ ≜ cℓ(b, r) and Eb,r ℓ ≜ Eℓ(b, r).

We define the error reduction ∆e and bit cost ∆c for each action as

Each candidate action is scored by density ρ = ∆e/∆c, and we greedily apply the best feasible action under the global budget. Algorithm 3 provides the full allocation routine.

4 Experiment

4.1 Experimental Setup

We evaluate BALANCER to validate its effectiveness and practicality. We report (i) main results at matched budgets versus state-of-the-art baselines, (ii) ablation studies of key design choices, (iii) flexibility under mixed-precision schedules, and (iv) deployability on resource-constrained hardware. Unless otherwise stated, "mean±SD" reflects variability over different calibration-set samplings (and corresponding allocation/quantization outcomes) with the same evaluation protocol.

Algorithm 3 ALLOCATE: budget-aware global greedy allocation

Require: Precomputed {eℓ(b)}, {gℓ,b(t)}; costs Cmeta,Clr; budget B; bit set B; total parameters N

Ensure: Choices {(bℓ, rℓ)}Lℓ=1

• 1: bℓ ←min(B), rℓ ← 0 ∀ℓ

• 2: bits ← ∑ℓ (mℓnℓbℓ + Cmeta(mℓ, bℓ)) {initial cost}
• 3: Initialize max-heap H
• 4: {Each heap element is an action a = (ℓ, type) with density ρ(a) = ∆e/∆c computed from Eq. 25.}
• 5: {PUSHRANK/PUSHBIT insert the next feasible rank/bit action for module ℓ; REFRESH recomputes actions for the updated module.}
• 6: for ℓ = 1 to L do
• 7: PUSHRANK(ℓ); PUSHBIT(ℓ)
• 8: end for
• 9: while H not empty do
• 10: Pop action a with largest density ρ
• 11: if bits + ∆c(a) > BN then
• 12: continue
• 13: end if
• 14: Apply a; bits ← bits +∆c(a)
• 15: REFRESH(ℓ(a)) {recompute next actions for affected module}
• 16: end while
• 17: return {(bℓ, rℓ)}Lℓ=1

Models and Datasets. Our primary evaluations are conducted on the LLaMA model family, specifically LLaMA-27B and LLaMA-3.1-8B. To demonstrate the generalizability of our method across different architectural families, we also verify the same trend on Mistral-7B, where BALANCER reduces WikiText-2 PPL from 5.58 (RTN Q4) to 5.35 at 4.0 bits. Perplexity (PPL) is the primary metric for language modeling quality, reported on WikiText-2 and C4.

EvaluationMetrics. We report "Bits" as the left-hand side of Eq. 3 divided by N, and CR as 16/Bits relative to FP16, and runtime performance including steady-state throughput ("Tok/s"), first-token latency ("Lat"), and resident VRAM.

Baselines. We compare toRTN (uniform round-to-nearest), GPTQ [Frantar et al., 2022], AWQ [Lin et al., 2024], CALDERA [Saha et al., 2024], LoftQ [Li et al., 2024], Atom [Zhao et al., 2024], and FlexQuant [Liu et al., 2025] under their recommended settings. The data-dependent signals for BALANCER are derived using a small calibration set (64 sequences from WikiText-2 and 1000 from C4). Teacher-guided gradients. Unless otherwise stated, we use the same FP16 base model as the teacher (self-teaching) and compute reference gradients with respect to a reconstructionstyle objective (matching the teacher logits on a small mixture of instruction corpora such as Alpaca [Taori et al., 2023], Dolly [Conover et al., 2023], and OpenAssistant [K¨opf et al., 2023]). This keeps profiling reproducible without requiring a larger external teacher; using a stronger teacher is compatible with the same profiling pipeline.

4.2 Main Results

BALANCER Configurations. We evaluate two configurations: (i) BALANCER (Q4+LR), our main setting at the 4-bit operating point with a uniform 4-bit backbone and globally allocated low-rank corrections; and (ii) BALANCER (Q-Mix), a high-compression variant that uses a mixedprecision backbone (e.g., 2-3 bits for less sensitive MLP projections) to reach tighter budgets (e.g., 3.6 bits). We report the main operating points in Tables 1-5.

At 4-bit average budgets, BALANCER consistently attains lower perplexity than uniform quantization (RTN) and remains competitive with or superior to other strong posttraining baselines. We highlight two key configurations from our results in Table 1. Our primary configuration, BALANCER (Q4+LR), is designed for a direct and fair comparison at the standard 4-bit operating point. To verify that BALANCER's quality improvements do not increase runtime cost, we compare BALANCER (Q4+LR) against the RTN Q4 baseline under identical kernels and settings. Table 3 shows matched throughput and VRAM, and latency that is comparable or lower. On LLaMA-2-7B, for instance, it successfully recovers over 50% of the perplexity gap between the uncompressed FP16 model and the naive round-tonearest (RTN) baseline. On the other hand, the BALANCER Q-Mix variant demonstrates the flexibility of our framework, pushing compression further to 3.6 bits with only a mild PPL increase.

Downstreamevaluation. In addition to perplexity and runtime, we report downstream accuracies to verify that budget allocation preserves task-level quality. We separate downstream results from the compression table for readability: Table 1 summarizes budget, perplexity, and runtime, while Table 2 reports accuracies on HS, WG, PIQA, ARC-E, ARCC, BoolQ, and RTE under the same compressed checkpoints. Overall, BALANCER remains competitive with strong posttraining baselines at the standard 4-bit operating point, and the mixed-precision configuration (Q-Mix) provides an additional operating point under tighter budgets while maintaining comparable task-level quality.

4.3 Performance under Different Budgets

We next analyze performance under different budgets on LLaMA-3.1-8B. Table 1 shows that as the budget tightens from 4.0 to 3.2, BALANCER degrades smoothly, unlike typical uniform schemes. At 4.0 bits, BALANCER (Q4+LR) closes the gap to FP16 more effectively than RTN. At 3.6 bits, BALANCER (Q-Mix) remains competitive with 4-bit baselines while increasing CR to 4.44×. Importantly, the results confirm that the performance decline of BALANCER is gradual and predictable: each reduction in budget leads to a proportional increase in perplexity, rather than sudden degradation. This demonstrates that the allocator consistently invests capacity in the most fragile regions, ensuring that the model remains usable even at aggressive compression levels. Signal ablations show that the full activation-gradientcurvature construction gives the lowest PPL, indicating that curvature provides complementary output-side sensitivity.

4.4 Mixed-Precision Analysis

While BALANCER is strong with a uniform 4-bit backbone, its global allocation is particularly well-suited for mixed-precision schedules. Fixing the budget to 3.6 bits for LLaMA-3.1-8B, we vary precision between Attention and MLP. The results are shown in Table 4. A balanced allocation, our BALANCER (Q-Mix) configuration, performs best at this budget.

4.5 Deployment Results

A core objective of BALANCER is to enable compressed models that retain quality while being practical on constrained hardware [Clements and Lao, 2024a; 2024b]. We therefore evaluate on a real-world edge device, the Jetson Orin Nano (8GB), and compare directly to the standard Q4 (RTN) baseline. Unless otherwise stated, we benchmark Jetson Orin Nano in the default 15W power mode and use TensorRT-LLM for inference with standard GEMM kernels. Importantly, BALANCER does not require custom CUDA kernels: it exports a single reconstructed checkpoint with the same runtime interface as uniform PTQ models, so deployment uses an unchanged serving stack.

On-Device Performance. As shown in Table 5, BALANCER achieves clear deployment gains. On LLaMA-2-7B, our Q4+LR variant improves throughput by 18% and reduces first-token latency by 14%, while also lowering memory usage by about 15%. On LLaMA-3.1-8B, the Q-Mix variant shows similar advantages, reaching 22% higher throughput and 15% lower latency. These improvements suggest that preserving fragile directions can improve both quality and practical deployability under the same budget.

5 Conclusion

We introduced BALANCER, a training-free framework that reframes post-training compression as a global, budget-aware allocation problem. The core innovation of BALANCER is converting calibration-time signals (activations, reference gradients, and a diagonal curvature proxy) into informationguided subspace matrices, which induce direction-wise gain curves and enable global budget allocation for mixedprecision quantization and low-rank correction under an explicit bpp budget. This map guides a global allocator that strategically preserves critical singular directions with low-rank corrections. Our experiments on LLaMA-2-7B and LLaMA-3.1-8B demonstrate that BALANCER achieves consistently strong performance among training-free posttraining compression methods under matched budgets, while remaining deployable on resource-constrained edge GPUs.

Acknowledgments

This research was supported in part by the National Science Foundation (NSF) SaTC-2426299 and SaTC-2413046.