SeGO: Sensitivity-Aware Golden Optimization for Large-Scale VLM Quantization

1 Introduction

Recently, large Vision-Language Models (VLMs) have shown excellent performance across various tasks [Brown et al., 2020; Yu et al., 2025]. However, their massive parameters, such as up to 72B for InternVL2 [Chen et al., 2024c], make deployment on resource-constrained devices extremely difficult. Post-Training Quantization (PTQ) has become a key method to reducing memory usage and accelerating inference without retraining [Shao et al., 2023; Yuan et al., 2023; Zhou et al., 2025].

Although PTQ has succeeded in Large Language Models (LLMs) such as SmoothQuant [Xiao et al., 2023], SpinQuant [Liu et al., 2024d], FlatQuant [Sun et al., 2024], Atom [Zhao et al., 2024], existing researches indicate that the introduction of the visual modality leads to extreme heterogeneity with respect to input feature distributions [Luo et al., 2024; Gong et al., 2024]. Traditional PTQ methods often assume uniform sensitivity across layers and use global heuristic searches. They fail to handle complex cross-modal interactions in VLMs, leading to severe quantization accuracy drops at low bits. Furthermore, as the parameter amount of model increases, systematic outliers emerge in Transformer layers [Dettmers et al., 2022], hence the performance degradation of traditional uniform quantization strategies becomes more pronounced, further exacerbating the difficulty of quantizing large-scale VLMs.

To address the problem of quantization accuracy degradation, some gradient-based PTQ methods such as OmniQuant [Shao et al., 2023], SignRound [Li et al., 2024c], QLLM [Liu et al., 2024b], PB-LLM [Shang et al., 2024b], NormTweaking [Li et al., 2024b] recover quantization accuracy through backpropagation, but their high memory cost and hours of calibration time become prohibitive as model size grow. While GPU cluster optimizations [Li et al., 2025] address hardware bottlenecks, algorithmic efficiency remains vital for VLM deployment. Moreover, from the perspective of model quantization coverage, most existing works [Zhou et al., 2024; Lin et al., 2024b; Li et al., 2024d; Chen et al., 2024b] focus on compressing the LLM while keeping the Vision Transformer (ViT) in FP16 format. This is mainly because, in early small-scale VLMs, the number of parameters in ViT (typically around 300M) is much smaller than that of LLMs. And visual inputs are more sensitive to quantization, the severe quantization accuracy degradation caused by quantizing the ViT is disproportionate to the limited memory reduction obtained. However, modern VLMs use much larger ViT. For instance, InternVL2-26B features a 6B-parameter ViT [Chen et al., 2024a], which alone consumes more memory than many 7B-class LLMs. Under this trend, quantizing only the LLM will make the ViT the new resource bottleneck, hindering the model's deployment on edge devices. Therefore, achieving high-precision deployment of

(a) Universal Structural Sensitivity Asymmetry across Modalities

full-model VLMs remains an urgent challenge. This is critical for the realization of 6G endogenous AI networks, where edge-cloud collaboration is essential for distributing intelligence across the network [Wang et al., 2025].

To solve these problems, this paper first conducts an indepth analysis of the sources of VLM quantization errors. We categorize the linear layers in VLMs into two types:

• Expansion Space: Map input hidden states to a highdimensional space. Their output usually connects to nonlinear activation functions, leading to highly sparse distributions of activation values and significant outliers.
• Projection Space: Aggregate high-dimensional features back to the model's hidden dimension. It is more robust to quantization error.

We analyze the error propagation mechanism within the Transformer blocks of the full multimodal model. Under the low-bit quantization configuration, we perform a layerby-layer sensitivity analysis on the 48-layer Transformer of the InternVL2-2B model (with the first 24 layers as the ViT encoder and the last 24 layers as the LLM encoder). In each layer, we test the layer-wise output errors generated by "quantizing only the Expansion Space" and "quantizing only the Projection Space". As shown in Fig. 1(a), the red line and its shaded area show the relative error increase percentage caused by quantizing the Expansion space alone, compared to quantizing the projection space alone at the current layer. Our analysis of the InternVL2-2Bmodel shows that quantization errors are dominated by the Expansion Space across both the ViT and LLM. Since the InternVL2 series models use the most typical Transformer architecture, the structural sensitivity asymmetry observed in these models are also applicable to other VLMs based on the Transformer architecture. We extend the same experiments to other variants of InternVL2 and the LLaVA series, the findings are consistent with the results illustrated in Fig. 1(a). This indicates that the Expansion Space is the key bottleneck causing the degradation in VLM quantization accuracy.

To validate this finding, we further conduct single-module perturbation experiments on different models. We observe same structural sensitivity asymmetry across various quantization configurations. The results shown in the Fig. 1(b), which correspond to quantization at 4 bits, demonstrate that quantizing only the Expansion Space to 4 bits while keeping the Projection Space in FP16 results in accuracy that is very close to full quantization. Quantizing only the Projection Space to 4 bits while keeping the Expansion Space in FP16 almost restores the accuracy to the FP16 level. This phenomenon suggests that the quantization of the Expansion Space is the main factor contributing to the accuracy degradation, while the Projection Space exhibits strong robustness. This suggests that sparse intervention focusing on the Expansion Space is sufficient to restore accuracy, thereby saving the overhead of applying scaling protection to all linear layers. Considering that VLM activations change rapidly with text token and visual token and compared to the static distribution of weight features, forcibly quantizing the activations leads to a significant drop in the accuracy of VLMs [Jiang et al., 2024; Shang et al., 2024a], we focus on weight-only quantization and treat activation quantization as complementary future work.

Based on the above insights, we propose SeGO, a unified structural sensitivity-aware sparse optimization framework. Specifically, for the LLM encoder and ViT encoder, it efficiently constructs the search interval for scaling factors based on static weight statistics to exclude invalid solutions. Building upon this, we design a unified golden-section solver, leveraging its linear convergence property to speedup the search of optimal scaling factors for the Expansion Space. By only scaling the sensitive Expansion Space while directly quantizing the Projection Space, SeGO achieves efficient and consistent compression for the entire model. Our main contributions are:

• We reveal structural sensitivity asymmetry across modalities and systematically demonstrate for the first time that in the LLM encoder and ViT encoder of VLMs, the Expansion Space is a key bottleneck for quantization error.
• We propose the SeGO framework: we develope a unified parameter-solving strategy for the entire model. For the LLM encoder and ViT encoder, we innovatively use static weight statistics to construct the search interval for scaling factors.
• We introduce a Golden-Section Solver to speedup the search of optimal scaling factors without requiring gradients. It address the cross-modal optimization unification problem and significantly reduce computational overhead.

2 Preliminaries and Analysis

2.1 Definition and Symbolic Representation

In the VLM Transformer architecture, we define the weight matrix set as Wm type , where m ∈ {LLM,ViT} and type ∈ {exp,proj}.

Expansion Space(Wexp): lt is responsible for mapping the input features to a higher-dimensional space and the output is typically directly connected to a nonlinear activation function. WLLM exp includes the projections of Q, K, and V in the Self-Attention and the Gate and Up projections in the MLP module; WViT exp includes the projections of q, k and v in the Multi-Head Attention and the first fully connected layer of the FFN. Due to the" dimensional Expansion + activation" operation, the weight distribution in the Expansion Space often has a large dynamic range and outliers.

Projection Space(Wproj): It is responsible for aggregating high-dimensional features back to the backbone dimension. WLLM proj /WViT proj corresponds to the Output projection in each Attention and the Down projection in the MLP. Since the Projection Space performs dimensionality reduction and Down projection in the MLP. Since the projection space performs dimensionality reduction and weighted summation, it has a smoothing effect on the introduced quantization errors.

2.2 Quantization Basis

This paper aims to address the PTQ problem of the entire VLMs. Based on the definitions in Section 2.1, we focus on the quantization of the Expansion Space weights (Wm exp , m represents the LLM and ViT). First, we define the per-channel maximum absolute value vector of the weight matrix vm ∈

RCin , where the c-th element is:

This statistic reflects the weight intensity distribution of the input channels in the Expansion Space across different modalities. We quantize the weights to b-bit integers, and the quantization function is represented as:

Where ∆ is the quantization step and z is the zero point. Our goal is to minimize the reconstruction error, as formulated in Eq.(3), under the given low-bit width. Since the structural sensitivity of Wm exp is higher than Wm proj , our optimization focus will be entirely on the Wm exp .

2.3 Mitigating Errors through Channel-Wise Scaling Equivalent

To reduce above error, we introduce a channel-wise scaling factor α for the Expansion Space Wm exp , We reorganize the weight distribution through mathematical equivalence:

The quantized output error e after transformation satisfies is:

Where E = Q(Wm expα) - Wm expα represents the scaled weight error and Q means the quantization function. By carefully designing the scaling factor α, we can redistribute the weight's range, suppress truncation errors, and reduce rounding errors. From this derivation, we conclude that: by specifically optimizing the scaling factor for the Expansion Space, we can achieve precise protection of the structurally sensitive layers.

3 SeGO Framework

This section provides a detailed introduction to the SeGO framework and the overview is shown in Fig. 2. Based on the previously revealed structural sensitivity asymmetry, SeGO aims to achieve efficient quantization for the entire model through sparse intervention in the Expansion Space. We construct a unified optimization paradigm for the entire model. First, we use static weight statistics to construct the search interval for scaling factors. Next, within the determined search interval, we efficiently locate the optimal scaling factor using the golden-section solver. Finally, we perform unified quantization to both the Expansion Space and Projection Space.

3.1 Unified Scaling interval Initialization Based on Weight Magnitudes

Since the Expansion Space maps input features to a higherdimensional space, its weight distribution often exhibits significant channel-wise variations. These variations are the cause of quantization errors. To construct an efficient and unified optimization framework, we use the statistical characteristics of multimodal model weights to initialize the scaling factors.

For both LLM and ViT Expansion Spaces weight Wexp, we calculate the channel-wise max absolute value vm c as formula(1). This metric directly reflects the weight intensity of the channel in the network transformation. Based on vm c , we define the initial scaling factor αinit c for each channel. Our goal is to introduce the scaling factor to reduce the values of channels with large weights while equivalently amplifying the input features, thereby balancing the numerical range across channels. The αinit c is calculated as follows:

Where ∥v∥2 is the L2 norm of the channel magnitude, used to normalize the relative proportion between channels; λ is a hyperparameter controlling the overall balance strength; αmin and αmax are the preset lower and upper bounds for the scaling factor.

Although the scaling factors are initialized based on the statistical characteristics of the weights, the optimal solution may deviate from this static estimate. Therefore, we restrict the subsequent search to a local neighborhood centered [αinit c - δ, αinit c + δ] around αinit c , providing a highquality starting point for fine-tuning the optimization for specific modal objectives.

3.2 Scaling Factor Solver Based on the Golden-Section

We need to find the optimal α∗ to minimize the reconstruction error. This problem can be formalized as an optimization task to find the extremum of a univariate function within a continuous, bounded interval. Such problems typically have two key characteristics: First, the objective function often exhibits unimodality within a reasonably initialized interval; second, the optimization process needs to be completed with minimal computational overhead. To verify the critical assumption that"the loss function exhibits unimodal characteristics within the search interval", we analyze the Expansion Space at different depths in the InternVL2-8B model. For each channel's scaling factor optimization, we define a modalityweighted loss function. Previous studies have shown that the token of the text modality is typically more sensitive to quantization errors, and it often dominates the overall error [Luo et al., 2024; Gong et al., 2024]. Therefore, we split the output error into text and visual components and define the output reconstruction error as follows:

Where yfp and yq represent the full-precision and quantized output activation values. The subscripts t and v refer to the output corresponding to text tokens and visual tokens. βt and βv are the weighting coefficients used to balance the importance of different modalities, and ∥ · ∥1 denotes the L1 norm, which is more robust to outlier errors.

For each channel, we perform grid sampling with a finegrained step size of 0.05 within its determined search interval. As illustrated in Fig. 3, the reconstruction error curves exhibit a pronounced unimodal shape (convex-like) with unique minimum points. Moreover, the region around this minimum point shows smooth variation, indicating good local convexity.

For this typical scenario of optimizing a unimodal function within a limited interval, there exists a class of efficient and stable interval contraction algorithms in the field of numerical optimization. We adopt the golden-section method, one of the most representative algorithms, as the core solver. The advantage of this method is that it does not require derivative information and only uses function value comparisons to shrink the search interval, avoiding the memory and computational overhead caused by backpropagation. Each iteration reduces the interval by a fixed proportion, ensuring linear convergence and achieving a high-precision solution with very few evaluations.

Specifically, for each channel, we use the golden-section ratio φ = √5-1 2 ≈ 0.618 to iteratively shrink the search interval [a, b] which is determined by Section 3.1:

1. Probe Point Generation: Compute two internal probe points:

1. Interval Reduction: Compare Lrec(x1) and Lrec(x2), whose calculation method is shown in Formula 6. If

Lrec(x1) < Lrec(x2), update the new interval as [a, x2], otherwise, update it as [x1, b].

3. Convergence Check: Repeat the above steps until the interval width is smaller than a pre-set convergence tolerance ϵ (for example ϵ = 10-2 ), which controls the precision of the optimal solution.

This solving process fully leverages the unimodal characteristic of the loss function within the interval. It efficiently locates the optimal scaling factor with logarithmic complexity O(log(1/ϵ)). And it perfectly meets the comprehensive requirements for precision and efficiency in VLM quantization.

3.3 Unified Quantization and Efficiency Analysis

After finding α∗ , SeGO enters the final execution stage. We apply a unified mathematical transformation to reorganize the model weight. For each Expansion Space, we first adjust the weight distribution using the scaling factor and incorporate the inverse scaling factor into the preceding layer's parameters. This aims to maintain mathematical equivalence with the inference results. Subsequently, we quantize the weights of all linear layers (including the reorganized Expansion Space and the untreated Projection Space) to the target bit-width. This "structure alignment first, then unified quantization" paradigm ensures that the quantized model can be directly deployed on existing high-performance inference kernels without modifying any underlying operators.

Time Complexity: Thanks to the weight statistics-based initialization strategy proposed in Section 3.1, SeGO does not require data forward propagation during this phase. The search interval can be constructed using static weight statistics. Furthermore, the golden-section solver leverages its linear convergence property, requiring only a few iterations within the determined search interval to locate the optimal solution.

Space Complexity: The peak memory usage of SeGO is only slightly higher than that of model inference, allowing it to easily calibrate models with billions of parameters on consumer-grade GPUs.

4 Experiments

4.1 Experimental Setup

Datasets and Models. To activate both visual and linguistic capabilities during calibration, we construct a set of 128 image-text pairs randomly sampled from MS COCO 2017 [Lin et al., 2014] and ShareGPT4V [Chen et al., 2023]. For a comprehensive evaluation, we select two VLM series across various scales: InternVL2 [Chen et al., 2024c] (1B, 2B, 8B, 26B) and LLaVA (onevision-7B [Li et al., 2024a] and v1.5-7B [Liu et al., 2024a]). We evaluate performance via the LMMs-Eval [Zhang et al., 2024] framework on diverse benchmarks, including OCRBench (text recognition) [Liu et al., 2024c], MMMU val (multi-discipline reasoning) [Yue et al., 2024], RealWorldQA (real-world QA) [Fu et al., 2024], and AI2D (diagram reasoning) [Kembhavi et al., 2016].

Baselines and Configuration. We compare SeGO with three mainstream PTQ methods: (1) RTN, the basic roundto-nearest quantization; (2) AWQ [Lin et al., 2024a], which utilizes channel-wise equalization to suppress outliers; and (3) GPTQ [Frantar et al., 2023], a second-order approximation method based on the Hessian matrix. For a fair comparison, all methods adopt weight-only, asymmetric group-wise quantization (group size 128). We conduct evaluations under both W3A16 and W4A16 bit-width configurations.

4.2 Main Results

We evaluate SeGO against FP16 and mainstream PTQ baselines under the unified settings. Results for the InternVL2 series are summarized in Table 1-2, while results for LLaVAonevision-7B and LLaVA-v1.5-7B are presented in Table 3-4.

Analysis of Results. As shown in Table 1-4, SeGO consistently outperforms baselines across different architectures and scales. In W4A16, all methods show some degradation on the small-scale model. The advantage of SeGO becomes more significant as model size increases (InternVL28B/26B). In the more challenging 3-bit scenario, the quantization error is significantly amplified. RTN almost fail on small models, while AWQ and GPTQ offer some relief but still exhibit significant accuracy degradation. SeGO maintains high accuracy. For InternVL2-8B on MMMU val, SeGO reaches 0.4378, far exceeding RTN (0.2384) and GPTQ (0.2274). More importantly, SeGO ranks first for both LLaVA-onevision and LLaVA-v1.5, proving it effectively handles structural sensitivity asymmetry in different model architectures.

4.3 Calibration Efficiency and Cost Comparison Experiment

In addition to its accuracy advantage, another contribution is its exceptional calibration efficiency. To show this advantage, we compare the calibration time overhead of different methods.

Weight-only Quantization vs Weight-Activation Joint Quantization

To validate the efficiency of SeGO as a weight-only quantization strategy, we conduct a systematic comparison between SeGO (W4A16) and the current mainstream weightactivation joint quantization method, SmoothQuant (W4A8). We record the total calibration time (including data loading, statistical computation, and parameter search) and the final average quantization accuracy for VLM models of different sizes. The comparison results are shown in the Fig. 4.

SmoothQuant requires online statistics of the activation value distribution and scaling factor search for each layer of the entire network. In contrast, SeGO only needs to optimize the parameters for the Expansion Space and avoids the dynamic quantization of activation values. Results show that SeGO achieves significant acceleration across all models and its accuracy performance under W4A16 significantly outperforms the W4A8 SmoothQuant. Specifically, SeGO achieve average accuracy improvements of +0.65%, +2.23%, and +2.00% on three models. This is primarily due to the structural sensitivity protection strategy we propose: by precisely optimizing the weight distribution of the key Expansion Space, SeGO reduces quantization errors caused by extreme outliers in VLM activation values.

Optimality Analysis of the Golden-Section Solver

To verify the superiority of the core solver, we design a set of rigorous controlled experiments. We compare the calibration time and quantization accuracy of three different optimization strategies on the InternVL2-8B model. The three strategies are: (1) Grid Search, (2) Gradient-Based Search and (3) our Golden-Section Solver. The comparison results are shown in the Table 5.

While grid search doesn't require gradients, its solution quality is limited by the predefined step size, often only finding suboptimal solutions at discrete points. In contrast, the golden-section solver iteratively approximates the optimal point in the continuous interval, achieving high numerical precision. Our golden-section solver reach nearly the same accuracy as the gradient-based search. It confirm that quantization error function exhibits good unimodal characteristics within a constrained local interval, allowing precise solutions without expensive gradient optimization.

Moreover, our solver's calibration time is just 1/85 that of the gradient-based method (135 seconds vs 3.2 hours). This means SeGO can perform dozens of model iterations within the same computational budget, significantly lowering the deployment threshold for VLMs on edge devices.

Table 6: Ablation study of scaling intervention on different Expansion Space for InternVL2-8B across various benchmarks.

4.4 Ablation Study

To verify whether SeGO's introduction of scaling only in the Expansion Space is truly "effective and necessary", we conduct a structural-level ablation experiment on the InternVL28B model. We compare scaling only MLP (SeGO-MLP), only Attention (SeGO-Attn), both (SeGO-Full) and Baseline PTQ. Results are shown in Table 6.

The results show that scaling only the Expansion Space of MLP and Attention brings improvements, confirming that the Expansion Space is more sensitive observed in earlier figure. Quantitatively, the precision recovery achieved by optimizing the Expansion Space is significantly higher than that of the Projection Space, which validates our hypothesis that the feature expansion process in Transformers is the primary source of quantization error. SeGO-Full consistently outperforms the case where scaling is applied to only a single Expansion Space across three benchmarks and two bit-widths. It indicates that the MLP and Attention Expansion Space complement each other in error propagation. Correcting one type of Expansion Space only mitigates part of the structural asymmetry, while joint processing maximizes error suppression. By concurrently stabilizing expansion layers in both MLP and Attention, SeGO preserves the consistency of semantic manifold representations.

5 Conclusion

This paper proposes SeGO, a unified structural sensitivityaware sparse optimization framework for VLMs quantization. We reveal that the Expansion Space is the key bottleneck for quantization error. By applying scaling protection only to Expansion Space and using a Golden-Section Solver, SeGO establishes a new benchmark for VLM parameter amountquantization accuracy-calibration efficiency trade-offs. It outperforms existing methods significantly in W3A16 configurations and provides an 85x speedup over gradient-based methods. SeGO enables the efficient deployment of large-scale VLMs at low calibration costs. Future work will explore its generalization to larger models (70B+) and other modalities.

Contribution Statement

Tianqi Zhao and Ruihan Hu contributed equally to this work. ∗Ruihan Hu is the corresponding author.

Ethical Statement

There are no ethical issues.

Acknowledgments

This work was supported in part by the National Key Research and Development Program under Grant 2023YFF0905500, in part by the National Key Research and Development Program of Heilongjiang under Grant 2022ZX01A11, in part by the Natural Science Foundation of Heilongjiang Province under Grant LH2023F001, in part by the ZTE Industry-University-Institute Cooperation Funds under Grant No. 2022H2ZTE02-01-P4, and in part by the Spring Goose Talent Team Program of Heilongjiang.