RIVS: Mitigating Hallucination in Large Vision-Language Models via Representation Intervention on Visual Grounding Shift

1 Introduction

In recent years, Large Vision-Language Models (LVLMs) [Liu et al., 2023; Zhu et al., 2023; Dai et al., 2023; Ye et al., 2023; Bai et al., 2023; Zhang et al., 2023; Chen et al., 2023; Liu et al., 2024b; Dong et al., 2024; Bai et al., 2025] have achieved remarkable success in various downstream tasks like image understanding or multimodal reasoning, by aligning visual encoders with large language models. Despite these advances, object hallucination persists in LVLMs, which often generate non-existent objects, attributes, or relations, especially in open-ended scenarios, thus significantly undermining LVLMs reliability in practice.

∗Corresponding author.

To address this issue, existing researches can be broadly categorized into three classes. Training-based methods often rely on data augmentation or fine-tuning, such as reconstructing data distributions [Liu et al., 2024a], introducing counterfactual samples [Yu et al., 2024a], or designing contrastive learning objective [Jiang et al., 2024], which incur substantial computation cost and limits scalability. Inferencebased methods, including contrastive decoding [Leng et al., 2024; Chen et al., 2024; Huo et al., 2025; Zhuang et al., 2025] , over-trust penalty [Huang et al., 2024], and attention reweighting [Zhang et al., 2024; Liu et al., 2024c; Yin et al., 2025], offering a lighter and more efficient solution, but provide limited interpretability of underlying model mechanisms. Post-hoc approaches commonly refine outputs with extra models [Yin et al., 2024] or hallucination correctors [Zhou et al., 2024], inevitably increasing computational complexity. Recently, some works begin to conduct transparency research on models through representation engineering [Zou et al., 2023], typically deriving steering vectors by computing averaged differences between positive and negative samples, relying on prior information [Khayatan et al., 2025], carefully designed contrastive data [Liu et al., 2025], or removing solely at specific individual objects [Jiang et al., 2025]. Although effective under predefined conditions, above methods offer limited insight into how hallucinations emerge from the model's internal representations during the auto-regressive generation.

To address this limitation, we study hallucination from the perspective of representation shift induced by the model itself, focusing on how visual grounding representations change during auto-regressive generation. By analyzing attention patterns in intermediate layers, we observe that visual attention gradually decays as generation progresses, and hallucinations tend to emerge shortly after a pronounced drop in visual attention.

Motivated by this observation, we propose a two-stage hallucination mitigation framework, termed Representation Intervention based on Visual Grounding Shift (RIVS). In the representation shift identification stage, we use a small calibration set to analyse visual attention dynamics and identify the generation step at which visual attention exhibits a significant decline for each sample. As shown in Fig. 1, such drop point is closely aligned with the generation of hallucinated token 'people', suggesting that hallucinations are not only triggered by individual token, but are associated with an internal representation shift from visual grounded to ungrounded. Using the identified drop point as an anchor, we extract difference vectors between hidden states before and after the decline to capture the direction of representation drift. By aggregating such vectors across samples and layers, we construct a low-dimensional hallucination-related subspace, without relying on explicit contrast sample construction or external supervision. In the representation intervention stage, we apply a lightweight and controllable online intervention during inference by projecting intermediate hidden states away from the constructed subspace. Our approach requires no model retraining or parameter modification, introducing minimal computational overhead.

The effectiveness of RIVS is evaluated on LLaVA-1.5, InstructBLIP, Qwen-VL and Qwen2.5-VL across six benchmarks: CHAIR [Rohrbach et al., 2018], POPE [Li et al., 2023], AMBER [Wang et al., 2023], MME [Fu et al., 2023], ScienceQA [Lu et al., 2022] and MM-Vet [Yu et al., 2024b]. Extensive experimental results validate the effectiveness of our RIVS in mitigating various types of hallucinations in LVLMs while preserving visual perception and reasoning capabilities.

To sum up, our main contributions are as follows:

• We present an analysis of hallucination in LVLMs from the perspective of internal representation shift during auto-regressive generation, demonstrating the shifts induced by visual attention decay can provide valuable guidance for mitigating hallucinations.
• We propose RIVS, a novel two-stage framework that identifies hallucination-related representation shifts using a small calibration set and constructs a low-
• dimensional hallucination subspace without explicit contrastive sample construction, enabling effective intervention during inference.
• Extensive experiments on six benchmarks show that RIVS consistently reduces hallucinations across multiple LVLMs, while preserving language quality and achieving favorable inference efficiency.

2 RelatedWork

Object hallucinations in LVLMs refer to models erroneously generating content that does not correspond to the input image. Fine-tuning methods aim to balance data distributions [Liu et al., 2024a], construct counterfactual data [Yu et al., 2024a], and modify optimisation objectives [Jiang et al., 2024] for fine-tuning LVLMs. Inference-time approaches intervene during decoding stage. VCD [Leng et al., 2024] contrasts hallucinations induced from language priors by introducing noise to images. SID [Huo et al., 2025] amplifies multi-modal hallucinations by retaining unimportant visual tokens, then eliminates them through contrastive decoding. HALC [Chen et al., 2024] mitigates hallucinations by focusing on the most critical image regions. VASparse [Zhuang et al., 2025] designs a token selection strategy based on visual perception and introduces a sparse visual contrast decoding method to calibrate the output distribution. OPERA [Huang et al., 2024] penalises generated tokens with attention sinking patterns, thereby adjusting the candidate level of decoded tokens. VAF [Yin et al., 2025] enhances attention to the visual tokens in the model's intermediate layers, thereby strengthening the model's visual representations. Post-hocs methods [Yin et al., 2024; Zhou et al., 2024; Lee et al., 2024] employ auxiliary models or train specialized hallucination correctors to rectify generated hallucinatory text.

Inspired by Representation Engineering [Zou et al., 2023], recent work [Khayatan et al., 2025] indicates behavioural changes in MLLMs can be characterised as representation shifts in the latent space. VTI [Liu et al., 2025] constructs positive and negative samples to estimate intervention directions and manipulate latent representations in visual and textual modules. Unlike aforementioned methods, we re-examine the mechanism of hallucination generation in LVLMs from the perspective of representation shift during auto-regressive generation. This design avoids additional supervision costs and minimises disruption to language generation capabilities.

3 Preliminary

LVLMs takes an image V and a text prompt T as input, the image is first encoded by a pre-trained visual encoder, then projected into the semantic space of LLM via a connector to yield a sequence of visual features Xv ∈ Rn×d , where n denotes the number of visual tokens. The text input is tokenized into a sequence of language features Xl ∈ Rm×d , with m text tokens. The final multimodal input is formed by concatenating visual and language sequence as Xvl = (Xv;Xl) ∈ R(n+m)×d .

The decoder consists of L stacked Transformer blocks and generates output sequence y = [y1, y2, . . . , yT ] autoregressively. Denote H(l) as the hidden states at layer l, each layer transforms the representations from the previous layer as:

4 Method

In this section, we first investigate the correlation between intermediate-layer representation shifts and hallucination generation in Sec. 4.1. Subsequently, in Sec. 4.2, we then introduce a lightweight representation intervention mechanism to mitigate hallucinations during inference.

4.1 Correlation Between Intermediate Representation Shifts and Hallucinations

Linear Artificial Tomography (LAT)

Linear Artificial Tomography (LAT) [Zou et al., 2023] is a technique in Representation Engineering for analysing and manipulating high-level concepts encoded in neural networks, which comprises two components: representation reading and representation control. Representation reading locates representations of high-level concepts within the network by: (1) designing stimulus and task, (2) collecting neural activity, and (3) constructting a linear model. Representation control subsequently leverages extracted reading vectors or contrast vectors to intervene in model activations through linear operations such as projection or linear combination.

Hallucination Analysis via Visual Attention-Guided Difference Vectors

Prior studies suggest that hallucinations in LVLMs are closely associated with forgetting or weakening of visual grounding during generation [Deng et al., 2024; Zou et al., 2024]. Instead of explicitly construct contrast inputs [Liu et al., 2025], we analyse hallucination generation by identifying significant visual attention decay points during generation and extracting representation difference vectors around these points.

Stimulus and Task. We consider an image captioning task, where LVLM is prompted with an image and a instruction (e.g., "Please describe this image in detail"). During generation, we track the model's attention allocation to visual tokens and identify generation step at which visual attention exhibits a significant decline.

Neural activity collection. To investigate the impact of representations on hallucinations, we identify points where visual attention significantly drops during generation, then extract difference vectors around these drop points to capture variations in representations. Through this process, we can capture the shift in the model's understanding of visual information during the generation process.

Adaptive threshold-based drop point detection. For each sample, we first detect visual drop points in generated tokens by tracking variations in visual attention. For each step t, we compute the average attention weight of the generated token towards visual tokens at layer l:

where A(l,h) denotes the attention weight matrix of the h-th head at layer l, and I indexes visual tokens. We concatenate over generation steps to obtain the visual attention trajectory, then normalize each visual attention curve as:

where µ(l) and σ(l) denote mean and standard deviation. We further apply Gaussian smoothing Gσ(·) to suppress noise:

Next, we define the first-order discrete derivative of the smoothed attention curve as:

where ∆(l) (t) < 0 indicates a declining trend in visual attention at generation step t. We define 'significant drop' as a marked negative deviation relative to the overall variation distribution of this curve. Specifically, we propose a drop-point detection method based on adaptive threshold τ (l) , defined as the γ-quantile (usually set to 0.1) of the derivative distribution, and dentify the earliest generation step satisfying the derivative value falls below the threshold, as the significant visual attention drop point t(l) ⋆ :

If no such point exists, we select the step with the minimum derivative value as the most notable turning point:

Difference vector Extraction. To characterise the representation shift induced by the visual grounding decay, we extract representations from windows before and after the drop point t(l) ⋆ , using them to derive difference vectors. The window preceding the visual drop point is denoted as B = {t(l) ⋆ -w, . . . , t(l) ⋆ -1}, and the window after the visual drop point as A = {t(l) ⋆ , . . . , t(l) ⋆ +w - 1}. We then average representations within each window as:

where h(l) t denotes the representation corresponding to the token generated at step t of layer l.

Next, the representation shift direction induced by visual grounding instability is then defined as:

Constructing a Linear Model. The objective of this step is to examine whether hallucination-related information is linearly encoded in the representation shift directions extracted from the model, using only internal neural activations as input. Following the representation reading paradigm of LAT, we conduct a linear probing experiment on the MSCOCO dataset [Lin et al., 2014]. Specifically, we construct a tokenanchored representation shift signal based on visual attention decay. Specifically, for each generated sequence si = {yi,1, yi,2, . . . , yi,T}, we identify the visual attention drop point ti,⋆ following the process described in neural activity collection. Using this position as an anchor, we extract representations from its adjacent windows to compute the difference vector di, which characterises the representation shift induced by visual grounding decay. For hallucination annotation, we map word-level annotations to token-level and assign each difference vector a binary label li based on whether hallucination occurs following its anchor point. After classbalanced sampling, we ultimately obtain a labelled dataset of difference vectors: D = {(di, li)}.

To assess the linear separability of hallucination-related signals, we perform a linear probing experiment using difference vectors extracted from decoder layer 25. Specifically, we train a logistic regression classifier to predict hallucination occurrence based solely on difference vectors and evaluated with 5-fold cross-validation. The probe achieves an average AUC of 81.67 and accuracy of 73.79, indicating that hallucination-raleted information is linearly separable from the shifting representation captured by visual attentionguided difference vectors. Fig. 2 compares probes trained on difference vectors with several control baselines, including label shuffling, feature shuffling, and random noise. Probes trained on representation differences significantly outperform other baselines, with all pairwise comparisons showing statistically significant improvements (p<0.001), indicating the robustness of the observed linear separability.

4.2 Representation Intervention Strategy based on Visual Grounding Shift

Motivated by the linear separability analysis in the previous section, we observe that representation difference vectors before and after the visual attention drop point are strongly correlated with hallucination occurrence in the latent space. Building on this observation and inspired by representation control in LAT, we propose a representation intervention strategy based on visual grounding shift, as illustrated in Fig. 3. This approach employs a two-stage framework to decouple identification of hallucination-related directions from online intervention. First, we extract and aggregate representation difference vectors associated with visual attention decay to construct a low-dimensional hallucination subspace. In the inference stage, we apply a lightweight projection-based intervention to intermediate hidden states, suppressing representation shifts along hallucination-related directions during generation, thereby reducing hallucination occurrence.

Construction of the Hallucination Subspace

Considering that hallucinations in LVLMs are not arise from representations at a single layer, but rather result from coordinated shifts across multiple layers, we collect representation difference vectors across a set of layers L = {l1, l2, . . . , lc}. Specifically, for each sample i and layer ∈ L, we first track the visual attention curve ˜m(l) over the generated sequence and detect the visual attention drop point t(l) i,⋆ using the adaptive threshold-based drop point detection method described in Sec. 4.1, and then extract hidden states from windows before and after the drop point to obtain the difference vector d(l) i .

To obtain a stable hallucination-related respresentation, we aggregate difference vectors across N samples in calibration set and |L| layers, all normalized vectors are concatenated as:

where M = N ×|L| denotes the number of direction vectors.

We then centre each difference vector as:

where dm denote the normalized difference vector. Then, we apply principal component analysis (PCA) to extract the most representative low-dimensional subspace. Specifically, we perform singular value decomposition (SVD) on ˜D, which denote the centred version of D, and select the top-k right singular vectors as an orthogonal basis:

This subspace characterises representation shift patterns associated with visual attention decay across layers and samples.

Language Subspace Modelling and Orthogonal Constraint

Since vision and language tokens in LVLMs are mapped into a shared semantic space, directly intervening on hidden representations may inadvertently undermine language generation. To mitigate this effect, we model a language-dominant subspace and constrain the hallucination space accordingly. Specifically, we collect the hidden states from L based solely on the text input, then perform PCA on these linguistic representations in the same manner as for the difference vectors, yielding a low-dimensional language subspace Ulang, which captures the principal language-driven direction during generation.

To ensure the intervention primarily affects representations related to visual grounding without disrupting language semantic structures, we orthogonalise the visual hallucination subspace with respect to the language subspace:

where β ∈ [0, 1] controls the strength of the orthogonal constraint. After normalization, Uhall serves as the final hallucination-related subspace, capturing representation shifts induced by visual grounding degradation while preserving natural language structures.

Projection-based Representation Intervention

After constructing the hallucination-related subspace, we employ it for online representation intervention during inference to suppress hallucination-related representation shifts. Unlike methods based on retraining or modifying model weights, we adopt a lightweight linear projection mechanism to modify the model's intermediate representations. For the hidden states H(l) at layer l, we apply the intervention as:

where α controls the intervention strength. As hallucinationrelated representation shifts emerge across multiple layers, the intervention is applied to the hidden states of all layers l ∈ L respectively.

5 Experiments

5.1 Experimental Settings

Models. Following previous works [Liu et al., 2025; Yin et al., 2025], we conduct experiments on four mainstream models: LLaVA-1.5 [Liu et al., 2024b], Instructblip [Dai et al., 2023], Qwen-VL [Bai et al., 2023] and Qwen2.5-VL [Bai et al., 2025]. All model are with 7B parameters.

Baselines. We compare performance with basic greedy search and several popular training-free methods, including VCD [Leng et al., 2024], OPERA [Huang et al., 2024], HALC [Chen et al., 2024], SID [Huo et al., 2025], VTI [Liu et al., 2025], VASparse [Zhuang et al., 2025] and VAF [Yin et al., 2025]. To ensure a fair comparison, we use the paper's default settings for these baselines.

Implementation Details. The sample size N for calibration set is set to 100. We uniformly set k = 3, α = 0.1, and set β to 0.5, 0.8, 0.2 and 1.0, w to 100, 6, 100, 100 for LLaVA-1.5, InstructBLIP, Qwen-VL, and Qwen2.5-VL respectively. We faithfully replicate all baselines using their released code and the hyperparameters reported in the original papers.

5.2 Experimental Results

We evaluate on four hallucination benchmarks and two reasoning benchmarks.

Hallucination Mitigation

CHAIR. [Rohrbach et al., 2018] We randomly select 500 images from MSCOCO val2014 [Lin et al., 2014], and use the prompt "Please describe this image in detail." to query LVLMs. To ensure fair comparison, we set the max new tokens to 512. Table 1 demonstrates that our method outperforms all baselines except VTI [Liu et al., 2025] in mitigating hallucinations on all LVLMs. While VTI yields a low hallucination score, it produces shorter outputs with noticeably reduced recall, which limit ground-truth object coverage. In contrast, our RIVS achieves a more favorable trade-off between factuality and content coverage.

POPE. [Li et al., 2023] We evaluate our method on MSCOCO using three settings: random, popular, and adversarial. We report the average performance across the three settings in Table 2, indicating that our approach achieves superior average performance across all models compared to the baselines, with particularly enhanced capabilities in the more challenging popular and adversarial settings.

AMBER. [Wang et al., 2023] AMBER is a benchmark for comprehensively evaluating discriminative and generative capabilities of LVLMs, and the AMBER Score provides a synthetic metric for this assessment. Similarly, as shown in Table 6, while VTI effectively eliminates hallucinations in the generation task, its coverage rate is significantly reduced compared to the original algorithm (40.0 vs 50.5). In contrast, our approach not only mitigates hallucinations effectively but also maintains coverage rate, further demonstrating our method's capability to reduce hallucinations while preserving content richness. Furthermore, our approach achieves the best overall performance, demonstrating that pre-extracting difference vectors from a small sample set is task-agnostic and performs excellently on both short and long sequence tasks. Moreover, whereas other approaches demonstrate effectiveness on LLaVA-1.5 yet exhibit performance degradation or marginal gains on InstructBLIP, our method consistently achieves superior performance across diverse LVLMs. MME. [Fu et al., 2023] Table 3 presents performance across ten perception tasks, where Hallu. denotes scores on hallucination subset. Our approach obtains considerable performance on hallucination subset and achieves the best overall performance. Specifically, it surpasses the optimal baseline by 22 and 39 points on LLaVA-1.5 and InstructBLIP respectively, demonstrating the superiority of the proposed method in visual perception.

Furthermore, we validate the effectiveness of our approach on the Qwen-VL and Qwen2.5-VL model, which possesses stronger multimodal capabilities. The results in Table 4 demonstrate that RIVS consistently achieves performance gains even on the less-hallucinatory LVLM.

Comprehensive Visual-Language Abilities Assessment

To evaluate our method's capability in vision-language reasoning, we further experiment on ScienceQA [Lu et al., 2022] and MM-Vet [Yu et al., 2024b]. The results in Table 5 demonstrate that while VCD and VTI mitigates hallucinations to some extent, it compromises model's reasoning ability. By contrast, our approach preserves reasoning and response accuracy, proving better adapted to complex reasoning tasks.

Inference efficiency

Fig.4 compares the inference efficiency between different methods. To ensure a fair comparison, inference latency was measured in a token-by-token manner. Compared to other methods, our approach achieves the best performance in inference speed and also maintains a significant advantage in GPU memory usage. The bubble area is proportional to F1 Score on POPE, and the result indicates that our method achieves an optimal balance between performance and computational cost overall.

Ablation Study

To validate the effectiveness of the proposed method, we conduct ablation studies with several controlled baselines, including: (1) randomize difference vectors, (2) hallucinationrelated subspace constructed from randomly sampled orthogonal bases, (3) difference vectors extracted from randomly selected positions, (4) employing the visual hallucinationreleated subspace directly for projection intervention without applying language orthogonalisation, and (5) projection amplification, where the projection onto the hallucinationrelated subspace is added rather than suppressed.

As shown in Table 7, random difference vectors yield limited improvements, indicating that hallucination mitigation cannot be achieved by arbitrary representation interventions. Random position and visual hallucination-related subspace reduce hallucination metrics to some extent, indicating that suppressing representation drift can partially alleviate hallucinations, but at the cost of modest decreases in recall and generation length. This indicates that inadequate precision in suppressing representations may lightly impair model's language generation capability. In contrast, random subspace and projection amplification substantially increase hallucination metrics, confirming that the identified subspace indeed encodes hallucination-related informations. Our RIVS achieves the lowest hallucination scores while maintaining comparable Recall and generation length, demonstrating effective hallucination suppression without sacrificing generation quality.

6 Conclusion

In this work, we investigate object hallucination in LVLMs from the perspective of dynamic representation shift during auto-regressive generation. By analyzing the decay of visual attention, we identify a shift in representations from visually grounded to language-dominated states, which is closely associated with hallucination. Building on this observation, we propose RIVS, a training-free representation intervention framework that constructs a hallucination-related subspace from representation differences around visual attention drop points and applies an projection-based intervention during inference. Extensive experiments across multiple hallucination and reasoning benchmarks demonstrate that RIVS effectively reduces hallucinations while preserving language quality and generation richness, offering a new representational perspective for understanding and mitigating hallucinations in LVLMs.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant No. 62276110 and in part by the fund of Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL). The authors would also like to thank the anonymous reviewers for their comments on improving the quality of this paper.