Online Self-Calibration Against Hallucination in Vision-Language Models

1 Introduction

Large Vision-Language Models (LVLMs) [Liu et al., 2023c; Zhu et al., 2023; Bai et al., 2023; Dai et al., 2023; Lu et al., 2024] integrate visual encoders with pre-trained Large Language Models (LLMs), achieving strong performance across a wide range of multimodal tasks, from image captioning to visual reasoning. However, these models frequently suffer from hallucinations [Rawte et al., 2023; Bai et al., 2024; Liu et al., 2024], generating content that is inconsistent with or absent from the visual input, such as fabricating nonexistent objects, misinterpreting spatial relationships, or incorrectly describing object attributes. This limitation poses a significant barrier to deploying LVLMs in safety-critical domains like autonomous driving, medical imaging, and robotics, where factual grounding is essential.

Recent efforts to mitigate hallucinations have largely relied on preference alignment techniques, including Reinforcement Learning from Human Feedback (RLHF) [Sun et

al., 2024] and DPO [Rafailov et al., 2023]. These methods typically construct preference datasets using human annotations [Sun et al., 2024; Gunjal et al., 2024] or responses distilled from stronger, proprietary models such as GPT [Achiam et al., 2023; Liu et al., 2023a; Li et al., 2023a; Zhou et al., 2024]. While effective to some extent, we argue that this reliance on offline supervision introduces a fundamental Supervision-Perception Mismatch. As illustrated in Fig. 1, teacher models with superior visual capabilities tend to produce highly detailed descriptions that capture subtle visual elements, such as small objects or fine-grained attributes that are difficult to discern. When a weaker student model is supervised by such data, it is compelled to reproduce these finegrained details that lie beyond its perceptual capacity. Unable to ground these descriptions in actual visual features, the student instead resorts to exploiting language priors and statistical shortcuts, generating different but equally ungrounded content. In essence, the model learns to guess rather than to see. This observation is further validated by our pilot experiments in Section 3.1. We find that fine-tuning LLaVA-1.5-7B on teacher-distilled data paradoxically increases hallucination rates, with performance degrading further as more training data is added. These findings motivate a shift toward online learning, where training data respects the model's intrinsic perceptual boundaries.

A natural question arises: can we obtain high-quality, truthful supervision from a model that is itself prone to hallucination? To address this, we identify a notable GenerativeDiscriminative Gap within LVLMs. Our analysis shows that while models often yield to language inertia during autoregressive generation, they exhibit considerably higher accuracy on discriminative tasks, such as verifying whether a specific object exists in an image. This gap arises because discriminative verification, by explicitly conditioning on a specific query, reduces the influence of unconstrained language priors that dominate open-ended generation. This finding suggests that LVLMs possess a latent capacity for selfverification that remains underutilized during generation.

While the Generative-Discriminative Gap addresses the question of where to obtain reliable supervision, another key challenge lies in how to construct high-quality training data. Standard decoding strategies like greedy or beam search optimize locally, presenting two limitations. First, some tokens that appear safe at the current step may carry high risk of inducing hallucinations in subsequent generation, a cascading effect that local optimization cannot foresee. Second, greedily selecting branches with the lowest immediate hallucination rate at each step may compromise overall response quality in terms of logical consistency and fluency. To overcome these limitations, we integrate Monte Carlo Tree Search (MCTS) to explore the generation space more strategically [Xie et al., 2024a; Tian et al., 2024; Zhang et al., 2024]. We design a Dual-Granularity Reward Mechanism to guide the search. At the node level, we leverage the model's discriminative capability by prompting it to verify whether each generated sentence mentions objects absent from the image, using the probability of a negative response as the process reward. At the trajectory level, we employ a Gated Outcome Reward that evaluates response quality only if the complete response passes a faithfulness check, and returns zero otherwise.

Through MCTS backpropagation, terminal rewards propagate from leaf nodes to the root, enabling the model to identify generation trajectories that balance visual faithfulness with descriptive richness. Building on these insights, we propose Online Self-CAlibRation (OSCAR), a unified framework for online preference learning. Specifically, OSCAR extracts preference pairs from the MCTS tree at two granularities: global path comparison selects complete trajectories with the highest and lowest cumulative values, while sibling comparison pairs nodes along the optimal path with their worst-performing siblings. These preference pairs are then used to update the model via DPO [Rafailov et al., 2023]. At each iteration, the updated model generates new preference data through MCTS, ensuring the training distribution evolves alongside the model's capabilities and enabling continuous self-improvement. Our contributions are as follows:

• We empirically demonstrate the necessity of online preference learning that respects the model's intrinsic perceptual boundaries, showing that offline distillation from stronger teachers can unexpectedly exacerbate hallucinations.
• We propose OSCAR, a novel training paradigm that exploits the Generative-Discriminative Gap. By integrating MCTS with a Dual-Granularity Reward Mechanism, OSCAR enables lookahead to suppress early tokens that risk inducing downstream hallucinations.
• Extensive experiments show that OSCAR achieves state-of-the-art performance on hallucination benchmarks while simultaneously preserving general multimodal capabilities.

2 RelatedWork

2.1 Hallucination in LVLMs

Large Vision-Language Models (LVLMs) frequently suffer from hallucinations, generating content that is inconsistent with the visual input [Rawte et al., 2023; Bai et al., 2024]. Various approaches have been proposed to mitigate this issue, including enhancing dataset quality [Liu et al., 2023b; Gunjal et al., 2024; Li et al., 2023a], manipulating the decoding process [Leng et al., 2024; Huang et al., 2024; Suo et al., 2025], leveraging external models for post-hoc correction [Yin et al., 2024; Zhou et al., 2024], and preference optimization [Xie et al., 2024b; Wang et al., 2025].

2.2 Self-Improvement for LLMs

Self-improvement methods enable models to enhance their capabilities using self-generated feedback, reducing reliance on external annotations [Huang et al., 2023; Yuan et al., 2024; Hu et al., 2024; Sun et al., 2025]. In the vision-language domain, recent works such as STIC [Deng et al., 2024] and SIMA [Wang et al., 2025] have explored self-improvement for hallucination mitigation. However, these methods typically construct preference data via simple sampling or beam search, which fails to account for the cascading nature of hallucinations. We instead integrate MCTS with a DualGranularity Reward Mechanism, enabling lookahead to suppress tokens that risk inducing downstream hallucinations.

2.3 Monte Carlo Tree Search

Monte Carlo Tree Search (MCTS) has emerged as a powerful technique for enhancing reasoning in LLMs, inspired by its success in game-playing agents [Silver et al., 2016]. Recent works have adapted MCTS to guide text generation by simulating future trajectories and backpropagating rewards, achieving improvements in mathematical reasoning [Xie et al., 2024a; Tian et al., 2024; Zhang et al., 2024] and task planning [Hao et al., 2023; Li and Ng, 2025; Li et al., 2025]. In the context of hallucination mitigation, we are the first to leverage MCTS for preference data construction, enabling lookahead to suppress locally plausible tokens that risk inducing downstream hallucinations.

3 Observations andMotivations

3.1 Supervision-PerceptionMismatch

Recent preference alignment methods predominantly rely on supervision signals distilled from stronger, proprietary models such as GPT [Achiam et al., 2023]. We hypothesize that this reliance on offline supervision introduces a fundamental Supervision-Perception Mismatch: when a target model with limited visual perception capacity is trained on data generated by a teacher model with superior perceptual abilities, the student is compelled to align with fine-grained visual details that exceed its intrinsic perceptual capabilities. Consequently, the model may learn to minimize training loss by exploiting language priors and statistical shortcuts rather than grounding its outputs in visual features. As illustrated in Fig. 1, teacher models with superior visual capabilities tend to capture subtle visual elements that weaker models cannot reliably perceive. For instance, the teacher model identifies a "mouse" near the laptop, a "green glass object" in the corner, and a "floor lamp" in the background. When supervised by such data, the student model fails to ground these details in actual visual features, instead generating different but equally ungrounded content, such as hallucinating a "mirror" above the fireplace, a "poster", and a "blue wall" that does not exist.

To empirically validate this hypothesis, we employ Qwen3-VL-8B-Instruct [Yang et al., 2025] to generate detailed image descriptions on randomly sampled images from the LLaVA-150k dataset, and use these teacher-generated descriptions to fine-tune LLaVA-1.5-7B with varying data scales (2.5k, 5k, 7.5k, and 10k samples). As shown in Fig. 2 (a), despite Qwen3-VL achieving 88.91% on POPE F1, the fine-tuned LLaVA models consistently underperform the original baseline (85.87%), with performance degrading further as more training data is added. Similar trends are observed on the AMBER benchmark, where both CHAIR and Cog scores deteriorate after fine-tuning (Fig. 2b-c). This counterintuitive finding confirms that offline supervision can paradoxically exacerbate hallucinations, motivating our shift toward online learning.

3.2 Generative-Discriminative Gap

To explore whether a hallucination-prone model can still provide useful self-supervision, we investigate its behavior across different inference paradigms. Our analysis reveals a notable Generative-Discriminative Gap: while models frequently yield to language inertia during autoregressive generation, where plausible linguistic patterns overshadow visual grounding, they exhibit improved accuracy when tasked with discriminative verification against visual evidence. Fig. 3 illustrates this gap with a concrete example. When prompted to describe an image in detail, LLaVA-1.5 hallucinates a "clock mounted on the wall" that does not exist. However, when the same model is explicitly asked "Is there a clock in the image?", it correctly answers "No". This discrepancy suggests that discriminative verification, by conditioning on a specific query, reduces the influence of unconstrained language priors that dominate open-ended generation.

To quantify this gap, we conduct the following analysis. We first generate image captions using LLaVA-1.5-7B on 500 randomly sampled images from the COCO dataset and compute the CHAIR metrics. For each hallucinated object x detected in the generated captions, we construct a discriminative query: "Is there a/an x in the image?" and prompt the same model to respond with "Yes" or "No". If the model correctly answers "No", we remove this hallucinated object from the caption and recompute the CHAIR metrics. As shown in Fig. 3, this simple self-verification procedure reduces CHAIRS from 49.0% to 36.0% and CHAIRI from 14.3% to 9.3%, confirming that LVLMs possess a latent capacity for self-verification that remains underutilized during standard generation. This finding motivates our approach: by leveraging the model's discriminative ability to curate training data, we can align the granularity of generated descriptions with the model's intrinsic perceptual capabilities while improving factual accuracy.

4 Methodology

Building upon the insights from Section 3, we present Online Self-CAlibRation (OSCAR) , a framework illustrated in Fig. 4 that leverages the model's capability to construct high-quality online preference data and iteratively refines the model through DPO. In this section, we first detail our MCTS-guided generation with a dual-granularity reward mechanism (§4.2), and then describe the preference learning procedure (§4.3).

4.1 LVLM Inference

A Large Vision-Language Model (LVLM) Mθ with parameters θ takes as input a visual image v and a textual instruction q, and generates a textual response y = (y1, y2, . . . , yT ) in an autoregressive manner. Specifically, at each generation step t, the model computes the probability distribution over the vocabulary conditioned on the visual input, the textual instruction, and the previously generated tokens:

where ℓt denotes the logit vector for the next token yt, and y<t = (y1, . . . , yt-1) represents the sequence of tokens generated before step t. The complete response is generated by sequentially sampling tokens from this distribution until the end-of-sequence token is produced.

4.2 MCTS-Guided Generation

A key challenge in constructing faithful training data is that standard decoding strategies such as greedy or beam search optimize locally, which suffers from two limitations. First, tokens that seem acceptable at the current step may induce hallucinations in later generation, a long-term risk that local optimization cannot anticipate. Second, prioritizing branches with minimal immediate hallucination at each step may degrade overall response quality, sacrificing logical consistency and fluency. To overcome these limitations, we integrate Monte Carlo Tree Search (MCTS) into the generation process. By simulating future trajectories and backpropagating terminal rewards, MCTS enables the model to evaluate the long-term value of each token, identifying generation paths that balance visual faithfulness with descriptive richness.

MCTS Procedure. We decompose the generation process into sentence-level steps, where each node in the search tree represents a partial response. Formally, let st = (v, q, a1, a2, . . . , at-1) denote the state at step t, where ai represents the i-th generated sentence. An action at corresponds to generating a complete sentence, delimited by terminal punctuation marks. Each MCTS iteration consists of four phases:

• Selection. Starting from the root node, we traverse the tree by selecting child nodes according to the PUCT (Predictor + Upper Confidence bounds applied to Trees) criterion [Rosin, 2011]:

where Q(s, a) denotes the action value, p(a|s) = πθ(a|s)/|a|λ is the length-normalized policy probability with penalty λ, N(s) and N(s, a) are visit counts, and cpuct controls the exploration-exploitation trade-off.

• Expansion. At a leaf node, we sample K candidate sentences from the policy πθ using temperature sampling. To ensure diversity, we filter candidates with embedding similarity exceeding a threshold τsim.
• Evaluation. Each expanded node receives a value score

through our Dual-Granularity Reward Mechanism, detailed below.

• Backpropagation. After evaluation, statistics are propagated from the leaf node back to the root. The Q-value and state value are updated as:

where r(st, a) = value(st+1)-value(st) represents the immediate reward and γ is the discount factor.

Dual-Granularity Reward Mechanism. Central to our approach is a reward mechanism that combines node-level process supervision with trajectory-level outcome evaluation. This design enables the search to identify early tokens that, while locally plausible, carry high risk of inducing downstream hallucinations.

Process Reward (Node-Level). For each generated sentence at, we employ the model's discriminative capability to assess whether the sentence mentions objects absent from the image. Specifically, we construct a verification prompt:

<image> Please determine if the following sentence mentions objects that are not present in the image: {sentence}

Answer Choices: (A) Yes (B) No

The process reward rproc is computed as the probability that the model judges the sentence as hallucination-free:

where Pproc(at) denotes the verification prompt instantiated with the candidate sentence at.

Gated Outcome Reward (Trajectory-Level). To evaluate the quality of a complete trajectory, we perform a greedy rollout from the current state to a terminal state, yielding a complete response yrollout. The outcome reward incorporates a gating mechanism that enforces strict faithfulness requirements. First, we assess whether the complete response contains any hallucinated content through a faithfulness check. Specifically, we extract all objects mentioned in the generated description and compare them against the ground-truth objects provided by the dataset. Specifically, we extract all object nouns from the generated description and map them to canonical COCO category names using a predefined synonym dictionary. If any mapped object does not appear in the ground-truth object set, the response is considered to contain hallucinations. The gating function is defined as:

where O(yrollout) denotes the set of canonical object names extracted from the generated response, and Ogt denotes the ground-truth object set. Second, for trajectories that pass the gate, we assess the response quality along dimensions of logical consistency, linguistic fluency, and redundancy:

<image> Please evaluate the following caption on three dimensions: logical consistency, linguistic fluency, and redundancy.

Caption: {yrollout}

Please provide a single overall score from 0 to 10, where 0 is extremely poor and 10 is excellent.

Let scorequality ∈ [0, 10] denote the extracted quality score. The gated outcome reward is defined as:

The final value assigned combines both granularities:

ThroughMCTS backpropagation, this trajectory-level reward signal propagates from leaf nodes to the root, elevating the estimated value of early tokens that lead to faithful and highquality completions.

4.3 Iterative Preference Learning

Preference Pair Extraction. We extract preference pairs from the MCTS tree at two levels of granularity. For global path comparison, we identify the complete path with the highest cumulative Q-value as the chosen response y+ and the one with the lowest Q-value as the rejected response y-:

where T denotes the set of all complete trajectories in the tree. For sibling comparison, we traverse each depth along the optimal path and pair the selected node with its worstperforming sibling, if their Q-value difference exceeds a threshold δQ:

where s<d denotes the partial response up to depth d, and ⊕ denotes concatenation. This step-wise comparison enables the extraction of multiple preference pairs from a single MCTS tree, maximizing the utilization of information accumulated during the search process.

DPO Training. Given the preference dataset D = {(vi, qi, y+ i , yi )}Ni=1 constructed via MCTS, we update the model using Direct Preference Optimization (DPO) [Rafailov et al., 2023]. The DPO objective directly optimizes the policy to prefer chosen responses over rejected ones:

where σ(·) is the sigmoid function, β is a temperature parameter, and:

Here, πref denotes the reference policy, initialized as the model checkpoint before the current iteration. The training proceeds iteratively: at each iteration m, we use the current policy π(m) θ to construct new preference data viaMCTS, then update the model to obtain π(m+1) θ . This online paradigm ensures that the training distribution evolves alongside the model's improving capabilities, progressively tightening the alignment between generated content and the model's perceptual boundaries.

5 Experiments

5.1 Experimental Setup

Evaluation Benchmarks. We conduct evaluations on both generative and discriminative hallucination tasks. For the generative task, we evaluate on Object-HalBench [Rohrbach et al., 2018], AMBER [Wang et al., 2023], and MM-VET [Yu et al., 2023]. For the discriminative task, we report results on AMBER [Wang et al., 2023] and POPE [Li et al., 2023b].

Baselines. We compare OSCAR with three categories of methods: (1) other open-source LVLMs, including InstructBLIP [Dai et al., 2023], MiniGPT-4 [Zhu et al., 2023], and mPLUG-Owl2 [Ye et al., 2024]; (2) SoTA data-driven preference learning methods for hallucination mitigation, including STIC [Deng et al., 2024], POVID [Zhou et al., 2024], and SIMA [Wang et al., 2025]; (3) a Self-Rewarding baseline that employs the same hallucination detection reward but constructs preference data via beam search instead of MCTS.

Implementation Details. We adopt LLaVA-1.5-7B and LLaVA-1.5-13B as base models. For preference data construction, we sample images and prompts from LLaVA150k [Liu et al., 2023c], yielding 120k preference pairs per iteration. The MCTS search is configured with cpuct = 1.0, length penalty λ = 1.25, and Q-value difference threshold δQ = 0.05. For DPO training, we employ LoRA [Hu et al., 2022] with rank 128 and α = 256, using the Adam optimizer with a learning rate of 1 × 10-5 and temperature β = 0.1. The model is iteratively trained for 3 iterations.

5.2 Main Results

Overall Performance. Tab. 1 presents comprehensive comparisons between OSCAR and existing methods on hallucination benchmarks. Our method achieves state-of-the-art performance on both generative and discriminative tasks, demonstrating its effectiveness in mitigating hallucinations while preserving general multimodal capabilities.

Generative Task. On Object-HalBench, OSCAR substantially reduces hallucination metrics for LLaVA-1.5-7B, decreasing CHAIRS from 49.0 to 27.6 and CHAIRI from 14.3 to 8.2. These improvements significantly surpass prior methods such as POVID (33.6/9.0) and SIMA (40.9/10.4). On AMBER generative metrics, OSCAR achieves the lowest Hal score (17.2) and Cog score (1.6), indicating fewer hallucinated responses and reduced reliance on human-like cognitive shortcuts. Notably, on MM-VET, which evaluates general multimodal understanding, OSCAR improves the overall score from 32.5 to 34.6, demonstrating that our hallucination mitigation does not compromise response quality or descriptive richness. For the larger LLaVA-1.5-13B model, OSCAR yields even more substantial gains, achieving CHAIRS of 5.4 and CHAIRI of 2.6, which represent reductions of 87.9% and 78.0% respectively compared to the baseline.

Discriminative Task. On discriminative benchmarks, OSCAR also demonstrates consistent improvements. For AMBER discrimination, OSCAR improves accuracy from 72.2% to 75.8% and F1 score from 75.5% to 80.2% on LLaVA-1.57B. On POPE, OSCAR achieves an F1 score of 86.22%, comparable to POVID (86.90%), while significantly outperforming it on generative tasks. These results confirm that OSCAR enhances the model's visual grounding capability across both generative and discriminative paradigms.

Iterative Improvement. A key advantage of our approach is its ability to enable continuous self-improvement through iterative training. As shown in Tab. 1, performance improves progressively from Iter1 to Iter3 across the majority of metrics. For LLaVA-1.5-7B, CHAIRS decreases from 32.0 (Iter1) to 28.6 (Iter2) and further to 27.6 (Iter3), while the AMBER Hal score drops from 22.1 to 19.4 to 17.2 over the same iterations. Similar trends are observed for LLaVA1.5-13B, where CHAIRS decreases from 16.4 (Iter1) to 7.8 (Iter2) and finally to 5.4 (Iter3). This progressive enhancement validates our online learning paradigm: as the model improves, the quality of MCTS-generated preference data also increases, creating a virtuous cycle that enables sustained capability growth.

5.3 Analysis

Ablation Studies To analyze the contribution of each component in OSCAR, we conduct ablation studies on LLaVA1.5-7B using a single iteration. Results are shown in Tab. 2. Effect of Process Reward. Comparing Index 4 and 5, adding the process reward substantially reduces CHAIRS from 44.0 to 32.0, demonstrating that node-level hallucination feedback is essential for fine-grained guidance during tree search. Effect of Gated Outcome Reward. Comparing Index 3 and 5, incorporating the gated outcome reward reduces CHAIRS from 45.6 to 32.0, ensuring trajectory-level faithfulness that complements the local process reward. Effect of MCTS. Comparing Index 2 and 5, integrating MCTS dramatically reduces CHAIRS from 46.7 to 32.0, confirming that lookahead search is crucial for identifying tokens that may induce downstream hallucinations. The full model significantly outperforms all partial configurations, indicating that the three components work synergistically.

Analysis of online Learning To validate the effectiveness of on-policy learning, we compare three training strategies using 10k samples: (1) SFT with Qwen3-VL-8B-Instruct distilled data, (2) SFT with LLaVA's own generated data, and (3) SFT with chosen samples from our OSCAR-constructed preference data. Results are shown in Tab. 3. SFT with Qwen3VL distilled data increases CHAIR from 7.6 to 9.2 and Hal from 31.2 to 62.7, confirming the Supervision-Perception Mismatch discussed in Section 3.1. SFT with LLaVA's own generated data maintains similar performance but yields no improvement. In contrast, SFT with our OSCAR-constructed data substantially reduces CHAIR to 4.5, Hal to 15.4, and Cog to 1.4, demonstrating that our MCTS-guided data construction effectively leverages the model's discriminative capability to generate high-quality online training data.

Case Study Fig. 5 presents a qualitative comparison between LLaVA-1.5 and our OSCAR method. LLaVA-1.5 generates numerous hallucinated objects (highlighted in red) that LLaVA-1.5

The image features a man sitting at a dining table, holding a cell phone to his ear. He is surrounded by various items, including a bowl of food, a spoon, a fork, and a book. There are also two couches in the room, one on the left side and another on the right side.\n\nAdditionally, there are two chairs in the scene, one near the man and another further away. A bottle can be seen on the table, and a cup is placed nearby. A backpack is also present in the room, located on the right side of the scene.

OSCAR (Ours)

The image depicts a man sitting at a dining table, surrounded by various items. He is holding a cell phone to his ear while eating from a bowl containing food. The table is cluttered with a bowl of popcorn and another bowl with food.\n\nThere are several chairs around the table, with one near the man and others placed at different angles. Some CDs can be seen on the table, and a remote control is placed nearby. The scene includes everyday objects and personal items.

Figure 5: Qualitative comparison between LLaVA-1.5 and OSCAR (Ours). Hallucinated content is highlighted in red, while correct descriptions are shown in green. Our method generates fewer hallucinations.

are entirely absent from the image, while in contrast, our OSCAR method produces significantly fewer hallucinations. Moreover, the response generated by OSCAR exhibits better fluency and reduced redundancy, resulting in a more concise and coherent description. This demonstrates that our dual-granularity reward mechanism effectively balances visual faithfulness with overall response quality.

6 Conclusion

In this paper, we identified two key observations: a Supervision-Perception Mismatch in offline preference learning that unexpectedly worsen hallucinations, and a Generative-Discriminative Gap that provides reliable selfsupervision signals. Building on these insights, we proposed Iterative Self-Calibration (OSCAR), which integrates MCTS with a Dual-Granularity Reward Mechanism for online preference learning. Experiments demonstrated that OSCAR achieves state-of-the-art performance on hallucination benchmarks while preserving general multimodal capabilities. Our work highlights the importance of respecting the model's intrinsic perceptual boundaries, offering a new perspective for building more reliable vision-language systems.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant 62472420.

Contribution Statement

Minghui Chen and Chenxu Yang contributed equally to this work. Dayan Wu is the corresponding author.