Dynamic Multi-Path Retrieval for Knowledge-based Visual Question Answering

1 Introduction

Visual Question Answering (VQA) [Antol et al., 2015; Wu et al., 2023] integrates visual perception and language understanding and has become a standard benchmark for evaluating the reasoning capability of Multimodal Large Language Models (MLLMs) [Liu et al., 2023; Wang et al., 2024]. Recent progress in large-scale multimodal pretraining has led to substantial improvements in both accuracy and efficiency on a range of perception-centric tasks, including object recognition, counting, and attribute prediction. In contrast, Knowledge-based Visual Question Answering (KBVQA) requires reasoning over external knowledge that is not directly observable in the image, and thus remains a persistent challenge [Wu et al., 2024]. Current models rely heavily on parameterized knowledge acquired during pretraining, which provides limited coverage of long-tail entities and time-sensitive facts. As a result, these models are prone to knowledge hallucination and factual obsolescence during inference, as illustrated in the top row of Figure 1.

∗Corresponding author.

Recent studies adopt Retrieval-Augmented Generation (RAG) [Fan et al., 2024] to tackle KB-VQA by supplementing an MLLM's parametric knowledge with evidence retrieved from external knowledge databases [Deng et al., 2026a]. By grounding generation in retrieved documents, RAG expands coverage over long-tail entities and timesensitive facts, thereby reducing hallucinations and enabling more accurate answer generation in open-domain scenarios where the answer cannot be inferred from the image alone.

Despite its promise, reliable retrieval in multimodal settings remains challenging. Existing RAG-based KB-VQA approaches often suffer from two issues. First, many methods [Yan and Xie, 2024; Yang et al., 2025] retrieve candidate evidence from a single modality, which fails to fully exploit the heterogeneous and complementary signals in multimodal knowledge bases and may miss question-critical cues. Second, even when multiple modalities are involved [Hong et al., 2025], fusion is frequently static or coarse-grained, and cannot adapt to how different questions rely on visual evidence versus semantic knowledge [Deng et al., 2026b]. Consequently, irrelevant information may dominate the retrieved set, while crucial evidence is overlooked at the initial recall stage, which further propagates to downstream reranking and answer generation and degrades overall performance.

To address these limitations, we propose the Dynamic Multi-Path Retrieval framework (DMRAG). DMRAG explicitly decomposes retrieval into complementary views and coordinates them adaptively [Shang et al., 2023; Zhao et al., 2023; Wu et al., 2024]. The framework comprises two retrieval paths. The first is a fine-grained visual retrieval path that localizes question-relevant image regions and retrieves visually matched knowledge. The second is a generative semantic retrieval path that produces a hypothetical textual description and uses it to perform text-to-text retrieval over an external knowledge base. This design is motivated by the observation that visual similarity alone is unreliable when questions hinge on abstract concepts or factual context not visible in the image, while direct text retrieval suffers from the mismatch between short questions and long encyclopedic passages. To bridge this gap, we adapt Hypothetical Document Embeddings (HyDE) [Gao et al., 2023]: we generate a hypothetical document that approximates the target evidence and use it as the semantic query. A Question-Adaptive Gated Fusion (QGF) network estimates the relative importance of visual and semantic evidence and dynamically fuses the retrieved candidates. The fused candidates are further refined by a multimodal rearrangement module to support answer generation. Overall, DMRAG enables flexible, querydependent use of multimodal knowledge during retrieval.

The main contributions are summarized as follows:

• We propose DMRAG, a dynamic multi-path retrieval framework for KB-VQA that retrieves candidates through complementary ROI-based visual and HyDEbased semantic pathways, and fuses their evidence via QGF module to improve retrieval relevance and enable more accurate answer generation.
• We introduce hypothetical document generation for textto-text retrieval in KB-VQA, which improves retrieval relevance over unstructured textual knowledge.
• Extensive experiments on the E-VQA and InfoSeek benchmarks show that DMRAG consistently improves both retrieval recall and answer accuracy in practice compared with existing methods.

2 RelatedWork

2.1 Multimodal Large Language Models

Recent NLP and LLM studies have explored knowledgeenhanced learning and structural-semantic reasoning to improve model reliability in complex tasks [Deng et al., 2025]. MLLMs integrate visual and language modalities through large-scale visual-language pretraining, demonstrating strong cross-modal understanding and generation capabilities. They support tasks such as visual question answering, image understanding, and cross-modal inference under zero-shot or few-shot settings. With improved scale and pretraining strategies, MLLMs also exhibit open-domain knowledge reasoning capabilities, making them a foundation for knowledgeenhanced visual question answering.

2.2 Knowledge-based Visual Question Answering

Knowledge-based Visual Question Answering (KB-VQA) focuses on problems that cannot be answered solely based on image content. Datasets such as OK-VQA [Marino et al., 2019] and A-OKVQA [Schwenk et al., 2022] involve questions requiring external knowledge, including general commonsense knowledge that is not unique. E-VQA [Mensink et al., 2023] and InfoSeek [Chen et al., 2023] further extend this by including images and textual content fromWikipedia, requiring models to retrieve relevant background knowledge of entities for reasoning and answer generation.

Early research mainly explored incorporating RAG into MLLMs. For example, Wiki LaVA [Caffagni et al., 2024] adopts a hierarchical retrieval strategy from coarse to fine, first performing document-level recall, followed by paragraph-level filtering, and injecting the retrieved text as context, effectively mitigating knowledge noise. EchoSight [Yan and Xie, 2024] proposes a retrieval pipeline of visual first, multimodal rearrangement to balance visual and semantic relevance while maintaining high recall. From the perspective of retrieval granularity, OMGM [Yang et al., 2025] gradually narrows the retrieval space through a multigranularity collaborative mechanism, achieving a more favorable trade-off between efficiency and accuracy.

Beyond retrieval structure design, some works focus on retrieval noise reduction and knowledge effectiveness. WikiPRF [Hong et al., 2025] introduces a three-stage processretrieval-filter framework, leveraging reinforcement learning to guide query formulation and explicitly filter candidate knowledge. MMKB-RAG [Ling et al., 2025] utilizes the intrinsic knowledge boundaries of the model to generate semantic labels and jointly screen retrieval results. In the generation stage, approaches such as ReflectiVA [Cocchi et al., 2025] and mR 2 AG [Zhang et al., 2024] incorporate reflection mechanisms, allowing the model to evaluate the adequacy of retrieved content and trigger re-retrieval when necessary, thus enhancing the feedback capability of the RAG system. Additionally, mkg-RAG [Yuan et al., 2025] integrates multimodal knowledge graphs into RAG to enhance multi-hop reasoning.

Overall, existing KB-VQA methods have progressively advanced in hierarchical retrieval, rearrangement and filtering, reflection mechanisms, and knowledge structuring. Nevertheless, fully exploiting multimodal knowledge bases and modeling queries from multiple complementary retrieval perspectives remains an open problem. Motivated by this, this work proposes DMRAG to enable dynamic collaborative fusion of retrieval results frommultiple sources.

3 Method

3.1 Overview

The task of KB-VQA requires reasoning over multimodal inputs by leveraging external knowledge. We utilize a multimodal knowledge base comprising paired text articles and associated images as the source for retrieval. As shown in Figure 2, we design two complementary retrieval pathways. The ROI-based visual pathway focuses on fine-grained entities by extracting salient regions and encoding region-level features for precise visual matching. In parallel, the HyDE-based semantic pathway generates a hypothetical encyclopedic paragraph to enrich the query, improving alignment with the knowledge base text and narrowing the visual-semantic gap. We then perform Question-Adaptive Gated Fusion (QGF) to merge candidates from both pathways into a unified evidence pool. Finally, a cross-modal reranker refines the fused set and selects the most relevant evidence for answer generation.

3.2 ROI-based Visual Retrieval

Global image embeddings often include background cues that dilute the evidence of the entity in the question [Zhu et al., 2015]. Thus, retrieval based only on global features may return irrelevant images. To address this issue, we adopt an Entity-Driven Salient Region Retrieval strategy. The core idea is to localize the Region of Interest (ROI) corresponding to the query entity and perform region-level image retrieval.

Entity Extraction

We first identify the key subject in the question Q. We use spaCy to parse Q and extract named entities and noun phrases. To cover the main entities in the question, we retain the top-M entities, where M = 3:

ROI Localization

To ground the extracted entities in the visual content, we use Grounding DINO [Liu et al., 2024b] as an open-set object detector. Given the image I and the entity set E, the detector outputs candidate bounding boxes with confidence scores:

where bk denotes the coordinates of the k-th box and sk is its confidence score. We select the ROI as the box with the highest confidence:

Visual Retrieval

We crop I using b∗ to suppress background distractors. The cropped region is encoded by the frozen EVA-CLIP image encoder fv, and the feature is ℓ2 normalized:

All knowledge base images are pre-encoded into {vKB j } with the same encoder and indexed by FAISS. We compute cosine similarity to score each candidate:

We then retrieve the top-K visual candidates:

3.3 HyDE-based Semantic Retrieval

Visual similarity alone is unreliable when the question depends on abstract concepts or factual context that is not visible in the image. Direct text retrieval also suffers from a mismatch between short questions and long encyclopedic passages. We bridge this mismatch using a generated hypothetical passage as the semantic query. Specifically, we generate a hypothetical document that approximates the target evidence, and we use it as the semantic query.

Hypothetical Document Generation

We prompt a Multimodal Large Language Model, denoted as M, to synthesize a paragraph dhypo. The generation is conditioned on the image I, the question Q, and an instruction prompt PHyDE that enforces a Wikipedia-style description:

This paragraph rewrites the user intent into a form that is closer to the knowledge base text distribution.

Semantic Retrieval

We encode dhypo with the frozen EVA-CLIP text encoder ft and normalize it to obtain the semantic query vector:

The knowledge base texts are pre-encoded as {tj}. We score each entry by cosine similarity:

We then retrieve the top-K semantic candidates:

3.4 QGFmodule

The ROI-based path emphasizes visual attributes, while the HyDE-based path captures factual context. Their relative importance varies across questions, so fixed fusion weights are suboptimal. We therefore learn an adaptive fusion module [Mi et al., 2025] to predict question-specific weights.

Explicit Supervision Construction

To supervise the gating module, we build oracle weights based on retrieval quality. Let k∗ be the ground-truth knowledge entry. For each path, we compute the rank of k∗ among its top-K results:

where path ∈ {ROI, HyDE}. We convert ranks into relevance scores via vpath = (rpath+ϵ)-1 and normalize them with Softmax to obtain oracle weights:

Gating Network and Optimization

We use a lightweight MLP Gθ as the gating network. It maps the question embedding q to predicted fusion weights:

We train Gθ by minimizing the KL divergence between w∗ and wpred, where wp denotes the p-th element of wpred:

Score Normalization and Fusion

At inference time, the two retrieval scores may follow different distributions, which makes them hard to combine directly. We apply Min-Max normalization within each path:

We then compute the fused score using the predicted weights:

sfused(kj) = wROI · ˜ sROI(kj) +wHyDE · ˜ sHyDE(kj ). (16) Candidates are ranked by sfused, and the top-K entries are passed to reranking.

3.5 Reranking and Answer Generation

The retrieval stage is designed for high recall, so its top results can still include distractors. To further improve precision, we adopt a Q-Former-based reranking module following EchoSight [Yan and Xie, 2024]. The reranker takes the image, the question, and each retrieved text candidate as input and computes fine-grained relevance through cross-attention. The highest-ranked knowledge entry is then used as context for the Large Language Model, which produces an answer explicitly grounded in the selected evidence.

4 Experiments

4.1 Datasets

We evaluate our method on two KB-VQA benchmark datasets, E-VQA [Mensink et al., 2023] and InfoSeek [Chen et al., 2023]. E-VQA contains 221,000 question-answer pairs associated with 16,700 fine-grained Wikipedia entities. The images in this dataset are sourced from iNaturalist 2021 (iNat21) and the Google Landmarks Dataset v2 (GLDv2). EVQA is equipped with a knowledge base consisting of approximately 2 million Wikipedia documents, each containing a title, textual paragraphs, and related images. In our experiments, we use the full knowledge base provided by E-VQA.

InfoSeek contains approximately 1.3 million visual information-seeking questions, covering more than 11K visual entities from the OVEN dataset [Hu et al., 2023]. We use a subset of 100,000 documents constructed in [Yan and Xie, 2024] as our knowledge base.

4.2 EvaluationMetrics

We evaluate our method from two aspects: retrieval performance and question answering performance.

Table 2: VQA accuracy scores on E-VQA test set and InfoSeek validation set where all results from retrieval-augmented models are reported without considering any re-ranking stage to reorder retrieved web pages. Bold indicates state-of-the-art, underline denotes second-best. results that are not directly comparable due to different knowledge bases. All retrieval-augmented results exclude re-ranking.

For the retrieval stage, we adopt Recall@K as the primary evaluation metric. A retrieval is considered successful in this setting only when the URL of the retrieved Wikipedia article exactly matches the target article.

For the question answering stage, to ensure fair comparison, we follow prior work by using VQA Accuracy [Goyal et al., 2017; Marino et al., 2019] and Relaxed Accuracy [Methani et al., 2020; Masry et al., 2022] for InfoSeek, and the BEM score [Zhang et al., 2019] for E-VQA.

4.3 Implementation Details

Initial Retrieval

In our experiments, we primarily employ EVA-CLIP-8B [Sun et al., 2023] for multimodal retrieval. For the ROI-based path, we adopt an image-to-image retrieval strategy, comparing the query image with images embedded in Wikipedia documents to retrieve the corresponding pages. For the HyDE

path, we perform text-to-text retrieval: generates hypothetical paragraphs using Qwen2.5-VL-7B-Instruct [Bai et al., 2025], which are then embedded using EVA-CLIP-8B. Both ROI and HyDE features are encoded in the same embedding space as the knowledge base's textual and visual vectors to ensure alignment. The ROI path additionally leverages SpaCy for text preprocessing and Grounding DINO [Liu et al., 2024b] for region proposals and ROI extraction.

Gating Network

The gating network is trained on a randomly sampled 50K subset from the InfoSeek and E-VQA training sets. For each training sample, the initial retrieval results from both ROI and HyDE paths are processed to compute oracle fusion weights based on the ranks of the ground-truth knowledge entries. These oracle weights serve as supervision for the gating network. The network itself is a simple and lightweight MLP with two hidden layers of size [512, 256], ReLU activations,

Retrieval Performance Comparison: HyDE vs Caption

and Dropout rate 0.1. It takes 1280-dimensional question embeddings extracted by EVA-CLIP-8B as input and outputs a 2-dimensional softmax vector representing the fusion weights for ROI and HyDE. The network is optimized using Adam with learning rate 10-3 , weight decay 10-5 , batch size 64, for a total of 50 epochs. The loss function is the KL divergence between the predicted and oracle fusion weights. A ReduceLROnPlateau scheduler with factor 0.5 and patience 5 is applied, and early stopping is used if the validation loss does not decrease for 10 consecutive epochs.

Reranker

Following the EchoSight paradigm [Yan and Xie, 2024], we adopt a Q-Former-based reranker to refine the top-K candidates obtained from the QGF module. The encoder is initialized from pre-trained Q-Former weights provided by the LAVIS library [Li et al., 2023a], and the first 32 embeddings are selected as the multimodal feature matrix corresponding to query token positions. Since InfoSeek training samples generally do not contain labeled evidence, the Q-Former is trained on the E-VQA training subset and then evaluated on both E-VQA and InfoSeek test sets. All data processing and hyperparameter settings are kept consistent with EchoSight to allow fair comparison.

4.4 Experimental Results

Retrieval Performance

Table 1 compares the recall performance of different methods on E-VQA and InfoSeek datasets. Overall, our method achieves consistent and substantial gains on both datasets, indicating that our multi-path retrieval, adaptive fusion, and cross-modal reranking align questions with knowledge entries more effectively. On E-VQA, even without reranking, our method reaches R@1=49.0% with only the two retrieval pathways and adaptive weighted fusion, surpassing all baselines and even exceeding the strong OMGM baseline with reranking (42.8%). This suggests that our method is already strong at the first-stage retrieval, pushing the target entry to higher ranks and thus providing higher quality evidence candidates for downstream answer generation. After applying the reranker, R@1 further increases to 55.0%. Similar trends are observed on InfoSeek, where our method achieves R@1 of 65.5% with reranking, surpassing previous state-of-the-art methods such as OMGM (64.0%) and ReflectiVA [Cocchi et al., 2025] (56.1%). Overall, the combination of ROI+HyDE with adaptive weighting and reranking enables our method to improve both retrieval coverage and precision.

VQA Accuracy

Table 2 reports the VQA accuracy of various zero-shot and retrieval-augmented models, considering only results without reranking. DMRAG consistently outperforms all baselines across both datasets, showing that improvements stem from stronger first-stage retrieval and better utilization of retrieved evidence rather than reranking. On E-VQA, our method achieves 48.4% in Single-Hop and 42.2% in Allquestion settings, higher than previous retrieval-augmented models [Caffagni et al., 2024; Yan and Xie, 2024; Ling et al., 2025; Cocchi et al., 2025], reflecting the benefits of combining ROI-based visual retrieval and HyDE-based semantic retrieval with the query-adaptive gating module. This enables DMRAG to capture both fine-grained visual cues and abstract textual knowledge, resulting in more accurate answers. On InfoSeek, our method achieves the highest scores across all categories, including 41.9% on unseen questions and 41.7% on unseen entities, highlighting strong generalization to unseen content and handling of long-tail entities. Compared with baselines, DMRAG narrows the gap between zero-shot and retrieval-augmented approaches, indicating that multi-path retrieval with adaptive fusion prioritizes relevant evidence for each query, mitigates noise from irrelevant candidates, and improves VQA accuracy. These results demonstrate that our framework strengthens first-stage retrieval and enhances answer generation even without reranking, establishing a foundation for further gains when incorporating post-retrieval refinements, and underscoring the effectiveness of dynamic multi-path retrieval and query-dependent evidence fusion in knowledge-based visual question answering.

4.5 Ablation Study

All ablation experiments are conducted on the E-VQA dataset to analyze the contribution of individual components in our retrieval framework. We first present a component-wise ablation by progressively introducing each module into the sys-

tem, with results summarized in Table 3. Based on this main ablation, we further investigate two key design choices: the effectiveness of HyDE for semantic retrieval and the necessity of QGF module.

Ablation of retrieval framework components

Table 3 reports the retrieval performance as different components are progressively introduced. Among single-path baselines, ROI-only performs the worst (R@20=50.08), indicating that pure visual matching is insufficient for encyclopedic KB-VQA. In contrast, HyDE-only achieves substantially higher recall at high K (R@20=69.38), further highlighting the importance of semantic retrieval. Building directly upon this, introducing the QGF module further improves high-K recall (R@20=77.70), demonstrating that query-dependent weighting effectively integrates complementary retrieval paths and increases candidate coverage. Finally, adding the reranking module leads to additional gains, particularly at lower K values, confirming that refining candidates after fusion improves overall ranking quality.

Impact of HyDE Compared with Image Caption

We justify the use of HyDE for the text retrieval branch by comparing it with a naive image-caption querying baseline, as shown in Figure 3. HyDE delivers substantially better retrieval performance across Recall@K, especially at higher K values. This is because image captions mainly describe visible content in the image, whereas HyDE generates a hypothetical encyclopedic paragraph that anticipates what answerrelevant evidence should look like in Wikipedia, resulting in stronger semantic alignment with knowledge-base text.

Impact of adaptive fusion

Finally, we analyze the necessity of adaptive gated fusion using the Recall@K growth curves in Figure 4 together with the results in Table 3. Although fixed 50%-50% fusion improves overall recall compared to ROI-only, it remains limited in high-K coverage, with R@20 reaching only 69.49, which is comparable to HyDE-only (69.38). This indicates that naive averaging fails to fully exploit the complementary strengths of different paths and may introduce noise from weaker signals. In contrast, adaptive gated fusion achieves a substantial improvement at high K (R@20=77.70), and its Recall@K curve gradually approaches the Best Fusion upper bound, demonstrating that the gating mechanism effectively approximates per-query optimal path selection.

4.6 Qualitative Results

We provide qualitative examples to illustrate the effect of multi-path retrieval and adaptive fusion in DMRAG. Figure 5 shows the top-1 retrieved document (Recall@1) fromHyDE, ROI, and the QGF module, together with the fusion weights. In our evaluation, a Recall@1 hit indicates a correct answer: ✓ denotes a successful retrieval and ✗ denotes a retrieval failure. These examples suggest that HyDE and ROI are complementary, and the gating module adaptively selects the more relevant source for each question.

5 Conclusion

We propose DMRAG, a dynamic multi-path retrieval framework for KB-VQA. By effectively combining ROI-based visual retrieval and HyDE-based semantic retrieval with the QGF module, the framework enables flexible and querydependent use of multimodal knowledge. Despite its effectiveness, DMRAG remains dependent on the coverage and quality of external knowledge sources; future work will explore more robust retrieval strategies and improved integration between retrieval and reasoning.

Acknowledgments

This work is in part funded by the National Natural Science Foundation of China (Grant No. 62372364) and the Technical Innovation Guidance Plan of Shaanxi Province, China (Grant No. 2024QCY-KXJ-199).

Contribution Statement

The first two authors, Zeyu Song and Yimin Deng, contributed equally to this work.