Localizing Before Answering: A Benchmark for Grounded Medical Visual Question Answering
Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 7670-7678.
https://doi.org/10.24963/ijcai.2025/853
Keywords
- Computer Vision: CV: Multimodal learning
- Multidisciplinary Topics and Applications: MTA: Health and medicine
Abstract
Medical Large Multi-modal Models (LMMs) have demonstrated remarkable capabilities in medical data interpretation. However, these models frequently generate hallucinations contradicting source evidence, particularly due to inadequate localization reasoning. This work reveals a critical limitation in current medical LMMs: instead of analyzing relevant pathological regions, they often rely on linguistic patterns or attend to irrelevant image areas when responding to disease-related queries.
To address this, we introduce HEAL-MedVQA (Hallucination Evaluation via Localization MedVQA), a comprehensive benchmark designed to evaluate LMMs' localization abilities and hallucination robustness. HEAL-MedVQA features (i) two innovative evaluation protocols to assess visual and textual shortcut learning, and (ii) a dataset of 67K VQA pairs, with doctor-annotated anatomical segmentation masks for pathological regions. To improve visual reasoning, we propose the Localize-before-Answer (LobA) framework, which trains LMMs to localize target regions of interest and self-prompt to emphasize segmented pathological areas, generating grounded and reliable answers. Experimental results demonstrate that our approach significantly outperforms state-of-the-art biomedical LMMs on the challenging HEAL-MedVQA benchmark, advancing robustness in medical VQA.
@inproceedings{ijcai2025p853,
title = {Localizing Before Answering: A Benchmark for Grounded Medical Visual Question Answering},
author = {Nguyen, Dung and Ho, Minh Khoi and Ta, Huy and Nguyen, Thanh Tam and Chen, Qi and Rav, Kumar and Dang, Quy Duong and Ramchandre, Satwik and Phung, Son Lam and Liao, Zhibin and To, Minh-Son and Verjans, Johan and Nguyen, Phi Le and Phan, Vu Minh Hieu},
booktitle = {Proceedings of the Thirty-Fourth International Joint Conference on
Artificial Intelligence, {IJCAI-25}},
publisher = {International Joint Conferences on Artificial Intelligence Organization},
editor = {James Kwok},
pages = {7670--7678},
year = {2025},
month = {8},
note = {Main Track},
doi = {10.24963/ijcai.2025/853},
url = {https://doi.org/10.24963/ijcai.2025/853},
}
TY - CPAPER TI - Localizing Before Answering: A Benchmark for Grounded Medical Visual Question Answering AU - Nguyen, Dung AU - Ho, Minh Khoi AU - Ta, Huy AU - Nguyen, Thanh Tam AU - Chen, Qi AU - Rav, Kumar AU - Dang, Quy Duong AU - Ramchandre, Satwik AU - Phung, Son Lam AU - Liao, Zhibin AU - To, Minh-Son AU - Verjans, Johan AU - Nguyen, Phi Le AU - Phan, Vu Minh Hieu T2 - Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25 PB - International Joint Conferences on Artificial Intelligence Organization PY - 2025 SP - 7670 EP - 7678 DO - 10.24963/ijcai.2025/853 UR - https://doi.org/10.24963/ijcai.2025/853 ER -Copied
Machine-readable versions of this paper, for tools and AI agents:
