{"type":"ScholarlyArticle","title":"Localizing Before Answering: A Benchmark for Grounded Medical Visual Question Answering","authors":[{"name":"Dung Nguyen","slug":"dung-nguyen","given":"Dung","family":"Nguyen","affiliation":""},{"name":"Minh Khoi Ho","slug":"minh-khoi-ho","given":"Minh Khoi","family":"Ho","affiliation":""},{"name":"Huy Ta","slug":"huy-ta","given":"Huy","family":"Ta","affiliation":""},{"name":"Thanh Tam Nguyen","slug":"thanh-tam-nguyen","given":"Thanh Tam","family":"Nguyen","affiliation":""},{"name":"Qi Chen","slug":"qi-chen","given":"Qi","family":"Chen","affiliation":""},{"name":"Kumar Rav","slug":"kumar-rav","given":"Kumar","family":"Rav","affiliation":""},{"name":"Quy Duong Dang","slug":"quy-duong-dang","given":"Quy Duong","family":"Dang","affiliation":""},{"name":"Satwik Ramchandre","slug":"satwik-ramchandre","given":"Satwik","family":"Ramchandre","affiliation":""},{"name":"Son Lam Phung","slug":"son-lam-phung","given":"Son Lam","family":"Phung","affiliation":""},{"name":"Zhibin Liao","slug":"zhibin-liao","given":"Zhibin","family":"Liao","affiliation":""},{"name":"Minh-Son To","slug":"minh-son-to","given":"Minh-Son","family":"To","affiliation":""},{"name":"Johan Verjans","slug":"johan-verjans","given":"Johan","family":"Verjans","affiliation":""},{"name":"Phi Le Nguyen","slug":"phi-le-nguyen","given":"Phi Le","family":"Nguyen","affiliation":""},{"name":"Vu Minh Hieu Phan","slug":"vu-minh-hieu-phan","given":"Vu Minh Hieu","family":"Phan","affiliation":""}],"abstract":"Medical Large Multi-modal Models (LMMs) have demonstrated remarkable capabilities in medical data interpretation. However, these models frequently generate hallucinations contradicting source evidence, particularly due to inadequate localization reasoning. This work reveals a critical limitation in current medical LMMs: instead of analyzing relevant pathological regions, they often rely on linguistic patterns or attend to irrelevant image areas when responding to disease-related queries.\r\nTo address this, we introduce HEAL-MedVQA (Hallucination Evaluation via Localization MedVQA), a comprehensive benchmark designed to evaluate LMMs' localization abilities and hallucination robustness. HEAL-MedVQA features (i) two innovative evaluation protocols to assess visual and textual shortcut learning, and (ii) a dataset of 67K VQA pairs, with doctor-annotated anatomical segmentation masks for pathological regions. To improve visual reasoning, we propose the Localize-before-Answer (LobA) framework, which trains LMMs to localize target regions of interest and self-prompt to emphasize segmented pathological areas, generating grounded and reliable answers. Experimental results demonstrate that our approach significantly outperforms state-of-the-art biomedical LMMs on the challenging HEAL-MedVQA benchmark, advancing robustness in medical VQA.","doi":"10.24963/ijcai.2025/853","url":"https://ijcai.org/proceedings/2025/853","pdf":"https://ijcai.org/proceedings/2025/0853.pdf","markdown":"https://ijcai.org/proceedings/2025/853.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/853","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":853,"pages":{"first":7670,"last":7678},"keywords":["Computer Vision: CV: Multimodal learning","Multidisciplinary Topics and Applications: MTA: Health and medicine"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}