Med-StepBench: A Hierarchical Reasoning Framework for Evaluating Hallucinations in Medical Vision-Language Models
Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. AI and Health. Pages 6841-6849.
https://doi.org/10.24963/ijcai.2026/761
Keywords
- Medical imaging: Medical imaging
- Multimodal data: Multimodal data
- Automated reasoning in clinical domains: Automated reasoning in clinical domains
Abstract
Large vision-language models (VLMs) demonstrate strong performance in medical image understanding, but frequently generate clinically plausible yet incorrect statements, raising significant safety concerns. Existing medical hallucination benchmarks primarily focus on 2D imaging with one-shot diagnostic questions, offering limited insight into whether predictions are grounded in correct localization and abnormality identification, allowing critical reasoning errors to remain hidden behind seemingly correct diagnoses. We introduce Med-StepBench, the first large-scale benchmark for step-wise hallucination detection in 3D oncological PET/CT, comprising over 12,000 images and more than 1,000,000 image–statement pairs across volumetric and multi-view 2D data, which decomposes clinical reasoning into four expert-designed diagnostic stages. Using clinician-verified annotations, we perform the first step-level evaluation of general-purpose and medical VLMs, revealing systematic failure modes obscured by aggregate accuracy metrics. Furthermore, we show that current VLMs are highly susceptible to adversarial yet clinically plausible intermediate explanations, which significantly amplify hallucinations despite contradictory visual evidence. Together, our findings highlight fundamental limitations in grounding multi-step clinical reasoning and establish Med-StepBench as a rigorous benchmark for developing safer and more reliable medical VLMs.
@inproceedings{ijcai2026p761,
title = {Med-StepBench: A Hierarchical Reasoning Framework for Evaluating Hallucinations in Medical Vision-Language Models},
author = {Nguyen, Minh Khoi and Le, Dai Lam and Jafari, Amir Reza and Nguyen, Tuan Dung and Mai, Hong Son and Mai, Huy Thong and Nguyen, Quang Huy and Nguyen, Thanh Trung and Farahbakhsh, Reza and Crespi, Noel and Nguyen, Phi Le},
booktitle = {Proceedings of the Thirty-Fifth International Joint Conference on
Artificial Intelligence, {IJCAI-26}},
publisher = {International Joint Conferences on Artificial Intelligence Organization},
editor = {Diego Calvanese},
pages = {6841--6849},
year = {2026},
month = {8},
note = {AI and Health},
doi = {10.24963/ijcai.2026/761},
url = {https://doi.org/10.24963/ijcai.2026/761},
}
TY - CPAPER TI - Med-StepBench: A Hierarchical Reasoning Framework for Evaluating Hallucinations in Medical Vision-Language Models AU - Nguyen, Minh Khoi AU - Le, Dai Lam AU - Jafari, Amir Reza AU - Nguyen, Tuan Dung AU - Mai, Hong Son AU - Mai, Huy Thong AU - Nguyen, Quang Huy AU - Nguyen, Thanh Trung AU - Farahbakhsh, Reza AU - Crespi, Noel AU - Nguyen, Phi Le T2 - Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26 PB - International Joint Conferences on Artificial Intelligence Organization PY - 2026 SP - 6841 EP - 6849 DO - 10.24963/ijcai.2026/761 UR - https://doi.org/10.24963/ijcai.2026/761 ER -Copied
Machine-readable versions of this paper, for tools and AI agents:
