Proceedings

MedDiT: A Knowledge-Controlled Diffusion Transformer Framework for Dynamic Medical Image Generation in Virtual Simulated Patient

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Demo Track. Pages 11082-11085.

https://doi.org/10.24963/ijcai.2025/1267

Abstract

Medical education relies heavily on Simulated Patients (SPs) to provide a safe environment for students to practice clinical skills, including medical image analysis. However, the high cost of recruiting qualified SPs and the lack of diverse medical imaging datasets have presented significant challenges. To address these issues, this paper introduces MedDiT, a novel knowledge-controlled conversational framework that can dynamically generate plausible medical images aligned with simulated patient symptoms, enabling diverse diagnostic skill training. Specifically, MedDiT integrates various patient Knowledge Graphs (KGs), which describe the attributes and symptoms of patients, to dynamically prompt Large Language Models' (LLMs) behavior and control the patient characteristics, mitigating hallucination during medical conversation. Additionally, a well-tuned Diffusion Transformer (DiT) model is incorporated to generate medical images according to the specified patient attributes in the KG. In this paper, we present the capabilities of MedDiT through a practical demonstration, showcasing its ability to act in diverse simulated patient cases and generate the corresponding medical images. This can provide an abundant and interactive learning experience for students, advancing medical education by offering an immersive simulation platform for future healthcare professionals. The work sheds light on the feasibility of incorporating advanced technologies like LLM, KG, and DiT in education applications, highlighting their potential to address the challenges faced in simulated patient-based medical education.

  1. [Baraheem et al., 2023] Samah Saeed Baraheem, Trung- Nghia Le, and Tam V Nguyen. Image synthesis: a review of methods, datasets, evaluation metrics, and future out- look. Artificial Intelligence Review, 56(10):10813-10865, 2023.
  2. [Benítez et al., 2024] Trista M Benítez, Yueyuan Xu, J Don- ald Boudreau, Alfred Wei Chieh Kow, Fernando Bello, Le Van Phuoc, Xiaofei Wang, Xiaodong Sun, Gilberto Ka- Kit Leung, Yanyan Lan, et al. Harnessing the potential of large language models in medical education: promise and pitfalls. Journal of the American Medical Informatics As- sociation, page ocad252, 2024.
  3. [Chen et al., 2023] Siyuan Chen, Mengyue Wu, Kenny Q Zhu, Kunyao Lan, Zhiling Zhang, and Lyuchun Cui. Llm-empowered chatbots for psychiatrist and patient sim- ulation: Application and evaluation. arXiv preprint arXiv:2305.13614, 2023.
  4. [Demner-Fushman et al., 2016] Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304-310, 2016.
  5. [Felix and Simon, 2019] Heidi M. Felix and Leslie V. Si- mon. Types of standardized patients and their recruitment in medical simulation. 2019.
  6. [Fensel et al., 2020] Dieter Fensel, Umutcan S ¸ims ¸ek, Kevin Angele, Elwin Huaman, Elias Kärle, Oleksandra Panasiuk, Ioan Toma, Jürgen Umbrich, Alexander Wahler, Dieter Fensel, et al. Introduction: what is a knowledge graph? Knowledge graphs: Methodology, tools and selected use cases, pages 1-10, 2020.
  7. [Gaba, 2007] David M Gaba. The future vision of simula- tion in healthcare. Simulation in Healthcare, 2(2):126- 135, 2007.
  8. [Glatard et al., 2012] Tristan Glatard, Carole Lartizien, Bernard Gibaud, Rafael Ferreira Da Silva, Germain Forestier, Frédéric Cervenansky, Martino Alessandrini, Hugues Benoit-Cattin, Olivier Bernard, Sorina Camarasu- Pop, et al. A virtual imaging platform for multi-modality medical image simulation. IEEE transactions on medical imaging, 32(1):110-118, 2012.
  9. [Gyrard et al., 2018] Amelie Gyrard, Manas Gaur, Saeedeh Shekarpour, Krishnaprasad Thirunarayan, and Amit Sheth. Personalized health knowledge graph. In CEUR workshop proceedings, volume 2317. NIH Public Access, 2018.
  10. [Hillier et al., 2020] Maureen Hillier, Tony L Williams, and Tiffani Chidume. Standardization of standardized patient training in medical simulation. 2020.
  11. [Holderried et al., 2024] Friederike Holderried, Christian Stegemann-Philipps, Lea Herschbach, Julia-Astrid Moldt, Andrew Nevins, Jan Griewatz, Martin Holderried, Anne Herrmann-Werner, Teresa Festl-Wietek, Moritz Mahling, et al. A generative pretrained transformer (gpt)-powered chatbot as a simulated patient to practice history taking: Prospective, mixed methods study. JMIR Medical Educa- tion, 10(1):e53961, 2024.
  12. [Hu et al., 2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  13. [Li et al., 2024a] Yaneng Li, Cheng Zeng, Jialun Zhong, Ruoyu Zhang, Minhao Zhang, and Lei Zou. Leveraging large language model as simulated patients for clinical ed- ucation. arXiv preprint arXiv:2404.13066, 2024.
  14. [Li et al., 2024b] Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, and Ying- fang Zhang et al. Hunyuan-dit: A powerful multi- resolution diffusion transformer with fine-grained chinese understanding, 2024.
  15. [Mesquita et al., 2010] Alessandra R Mesquita, Divaldo P Lyra Jr, Giselle C Brito, Blcie J Balisa-Rocha, Patrcia M Aguiar, and Abilio C de Almeida Neto. Developing com- munication skills in pharmacy: a systematic review of the use of simulated patient methods. Patient education and counseling, 78(2):143-148, 2010.
  16. [Pan et al., 2023] Shaoyan Pan, Tonghe Wang, Richard LJ Qiu, Marian Axente, Chih-Wei Chang, Junbo Peng, Ashish B Patel, Joseph Shelton, Sagar A Patel, Justin Roper, et al. 2d medical image synthesis us- ing transformer-based denoising diffusion probabilistic model. Physics in Medicine & Biology, 68(10):105004, 2023.
  17. [Sanko et al., 2013] Jill S Sanko, Ilya Shekhter, Richard R Kyle Jr, Stephen Di Benedetto, and David J Birnbach. Es- tablishing a convention for acting in healthcare simula- tion: merging art and science. Simulation in Healthcare, 8(4):215-220, 2013.
  18. [Wang et al., 2021] Tonghe Wang, Yang Lei, Yabo Fu, Ja- cob F Wynne, Walter J Curran, Tian Liu, and Xiaofeng Yang. A review on medical imaging synthesis using deep learning and its clinical applications. Journal of applied clinical medical physics, 22(1):11-36, 2021.
  19. [Yang et al., 2023a] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023.
  20. [Yang et al., 2023b] Shuxin Yang, Xian Wu, Shen Ge, Zhuozhao Zheng, S Kevin Zhou, and Li Xiao. Radi- ology report generation with a learned knowledge base and multi-modal alignment. Medical Image Analysis, 86:102798, 2023.
  21. [Yang et al., 2024] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, and Chang Zhou et al. Qwen2 tech- nical report. arXiv preprint arXiv:2407.10671, 2024.
  22. [Ziv et al., 2006] Amitai Ziv, Paul Root Wolpe, Stephen D Small, and Shimon Glick. Simulation-based medical ed- ucation: an ethical imperative. Simulation in Healthcare, 1(4):252-256, 2006.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards