A Smart Multimodal Healthcare Copilot with Powerful LLM Reasoning

Zhao, Xuejiao; Liu, Siyan; Yang, Su-Yin; Miao, Chunyan

doi:10.24963/ijcai.2025/1278

A Smart Multimodal Healthcare Copilot with Powerful LLM Reasoning

Xuejiao Zhao, Siyan Liu, Su-Yin Yang, Chunyan Miao

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Demo Track. Pages 11132-11136.

https://doi.org/10.24963/ijcai.2025/1278

Keywords

Data Mining: DM: Knowledge graphs and knowledge base completion
Computer Vision: CV: Multimodal learning
Natural Language Processing: NLP: Language models
Knowledge Representation and Reasoning: KRR: Applications

Abstract

Misdiagnosis causes significant harm to healthcare systems worldwide, leading to increased costs and patient risks. MedRAG is a smart multimodal healthcare copilot equipped with powerful large language model (LLM) reasoning, designed to enhance medical decision-making. It supports multiple input modalities, including non-intrusive voice monitoring, general medical queries, and electronic health records. MedRAG provides recommendations on diagnosis, treatment, medication, and follow-up questioning. Leveraging retrieval-augmented generation enhanced by knowledge graph-elicited reasoning, MedRAG retrieves and integrates critical diagnostic insights, reducing the risk of misdiagnosis. It has been evaluated on both public and private datasets, outperforming existing models and offering more specific and accurate healthcare assistance. A demonstration video of MedRAG is available at: https://www.youtube.com/watch?v=PNIBDMYRfDM. The source code is available at: https://github.com/SNOWTEAM2023/MedRAG.

Xuejiao Zhao, Siyan Liu, Su-Yin Yang, and Chunyan Miao. 2025. A Smart Multimodal Healthcare Copilot with Powerful LLM Reasoning. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. IJCAI, 11132–11136. https://doi.org/10.24963/ijcai.2025/1278

@inproceedings{ijcai2025p1278,
  title     = {A Smart Multimodal Healthcare Copilot with Powerful LLM Reasoning},
  author    = {Zhao, Xuejiao and Liu, Siyan and Yang, Su-Yin and Miao, Chunyan},
  booktitle = {Proceedings of the Thirty-Fourth International Joint Conference on
               Artificial Intelligence, {IJCAI-25}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {James Kwok},
  pages     = {11132--11136},
  year      = {2025},
  month     = {8},
  note      = {Demo Track},
  doi       = {10.24963/ijcai.2025/1278},
  url       = {https://doi.org/10.24963/ijcai.2025/1278},
}

TY  - CPAPER
TI  - A Smart Multimodal Healthcare Copilot with Powerful LLM Reasoning
AU  - Zhao, Xuejiao
AU  - Liu, Siyan
AU  - Yang, Su-Yin
AU  - Miao, Chunyan
T2  - Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2025
SP  - 11132
EP  - 11136
DO  - 10.24963/ijcai.2025/1278
UR  - https://doi.org/10.24963/ijcai.2025/1278
ER  -