Cyclic Vision-Language Manipulator: Towards Reliable and Fine-Grained Image Interpretation for Automated Report Generation

Fang, Yingying; Jin, Zihao; Guo, Shaojie; Liu, Jinda; Yue, Zhiling; Gao, Yijian; Ning, Junzhi; Li, Zhi; Walsh, Simon; Yang, Guang

doi:10.24963/ijcai.2025/41

Cyclic Vision-Language Manipulator: Towards Reliable and Fine-Grained Image Interpretation for Automated Report Generation

Yingying Fang, Zihao Jin, Shaojie Guo, Jinda Liu, Zhiling Yue, Yijian Gao, Junzhi Ning, Zhi Li, Simon Walsh, Guang Yang

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 357-366.

https://doi.org/10.24963/ijcai.2025/41

Keywords

AI Ethics, Trust, Fairness: ETF: Explainability and interpretability
AI Ethics, Trust, Fairness: ETF: Trustworthy AI
Computer Vision: CV: Vision, language and reasoning

Abstract

Despite significant advancements in automated report generation, the opaqueness of text interpretability continues to cast doubt on the reliability of the content produced. This paper introduces a novel approach to identify specific image features in X-ray images that influence the outputs of report generation models. Specifically, we propose Cyclic Vision-Language Manipulator (CVLM), a module to generate a manipulated X-ray from an original X-ray and its report from a designated report generator. The essence of CVLM is that cycling manipulated X-rays to the report generator produces altered reports aligned with the alterations pre-injected into the reports for X-ray generation, achieving the term ``cyclic manipulation''. This process allows direct comparison between original and manipulated X-rays, clarifying the critical image features driving changes in reports and enabling model users to assess the reliability of the generated texts. Empirical evaluations demonstrate that CVLM can identify more precise and reliable features compared to existing explanation methods, significantly enhancing the transparency and applicability of AI-generated reports.

Yingying Fang, Zihao Jin, Shaojie Guo, Jinda Liu, Zhiling Yue, Yijian Gao, Junzhi Ning, Zhi Li, Simon Walsh, and Guang Yang. 2025. Cyclic Vision-Language Manipulator: Towards Reliable and Fine-Grained Image Interpretation for Automated Report Generation. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. IJCAI, 357–366. https://doi.org/10.24963/ijcai.2025/41

@inproceedings{ijcai2025p41,
  title     = {Cyclic Vision-Language Manipulator: Towards Reliable and Fine-Grained Image Interpretation for Automated Report Generation},
  author    = {Fang, Yingying and Jin, Zihao and Guo, Shaojie and Liu, Jinda and Yue, Zhiling and Gao, Yijian and Ning, Junzhi and Li, Zhi and Walsh, Simon and Yang, Guang},
  booktitle = {Proceedings of the Thirty-Fourth International Joint Conference on
               Artificial Intelligence, {IJCAI-25}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {James Kwok},
  pages     = {357--366},
  year      = {2025},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2025/41},
  url       = {https://doi.org/10.24963/ijcai.2025/41},
}

TY  - CPAPER
TI  - Cyclic Vision-Language Manipulator: Towards Reliable and Fine-Grained Image Interpretation for Automated Report Generation
AU  - Fang, Yingying
AU  - Jin, Zihao
AU  - Guo, Shaojie
AU  - Liu, Jinda
AU  - Yue, Zhiling
AU  - Gao, Yijian
AU  - Ning, Junzhi
AU  - Li, Zhi
AU  - Walsh, Simon
AU  - Yang, Guang
T2  - Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2025
SP  - 357
EP  - 366
DO  - 10.24963/ijcai.2025/41
UR  - https://doi.org/10.24963/ijcai.2025/41
ER  -