M4Bench: A Benchmark of Multi-domain Multi-granularity Multi-image Understanding for Multi-modal Large Language Models

Ye, Xiaojun; Liang, Guanbao; Wang, Chun; Li, Liangcheng; Ke, Pengfei; Wang, Rui; Jia, Bingxin; Huang, Gang; Sun, Qiao; Zhou, Sheng

doi:10.24963/ijcai.2025/762

M4Bench: A Benchmark of Multi-domain Multi-granularity Multi-image Understanding for Multi-modal Large Language Models

Xiaojun Ye, Guanbao Liang, Chun Wang, Liangcheng Li, Pengfei Ke, Rui Wang, Bingxin Jia, Gang Huang, Qiao Sun, Sheng Zhou

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 6848-6856.

https://doi.org/10.24963/ijcai.2025/762

Keywords

Machine Learning: ML: Benchmarks

Abstract

The increasing demands in analyzing complex associated scenes pose necessities to researching multi-image understanding abilities. Compared with understanding individual images, both the alignments and differences between images are essential aspects of understanding the intricate relationships for multi-image inference tasks. However, existing benchmarks face difficulties in addressing both of these aspects simultaneously, resulting in obstacles to modeling relationships under various granularities and domains of images. In this paper, we introduce M4Bench to enhance the capability of aligning and distinguishing multi-images with multi-domain multi-granularity comparison. We carefully design five comparison tasks related to coarse and fine-grained granularities in single and multiple domains of images and evaluate them on 13 state-of-the-art multi-modal large language models with various sizes. Besides, we analyze the evaluation results and provide several observations and viewpoints for the multi-image understanding research. The data and evaluation code are available at https://github.com/eaglelab-zju/M4Bench.

Xiaojun Ye, Guanbao Liang, Chun Wang, Liangcheng Li, Pengfei Ke, Rui Wang, Bingxin Jia, Gang Huang, Qiao Sun, and Sheng Zhou. 2025. M4Bench: A Benchmark of Multi-domain Multi-granularity Multi-image Understanding for Multi-modal Large Language Models. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. IJCAI, 6848–6856. https://doi.org/10.24963/ijcai.2025/762

@inproceedings{ijcai2025p762,
  title     = {M4Bench: A Benchmark of Multi-domain Multi-granularity Multi-image Understanding for Multi-modal Large Language Models},
  author    = {Ye, Xiaojun and Liang, Guanbao and Wang, Chun and Li, Liangcheng and Ke, Pengfei and Wang, Rui and Jia, Bingxin and Huang, Gang and Sun, Qiao and Zhou, Sheng},
  booktitle = {Proceedings of the Thirty-Fourth International Joint Conference on
               Artificial Intelligence, {IJCAI-25}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {James Kwok},
  pages     = {6848--6856},
  year      = {2025},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2025/762},
  url       = {https://doi.org/10.24963/ijcai.2025/762},
}

TY  - CPAPER
TI  - M4Bench: A Benchmark of Multi-domain Multi-granularity Multi-image Understanding for Multi-modal Large Language Models
AU  - Ye, Xiaojun
AU  - Liang, Guanbao
AU  - Wang, Chun
AU  - Li, Liangcheng
AU  - Ke, Pengfei
AU  - Wang, Rui
AU  - Jia, Bingxin
AU  - Huang, Gang
AU  - Sun, Qiao
AU  - Zhou, Sheng
T2  - Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2025
SP  - 6848
EP  - 6856
DO  - 10.24963/ijcai.2025/762
UR  - https://doi.org/10.24963/ijcai.2025/762
ER  -