{"type":"ScholarlyArticle","title":"CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning","authors":[{"name":"Zheqi He","slug":"zheqi-he","given":"Zheqi","family":"He","affiliation":"Beijing Academy of Artificial Intelligence"},{"name":"Xinya Wu","slug":"xinya-wu","given":"Xinya","family":"Wu","affiliation":"Beijing Academy of Artificial Intelligence"},{"name":"Pengfei Zhou","slug":"pengfei-zhou","given":"Pengfei","family":"Zhou","affiliation":"Beijing Academy of Artificial Intelligence; Beijing University of Post and Telecommunication"},{"name":"Richeng Xuan","slug":"richeng-xuan","given":"Richeng","family":"Xuan","affiliation":"Beijing Academy of Artificial Intelligence"},{"name":"Guang Liu","slug":"guang-liu","given":"Guang","family":"Liu","affiliation":"Beijing Academy of Artificial Intelligence"},{"name":"Xi Yang","slug":"xi-yang","given":"Xi","family":"Yang","affiliation":"Beijing Academy of Artificial Intelligence"},{"name":"Qiannan Zhu","slug":"qiannan-zhu","given":"Qiannan","family":"Zhu","affiliation":"Beijing Normal University"},{"name":"Hua Huang","slug":"hua-huang","given":"Hua","family":"Huang","affiliation":"Beijing Normal University"}],"abstract":"Multi-modal large language models(MLLMs) have achieved remarkable progress and demonstrated powerful knowledge comprehension and reasoning abilities. However, the mastery of domain-specific knowledge, which is essential for evaluating the intelligence of MLLMs, continues to be a challenge. Current multi-modal benchmarks for domain-specific knowledge concentrate on multiple-choice questions and are predominantly available in English, which imposes limitations on the comprehensiveness of the evaluation. To this end, we introduce CMMU, a novel benchmark for multi-modal and multi-type question understanding and reasoning in Chinese. CMMU consists of 3,603 questions in 7 subjects, covering knowledge from primary to high school. The questions can be categorized into 3 types: multiple-choice, multiple-response, and fill-in-the-blank, bringing greater challenges to MLLMs. In addition, we propose an evaluation strategy called Positional Error Variance for assessing multiple-choice questions. The strategy aims to perform a quantitative analysis of position bias. We evaluate seven open-source MLLMs along with GPT4-V, Gemini-Pro, and Qwen-VL-Plus. The results demonstrate that CMMU poses a significant challenge to the recent MLLMs. The data and code are available at https://github.com/FlagOpen/CMMU.","doi":"10.24963/ijcai.2024/92","url":"https://ijcai.org/proceedings/2024/92","pdf":"https://ijcai.org/proceedings/2024/0092.pdf","markdown":"https://ijcai.org/proceedings/2024/92.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/92","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":92,"pages":{"first":830,"last":838},"keywords":["Computer Vision: CV: Multimodal learning","Multidisciplinary Topics and Applications: MTA: Education"],"videos":[],"references":[{"position":1,"raw":"[Agrawal et al., 2019] Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. No- caps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8948-8957, 2019.","title":"Nocaps: Novel object captioning at scale","authors":"Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, Peter Anderson","year":"2019","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bai et al., 2023] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023.","title":"Qwen-vl: A versatile vision-language model for understanding, localization","authors":"Bai","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cai et al., 2023] Rizhao Cai, Zirui Song, Dayan Guan, Zhen- hao Chen, Xing Luo, Chenyu Yi, and Alex Kot. Benchlmm: Benchmarking cross-style visual capability of large multi- modal models. arXiv preprint arXiv:2312.02896, 2023.","title":"Benchlmm: Benchmarking cross-style visual capability of large multimodal models","authors":"Rizhao Cai, Zirui Song, Dayan Guan, Zhenhao Chen, Xing Luo, Chenyu Yi, Alex Kot","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chiang et al., 2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (ac- cessed 14 April 2023), 2023.","title":"Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality","authors":"Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez","year":"2023","venue":null,"doi":null,"url":"https://vicuna.lmsys.org(ac-cessed","internal_url":null},{"position":5,"raw":"[Dai et al., 2024] Wenliang Dai, Junnan Li, Dongxu Li, An- thony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with in- struction tuning. Advances in Neural Information Process- ing Systems, 36, 2024.","title":"Instructblip: Towards general-purpose vision-language models with instruction tuning","authors":"Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng, Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, Steven Hoi","year":"2024","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fu et al., 2023] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive eval- uation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023.","title":"Mme: A comprehensive evaluation benchmark for multimodal large language models","authors":"Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goyal et al., 2017] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understand- ing in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 6904-6913, 2017.","title":"Making the v in vqa matter: Elevating the role of image understanding in visual question answering","authors":"Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, Devi Parikh","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hudson and Manning, 2019] Drew A Hudson and Christo- pher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.","title":"Gqa: A new dataset for real-world visual reasoning and compositional question answering","authors":"A Drew, Christopher D Hudson, Manning","year":"2019","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kafle and Kanan, 2017] Kushal Kafle and Christopher Kanan. An analysis of visual question answering algo- rithms. In ICCV, 2017.","title":"An analysis of visual question answering algorithms","authors":"Kushal Kafle, Christopher Kanan","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kazemzadeh et al., 2014] Sahar Kazemzadeh, Vicente Or- donez, Mark Matten, and Tamara Berg. Referitgame: Re- ferring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787-798, 2014.","title":"Referitgame: Referring to objects in photographs of natural scenes","authors":"Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, Tamara Berg","year":"2014","venue":"Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2023a] Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed- bench-2: Benchmarking multimodal large language models. arXiv preprint arXiv:2311.17092, 2023.","title":"Seed-bench-2: Benchmarking multimodal large language models","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2023b] Bohao Li, Rui Wang, Guangzhi Wang, Yuy- ing Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023.","title":"Seed-bench: Benchmarking multimodal llms with generative comprehension","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2023c] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.","title":"Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2023d] Yanyang Li, Jianqiao Zhao, Duo Zheng, Zi- Yuan Hu, Zhi Chen, Xiaohui Su, Yongfeng Huang, Shijia Huang, Dahua Lin, Michael R Lyu, et al. Cleva: Chi- nese language models evaluation platform. arXiv preprint arXiv:2308.04813, 2023.","title":"Cleva: Chinese language models evaluation platform","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liang et al., 2022] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022.","title":"Holistic evaluation of language models","authors":"Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"Computer Vision-ECCV 2014: 13th European Conference","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2023a] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023.","title":"Improved baselines with visual instruction tuning","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2023b] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023.","title":"Visual instruction tuning","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2023c] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023.","title":"Mmbench: Is your multi-modal model an all-around player","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lu et al., 2022] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Mul- timodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.","title":"Learn to explain: Multimodal reasoning via thought chains for science question answering","authors":"Lu","year":"2022","venue":"The 36th Conference on Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Macfarlane, 2023] Donald Macfarlane. Professional report generation using lexeme theories and openai's generative pretrained transformer, gpt-4: A comparison. Medical Research Archives, 11(11), 2023.","title":"Professional report generation using lexeme theories and openai's generative pretrained transformer, gpt-4: A comparison","authors":"Donald Macfarlane","year":"2023","venue":"Medical Research Archives","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mialon et al., 2023] Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. arXiv preprint arXiv:2311.12983, 2023.","title":"Gaia: a benchmark for general ai assistants","authors":"Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann Lecun, Thomas Scialom","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[OpenAI, 2023] R OpenAI. Gpt-4 technical report. arXiv, pages 2303-08774, 2023.","title":"Gpt-4 technical report","authors":"Openai","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Plummer et al., 2015] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collect- ing region-to-phrase correspondences for richer image-to- sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649, 2015.","title":"Flickr30k entities: Collecting region-to-phrase correspondences for richer image-tosentence models","authors":"Liwei Bryan A Plummer, Chris M Wang, Juan C Cervantes, Julia Caicedo, Svetlana Hockenmaier, Lazebnik","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Singh et al., 2019] Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317-8326, 2019.","title":"Towards vqa models that can read","authors":"Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, Marcus Rohrbach","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sun et al., 2023] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, et al. Gen- erative multimodal models are in-context learners. arXiv preprint arXiv:2312.13286, 2023.","title":"Generative multimodal models are in-context learners","authors":"Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Team et al., 2023] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multi- modal models. arXiv preprint arXiv:2312.11805, 2023.","title":"Gemini: a family of highly capable multimodal models","authors":"Team","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Touvron et al., 2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023.","title":"Llama 2: Open foundation and finetuned chat models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2023] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023.","title":"Cogvlm: Visual expert for pretrained language models","authors":"Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xu et al., 2023] Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlm-ehub: A compre- hensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09265, 2023.","title":"Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models","authors":"Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, Ping Luo","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Ye et al., 2023] Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jin- gren Zhou. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arXiv:2311.04257, 2023.","title":"mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration","authors":"Ye","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yue et al., 2023] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Bo- tao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi- discipline multimodal understanding and reasoning bench- mark for expert agi. arXiv preprint arXiv:2311.16502, 2023.","title":"Mmmu: A massive multidiscipline multimodal understanding and reasoning benchmark for expert agi","authors":"Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, Wenhu Chen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2023] Wenxuan Zhang, Sharifah Mahani Alju- nied, Chang Gao, Yew Ken Chia, and Lidong Bing. M3exam: A multilingual, multimodal, multilevel bench- mark for examining large language models. arXiv preprint arXiv:2306.05179, 2023.","title":"M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models","authors":"Wenxuan Zhang, Sharifah Mahani Aljunied, Chang Gao, Ken Yew, Lidong Chia, Bing","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zheng et al., 2023] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023.","title":"Judging llm-as-a-judge with mt-bench and chatbot arena","authors":"Zheng","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhu et al., 2016] Yuke Zhu, Oliver Groth, Michael Bern- stein, and Li Fei-Fei. Visual7w: Grounded question answer- ing in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4995-5004, 2016.","title":"Visual7w: Grounded question answering in images","authors":"Yuke Zhu, Oliver Groth, Michael Bernstein, Li Fei-Fei","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhu et al., 2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xi- ang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models. arXiv preprint arXiv:2304.10592, 2023.","title":"Minigpt-4: Enhancing vision-language understanding with advanced large language models","authors":"Zhu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}