{"type":"ScholarlyArticle","title":"3DBench: A Scalable 3D Benchmark and Instruction-Tuning Dataset","authors":[{"name":"Junjie Zhang","slug":"junjie-zhang","given":"Junjie","family":"Zhang","affiliation":"Shanghai University"},{"name":"Tianci Hu","slug":"tianci-hu","given":"Tianci","family":"Hu","affiliation":"Shanghai University"},{"name":"Xiaoshui Huang","slug":"xiaoshui-huang","given":"Xiaoshui","family":"Huang","affiliation":"Shanghai AI Laboratory"},{"name":"Yongshun Gong","slug":"yongshun-gong","given":"Yongshun","family":"Gong","affiliation":"Shandong University"},{"name":"Dan Zeng","slug":"dan-zeng","given":"Dan","family":"Zeng","affiliation":"Shanghai University"}],"abstract":"Evaluating the performance of Multi-modal Large Language Models (MLLMs), integrating both point cloud and language, presents significant challenges. The lack of a comprehensive assessment hampers determining whether these models truly represent advancements, thereby impeding further progress in the field. Current evaluations heavily rely on classification and caption tasks, falling short in providing a thorough assessment of MLLMs. A pressing need exists for a more sophisticated evaluation method capable of thoroughly analyzing the spatial understanding and expressive capabilities of these models. To address these issues, we introduce a scalable 3D benchmark, accompanied by a large-scale instruction-tuning dataset known as 3DBench, providing an extensible platform for a comprehensive evaluation of MLLMs. Specifically, we establish the benchmark that spans a wide range of spatial and semantic scales, from object-level to scene-level, addressing both perception and planning tasks. Furthermore, we present a rigorous pipeline for automatically constructing scalable 3D instruction-tuning datasets, covering 10 diverse multi-modal tasks with more than 0.23 million QA pairs generated in total. Thorough experiments evaluating trending MLLMs, comparisons against existing datasets, and variations of training protocols demonstrate the superiority of 3DBench, offering valuable insights into current limitations and potential research directions. Codes are available at https://github.com/Inshsang/3DBench.","doi":"10.24963/ijcai.2024/189","url":"https://ijcai.org/proceedings/2024/189","pdf":"https://ijcai.org/proceedings/2024/0189.pdf","markdown":"https://ijcai.org/proceedings/2024/189.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/189","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":189,"pages":{"first":1706,"last":1714},"keywords":["Computer Vision: CV: Scene analysis and understanding","Computer Vision: CV: 3D computer vision","Computer Vision: CV: Multimodal learning"],"videos":[],"references":[{"position":1,"raw":"[Banerjee and Lavie, 2005] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. Meeting of the Association for Computational Linguistics,Meeting of the Association for Computational Linguistics, Jun 2005.","title":"Meteor: An automatic metric for mt evaluation with improved correlation with human judgments","authors":"Lavie Banerjee, Satanjeev Banerjee, Alon Lavie","year":"2005","venue":"Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chang et al., 2015] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015.","title":"Shapenet: An information-rich 3d model repository","authors":"Thomas Angel X Chang, Leonidas Funkhouser, Pat Guibas, Qixing Hanrahan, Zimo Huang, Silvio Li, Manolis Savarese, Shuran Savva, Hao Song, Su","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dai et al., 2024] Wenliang Dai, Junnan Li, Dongxu Li, An- thony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with in- struction tuning. Advances in Neural Information Process- ing Systems, 36, 2024.","title":"Instructblip: Towards general-purpose vision-language models with instruction tuning","authors":"Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng, Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, Steven Hoi","year":"2024","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deitke et al., 2022] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Win- son Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using pro- cedural generation. Advances in Neural Information Pro- cessing Systems, 35:5982-5994, 2022.","title":"Procthor: Large-scale embodied ai using procedural generation","authors":"Matt Deitke, Eli Vanderbilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, Roozbeh Mottaghi","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deitke et al., 2023] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Lud- wig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d ob- jects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142- 13153, 2023.","title":"Objaverse: A universe of annotated 3d objects","authors":"Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli Vanderbilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, Ali Farhadi","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dubois et al., 2023] Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Al- pacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387, 2023.","title":"Alpacafarm: A simulation framework for methods that learn from human feedback","authors":"Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, Tatsunori B Hashimoto","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fu et al., 2022] Yao Fu, Hao Peng, and Tushar Khot. How does gpt obtain its ability? tracing emergent abilities of language models to their sources. Yao Fu's Notion, 2022.","title":"How does gpt obtain its ability? tracing emergent abilities of language models to their sources","authors":"Yao Fu, Hao Peng, Tushar Khot","year":"2022","venue":"Yao Fu's Notion","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fu et al., 2023] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language mod- els. arXiv preprint arXiv:2306.13394, 2023.","title":"Mme: A comprehensive evaluation benchmark for multimodal large language models","authors":"Fu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guo et al., 2023] Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309.00615, 2023.","title":"Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following","authors":"Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hao et al., 2022] Yaru Hao, Haoyu Song, Li Dong, Shao- han Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. Language models are general-purpose inter- faces. arXiv preprint arXiv:2206.06336, 2022.","title":"Language models are general-purpose interfaces","authors":"Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, Furu Wei","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kolve et al., 2017] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. AI2- THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017.","title":"AI2-THOR: An Interactive 3D Environment for Visual AI","authors":"Kolve","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.","title":"Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, Septem- ber 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"Computer Vision-ECCV 2014: 13th European Conference","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin, 2004] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. Meeting of the Association for Computational Linguistics,Meeting of the Association for Computational Linguistics, Jul 2004.","title":"Rouge: A package for automatic evaluation of summaries","authors":"Chin-Yew Lin","year":"2004","venue":"Meeting of the Association for Computational Linguistics,Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2023] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023.","title":"Mmbench: Is your multi-modal model an all-around player","authors":"Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Luo et al., 2023] Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pre- trained models. arXiv preprint arXiv:2306.07279, 2023.","title":"Scalable 3d captioning with pretrained models","authors":"Luo","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[OpenAI, 2023] R OpenAI. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2:13, 2023.","title":"Gpt-4 technical report","authors":"Openai","year":"2023","venue":"View in Article","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Papineni et al., 2001] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu. In Proceedings of the 40th Annual Meeting on Association for Computa- tional Linguistics -ACL '02, Jan 2001.","title":"Proceedings of the 40th Annual Meeting on Association for Computational Linguistics -ACL '02","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu, Bleu","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2023] Bin Wang, Fan Wu, Xiao Han, Jiahui Peng, Huaping Zhong, Pan Zhang, Xiaoyi Dong, Weijia Li, Wei Li, Jiaqi Wang, et al. Vigc: Visual instruction gen- eration and correction. arXiv preprint arXiv:2308.12714, 2023.","title":"Vigc: Visual instruction generation and correction","authors":"Bin Wang, Fan Wu, Xiao Han, Jiahui Peng, Huaping Zhong, Pan Zhang, Xiaoyi Dong, Weijia Li, Wei Li, Jiaqi Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wei et al., 2021] Jason Wei, Maarten Bosma, VincentY. Zhao, Kelvin Guu, AdamsWei Yu, Brian Lester, Nan Du, AndrewM. Dai, and QuocV. Le. Finetuned language mod- els are zero-shot learners. Learning,Learning, Sep 2021.","title":"Finetuned language models are zero-shot learners","authors":"Jason Wei, Maarten Bosma, Vincenty, Kelvin Zhao, Adamswei Guu, Brian Yu, Nan Lester, Andrewm Du, Quocv Dai, Le","year":"2021","venue":"Learning,Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu et al., 2023] Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. arXiv preprint arXiv:2308.16911, 2023.","title":"Pointllm: Empowering large language models to understand point clouds","authors":"Xu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xue et al., 2023] Le Xue, Ning Yu, Shu Zhang, Junnan Li, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip-2: To- wards scalable multimodal pre-training for 3d understand- ing. arXiv preprint arXiv:2305.08275, 2023.","title":"Ulip-2: Towards scalable multimodal pre-training for 3d understanding","authors":"Le Xue, Ning Yu, Shu Zhang, Junnan Li, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, Silvio Savarese","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yin et al., 2023] Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Lu Sheng, Lei Bai, Xiaoshui Huang, Zhiyong Wang, et al. Lamm: Language- assisted multi-modal instruction-tuning dataset, frame- work, and benchmark. arXiv preprint arXiv:2306.06687, 2023.","title":"Lamm: Languageassisted multi-modal instruction-tuning dataset, framework, and benchmark","authors":"Yin","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhu et al., 2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xi- ang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models. arXiv preprint arXiv:2304.10592, 2023.","title":"Minigpt-4: Enhancing vision-language understanding with advanced large language models","authors":"Zhu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}