{"type":"ScholarlyArticle","title":"GRASP: A Novel Benchmark for Evaluating Language GRounding and Situated Physics Understanding in Multimodal Language Models","authors":[{"name":"Serwan Jassim","slug":"serwan-jassim","given":"Serwan","family":"Jassim","affiliation":"Osnabrück University"},{"name":"Mario Holubar","slug":"mario-holubar","given":"Mario","family":"Holubar","affiliation":"University of Amsterdam"},{"name":"Annika Richter","slug":"annika-richter","given":"Annika","family":"Richter","affiliation":"Osnabrück University"},{"name":"Cornelius Wolff","slug":"cornelius-wolff","given":"Cornelius","family":"Wolff","affiliation":"Osnabrück University"},{"name":"Xenia Ohmer","slug":"xenia-ohmer","given":"Xenia","family":"Ohmer","affiliation":"Osnabrück University"},{"name":"Elia Bruni","slug":"elia-bruni","given":"Elia","family":"Bruni","affiliation":"Osnabrück University"}],"abstract":"This paper presents GRASP, a novel benchmark to evaluate the language grounding and physical understanding capabilities of video-based multimodal large language models (LLMs). This evaluation is accomplished via a two-tier approach leveraging Unity simulations. The first level tests for language grounding by assessing a model's ability to relate simple textual descriptions with visual information. The second level evaluates the model's understanding of \"Intuitive Physics\" principles, such as object permanence and continuity. In addition to releasing the benchmark, we use it to evaluate several state-of-the-art multimodal LLMs. Our evaluation reveals significant shortcomings in the language grounding and intuitive physics capabilities of these models. Although they exhibit at least some grounding capabilities, particularly for colors and shapes, these capabilities depend heavily on the prompting strategy. At the same time, all models perform below or at the chance level of 50% in the Intuitive Physics tests, while human subjects are on average 80% correct. These identified limitations underline the importance of using benchmarks like GRASP to monitor the progress of future models in developing these competencies.","doi":"10.24963/ijcai.2024/696","url":"https://ijcai.org/proceedings/2024/696","pdf":"https://ijcai.org/proceedings/2024/0696.pdf","markdown":"https://ijcai.org/proceedings/2024/696.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/696","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Natural Language Processing","track":"Main Track","number":696,"pages":{"first":6297,"last":6305},"keywords":["Computer Vision: CV: Vision, language and reasoning","Natural Language Processing: NLP: Language grounding","Natural Language Processing: NLP: Resources and evaluation"],"videos":[],"references":[{"position":1,"raw":"[Agrawal et al., 2016] Pulkit Agrawal, Ashvin V. Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Gar- nett, editors, Proceedings of the 30th International Confer- ence on Neural Information Processing Systems (NeurIPS), volume 29.","title":"Learning to poke by poking: Experiential learning of intuitive physics","authors":"Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, Sergey Levine","year":null,"venue":"Proceedings of the 30th International Conference on Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"Curran Associates, Inc., 2016.","title":null,"authors":"Curran Associates, Inc","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Alayrac et al., 2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Ja- cob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ri- cardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. ArXiv Preprint, arXiv:2204.14198, 2022.","title":"Flamingo: a visual language model for few-shot learning","authors":"Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, Karen Simonyan","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Anderson et al., 2018] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision- and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceed- ings of the IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), 2018.","title":"Visionand-language navigation: Interpreting visually-grounded navigation instructions in real environments","authors":"Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, Anton Van Den, Hengel","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015.","title":"VQA: Visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Baillargeon et al., 1985] Renée Baillargeon, Elizabeth S. Spelke, and Stanley Wasserman. Object permanence in five-month-old infants. Cognition, 20:191-208, 1985.","title":"Object permanence in five-month-old infants","authors":"Renée Baillargeon, Elizabeth S Spelke, Stanley Wasserman","year":"1985","venue":"Cognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Baillargeon et al., 2012] Renée Baillargeon, Maayan Sta- vans, Di Wu, Yael Gertner, Peipei Setoh, Audrey K. Kit- tredge, and Amélie Bernard. Object individuation and physical reasoning in infancy: An integrative account. Lan- guage learning and development: The official journal of the Society for Language Development, 8(1):4-46, 2012.","title":"Object individuation and physical reasoning in infancy: An integrative account. Language learning and development","authors":"Renée Baillargeon, Maayan Stavans, Di Wu, Yael Gertner, Peipei Setoh, Audrey K Kittredge, Amélie Bernard","year":"2012","venue":"The official journal of the Society for Language Development","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Baillargeon, 1987] Renée Baillargeon. Object permanence in 3½-and 4½-month-old infants. Developmental Psychol- ogy, 23(5):655-664, 1987.","title":"Object permanence in 3½-and 4½-month-old infants","authors":"Renée Baillargeon, Baillargeon","year":"1987","venue":"Developmental Psychology","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Baillargeon, 1995] Renee Baillargeon. Physical reasoning in infancy. In Michael S. Gazzaniga, editor, The Cognitive Neurosciences, pages 181-204. MIT Press, 1995.","title":"Physical reasoning in infancy","authors":"Renee Baillargeon","year":"1995","venue":"The Cognitive Neurosciences","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Battaglia et al., 2016] Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray kavukcuoglu. Interaction networks for learning about ob- jects, relations and physics. In Proceedings of the 30th International Conference on Neural Information Process- ing Systems (NeurIPS), pages 4509-4517, Red Hook, NY, USA, 2016.","title":"Interaction networks for learning about objects, relations and physics","authors":"Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, Koray Kavukcuoglu","year":"2016","venue":"Proceedings of the 30th International Conference on Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Curran Associates Inc. [Bisk et al., 2020]","title":null,"authors":"Curran Associates, Inc [ Bisk","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Yonatan Bisk, Ari Holtzman, Jesse Thoma- son, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nis- nevich, Nicolas Pinto, and Joseph Turian. Experience grounds language. In Bonnie Webber, Trevor Cohn, Yu- lan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8718-8735, Online, 2020. As- sociation for Computational Linguistics.","title":"Experience grounds language","authors":"Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, Joseph Turian","year":"2020","venue":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Chiang et al., 2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. LMSYS Org Blog, March 2023.","title":"Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality","authors":"Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, Ion Stoica, Eric P Xing","year":"2023","venue":"LMSYS Org Blog","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Das et al., 2018] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Em- bodied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.","title":"Embodied question answering","authors":"Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, Dhruv Batra","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Dasgupta et al., 2021a] Arijit Dasgupta, Jiafei Duan, Marcelo H Ang Jr, Yi Lin, Su-hua Wang, Renée Bail- largeon, and Cheston Tan. A benchmark for modeling violation-of-expectation in physical reasoning across event categories. ArXiv Preprint, arXiv:2111.0882, 2021.","title":"A benchmark for modeling violation-of-expectation in physical reasoning across event categories","authors":"Dasgupta","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Dasgupta et al., 2021b] Arijit Dasgupta, Jiafei Duan, Marcelo H Ang Jr, and Cheston Tan. Avoe: A synthetic 3d dataset on understanding violation of expectation for artificial cognition. ArXiv Preprint, arXiv:2110.05836, 2021.","title":"Avoe: A synthetic 3d dataset on understanding violation of expectation for artificial cognition","authors":"Dasgupta","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Duan et al., 2022] Jiafei Duan, Arijit Dasgupta, Jason Fis- cher, and Cheston Tan. A survey on machine learning approaches for modelling intuitive physics. In Lud De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 5444-5452. International Joint Conferences on Artificial Intelligence Organization, 2022. Survey Track.","title":"A survey on machine learning approaches for modelling intuitive physics","authors":"Duan","year":"2022","venue":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22","doi":null,"url":null,"internal_url":"/proceedings/2022/763"},{"position":18,"raw":"[Girdhar et al., 2023] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Ar- mand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. ArXiv Preprint, arXiv:2305.05665, 2023.","title":"Imagebind: One embedding space to bind them all","authors":"Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, Ishan Misra","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Gordon et al., 2012] Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In Eneko Agirre, Johan Bos, Mona Diab, Suresh Manandhar, Yuval Marton, and Deniz Yuret, edi- tors, *SEM 2012: The First Joint Conference on Lexical and Computational Semantics -Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Seman- tic Evaluation (SemEval 2012), pages 394-398, Montréal, Canada, 2012. Association for Computational Linguistics.","title":"SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning","authors":"Andrew Gordon, Zornitsa Kozareva, Melissa Roemmele","year":"2012","venue":"The First Joint Conference on Lexical and Computational Semantics -Volume 1: Proceedings of the main conference and the shared task","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Hendrycks et al., 2021] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask lan- guage understanding. In International Conference on Learning Representations (ICLR), 2021.","title":"Measuring massive multitask language understanding","authors":"Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt","year":"2021","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Huang et al., 2023] Bin Huang, Xin Wang, Hong Chen, Zi- han Song, and Wenwu Zhu. Vtimellm: Empower llm to grasp video moments, 2023.","title":"Vtimellm: Empower llm to grasp video moments","authors":"Bin Huang, Xin Wang, Hong Chen, Zihan Song, Wenwu Zhu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Johnson et al., 2017] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for com- positional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.","title":"CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning","authors":"Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, Ross Girshick","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Kim and Spelke, 1992] In K. Kim and Elizabeth S. Spelke. Infants' sensitivity to effects of gravity on visible object motion. Journal of Experimental Psychology: Human Perception and Performance, 18(2):385-393, 1992.","title":"Infants' sensitivity to effects of gravity on visible object motion","authors":"Spelke Kim, K Kim, Elizabeth S Spelke","year":"1992","venue":"Journal of Experimental Psychology: Human Perception and Performance","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Kim and Spelke, 1999] In K. Kim and Elizabeth S. Spelke. Perception and understanding of effects of gravity and iner- tia on object motion. Developmental Science, 2(3):339-362, 1999.","title":"Perception and understanding of effects of gravity and inertia on object motion","authors":"Spelke Kim, K Kim, Elizabeth S Spelke","year":"1999","venue":"Developmental Science","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Lerer et al., 2016] Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition of block towers by example. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Ma- chine Learning (ICML), volume 48, pages 430-438. PMLR, 2016.","title":"Learning physical intuition of block towers by example","authors":"Adam Lerer, Sam Gross, Rob Fergus","year":"2016","venue":"Proceedings of The 33rd International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Li et al., 2023a] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. ArXiv Preprint, arXiv:2301.12597, 2023.","title":"Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Li et al., 2023b] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. ArXiv Preprint, arXiv:2305.06355, 2023.","title":"Videochat: Chat-centric video understanding","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Liu et al., 2023] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. ArXiv Preprint, arXiv:2304.08485, 2023.","title":"Visual instruction tuning","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Maaz et al., 2023] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-ChatGPT: Towards detailed video understanding via large vision and language models. ArXiv Preprint, arXiv:2306.05424, 2023.","title":"Video-ChatGPT: Towards detailed video understanding via large vision and language models","authors":"Maaz","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Mitchell et al., 2012] Margaret Mitchell, Jesse Dodge, Amit Goyal, Kota Yamaguchi, Karl Stratos, Xufeng Han, Alyssa Mensch, Alex Berg, Tamara Berg, and Hal Daumé III. Midge: Generating image descriptions from computer vi- sion detections. In Walter Daelemans, editor, Proceedings of the 13th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 747-756, Avignon, France, 2012. Association for Computational Lin- guistics.","title":"Midge: Generating image descriptions from computer vision detections","authors":"Margaret Mitchell, Jesse Dodge, Amit Goyal, Kota Yamaguchi, Karl Stratos, Xufeng Han, Alyssa Mensch, Alex Berg, Tamara Berg, Hal Daumé, Iii","year":"2012","venue":"Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Naveed et al., 2023] Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Us- man, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models, 2023.","title":"A comprehensive overview of large language models","authors":"Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, Ajmal Mian","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Piloto et al., 2022] Luis S. Piloto, Ari Weinstein, Peter Battaglia, and Matthew Botvinick. Intuitive physics learn- ing in a deep-learning model inspired by developmental psychology. Nature Human Behaviour, 6(9):1257-1267, Sep 2022.","title":"Intuitive physics learning in a deep-learning model inspired by developmental psychology","authors":"Luis S Piloto, Ari Weinstein, Peter Battaglia, Matthew Botvinick","year":"2022","venue":"Nature Human Behaviour","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning trans- ferable visual models from natural language supervision. ArXiv Preprint, arXiv:2103.00020, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Riochet et al., 2022] Ronan Riochet, Mario Ynocente Cas- tro, Mathieu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. Intphys 2019: A bench- mark for visual intuitive physics understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5016-5025, 2022.","title":"Intphys 2019: A benchmark for visual intuitive physics understanding","authors":"Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, Emmanuel Dupoux","year":"2022","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Spelke et al., 1992] Elizabeth S. Spelke, Karen Breinlinger, Janet Macomber, and Kristen Jacobson. Origins of knowl- edge. Psychological review, 99(4):605-632, 1992.","title":"Origins of knowledge","authors":"Elizabeth S Spelke, Karen Breinlinger, Janet Macomber, Kristen Jacobson","year":"1992","venue":"Psychological review","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Spelke et al., 1994] Elizabeth S. Spelke, Gary Katz, Susan E. Purcell, Sheryl M. Ehrlich, and Karen Breinlinger. Early knowledge of object motion: continuity and inertia. Cogni- tion, 51(1):131-176, 1994.","title":"Early knowledge of object motion: continuity and inertia","authors":"Elizabeth S Spelke, Gary Katz, Susan E Purcell, Sheryl M Ehrlich, Karen Breinlinger","year":"1994","venue":"Cognition","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Su et al., 2023] Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. PandaGPT: One model to instruction-follow them all. ArXiv Preprint, arXiv:2305.16355, 2023.","title":"PandaGPT: One model to instruction-follow them all","authors":"Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, Deng Cai","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Watters et al., 2017] Nicholas Watters, Andrea Tacchetti, Théophane Weber, Razvan Pascanu, Peter Battaglia, and Daniel Zoran. Visual interaction networks: Learning a physics simulator from video. In Proceedings of the 31st International Conference on Neural Information Process- ing Systems (NeurIPS), pages 4542-4550, Red Hook, NY, USA, 2017.","title":"Visual interaction networks: Learning a physics simulator from video","authors":"Nicholas Watters, Andrea Tacchetti, Théophane Weber, Razvan Pascanu, Peter Battaglia, Daniel Zoran","year":"2017","venue":"Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"Curran Associates Inc. [Weihs et al., 2022]","title":null,"authors":"Curran Associates, Inc [ Weihs","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"Luca Weihs, Amanda Yuile, Renée Bail- largeon, Cynthia Fisher, Gary Marcus, Roozbeh Mottaghi, and Aniruddha Kembhavi. Benchmarking progress to infant-level physical reasoning in AI. Transactions on Machine Learning Research, 2022.","title":"Benchmarking progress to infant-level physical reasoning in AI","authors":"Luca Weihs, Amanda Yuile, Renée Baillargeon, Cynthia Fisher, Gary Marcus, Roozbeh Mottaghi, Aniruddha Kembhavi","year":"2022","venue":"Transactions on Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Yi et al., 2020] Kexin Yi, Chuang Gan, Yunzhu Li, Push- meet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. CLEVRER: Collision events for video repre- sentation and reasoning. In International Conference on Learning Representations (ICLR), 2020.","title":"CLEVRER: Collision events for video representation and reasoning","authors":"Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, Joshua B Tenenbaum","year":"2020","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Zellers et al., 2019] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.","title":"From recognition to cognition: Visual commonsense reasoning","authors":"Rowan Zellers, Yonatan Bisk, Ali Farhadi, Yejin Choi","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Zhang et al., 2023] Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMa: An instruction-tuned audio-vidual lan- guage model for video understanding. ArXiv Preprint, arXiv:2306.02858, 2023.","title":"Video-LLaMa: An instruction-tuned audio-vidual language model for video understanding","authors":"Hang Zhang, Xin Li, Lidong Bing","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Zhu et al., 2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xi- ang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large lan- guage models. ArXiv Preprint, arXiv:2304.10592, 2023.","title":"MiniGPT-4: Enhancing vision-language understanding with advanced large language models","authors":"Zhu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}