{"type":"ScholarlyArticle","title":"Reframing Spatial Reasoning Evaluation in Language Models: A Real-World Simulation Benchmark for Qualitative Reasoning","authors":[{"name":"Fangjun Li","slug":"fangjun-li","given":"Fangjun","family":"Li","affiliation":"School of Computing , University of Leeds , UK"},{"name":"David C. Hogg","slug":"david-c-hogg","given":"David C.","family":"Hogg","affiliation":"School of Computing , University of Leeds , UK"},{"name":"Anthony G. Cohn","slug":"anthony-g-cohn","given":"Anthony G.","family":"Cohn","affiliation":"School of Computing , University of Leeds , UK; Alan Turing Institute , UK"}],"abstract":"Spatial reasoning plays a vital role in both human cognition and machine intelligence, prompting new research into language models' (LMs) capabilities in this regard. However, existing benchmarks reveal shortcomings in evaluating qualitative spatial reasoning (QSR). These benchmarks typically present oversimplified scenarios or unclear natural language descriptions, hindering effective evaluation. We present a novel benchmark for assessing QSR in LMs, which is grounded in realistic 3D simulation data, offering a series of diverse room layouts with various objects and their spatial relationships. This approach provides a more detailed and context-rich narrative for spatial reasoning evaluation, diverging from traditional, toy-task-oriented scenarios. Our benchmark encompasses a broad spectrum of qualitative spatial relationships, including topological, directional, and distance relations. These are presented with different viewing points, varied granularities, and density of relation constraints to mimic real-world complexities. A key contribution is our logic-based consistency-checking tool, which enables the assessment of multiple plausible solutions, aligning with real-world scenarios where spatial relationships are often open to interpretation. Our benchmark evaluation of advanced LMs reveals their strengths and limitations in spatial reasoning. They face difficulties with multi-hop spatial reasoning and interpreting a mix of different view descriptions, pointing to areas for future improvement.","doi":"10.24963/ijcai.2024/701","url":"https://ijcai.org/proceedings/2024/701","pdf":"https://ijcai.org/proceedings/2024/0701.pdf","markdown":"https://ijcai.org/proceedings/2024/701.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/701","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Natural Language Processing","track":"Main Track","number":701,"pages":{"first":6342,"last":6349},"keywords":["Knowledge Representation and Reasoning: KRR: Learning and reasoning","Constraint Satisfaction and Optimization: CSO: Applications","Natural Language Processing: NLP: Resources and evaluation","Knowledge Representation and Reasoning: KRR: Qualitative, geometric, spatial, and temporal reasoning"],"videos":[],"references":[{"position":1,"raw":"[Alomari et al., 2022] Muhannad Alomari, Fangjun Li, David C Hogg, and Anthony G Cohn. Online perceptual learning and natural language acquisition for autonomous robots. Artificial Intelligence, 303:103637, 2022.","title":"Online perceptual learning and natural language acquisition for autonomous robots","authors":"Muhannad Alomari, Fangjun Li, David C Hogg, Anthony G Cohn","year":"2022","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bang et al., 2023] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023, 2023.","title":"A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity","authors":"Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bommasani et al., 2021] Rishi Bommasani, Drew A Hud- son, Ehsan Adeli, Russ Altman, Simran Arora, Syd- ney von Arx, Michael S Bernstein, Jeannette Bohg, An- toine Bosselut, Emma Brunskill, et al. On the oppor- tunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.","title":"On the opportunities and risks of foundation models","authors":"Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Von Arx, Jeannette Michael S Bernstein, Antoine Bohg, Emma Bosselut, Brunskill","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing sys- tems, 33:1877-1901, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cohn and Hernandez-Orallo, 2023] Anthony G Cohn and Jose Hernandez-Orallo. Dialectical language model evaluation: An initial appraisal of the commonsense spatial reasoning abilities of LLMs. arXiv preprint arXiv:2304.11164, 2023.","title":"Dialectical language model evaluation: An initial appraisal of the commonsense spatial reasoning abilities of LLMs","authors":"G Anthony, Jose Cohn, Hernandez-Orallo","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cohn and Renz, 2008] Anthony G Cohn and Jochen Renz. Qualitative spatial representation and reasoning. In Frank Van Harmelen, Vladimir Lifschitz, and Bruce Porter, ed- itors, Handbook of knowledge representation, pages 551- 596. Elsevier, 2008.","title":"Qualitative spatial representation and reasoning","authors":"G Anthony, Jochen Cohn, Renz","year":"2008","venue":"Handbook of knowledge representation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Deitke et al., 2022] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, et al. Procthor: Large-scale embodied ai using procedural gen- eration. Advances in Neural Information Processing Sys- tems, 35:5982-5994, 2022.","title":"Procthor: Large-scale embodied ai using procedural generation","authors":"Matt Deitke, Eli Vanderbilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Haviland, 1998] J B Haviland. Guugu Yimithirr cardinal directions. Ethos, 26(1):25-47, 1998.","title":"Guugu Yimithirr cardinal directions","authors":"J B Haviland","year":"1998","venue":"Ethos","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2022] Fangjun Li, DC Hogg, and AG Cohn. Ontol- ogy knowledge-enhanced in-context learning for action- effect prediction. Advances in Cognitive Systems. ACS- 2022, 2022.","title":"Ontology knowledge-enhanced in-context learning for actioneffect prediction","authors":"Li","year":"2022","venue":"Advances in Cognitive Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2024] Fangjun Li, David C Hogg, and Anthony G Cohn. Advancing spatial reasoning in large language mod- els: An in-depth evaluation and enhancement using the stepgame benchmark. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 38, pages 18500- 18507, 2024.","title":"Advancing spatial reasoning in large language models: An in-depth evaluation and enhancement using the stepgame benchmark","authors":"Li","year":"2024","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ligozat, 1998] Gerard Ligozat. Reasoning about cardinal directions. Journal of Visual Languages & Computing, 9(1):23-44, 1998.","title":"Gerard Ligozat. Reasoning about cardinal directions","authors":"Ligozat","year":"1998","venue":"Journal of Visual Languages & Computing","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mirzaee and Kordjamshidi, 2022] Roshanak Mirzaee and Parisa Kordjamshidi. Transfer learning with synthetic cor- pora for spatial role labeling and reasoning. In Proceed- ings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 6148-6165. Association for Computational Linguistics, December 2022.","title":"Transfer learning with synthetic corpora for spatial role labeling and reasoning","authors":"Kordjamshidi Mirzaee, Roshanak Mirzaee, Parisa Kordjamshidi","year":"2022","venue":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mirzaee et al., 2021] Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjamshidi. SpartQA: a textual question answering benchmark for spatial reason- ing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4582- 4598, 2021.","title":"SpartQA: a textual question answering benchmark for spatial reasoning","authors":"Roshanak Mirzaee, Rajaby Hossein, Qiang Faghihi, Parisa Ning, Kordjamshidi","year":"2021","venue":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[OpenAI, 2023] OpenAI. GPT-4 technical report. ArXiv, abs/2303.08774, 2023.","title":"OpenAI. GPT-4 technical report","authors":"Openai","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Randell et al., 1992] David A Randell, Zhan Cui, and An- thony G Cohn. A spatial logic based on regions and con- nection. KR, 92:165-176, 1992.","title":"A spatial logic based on regions and connection","authors":"Zhan David A Randell, Anthony G Cui, Cohn","year":"1992","venue":"KR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shi et al., 2022] Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. Stepgame: A new benchmark for robust multi- hop spatial reasoning in texts. In Proceedings of the AAAI conference on Artificial Intelligence, volume 36, pages 11321-11329, 2022.","title":"Stepgame: A new benchmark for robust multihop spatial reasoning in texts","authors":"Shi","year":"2022","venue":"Proceedings of the AAAI conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Touvron et al., 2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, et al. Llama 2: Open foundation and fine-tuned chat models, 2023.","title":"Llama 2: Open foundation and fine-tuned chat models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Weston et al., 2016] Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Ar- mand Joulin, and Tomas Mikolov. Towards AI-complete question answering: A set of prerequisite toy tasks. In 4th International Conference on Learning Representations, ICLR, 2016.","title":"Towards AI-complete question answering: A set of prerequisite toy tasks","authors":"Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, Tomas Mikolov","year":"2016","venue":"4th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}