{"type":"ScholarlyArticle","title":"What is Right for Me is Not Yet Right for You: A Dataset for Grounding Relative Directions via Multi-Task Learning","authors":[{"name":"Jae Hee Lee","slug":"jae-hee-lee","given":"Jae Hee","family":"Lee","affiliation":"University of Hamburg"},{"name":"Matthias Kerzel","slug":"matthias-kerzel","given":"Matthias","family":"Kerzel","affiliation":"University of Hamburg"},{"name":"Kyra Ahrens","slug":"kyra-ahrens","given":"Kyra","family":"Ahrens","affiliation":"University of Hamburg"},{"name":"Cornelius Weber","slug":"cornelius-weber","given":"Cornelius","family":"Weber","affiliation":"University of Hamburg"},{"name":"Stefan Wermter","slug":"stefan-wermter","given":"Stefan","family":"Wermter","affiliation":"University of Hamburg"}],"abstract":"Understanding spatial relations is essential for intelligent agents to act and communicate in the physical world. Relative directions are spatial relations that describe the relative positions of target objects with regard to the intrinsic orientation of reference objects. Grounding relative directions is more difficult than grounding absolute directions because it not only requires a model to detect objects in the image and to identify spatial relation based on this information, but it also needs to recognize the orientation of objects and integrate this information into the reasoning process. We investigate the challenging problem of grounding relative directions with end-to-end neural networks. To this end, we provide GRiD-3D, a novel dataset that features relative directions and complements existing visual question answering (VQA) datasets, such as CLEVR, that involve only absolute directions. We also provide baselines for the dataset with two established end-to-end VQA models. Experimental evaluations show that answering questions on relative directions is feasible when questions in the dataset simulate the necessary subtasks for grounding relative directions. We discover that those subtasks are learned in an order that reflects the steps of an intuitive pipeline for processing relative directions.","doi":"10.24963/ijcai.2022/145","url":"https://ijcai.org/proceedings/2022/145","pdf":"https://ijcai.org/proceedings/2022/0145.pdf","markdown":"https://ijcai.org/proceedings/2022/145.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/145","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":145,"pages":{"first":1039,"last":1045},"keywords":["Machine Learning: Multi-modal learning","Natural Language Processing: Language Grounding","Machine Learning: Relational Learning","Computer Vision: Vision and language","Natural Language Processing: Question Answering"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986867","length_sec":91,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985010","length_sec":335,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bloom et al., 1999] Paul Bloom, Merrill F. Garrett, Lynn Nadel, and Mary A. Peterson, editors. Language and Space. The MIT Press, 1999.","title":"Language and Space","authors":null,"year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Coventry and Garrod, 2004] Kenny R. Coventry and Si- mon C. Garrod. Saying, Seeing and Acting: The Psycholog- ical Semantics of Spatial Prepositions. Psychology Press, July 2004.","title":"Saying, Seeing and Acting: The Psycholog-ical Semantics of Spatial Prepositions","authors":"Kenny R Coventry, Simon C Garrod","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Freksa et al., 2018] Christian Freksa, Jasper van de Ven, and Diedrich Wolter. Formal representation of qualitative direc- tion. International Journal of Geographical Information Science, 32(12), December 2018.","title":"Christian Freksa, Jasper van de Ven, and Diedrich Wolter. Formal representation of qualitative direction","authors":"Freksa","year":"2018","venue":"International Journal of Geographical Information Science","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Goyal et al., 2017] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image under- standing in visual question answering. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.","title":"Making the V in VQA matter: Elevating the role of image understanding in visual question answering","authors":"Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, Devi Parikh","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goyal et al., 2020] Ankit Goyal, Kaiyu Yang, Dawei Yang, and Jia Deng. Rel3D: A minimally contrastive benchmark for grounding spatial relations in 3D. Advances in Neural Information Processing Systems, 33, 2020.","title":"Rel3D: A minimally contrastive benchmark for grounding spatial relations in 3D","authors":"Ankit Goyal, Kaiyu Yang, Dawei Yang, Jia Deng","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hong et al., 2021] Yining Hong, Li Yi, Joshua B. Tenen- baum, Antonio Torralba, and Chuang Gan. PTR: A bench- mark for part-based conceptual, relational, and physical reasoning. arXiv:2112.05136 [cs], December 2021.","title":"PTR: A benchmark for part-based conceptual, relational, and physical reasoning","authors":"Yining Hong, Li Yi, Joshua B Tenenbaum, Antonio Torralba, Chuang Gan","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2017] Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In IEEE International Conference on Computer Vision (ICCV), Venice, October 2017. IEEE.","title":"Learning to reason: End-to-end module networks for visual question answering","authors":"Hu","year":"2017","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hua et al., 2018] Hua Hua, Jochen Renz, and Xiaoyu Ge. Qualitative representation and reasoning over direction re- lations across different frames of reference. In Sixteenth International Conference on Principles of Knowledge Rep- resentation and Reasoning, September 2018.","title":"Qualitative representation and reasoning over direction relations across different frames of reference","authors":"Hua Hua, Jochen Renz, Xiaoyu Ge","year":"2018","venue":"Sixteenth International Conference on Principles of Knowledge Representation and Reasoning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hudson and Manning, 2018] Drew A. Hudson and Christo- pher D. Manning. Compositional attention networks for machine reasoning. In International Conference on Learn- ing Representations, February 2018.","title":"Compositional attention networks for machine reasoning","authors":"Drew A Hudson, Christopher D Manning","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hudson and Manning, 2019] Drew A. Hudson and Christo- pher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6693-6702, 2019.","title":"GQA: A new dataset for real-world visual reasoning and compositional question answering","authors":"Drew A Hudson, Christopher D Manning","year":"2019","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Johnson et al., 2017a] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for com- positional language and elementary visual reasoning. In IEEE Conference on Computer Vision and Pattern Recog- nition, 2017.","title":"CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning","authors":"Johnson","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Johnson et al., 2017b] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reasoning. In IEEE Inter- national Conference on Computer Vision (ICCV), October 2017.","title":"Inferring and executing programs for visual reasoning","authors":"Johnson","year":"2017","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kerzel et al., 2018] Matthias Kerzel, Hadi Beik Moham- madi, Mohammad Ali Zamani, and Stefan Wermter. Ac- celerating deep continuous reinforcement learning through task simplification. In 2018 International Joint Conference on Neural Networks (IJCNN), pages 1-6. IEEE, 2018.","title":"Accelerating deep continuous reinforcement learning through task simplification","authors":"Matthias Kerzel, Beik Hadi, Mohammad Mohammadi, Stefan Ali Zamani, Wermter","year":"2018","venue":"2018 International Joint Conference on Neural Networks (IJCNN)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krishna et al., 2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual Genome: Connecting language and vision using crowd- sourced dense image annotations. International Journal of Computer Vision, 123(1), May 2017.","title":"Visual Genome: Connecting language and vision using crowdsourced dense image annotations","authors":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael S Bernstein, Li Fei-Fei","year":"2017","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lee et al., 2013] Jae Hee Lee, Jochen Renz, and Diedrich Wolter. StarVars: Effective reasoning about relative direc- tions. In Twenty-Third International Joint Conference on Artificial Intelligence, Beijing, China, 2013. AAAI Press.","title":"StarVars: Effective reasoning about relative directions","authors":"Jae Hee, Lee, Jochen Renz, Diedrich Wolter","year":"2013","venue":"Twenty-Third International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2013/149"},{"position":16,"raw":"[Moratz and Tenbrink, 2006] Reinhard Moratz and Thora Tenbrink. Spatial reference in linguistic human-robot inter- action: Iterative, empirically supported development of a model of projective relations. Spatial Cognition & Compu- tation, 6(1), March 2006.","title":"Spatial reference in linguistic human-robot interaction: Iterative, empirically supported development of a model of projective relations","authors":"Tenbrink Moratz, Reinhard Moratz, Thora Tenbrink","year":"2006","venue":"Spatial Cognition & Computation","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Moratz, 2006] Reinhard Moratz. Representing relative di- rection as a binary relation of oriented points. In 17th European Conference on Artificial Intelligence. IOS Press, May 2006.","title":"Representing relative direction as a binary relation of oriented points","authors":"Reinhard Moratz","year":"2006","venue":"17th European Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Perez et al., 2018] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual reasoning with a general conditioning layer. In Thirty-Second AAAI Conference on Artificial Intelligence, April 2018.","title":"FiLM: Visual reasoning with a general conditioning layer","authors":"Ethan Perez, Florian Strub, Vincent Harm De Vries, Aaron Dumoulin, Courville","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Suhr et al., 2019] Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. arXiv:1811.00491 [cs], July 2019.","title":"A corpus for reasoning about natural language grounded in photographs","authors":"Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, Yoav Artzi","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wu et al., 2017] Qi Wu, Damien Teney, Peng Wang, Chun- hua Shen, Anthony Dick, and Anton van den Hengel. Vi- sual question answering: A survey of methods and datasets. Computer Vision and Image Understanding, 163, October 2017.","title":"Visual question answering: A survey of methods and datasets","authors":"Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, Anton Van Den, Hengel","year":"2017","venue":"Computer Vision and Image Understanding","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2019] Kaiyu Yang, Olga Russakovsky, and Jia Deng. SpatialSense: An adversarially crowdsourced bench- mark for spatial relation recognition. In IEEE/CVF Inter- national Conference on Computer Vision, 2019.","title":"SpatialSense: An adversarially crowdsourced benchmark for spatial relation recognition","authors":"Kaiyu Yang, Olga Russakovsky, Jia Deng","year":"2019","venue":"IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yang et al., 2021] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learn- ing to answer questions from millions of narrated videos. In IEEE/CVF International Conference on Computer Vision, pages 1686-1697, 2021.","title":"Just ask: Learning to answer questions from millions of narrated videos","authors":"Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, Cordelia Schmid","year":"2021","venue":"IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yi et al., 2018] Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic VQA: Disentangling reasoning from vision and language understanding. In Advances in Neural Informa- tion Processing Systems 31.","title":"Neuralsymbolic VQA: Disentangling reasoning from vision and language understanding","authors":"Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, Josh Tenenbaum","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"Curran Associates, Inc., 2018.","title":null,"authors":"Curran Associates, Inc","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}