{"type":"ScholarlyArticle","title":"Vision beyond Pixels: Visual Reasoning via Blocksworld Abstractions","authors":[{"name":"Tejas Gokhale","slug":"tejas-gokhale","given":"Tejas","family":"Gokhale","affiliation":"Arizona State University"}],"abstract":"Deep neural networks trained in an end-to-end fashion have brought about exceptional advances in computer vision, especially in computational perception. We go beyond perception and seek to enable vision modules to reason about perceived visual entities such as scenes, objects and actions. We introduce a challenging visual reasoning task, Image-Based Event Sequencing (IES) and compile the first IES dataset, Blocksworld Image Reasoning Dataset (BIRD). Motivated by the blocksworld concept, we propose a modular approach supported by literature in cognitive psychology and children's development. We decompose the problem into two stages - visual perception and event sequencing, and show that our approach can be extended to natural images without re-training.","doi":"10.24963/ijcai.2019/907","url":"https://ijcai.org/proceedings/2019/907","pdf":"https://ijcai.org/proceedings/2019/0907.pdf","markdown":"https://ijcai.org/proceedings/2019/907.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/907","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Doctoral Consortium","track":"Doctoral Consortium","number":907,"pages":{"first":6436,"last":6437},"keywords":["Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[He et al., 2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In International Conference on Computer Vision, pages 2980-2988, 2017.","title":"Kaiming He, Georgia Gkioxari","authors":"He","year":"2017","venue":"International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Johnson et al., 2017] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for com- positional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2901-2910, 2017.","title":"Clevr: A diagnostic dataset for compositional language and elementary visual reasoning","authors":"Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, Ross Girshick","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Johnson, 1983] Harriet Johnson. The Art of Block Building. The John Day Company, New York, 1983.","title":"The Art of Block Building. The John Day Company","authors":"Harriet Johnson","year":"1983","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Muggleton, 1991] Stephen Muggleton. Inductive logic pro- gramming. New generation computing, 8(4):295-318, 1991.","title":"Inductive logic programming","authors":"Stephen Muggleton","year":"1991","venue":"New generation computing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Park et al., 2019] Dong Huk Park, Trevor Darrell, and Anna Rohrbach. Viewpoint invariant change captioning. arXiv preprint arXiv:1901.02527, 2019.","title":"Viewpoint invariant change captioning","authors":"Dong Huk, Park, Trevor Darrell, Anna Rohrbach","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Santoro et al., 2017] Adam Santoro, David Raposo, David Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Advances in neural information processing systems, pages 4967-4976, 2017.","title":"A simple neural network module for relational reasoning","authors":"Adam Santoro, David Raposo, David Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, Timothy Lillicrap","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Sarama and Clements, 2001] Julie Sarama and Douglas Clements. Building blocks and cognitive building blocks -playing to know the world mathematically. American Journal of Play, 1(3):313-337, Winter 2001.","title":"Building blocks and cognitive building blocks -playing to know the world mathematically","authors":"Clements Sarama, Julie Sarama, Douglas Clements","year":"2001","venue":"American Journal of Play","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Zhang et al., 2018] Ji Zhang, Yannis Kalantidis, Marcus Rohrbach, Manohar Paluri, Ahmed Elgammal, and Mo- hamed Elhoseiny. Large-scale visual relationship under- standing. CoRR, abs/1804.10660, 2018.","title":"Large-scale visual relationship understanding","authors":"Ji Zhang, Yannis Kalantidis, Marcus Rohrbach, Manohar Paluri, Ahmed Elgammal, Mohamed Elhoseiny","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Zhou et al., 2018] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In Proceedings of the European Conference on Computer Vision, pages 803-818, 2018.","title":"Temporal relational reasoning in videos","authors":"Zhou","year":"2018","venue":"Proceedings of the European Conference on Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}