{"type":"ScholarlyArticle","title":"Commonsense Reasoning to Guide Deep Learning for Scene Understanding (Extended Abstract)","authors":[{"name":"Mohan Sridharan","slug":"mohan-sridharan","given":"Mohan","family":"Sridharan","affiliation":"School of Computer Science , University of Birmingham , UK"},{"name":"Tiago Mota","slug":"tiago-mota","given":"Tiago","family":"Mota","affiliation":"Electrical and Computer Engineering , The University of Auckland , NZ"}],"abstract":"Our architecture uses non-monotonic logical reasoning with incomplete commonsense domain knowledge, and incremental inductive learning, to guide the construction of deep network models from a small number of training examples. Experimental results in the context of a robot reasoning about the partial occlusion of objects and the stability of object configurations in simulated images indicate an improvement in reliability and a reduction in computational effort in comparison with an architecture based just on deep networks.","doi":"10.24963/ijcai.2020/661","url":"https://ijcai.org/proceedings/2020/661","pdf":"https://ijcai.org/proceedings/2020/0661.pdf","markdown":"https://ijcai.org/proceedings/2020/661.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/661","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":661,"pages":{"first":4760,"last":4764},"keywords":["Machine Learning: Deep Learning","Machine Learning: Online Learning","Robotics: Robotics and Vision","Knowledge Representation and Reasoning: Non-monotonic Reasoning, Common-Sense Reasoning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/27006/file_27006.m3u8","length_sec":936,"provider":"hls"}],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martin Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. TensorFlow: Large-scale Machine Learning on Het- erogeneous Distributed Systems. Technical report, https: //arxiv.org/abs/1603.04467, 2016.","title":"TensorFlow: Large-scale Machine Learning on Heterogeneous Distributed Systems","authors":"Martin Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin","year":null,"venue":null,"doi":null,"url":"https://arxiv.org/abs/1603.04467,2016","internal_url":null},{"position":2,"raw":"[Calli et al., 2015] Berk Calli, Aaron Wallsman, Arjun Singfh, and Siddhartha S. Srinivasa. Benchmarking in Manipulation Research. IEEE Robotics and Automation Magazine, (September):36-52, 2015.","title":"Benchmarking in Manipulation Research","authors":"Berk Calli, Aaron Wallsman, Arjun Singfh, Siddhartha S Srinivasa","year":"2015","venue":"IEEE Robotics and Automation Magazine","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chai et al., 2018] Joyce Y. Chai, Qiaozi Gao, Lanbo She, Shaohua Yang, Sari Saba-Sadiya, and Guangyue Xu. Lan- guage to Action: Towards Interactive Task Learning with Physical Agents. In International Joint Conference on Ar- tificial Intelligence (IJCAI), Stockholm, Sweden, July 13- 19, 2018.","title":"Language to Action: Towards Interactive Task Learning with Physical Agents","authors":"Joyce Y Chai, Qiaozi Gao, Lanbo She, Shaohua Yang, Sari Saba-Sadiya, Guangyue Xu","year":"2018","venue":"International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2018/1"},{"position":4,"raw":"[Erdem et al., 2016] Esra Erdem, Michael Gelfond, and Nicola Leone. Applications of Answer Set Programming. AI Magazine, 37(3):53-68, 2016.","title":"Applications of Answer Set Programming","authors":"Esra Erdem, Michael Gelfond, Nicola Leone","year":"2016","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gelfond and Kahl, 2014] Michael Gelfond and Yulia Kahl. Knowledge Representation, Reasoning and the Design of Intelligent Agents. Cambridge University Press, 2014.","title":"Knowledge Representation, Reasoning and the Design of Intelligent Agents","authors":"Kahl Gelfond, Michael Gelfond, Yulia Kahl","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gil, 1994] Yolanda Gil. Learning by Experimentation: In- cremental Refinement of Incomplete Planning Domains. In International Conference on Machine Learning, pages 87-95, New Brunswick, USA, July 10-13, 1994.","title":"Learning by Experimentation: Incremental Refinement of Incomplete Planning Domains","authors":"Yolanda Gil","year":"1994","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jund et al., 2018] Philipp Jund, Andreas Eitel, Nichola Abdo, and Wolfram Burgard. Optimization Beyond the Convolution: Generalizing Spatial Relations with End- to-End Metric Learning. In International Conference on Robotics and Automation, 2018.","title":"Optimization Beyond the Convolution: Generalizing Spatial Relations with Endto-End Metric Learning","authors":"Philipp Jund, Andreas Eitel, Nichola Abdo, Wolfram Burgard","year":"2018","venue":"International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural in- formation processing systems, pages 1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lerer et al., 2016] Adam Lerer, Sam Gross, and Rob Fer- gus. Learning physical intuition of block towers by ex- ample. Technical report, https://arxiv.org/abs/1603.01312, 2016.","title":"Learning physical intuition of block towers by example","authors":"Adam Lerer, Sam Gross, Rob Fergus","year":null,"venue":null,"doi":null,"url":"https://arxiv.org/abs/1603.01312,2016","internal_url":null},{"position":11,"raw":"[Mota and Sridharan, 2018] Tiago Mota and Mohan Sridha- ran. Incrementally Grounding Expressions for Spatial Re- lations between Objects. In International Joint Conference on Artificial Intelligence, pages 1928-1934, 2018.","title":"Incrementally Grounding Expressions for Spatial Relations between Objects","authors":"Tiago Mota, Mohan Sridharan","year":"2018","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/266"},{"position":12,"raw":"[Mota and Sridharan, 2019a] Tiago Mota and Mohan Srid- haran. Commonsense Reasoning and Knowledge Acquisi- tion to Guide Deep Learning on Robots. In Robotics Sci- ence and Systems, Freiburg, Germany, June 22-26, 2019.","title":"Commonsense Reasoning and Knowledge Acquisition to Guide Deep Learning on Robots","authors":"Sridharan Mota","year":"2019","venue":"Robotics Science and Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mota and Sridharan, 2019b] Tiago Mota and Mohan Srid- haran. Software related to the paper, 2019. https:// github.com/tmot987/Scenes-Understanding.","title":"Tiago Mota and Mohan Sridharan","authors":"Sridharan Mota","year":"2019","venue":null,"doi":null,"url":"https://github.com/tmot987/Scenes-Understanding","internal_url":null},{"position":14,"raw":"[Paul et al., 2016] Rohan Paul, Jacob Arkin, Nicholas Roy, and Thomas Howard. Efficient Grounding of Abstract Spatial Concepts for Natural Language Interaction with Robot Manipulators. In Robotics: Science and Systems, Ann Arbor, USA, June 18-22, 2016.","title":"Efficient Grounding of Abstract Spatial Concepts for Natural Language Interaction with Robot Manipulators","authors":"Rohan Paul, Jacob Arkin, Nicholas Roy, Thomas Howard","year":"2016","venue":"Robotics: Science and Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Riley and Sridharan, 2019] Heather Riley and Mohan Srid- haran. Integrating Non-monotonic Logical Reasoning and Inductive Learning with Deep Learning for Explainable Visual Question Answering. In Frontiers in Robotics and AI, special issue on Combining Symbolic Reasoning and Data-Driven Learning for Decision-Making, 6:125, 2019.","title":"Integrating Non-monotonic Logical Reasoning and Inductive Learning with Deep Learning for Explainable Visual Question Answering","authors":"Sridharan Riley","year":"2019","venue":"Frontiers in Robotics and AI, special issue on Combining Symbolic Reasoning and Data-Driven Learning for Decision-Making","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Santoro et al., 2017] Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Tim Lillicrap. A simple neural network module for relational reasoning. In Advances in neural information processing systems, pages 4967-4976, 2017.","title":"A simple neural network module for relational reasoning","authors":"Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, Tim Lillicrap","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sridharan and Meadows, 2018] Mohan Sridharan and Ben Meadows. Knowledge Representation and Interactive Learning of Domain Knowledge for Human-Robot Col- laboration. Advances in Cognitive Systems, 7:77-96, De- cember 2018.","title":"Knowledge Representation and Interactive Learning of Domain Knowledge for Human-Robot Collaboration","authors":"Mohan Sridharan, Ben Meadows","year":"2018","venue":"Advances in Cognitive Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sridharan and Meadows, 2019] Mohan Sridharan and Ben- jamin Meadows. Towards a Theory of Explanations for Human-Robot Collaboration. Kunstliche Intelligenz, 33(4):331-342, December 2019.","title":"Towards a Theory of Explanations for Human-Robot Collaboration","authors":"Mohan Sridharan, Benjamin Meadows","year":"2019","venue":"Kunstliche Intelligenz","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sridharan et al., 2019] Mohan Sridharan, Michael Gelfond, Shiqi Zhang, and Jeremy Wyatt. REBA: A Refinement- Based Architecture for Knowledge Representation and Reasoning in Robotics. Journal of Artificial Intelligence Research, 65:87-180, May 2019.","title":"REBA: A Refinement-Based Architecture for Knowledge Representation and Reasoning in Robotics","authors":"Mohan Sridharan, Michael Gelfond, Shiqi Zhang, Jeremy Wyatt","year":"2019","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sünderhauf et al., 2018] Niko Sünderhauf, Oliver Brock, Walter Scheirer, Raia Hadsell, Dieter Fox, Jürgen Leitner, Ben Upcroft, Pieter Abbeel, Wolfram Burgard, Michael Milford, et al. The limits and potentials of deep learning for robotics. The International Journal of Robotics Re- search, 37(4-5):405-420, 2018.","title":"The limits and potentials of deep learning for robotics","authors":"Niko Sünderhauf, Oliver Brock, Walter Scheirer, Raia Hadsell, Dieter Fox, Jürgen Leitner, Ben Upcroft, Pieter Abbeel, Wolfram Burgard, Michael Milford","year":"2018","venue":"The International Journal of Robotics Research","doi":"10.1177/0278364918770733","url":null,"internal_url":null},{"position":21,"raw":"[Wagner et al., 2018] Misha Wagner, Hector Basevi, Rak- shith Shetty, Wenbin Li, Mateusz Malinowski, Mario Fritz, and Ales Leonardis. Answering Visual What-If Questions: From Actions to Predicted Scene Descrip- tions. In Visual Learning and Embodied Agents in Simula- tion Environments Workshop at ECCV, Munich, Germany, September 9, 2018. https://arxiv.org/abs/1809.03707.","title":"Answering Visual What-If Questions: From Actions to Predicted Scene Descriptions","authors":"Misha Wagner, Hector Basevi, Rakshith Shetty, Wenbin Li, Mateusz Malinowski, Mario Fritz, Ales Leonardis","year":"2018","venue":"Visual Learning and Embodied Agents in Simulation Environments Workshop at ECCV","doi":null,"url":"https://arxiv.org/abs/1809.03707","internal_url":null},{"position":22,"raw":"[Wu et al., 2015] Jiajun Wu, Ilker Yildirim, Joseph J Lim, Bill Freeman, and Josh Tenenbaum. Galileo: Perceiving physical object properties by integrating a physics engine with deep learning. In Advances in neural information pro- cessing systems, pages 127-135, 2015.","title":"Galileo: Perceiving physical object properties by integrating a physics engine with deep learning","authors":"Jiajun Wu, Ilker Yildirim, Joseph J Lim, Bill Freeman, Josh Tenenbaum","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2016] Renqiao Zhang, Jiajun Wu, Chengkai Zhang, William T Freeman, and Joshua B Tenenbaum. A Comparative Evaluation of Approximate Probabilistic Simulation and Deep Neural Networks as Accounts of Human Physical Scene Understanding. Technical report, https://arxiv.org/abs/1605.01138, 2016.","title":"A Comparative Evaluation of Approximate Probabilistic Simulation and Deep Neural Networks as Accounts of Human Physical Scene Understanding","authors":"Zhang","year":"2016","venue":null,"doi":null,"url":"https://arxiv.org/abs/1605.01138,2016","internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}