{"type":"ScholarlyArticle","title":"Teaching Robots through Situated Interactive Dialogue and Visual Demonstrations","authors":[{"name":"Jose L. Part","slug":"jose-l-part","given":"Jose L.","family":"Part","affiliation":"HRI Lab , Heriot-Watt University Edinburgh Centre for Robotics Edinburgh , Scotland , UK"},{"name":"Oliver Lemon","slug":"oliver-lemon","given":"Oliver","family":"Lemon","affiliation":"HRI Lab , Heriot-Watt University Edinburgh Centre for Robotics Edinburgh , Scotland , UK"}],"abstract":"The ability to quickly adapt to new environments and incorporate new knowledge is of great importance for robots operating in unstructured environments and interacting with non-expert users. This paper reports on our current progress in tackling this problem. We propose the development of a framework for teaching robots to perform tasks using natural language instructions, visual demonstrations and interactive dialogue. Moreover, we present a module for learning objects incrementally and on-the-fly that would enable robots to ground referents in the natural language instructions and reason about the state of the world.","doi":"10.24963/ijcai.2017/760","url":"https://ijcai.org/proceedings/2017/760","pdf":"https://ijcai.org/proceedings/2017/0760.pdf","markdown":"https://ijcai.org/proceedings/2017/760.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/760","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Doctoral Consortium","track":"Doctoral Consortium","number":760,"pages":{"first":5201,"last":5202},"keywords":["Artificial Intelligence: artificial intelligence","Artificial Intelligence: knowledge representation","Artificial Intelligence: natural language processing","Artificial Intelligence: robotics"],"videos":[],"references":[{"position":1,"raw":"[Antunes et al., 2016] Alexandre Antunes, Lorenzo Jamone, Giovanni Saponaro, Alexandre Bernardino, and Rodrigo Ventura. From Human Instructions to Robot Actions: For- mulation of Goals, Affordances and Probabilistic Plan- ning. In ICRA, pages 5449-5454, 2016.","title":"From Human Instructions to Robot Actions: Formulation of Goals, Affordances and Probabilistic Planning","authors":"Alexandre Antunes, Lorenzo Jamone, Giovanni Saponaro, Alexandre Bernardino, Rodrigo Ventura","year":"2016","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Forbes et al., 2015] Maxwell Forbes, Rajesh P. N. Rao, Luke Zettlemoyer, and Maya Cakmak. Robot Program- ming by Demonstration with Situated Spatial Language Understanding. In ICRA, pages 2014-2020, 2015.","title":"Robot Programming by Demonstration with Situated Spatial Language Understanding","authors":"Maxwell Forbes, P N Rajesh, Luke Rao, Maya Zettlemoyer, Cakmak","year":"2015","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Liu and Yang, 2016] Changsong Liu and Shaohua Yang. Joint Learning Task Structures from Language Instruction and Visual Demonstration. In EMNLP, pages 1-10, 2016.","title":"Joint Learning Task Structures from Language Instruction and Visual Demonstration","authors":"Yang ; Liu, Changsong Liu, Shaohua Yang","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Mohseni-Kabir et al., 2015] Anahita Mohseni-Kabir, Charles Rich, Sonia Chernova, Candace L. Sidner, and Daniel Miller. Interactive Hierarchical Task Learning from a Single Demonstration. In HRI, pages 205-212, 2015.","title":"Interactive Hierarchical Task Learning from a Single Demonstration","authors":"Anahita Mohseni-Kabir, Charles Rich, Sonia Chernova, Candace L Sidner, Daniel Miller","year":"2015","venue":"HRI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Part and Lemon, 2016] Jose L. Part and Oliver Lemon. In- cremental On-Line Learning of Object Classes using a Combination of Self-Organizing Incremental Neural Net- works and Deep Convolutional Neural Networks. In Work- shop on Bio-inspired Social Robot Learning in Home Scenarios (IROS), Daejeon, Korea, October 2016.","title":"Incremental On-Line Learning of Object Classes using a Combination of Self-Organizing Incremental Neural Networks and Deep Convolutional Neural Networks","authors":"Jose L Part, Oliver Lemon","year":"2016","venue":"Workshop on Bio-inspired Social Robot Learning in Home Scenarios (IROS)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Tellex et al., 2011] S. Tellex, T. Kollar, S. Dickerson, M.R. Walter, A.G. Banerjee, S. Teller, and N. Roy. Understand- ing natural language commands for robotic navigation and mobile manipulation. In AAAI, 2011.","title":"Understanding natural language commands for robotic navigation and mobile manipulation","authors":"S Tellex, T Kollar, S Dickerson, M R Walter, A G Banerjee, S Teller, N Roy","year":"2011","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Yang et al., 2015] Yezhou Yang, Yi Li, Cornelia Fermuller, and Yiannis Aloimonos. Robot Learning Manipulation Action Plans by \"Watching\" Unconstrained Videos from the World Wide Web. In AAAI, pages 3686-3692, 2015.","title":"Robot Learning Manipulation Action Plans by \"Watching\" Unconstrained Videos from the World Wide Web","authors":"Yang","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}