{"type":"ScholarlyArticle","title":"Grounded Language Learning: Where Robotics and NLP Meet","authors":[{"name":"Cynthia Matuszek","slug":"cynthia-matuszek","given":"Cynthia","family":"Matuszek","affiliation":"University of Maryland , Baltimore County , Baltimore , MD"}],"abstract":"Grounded language acquisition is concerned with learning the meaning of language as it applies to the physical world. As robots become more capable and ubiquitous, there is an increasing need for non-specialists to interact with and control them, and natural language is an intuitive, flexible, and customizable mechanism for such communication. At the same time, physically embodied agents offer a way to learn to understand natural language in the context of the world to which it refers. This paper gives an overview of the research area, selected recent advances, and some future directions and challenges that remain.","doi":"10.24963/ijcai.2018/810","url":"https://ijcai.org/proceedings/2018/810","pdf":"https://ijcai.org/proceedings/2018/0810.pdf","markdown":"https://ijcai.org/proceedings/2018/810.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/810","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Early Career","track":"Early Career","number":810,"pages":{"first":5687,"last":5691},"keywords":["Robotics: Human Robot Interaction","Natural Language Processing: Natural Language Processing","Machine Learning Applications: Other Applications","Robotics: Learning in Robotics"],"videos":[],"references":[{"position":1,"raw":"[Arumugam et al., 2017] Dilip Arumugam, Siddharth Karamcheti, Nakul Gopalan, Lawson L.S. Wong, and Stefanie Tellex. Accurately and efficiently interpreting human-robot instructions of varying granularities. In Robotics: Science and Systems (RSS), 2017.","title":"Accurately and efficiently interpreting human-robot instructions of varying granularities","authors":"Dilip Arumugam, Siddharth Karamcheti, Nakul Gopalan, L S Lawson, Stefanie Wong, Tellex","year":"2017","venue":"Robotics: Science and Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Boteanu et al., 2017] Adrian Boteanu, Jacob Arkin, Sid- dharth Patki, Thomas Howard, and Hadas Kress-Gazit. Robot-initiated specification repair through grounded lan- guage interaction. In AAAI Fall Symposium on Natural Communication for Human-Robot Collaboration, 2017.","title":"Robot-initiated specification repair through grounded language interaction","authors":"Adrian Boteanu, Jacob Arkin, Siddharth Patki, Thomas Howard, Hadas Kress-Gazit","year":"2017","venue":"AAAI Fall Symposium on Natural Communication for Human-Robot Collaboration","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bozcuoglu et al., 2018] Asil Kaan Bozcuoglu, Gayane Kazhoyan, Yuki Furuta, Simon Stelter, Michael Beetz, Kei Okada, and Masayuki Inaba. The exchange of knowledge using cloud robotics. IEEE Robotics and Automation Let- ters (RA-L), 3(2):1072-1079, 2018.","title":"The exchange of knowledge using cloud robotics","authors":"Gayane Asil Kaan Bozcuoglu, Yuki Kazhoyan, Simon Furuta, Michael Stelter, Kei Beetz, Masayuki Okada, Inaba","year":"2018","venue":"IEEE Robotics and Automation Letters (RA-L)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Donahue et al., 2014] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Proc. of the 31 st International Conference on Machine Learning, 2014.","title":"Decaf: A deep convolutional activation feature for generic visual recognition","authors":"Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, Trevor Darrell","year":"2014","venue":"Proc. of the 31 st International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Harnad, 1990] Stevan Harnad. The symbol grounding prob- lem. Physica D: Nonlinear Phenomena, 42(1), 1990.","title":"The symbol grounding problem","authors":"Stevan Harnad","year":"1990","venue":"Physica D: Nonlinear Phenomena","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Long et al., 2015] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I Jordan. Learning transferable fea- tures with deep adaptation networks. In Proc. of the 32 nd International Conference on Machine Learning, 2015.","title":"Learning transferable features with deep adaptation networks","authors":"Mingsheng Long, Yue Cao, Jianmin Wang, Michael I Jordan","year":"2015","venue":"Proc. of the 32 nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Matuszek et al., 2012] Cynthia Matuszek, Nicholas Fitz- Gerald, Luke Zettlemoyer, Liefeng Bo, and Dieter Fox. A Joint Model of Language and Perception for Grounded Attribute Learning. In Proc. of the 29 th International Con- ference on Machine Learning (ICML), 2012.","title":"A Joint Model of Language and Perception for Grounded Attribute Learning","authors":"Cynthia Matuszek, Nicholas Fitz-Gerald, Luke Zettlemoyer, Liefeng Bo, Dieter Fox","year":"2012","venue":"Proc. of the 29 th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Matuszek et al., 2014] Cynthia Matuszek, Liefeng Bo, Luke Zettlemoyer, and Dieter Fox. Learning from un- scripted deictic gesture and language for human-robot in- teractions. In Proc. of the 28 th National Conference on Artificial Intelligence (AAAI), March 2014.","title":"Learning from unscripted deictic gesture and language for human-robot interactions","authors":"Cynthia Matuszek, Liefeng Bo, Luke Zettlemoyer, Dieter Fox","year":"2014","venue":"Proc. of the 28 th National Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositional- ity. In Advances in Neural Information Processing Systems (NIPS), 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Misra et al., 2017] Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with reinforcement learning. In Proc. of the 34 th Conference on Empirical Methods in Natural Language Processing (EMNLP), 2017.","title":"Mapping instructions and visual observations to actions with reinforcement learning","authors":"Dipendra Misra, John Langford, Yoav Artzi","year":"2017","venue":"Proc. of the 34 th Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mohan and Laird, 2014] Shiwali Mohan and John E Laird. Learning goal-oriented hierarchical tasks from situated in- teractive instruction. In Proc. of the 28 th National Confer- ence on Artificial Intelligence (AAAI), 2014.","title":"Learning goal-oriented hierarchical tasks from situated interactive instruction","authors":"Mohan, Laird ;, Shiwali Mohan, John E Laird","year":"2014","venue":"Proc. of the 28 th National Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mooney, 2008] Raymond J. Mooney. Learning to connect language and perception. In Dieter Fox and Carla P. Gomes, editors, Proc. of the 23 rd Conference on Artificial Intelligence (AAAI), pages 1598-1601, 2008.","title":"Learning to connect language and perception","authors":"Raymond J Mooney","year":"2008","venue":"Proc. of the 23 rd Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Newell and Simon, 1976] Allen Newell and Herbert A Si- mon. Computer science as empirical inquiry: Symbols and search. Communications of the ACM, 19(3), 1976.","title":"Computer science as empirical inquiry: Symbols and search","authors":"Simon Newell","year":"1976","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Paul et al., 2017] Rohan Paul, Andrei Barbu, Sue Felshin, Boris Katz, and Nicholas Roy. Temporal grounding graphs for language understanding with accrued visual-linguistic context. In Proc. of the 26 th International Joint Conference on Artificial Intelligence (IJCAI), 2017.","title":"Temporal grounding graphs for language understanding with accrued visual-linguistic context","authors":"Rohan Paul, Andrei Barbu, Sue Felshin, Boris Katz, Nicholas Roy","year":"2017","venue":"Proc. of the 26 th International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2017/629"},{"position":15,"raw":"[Pillai and Matuszek, 2018] Nisha Pillai and Cynthia Ma- tuszek. Unsupervised end-to-end data selection for grounded language learning. In Proc. of the 32 nd Na- tional Conference on Artificial Intelligence (AAAI), New Orleans, USA, February 2018.","title":"Unsupervised end-to-end data selection for grounded language learning","authors":"Nisha Pillai, Cynthia Matuszek","year":"2018","venue":"Proc. of the 32 nd National Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Roy, 2002] Deb K Roy. Learning visually grounded words and syntax for a scene description task. Computer Speech & Language, 16(3):353-385, 2002.","title":"Learning visually grounded words and syntax for a scene description task","authors":"Deb K Roy","year":"2002","venue":"Computer Speech & Language","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Schlangen et al., 2016] David Schlangen, Sina Zarrieß, and Casey Kennington. Resolving references to objects in pho- tographs using the words-as-classifiers model. In Proc. of the 54 th Annual Meeting of the Association for Computa- tional Linguistics (ACL 2016), 2016.","title":"Resolving references to objects in photographs using the words-as-classifiers model","authors":"David Schlangen, Sina Zarrieß, Casey Kennington","year":"2016","venue":"Proc. of the 54 th Annual Meeting of the Association for Computational Linguistics (ACL 2016)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Searle, 1980] John R Searle. Minds, brains, and programs. Behavioral and brain sciences, 3(03):417-424, 1980.","title":"Minds, brains, and programs","authors":"John R Searle","year":"1980","venue":"Behavioral and brain sciences","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Settles, 2012] Burr Settles. Active learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 6(1):1-114, 2012.","title":"Burr Settles. Active learning","authors":"Settles","year":"2012","venue":"Synthesis Lectures on Artificial Intelligence and Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[She and Chai, 2016] Lanbo She and Joyce Y Chai. Incre- mental acquisition of verb hypothesis space towards phys- ical world interaction. In Proc. of the 54 th Meeting of the Association for Computational Linguistics (ACL), 2016.","title":"Incremental acquisition of verb hypothesis space towards physical world interaction","authors":"Chai She, Lanbo She, Joyce Y Chai","year":"2016","venue":"Proc. of the 54 th Meeting of the Association for Computational Linguistics (ACL)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sinapov et al., 2016] Jivko Sinapov, Priyanka Khante, Maxwell Svetlik, and Peter Stone. Learning to order objects using haptic and proprioceptive exploratory behaviors. In Proc. of the 25 th International Joint Conference on Artificial Intelligence (IJCAI), 2016.","title":"Learning to order objects using haptic and proprioceptive exploratory behaviors","authors":"Jivko Sinapov, Priyanka Khante, Maxwell Svetlik, Peter Stone","year":"2016","venue":"Proc. of the 25 th International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2016/489"},{"position":22,"raw":"[Spranger and Steels, 2015] Michael Spranger and Luc Steels. Co-acquisition of syntax and semantics-an investigation in spatial language. In Proc. of the 24 th International Joint Conference on Artificial Intelligence (IJCAI), Buenos Aires, Argentina, 2015.","title":"Co-acquisition of syntax and semantics-an investigation in spatial language","authors":"Steels Spranger, Michael Spranger, Luc Steels","year":"2015","venue":"Proc. of the 24 th International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2015/271"},{"position":23,"raw":"[Thomason et al., 2018] Jesse Thomason, Jivko Sinapov, Raymond J Mooney, and Peter Stone. Guiding exploratory behaviors for multi-modal grounding of linguistic descrip- tions. In Proc. of the 32 nd National Conference on Artifi- cial Intelligence (AAAI), 2018.","title":"Guiding exploratory behaviors for multi-modal grounding of linguistic descriptions","authors":"Jesse Thomason, Jivko Sinapov, Raymond J Mooney, Peter Stone","year":"2018","venue":"Proc. of the 32 nd National Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tucker et al., 2017] Mycal Tucker, Derya Aksaray, Rohan Paul, Gregory J Stein, and Nicholas Roy. Learning un- known groundings for natural language interaction with mobile robots. In International Symposium on Robotics Research (ISRR), 2017.","title":"Learning unknown groundings for natural language interaction with mobile robots","authors":"Mycal Tucker, Derya Aksaray, Rohan Paul, Gregory J Stein, Nicholas Roy","year":"2017","venue":"International Symposium on Robotics Research (ISRR)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Williams and Scheutz, 2016] Tom Williams and Matthias Scheutz. A framework for resolving open-world refer- ential expressions in distributed heterogeneous knowledge bases. In Proc. of the 30 th Conference on Artificial Intelli- gence (AAAI), 2016.","title":"A framework for resolving open-world referential expressions in distributed heterogeneous knowledge bases","authors":"Scheutz Williams, Tom Williams, Matthias Scheutz","year":"2016","venue":"Proc. of the 30 th Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}