{"type":"ScholarlyArticle","title":"Temporal Grounding Graphs for Language Understanding with Accrued Visual-Linguistic Context","authors":[{"name":"Rohan Paul","slug":"rohan-paul","given":"Rohan","family":"Paul","affiliation":"Massachusetts Institute of Technology , Cambridge , MA"},{"name":"Andrei Barbu","slug":"andrei-barbu","given":"Andrei","family":"Barbu","affiliation":"Massachusetts Institute of Technology , Cambridge , MA"},{"name":"Sue Felshin","slug":"sue-felshin","given":"Sue","family":"Felshin","affiliation":"Massachusetts Institute of Technology , Cambridge , MA"},{"name":"Boris Katz","slug":"boris-katz","given":"Boris","family":"Katz","affiliation":"Massachusetts Institute of Technology , Cambridge , MA"},{"name":"Nicholas Roy","slug":"nicholas-roy","given":"Nicholas","family":"Roy","affiliation":"Massachusetts Institute of Technology , Cambridge , MA"}],"abstract":"A robot’s ability to understand or ground natural language instructions is fundamentally tied to its knowledge about the surrounding world. We present an approach to grounding natural language utterances in the context of factual information gathered through natural-language interactions and past visual observations. A probabilistic model estimates, from a natural language utterance, the objects, relations, and actions that the utterance refers to, the objectives for future robotic actions it implies, and generates a plan to execute those actions while updating a state representation to include newly acquired knowledge from the visual-linguistic context. Grounding a command necessitates a representation for past observations and interactions; however, maintaining the full context consisting of all possible observed objects, attributes, spatial relations, actions, etc., over time is intractable. Instead, our model, Temporal Grounding Graphs, maintains a learned state representation for a belief over factual groundings, those derived from natural-language interactions, and lazily infers new groundings from visual observations using the context implied by the utterance. This work significantly expands the range of language that a robot can understand by incorporating factual knowledge and observations of its workspace into its inference about the meaning and grounding of natural-language utterances.","doi":"10.24963/ijcai.2017/629","url":"https://ijcai.org/proceedings/2017/629","pdf":"https://ijcai.org/proceedings/2017/0629.pdf","markdown":"https://ijcai.org/proceedings/2017/629.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/629","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Robotics and Vision","track":"Main track","number":629,"pages":{"first":4506,"last":4514},"keywords":["Robotics and Vision: Human Robot Interaction","Robotics and Vision: Cognitive Robotics"],"videos":[],"references":[{"position":1,"raw":"[Andreas and Klein, 2015] Jacob Andreas and Dan Klein. Alignment-based compositional semantics for instruction following. In Empirical Methods in Natural Language Processing (EMNLP), 2015.","title":"Alignment-based compositional semantics for instruction following","authors":"Andreas, Klein, Jacob Andreas, Dan Klein","year":"2015","venue":"Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Artzi and Zettlemoyer, 2013] Yoav Artzi and Luke Zettlemoyer. Weakly supervised learning of semantic parsers for mapping in- structions to actions. Transactions of the Association for Compu- tational Linguistics, 1:49-62, 2013.","title":"Weakly supervised learning of semantic parsers for mapping instructions to actions","authors":"Zettlemoyer Artzi, Yoav Artzi, Luke Zettlemoyer","year":"2013","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Barrett et al., 2016] D. P. Barrett, A. Barbu, N. Siddharth, and J. M. Siskind. Saying what you're looking for: Linguistics meets video search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(10):2069-2081, Oct 2016.","title":"Saying what you're looking for: Linguistics meets video search","authors":"D P Barrett, A Barbu, N Siddharth, J M Siskind","year":"2016","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Berant et al., 2013] J. Berant, A. Chou, R. Frostig, and P. Liang. Semantic parsing on Freebase from question-answer pairs. In Empirical Methods in Natural Language Processing (EMNLP), 2013.","title":"Semantic parsing on Freebase from question-answer pairs","authors":"J Berant, A Chou, R Frostig, P Liang","year":"2013","venue":"Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Berzak et al., 2016] Yevgeni Berzak, Andrei Barbu, Daniel Harari, Boris Katz, and Shimon Ullman. Do you see what I mean? Visual resolution of linguistic ambiguities. arXiv preprint arXiv:1603.08079, 2016.","title":"Do you see what I mean? Visual resolution of linguistic ambiguities","authors":"Yevgeni Berzak, Andrei Barbu, Daniel Harari, Boris Katz, Shimon Ullman","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Calli et al., 2015] Berk Calli, Arjun Singh, Aaron Walsman, Sid- dhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. The YCB object and model set: Towards common benchmarks for manipula- tion research. In International Conference on Advanced Robotics (ICAR), pages 510-517. IEEE, 2015.","title":"The YCB object and model set: Towards common benchmarks for manipulation research","authors":"Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, Aaron M Dollar","year":"2015","venue":"International Conference on Advanced Robotics (ICAR)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cantrell et al., 2010] Rehj Cantrell, Matthias Scheutz, Paul Scher- merhorn, and Xuan Wu. Robust spoken instruction understanding for hri. In Proceedings of the 5th ACM/IEEE international con- ference on Human-robot interaction, pages 275-282. IEEE Press, 2010.","title":"Robust spoken instruction understanding for hri","authors":"Cantrell","year":"2010","venue":"Proceedings of the 5th ACM/IEEE international conference on Human-robot interaction","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen and Mooney, 2011] David L Chen and Raymond J Mooney. Learning to interpret natural language navigation instructions from observations. In AAAI, volume 2, pages 1-2, 2011.","title":"Learning to interpret natural language navigation instructions from observations","authors":"L David, Raymond J Chen, Mooney","year":"2011","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2010] David L Chen, Joohyun Kim, and Raymond J Mooney. Training a multilingual sportscaster: Using perceptual context to learn language. Journal of Artificial Intelligence Re- search, pages 397-435, 2010.","title":"Training a multilingual sportscaster: Using perceptual context to learn language","authors":"Joohyun David L Chen, Raymond J Kim, Mooney","year":"2010","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chung et al., 2015] Istvan Chung, Oron Propp, Matthew R Walter, and Thomas M Howard. On the performance of hierarchical distributed correspondence graphs for efficient symbol grounding of robot instructions. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5247-5252. IEEE, 2015.","title":"On the performance of hierarchical distributed correspondence graphs for efficient symbol grounding of robot instructions","authors":"Istvan Chung, Oron Propp, Matthew R Walter, Thomas M Howard","year":"2015","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ghahramani et al., 1997] Zoubin Ghahramani, Michael I Jordan, and Padhraic Smyth. Factorial hidden Markov models. Machine learning, 29(2-3):245-273, 1997.","title":"Factorial hidden Markov models","authors":"Zoubin Ghahramani, Padhraic Michael I Jordan, Smyth","year":"1997","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Guadarrama et al., 2013] Sergio Guadarrama, Lorenzo Riano, Dave Golland, Daniel Gouhring, Yangqing Jia, David Klein, Pieter Abbeel, and Trevor Darrell. Grounding spatial relations for human-robot interaction. In International Conference on In- telligent Robots and Systems (IROS), pages 1640-1647. IEEE, 2013.","title":"Grounding spatial relations for human-robot interaction","authors":"Sergio Guadarrama, Lorenzo Riano, Dave Golland, Daniel Gouhring, Yangqing Jia, David Klein, Pieter Abbeel, Trevor Darrell","year":"2013","venue":"International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hemachandra et al., 2015] Sachithra Hemachandra, Felix Duval- let, Thomas M. Howard, Nicholas Roy, Anthony Stentz, and Matthew R Walter. Learning models for following natural lan- guage directions in unknown environments. In Robotics and Au- tomation (ICRA), 2015 IEEE International Conference on, pages 5608-5615. IEEE, 2015.","title":"Learning models for following natural language directions in unknown environments","authors":"Hemachandra","year":"2015","venue":"Robotics and Automation (ICRA), 2015 IEEE International Conference on","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Howard et al., 2014] Thomas M Howard, Istvan Chung, Oron Propp, Matthew R Walter, and Nicholas Roy. Efficient natural language interfaces for assistive robots. In IEEE/RSJ Intl Conf. on Intelligent Robots and Systems (IROS) Work. on Rehabilitation and Assistive Robotics, 2014.","title":"Efficient natural language interfaces for assistive robots","authors":"Istvan Thomas M Howard, Oron Chung, Matthew R Propp, Nicholas Walter, Roy","year":"2014","venue":"IEEE/RSJ Intl Conf. on Intelligent Robots and Systems (IROS) Work. on Rehabilitation and Assistive Robotics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Katz, 1988] Boris Katz. Using English for indexing and retrieving. 1988.","title":"Using English for indexing and retrieving","authors":"Boris Katz","year":"1988","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kim and Mooney, 2012] Joohyun Kim and Raymond J Mooney. Unsupervised PCFG induction for grounded language learning with highly ambiguous supervision. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 433-444. Association for Computational Linguistics, 2012.","title":"Unsupervised PCFG induction for grounded language learning with highly ambiguous supervision","authors":"Mooney Kim, Joohyun Kim, Raymond J Mooney","year":"2012","venue":"Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2016] C. Liu, S. Yang1, S. Saba-Sadiya1, N. Shukla, Y. He, Song-Chun Zhu, , and J. Y. Chai. Jointly learning grounded task structures from language instruction and visual demonstration. In Empirical Methods in Natural Language Processing (EMNLP), 2016.","title":"Jointly learning grounded task structures from language instruction and visual demonstration","authors":"C Liu, S Yang1, S Saba-Sadiya1, N Shukla, Y He, J Y Song-Chun Zhu, Chai","year":"2016","venue":"Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Matuszek et al., 2014] Cynthia Matuszek, Liefeng Bo, Luke Zetlle- moyer, and Dieter Fox. Learning from unscripted deictic gesture and language for human-robot interactions. In Proceedings of the National Conference on Artificial Intelligence (AAAI), Québec City, Quebec, Canada, March 2014.","title":"Learning from unscripted deictic gesture and language for human-robot interactions","authors":"Cynthia Matuszek, Liefeng Bo, Luke Zetllemoyer, Dieter Fox","year":"2014","venue":"Proceedings of the National Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Misra et al., 2016] Dipendra K Misra, Jaeyong Sung, Kevin Lee, and Ashutosh Saxena. Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions. The International Journal of Robotics Research, 35(1-3):281-300, 2016.","title":"Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions","authors":"Jaeyong Dipendra K Misra, Kevin Sung, Ashutosh Lee, Saxena","year":"2016","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Paul et al., 2016] Rohan Paul, Jacob Arkin, Nicholas Roy, and Thomas M. Howard. Efficient grounding of abstract spatial con- cepts for natural language interaction with robot manipulators. In Proceedings of Robotics: Science and Systems, Ann Arbor, Michigan, June 2016.","title":"Efficient grounding of abstract spatial concepts for natural language interaction with robot manipulators","authors":"Rohan Paul, Jacob Arkin, Nicholas Roy, Thomas M Howard","year":"2016","venue":"Proceedings of Robotics: Science and Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Russell and Norvig, 1995] Stuart Russell and Peter Norvig. Artifi- cial intelligence: A modern approach. Prentice-Hall, Egnlewood Cliffs, 25:27, 1995.","title":"Artificial intelligence: A modern approach","authors":"Stuart Russell, Peter Norvig","year":"1995","venue":"Egnlewood Cliffs","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tellex et al., 2011] Stefanie Tellex, Thomas Kollar, Steven Dick- erson, Matthew R. Walter, Ashis Gopal Banerjee, Seth J Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the National Conference on Artificial Intelligence (AAAI), 2011.","title":"Understanding natural language commands for robotic navigation and mobile manipulation","authors":"Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R Walter, Ashis Gopal Banerjee, Seth J Teller, Nicholas Roy","year":"2011","venue":"Proceedings of the National Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Walter et al., 2014] Matthew R Walter, Sachithra Hemachandra, Bianca Homberg, Stefanie Tellex, and Seth Teller. A framework for learning semantic maps from grounded natural language de- scriptions. The International Journal of Robotics Research (IJRR), 33(9):1167-1190, August 2014.","title":"A framework for learning semantic maps from grounded natural language descriptions","authors":"Sachithra Matthew R Walter, Bianca Hemachandra, Stefanie Homberg, Seth Tellex, Teller","year":"2014","venue":"The International Journal of Robotics Research (IJRR)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yu et al., 2015] Haonan Yu, N Siddharth, Andrei Barbu, and Jef- frey Mark Siskind. A compositional framework for grounding language inference, generation, and acquisition in video. J. Artif. Intell. Res. (JAIR), 52:601-713, 2015.","title":"A compositional framework for grounding language inference, generation, and acquisition in video","authors":"Haonan Yu, N Siddharth, Andrei Barbu, Jeffrey Mark Siskind","year":"2015","venue":"J. Artif. Intell. Res. (JAIR)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zettlemoyer and Collins, 2007] Luke S Zettlemoyer and Michael Collins. Online learning of relaxed CCG grammars for parsing to logical form. In Proceedings of Empirical Methods in Natu- ral Language Processing and Computational Nautal Language Learning (EMNLP-CoNLL), pages 678-687, Prague, 2007.","title":"Online learning of relaxed CCG grammars for parsing to logical form","authors":"Collins ; Zettlemoyer, S Luke, Michael Zettlemoyer, Collins","year":"2007","venue":"Proceedings of Empirical Methods in Natural Language Processing and Computational Nautal Language Learning (EMNLP-CoNLL)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}