{"type":"ScholarlyArticle","title":"Learning Multi-Modal Grounded Linguistic Semantics by Playing \"I Spy\"","authors":[{"name":"Jesse Thomason","slug":"jesse-thomason","given":"Jesse","family":"Thomason","affiliation":""},{"name":"Jivko Sinapov","slug":"jivko-sinapov","given":"Jivko","family":"Sinapov","affiliation":""},{"name":"Maxwell Svetlik","slug":"maxwell-svetlik","given":"Maxwell","family":"Svetlik","affiliation":""},{"name":"Peter Stone","slug":"peter-stone","given":"Peter","family":"Stone","affiliation":""},{"name":"Raymond J. Mooney","slug":"raymond-j-mooney","given":"Raymond J.","family":"Mooney","affiliation":""}],"abstract":"Grounded language learning bridges words like red and square with robot perception. The vast majority of existing work in this space limits robot perception to vision. In this paper, we build perceptual models that use haptic, auditory, and proprioceptive data acquired through robot exploratory behaviors to go beyond vision. Our system learns to ground natural language words describing objects using supervision from an interactive human-robot I Spy game. In this game, the human and robot take turns describing one object among several, then trying to guess which object the other has described. All supervision labels were gathered from human participants physically present to play this game with a robot. We demonstrate that our multi-modal system for grounding natural language outperforms a traditional, vision-only grounding framework by comparing the two on the I Spy task. We also provide a qualitative analysis of the groundings learned in the game, visualizing what words are understood better with multi-modal sensory information as well as identifying learned word meanings that correlate with physical object properties (e.g. small negatively correlates with object weight).","doi":null,"url":"https://ijcai.org/proceedings/2016/491","pdf":"https://ijcai.org/proceedings/2016/0491.pdf","markdown":"https://ijcai.org/proceedings/2016/491.md","bibtex":"https://ijcai.org/proceedings/2016/bibtex/491","volume":{"slug":"2016","title":"Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2016","url":"https://ijcai.org/proceedings/2016/","year":2016,"location":"New York, NY, USA","dates":"9–15 July 2016","editors":"Subbarao Kambhampati","isbn":"978-1-57735-770-4; 978-1-57735-771-1","publisher":"AAAI Press / International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":null,"track":"","number":491,"pages":{"first":3477,"last":3482},"keywords":[],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}