{"type":"ScholarlyArticle","title":"Grounding of Human Environments and Activities for Autonomous Robots","authors":[{"name":"Muhannad Alomari","slug":"muhannad-alomari","given":"Muhannad","family":"Alomari","affiliation":"University of Leeds , United Kingdom.; {scmara ,"},{"name":"Paul Duckworth","slug":"paul-duckworth","given":"Paul","family":"Duckworth","affiliation":"University of Leeds , United Kingdom.; {scmara ,"},{"name":"Nils Bore","slug":"nils-bore","given":"Nils","family":"Bore","affiliation":"Royal Institute of Technology (KTH) , Sweden"},{"name":"Majd Hawasly","slug":"majd-hawasly","given":"Majd","family":"Hawasly","affiliation":"University of Leeds , United Kingdom.; {scmara ,"},{"name":"David C. Hogg","slug":"david-c-hogg","given":"David C.","family":"Hogg","affiliation":"University of Leeds , United Kingdom.; {scmara ,"},{"name":"Anthony G. Cohn","slug":"anthony-g-cohn","given":"Anthony G.","family":"Cohn","affiliation":"University of Leeds , United Kingdom.; {scmara ,"}],"abstract":"With the recent proliferation of human-oriented robotic applications in domestic and industrial scenarios, it is vital for robots to continually learn about their environments and about the humans they share their environments with. In this paper, we present a novel, online, incremental framework for unsupervised symbol grounding in real-world, human environments for autonomous robots. We demonstrate the flexibility of the framework by learning about colours, people names, usable objects and simple human activities, integrating state-of-the-art object segmentation, pose estimation, activity analysis along with a number of sensory input encodings into a continual learning framework. Natural language is grounded to the learned concepts, enabling the robot to communicate in a human-understandable way. We show, using a challenging real-world dataset of human activities as perceived by a mobile robot, that our framework is able to extract useful concepts, ground natural language descriptions to them, and, as a proof-of-concept, generate simple sentences from templates to describe people and the activities they are engaged in.","doi":"10.24963/ijcai.2017/193","url":"https://ijcai.org/proceedings/2017/193","pdf":"https://ijcai.org/proceedings/2017/0193.pdf","markdown":"https://ijcai.org/proceedings/2017/193.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/193","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":193,"pages":{"first":1395,"last":1402},"keywords":["Robotics and Vision: Robotics","Machine Learning: Unsupervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Aksoy et al., 2017] Eren E. Aksoy, Ekaterina Ovchin- nikova, Adil Orhan, Yezhou Yang, and Tamim Asfour. Un- supervised linking of visual features to textual descriptions in long manipulation activities. IEEE RA-L, 2017.","title":"Unsupervised linking of visual features to textual descriptions in long manipulation activities","authors":"E Eren, Ekaterina Aksoy, Adil Ovchinnikova, Yezhou Orhan, Tamim Yang, Asfour","year":"2017","venue":"IEEE RA-L","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Allen, 1983] James F. Allen. Maintaining knowledge about temporal intervals. CACM, 26(11):832-843, 1983.","title":"Maintaining knowledge about temporal intervals","authors":"James F Allen","year":"1983","venue":"CACM","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Alomari et al., 2017] Muhannad Alomari, Paul Duckworth, David C. Hogg, and Anthony G. Cohn. Natural language acquisition and grounding for embodied robotic systems. In AAAI, 2017.","title":"Natural language acquisition and grounding for embodied robotic systems","authors":"Muhannad Alomari, Paul Duckworth, David C Hogg, Anthony G Cohn","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Beetz et al., 2011] Michael Beetz, Ulrich Klank, Ingo Kresse, Alexis Maldonado, Lorenz Mosenlechner, Dejan Pangercic, Thomas Ruhr, and M. Tenorth. Robotic room- mates making pancakes. In Humanoid Robots, 2011.","title":"Robotic roommates making pancakes","authors":"Michael Beetz, Ulrich Klank, Ingo Kresse, Alexis Maldonado, Lorenz Mosenlechner, Dejan Pangercic, Thomas Ruhr, M Tenorth","year":"2011","venue":"Humanoid Robots","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Blei et al., 2003] David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latent Dirichlet Allocation. JMLR, 3, 2003.","title":"Latent Dirichlet Allocation","authors":"David M Blei, Andrew Y Ng, Michael I Jordan","year":"2003","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bore et al., 2017] Nils Bore, Rares Ambrus, Patric Jensfelt, and John Folkesson. Efficient retrieval of arbitrary ob- jects from long-term robot observations. Robot Auton Syst, 2017.","title":"Efficient retrieval of arbitrary objects from long-term robot observations","authors":"Nils Bore, Rares Ambrus, Patric Jensfelt, John Folkesson","year":"2017","venue":"Robot Auton Syst","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2015] Juan Chen, Anthony G. Cohn, Dayou Liu, Shengsheng Wang, Jihong Ouyang, and Qiangyuan Yu. A survey of qualitative spatial representations. The Knowledge Engineering Review, 30(01):106-136, 2015.","title":"A survey of qualitative spatial representations","authors":"Juan Chen, Anthony G Cohn, Dayou Liu, Shengsheng Wang, Jihong Ouyang, Qiangyuan Yu","year":"2015","venue":"The Knowledge Engineering Review","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Clementini et al., 1997] Eliseo Clementini, Paolino Di Fe- lice, and Daniel Hernández. Qualitative representation of positional information. Artif Intell, 95(2):317 -356, 1997.","title":"Qualitative representation of positional information","authors":"Eliseo Clementini, Paolino Di Felice, Daniel Hernández","year":"1997","venue":"Artif Intell","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Craye et al., 2015] Céline Craye, David Filliat, and Jean- Franc ¸ois Goudou. Exploration strategies for incremental learning of object-based visual saliency. In ICDL-EpiRob, 2015.","title":"Exploration strategies for incremental learning of object-based visual saliency","authors":"Craye","year":"2015","venue":"ICDL-EpiRob","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Delafontaine et al., 2011] Matthias Delafontaine, An- thony G. Cohn, and Nico Van de Weghe. Implementing a qualitative calculus to analyse moving point objects. Expert Syst Appl, 38(5):5187 -5196, 2011.","title":"Implementing a qualitative calculus to analyse moving point objects","authors":"Matthias Delafontaine, Anthony G Cohn, Nico Van De Weghe","year":"2011","venue":"Expert Syst Appl","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Duckworth et al., 2017] Paul Duckworth, Muhannad Alo- mari, James Charles, David Hogg, and Anthony Cohn. La- tent Dirichlet Allocation for unsupervised activity analysis on an autonomous mobile robot. In AAAI, 2017.","title":"Latent Dirichlet Allocation for unsupervised activity analysis on an autonomous mobile robot","authors":"Duckworth","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gatsoulis et al., 2016] Yiannis Gatsoulis, Muhannad Alo- mari, Chris Burbridge, Christian Dondrup, Paul Duck- worth, Peter Lightbody, Marc Hanheide, Nick Hawes, and Anthony G. Cohn. QSRlib: a software library for online acquisition of Qualitative Spatial Relations from Video. In Workshop on Qualitative Reasoning, at IJCAI, 2016.","title":"QSRlib: a software library for online acquisition of Qualitative Spatial Relations from Video","authors":"Yiannis Gatsoulis, Muhannad Alomari, Chris Burbridge, Christian Dondrup, Paul Duckworth, Peter Lightbody, Marc Hanheide, Nick Hawes, Anthony G Cohn","year":"2016","venue":"Workshop on Qualitative Reasoning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gelman et al., 2014] Andrew Gelman, John B. Carlin, Hal S. Stern, and Donald B. Rubin. Bayesian data analy- sis, volume 2. Chapman & Hall/CRC Boca Raton, 2014.","title":"Bayesian data analysis","authors":"Andrew Gelman, John B Carlin, Hal S Stern, Donald B Rubin","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hemachandra et al., 2015] Sachithra Hemachandra, Felix Duvallet, Thomas M. Howard, Nicholas Roy, Anthony Stentz, and Matthew R. Walter. Learning models for fol- lowing natural language directions in unknown environ- ments. In ICRA, 2015.","title":"Learning models for following natural language directions in unknown environments","authors":"Hemachandra","year":"2015","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hoffman et al., 2010] Matthew Hoffman, Francis R. Bach, and David M. Blei. Online learning for Latent Dirichlet Allocation. In NIPS, 2010.","title":"Online learning for Latent Dirichlet Allocation","authors":"Matthew Hoffman, Francis R Bach, David M Blei","year":"2010","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jovan et al., 2016] Ferdian Jovan, Jeremy Wyatt, Nick Hawes, and Tomáš Krajník. A Poisson-Spectral Model for Modelling the Spatio-Temporal Patterns in Human Data Observed by a Robot. In IROS, 2016.","title":"A Poisson-Spectral Model for Modelling the Spatio-Temporal Patterns in Human Data Observed by a Robot","authors":"Jovan","year":"2016","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lauria et al., 2002] Stanislao Lauria, Guido Bugmann, Theocharis Kyriacou, and Ewan Klein. Mobile robot pro- gramming using natural language. Robot Auton Syst, 2002.","title":"Mobile robot programming using natural language","authors":"Stanislao Lauria, Guido Bugmann, Theocharis Kyriacou, Ewan Klein","year":"2002","venue":"Robot Auton Syst","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lienhart and Maydt, 2002] Rainer Lienhart and Jochen Maydt. An extended set of Haar-like features for rapid object detection. In Image Processing, 2002.","title":"An extended set of Haar-like features for rapid object detection","authors":"Rainer Lienhart, Jochen Maydt","year":"2002","venue":"Image Processing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Matuszek et al., 2013] Cynthia Matuszek, Evan Herbst, Luke Zettlemoyer, and Dieter Fox. Learning to parse nat- ural language commands to a robot control system. In Ex- perimental Robotics, 2013. [MetraLabs, 2016] MetraLabs. www.metralabs.com, 2016.","title":"Learning to parse natural language commands to a robot control system","authors":"Cynthia Matuszek, Evan Herbst, Luke Zettlemoyer, Dieter Fox","year":"2013","venue":"Experimental Robotics","doi":null,"url":"www.metralabs.com","internal_url":null},{"position":20,"raw":"[Moratz and Ragni, 2008] Reinhard Moratz and Marco Ragni. Qualitative spatial reasoning about relative point position. J Visual Lang Comput, 19(1):75-98, 2008.","title":"Qualitative spatial reasoning about relative point position","authors":"Ragni Moratz, Reinhard Moratz, Marco Ragni","year":"2008","venue":"J Visual Lang Comput","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[OpenNI, 2016] OpenNI. www.openni.org, 2016.","title":"OpenNI","authors":"Openni","year":"2016","venue":null,"doi":null,"url":"www.openni.org","internal_url":null},{"position":22,"raw":"[Papon et al., 2013] Jeremie Papon, Alexey Abramov, Markus Schoeler, and Florentin Worgotter. Voxel cloud connectivity segmentation-supervoxels for point clouds. In CVPR, 2013.","title":"Voxel cloud connectivity segmentation-supervoxels for point clouds","authors":"Jeremie Papon, Alexey Abramov, Markus Schoeler, Florentin Worgotter","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pfister et al., 2000] Hanspeter Pfister, Matthias Zwicker, Jeroen Van Baar, and Markus Gross. Surfels: Surface ele- ments as rendering primitives. In SIGGRAPH, 2000.","title":"Surfels: Surface elements as rendering primitives","authors":"Hanspeter Pfister, Matthias Zwicker, Jeroen Van Baar, Markus Gross","year":"2000","venue":"SIGGRAPH","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Posada and Buckley, 2004] David Posada and Thomas R. Buckley. Model selection and model averaging in phylo- genetics: advantages of Akaike information criterion and Bayesian approaches over likelihood ratio tests. System- atic Biology, 53(5):793-808, 2004.","title":"Model selection and model averaging in phylogenetics: advantages of Akaike information criterion and Bayesian approaches over likelihood ratio tests","authors":"David Posada, Thomas R Buckley","year":"2004","venue":"Systematic Biology","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rosenberg and Hirschberg, 2007] Andrew Rosenberg and Julia Bell Hirschberg. V-measure: A conditional entropy- based external cluster evaluation measure. In EMNLP- CoNLL, 2007.","title":"V-measure: A conditional entropybased external cluster evaluation measure","authors":"Andrew Rosenberg, Julia Bell Hirschberg","year":"2007","venue":"EMNLP-CoNLL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Schoeler et al., 2015] Markus Schoeler, Jeremie Papon, and Florentin Worgotter. Constrained planar cuts-object parti- tioning for point clouds. In CVPR, 2015.","title":"Constrained planar cuts-object partitioning for point clouds","authors":"Markus Schoeler, Jeremie Papon, Florentin Worgotter","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Schuster and Manning, 2016] Sebastian Schuster and","title":null,"authors":"Sebastian Schuster","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"Christopher D. Manning. Enhanced English universal dependencies: An improved representation for natural language understanding tasks. In LREC, 2016.","title":"Enhanced English universal dependencies: An improved representation for natural language understanding tasks","authors":"D Christopher, Manning","year":"2016","venue":"LREC","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[She et al., 2014] Lanbo She, Shaohua Yang, Yu Cheng, Yunyi Jia, Joyce Y. Chai, and Ning Xi. Back to the blocks world: Learning new actions through situated human- robot dialogue. In Meeting of the SIG on Discourse and Dialogue, 2014.","title":"Back to the blocks world: Learning new actions through situated humanrobot dialogue","authors":"Lanbo She, Shaohua Yang, Yu Cheng, Yunyi Jia, Joyce Y Chai, Ning Xi","year":"2014","venue":"Meeting of the SIG on Discourse and Dialogue","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sinapov et al., 2014] Jivko Sinapov, Connor Schenck, and Alexander Stoytchev. Learning relational object categories using behavioral exploration and multimodal perception. In ICRA, 2014.","title":"Learning relational object categories using behavioral exploration and multimodal perception","authors":"Jivko Sinapov, Connor Schenck, Alexander Stoytchev","year":"2014","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Song and Wang, 2005] Mingzhou Song and Hongbin Wang. Highly efficient incremental estimation of Gaus- sian mixture models for online data stream clustering. In Defense and Security, 2005.","title":"Highly efficient incremental estimation of Gaussian mixture models for online data stream clustering","authors":"Wang ; Song, Mingzhou Song, Hongbin Wang","year":"2005","venue":"Defense and Security","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Song et al., 2016] Young C. Song, Iftekhar Naim, Abdullah Al Mamun, Kaustubh Kulkarni, Parag Singla, Jiebo Luo, Daniel Gildea, and Henry Kautz. Unsupervised alignment of actions in video with text descriptions. In IJCAI, 2016.","title":"Unsupervised alignment of actions in video with text descriptions","authors":"Young C Song, Iftekhar Naim, Abdullah Al Mamun, Kaustubh Kulkarni, Parag Singla, Jiebo Luo, Daniel Gildea, Henry Kautz","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/289"},{"position":33,"raw":"[Spranger and Steels, 2015] Michael Spranger and Luc Steels. Co-Acquisition of Syntax and Semantics -An Investigation in Spatial Language. In IJCAI, pages 1909-1905. 2015.","title":"Co-Acquisition of Syntax and Semantics -An Investigation in Spatial Language","authors":"Steels Spranger, Michael Spranger, Luc Steels","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/271"},{"position":34,"raw":"[Tellex et al., 2011] Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R. Walter, Ashis G. Banerjee, Seth Teller, and Nicholas Roy. Approaching the symbol grounding problem with probabilistic graphical models. AI magazine, 32(4):64-76, 2011.","title":"Approaching the symbol grounding problem with probabilistic graphical models","authors":"Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R Walter, Ashis G Banerjee, Seth Teller, Nicholas Roy","year":"2011","venue":"AI magazine","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Turk and Pentland, 1991] Matthew Turk and Alex Pentland. Eigenfaces for recognition. J cognitive neurosci, 3(1), 1991.","title":"Matthew Turk and Alex Pentland. Eigenfaces for recognition","authors":"Pentland Turk","year":"1991","venue":"J cognitive neurosci","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Vinh et al., 2009] Nguyen Xuan Vinh, Julien Epps, and James Bailey. Information theoretic measures for cluster- ings comparison: Is a correction for chance necessary? In ICML, 2009.","title":"Information theoretic measures for clusterings comparison: Is a correction for chance necessary?","authors":"Xuan Nguyen, Julien Vinh, James Epps, Bailey","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wei et al., 2016] Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. Convolutional pose machines. In CVPR, 2016.","title":"Convolutional pose machines","authors":"Shih-En Wei, Varun Ramakrishna, Takeo Kanade, Yaser Sheikh","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Young et al., 2016] Jay Young, Valerio Basile, Lars Kunze, Elena Cabrio, and Nick Hawes. Towards lifelong object learning by integrating situated robot perception and se- mantic web mining. In ECAI, 2016.","title":"Towards lifelong object learning by integrating situated robot perception and semantic web mining","authors":"Jay Young, Valerio Basile, Lars Kunze, Elena Cabrio, Nick Hawes","year":"2016","venue":"ECAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}