{"type":"ScholarlyArticle","title":"Dual Track Multimodal Automatic Learning through Human-Robot Interaction","authors":[{"name":"Shuqiang Jiang","slug":"shuqiang-jiang","given":"Shuqiang","family":"Jiang","affiliation":"Key Laboratory of Intelligent Information Processing , Institute of Computing Technology , CAS , China; University of Chinese Academy of Sciences , China"},{"name":"Weiqing Min","slug":"weiqing-min","given":"Weiqing","family":"Min","affiliation":"Key Laboratory of Intelligent Information Processing , Institute of Computing Technology , CAS , China; University of Chinese Academy of Sciences , China"},{"name":"Xue Li","slug":"xue-li","given":"Xue","family":"Li","affiliation":"Key Laboratory of Intelligent Information Processing , Institute of Computing Technology , CAS , China; Shandong University of Science and Technology , China"},{"name":"Huayang Wang","slug":"huayang-wang","given":"Huayang","family":"Wang","affiliation":"Key Laboratory of Intelligent Information Processing , Institute of Computing Technology , CAS , China; University of Chinese Academy of Sciences , China"},{"name":"Jian Sun","slug":"jian-sun","given":"Jian","family":"Sun","affiliation":"Key Laboratory of Intelligent Information Processing , Institute of Computing Technology , CAS , China; Shandong University of Science and Technology , China"},{"name":"Jiaqi Zhou","slug":"jiaqi-zhou","given":"Jiaqi","family":"Zhou","affiliation":"Key Laboratory of Intelligent Information Processing , Institute of Computing Technology , CAS , China"}],"abstract":"Human beings are constantly improving their cognitive ability via automatic learning from the interaction with the environment. Two important aspects of automatic learning are the visual perception and knowledge acquisition. The fusion of these two aspects is vital for improving the intelligence and interaction performance of robots. Many automatic knowledge extraction and recognition methods have been widely studied. However, little work focuses on integrating automatic knowledge extraction and recognition into a unified framework to enable jointly visual perception and knowledge acquisition. To solve this problem, we propose a Dual Track Multimodal Automatic Learning (DTMAL) system, which consists of two components: Hybrid Incremental Learning (HIL) from the vision track and Multimodal Knowledge Extraction (MKE) from the knowledge track. HIL can incrementally improve recognition ability of the system by learning new object samples and new object concepts. MKE is capable of constructing and updating the multimodal knowledge items based on the recognized new objects from HIL and other knowledge by exploring the multimodal signals. The fusion of the two tracks is a mutual promotion process and jointly devote to the dual track learning. We have conducted the experiments through human-machine interaction and the experimental results  validated the effectiveness of our proposed system.","doi":"10.24963/ijcai.2017/626","url":"https://ijcai.org/proceedings/2017/626","pdf":"https://ijcai.org/proceedings/2017/0626.pdf","markdown":"https://ijcai.org/proceedings/2017/626.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/626","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Robotics and Vision","track":"Main track","number":626,"pages":{"first":4485,"last":4491},"keywords":["Robotics and Vision: Developmental Robotics","Robotics and Vision: Robotics and Vision","Robotics and Vision: Human Robot Interaction","Robotics and Vision: Cognitive Robotics"],"videos":[],"references":[{"position":1,"raw":"[Al-Omari et al., 2017] M. Al-Omari, P. Duckworth, D. C. Hogg, and A. G. Cohn. Natural language acquisition and grounding for embodied robotic systems. In In Proceed- ings of the Thirty-First AAAI Conference on Artificial In- telligence, pages 4349-4356, 2017.","title":"Natural language acquisition and grounding for embodied robotic systems","authors":"M Al-Omari, P Duckworth, D C Hogg, A G Cohn","year":"2017","venue":"Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Barsalou, 1999] L. W. Barsalou. Perceptual Symbol Sys- tems. Behavioral and Brain Sciences, 22:577-660, 1999.","title":"Perceptual Symbol Systems","authors":"L W Barsalou","year":"1999","venue":"Behavioral and Brain Sciences","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bordes et al., 2005] Antoine Bordes, Seyda Ertekin, Jason Weston, and Léon Bottou. Fast kernel classifiers with on- line and active learning. Journal of Machine Learning Re- search, 6:1579-1619, December 2005.","title":"Fast kernel classifiers with online and active learning","authors":"Antoine Bordes, Seyda Ertekin, Jason Weston, Léon Bottou","year":"2005","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cauwenberghs and Poggio, 2000] Gert Cauwenberghs and Tomaso A. Poggio. Incremental and decremental support vector machine learning. In Conference on Neural Infor- mation Processing Systems, pages 409-415, 2000.","title":"Incremental and decremental support vector machine learning","authors":"Poggio Cauwenberghs, Gert Cauwenberghs, Tomaso A Poggio","year":"2000","venue":"Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gotts, 2016] Stephen J. Gotts. Incremental learning of per- ceptual and conceptual representations and the puzzle of neural repetition suppression. Psychonomic Bulletin & Re- view, 23(4):1055-1071, 2016.","title":"Incremental learning of perceptual and conceptual representations and the puzzle of neural repetition suppression","authors":"Stephen J Gotts","year":"2016","venue":"Psychonomic Bulletin & Review","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hakkani-Tür et al., 2014] Dilek Hakkani-Tür, Asli C ¸elikyilmaz, Larry P. Heck, Gökhan Tür, and Geof- frey Zweig. Probabilistic enrichment of knowledge graph entities for relation detection in conversational understanding. In Annual Conference of the International Speech Communication Association, pages 2113-2117, 2014.","title":"Probabilistic enrichment of knowledge graph entities for relation detection in conversational understanding","authors":"Dilek Hakkani-Tür, C Asli, Larry P ¸elikyilmaz, Gökhan Heck, Geoffrey Tür, Zweig","year":"2014","venue":"Annual Conference of the International Speech Communication Association","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hixon et al., 2015] Ben Hixon, Peter Clark, and Hannaneh Hajishirzi. Learning knowledge graphs for question an- swering through conversational dialog. In The Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, pages 851-861, 2015.","title":"Learning knowledge graphs for question answering through conversational dialog","authors":"Hixon","year":"2015","venue":"The Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Johnson et al., 2015] Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Image retrieval using scene graphs. In In Proceedings IEEE Conference on Computer Vision and Pattern Recognition, pages 3668-3678, 2015.","title":"Image retrieval using scene graphs","authors":"Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David A Shamma, Michael S Bernstein, Fei-Fei Li","year":"2015","venue":"Proceedings IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Klein and Manning, 2003] Dan Klein and Christopher D. Manning. A* parsing: Fast exact viterbi parse selection. In Proceedings of the 2003 Conference of the North Amer- ican Chapter of the Association for Computational Lin- guistics on Human Language Technology, 2003.","title":"A* parsing: Fast exact viterbi parse selection","authors":"Dan Klein, Christopher D Manning","year":"2003","venue":"Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Conference on Neural Information Processing Systems, pages 1106-1114, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kuzborskij et al., 2013] I. Kuzborskij, F. Orabona, and B. Caputo. From n to n+1: Multiclass transfer incremental learning. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition, pages 3358-3365, June 2013.","title":"From n to n+1: Multiclass transfer incremental learning","authors":"Kuzborskij","year":"2013","venue":"Proceedings IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lampert et al., 2009] C.H. Lampert, H. Nickisch, and S. Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition, pages 951-958, June 2009.","title":"Learning to detect unseen object classes by between-class attribute transfer","authors":"C H Lampert, H Nickisch, S Harmeling","year":"2009","venue":"Proceedings IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lewis and Steedman, 2014] Mike Lewis and Mark Steed- man. A* CCG parsing with a supertag-factored model. In Proceedings of the 2014 Conference on Empirical Meth- ods in Natural Language Processing, pages 990-1000, 2014.","title":"A* CCG parsing with a supertag-factored model","authors":"Mike Lewis, Mark Steedman","year":"2014","venue":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2016] Xin Liu, Meina Kan, Wanglong Wu, Shiguang Shan, and Xilin Chen. VIPLFacenet: An open source deep face recognition SDK. Computing Research Repository, abs/1609.03892, 2016.","title":"VIPLFacenet: An open source deep face recognition SDK","authors":"Liu","year":"2016","venue":"Computing Research Repository","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lv et al., 2015] Xiong Lv, Shuqiang Jiang, Luis Herranz, and Shuang Wang. Rgb-d hand-held object recognition based on heterogeneous feature fusion. Journal of Com- puter Science and Technology, 30(2):340-352, 2015.","title":"Rgb-d hand-held object recognition based on heterogeneous feature fusion","authors":"Xiong Lv, Shuqiang Jiang, Luis Herranz, Shuang Wang","year":"2015","venue":"Journal of Computer Science and Technology","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Murata et al., 2002] Noboru Murata, Motoaki Kawanabe, Andreas Ziehe, Klaus-Robert Müller, and Shun ichi Amar- i. On-line learning in changing environments with applica- tions in supervised and unsupervised learning. volume 15, pages 743-760, 2002.","title":"On-line learning in changing environments with applications in supervised and unsupervised learning","authors":"Noboru Murata, Motoaki Kawanabe, Andreas Ziehe, Klaus-Robert Müller, Shun Ichi, Amari","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Parde et al., 2015] Natalie Parde, Adam Hair, Michalis Pa- pakostas, Konstantinos Tsiakas, Maria Dagioglou, Vange- lis Karkaletsis, and Rodney D Nielsen. Grounding the meaning of words through vision and interactive game- play. In International Conference on Artificial Intelli- gence, pages 1895-1901, 2015.","title":"Grounding the meaning of words through vision and interactive gameplay","authors":"Natalie Parde, Adam Hair, Michalis Papakostas, Konstantinos Tsiakas, Maria Dagioglou, Vangelis Karkaletsis, Rodney D Nielsen","year":"2015","venue":"International Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2015/269"},{"position":18,"raw":"[Paul et al., 2016] Rohan Paul, Jacob Arkin, Nicholas Roy, and Thomas M. Howard. Efficient grounding of abstract spatial concepts for natural language interaction with robot manipulators. In Robotics: Science and Systems, 2016.","title":"Efficient grounding of abstract spatial concepts for natural language interaction with robot manipulators","authors":"Rohan Paul, Jacob Arkin, Nicholas Roy, Thomas M Howard","year":"2016","venue":"Robotics: Science and Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ristin et al., 2015] M. Ristin, M. Guillaumin, J. Gall, and L. Van Gool. Incremental learning of random forests for large-scale image classification. Proceedings IEEE Trans- actions on Pattern Analysis and Machine Intelligence, PP(99):1-1, 2015.","title":"Incremental learning of random forests for large-scale image classification","authors":"M Ristin, M Guillaumin, J Gall, L Van Gool","year":"2015","venue":"Proceedings IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ruping, 2001] S. Ruping. Incremental learning with sup- port vector machines. In Proceedings IEEE International Conference on Data Mining, pages 641-642, 2001.","title":"Incremental learning with support vector machines","authors":"S Ruping","year":"2001","venue":"Proceedings IEEE International Conference on Data Mining","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Saxena et al., 2014] Ashutosh Saxena, Ashesh Jain, Ozan Sener, Aditya Jami, Dipendra K. Misra, and Hema S. Kop- pula. Robobrain: Large-scale knowledge engine for robot- s, April 12 2014.","title":"Robobrain: Large-scale knowledge engine for robots","authors":"Ashutosh Saxena, Ashesh Jain, Ozan Sener, Aditya Jami, K Dipendra, Hema S Misra, Koppula","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. Computing Research Repository, abs/1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":"Computing Research Repository","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tenorth and Beetz, 2013] Moritz Tenorth and Michael Beetz. Knowrob: A knowledge processing infrastructure for cognition-enabled robots. International Journal of Robotics Research, 32(5):566-590, 2013.","title":"Knowrob: A knowledge processing infrastructure for cognition-enabled robots","authors":"Beetz Tenorth, Moritz Tenorth, Michael Beetz","year":"2013","venue":"International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Thomason et al., 2016] Jesse Thomason, Jivko Sinapov, Maxwell Svetlik, Peter Stone, and Raymond Mooney. Learning multi-modal grounded linguistic semantics by playing \"I spy\". In Proceedings of the 25th Internation- al Joint Conference on Artificial Intelligence, pages 3477- 3483, July 2016.","title":"Learning multi-modal grounded linguistic semantics by playing \"I spy","authors":"Jesse Thomason, Jivko Sinapov, Maxwell Svetlik, Peter Stone, Raymond Mooney","year":"2016","venue":"Proceedings of the 25th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2016/491"},{"position":25,"raw":"[Zhu et al., 2015] Yuke Zhu, Ce Zhang, Christopher Ré, and Li Fei-Fei. Building a large-scale multimodal knowledge base for visual question answering. Computing Research Repository, abs/1507.05670, 2015.","title":"Building a large-scale multimodal knowledge base for visual question answering","authors":"Yuke Zhu, Ce Zhang, Christopher Ré, Li Fei-Fei","year":"2015","venue":"Computing Research Repository","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}