{"type":"ScholarlyArticle","title":"Deep Learning Based Multi-modal Addressee Recognition in Visual Scenes with Utterances","authors":[{"name":"Thao Le Minh","slug":"thao-le-minh","given":"Thao","family":"Le Minh","affiliation":"Tokyo Institute of Technology , Tokyo , Japan; JAPAN Research , Tokyo , Japan"},{"name":"Nobuyuki Shimizu","slug":"nobuyuki-shimizu","given":"Nobuyuki","family":"Shimizu","affiliation":"Yahoo Japan Corporation; Yahoo!"},{"name":"Takashi Miyazaki","slug":"takashi-miyazaki","given":"Takashi","family":"Miyazaki","affiliation":"Yahoo Japan Corporation"},{"name":"Koichi Shinoda","slug":"koichi-shinoda","given":"Koichi","family":"Shinoda","affiliation":"Tokyo Institute of Technology , Tokyo , Japan"}],"abstract":"With the widespread use of intelligent systems, such as smart speakers, addressee recognition has become a concern in human-computer interaction, as more and more people expect such systems to understand complicated social scenes, including those outdoors, in cafeterias, and hospitals. Because previous studies typically focused only on pre-specified tasks with limited conversational situations such as controlling smart homes, we created a mock dataset called Addressee Recognition in Visual Scenes with Utterances (ARVSU) that contains a vast body of image variations in visual scenes with an annotated utterance and a corresponding addressee for each scenario. We also propose a multi-modal deep-learning-based model that takes different human cues, specifically eye gazes and transcripts of an utterance corpus, into account to predict the conversational addressee from a specific speaker's view in various real-life conversational scenarios. To the best of our knowledge, we are the first to introduce an end-to-end deep learning model that combines vision and transcripts of utterance for addressee recognition. As a result, our study suggests that future addressee recognition can reach the ability to understand human intention in many social situations previously unexplored, and our modality dataset is a first step in promoting research in this field.","doi":"10.24963/ijcai.2018/214","url":"https://ijcai.org/proceedings/2018/214","pdf":"https://ijcai.org/proceedings/2018/0214.pdf","markdown":"https://ijcai.org/proceedings/2018/214.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/214","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Humans and AI","track":"Main track","number":214,"pages":{"first":1546,"last":1553},"keywords":["Humans and AI: Human-Computer Interaction","Computer Vision: Language and Vision","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Akhtiamov et al., 2017] Oleg Akhtiamov, Maxim Sidorov, Alexey Karpov, and Wolfgang Minker. Speech and text analysis for multimodal addressee detection in human- human-computer interaction. pages 2521-2525, 2017.","title":"Speech and text analysis for multimodal addressee detection in humanhuman-computer interaction","authors":"Oleg Akhtiamov, Maxim Sidorov, Alexey Karpov, Wolfgang Minker","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Antol et al., 2015a] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Ba- tra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In the IEEE International Conference on Computer Vision, pages 2425-2433, 2015.","title":"Vqa: Visual question answering","authors":"Antol","year":"2015","venue":"the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Antol et al., 2015b] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answer- ing. In the International Conference on Computer Vision (ICCV), 2015.","title":"Vqa: Visual question answering","authors":"Antol","year":"2015","venue":"the International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bobicev and Sokolova, 2017] Victoria Bobicev and Marina Sokolova. Inter-annotator agreement in sentiment analy- sis: Machine learning perspective. In the International Conference Recent Advances in Natural Language Pro- cessing, RANLP 2017, pages 97-102, 2017.","title":"Victoria Bobicev and Marina Sokolova. Inter-annotator agreement in sentiment analysis: Machine learning perspective","authors":"Sokolova Bobicev","year":"2017","venue":"the International Conference Recent Advances in Natural Language Processing, RANLP 2017","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cohen, 1960] Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measure- ment, 20(1):37-46, 1960.","title":"A coefficient of agreement for nominal scales","authors":"Jacob Cohen","year":"1960","venue":"Educational and Psychological Measurement","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Donahue et al., 2014] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference on machine learning (ICML), pages 647-655, 2014.","title":"Decaf: A deep convolutional activation feature for generic visual recognition","authors":"Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, Trevor Darrell","year":"2014","venue":"International conference on machine learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ganganwar, 2012] Vaishali Ganganwar. An overview of classification algorithms for imbalanced datasets. Inter- national Journal of Emerging Technology and Advanced Engineering, 2:42-47, 01 2012.","title":"An overview of classification algorithms for imbalanced datasets","authors":"Vaishali Ganganwar","year":"2012","venue":"International Journal of Emerging Technology and Advanced Engineering","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gupta et al., 2017] A. Gupta, Y. Miao, L. Neves, and F. Metze. Visual features for context-aware speech recog- nition. In 2017 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pages 5020- 5024, March 2017.","title":"Visual features for context-aware speech recognition","authors":"A Gupta, Y Miao, L Neves, F Metze","year":"2017","venue":"IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hallgren, 2012] Kevin Hallgren. Computing inter-rater re- liability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8:23- 34, 07 2012.","title":"Computing inter-rater reliability for observational data: An overview and tutorial","authors":"Kevin Hallgren","year":"2012","venue":"Tutorials in Quantitative Methods for Psychology","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Holthaus et al., 2016] Patrick Holthaus, Christian Leich- senring, Jasmin Bernotat, Viktor Richter, Marian Pohling, Birte Carlmeyer, Norman Köster, Sebastian Meyer zu Borgsen, René Zorn, Birte Schiffhauer, Kai Frederic Engelmann, Florian Lier, Simon Schulz, Philipp Cimi- ano, Friederike Eyssel, Thomas Hermann, Franz Kum- mert, David Schlangen, Sven Wachsmuth, Petra Wagner, Britta Wrede, and Sebastian Wrede. How to address smart homes with a social robot? a multi-modal corpus of user interactions with an intelligent environment. In the Inter- national Conference on Language Resources and Evalua- tion (LREC), 2016.","title":"How to address smart homes with a social robot? a multi-modal corpus of user interactions with an intelligent environment","authors":"Patrick Holthaus, Christian Leichsenring, Jasmin Bernotat, Viktor Richter, Marian Pohling, Birte Carlmeyer, Norman Köster, Sebastian Meyer Zu Borgsen, René Zorn, Birte Schiffhauer, Kai Frederic Engelmann, Florian Lier, Simon Schulz, Philipp Cimiano, Friederike Eyssel, Thomas Hermann, Franz Kummert, David Schlangen, Sven Wachsmuth, Petra Wagner, Britta Wrede, Sebastian Wrede","year":"2016","venue":"the International Conference on Language Resources and Evaluation (LREC)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jayagopi et al., 2013] Dinesh Babu Jayagopi, Samira Sheiki, David Klotz, Johannes Wienke, Jean-Marc Odobez, Sebastien Wrede, Vasil Khalidov, Laurent Nyugen, Britta Wrede, and Daniel Gatica-Perez. The vernissage corpus: A conversational human-robot- interaction dataset. In the 8th ACM/IEEE International Conference on Human-robot Interaction, HRI '13, pages 149-150, 2013.","title":"The vernissage corpus: A conversational human-robotinteraction dataset","authors":"Dinesh Babu, Jayagopi, Samira Sheiki, David Klotz, Johannes Wienke, Jean-Marc Odobez, Sebastien Wrede, Vasil Khalidov, Laurent Nyugen, Britta Wrede, Daniel Gatica-Perez","year":"2013","venue":"the 8th ACM/IEEE International Conference on Human-robot Interaction, HRI '13","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Johansson et al., 2013] Martin Johansson, Gabriel Skantze, and Joakim Gustafson. Head pose patterns in multiparty human-robot team-building interactions. In International Conference on Social Robotics, pages 351-360. Springer, 2013.","title":"Head pose patterns in multiparty human-robot team-building interactions","authors":"Martin Johansson, Gabriel Skantze, Joakim Gustafson","year":"2013","venue":"International Conference on Social Robotics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jovanovic et al., 2006] Natasa Jovanovic, Rieks op den Akker, and Anton Nijholt. Addressee identification in face-to-face meetings. In 11th Conference of the European Chapter of the Association for Computational Linguistics, 2006.","title":"Addressee identification in face-to-face meetings","authors":"Natasa Jovanovic, Rieks Op Den Akker, Anton Nijholt","year":"2006","venue":"11th Conference of the European Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Oquab et al., 2014] Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid- level image representations using convolutional neural net- works. In the IEEE conference on computer vision and pattern recognition, pages 1717-1724, 2014.","title":"Learning and transferring midlevel image representations using convolutional neural networks","authors":"Maxime Oquab, Leon Bottou, Ivan Laptev, Josef Sivic","year":"2014","venue":"the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ouchi and Tsuboi, 2016] Hiroki Ouchi and Yuta Tsuboi. Addressee and response selection for multi-party conver- sation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2133-2143, 2016.","title":"Addressee and response selection for multi-party conver-sation","authors":"Tsuboi Ouchi, Hiroki Ouchi, Yuta Tsuboi","year":"2016","venue":"Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pan and Yang, 2010] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowl- edge and data engineering, 22(10):1345-1359, 2010.","title":"A survey on transfer learning","authors":"Jialin Sinno, Qiang Pan, Yang","year":"2010","venue":"IEEE Transactions on knowledge and data engineering","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Parkhi et al., 2015] O. M. Parkhi, A. Vedaldi, and A. Zis- serman. Deep face recognition. In British Machine Vision Conference, 2015.","title":"Deep face recognition","authors":"O M Parkhi, A Vedaldi, A Zisserman","year":"2015","venue":"British Machine Vision Conference","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Conference on Empir- ical Methods in Natural Language Processing (EMNLP), 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pugachev et al., 2017] Aleksei Pugachev, Oleg Akhtiamov, Alexey Karpov, and Wolfgang Minker. Deep learning for acoustic addressee detection in spoken dialogue systems. In Conference on Artificial Intelligence and Natural Lan- guage, pages 45-53. Springer, 2017.","title":"Deep learning for acoustic addressee detection in spoken dialogue systems","authors":"Aleksei Pugachev, Oleg Akhtiamov, Alexey Karpov, Wolfgang Minker","year":"2017","venue":"Conference on Artificial Intelligence and Natural Language","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Recasens et al., 2015] Adria Recasens, Aditya Khosla, Carl Vondrick, and Antonio Torralba. Where are they looking? In Advances in Neural Information Processing Systems, pages 199-207, 2015.","title":"Where are they looking?","authors":"Adria Recasens, Aditya Khosla, Carl Vondrick, Antonio Torralba","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sheikhi and Odobez, 2015] Samira Sheikhi and Jean-Marc Odobez. Combining dynamic head pose-gaze mapping with the robot conversational state for attention recogni- tion in human-robot interactions. Pattern Recognition Let- ters, 66:81-90, 2015.","title":"Combining dynamic head pose-gaze mapping with the robot conversational state for attention recognition in human-robot interactions","authors":"Samira Sheikhi, Jean-Marc Odobez","year":"2015","venue":"Pattern Recognition Letters","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tsai et al., 2015a] T. J. Tsai, Andreas Stolcke, and Malcolm Slaney. Multimodal addressee detection in multiparty di- alogue systems. 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2314-2318, 2015.","title":"Multimodal addressee detection in multiparty dialogue systems","authors":"Tsai","year":"2015","venue":"IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tsai et al., 2015b] TJ Tsai, Andreas Stolcke, and Malcolm Slaney. A study of multimodal addressee detection in human-human-computer interaction. IEEE Transactions on Multimedia, 17(9):1550-1561, 2015.","title":"A study of multimodal addressee detection in human-human-computer interaction","authors":"Tsai","year":"2015","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[van Turnhout et al., 2005] Koen van Turnhout, Jacques Terken, Ilse Bakx, and Berry Eggen. Identifying the intended addressee in mixed human-human and human- computer interaction from non-verbal features. In the 7th International Conference on Multimodal Interfaces, ICMI '05, pages 175-182, 2005.","title":"Identifying the intended addressee in mixed human-human and humancomputer interaction from non-verbal features","authors":"Jacques Koen Van Turnhout, Ilse Terken, Berry Bakx, Eggen","year":"2005","venue":"the 7th International Conference on Multimodal Interfaces, ICMI '05","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In the IEEE conference on computer vision and pattern recognition, pages 3156- 3164, 2015.","title":"Show and tell: A neural image caption generator","authors":"Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan","year":"2015","venue":"the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}