{"type":"ScholarlyArticle","title":"Inferring Human Attention by Learning Latent Intentions","authors":[{"name":"Ping Wei","slug":"ping-wei","given":"Ping","family":"Wei","affiliation":"Xi'an Jiaotong University , Xi'an , China; University of California , Los Angeles , Los Angeles , USA"},{"name":"Dan Xie","slug":"dan-xie","given":"Dan","family":"Xie","affiliation":"University of California , Los Angeles , Los Angeles , USA"},{"name":"Nanning Zheng","slug":"nanning-zheng","given":"Nanning","family":"Zheng","affiliation":"Xi'an Jiaotong University , Xi'an , China"},{"name":"Song-Chun Zhu","slug":"song-chun-zhu","given":"Song-Chun","family":"Zhu","affiliation":"University of California , Los Angeles , Los Angeles , USA"}],"abstract":"This paper addresses the problem of inferring 3D human attention in RGB-D videos at scene scale. 3D human attention describes where a human is looking in 3D scenes. We propose a probabilistic method to jointly model attention, intentions, and their interactions. Latent intentions guide human attention which conversely reveals the intention features. This mutual interaction makes attention inference a joint optimization with latent intentions. An EM-based approach is adopted to learn the latent intentions and model parameters. Given an RGB-D video with 3D human skeletons, a joint-state dynamic programming algorithm is utilized to jointly infer the latent intentions, the 3D attention directions, and the attention voxels in scene point clouds. Experiments on a new 3D human attention dataset prove the strength of our method.","doi":"10.24963/ijcai.2017/180","url":"https://ijcai.org/proceedings/2017/180","pdf":"https://ijcai.org/proceedings/2017/0180.pdf","markdown":"https://ijcai.org/proceedings/2017/180.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/180","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra, IIIA-CSIC","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Knowledge Representation, Reasoning, and Logic","track":"Main track","number":180,"pages":{"first":1297,"last":1303},"keywords":["Knowledge Representation, Reasoning, and Logic: Geometric, Spatial, and Temporal Reasoning","Robotics and Vision: Human Robot Interaction"],"videos":[],"references":[{"position":1,"raw":"[Arulampalam et al., 2002] M. Sanjeev Arulampalam, Si- mon Maskell, Neil Gordon, and Tim Clapp. A tuto- rial on particle filters for online nonlinear/non-gaussian bayesian tracking. IEEE Transactions on Signal Process- ing, 50(2):174-188, 2002.","title":"A tutorial on particle filters for online nonlinear/non-gaussian bayesian tracking","authors":"M Sanjeev Arulampalam, Simon Maskell, Neil Gordon, Tim Clapp","year":"2002","venue":"IEEE Transactions on Signal Processing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Belardinelli et al., 2015] Anna Belardinelli, Oliver Herbort, and Martin V. Butz. Goal-oriented gaze strategies afforded by object interaction. Vision Research, 106:47-57, 2015.","title":"Goal-oriented gaze strategies afforded by object interaction","authors":"Anna Belardinelli, Oliver Herbort, Martin V Butz","year":"2015","venue":"Vision Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Benfold and Reid, 2009] Ben Benfold and Ian Reid. Guid- ing visual surveillance by tracking human attention. In Proceedings of the 20th British Machine Vision Confer- ence, pages 1-11, 2009.","title":"Guiding visual surveillance by tracking human attention","authors":"Reid ; Benfold, Ben Benfold, Ian Reid","year":"2009","venue":"Proceedings of the 20th British Machine Vision Conference","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bishop, 2006] Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer-Verlag New York, 2006.","title":"Pattern Recognition and Machine Learning","authors":"Christopher M Bishop","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Borji et al., 2012] Ali Borji, Dicky N. Sihite, and Lauren- t Itti. Probabilistic learning of task-specic visual atten- tion. In Proceedings of Conference on Computer Vision and Pattern Recognition, pages 470-477, 2012.","title":"Probabilistic learning of task-specic visual attention","authors":"Ali Borji, N Dicky, Laurent Sihite, Itti","year":"2012","venue":"Proceedings of Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chang and Lin, 2011] Chih-Chung Chang and Chih-Jen Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1-27:27, 2011.","title":"LIBSVM: A library for support vector machines","authors":"Chang, Lin ; ] Chih-Chung Chang, Chih-Jen Lin","year":"2011","venue":"ACM Transactions on Intelligent Systems and Technology","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2008] Jixu Chen, Yan Tong, Wayne Gray, and Qiang Ji. A robust 3d eye gaze tracking system using noise reduction. In Proceedings of the Symposium on Eye Track- ing Research & Applications, pages 189-196, 2008.","title":"A robust 3d eye gaze tracking system using noise reduction","authors":"Chen","year":"2008","venue":"Proceedings of the Symposium on Eye Tracking Research & Applications","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Damen et al., 2016] Dima Damen, Teesid Leelasawassuk, and Walterio Mayol-Cuevas. You-do, i-learn: egocentric unsupervised discovery of objects and their modes of in- teraction towards video-based guidance. Computer Vision and Image Understanding, 149:98-112, 2016.","title":"You-do, i-learn: egocentric unsupervised discovery of objects and their modes of interaction towards video-based guidance","authors":"Dima Damen, Teesid Leelasawassuk, Walterio Mayol-Cuevas","year":"2016","venue":"Computer Vision and Image Understanding","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Duan et al., 2013] Dingrui Duan, Lu Tian, Jinshi Cui, Li Wang, Hongbin Zha, and Hamid Aghajan. Gaze es- timation in childrens peer-play scenarios. In Proceedings of the 2nd Asian Conference on Pattern Recognition, pages 760-764, 2013.","title":"Gaze estimation in childrens peer-play scenarios","authors":"Dingrui Duan, Lu Tian, Jinshi Cui, Li Wang, Hongbin Zha, Hamid Aghajan","year":"2013","venue":"Proceedings of the 2nd Asian Conference on Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fathi et al., 2012] Alireza Fathi, Yin Li, and James M. Re- hg. Learning to recognize daily actions using gaze. In Proceedings of the 12th European Conference on Comput- er Vision, pages 314-327, 2012.","title":"Rehg. Learning to recognize daily actions using gaze","authors":"Alireza Fathi, Yin Li, James M","year":"2012","venue":"Proceedings of the 12th European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Forney, 1973] G. David Forney. The viterbi algorithm. Pro- ceedings of the IEEE, 61(3):268-278, 1973.","title":"The viterbi algorithm","authors":"G David Forney","year":"1973","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Funes-Mora and Odobez, 2016] Kenneth A. Funes-Mora and Jean-Marc Odobez. Gaze estimation in the 3d space using rgb-d sensors. International Journal of Computer Vision, 118(2):194-216, 2016.","title":"Gaze estimation in the 3d space using rgb-d sensors","authors":"A Kenneth, Jean-Marc Funes-Mora, Odobez","year":"2016","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ghahramani and Hinton, 2000] Zoubin Ghahramani and Geoffrey E. Hinton. Variational learning for switching state-space models. Neural Computation, 12(4):831-864, 2000.","title":"Variational learning for switching state-space models","authors":"Zoubin Ghahramani, Geoffrey E Hinton","year":"2000","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hou and Zhang, 2008] Xiaodi Hou and Liqing Zhang. Dy- namic visual attention: searching for coding length in- crements. In Proceedings of the 21th International Con- ference on Neural Information Processing Systems, pages 681-688, 2008.","title":"Dynamic visual attention: searching for coding length increments","authors":"Zhang Hou, Xiaodi Hou, Liqing Zhang","year":"2008","venue":"Proceedings of the 21th International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Itti et al., 1998] Laurent Itti, Christof Koch, and Ernst Niebur. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 20(11):1254-1259, 1998.","title":"A model of saliency-based visual attention for rapid scene analysis","authors":"Laurent Itti, Christof Koch, Ernst Niebur","year":"1998","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jeni and Cohn, 2016] László A. Jeni and Jeffrey F. Cohn. Person-independent 3d gaze estimation using face frontal- ization. In Proceedings of IEEE Conference on Comput- er Vision and Pattern Recognition Workshops, pages 792- 800, 2016.","title":"Person-independent 3d gaze estimation using face frontalization","authors":"László A Jeni, Jeffrey F Cohn","year":"2016","venue":"Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kim, 1994] Chang-Jin Kim. Dynamic linear models with markov-switching. Journal of Econometrics, 60(1):1 -22, 1994.","title":"Dynamic linear models with markov-switching","authors":"Chang-Jin Kim","year":"1994","venue":"Journal of Econometrics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Land et al., 1999] Michael Land, Neil Mennie, and Jennifer Rusted. The roles of vision and eye movements in the con- trol of activities of daily living. Perception, 28(11):1311- 1328, 1999.","title":"The roles of vision and eye movements in the control of activities of daily living","authors":"Michael Land, Neil Mennie, Jennifer Rusted","year":"1999","venue":"Perception","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lanillos et al., 2015] Pablo Lanillos, João Filipe Ferreira, and Jorge Dias. Multisensory 3d saliency for artificial attention systems. In Proceedings of 3rd Workshop on Recognition and Action for Scene Understanding, 16th In- ternational Conference of Computer Analysis of Images and Patterns, pages 1-13, 2015.","title":"Pablo Lanillos, João Filipe Ferreira, and Jorge Dias. Multisensory 3d saliency for artificial attention systems","authors":"Lanillos","year":"2015","venue":"Proceedings of 3rd Workshop on Recognition and Action for Scene Understanding, 16th International Conference of Computer Analysis of Images and Patterns","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2013] Yin Li, Alireza Fathi, and James M. Rehg. Learning to predict gaze in egocentric video. In Proceed- ings of the IEEE International Conference on Computer Vision, pages 3216-3223, 2013.","title":"Learning to predict gaze in egocentric video","authors":"Yin Li, Alireza Fathi, James M Rehg","year":"2013","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2011] Tie Liu, Zejian Yuan, Jian Sun, Jingdong Wang, Nanning Zheng, Xiaoou Tang, and Heung-Yeung Shum. Learning to detect a salient object. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 33(2):353-367, 2011.","title":"Learning to detect a salient object","authors":"Liu","year":"2011","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mansouryar et al., 2016] Mohsen Mansouryar, Julian Steil, Yusuke Sugano, and Andreas Bulling. 3d gaze estimation from 2d pupil positions on monocular head-mounted eye trackers. In Proceedings of the Ninth Biennial ACM Sym- posium on Eye Tracking Research & Applications, pages 197-200, 2016.","title":"3d gaze estimation from 2d pupil positions on monocular head-mounted eye trackers","authors":"Mohsen Mansouryar, Julian Steil, Yusuke Sugano, Andreas Bulling","year":"2016","venue":"Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Marin-Jimenez et al., 2014] Manuel J. Marin-Jimenez, An- drew Zisserman, Marcin Eichner, and Vittorio Ferrari. De- tecting people looking at each other in videos. Internation- al Journal of Computer Vision, 106(3):282-296, 2014.","title":"Detecting people looking at each other in videos","authors":"J Manuel, ; Marin-Jimenez, Vittorio Ferrari","year":"2014","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mnih et al., 2014] Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. Recurrent models of vi- sual attention. In Proceedings of the International Con- ference on Neural Information Processing Systems, pages 2204-2212, 2014.","title":"Recurrent models of visual attention","authors":"Volodymyr Mnih, Nicolas Heess, Alex Graves, Koray Kavukcuoglu","year":"2014","venue":"Proceedings of the International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Recasens et al., 2015] Adrià Recasens, Aditya Khosla, Carl Vondrick, and Antonio Torralba. Where are they looking? In Proceedings of International Conference on Neural In- formation Processing Systems, pages 199-207, 2015.","title":"Where are they looking","authors":"Adrià Recasens, Aditya Khosla, Carl Vondrick, Antonio Torralba","year":"2015","venue":"Proceedings of International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Scholl, 2001] Brian J Scholl. Objects and attention: the s- tate of the art. Cognition, 80(12):1-46, 2001.","title":"Objects and attention: the state of the art","authors":"Brian J Scholl","year":"2001","venue":"Cognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sugano et al., 2014] Yusuke Sugano, Yasuyuki Matsushita, and Yoichi Sato. Learning-by-synthesis for appearance- based 3d gaze estimation. In Proceedings of IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 1821-1828, 2014.","title":"Learning-by-synthesis for appearancebased 3d gaze estimation","authors":"Sugano","year":"2014","venue":"Proceedings of IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yarbus, 1967] Alfred L. Yarbus. Eye Movements and Vision. Plenum Press, 1967.","title":"Eye Movements and Vision","authors":"Alfred L Yarbus","year":"1967","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yu and Smith, 2015] Chen Yu and Linda B. Smith. Linking joint attention with hand-eye coordination a sensorimotor approach to understanding child-parent social interaction. In Proceedings of the 37th Annual Meeting of the Cogni- tive Science Society, pages 2763-2768, 2015.","title":"Linking joint attention with hand-eye coordination a sensorimotor approach to understanding child-parent social interaction","authors":"Chen Yu, Linda B Smith","year":"2015","venue":"Proceedings of the 37th Annual Meeting of the Cognitive Science Society","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2015] Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. Appearance-based gaze es- timation in the wild. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 4511- 4520, 2015.","title":"Appearance-based gaze estimation in the wild","authors":"Zhang","year":"2015","venue":"Proceedings of IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}