{"type":"ScholarlyArticle","title":"Multi-Type Activity Recognition from a Robot's Viewpoint","authors":[{"name":"Ilaria Gori","slug":"ilaria-gori","given":"Ilaria","family":"Gori","affiliation":"Computer and Vision Research Center , The University of Texas at Austin , Austin , TX , USA"},{"name":"J. K. Aggarwal","slug":"j-k-aggarwal","given":"J. K.","family":"Aggarwal","affiliation":"Computer and Vision Research Center , The University of Texas at Austin , Austin , TX , USA"},{"name":"Larry Matthies","slug":"larry-matthies","given":"Larry","family":"Matthies","affiliation":"Jet Propulsion Laboratory , California Institute of Technology , Pasadena , CA , USA"},{"name":"Michael. S. Ryoo","slug":"michael-s-ryoo","given":"Michael. S.","family":"Ryoo","affiliation":"School of Informatics and Computing , Indiana University , Bloomington , IN , USA"}],"abstract":"The literature in computer vision is rich of works where different types of activities -- single actions, two persons interactions or ego-centric activities, to name a few -- have been analyzed. However, traditional methods treat such types of activities separately, while in real settings detecting and recognizing different types of activities simultaneously is necessary. We first design a new unified descriptor, called Relation History Image (RHI), which can be extracted from all the activity types we are interested in. We then formulate an optimization procedure to detect and recognize activities of different types. We assess our approach on a new dataset recorded from a robot-centric perspective as well as on publicly available datasets, and evaluate its quality compared to multiple baselines.","doi":"10.24963/ijcai.2017/680","url":"https://ijcai.org/proceedings/2017/680","pdf":"https://ijcai.org/proceedings/2017/0680.pdf","markdown":"https://ijcai.org/proceedings/2017/680.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/680","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Best Sister Conferences","track":"Best Sister Conferences","number":680,"pages":{"first":4849,"last":4853},"keywords":["Artificial Intelligence: computer vision"],"videos":[],"references":[{"position":1,"raw":"[Chaaraoui et al., 2013] Alexandros Andre Chaaraoui, Jose Ramon Padilla-Lopez, and Francisco Florez- Revuelta. Fusion of skeletal and silhouette-based features for human action recognition with rgb-d devices. In IEEE International Conference on Computer Vision Workshop (ICCVW), 2013.","title":"Fusion of skeletal and silhouette-based features for human action recognition with rgb-d devices","authors":"Alexandros Andre Chaaraoui, Jose Ramon Padilla-Lopez, Francisco Florez-Revuelta","year":"2013","venue":"IEEE International Conference on Computer Vision Workshop (ICCVW)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Evangelidis et al., 2014] Georgios Evangelidis, Gurkirt Singh, and Radu Horaud. Skeletal quads: Human action recognition using joint quadruples. International Conference on Pattern Recognition (ICPR), 2014.","title":"Skeletal quads: Human action recognition using joint quadruples","authors":"Georgios Evangelidis, Gurkirt Singh, Radu Horaud","year":"2014","venue":"International Conference on Pattern Recognition (ICPR)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fathi et al., 2012] Alireza Fathi, Jessica K. Hodgins, and James M. Rehg. Social interactions: A first-person per- spective. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.","title":"Social interactions: A first-person perspective","authors":"Alireza Fathi, Jessica K Hodgins, James M Rehg","year":"2012","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gori et al., 2016] Ilaria Gori, J. K. Aggarwal, Larry Matthies, and Michael S. Ryoo. Multitype activity recog- nition in robot-centric scenarios. IEEE Robotics and Au- tomation Letters, 1(1):593-600, Jan 2016.","title":"Multitype activity recognition in robot-centric scenarios","authors":"J K Ilaria Gori, Larry Aggarwal, Michael S Matthies, Ryoo","year":"2016","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ji et al., 2014] Yanli Ji, Guo Ye, and Hong Cheng. Inter- active body part contrast mining for human interaction recognition. IEEE International Conference on Multime- dia and Expo Workshops (ICMEW), 2014.","title":"Interactive body part contrast mining for human interaction recognition","authors":"Yanli Ji, Guo Ye, Hong Cheng","year":"2014","venue":"IEEE International Conference on Multimedia and Expo Workshops (ICMEW)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Khamis et al., 2012] Sameh Khamis, Vlad I. Morariu, and Larry S. Davis. Combining per-frame and per-track cues for multi-person action recognition. In European Confer- ence on Computer Vision (ECCV), 2012.","title":"Combining per-frame and per-track cues for multi-person action recognition","authors":"Sameh Khamis, Vlad I Morariu, Larry S Davis","year":"2012","venue":"European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kong et al., 2014] Yu Kong, Yunde Jia, and Yun Fu. Inter- active phrases: Semantic descriptions for human interac- tion recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014.","title":"Interactive phrases: Semantic descriptions for human interaction recognition","authors":"Yu Kong, Yunde Jia, Yun Fu","year":"2014","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lan et al., 2012] Tian Lan, Leonid Sigal, and Greg Mori. Social roles in hierarchical models for human activity recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.","title":"Social roles in hierarchical models for human activity recognition","authors":"Tian Lan, Leonid Sigal, Greg Mori","year":"2012","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Laptev and Lindeberg, 2003] Ivan Laptev and Tony Linde- berg. Space-time interest points. In IEEE International Conference on Computer Vision (ICCV), 2003.","title":"Space-time interest points","authors":"Laptev, Lindeberg ;, Tony Lindeberg","year":"2003","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2010] Wanqing Li, Zhengyou Zhang, and Zicheng Liu. Action recognition based on a bag of 3d points. In IEEE Conference on Computer Vision and Pattern Recog- nition Workshops (CVPRW), 2010.","title":"Action recognition based on a bag of 3d points","authors":"Wanqing Li, Zhengyou Zhang, Zicheng Liu","year":"2010","venue":"IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lu and Grauman, 2013] Zheng Lu and Kristen Grauman. Story-driven summarization for egocentric video. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013.","title":"Story-driven summarization for egocentric video","authors":"Zheng Lu, Kristen Grauman","year":"2013","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lu et al., 2014] Cewu Lu, Jiaya Jia, and Chi-Keung Tang. Range-sample depth feature for action recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.","title":"Range-sample depth feature for action recognition","authors":"Lu","year":"2014","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Luo et al., 2013] Jiajia Luo, Wei Wang, and Hairong Qi. Group sparsity and geometry constrained dictionary learn- ing for action recognition from depth maps. In IEEE Inter- national Conference on Computer Vision (ICCV), 2013.","title":"Group sparsity and geometry constrained dictionary learning for action recognition from depth maps","authors":"Jiajia Luo, Wei Wang, Hairong Qi","year":"2013","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Oreifej and Liu, 2013] Omar Oreifej and Zicheng Liu. Hon4d: Histogram of oriented 4d normals for activity recognition from depth sequences. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013.","title":"Omar Oreifej and Zicheng Liu. Hon4d: Histogram of oriented 4d normals for activity recognition from depth sequences","authors":"Liu Oreifej","year":"2013","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ryoo and Aggarwal, 2009] Michael S. Ryoo and J. K. Ag- garwal. Spatio-temporal relationship match: Video struc- ture comparison for recognition of complex human activ- ities. In IEEE International Conference on Computer Vi- sion (ICCV), 2009.","title":"Spatio-temporal relationship match: Video structure comparison for recognition of complex human activities","authors":"Michael S Ryoo, J K","year":"2009","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ryoo and Matthies, 2013] Michael S. Ryoo and Larry Matthies. First-person activity recognition: What are they doing to me? IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013.","title":"First-person activity recognition: What are they doing to me","authors":"Michael S Ryoo, Larry Matthies","year":"2013","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Vemulapalli et al., 2014] Raviteja Vemulapalli, Felipe Ar- rate, and Rama Chellappa. Human action recognition by representing 3d skeletons as points in a lie group. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.","title":"Raviteja Vemulapalli, Felipe Arrate, and Rama Chellappa. Human action recognition by representing 3d skeletons as points in a lie group","authors":"Vemulapalli","year":"2014","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2012a] Jiang Wang, Zicheng Liu, Jan Chorowski, Zhuoyuan Chen, and Ying Wu. Robust 3d action recognition with random occupancy patterns. In European Conference on Computer Vision (ECCV), 2012.","title":"Robust 3d action recognition with random occupancy patterns","authors":"Wang","year":"2012","venue":"European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2012b] Jiang Wang, Zicheng Liu, Ying Wu, and Junsong Yuan. Mining actionlet ensemble for action recognition with depth cameras. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.","title":"Mining actionlet ensemble for action recognition with depth cameras","authors":"Wang","year":"2012","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2013] Chunyu Wang, Yizhou Wang, and Alan L. Yuille. An approach to pose-based action recogni- tion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013.","title":"An approach to pose-based action recognition","authors":"Chunyu Wang, Yizhou Wang, Alan L Yuille","year":"2013","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xia and Aggarwal, 2013] Lu Xia and J. K. Aggarwal. Spatio-temporal depth cuboid similarity feature for activ- ity recognition using depth camera. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013.","title":"Spatio-temporal depth cuboid similarity feature for activity recognition using depth camera","authors":"Lu Xia, J K Aggarwal","year":"2013","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xia et al., 2011] Lu Xia, Chia-Chih Chen, and J. K. Aggar- wal. View invariant human action recognition using his- tograms of 3d joints. IEEE Conference on Computer Vi- sion and Pattern Recognition Workshop (CVPRW), 2011.","title":"View invariant human action recognition using histograms of 3d joints","authors":"Lu Xia, Chia-Chih Chen, J K Aggarwal","year":"2011","venue":"IEEE Conference on Computer Vision and Pattern Recognition Workshop","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xia et al., 2015] Lu Xia, Ilaria Gori, J. K. Aggarwal, and Michael S. Ryoo. Robot-centric activity recognition from first-person rgb-d videos. In IEEE Winter Conference on Applications of Computer Vision (WACV), 2015.","title":"Robot-centric activity recognition from first-person rgb-d videos","authors":"Lu Xia, Ilaria Gori, J K Aggarwal, Michael S Ryoo","year":"2015","venue":"IEEE Winter Conference on Applications of Computer Vision (WACV)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yun et al., 2012] Kiwon Yun, Jean Honorio, Debaleena Chattopadhyay, Tamara L. Berg, and Dimitris Samaras. Two-person interaction detection using body-pose fea- tures and multiple instance learning. In IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2012.","title":"Two-person interaction detection using body-pose features and multiple instance learning","authors":"Kiwon Yun, Jean Honorio, Debaleena Chattopadhyay, Tamara L Berg, Dimitris Samaras","year":"2012","venue":"IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}