{"type":"ScholarlyArticle","title":"Predicting Human Interaction via Relative Attention Model","authors":[{"name":"Yichao Yan","slug":"yichao-yan","given":"Yichao","family":"Yan","affiliation":"University , Shanghai , China"},{"name":"Bingbing Ni","slug":"bingbing-ni","given":"Bingbing","family":"Ni","affiliation":"University , Shanghai , China"},{"name":"Xiaokang Yang","slug":"xiaokang-yang","given":"Xiaokang","family":"Yang","affiliation":""}],"abstract":"Predicting human interaction is challenging as the on-going activity has to be inferred based on a partially observed video. Essentially, a good algorithm should effectively model the mutual influence between the two interacting subjects. Also, only a small region in the scene is discriminative for identifying the on-going interaction. In this work, we propose a relative attention model to explicitly address these difficulties. Built on a tri-coupled deep recurrent structure representing both interacting subjects and global interaction status, the proposed network collects spatio-temporal information from each subject, rectified with global interaction information, yielding effective interaction representation. Moreover, the proposed network also unifies an attention module to assign higher importance to the regions which are relevant to the on-going action. Extensive experiments have been conducted on two public datasets, and the results demonstrate that the proposed relative attention network successfully predicts informative regions between interacting subjects, which in turn yields superior human interaction prediction accuracy.","doi":"10.24963/ijcai.2017/453","url":"https://ijcai.org/proceedings/2017/453","pdf":"https://ijcai.org/proceedings/2017/0453.pdf","markdown":"https://ijcai.org/proceedings/2017/453.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/453","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":453,"pages":{"first":3245,"last":3251},"keywords":["Machine Learning: Deep Learning","Machine Learning: Feature Selection/Construction","Machine Learning: Neural Networks"],"videos":[],"references":[{"position":1,"raw":"[Ba et al., 2014] Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. Multiple object recognition with visual at- tention. CoRR, abs/1412.7755, 2014.","title":"Multiple object recognition with visual attention","authors":"Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baccouche et al., 2010] Moez Baccouche, Franck Mamalet, Christian Wolf, Christophe Garcia, and Atilla Baskurt. Ac- tion classification in soccer videos with long short-term memory recurrent neural networks. In ICANN, 2010.","title":"Action classification in soccer videos with long short-term memory recurrent neural networks","authors":"Moez Baccouche, Franck Mamalet, Christian Wolf, Christophe Garcia, Atilla Baskurt","year":"2010","venue":"ICANN","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brox et al., 2004] Thomas Brox, Andrés Bruhn, Nils Pa- penberg, and Joachim Weickert. High accuracy optical flow estimation based on a theory for warping. In ECCV, 2004.","title":"High accuracy optical flow estimation based on a theory for warping","authors":"Thomas Brox, Andrés Bruhn, Nils Papenberg, Joachim Weickert","year":"2004","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cao et al., 2013] Yu Cao, Daniel Paul Barrett, Andrei Barbu, Siddharth Narayanaswamy, Haonan Yu, Aaron Michaux, Yuewei Lin, Sven J. Dickinson, Jeffrey Mark Siskind, and Song Wang. Recognize human activities from partially observed videos. In CVPR, 2013.","title":"Recognize human activities from partially observed videos","authors":"Yu Cao, Paul Daniel, Andrei Barrett, Siddharth Barbu, Haonan Narayanaswamy, Aaron Yu, Yuewei Michaux, Sven J Lin, Jeffrey Mark Dickinson, Song Siskind, Wang","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen, 2015] Jia-Lin Chen. Weakly supervised learning of part-based models for interaction prediction via LDA. In ACM MM, 2015.","title":"Weakly supervised learning of part-based models for interaction prediction via LDA","authors":"Jia-Lin Chen","year":"2015","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Donahue et al., 2015] Jeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venu- gopalan, Trevor Darrell, and Kate Saenko. Long-term re- current convolutional networks for visual recognition and description. In CVPR, 2015.","title":"Long-term recurrent convolutional networks for visual recognition and description","authors":"Jeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Trevor Darrell, Kate Saenko","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ibrahim et al., 2016] Mostafa S. Ibrahim, Srikanth Muralid- haran, Zhiwei Deng, Arash Vahdat, and Greg Mori. A hi- erarchical deep temporal model for group activity recogni- tion. In CVPR, 2016.","title":"A hierarchical deep temporal model for group activity recognition","authors":"Mostafa S Ibrahim, Srikanth Muralidharan, Zhiwei Deng, Arash Vahdat, Greg Mori","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ji et al., 2013] Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3d convolutional neural networks for human action recognition. TPAMI, 35(1):221-31, 2013.","title":"3d convolutional neural networks for human action recognition","authors":"Ji","year":"2013","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jia et al., 2014] Yangqing Jia, Evan Shelhamer, Jeff Don- ahue, Sergey Karayev, Jonathan Long, Ross Girshick, Ser- gio Guadarrama, and Trevor Darrell. Caffe: Convolution- al architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.","title":"Caffe: Convolutional architecture for fast feature embedding","authors":"Jia","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Karpathy et al., 2014] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Fei-Fei Li. Large-scale video classification with convo- lutional neural networks. In CVPR, 2014.","title":"Large-scale video classification with convolutional neural networks","authors":"Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, Fei-Fei Li","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kong and Fu, 2016] Yu Kong and Yun Fu. Max-margin ac- tion prediction machine. TPAMI, 38(9):1844-1858, 2016.","title":"Max-margin action prediction machine","authors":"Yu Kong, Yun Fu","year":"2016","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kong et al., 2012] Yu Kong, Yunde Jia, and Yun Fu. Learn- ing human interaction by interactive phrases. In ECCV, 2012.","title":"Learning human interaction by interactive phrases","authors":"Yu Kong, Yunde Jia, Yun Fu","year":"2012","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kong et al., 2014] Yu Kong, Dmitry Kit, and Yun Fu. A dis- criminative model with multiple temporal scales for action prediction. In ECCV, 2014.","title":"A discriminative model with multiple temporal scales for action prediction","authors":"Yu Kong, Dmitry Kit, Yun Fu","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lan et al., 2014] Tian Lan, Tsung-Chuan Chen, and Silvio Savarese. A hierarchical representation for future action prediction. In ECCV, 2014.","title":"A hierarchical representation for future action prediction","authors":"Tian Lan, Tsung-Chuan Chen, Silvio Savarese","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ma et al., 2016] Shugao Ma, Leonid Sigal, and Stan Sclaroff. Learning activity progression in lstms for activity detection and early detection. In CVPR, 2016.","title":"Learning activity progression in lstms for activity detection and early detection","authors":"Ma","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mnih et al., 2014] Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. Recurrent models of vi- sual attention. In NIPS, 2014.","title":"Recurrent models of visual attention","authors":"Volodymyr Mnih, Nicolas Heess, Alex Graves, Koray Kavukcuoglu","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ng et al., 2015] Joe Yue-Hei Ng, Matthew J. Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Mon- ga, and George Toderici. Beyond short snippets: Deep networks for video classification. In CVPR, 2015.","title":"Beyond short snippets: Deep networks for video classification","authors":"Joe Yue-Hei Ng, Matthew J Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, George Toderici","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nguyen and la Torre, 2012] Minh Hoai Nguyen and Fer- nando De la Torre. Max-margin early event detectors. In CVPR, 2012.","title":"Max-margin early event detectors","authors":"La Nguyen, ; Torre, Torre De La","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ryoo and Aggarwal, 2010] M. S. Ryoo and J. K. Aggarw- al. UT-Interaction Dataset, ICPR contest on Semantic De- scription of Human Activities (SDHA), 2010.","title":"UT-Interaction Dataset","authors":"M S Ryoo, J K Aggarwal","year":"2010","venue":"ICPR contest on Semantic Description of Human Activities (SDHA)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ryoo, 2011] M. S. Ryoo. Human activity prediction: Early recognition of ongoing activities from streaming videos. In ICCV, 2011.","title":"Human activity prediction: Early recognition of ongoing activities from streaming videos","authors":"M S Ryoo","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sharma et al., 2015] Shikhar Sharma, Ryan Kiros, and Rus- lan Salakhutdinov. Action recognition using visual atten- tion. CoRR, abs/1511.04119, 2015.","title":"Action recognition using visual attention","authors":"Shikhar Sharma, Ryan Kiros, Ruslan Salakhutdinov","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Two-stream convolutional networks for action recognition in videos. In NIPS, 2014.","title":"Two-stream convolutional networks for action recognition in videos","authors":"Zisserman ; Simonyan, Zisserman","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Song et al., 2016] Sijie Song, Cuiling Lan, Junliang Xing, Wenjun Zeng, and Jiaying Liu. An end-to-end spatio- temporal attention model for human action recognition from skeleton data. CoRR, abs/1611.06067, 2016.","title":"An end-to-end spatiotemporal attention model for human action recognition from skeleton data","authors":"Sijie Song, Cuiling Lan, Junliang Xing, Wenjun Zeng, Jiaying Liu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wu et al., 2015] Zuxuan Wu, Xi Wang, Yu-Gang Jiang, Hao Ye, and Xiangyang Xue. Modeling spatial-temporal clues in a hybrid deep learning framework for video classifica- tion. In ACM MM, 2015.","title":"Modeling spatial-temporal clues in a hybrid deep learning framework for video classification","authors":"Zuxuan Wu, Xi Wang, Yu-Gang Jiang, Hao Ye, Xiangyang Xue","year":"2015","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xu et al., 2015] Zhen Xu, Laiyun Qing, and Jun Miao. Ac- tivity auto-completion: Predicting human activities from partial videos. In ICCV, 2015.","title":"Activity auto-completion: Predicting human activities from partial videos","authors":"Zhen Xu, Laiyun Qing, Jun Miao","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}