{"type":"ScholarlyArticle","title":"Deeply-Supervised CNN Model for Action Recognition with Trainable Feature Aggregation","authors":[{"name":"Yang Li","slug":"yang-li","given":"Yang","family":"Li","affiliation":"School of Computer Science , Beijing Institute of Technology , Beijing , China"},{"name":"Kan Li","slug":"kan-li","given":"Kan","family":"Li","affiliation":"School of Computer Science , Beijing Institute of Technology , Beijing , China"},{"name":"Xinxin Wang","slug":"xinxin-wang","given":"Xinxin","family":"Wang","affiliation":"School of Computer Science , Beijing Institute of Technology , Beijing , China"}],"abstract":"In this paper, we propose a deeply-supervised CNN model for action recognition that fully exploits powerful hierarchical features of CNNs. In this model, we build multi-level video representations by applying our proposed aggregation module at different convolutional layers. Moreover, we train this model in a deep supervision manner, which brings improvement in both performance and efficiency. Meanwhile, in order to capture the temporal structure as well as preserve more details about actions, we propose a trainable aggregation module. It models the temporal evolution of each spatial location and projects them into a semantic space using the Vector of Locally Aggregated Descriptors (VLAD) technique. This deeply-supervised CNN model integrating the powerful aggregation module provides a promising solution to recognize actions in videos. We conduct experiments on two action recognition datasets: HMDB51 and UCF101. Results show that our model outperforms the state-of-the-art methods.","doi":"10.24963/ijcai.2018/112","url":"https://ijcai.org/proceedings/2018/112","pdf":"https://ijcai.org/proceedings/2018/0112.pdf","markdown":"https://ijcai.org/proceedings/2018/112.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/112","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":112,"pages":{"first":807,"last":813},"keywords":["Computer Vision: Action Recognition","Computer Vision: Computer Vision","Computer Vision: Video: Events, Activities and Surveillance"],"videos":[],"references":[{"position":1,"raw":"[Bau et al., 2017] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In CVPR, pages 3319-3327, 2017.","title":"Network dissection: Quantifying interpretability of deep visual representations","authors":"David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, Antonio Torralba","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bilen et al., 2016] Hakan Bilen, Basura Fernando, Efstra- tios Gavves, Andrea Vedaldi, and Stephen Gould. Dy- namic image networks for action recognition. In CVPR, pages 3034-3042, 2016.","title":"Dynamic image networks for action recognition","authors":"Hakan Bilen, Basura Fernando, Efstratios Gavves, Andrea Vedaldi, Stephen Gould","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carreira and Zisserman, 2017] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, pages 4724-4733, 2017.","title":"Quo vadis, action recognition? a new model and the kinetics dataset","authors":"Joao Carreira, Andrew Zisserman","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Donahue et al., 2015] Jeffrey Donahue, Lisa Anne Hen- dricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long- term recurrent convolutional networks for visual recogni- tion and description. In CVPR, pages 2625-2634, 2015.","title":"Longterm recurrent convolutional networks for visual recognition and description","authors":"Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, Trevor Darrell","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Duta et al., 2017] Ionut Cosmin Duta, Bogdan Ionescu, Kiyoharu Aizawa, Nicu Sebe, et al. Spatio-temporal vec- tor of locally max pooled features for action recognition in videos. In CVPR, pages 3205-3214, 2017.","title":"Spatio-temporal vector of locally max pooled features for action recognition in videos","authors":"Bogdan Ionut Cosmin Duta, Kiyoharu Ionescu, Nicu Aizawa, Sebe","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Feichtenhofer et al., 2016a] Christoph Feichtenhofer, Axel Pinz, and Richard Wildes. Spatiotemporal residual net- works for video action recognition. In NIPS, pages 3468- 3476, 2016.","title":"Spatiotemporal residual networks for video action recognition","authors":"Feichtenhofer","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Feichtenhofer et al., 2016b] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. In CVPR, pages 1933-1941, 2016.","title":"Convolutional two-stream network fusion for video action recognition","authors":"Feichtenhofer","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fernando et al., 2015] Basura Fernando, Stratis Gavves, Oramas Mogrovejo, José Antonio, Amir Ghodrati, and Tinne Tuytelaars. Modeling video evolution for action recognition. In CVPR, pages 5378-5387, 2015.","title":"Modeling video evolution for action recognition","authors":"Basura Fernando, Stratis Gavves, Oramas Mogrovejo, José Antonio, Amir Ghodrati, Tinne Tuytelaars","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Girdhar and Ramanan, 2017] Rohit Girdhar and Deva Ra- manan. Attentional pooling for action recognition. In NIPS, pages 33-44, 2017.","title":"Attentional pooling for action recognition","authors":"Ramanan Girdhar","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Girdhar et al., 2017] Rohit Girdhar, Deva Ramanan, Abhi- nav Gupta, Josef Sivic, and Bryan Russell. Actionvlad: Learning spatio-temporal aggregation for action classifi- cation. In CVPR, pages 3165-3174, 2017.","title":"Actionvlad: Learning spatio-temporal aggregation for action classification","authors":"Rohit Girdhar, Deva Ramanan, Abhinav Gupta, Josef Sivic, Bryan Russell","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In ICML, pages 448-456, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Karpathy et al., 2014] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolu- tional neural networks. In CVPR, pages 1725-1732, 2014.","title":"Large-scale video classification with convolutional neural networks","authors":"Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, Li Fei-Fei","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kuehne et al., 2011] Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recog- nition. In ICCV, pages 2556-2563, 2011.","title":"Hmdb: a large video database for human motion recognition","authors":"Kuehne","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lan et al., 2017] Zhenzhong Lan, Yi Zhu, Alexander G Hauptmann, and Shawn Newsam. Deep local video fea- ture for action recognition. In CVPRW, pages 1219-1225, 2017.","title":"Deep local video feature for action recognition","authors":"Zhenzhong Lan, Yi Zhu, Alexander G Hauptmann, Shawn Newsam","year":"2017","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lee et al., 2015] Chen-Yu Lee, Saining Xie, Patrick Gal- lagher, Zhengyou Zhang, and Zhuowen Tu. Deeply- supervised nets. In AISTATS, pages 562-570, 2015.","title":"Deeplysupervised nets","authors":"Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, Zhuowen Tu","year":"2015","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Maaten and Hinton, 2008] Laurens van der Maaten and Ge- offrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2605, 2008.","title":"Laurens van der Maaten and Geoffrey Hinton","authors":"Hinton Maaten","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Richard et al., 2017] Alexander Richard, Hildegard Kuehne, and Juergen Gall. Weakly supervised action learning with rnn based fine-to-coarse modeling. In CVPR, pages 1273-1282, 2017.","title":"Weakly supervised action learning with rnn based fine-to-coarse modeling","authors":"Alexander Richard, Hildegard Kuehne, Juergen Gall","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Two-stream convolutional networks for action recognition in videos. In NIPS, pages 568-576, 2014.","title":"Two-stream convolutional networks for action recognition in videos","authors":"Zisserman Simonyan","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Soomro et al., 2012] Khurram Soomro, Amir Roshan Za- mir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.","title":"Ucf101: A dataset of 101 human actions classes from videos in the wild","authors":"Khurram Soomro, Mubarak Amir Roshan Zamir, Shah","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Srivastava et al., 2015] Nitish Srivastava, Elman Mansi- mov, and Ruslan Salakhudinov. Unsupervised learning of video representations using lstms. In ICML, pages 843- 852, 2015.","title":"Unsupervised learning of video representations using lstms","authors":"Nitish Srivastava, Elman Mansimov, Ruslan Salakhudinov","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tran et al., 2015] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In ICCV, pages 4489-4497, 2015.","title":"Learning spatiotemporal features with 3d convolutional networks","authors":"Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2016] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, pages 20- 36, 2016.","title":"Temporal segment networks: Towards good practices for deep action recognition","authors":"Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, Luc Van Gool","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yue-Hei Ng et al., 2015] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification. In CVPR, pages 4694-4702, 2015.","title":"Beyond short snippets: Deep networks for video classification","authors":"Yue-Hei Ng","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2016] Bowen Zhang, Limin Wang, Zhe Wang, Yu Qiao, and Hanli Wang. Real-time action recognition with enhanced motion vector cnns. In CVPR, pages 2718- 2726, 2016.","title":"Real-time action recognition with enhanced motion vector cnns","authors":"Bowen Zhang, Limin Wang, Zhe Wang, Yu Qiao, Hanli Wang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}