{"type":"ScholarlyArticle","title":"Watching a Small Portion could be as Good as Watching All: Towards Efficient Video Classification","authors":[{"name":"Hehe Fan","slug":"hehe-fan","given":"Hehe","family":"Fan","affiliation":"SUSTech-UTS Joint Centre of CIS , Southern University of Science and Technology; Centre for Artificial Intelligence , University of Technology Sydney , Australia"},{"name":"Zhongwen Xu","slug":"zhongwen-xu","given":"Zhongwen","family":"Xu","affiliation":"Centre for Artificial Intelligence , University of Technology Sydney , Australia"},{"name":"Linchao Zhu","slug":"linchao-zhu","given":"Linchao","family":"Zhu","affiliation":"SUSTech-UTS Joint Centre of CIS , Southern University of Science and Technology; Centre for Artificial Intelligence , University of Technology Sydney , Australia"},{"name":"Chenggang Yan","slug":"chenggang-yan","given":"Chenggang","family":"Yan","affiliation":"Institute of Information and Control , Hangzhou Dianzi University , China"},{"name":"Jianjun Ge","slug":"jianjun-ge","given":"Jianjun","family":"Ge","affiliation":"Information Science Academy , CETC , China"},{"name":"Yi Yang","slug":"yi-yang","given":"Yi","family":"Yang","affiliation":"SUSTech-UTS Joint Centre of CIS , Southern University of Science and Technology; Centre for Artificial Intelligence , University of Technology Sydney , Australia"}],"abstract":"We aim to significantly reduce the computational cost for classification of temporally untrimmed videos while retaining similar accuracy. Existing video classification methods sample frames with a predefined frequency over entire video. Differently, we propose an end-to-end deep reinforcement approach which enables an agent to classify videos by watching a very small portion of frames like what we do. We make two main contributions.  First, information is not equally distributed in video frames along time. An agent needs to watch more carefully when a clip is informative and skip the frames if they are redundant or irrelevant. The proposed approach enables the agent to adapt sampling rate to video content and skip most of the frames without the loss of information. Second, in order to have a confident decision, the number of frames that should be watched by an agent varies greatly from one video to another. We incorporate an adaptive stop network to measure confidence score and generate timely trigger to stop the agent watching videos, which improves efficiency without loss of accuracy. Our approach reduces the computational cost significantly for the large-scale YouTube-8M dataset, while the accuracy remains the same.","doi":"10.24963/ijcai.2018/98","url":"https://ijcai.org/proceedings/2018/98","pdf":"https://ijcai.org/proceedings/2018/0098.pdf","markdown":"https://ijcai.org/proceedings/2018/98.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/98","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":98,"pages":{"first":705,"last":711},"keywords":["Machine Learning Applications: Applications of Reinforcement Learning","Computer Vision: Video: Events, Activities and Surveillance"],"videos":[],"references":[{"position":1,"raw":"[Abu-El-Haija et al., 2016] Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijaya- narasimhan. Youtube-8m: A large-scale video classifica- tion benchmark. CoRR, abs/1609.08675, 2016.","title":"Youtube-8m: A large-scale video classification benchmark","authors":"Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, Sudheendra Vijayanarasimhan","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bhatnagar et al., 2017] Bharat Lal Bhatnagar, Suriya Singh, Chetan Arora, and C. V. Jawahar. Unsupervised learning of deep feature representation for clustering egocentric ac- tions. In IJCAI, 2017.","title":"Unsupervised learning of deep feature representation for clustering egocentric actions","authors":"Bharat Lal Bhatnagar, Suriya Singh, Chetan Arora, C V Jawahar","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/200"},{"position":3,"raw":"[Chang et al., 2016] Xiaojun Chang, Yi Yang, Guodong Long, Chengqi Zhang, and Alexander G. Hauptmann. Dy- namic concept composition for zero-example event detec- tion. In AAAI, 2016.","title":"Dynamic concept composition for zero-example event detection","authors":"Xiaojun Chang, Yi Yang, Guodong Long, Chengqi Zhang, Alexander G Hauptmann","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fan et al., 2017] Hehe Fan, Xiaojun Chang, De Cheng, Yi Yang, Dong Xu, and Alexander G. Hauptmann. Com- plex event detection by identifying reliable shots from untrimmed videos. In ICCV, 2017.","title":"Complex event detection by identifying reliable shots from untrimmed videos","authors":"Fan","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Howard et al., 2017] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mo- bilenets: Efficient convolutional neural networks for mo- bile vision applications. arXiv preprint arXiv:1704.04861, 2017.","title":"Mobilenets: Efficient convolutional neural networks for mobile vision applications","authors":"Menglong Andrew G Howard, Bo Zhu, Dmitry Chen, Weijun Kalenichenko, Tobias Wang, Marco Weyand, Hartwig Andreetto, Adam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jouppi et al., 2017] Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor processing unit. arXiv preprint arXiv:1704.04760, 2017.","title":"In-datacenter performance analysis of a tensor processing unit","authors":"Cliff Norman P Jouppi, Nishant Young, David Patil, Gaurav Patterson, Raminder Agrawal, Sarah Bajwa, Suresh Bates, Nan Bhatia, Al Boden, Borchers","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Karpathy et al., 2014] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convo- lutional neural networks. In CVPR, 2014.","title":"Large-scale video classification with convolutional neural networks","authors":"Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, Li Fei-Fei","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mnih et al., 2014] Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. Recurrent models of vi- sual attention. In NIPS, 2014.","title":"Recurrent models of visual attention","authors":"Volodymyr Mnih, Nicolas Heess, Alex Graves, Koray Kavukcuoglu","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In ICML, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ng et al., 2015] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification. In CVPR, 2015.","title":"Beyond short snippets: Deep networks for video classification","authors":"Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, George Toderici","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pan et al., 2016] Pingbo Pan, Zhongwen Xu, Yi Yang, Fei Wu, and Yueting Zhuang. Hierarchical recurrent neural encoder for video representation with application to cap- tioning. In CVPR, 2016.","title":"Hierarchical recurrent neural encoder for video representation with application to captioning","authors":"Pan","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Two-stream convolutional networks for action recognition in videos. In NIPS, 2014.","title":"Two-stream convolutional networks for action recognition in videos","authors":"Zisserman ; Simonyan, Zisserman","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Szegedy et al., 2016] Christian Szegedy, Vincent Van- houcke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.","title":"Rethinking the inception architecture for computer vision","authors":"Szegedy","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tran et al., 2015] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In ICCV, 2015.","title":"Learning spatiotemporal features with 3d convolutional networks","authors":"Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tran et al., 2017] Du Tran, Jamie Ray, Zheng Shou, Shih- Fu Chang, and Manohar Paluri. Convnet architecture search for spatiotemporal feature learning. arXiv preprint arXiv:1708.05038, 2017.","title":"Convnet architecture search for spatiotemporal feature learning","authors":"Du Tran, Jamie Ray, Zheng Shou, Shih-Fu Chang, Manohar Paluri","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Venugopalan et al., 2015] Subhashini Venugopalan, Mar- cus Rohrbach, Jeff Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence -video to text. In ICCV, 2015.","title":"Sequence to sequence -video to text","authors":"Subhashini Venugopalan, Marcus Rohrbach, Jeff Donahue, Raymond Mooney, Trevor Darrell, Kate Saenko","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang and Schmid, 2013] Heng Wang and Cordelia Schmid. Action recognition with improved trajectories. In ICCV, 2013.","title":"Action recognition with improved trajectories","authors":"Heng Wang, Cordelia Schmid","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2011] Heng Wang, Alexander Kläser, Cordelia Schmid, and Cheng-Lin Liu. Action recognition by dense trajectories. In CVPR, 2011.","title":"Action recognition by dense trajectories","authors":"Heng Wang, Alexander Kläser, Cordelia Schmid, Cheng-Lin Liu","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2016] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016.","title":"Temporal segment networks: Towards good practices for deep action recognition","authors":"Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, Luc Van Gool","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Williams, 1992] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine Learning, 8:229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Ronald J Williams","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2015] Zhongwen Xu, Yi Yang, and Alexander G. Hauptmann. A discriminative CNN video representation for event detection. In CVPR, 2015.","title":"A discriminative CNN video representation for event detection","authors":"Zhongwen Xu, Yi Yang, Alexander G Hauptmann","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yeung et al., 2016] Serena Yeung, Olga Russakovsky, Greg Mori, and Li Fei-Fei. End-to-end learning of action detec- tion from frame glimpses in videos. In CVPR, 2016.","title":"End-to-end learning of action detection from frame glimpses in videos","authors":"Serena Yeung, Olga Russakovsky, Greg Mori, Li Fei-Fei","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yue-Hei Ng et al., 2015] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification. In CVPR, 2015.","title":"Beyond short snippets: Deep networks for video classification","authors":"Yue-Hei Ng","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhu et al., 2017] Linchao Zhu, Zhongwen Xu, Yi Yang, and Alexander G. Hauptmann. Uncovering the temporal con- text for video question answering. IJCV, 124(3):409-421, 2017.","title":"Uncovering the temporal context for video question answering","authors":"Zhu","year":"2017","venue":"IJCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}