{"type":"ScholarlyArticle","title":"Hallucinating Optical Flow Features for Video Classification","authors":[{"name":"Yongyi Tang","slug":"yongyi-tang","given":"Yongyi","family":"Tang","affiliation":"Tencent AI Lab"},{"name":"Lin Ma","slug":"lin-ma","given":"Lin","family":"Ma","affiliation":"Tencent AI Lab"},{"name":"Lianqiang Zhou","slug":"lianqiang-zhou","given":"Lianqiang","family":"Zhou","affiliation":"Tencent AI Lab"}],"abstract":"Appearance and motion are two key components to depict and characterize the video content. Currently, the two-stream models have achieved state-of-the-art performances on video classification. However, extracting motion information, specifically in the form of optical flow features, is extremely computationally expensive, especially for large-scale video classification.  In this paper, we propose a motion hallucination network, namely MoNet, to imagine the optical flow features from the appearance features, with no reliance on the optical flow computation. Specifically, MoNet models the temporal relationships of the appearance features and exploits the contextual relationships of the optical flow features with concurrent connections. Extensive experimental results demonstrate that the proposed MoNet can effectively and efficiently hallucinate the optical flow features, which together with the appearance features consistently improve the video classification performances. \r\nMoreover, MoNet can help cutting down almost a half of computational and data-storage burdens for the two-stream video classification. Our code is available at: https://github.com/YongyiTang92/MoNet-Features","doi":"10.24963/ijcai.2019/130","url":"https://ijcai.org/proceedings/2019/130","pdf":"https://ijcai.org/proceedings/2019/0130.pdf","markdown":"https://ijcai.org/proceedings/2019/130.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/130","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":130,"pages":{"first":926,"last":932},"keywords":["Computer Vision: Video: Events, Activities and Surveillance","Computer Vision: Action Recognition","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Abu-El-Haija et al., 2016] Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijaya- narasimhan. Youtube-8m: A large-scale video classifica- tion benchmark. arXiv preprint 1609.08675, 2016.","title":"Youtube-8m: A large-scale video classification benchmark","authors":"Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, Sudheendra Vijayanarasimhan","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brox et al., 2004] Thomas Brox, Andrés Bruhn, Nils Pa- penberg, and Joachim Weickert. High accuracy optical flow estimation based on a theory for warping. In ECCV, 2004.","title":"High accuracy optical flow estimation based on a theory for warping","authors":"Thomas Brox, Andrés Bruhn, Nils Papenberg, Joachim Weickert","year":"2004","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carreira and Zisserman, 2017] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017.","title":"Quo vadis, action recognition? a new model and the kinetics dataset","authors":"Joao Carreira, Andrew Zisserman","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2018a] Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat-Seng Chua. Temporally grounding nat- ural sentence in video. In EMNLP, 2018.","title":"Temporally grounding natural sentence in video","authors":"Chen","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2018b] Xinpeng Chen, Lin Ma, Wenhao Jiang, Jian Yao, and Wei Liu. Regularizing rnns for caption gen- eration by reconstructing the past with the present. In CVPR, 2018.","title":"Regularizing rnns for caption generation by reconstructing the past with the present","authors":"Chen","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2019] Jingyuan Chen, Lin Ma, Xinpeng Chen, Zequn Jie, and Jiebo Luo. Localizing natural language in videos. In AAAI, 2019.","title":"Localizing natural language in videos","authors":"Jingyuan Chen, Lin Ma, Xinpeng Chen, Zequn Jie, Jiebo Luo","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint 1406.1078, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Donahue et al., 2015] Jeffrey Donahue, Lisa Anne Hen- dricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long- term recurrent convolutional networks for visual recogni- tion and description. In CVPR, 2015.","title":"Longterm recurrent convolutional networks for visual recognition and description","authors":"Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, Trevor Darrell","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dosovitskiy et al., 2015] Alexey Dosovitskiy, Philipp Fis- cher, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with con- volutional networks. In CVPR, 2015.","title":"Flownet: Learning optical flow with convolutional networks","authors":"Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der, Daniel Smagt, Thomas Cremers, Brox","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Feng et al., 2018] Yang Feng, Lin Ma, Wei Liu, Tong Zhang, and Jiebo Luo. Video re-localization. In ECCV, 2018.","title":"Video re-localization","authors":"Yang Feng, Lin Ma, Wei Liu, Tong Zhang, Jiebo Luo","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Feng et al., 2019] Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. Spatio-temporal video re-localization by warp lstm. In CVPR, 2019.","title":"Spatio-temporal video re-localization by warp lstm","authors":"Yang Feng, Lin Ma, Wei Liu, Jiebo Luo","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gao et al., 2018] Ruohan Gao, Bo Xiong, and Kristen Grauman. Im2flow: Motion hallucination from static im- ages for action recognition. In CVPR, 2018.","title":"Im2flow: Motion hallucination from static images for action recognition","authors":"Gao","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gilmer et al., 2017] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. ICML, 2017.","title":"Neural message passing for quantum chemistry","authors":"Justin Gilmer, S Samuel, Patrick F Schoenholz, Oriol Riley, George E Vinyals, Dahl","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Graves and Jaitly, 2014] Alex Graves and Navdeep Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In ICML, 2014.","title":"Towards end-to-end speech recognition with recurrent neural networks","authors":"Alex Graves, Navdeep Jaitly","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ilg et al., 2017] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In CVPR, 2017.","title":"Flownet 2.0: Evolution of optical flow estimation with deep networks","authors":"Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, Thomas Brox","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Jiang et al., 2018] Wenhao Jiang, Lin Ma, Yu-Gang Jiang, Wei Liu, and Tong Zhang. Recurrent fusion network for image captioning. In ECCV, 2018.","title":"Recurrent fusion network for image captioning","authors":"Wenhao Jiang, Lin Ma, Yu-Gang Jiang, Wei Liu, Tong Zhang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kay et al., 2017] Will Kay, Joao Carreira, Karen Si- monyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 1705.06950, 2017.","title":"The kinetics human action video dataset","authors":"Kay","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2018] Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indrnn): Building a longer and deeper rnn. In CVPR, 2018.","title":"Independently recurrent neural network (indrnn): Building a longer and deeper rnn","authors":"Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, Yanbo Gao","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lin et al., 2018] Rongcheng Lin, Jing Xiao, and Jianping Fan. Nextvlad: An efficient neural network to aggre- gate frame-level features for large-scale video classifica- tion. arXiv:1811.05014, 2018.","title":"Nextvlad: An efficient neural network to aggregate frame-level features for large-scale video classification","authors":"Lin","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2019] Yuan Liu, Lin Ma, Yifeng Zhang, Wei Liu, and Shih-Fu Chang. Multi-granulairty generator for tem- poral action proposal. In CVPR, 2019.","title":"Multi-granulairty generator for temporal action proposal","authors":"Yuan Liu, Lin Ma, Yifeng Zhang, Wei Liu, Shih-Fu Chang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Miech et al., 2017] Antoine Miech, Ivan Laptev, and Josef Sivic. Learnable pooling with context gating for video classification. arXiv preprint 1706.06905, 2017.","title":"Learnable pooling with context gating for video classification","authors":"Antoine Miech, Ivan Laptev, Josef Sivic","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Peng and Schmid, 2016] Xiaojiang Peng and Cordelia Schmid. Multi-region two-stream r-cnn for action detection. In ECCV, 2016.","title":"Multi-region two-stream r-cnn for action detection","authors":"Schmid Peng, Xiaojiang Peng, Cordelia Schmid","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Schuster and Paliwal, 1997] Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673-2681, 1997.","title":"Bidirectional recurrent neural networks","authors":"Mike Schuster, Kuldip K Paliwal","year":"1997","venue":"IEEE Transactions on Signal Processing","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Two-stream convolutional networks for action recognition in videos. In NeurIPS, 2014.","title":"Two-stream convolutional networks for action recognition in videos","authors":"Zisserman ; Simonyan, Zisserman","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In CVPR, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tang et al., 2018] Yongyi Tang, Xing Zhang, Jingwen Wang, Shaoxiang Chen, Lin Ma, and Yu-Gang Jiang. Non-local netvlad encoding for video classification. arXiv:1810.00207, 2018.","title":"Non-local netvlad encoding for video classification","authors":"Yongyi Tang, Xing Zhang, Jingwen Wang, Shaoxiang Chen, Lin Ma, Yu-Gang Jiang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2018] Bairui Wang, Lin Ma, Wei Zhang, and Wei Liu. Reconstruction network for video captioning. In CVPR, 2018.","title":"Reconstruction network for video captioning","authors":"Bairui Wang, Lin Ma, Wei Zhang, Wei Liu","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wu et al., 2018] Chao-Yuan Wu, Manzil Zaheer, Hexiang Hu, R Manmatha, Alexander J Smola, and Philipp Krähenbühl. Compressed video action recognition. In CVPR, 2018.","title":"Compressed video action recognition","authors":"Chao-Yuan, Manzil Wu, Hexiang Zaheer, R Hu, Alexander J Manmatha, Philipp Smola, Krähenbühl","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zach et al., 2007] Christopher Zach, Thomas Pock, and Horst Bischof. A duality based approach for realtime tv-l 1 optical flow. In Joint Pattern Recognition Symposium, pages 214-223. Springer, 2007.","title":"A duality based approach for realtime tv-l 1 optical flow","authors":"Christopher Zach, Thomas Pock, Horst Bischof","year":"2007","venue":"Joint Pattern Recognition Symposium","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zheng et al., 2015] Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip HS Torr. Con- ditional random fields as recurrent neural networks. In ICCV, 2015.","title":"Conditional random fields as recurrent neural networks","authors":"Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, Philip Hs Torr","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhou et al., 2018] Bolei Zhou, Alex Andonian, and Antonio Torralba. Temporal relational reasoning in videos. ECCV, 2018.","title":"Temporal relational reasoning in videos","authors":"Zhou","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhu et al., 2017] Yi Zhu, Zhenzhong Lan, Shawn Newsam, and Alexander G Hauptmann. Hidden two-stream convo- lutional networks for action recognition. arXiv preprint 1704.00389, 2017.","title":"Hidden two-stream convolutional networks for action recognition","authors":"Yi Zhu, Zhenzhong Lan, Shawn Newsam, Alexander G Hauptmann","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}