{"type":"ScholarlyArticle","title":"Representation Learning for Compressed Video Action Recognition via Attentive Cross-modal Interaction with Motion Enhancement","authors":[{"name":"Bing Li","slug":"bing-li","given":"Bing","family":"Li","affiliation":"State Key Laboratory of Software Development Environment , Beihang University , Beijing , China; School of Computer Science and Engineering , Beihang University , Beijing , China"},{"name":"Jiaxin Chen","slug":"jiaxin-chen","given":"Jiaxin","family":"Chen","affiliation":"School of Computer Science and Engineering , Beihang University , Beijing , China"},{"name":"Dongming Zhang","slug":"dongming-zhang","given":"Dongming","family":"Zhang","affiliation":"National Computer Network Emergency Response Technical Team , Coordination Center of China , Beijing , China"},{"name":"Xiuguo Bao","slug":"xiuguo-bao","given":"Xiuguo","family":"Bao","affiliation":"National Computer Network Emergency Response Technical Team , Coordination Center of China , Beijing , China"},{"name":"Di Huang","slug":"di-huang","given":"Di","family":"Huang","affiliation":"State Key Laboratory of Software Development Environment , Beihang University , Beijing , China; School of Computer Science and Engineering , Beihang University , Beijing , China"}],"abstract":"Compressed video action recognition has recently drawn growing attention, since it remarkably reduces the storage and computational cost via replacing raw videos by sparsely sampled RGB frames and compressed motion cues (e.g., motion vectors and residuals). However, this task severely suffers from the coarse and noisy dynamics and the insufficient fusion of the heterogeneous RGB and motion modalities. To address the two issues above, this paper proposes a novel framework, namely Attentive Cross-modal Interaction Network with Motion Enhancement (MEACI-Net). It follows the two-stream architecture, i.e. one for the RGB modality and the other for the motion modality. Particularly, the motion stream employs a multi-scale block embedded with a denoising module to enhance representation learning. The interaction between the two streams is then strengthened by introducing the Selective Motion Complement (SMC) and Cross-Modality Augment (CMA) modules, where SMC complements the RGB modality with spatio-temporally attentive local motion features and CMA further combines the two modalities with selective feature augmentation. Extensive experiments on the UCF-101, HMDB-51 and Kinetics-400 benchmarks demonstrate the effectiveness and efficiency of MEACI-Net.","doi":"10.24963/ijcai.2022/148","url":"https://ijcai.org/proceedings/2022/148","pdf":"https://ijcai.org/proceedings/2022/0148.pdf","markdown":"https://ijcai.org/proceedings/2022/148.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/148","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":148,"pages":{"first":1060,"last":1066},"keywords":["Computer Vision: Video analysis and understanding"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986861","length_sec":49,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985004","length_sec":270,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Battash et al., 2020] Barak Battash, Haim Barad, Hanlin Tang, and Amit Bleiweiss. Mimic the raw domain: Ac- celerating action recognition in the compressed domain. In CVPR Workshops, pages 684-685, 2020.","title":"Mimic the raw domain: Accelerating action recognition in the compressed domain","authors":"Barak Battash, Haim Barad, Hanlin Tang, Amit Bleiweiss","year":"2020","venue":"CVPR Workshops","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao et al., 2019] Haoyuan Cao, Shining Yu, and Jiashi Feng. Compressed video action recognition with refined motion vector. arXiv preprint arXiv:1910.02533, 2019.","title":"Compressed video action recognition with refined motion vector","authors":"Cao","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carreira and Zisserman, 2017] João Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. In CVPR, pages 4724-4733, 2017.","title":"Quo vadis, action recognition? A new model and the kinetics dataset","authors":"João Carreira, Andrew Zisserman","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Feichtenhofer et al., 2019] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In CVPR, pages 6202-6211, 2019.","title":"Slowfast networks for video recognition","authors":"Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, Kaiming He","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Huang et al., 2019] Shiyuan Huang, Xudong Lin, Svebor Karaman, and Shih-Fu Chang. Flow-distilled ip two- stream networks for compressed video actionrecognition. arXiv preprint arXiv:1912.04462, 2019.","title":"Flow-distilled ip twostream networks for compressed video actionrecognition","authors":"Shiyuan Huang, Xudong Lin, Svebor Karaman, Shih-Fu Chang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Huo et al., 2019] Yuqi Huo, Xiaoli Xu, Yao Lu, Yulei Niu, Zhiwu Lu, and Ji-Rong Wen. Mobile video action recog- nition. arXiv preprint arXiv:1908.10155, 2019.","title":"Mobile video action recognition","authors":"Yuqi Huo, Xiaoli Xu, Yao Lu, Yulei Niu, Zhiwu Lu, Ji-Rong Wen","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jiang et al., 2019] Boyuan Jiang, MengMeng Wang, Wei- hao Gan, Wei Wu, and Junjie Yan. Stm: Spatiotempo- ral and motion encoding for action recognition. In CVPR, pages 2000-2009, 2019.","title":"Stm: Spatiotemporal and motion encoding for action recognition","authors":"Boyuan Jiang, Mengmeng Wang, Weihao Gan, Wei Wu, Junjie Yan","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2020a] Jiapeng Li, Ping Wei, Yongchi Zhang, and Nanning Zheng. A slow-i-fast-p architecture for com- pressed video action recognition. In ACM MM, pages 2039-2047, 2020.","title":"A slow-i-fast-p architecture for compressed video action recognition","authors":"Li","year":"2020","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2020b] Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. Tea: Temporal excitation and aggregation for action recognition. In CVPR, pages 909- 918, 2020.","title":"Tea: Temporal excitation and aggregation for action recognition","authors":"Li","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin et al., 2019] Ji Lin, Chuang Gan, and Song Han. TSM: temporal shift module for efficient video understanding. In ICCV, pages 7082-7092, 2019.","title":"TSM: temporal shift module for efficient video understanding","authors":"Ji Lin, Chuang Gan, Song Han","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Moon et al., 2013] Young-Su Moon, Shi-Hwa Lee, Yong- Min Tai, and Junguk Cho. A fast low-light multi-image fusion with online image restoration. In ICCE, 2013.","title":"A fast low-light multi-image fusion with online image restoration","authors":"Moon","year":"2013","venue":"ICCE","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Shou et al., 2019] Zheng Shou, Xudong Lin, Yannis Kalan- tidis, Laura Sevilla-Lara, Marcus Rohrbach, Shih-Fu Chang, and Zhicheng Yan. Dmc-net: Generating discrimi- native motion cues for fast compressed video action recog- nition. In CVPR, pages 1268-1277, 2019.","title":"Dmc-net: Generating discriminative motion cues for fast compressed video action recognition","authors":"Zheng Shou, Xudong Lin, Yannis Kalantidis, Laura Sevilla-Lara, Marcus Rohrbach, Shih-Fu Chang, Zhicheng Yan","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Two-stream convolutional networks for action recognition in videos. In NeurIPS, pages 568-576, 2014.","title":"Two-stream convolutional networks for action recognition in videos","authors":"Zisserman Simonyan","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sun et al., 2018] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. Pwc-net: Cnns for optical flow us- ing pyramid, warping, and cost volume. In CVPR, pages 8934-8943, 2018.","title":"Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume","authors":"Deqing Sun, Xiaodong Yang, Ming-Yu Liu, Jan Kautz","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Tran et al., 2018] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In CVPR, pages 6450-6459, 2018.","title":"A closer look at spatiotemporal convolutions for action recognition","authors":"Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann Lecun, Manohar Paluri","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wang and Schmid, 2013] Heng Wang and Cordelia Schmid. Action recognition with improved trajectories. In ICCV, pages 3551-3558, 2013.","title":"Action recognition with improved trajectories","authors":"Wang, Schmid, Heng Wang, Cordelia Schmid","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2016] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, pages 20- 36, 2016.","title":"Temporal segment networks: Towards good practices for deep action recognition","authors":"Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, Luc Van Gool","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2018] Limin Wang, Wei Li, Wen Li, and Luc Van Gool. Appearance-and-relation networks for video classification. In CVPR, pages 1430-1439, 2018.","title":"Appearance-and-relation networks for video classification","authors":"Limin Wang, Wei Li, Wen Li, Luc Van Gool","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2020] Limin Wang, Zhan Tong, Bin Ji, and Gangshan Wu. Tdn: Temporal difference net- works for efficient action recognition. arXiv preprint arXiv:2012.10071, 2020.","title":"Tdn: Temporal difference networks for efficient action recognition","authors":"Limin Wang, Zhan Tong, Bin Ji, Gangshan Wu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wu et al., 2018] Chao-Yuan Wu, Manzil Zaheer, Hexiang Hu, R. Manmatha, Alexander J. Smola, and Philipp Krähenbühl. Compressed video action recognition. In CVPR, pages 6026-6035, 2018.","title":"Compressed video action recognition","authors":"Chao-Yuan, Manzil Wu, Hexiang Zaheer, R Hu, Alexander J Manmatha, Philipp Smola, Krähenbühl","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yue-Hei Ng et al., 2015] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification. In CVPR, pages 4694-4702, 2015.","title":"Beyond short snippets: Deep networks for video classification","authors":"Yue-Hei Ng","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2016] Bowen Zhang, Limin Wang, Zhe Wang, Yu Qiao, and Hanli Wang. Real-time action recognition with enhanced motion vector cnns. In CVPR, pages 2718- 2726, 2016.","title":"Real-time action recognition with enhanced motion vector cnns","authors":"Bowen Zhang, Limin Wang, Zhe Wang, Yu Qiao, Hanli Wang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2018] Bowen Zhang, Limin Wang, Zhe Wang, Yu Qiao, and Hanli Wang. Real-time action recognition with deeply transferred motion vector cnns. IEEE Trans- actions on Image Processing, 27(5):2326-2339, 2018.","title":"Real-time action recognition with deeply transferred motion vector cnns","authors":"Bowen Zhang, Limin Wang, Zhe Wang, Yu Qiao, Hanli Wang","year":"2018","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}