{"type":"ScholarlyArticle","title":"Long-Short Term Cross-Transformer in Compressed Domain for Few-Shot Video Classification","authors":[{"name":"Wenyang Luo","slug":"wenyang-luo","given":"Wenyang","family":"Luo","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation , Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences"},{"name":"Yufan Liu","slug":"yufan-liu","given":"Yufan","family":"Liu","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation , Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences"},{"name":"Bing Li","slug":"bing-li","given":"Bing","family":"Li","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation , Chinese Academy of Sciences; PeopleAI, Inc ."},{"name":"Weiming Hu","slug":"weiming-hu","given":"Weiming","family":"Hu","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation , Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences; CAS Center for Excellence in Brain Science and Intelligence Technology"},{"name":"Yanan Miao","slug":"yanan-miao","given":"Yanan","family":"Miao","affiliation":"National Computer Network Emergency Response Technical Team/Coordination Center of China"},{"name":"Yangxi Li","slug":"yangxi-li","given":"Yangxi","family":"Li","affiliation":"National Computer Network Emergency Response Technical Team/Coordination Center of China"}],"abstract":"Compared with image few-shot learning, most of the existing few-shot video classification methods perform worse on feature matching, because they fail to sufficiently exploit the temporal information and relation. Specifically, frames are usually evenly sampled, which may miss important frames. On the other hand, the heuristic model simply encodes the equally treated frames in sequence, which results in the lack of both long-term and short-term temporal modeling and interaction. To alleviate these limitations, we take advantage of the compressed domain knowledge and propose a long-short term Cross-Transformer (LSTC) for few-shot video classification. For short terms, the motion vector (MV) contains temporal cues and reflects the importance of each frame. For long terms, a video can be natively divided into a sequence of GOPs (Group Of Picture). Using this compressed domain knowledge helps to obtain a more accurate spatial-temporal feature space. Consequently, we design the long-short term selection module, short-term module, and long-term module to comprise the LSTC. Long-short term selection is performed to select informative compressed domain data. Long/short-term modules are utilized to sufficiently exploit the temporal information so that the query and support can be well-matched by cross-attention. Experimental results show the superiority of our method on various datasets.","doi":"10.24963/ijcai.2022/174","url":"https://ijcai.org/proceedings/2022/174","pdf":"https://ijcai.org/proceedings/2022/0174.pdf","markdown":"https://ijcai.org/proceedings/2022/174.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/174","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":174,"pages":{"first":1247,"last":1253},"keywords":["Computer Vision: Recognition (object detection, categorization)","Machine Learning: Few-shot learning","Computer Vision: Video analysis and understanding"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986552","length_sec":82,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984695","length_sec":348,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bishay et al., 2019] Mina Bishay, Georgios Zoumpourlis, and Ioannis Patras. TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognition. In BMVC, 2019.","title":"TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognition","authors":"Bishay","year":"2019","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bottou, 2010] Léon Bottou. Large-Scale Machine Learning with Stochastic Gradient Descent. In COMPSTAT, 2010.","title":"Large-Scale Machine Learning with Stochastic Gradient Descent","authors":"Léon Bottou, Bottou","year":"2010","venue":"COMPSTAT","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cao et al., 2020] Kaidi Cao, Jingwei Ji, Zhangjie Cao, Chien-Yi Chang, and Juan Carlos Niebles. Few-Shot Video Classification via Temporal Alignment. In CVPR, 2020.","title":"Few-Shot Video Classification via Temporal Alignment","authors":"Kaidi Cao, Jingwei Ji, Zhangjie Cao, Chien-Yi Chang, Juan Carlos Niebles","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Carreira and Zisserman, 2017] João Carreira and Andrew Zisserman. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In CVPR, 2017.","title":"Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset","authors":"João Carreira, Andrew Zisserman","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Fei-Fei Li. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.","title":"ImageNet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Fei-Fei Li","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Doersch et al., 2020] Carl Doersch, Ankush Gupta, and An- drew Zisserman. CrossTransformers: Spatially-Aware Few-Shot Transfer. In NeurIPS, 2020.","title":"CrossTransformers: Spatially-Aware Few-Shot Transfer","authors":"Carl Doersch, Ankush Gupta, Andrew Zisserman","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dwivedi et al., 2019] Sai Kumar Dwivedi, Vikram Gupta, Rahul Mitra, Shuaib Ahmed, and Arjun Jain. ProtoGAN: Towards Few Shot Learning for Action Recognition. In ICCV Workshop, 2019.","title":"ProtoGAN: Towards Few Shot Learning for Action Recognition","authors":"Sai Kumar Dwivedi, Vikram Gupta, Rahul Mitra, Shuaib Ahmed, Arjun Jain","year":"2019","venue":"ICCV Workshop","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Feichtenhofer et al., 2019] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. SlowFast Networks for Video Recognition. In ICCV, 2019.","title":"SlowFast Networks for Video Recognition","authors":"Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, Kaiming He","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adapta- tion of Deep Networks. In ICML, 2017.","title":"Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks","authors":"Finn","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fu et al., 2019] Yuqian Fu, Chengrong Wang, Yanwei Fu, Yu-Xiong Wang, Cong Bai, Xiangyang Xue, and Yu-Gang Jiang. Embodied one-shot video recognition: Learning from actions of a virtual embodied agent. In ACM Multi- media, 2019.","title":"Embodied one-shot video recognition: Learning from actions of a virtual embodied agent","authors":"Yuqian Fu, Chengrong Wang, Yanwei Fu, Yu-Xiong Wang, Cong Bai, Xiangyang Xue, Yu-Gang Jiang","year":"2019","venue":"ACM Multimedia","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fu et al., 2020] Yuqian Fu, Li Zhang, Junke Wang, Yanwei Fu, and Yu-Gang Jiang. Depth guided adaptive meta- fusion network for few-shot video recognition. In ACM Multimedia, 2020.","title":"Depth guided adaptive metafusion network for few-shot video recognition","authors":"Yuqian Fu, Li Zhang, Junke Wang, Yanwei Fu, Yu-Gang Jiang","year":"2020","venue":"ACM Multimedia","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Goyal et al., 2017] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Chris- tian Thurau, Ingo Bax, and Roland Memisevic. The \"Something Something\" Video Database for Learning and Evaluating Visual Common Sense. In ICCV, 2017. [He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.","title":"The \"Something Something\" Video Database for Learning and Evaluating Visual Common Sense","authors":"Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, Roland Memisevic","year":"2016","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kuehne et al., 2011] Hilde Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. HMDB: A large video database for human motion recognition. In ICCV, 2011.","title":"HMDB: A large video database for human motion recognition","authors":"Hilde Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, Thomas Serre","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2020] Jiapeng Li, Ping Wei, Yongchi Zhang, and Nanning Zheng. A Slow-I-Fast-P Architecture for Com- pressed Video Action Recognition. In ACM Multimedia, 2020.","title":"A Slow-I-Fast-P Architecture for Compressed Video Action Recognition","authors":"Li","year":"2020","venue":"ACM Multimedia","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Perrett et al., 2021] Toby Perrett, Alessandro Masullo, Tilo Burghardt, Majid Mirmehdi, and Dima Damen. Temporal-Relational CrossTransformers for Few-Shot Ac- tion Recognition. In CVPR, 2021.","title":"Temporal-Relational CrossTransformers for Few-Shot Action Recognition","authors":"Toby Perrett, Alessandro Masullo, Tilo Burghardt, Majid Mirmehdi, Dima Damen","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shou et al., 2019] Zheng Shou, Zhicheng Yan, Yannis Kalantidis, Laura Sevilla-Lara, Marcus Rohrbach, Xudong Lin, and Shih-Fu Chang. DMC-Net: Generating Discrim- inative Motion Cues for Fast Compressed Video Action Recognition. In CVPR, 2019.","title":"DMC-Net: Generating Discriminative Motion Cues for Fast Compressed Video Action Recognition","authors":"Zheng Shou, Zhicheng Yan, Yannis Kalantidis, Laura Sevilla-Lara, Marcus Rohrbach, Xudong Lin, Shih-Fu Chang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Soomro et al., 2012] Khurram Soomro, Amir Roshan Za- mir, and Mubarak Shah. UCF101: A Dataset of 101 Hu- man Actions Classes From Videos in The Wild. ArXiv, 2012.","title":"UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild","authors":"Khurram Soomro, Mubarak Amir Roshan Zamir, Shah","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sung et al., 2018] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H. S. Torr, and Timothy M. Hospedales. Learning to Compare: Relation Network for Few-Shot Learning. In CVPR, 2018.","title":"Learning to Compare: Relation Network for Few-Shot Learning","authors":"Sung","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In NeurIPS, 2017.","title":"Attention is All you Need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vinyals et al., 2016] Oriol Vinyals, Charles Blundell, Tim- othy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching Networks for One Shot Learning. In NeurIPS, 2016.","title":"Matching Networks for One Shot Learning","authors":"Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, Daan Wierstra","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2016] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal Segment Networks: Towards Good Prac- tices for Deep Action Recognition. In ECCV, 2016.","title":"Temporal Segment Networks: Towards Good Practices for Deep Action Recognition","authors":"Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, Luc Van Gool","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu et al., 2018] Chao-Yuan Wu, Manzil Zaheer, Hexiang Hu, R. Manmatha, Alex Smola, and Philipp Krähenbühl. Compressed Video Action Recognition. In CVPR, 2018.","title":"Compressed Video Action Recognition","authors":"Chao-Yuan, Manzil Wu, Hexiang Zaheer, R Hu, Alex Manmatha, Philipp Smola, Krähenbühl","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2020] Hongguang Zhang, Li Zhang, Xiaojuan Qi, Hongdong Li, Philip H. S. Torr, and Piotr Koniusz. Few-Shot Action Recognition with Permutation-Invariant Attention. In ECCV, 2020.","title":"Few-Shot Action Recognition with Permutation-Invariant Attention","authors":"Hongguang Zhang, Li Zhang, Xiaojuan Qi, Hongdong Li, H S Philip, Piotr Torr, Koniusz","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhu and Yang, 2018] Linchao Zhu and Yi Yang. Compound Memory Networks for Few-Shot Video Classification. In ECCV, 2018.","title":"Compound Memory Networks for Few-Shot Video Classification","authors":"Yang ; Zhu, Linchao Zhu, Yi Yang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhu et al., 2021] Zhenxi Zhu, Limin Wang, Sheng Guo, and Gangshan Wu. A Closer Look at Few-Shot Video Classifi- cation: A New Baseline and Benchmark. In BMVC, 2021.","title":"A Closer Look at Few-Shot Video Classification: A New Baseline and Benchmark","authors":"Zhenxi Zhu, Limin Wang, Sheng Guo, Gangshan Wu","year":"2021","venue":"BMVC","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}