{"type":"ScholarlyArticle","title":"Part Aware Contrastive Learning for Self-Supervised Action Recognition","authors":[{"name":"Yilei Hua","slug":"yilei-hua","given":"Yilei","family":"Hua","affiliation":"School of Information Science and Engineering , Wuhan University of Science and Technology"},{"name":"Wenhan Wu","slug":"wenhan-wu","given":"Wenhan","family":"Wu","affiliation":"University of North Carolina at Charlotte"},{"name":"Ce Zheng","slug":"ce-zheng","given":"Ce","family":"Zheng","affiliation":"Center for Research in Computer Vision , University of Central Florida"},{"name":"Aidong Lu","slug":"aidong-lu","given":"Aidong","family":"Lu","affiliation":"University of North Carolina at Charlotte"},{"name":"Mengyuan Liu","slug":"mengyuan-liu","given":"Mengyuan","family":"Liu","affiliation":"Peking University , Shenzhen Graduate School"},{"name":"Chen Chen","slug":"chen-chen","given":"Chen","family":"Chen","affiliation":"Center for Research in Computer Vision , University of Central Florida"},{"name":"Shiqian Wu","slug":"shiqian-wu","given":"Shiqian","family":"Wu","affiliation":"School of Information Science and Engineering , Wuhan University of Science and Technology"}],"abstract":"In recent years, remarkable results have been achieved in self-supervised action recognition using skeleton sequences with contrastive learning. It has been observed that the semantic distinction of human action features is often represented by local body parts, such as legs or hands, which are advantageous for skeleton-based action recognition. This paper proposes an attention-based contrastive learning framework for skeleton representation learning, called SkeAttnCLR, which integrates local similarity and global features for skeleton-based action representations. To achieve this, a multi-head attention mask module is employed to learn the soft attention mask features from the skeletons, suppressing non-salient local features while accentuating local salient features, thereby bringing similar local features closer in the feature space. Additionally, ample contrastive pairs are generated by expanding contrastive pairs based on salient and non-salient features with global features, which guide the network to learn the semantic representations of the entire skeleton. Therefore, with the attention mask mechanism, SkeAttnCLR learns local features under different data augmentation views. The experiment results demonstrate that the inclusion of local feature similarity significantly enhances skeleton-based action representation. Our proposed SkeAttnCLR outperforms state-of-the-art methods on NTURGB+D, NTU120-RGB+D, and PKU-MMD datasets. The code and settings are available at this repository: https://github.com/GitHubOfHyl97/SkeAttnCLR.","doi":"10.24963/ijcai.2023/95","url":"https://ijcai.org/proceedings/2023/95","pdf":"https://ijcai.org/proceedings/2023/0095.pdf","markdown":"https://ijcai.org/proceedings/2023/95.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/95","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":95,"pages":{"first":855,"last":863},"keywords":["Computer Vision: CV: Action and behavior recognition","Machine Learning: ML: Self-supervised Learning","Computer Vision: CV: 3D computer vision","Computer Vision: CV: Representation learning"],"videos":[],"references":[{"position":1,"raw":"[Bao et al., 2021] Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021.","title":"Beit: Bert pre-training of image transformers","authors":"Bao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao et al., 2017] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In CVPR, pages 7291-7299, 2017.","title":"Realtime multi-person 2d pose estimation using part affinity fields","authors":"Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Caron et al., 2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In CVPR, pages 9650-9660, 2021.","title":"Emerging properties in self-supervised vision transformers","authors":"Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen and He, 2021] Xinlei Chen and Kaiming He. Explor- ing simple siamese representation learning. In CVPR, pages 15750-15758, 2021.","title":"Exploring simple siamese representation learning","authors":"Chen, He ;, Xinlei Chen, Kaiming He","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2020a] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, pages 1597-1607. PMLR, 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Chen","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2020b] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020.","title":"Improved baselines with momentum contrastive learning","authors":"Chen","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2022] Zhan Chen, Hong Liu, Tianyu Guo, Zhengyan Chen, Pinhao Song, and Hao Tang. Contrastive learning from spatio-temporal mixed skeleton sequences for self-supervised skeleton-based action recognition. arXiv preprint arXiv:2207.03065, 2022.","title":"Contrastive learning from spatio-temporal mixed skeleton sequences for self-supervised skeleton-based action recognition","authors":"Zhan Chen, Hong Liu, Tianyu Guo, Zhengyan Chen, Pinhao Song, Hao Tang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Guo et al., 2022] Tianyu Guo, Hong Liu, Zhan Chen, Mengyuan Liu, Tao Wang, and Runwei Ding. Contrastive learning from extremely augmented skeleton sequences for self-supervised action recognition. In AAAI, volume 36, pages 762-770, 2022.","title":"Contrastive learning from extremely augmented skeleton sequences for self-supervised action recognition","authors":"Tianyu Guo, Hong Liu, Zhan Chen, Mengyuan Liu, Tao Wang, Runwei Ding","year":"2022","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsu- pervised visual representation learning. In CVPR, pages 9729-9738, 2020.","title":"Momentum contrast for unsupervised visual representation learning","authors":"Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 16000-16009, 2022.","title":"Masked autoencoders are scalable vision learners","authors":"Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hochreiter et al., 2001] Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, J urgen Schmidhuber, and Corso Elvezia. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. A Field Guide to Dynamical Re- current Neural Networks, pages 237-244, 2001.","title":"Gradient flow in recurrent nets: the difficulty of learning long-term dependencies","authors":"Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, J Urgen Schmidhuber, Corso Elvezia","year":"2001","venue":"A Field Guide to Dynamical Recurrent Neural Networks","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huang et al., 2022] Lang Huang, Shan You, Mingkai Zheng, Fei Wang, Chen Qian, and Toshihiko Yamasaki. Learning where to learn in cross-view self-supervised learning. In CVPR, pages 14451-14460, 2022.","title":"Learning where to learn in cross-view self-supervised learning","authors":"Lang Huang, Shan You, Mingkai Zheng, Fei Wang, Chen Qian, Toshihiko Yamasaki","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ke et al., 2017] Qiuhong Ke, Mohammed Bennamoun, Sen- jian An, Ferdous Sohel, and Farid Boussaid. A new repre- sentation of skeleton sequences for 3d action recognition. In CVPR, pages 3288-3297, 2017.","title":"A new representation of skeleton sequences for 3d action recognition","authors":"Ke","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kim et al., 2021] Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. Self-guided contrastive learning for bert sentence representations. In ACL (Volume 1: Long Papers), pages 2528-2540, 2021.","title":"Self-guided contrastive learning for bert sentence representations","authors":"Kim","year":"2021","venue":"Long Papers)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2017] Chao Li, Qiaoyong Zhong, Di Xie, and Shil- iang Pu. Skeleton-based action recognition with convolu- tional neural networks. In ICMEW, pages 597-600. IEEE, 2017.","title":"Skeleton-based action recognition with convolutional neural networks","authors":"Chao Li, Qiaoyong Zhong, Di Xie, Shiliang Pu","year":"2017","venue":"ICMEW","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2019] Maosen Li, Siheng Chen, Xu Chen, Ya Zhang, Yanfeng Wang, and Qi Tian. Actional-structural graph convolutional networks for skeleton-based action recognition. In CVPR, pages 3595-3603, 2019.","title":"Actional-structural graph convolutional networks for skeleton-based action recognition","authors":"Li","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2021a] Linguo Li, Minsi Wang, Bingbing Ni, Hang Wang, Jiancheng Yang, and Wenjun Zhang. 3d human action representation learning via cross-view consistency pursuit. In CVPR, pages 4741-4750, 2021.","title":"3d human action representation learning via cross-view consistency pursuit","authors":"Li","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2021b] Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive cluster- ing. In AAAI, volume 35, pages 8547-8555, 2021.","title":"Contrastive clustering","authors":"Li","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liang et al., 2019] Duohan Liang, Guoliang Fan, Guangfeng Lin, Wanjun Chen, Xiaorong Pan, and Hong Zhu. Three- stream convolutional neural network with multi-task and ensemble learning for 3d action recognition. In CVPRW, pages 0-0, 2019.","title":"Threestream convolutional neural network with multi-task and ensemble learning for 3d action recognition","authors":"Liang","year":"2019","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2019] Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yu Duan, and Alex C Kot. Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding. IEEE TPAMI, 42(10):2684-2701, 2019.","title":"Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding","authors":"Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yu Duan, Alex C Kot","year":"2019","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2020] Jiaying Liu, Sijie Song, Chunhui Liu, Yang- hao Li, and Yueyu Hu. A benchmark dataset and compari- son study for multi-modal human action analytics. ACM TOMM, 16(2):1-24, 2020.","title":"A benchmark dataset and comparison study for multi-modal human action analytics","authors":"Jiaying Liu, Sijie Song, Chunhui Liu, Yanghao Li, Yueyu Hu","year":"2020","venue":"ACM TOMM","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Oord et al., 2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.","title":"Representation learning with contrastive predictive coding","authors":"Aaron Van Den Oord, Yazhe Li, Oriol Vinyals","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Paszke et al., 2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. NIPS, 32, 2019.","title":"Pytorch: An imperative style, highperformance deep learning library","authors":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga","year":"2019","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pathak et al., 2016] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, pages 2536-2544, 2016.","title":"Context encoders: Feature learning by inpainting","authors":"Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, Alexei A Efros","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Plizzari et al., 2021] Chiara Plizzari, Marco Cannici, and Matteo Matteucci. Spatial temporal transformer network for skeleton-based action recognition. ICPR, 2021.","title":"Spatial temporal transformer network for skeleton-based action recognition","authors":"Chiara Plizzari, Marco Cannici, Matteo Matteucci","year":"2021","venue":"ICPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Richemond et al., 2020] Pierre H Richemond, Jean-Bastien Grill, Florent Altché, Corentin Tallec, Florian Strub, An- drew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, et al. Byol works even without batch statistics. arXiv preprint arXiv:2010.10241, 2020.","title":"Byol works even without batch statistics","authors":"Jean-Bastien Pierre H Richemond, Florent Grill, Corentin Altché, Florian Tallec, Andrew Strub, Samuel Brock, Soham Smith, Razvan De, Bilal Pascanu, Piot","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ruder, 2016] Sebastian Ruder. An overview of gradi- ent descent optimization algorithms. arXiv preprint arXiv:1609.04747, 2016.","title":"An overview of gradient descent optimization algorithms","authors":"Sebastian Ruder","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Shahroudy et al., 2016] Amir Shahroudy, Jun Liu, Tian- Tsong Ng, and Gang Wang. Ntu rgb+ d: A large scale dataset for 3d human activity analysis. In CVPR, pages 1010-1019, 2016.","title":"Ntu rgb+ d: A large scale dataset for 3d human activity analysis","authors":"Amir Shahroudy, Jun Liu, Tian-Tsong Ng, Gang Wang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Shi et al., 2017] Yemin Shi, Yonghong Tian, Yaowei Wang, Wei Zeng, and Tiejun Huang. Learning long-term depen- dencies for action recognition with a biologically-inspired deep network. In ICCV, pages 716-725, 2017.","title":"Learning long-term dependencies for action recognition with a biologically-inspired deep network","authors":"Yemin Shi, Yonghong Tian, Yaowei Wang, Wei Zeng, Tiejun Huang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Shi et al., 2020] Lei Shi, Yifan Zhang, Jian Cheng, and Han- qing Lu. Decoupled spatial-temporal attention network for skeleton-based action-gesture recognition. In ACCV, 2020.","title":"Decoupled spatial-temporal attention network for skeleton-based action-gesture recognition","authors":"Lei Shi, Yifan Zhang, Jian Cheng, Hanqing Lu","year":"2020","venue":"ACCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Su et al., 2020] Kun Su, Xiulong Liu, and Eli Shlizerman. Predict & cluster: Unsupervised skeleton based action recognition. In CVPR, pages 9631-9640, 2020.","title":"Predict & cluster: Unsupervised skeleton based action recognition","authors":"Kun Su, Xiulong Liu, Eli Shlizerman","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Sun et al., 2019] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In CVPR, pages 5693-5703, 2019.","title":"Deep high-resolution representation learning for human pose estimation","authors":"Ke Sun, Bin Xiao, Dong Liu, Jingdong Wang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Van der Maaten and Hinton, 2008] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.","title":"Visualizing data using t-sne","authors":"Van Der Maaten, Laurens Hinton, Geoffrey Van Der Maaten, Hinton","year":"2008","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NIPS, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wei et al., 2019] Chen Wei, Lingxi Xie, Xutong Ren, Yingda Xia, Chi Su, Jiaying Liu, Qi Tian, and Alan L Yuille. Iterative reorganization with weak spatial constraints: Solv- ing arbitrary jigsaw puzzles for unsupervised representation learning. In CVPR, pages 1910-1919, 2019.","title":"Iterative reorganization with weak spatial constraints: Solving arbitrary jigsaw puzzles for unsupervised representation learning","authors":"Chen Wei, Lingxi Xie, Xutong Ren, Yingda Xia, Chi Su, Jiaying Liu, Qi Tian, Alan L Yuille","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Yan et al., 2018] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton- based action recognition. In AAAI, 2018.","title":"Spatial temporal graph convolutional networks for skeletonbased action recognition","authors":"Sijie Yan, Yuanjun Xiong, Dahua Lin","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhang et al., 2020] Pengfei Zhang, Cuiling Lan, Wenjun Zeng, Junliang Xing, Jianru Xue, and Nanning Zheng. Semantics-guided neural networks for efficient skeleton- based human action recognition. In CVPR, pages 1112- 1121, 2020.","title":"Semantics-guided neural networks for efficient skeletonbased human action recognition","authors":"Pengfei Zhang, Cuiling Lan, Wenjun Zeng, Junliang Xing, Jianru Xue, Nanning Zheng","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhang et al., 2022] Jiahang Zhang, Lilang Lin, and Jiay- ing Liu. Hierarchical consistent contrastive learning for skeleton-based action recognition with growing augmenta- tions. arXiv preprint arXiv:2211.13466, 2022.","title":"Hierarchical consistent contrastive learning for skeleton-based action recognition with growing augmentations","authors":"Zhang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zheng et al., 2022] Ce Zheng, Matias Mendieta, Pu Wang, Aidong Lu, and Chen Chen. A lightweight graph trans- former network for human mesh reconstruction from 2d human pose. In Proceedings of the 30th ACM International Conference on Multimedia, pages 5496-5507, 2022.","title":"A lightweight graph transformer network for human mesh reconstruction from 2d human pose","authors":"Ce Zheng, Matias Mendieta, Pu Wang, Aidong Lu, Chen Chen","year":"2022","venue":"Proceedings of the 30th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zheng et al., 2023] Ce Zheng, Matias Mendieta, Taojiannan Yang, Guo-Jun Qi, and Chen Chen. Feater: An efficient network for human reconstruction feature map-based trans- former. In CVPR, 2023.","title":"Feater: An efficient network for human reconstruction feature map-based transformer","authors":"Ce Zheng, Matias Mendieta, Taojiannan Yang, Guo-Jun Qi, Chen Chen","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}