{"type":"ScholarlyArticle","title":"Learning Graph-based Residual Aggregation Network for Group Activity Recognition","authors":[{"name":"Wei Li","slug":"wei-li","given":"Wei","family":"Li","affiliation":"School of Computing and Artificial Intelligence , Southwest Jiaotong University"},{"name":"Tianzhao Yang","slug":"tianzhao-yang","given":"Tianzhao","family":"Yang","affiliation":"School of Computing and Artificial Intelligence , Southwest Jiaotong University"},{"name":"Xiao Wu","slug":"xiao-wu","given":"Xiao","family":"Wu","affiliation":"School of Computing and Artificial Intelligence , Southwest Jiaotong University"},{"name":"Zhaoquan Yuan","slug":"zhaoquan-yuan","given":"Zhaoquan","family":"Yuan","affiliation":"School of Computing and Artificial Intelligence , Southwest Jiaotong University"}],"abstract":"Group activity recognition aims to understand the overall behavior performed by a group of people. Recently, some graph-based methods have made progress by learning the relation graphs among multiple persons. However, the differences between an individual and others play an important role in identifying confusable group activities, which have not been elaborately explored by previous methods. In this paper, a novel Graph-based Residual AggregatIon Network (GRAIN) is proposed to model the differences among all persons of the whole group, which is end-to-end trainable. Specifically, a new local residual relation module is explicitly proposed to capture the local spatiotemporal differences of relevant persons, which is further combined with the multi-graph relation networks. Moreover, a weighted aggregation strategy is devised to adaptively select multi-level spatiotemporal features from the appearance-level information to high level relations. Finally, our model is capable of extracting a comprehensive representation and inferring the group activity in an end-to-end manner. The experimental results on two popular benchmarks for group activity recognition clearly demonstrate the superior performance of our method in comparison with the state-of-the-art methods.","doi":"10.24963/ijcai.2022/154","url":"https://ijcai.org/proceedings/2022/154","pdf":"https://ijcai.org/proceedings/2022/0154.pdf","markdown":"https://ijcai.org/proceedings/2022/154.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/154","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":154,"pages":{"first":1102,"last":1108},"keywords":["Computer Vision: Video analysis and understanding","Computer Vision: Action and Behaviour Recognition"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986539","length_sec":85,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984682","length_sec":354,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Amer and Todorovic, 2015] Mohamed R Amer and Sinisa Todorovic. Sum product networks for activity recognition. IEEE TPAMI, 38(4):800-813, 2015.","title":"Sum product networks for activity recognition","authors":"R Mohamed, Sinisa Amer, Todorovic","year":"2015","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arandjelovič et al., 2018] Relja Arandjelovič, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic. Netvlad: Cnn architecture for weakly supervised place recognition. IEEE TPAMI, 40(6):1437-1451, 2018.","title":"Netvlad: Cnn architecture for weakly supervised place recognition","authors":"Relja Arandjelovič, Petr Gronat, Akihiko Torii, Tomas Pajdla, Josef Sivic","year":"2018","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bagautdinov et al., 2017] Timur Bagautdinov, Alexandre Alahi, Franc ¸ois Fleuret, Pascal Fua, and Silvio Savarese. Social scene understanding: End-to-end multi-person ac- tion localization and collective activity recognition. In CVPR, pages 4315-4324, 2017.","title":"Social scene understanding: End-to-end multi-person action localization and collective activity recognition","authors":"Timur Bagautdinov, Alexandre Alahi, Pascal Franc ¸ois Fleuret, Silvio Fua, Savarese","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Choi et al., 2009] Wongun Choi, Khuram Shahid, and Sil- vio Savarese. What are they doing?: Collective activ- ity classification using spatio-temporal relationship among people. In ICCVW, pages 1282-1289, 2009.","title":"What are they doing?: Collective activity classification using spatio-temporal relationship among people","authors":"Choi","year":"2009","venue":"ICCVW","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dang et al., 2021] Long Hoang Dang, Thao Minh Le, Vuong Le, and Truyen Tran. Hierarchical object-oriented spatio-temporal reasoning for video question answering. In IJCAI, pages 636-642, 2021.","title":"Hierarchical object-oriented spatio-temporal reasoning for video question answering","authors":"Long Hoang Dang, Thao, Minh Le, Vuong Le, Truyen Tran","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/88"},{"position":6,"raw":"[Gavrilyuk et al., 2020] Kirill Gavrilyuk, Ryan Sanford, Mehrsan Javan, and Cees GM Snoek. Actor-transformers for group activity recognition. In CVPR, pages 839-848, 2020.","title":"Actor-transformers for group activity recognition","authors":"Kirill Gavrilyuk, Ryan Sanford, Mehrsan Javan, G M Cees, Snoek","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2020] Guyue Hu, Bo Cui, Yuan He, and Shan Yu. Progressive relation learning for group activity recogni- tion. In CVPR, pages 980-989, 2020.","title":"Progressive relation learning for group activity recognition","authors":"Guyue Hu, Bo Cui, Yuan He, Shan Yu","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ibrahim et al., 2016] Mostafa S Ibrahim, Srikanth Muralid- haran, Zhiwei Deng, Arash Vahdat, and Greg Mori. A hierarchical deep temporal model for group activity recog- nition. In CVPR, pages 1971-1980, 2016.","title":"A hierarchical deep temporal model for group activity recognition","authors":"Srikanth Mostafa S Ibrahim, Zhiwei Muralidharan, Arash Deng, Greg Vahdat, Mori","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2019] Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In CVPR, pages 510-519, 2019.","title":"Selective kernel networks","authors":"Xiang Li, Wenhai Wang, Xiaolin Hu, Jian Yang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2020] Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. Tea: Temporal excitation and aggregation for action recognition. In CVPR, pages 909- 918, 2020.","title":"Tea: Temporal excitation and aggregation for action recognition","authors":"Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, Limin Wang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2021a] Dong Li, Zhaofan Qiu, Yingwei Pan, Ting Yao, Houqiang Li, and Tao Mei. Representing videos as discriminative sub-graphs for action recognition. In CVPR, pages 3310-3319, 2021.","title":"Representing videos as discriminative sub-graphs for action recognition","authors":"Li","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2021b] Shuaicheng Li, Qianggang Cao, Lingbo Liu, Kunlin Yang, Shinan Liu, Jun Hou, and Shuai Yi. Groupformer: Group activity recognition with clustered spatial-temporal transformer. In ICCV, pages 13668- 13677, 2021.","title":"Groupformer: Group activity recognition with clustered spatial-temporal transformer","authors":"Li","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2021] Qianhui Liu, Dong Xing, Huajin Tang, De Ma, and Gang Pan. Event-based action recognition using motion information and spiking neural networks. In IJCAI, pages 1743-1749, 2021.","title":"Event-based action recognition using motion information and spiking neural networks","authors":"Liu","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/240"},{"position":15,"raw":"[Masato et al., 2011] Daniele Masato, Timothy J Norman, Wamberto W Vasconcelos, and Katia Sycara. Agent- oriented incremental team and activity recognition. In IJ- CAI, pages 1402-1407, 2011.","title":"Agentoriented incremental team and activity recognition","authors":"Daniele Masato, Timothy J Norman, Wamberto W Vasconcelos, Katia Sycara","year":"2011","venue":"IJ-CAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Qi et al., 2020] Mengshi Qi, Yunhong Wang, Jie Qin, An- nan Li, Jiebo Luo, and Luc Van Gool. stagnet: an atten- tive semantic rnn for group activity and individual action recognition. IEEE TCSVT, 30(2):549-565, 2020.","title":"stagnet: an attentive semantic rnn for group activity and individual action recognition","authors":"Qi","year":"2020","venue":"IEEE TCSVT","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shu et al., 2015] Tianmin Shu, Dan Xie, Brandon Rothrock, Sinisa Todorovic, and Chun Song Zhu. Joint inference of groups, events and human roles in aerial videos. In CVPR, pages 4576-4584, 2015.","title":"Joint inference of groups, events and human roles in aerial videos","authors":"Shu","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shu et al., 2020] Xiangbo Shu, Liyan Zhang, Yunlian Sun, and Jinhui Tang. Host-parasite: Graph lstm-in-lstm for group activity recognition. IEEE TNNLS, 32(2):663-674, 2020.","title":"Host-parasite: Graph lstm-in-lstm for group activity recognition","authors":"Shu","year":"2020","venue":"IEEE TNNLS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, pages 1-14, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tang et al., 2019] Jinhui Tang, Xiangbo Shu, Rui Yan, and Liyan Zhang. Coherence constrained graph lstm for group activity recognition. IEEE TPAMI, pages 1-12, 2019.","title":"Coherence constrained graph lstm for group activity recognition","authors":"Jinhui Tang, Xiangbo Shu, Rui Yan, Liyan Zhang","year":"2019","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang and Wang, 2017] Hongsong Wang and Liang Wang. Modeling temporal dynamics and spatial configurations of actions using two-stream recurrent neural networks. In CVPR, pages 499-508, 2017.","title":"Modeling temporal dynamics and spatial configurations of actions using two-stream recurrent neural networks","authors":"Wang, Wang ;, Hongsong Wang, Liang Wang","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2017] Minsi Wang, Bingbing Ni, and Xi- aokang Yang. Recurrent modeling of interaction context for collective activity recognition. In CVPR, pages 3048- 3056, 2017.","title":"Recurrent modeling of interaction context for collective activity recognition","authors":"Minsi Wang, Bingbing Ni, Xiaokang Yang","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2019] Jianchao Wu, Limin Wang, Li Wang, Jie Guo, and Gangshan Wu. Learning actor relation graphs for group activity recognition. In CVPR, pages 9964-9974, 2019.","title":"Learning actor relation graphs for group activity recognition","authors":"Jianchao Wu, Limin Wang, Li Wang, Jie Guo, Gangshan Wu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yan et al., 2020] Rui Yan, Lingxi Xie, Jinhui Tang, Xi- angbo Shu, and Qi Tian. Higcin: hierarchical graph- based cross inference network for group activity recogni- tion. IEEE TPAMI, pages 1-14, 2020.","title":"Higcin: hierarchical graphbased cross inference network for group activity recognition","authors":"Rui Yan, Lingxi Xie, Jinhui Tang, Xiangbo Shu, Qi Tian","year":"2020","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yan et al., 2021] Rui Yan, Xiangbo Shu, Chengcheng Yuan, Qi Tian, and Jinhui Tang. Position-aware participation- contributed temporal dynamic model for group activity recognition. IEEE TNNLS, pages 1-15, 2021.","title":"Position-aware participationcontributed temporal dynamic model for group activity recognition","authors":"Rui Yan, Xiangbo Shu, Chengcheng Yuan, Qi Tian, Jinhui Tang","year":"2021","venue":"IEEE TNNLS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yuan and Ni, 2021] Hangjie Yuan and Dong Ni. Learning visual context for group activity recognition. In AAAI, pages 3261-3269, 2021.","title":"Learning visual context for group activity recognition","authors":"Ni ; ] Hangjie Yuan, Dong Yuan, Ni","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yuan et al., 2021] Hangjie Yuan, Dong Ni, and Mang Wang. Spatio-temporal dynamic inference network for group activity recognition. In ICCV, pages 7476-7485, 2021.","title":"Spatio-temporal dynamic inference network for group activity recognition","authors":"Yuan","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou et al., 2021] Jiaming Zhou, Kun-Yu Lin, Haoxin Li, and Wei-Shi Zheng. Graph-based high-order relation mod- eling for long-term action recognition. In CVPR, pages 8984-8993, 2021.","title":"Graph-based high-order relation modeling for long-term action recognition","authors":"Jiaming Zhou, Kun-Yu Lin, Haoxin Li, Wei-Shi Zheng","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}