{"type":"ScholarlyArticle","title":"I²R-Net: Intra- and Inter-Human Relation Network for Multi-Person Pose Estimation","authors":[{"name":"Yiwei Ding","slug":"yiwei-ding","given":"Yiwei","family":"Ding","affiliation":"School of Informatics , Xiamen University"},{"name":"Wenjin Deng","slug":"wenjin-deng","given":"Wenjin","family":"Deng","affiliation":"School of Informatics , Xiamen University"},{"name":"Yinglin Zheng","slug":"yinglin-zheng","given":"Yinglin","family":"Zheng","affiliation":"School of Informatics , Xiamen University"},{"name":"Pengfei Liu","slug":"pengfei-liu","given":"Pengfei","family":"Liu","affiliation":"School of Informatics , Xiamen University"},{"name":"Meihong Wang","slug":"meihong-wang","given":"Meihong","family":"Wang","affiliation":"School of Informatics , Xiamen University"},{"name":"Xuan Cheng","slug":"xuan-cheng","given":"Xuan","family":"Cheng","affiliation":"School of Informatics , Xiamen University"},{"name":"Jianmin Bao","slug":"jianmin-bao","given":"Jianmin","family":"Bao","affiliation":"Microsoft Research Asia"},{"name":"Dong Chen","slug":"dong-chen","given":"Dong","family":"Chen","affiliation":"Microsoft Research Asia"},{"name":"Ming Zeng","slug":"ming-zeng","given":"Ming","family":"Zeng","affiliation":"School of Informatics , Xiamen University"}],"abstract":"In this paper, we present the Intra- and Inter-Human Relation Networks I²R-Net for Multi-Person Pose Estimation. It involves two basic modules. First, the Intra-Human Relation Module operates on a single person and aims to capture Intra-Human dependencies. Second, the Inter-Human Relation Module considers the relation between multiple instances and focuses on capturing Inter-Human interactions. The Inter-Human Relation Module can be designed very lightweight by reducing the resolution of feature map, yet learn useful relation information to significantly boost the performance of the Intra-Human Relation Module. Even without bells and whistles, our method can compete or outperform current competition winners. We conduct extensive experiments on COCO, CrowdPose, and OCHuman datasets. The results demonstrate that the proposed model surpasses all the state-of-the-art methods. Concretely, the proposed method achieves 77.4% AP on CrowPose dataset and 67.8% AP on OCHuman dataset respectively, outperforming existing methods by a large margin. Additionally, the ablation study and visualization analysis also prove the effectiveness of our model.","doi":"10.24963/ijcai.2022/120","url":"https://ijcai.org/proceedings/2022/120","pdf":"https://ijcai.org/proceedings/2022/0120.pdf","markdown":"https://ijcai.org/proceedings/2022/120.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/120","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":120,"pages":{"first":855,"last":862},"keywords":["Computer Vision: Action and Behaviour Recognition","Computer Vision: Biometrics, Face, Gesture and Pose Recognition","Computer Vision: Recognition (object detection, categorization)"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986465","length_sec":78,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984608","length_sec":328,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bello et al., 2019] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V Le. Attention aug- mented convolutional networks. In CVPR, 2019.","title":"Attention augmented convolutional networks","authors":"Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, Quoc V Le","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao et al., 2019] Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Openpose: realtime multi-person 2d pose estimation using part affinity fields. IEEE transactions on pattern analysis and machine intel- ligence, 2019.","title":"Openpose: realtime multi-person 2d pose estimation using part affinity fields","authors":"Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, Yaser Sheikh","year":"2019","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018] Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun. Cas- caded pyramid network for multi-person pose estimation. In CVPR, 2018.","title":"Cascaded pyramid network for multi-person pose estimation","authors":"Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, Jian Sun","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cheng et al., 2020] Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas S Huang, and Lei Zhang. Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation. In CVPR, 2020.","title":"Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation","authors":"Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas S Huang, Lei Zhang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fang et al., 2017] Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. Rmpe: Regional multi-person pose estima- tion. In ICCV, 2017.","title":"Rmpe: Regional multi-person pose estimation","authors":"Shuqin Hao-Shu Fang, Yu-Wing Xie, Cewu Tai, Lu","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Geng et al., 2021] Zigang Geng, Ke Sun, Bin Xiao, Zhaox- iang Zhang, and Jingdong Wang. Bottom-up human pose estimation via disentangled keypoint regression. In CVPR, 2021.","title":"Bottom-up human pose estimation via disentangled keypoint regression","authors":"Zigang Geng, Ke Sun, Bin Xiao, Zhaoxiang Zhang, Jingdong Wang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.","title":"Kaiming He, Georgia Gkioxari","authors":"He","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jin et al., 2020] Sheng Jin, Wentao Liu, Enze Xie, Wenhai Wang, Chen Qian, Wanli Ouyang, and Ping Luo. Differ- entiable hierarchical graph grouping for multi-person pose estimation. In ECCV, 2020.","title":"Differentiable hierarchical graph grouping for multi-person pose estimation","authors":"Sheng Jin, Wentao Liu, Enze Xie, Wenhai Wang, Chen Qian, Wanli Ouyang, Ping Luo","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2019] Jiefeng Li, Can Wang, Hao Zhu, Yihuan Mao, Hao-Shu Fang, and Cewu Lu. Crowdpose: Efficient crowded scenes pose estimation and a new benchmark. In CVPR, 2019.","title":"Crowdpose: Efficient crowded scenes pose estimation and a new benchmark","authors":"Li","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In ECCV, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mihai et al., 2021] Fieraru Mihai, Zanfir Mihai, Alexan- dru Szente Teodor, Gabriel Bazavan Eduard, Olaru Vlad, and Cristian Sminchisescu. Remips: Physically consis- tent 3d reconstruction of multiple interacting people under weak supervision. NeurIPS, 2021.","title":"Remips: Physically consistent 3d reconstruction of multiple interacting people under weak supervision","authors":"Mihai","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Newell et al., 2016a] Alejandro Newell, Zhiao Huang, and Jia Deng. Associative embedding: End-to-end learning for joint detection and grouping. NeurIPS, 2016.","title":"Associative embedding: End-to-end learning for joint detection and grouping","authors":"Newell","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Newell et al., 2016b] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose es- timation. In ECCV, 2016.","title":"Stacked hourglass networks for human pose estimation","authors":"Newell","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nie et al., 2019] Xuecheng Nie, Jiashi Feng, Jianfeng Zhang, and Shuicheng Yan. Single-stage multi-person pose machines. In CVPR, 2019.","title":"Single-stage multi-person pose machines","authors":"Nie","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Papandreou et al., 2018] George Papandreou, Tyler Zhu, Liang-Chieh Chen, Spyros Gidaris, Jonathan Tompson, and Kevin Murphy. Personlab: Person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model. In ECCV, 2018.","title":"Personlab: Person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model","authors":"George Papandreou, Tyler Zhu, Liang-Chieh Chen, Spyros Gidaris, Jonathan Tompson, Kevin Murphy","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Parmar et al., 2018] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In ICML, 2018.","title":"Image transformer","authors":"Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, Dustin Tran","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Qiu et al., 2020] Lingteng Qiu, Xuanye Zhang, Yanran Li, Guanbin Li, Xiaojun Wu, Zixiang Xiong, Xiaoguang Han, and Shuguang Cui. Peeking into occluded joints: A novel framework for crowd pose estimation. In ECCV, 2020.","title":"Peeking into occluded joints: A novel framework for crowd pose estimation","authors":"Qiu","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":"OpenAI blog","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sun et al., 2019] Ke Sun, Bin Xiao, Dong Liu, and Jing- dong Wang. Deep high-resolution representation learning for human pose estimation. In CVPR, 2019.","title":"Deep high-resolution representation learning for human pose estimation","authors":"Ke Sun, Bin Xiao, Dong Liu, Jingdong Wang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Touvron et al., 2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablay- rolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.","title":"Training data-efficient image transformers & distillation through attention","authors":"Touvron","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2021a] Dongkai Wang, Shiliang Zhang, and Gang Hua. Robust pose estimation in crowded scenes with direct pose-level inference. NeurIPS, 34, 2021.","title":"Robust pose estimation in crowded scenes with direct pose-level inference","authors":"Wang","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2021b] Jiashun Wang, Huazhe Xu, Medhini Narasimhan, and Xiaolong Wang. Multi-person 3d mo- tion prediction with multi-range transformers. NeurIPS, 2021.","title":"Multi-person 3d motion prediction with multi-range transformers","authors":"Wang","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xiao et al., 2018] Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In ECCV, 2018.","title":"Simple baselines for human pose estimation and tracking","authors":"Bin Xiao, Haiping Wu, Yichen Wei","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2021] Sen Yang, Zhibin Quan, Mu Nie, and Wankou Yang. Transpose: Keypoint localization via trans- former. In ICCV, 2021.","title":"Transpose: Keypoint localization via transformer","authors":"Sen Yang, Zhibin Quan, Mu Nie, Wankou Yang","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yuan et al., 2021] Yuhui Yuan, Rao Fu, Lang Huang, Wei- hong Lin, Chao Zhang, Xilin Chen, and Jingdong Wang. Hrformer: High-resolution transformer for dense predic- tion. NeurIPS, 2021.","title":"Hrformer: High-resolution transformer for dense prediction","authors":"Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, Jingdong Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2019] Song-Hai Zhang, Ruilong Li, Xin Dong, Paul Rosin, Zixi Cai, Xi Han, Dingcheng Yang, Haozhi Huang, and Shi-Min Hu. Pose2seg: Detection free human instance segmentation. In CVPR, 2019.","title":"Pose2seg: Detection free human instance segmentation","authors":"Zhang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2020] Feng Zhang, Xiatian Zhu, Hanbin Dai, Mao Ye, and Ce Zhu. Distribution-aware coordinate rep- resentation for human pose estimation. In CVPR, 2020.","title":"Distribution-aware coordinate representation for human pose estimation","authors":"Feng Zhang, Xiatian Zhu, Hanbin Dai, Mao Ye, Ce Zhu","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}