{"type":"ScholarlyArticle","title":"Semi-Dynamic Hypergraph Neural Network for 3D Pose Estimation","authors":[{"name":"Shengyuan Liu","slug":"shengyuan-liu","given":"Shengyuan","family":"Liu","affiliation":"Center for Interdisciplinary Information Science Research , Zhengzhou University"},{"name":"Pei Lv","slug":"pei-lv","given":"Pei","family":"Lv","affiliation":"Center for Interdisciplinary Information Science Research , Zhengzhou University"},{"name":"Yuzhen Zhang","slug":"yuzhen-zhang","given":"Yuzhen","family":"Zhang","affiliation":"Center for Interdisciplinary Information Science Research , Zhengzhou University"},{"name":"Jie Fu","slug":"jie-fu","given":"Jie","family":"Fu","affiliation":"Center for Interdisciplinary Information Science Research , Zhengzhou University"},{"name":"Junjin Cheng","slug":"junjin-cheng","given":"Junjin","family":"Cheng","affiliation":"Center for Interdisciplinary Information Science Research , Zhengzhou University"},{"name":"Wanqing Li","slug":"wanqing-li","given":"Wanqing","family":"Li","affiliation":"Advanced Multimedia Research Lab , University of Wollongong clown"},{"name":"Bing Zhou","slug":"bing-zhou","given":"Bing","family":"Zhou","affiliation":"Center for Interdisciplinary Information Science Research , Zhengzhou University"},{"name":"Mingliang Xu","slug":"mingliang-xu","given":"Mingliang","family":"Xu","affiliation":"Center for Interdisciplinary Information Science Research , Zhengzhou University"}],"abstract":"This paper proposes a novel Semi-Dynamic Hypergraph Neural Network (SD-HNN) to estimate 3D human pose from a single image. SD-HNN adopts hypergraph to represent the human body to effectively exploit the kinematic constrains among adjacent and non-adjacent joints. Specifically, a pose hypergraph in SD-HNN has two components. One is a static hypergraph constructed according to the conventional tree body structure. The other is the semi-dynamic hypergraph representing the dynamic kinematic constrains among different joints. These two hypergraphs are combined together to be trained in an end-to-end fashion. Unlike traditional Graph Convolutional Networks (GCNs) that are based on a fixed tree structure, the SD-HNN can deal with ambiguity in human pose estimation. Experimental results demonstrate that the proposed method achieves state-of-the-art performance both on the Human3.6M and MPI-INF-3DHP datasets.","doi":"10.24963/ijcai.2020/109","url":"https://ijcai.org/proceedings/2020/109","pdf":"https://ijcai.org/proceedings/2020/0109.pdf","markdown":"https://ijcai.org/proceedings/2020/109.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/109","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":109,"pages":{"first":782,"last":788},"keywords":["Computer Vision: Biometrics, Face and Gesture Recognition","Computer Vision: 2D and 3D Computer Vision","Machine Learning: Deep Learning: Convolutional networks"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25455/file_25455.m3u8","length_sec":289,"provider":"hls"}],"references":[{"position":1,"raw":"Method Direct Discuss Eating Greet Phone Photo Pose Purch Sitting SittingD Smoke Wait WalkD Walk WalkT Avg Martinez [Martinez et al., 2017] 51.8 56.2 58.1 59.0 69.5 78.4 55.2 58.1 74.0 94.6 62.3 59.1 65.1 49.5 52.4 62.9 Pavlakos [Pavlakos et al., 2018] 67.4 71.9 66.7 69.1 72.0 77.0 65.0 68.3 83.7 96.5 71.7 65.8 74.9 59.1 63.2 71.9 Fang [Fang et al., 2018] 50.1 54.3 57.0 57.1 66.6 73.3 53.4 55.7 72.8 88.6 60.3 57.7 62.7 47.5 50.6 60.4 Yang [Yang et al., 2018] 51.5 58.9 50.4 57.0 62.1 65.4 49.8 52.7 69.2 85.2 57.4 58.4 43.6 60.1 47.7 58.6 Lee [Lee et al., 2018] 43.8 51.7 48.8 53.1 52.2 74.9 52.7 44.6 56.9 74.3 56.7 66.4 47.5 68.4 45.6 55.8 Trumble [Trumble et al., 2018] 41.7 43.2 52.9 70.0 64.9 83.0 57.3 63.5 61.0 95.0 70.0 62.3 66.2 53.7 52.4 62.5 Chen [Chen et al., 2019] 41.1 44.2 44.9 45.9 46.5 39.3 41.6 54.8 73.2 46.2 48.7 42.1 35.8 46.6 38.5 46.3 Pavllo* [Pavllo et al., 2019] 47.1 50.6 49.0 51.8 53.6 61.4 49.4 47.4 59.3 67.4 52.4 49.5 55.3 39.5 42.7 51.8 Wandt [Wandt and Rosenhahn, 2019] 53.0 58.3 59.6 66.5 72.8 71.0 56.7 69.6 78.3 95.2 66.6 58.5 63.2 57.5 49.9 65.1 Habibie [Habibie et al., 2019] 54.0 65.1 58.5 62.9 67.9 54.0 75.0 60.6 82.7 98.2 63.3 61.2 66.9 50.0 56.5 65.7 Zhao [Zhao et al., 2019] 48.2 60.8 51.8 64.0 64.6 53.6 51.1 67.4 88.7 57.7 73.2 65.6 48.9 64.8 51.9 60.8 Cai* [Cai et al., 2019] 46.5 48.8 47.6 50.9 52.9 61.3 48.3 45.8 59.2 64.4 51.2 48.4 53.5 39.2 41.2 50.6 Ci [Ci et al., 2019] 46.8 52.3 44.7 50.4 52.9 68.9 49.6 46.4 60.2 78.9 51.2 50.0 54.8 40.4 43.3 52.7 SD-HNN 46.5 55.0 54.7 60.2 63.3 48.9 50.2 64.2 54.0 76.0 63.2 55.5 47.6 59.3 44.0 56.2 Martinez [Martinez et al., 2017] (GT) 37.7 44.4 40.3 42.1 48.2 54.9 44.4 42.1 54.6 58.0 45.1 46.4 47.6 36.4 40.4 45.5 Zhao [Zhao et al., 2019](GT) 37.8 49.4 37.6 40.9 45.1 41.4 40.1 48.3 50.1 42.2 53.5 44.3 40.5 47.3 39.0 43.8 SD-HNN (GT) 42.1 45.6 38.2 41.4 41.5 47.4 45.8 39.9 44.7 53.0 42.6 44.0 42.1 34.0 37.6 42.7 Table 1: Quantitative comparisons of Mean Per Joint Position Error (MPJPE) in millimeter between the estimated 3D pose and the ground truth on Human3.6M under Protocol 1. SD-HNN shows the results taking as the input 2D pose estimated by the Stacked Hourglass network. GT means the 2d poses are from the ground truth. The results of Pavllo* and Cai* are based on single frame. Method Direct Discuss Eating Greet Phone Photo Pose Purch Sitting SittingD Smoke Wait WalkD Walk WalkT Avg Martinez [Martinez et al., 2017] 39.5 43.2 46.4 47.0 51.0 56.0 41.4 40.6 56.5 69.4 49.2 45.0 49.5 38.0 43.1 47.7 Lee [Lee et al., 2018] 38.0 39.3 46.3 44.4 49.0 55.1 40.2 41.1 53.2 68.9 51.0 39.1 33.9 56.4 38.5 46.2 Fang [Fang et al., 2018] 38.2 41.7 43.7 44.9 48.5 55.3 40.2 38.2 54.5 64.4 47.2 44.3 47.3 36.7 41.7 45.7 Rayat [Rayat Imtiaz Hossain and Little, 2018] (GT) 35.2 40.8 37.2 37.4 43.2 44.0 38.9 35.6 42.3 44.6 39.7 39.7 40.2 32.8 35.5 39.2 Chen [Chen et al., 2019] 36.5 41.0 40.9 43.9 45.6 53.8 38.5 37.3 53.0 65.2 44.6 40.9 44.3 32.0 38.4 44.1 Wandt [Wandt and Rosenhahn, 2019] (GT) 33.6 38.8 32.6 37.5 36.0 44.1 37.8 34.9 39.2 52.0 37.5 39.8 34.1 40.3 34.9 38.2 SD-HNN 43.7 46.4 39.3 42.7 42.6 48.6 46.6 41.7 45.4 52.5 43.3 45.0 42.4 34.9 38.3 43.5 SD-HNN (GT) 29.6 34.9 31.7 31.6 32.9 37.4 33.3 30.5 37.6 43.0 34.2 34.3 33.2 27.0 29.2 33.4","title":"Method Direct Discuss Eating Greet Phone Photo Pose Purch Sitting SittingD Smoke Wait WalkD Walk WalkT Avg Martinez","authors":"Martinez","year":"2017","venue":"Wandt and Rosenhahn","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alex et al., 2012] Krizhevsky Alex, Ilya Sutskever, and Ge- offrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurIPS, pages 1097-1105, 2012.","title":"Krizhevsky Alex, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks","authors":"Alex","year":"2012","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Atwood and Towsley, 2016] James Atwood and Don Towsley. Diffusion-convolutional neural networks. In NeurIPS, pages 1993-2001, 2016.","title":"Diffusion-convolutional neural networks","authors":"Towsley Atwood, James Atwood, Don Towsley","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cai et al., 2019] Yujun Cai, Liuhao Ge, Jun Liu, Jianfei Cai, Tat-Jen Cham, Junsong Yuan, and Nadia Magnenat Thal- mann. Exploiting spatial-temporal relationships for 3d pose estimation via graph convolutional networks. In IC- CV, pages 2272-2281, 2019.","title":"Exploiting spatial-temporal relationships for 3d pose estimation via graph convolutional networks","authors":"Yujun Cai, Liuhao Ge, Jun Liu, Jianfei Cai, Tat-Jen Cham, Junsong Yuan, Nadia Magnenat Thalmann","year":"2019","venue":"IC-CV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cao et al., 2017] Zhe Cao, Tomas Simon, Shih En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In CVPR, pages 1302-1310, 2017.","title":"Realtime multi-person 2d pose estimation using part affinity fields","authors":"Zhe Cao, Tomas Simon, Shih En Wei, Yaser Sheikh","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen and Ramanan, 2017] Ching-Hang Chen and Deva Ra- manan. 3d human pose estimation= 2d pose estimation+ matching. In CVPR, pages 5759-5767. IEEE, 2017.","title":"Ching-Hang Chen and Deva Ramanan. 3d human pose estimation= 2d pose estimation+ matching","authors":"Ramanan Chen","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2019] Xipeng Chen, Kwan-Yee Lin, Wentao Liu, Chen Qian, and Liang Lin. Weakly-supervised dis- covery of geometry-aware representation for 3d human pose estimation. In CVPR, pages 10895-10904, 2019.","title":"Weakly-supervised discovery of geometry-aware representation for 3d human pose estimation","authors":"Chen","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chu et al., 2017] Xiao Chu, Wei Yang, Wanli Ouyang, Cheng Ma, Alan L Yuille, and Xiaogang Wang. Multi- context attention for human pose estimation. In CVPR, pages 1831-1840, 2017.","title":"Multicontext attention for human pose estimation","authors":"Xiao Chu, Wei Yang, Wanli Ouyang, Cheng Ma, Alan L Yuille, Xiaogang Wang","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ci et al., 2019] Hai Ci, Chunyu Wang, Xiaoxuan Ma, and Yizhou Wang. Optimizing network structure for 3d human pose estimation. In CVPR, pages 2262-2271, 2019.","title":"Optimizing network structure for 3d human pose estimation","authors":"Hai Ci, Chunyu Wang, Xiaoxuan Ma, Yizhou Wang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fang et al., 2018] Haoshu Fang, Yuanlu Xu, Wenguan Wang, Xiaobai Liu, and Song-Chun Zhu. Learning knowledge-guided pose grammar machine for 3d human pose estimation. In AAAI, 2018.","title":"Learning knowledge-guided pose grammar machine for 3d human pose estimation","authors":"Fang","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Feng et al., 2019] Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph neural net- works. In AAAI, volume 33, pages 3558-3565, 2019.","title":"Hypergraph neural networks","authors":"Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, Yue Gao","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Habibie et al., 2019] Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Gerard Pons-Moll, and Christian Theobalt. In the wild human pose estimation using explicit 2d features and intermediate 3d representations. In CVPR, June 2019.","title":"In the wild human pose estimation using explicit 2d features and intermediate 3d representations","authors":"Habibie","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Insafutdinov et al., 2016] Eldar Insafutdinov, Leonid Pishchulin, Bjoern Andres, Mykhaylo Andriluka, and Bernt Schiele. Deepercut: A deeper, stronger, and faster multi-person pose estimation model. In ECCV, pages 34-50. Springer, 2016.","title":"Deepercut: A deeper, stronger, and faster multi-person pose estimation model","authors":"Insafutdinov","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kipf and Welling, 2016] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.","title":"Semi-supervised classification with graph convolutional networks","authors":"N Thomas, Max Kipf, Welling","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lee et al., 2018] Kyoungoh Lee, Inwoong Lee, and Sanghoon Lee. Propagating lstm: 3d pose estimation based on joint interdependency. In ECCV, pages 119-135, 2018.","title":"Propagating lstm: 3d pose estimation based on joint interdependency","authors":"Lee","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Martinez et al., 2017] Julieta Martinez, Rayat Hossain, Javier Romero, and James J Little. A simple yet effective baseline for 3d human pose estimation. In ICCV, pages 2640-2649, 2017.","title":"A simple yet effective baseline for 3d human pose estimation","authors":"Julieta Martinez, Rayat Hossain, Javier Romero, James J Little","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Michel et al., 2015] Frank Michel, Alexander Krull, Eric Brachmann, Michael Ying Yang, Stefan Gumhold, and Carsten Rother. Pose estimation of kinematic chain in- stances via object coordinate regression. In BMVC, 2015.","title":"Pose estimation of kinematic chain instances via object coordinate regression","authors":"Frank Michel, Alexander Krull, Eric Brachmann, Ying Michael, Stefan Yang, Carsten Gumhold, Rother","year":"2015","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Moreno-Noguer, 2017] Francesc Moreno-Noguer. 3d hu- man pose estimation from a single image via distance ma- trix regression. In CVPR, pages 1561-1570, 2017.","title":"Francesc Moreno-Noguer. 3d human pose estimation from a single image via distance matrix regression","authors":"Moreno-Noguer","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Newell et al., 2016] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose esti- mation. In ECCV, pages 483-499. Springer, 2016.","title":"Stacked hourglass networks for human pose estimation","authors":"Alejandro Newell, Kaiyu Yang, Jia Deng","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pavlakos et al., 2017] Georgios Pavlakos, Xiaowei Zhou, Konstantinos G. Derpanis, and Kostas Daniilidis. Coarse- to-fine volumetric prediction for single-image 3d human pose. In CVPR, pages 1263-1272, 2017.","title":"Coarseto-fine volumetric prediction for single-image 3d human pose","authors":"Georgios Pavlakos, Xiaowei Zhou, Konstantinos G Derpanis, Kostas Daniilidis","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pavlakos et al., 2018] Georgios Pavlakos, Xiaowei Zhou, and Kostas Daniilidis. Ordinal depth supervision for 3d human pose estimation. In CVPR, pages 7307-7316, 2018.","title":"Ordinal depth supervision for 3d human pose estimation","authors":"Georgios Pavlakos, Xiaowei Zhou, Kostas Daniilidis","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pavllo et al., 2019] Dario Pavllo, Christoph Feichtenhofer, David Grangier, and Michael Auli. 3d human pose es- timation in video with temporal convolutions and semi- supervised training. In CVPR, June 2019.","title":"3d human pose estimation in video with temporal convolutions and semisupervised training","authors":"Dario Pavllo, Christoph Feichtenhofer, David Grangier, Michael Auli","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rayat Imtiaz Hossain and Little, 2018] Mir Rayat Imti- az Hossain and James J Little. Exploiting temporal information for 3d human pose estimation. In ECCV, pages 68-84, 2018.","title":"Exploiting temporal information for 3d human pose estimation","authors":"Rayat Imtiaz, Hossain, Little ; Mir Rayat, Imtiaz Hossain, James J Little","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sun et al., 2018] Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. Integral human pose regression. In ECCV, September 2018.","title":"Integral human pose regression","authors":"Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, Yichen Wei","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Trumble et al., 2018] Matthew Trumble, Andrew Gilbert, Adrian Hilton, and John Collomosse. Deep autoencoder for combined human pose estimation and body model up- scaling. In ECCV, pages 784-800, 2018.","title":"Deep autoencoder for combined human pose estimation and body model upscaling","authors":"Matthew Trumble, Andrew Gilbert, Adrian Hilton, John Collomosse","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wandt and Rosenhahn, 2019] Bastian Wandt and Bodo Rosenhahn. Repnet: Weakly supervised training of an ad- versarial reprojection network for 3d human pose estima- tion. In CVPR, pages 7782-7791, 2019.","title":"Repnet: Weakly supervised training of an adversarial reprojection network for 3d human pose estimation","authors":"Rosenhahn Wandt, Bastian Wandt, Bodo Rosenhahn","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yan et al., 2018] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. In AAAI, 2018.","title":"Spatial temporal graph convolutional networks for skeleton-based action recognition","authors":"Sijie Yan, Yuanjun Xiong, Dahua Lin","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yang et al., 2018] Wei Yang, Wanli Ouyang, Xiaolong Wang, Jimmy Ren, Hongsheng Li, and Xiaogang Wang. 3d human pose estimation in the wild by adversarial learn- ing. In CVPR, pages 5255-5264, 2018.","title":"3d human pose estimation in the wild by adversarial learning","authors":"Wei Yang, Wanli Ouyang, Xiaolong Wang, Jimmy Ren, Hongsheng Li, Xiaogang Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhao et al., 2019] Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris N Metaxas. Semantic graph convo- lutional networks for 3d human pose regression. In CVPR, pages 3425-3435, 2019.","title":"Semantic graph convolutional networks for 3d human pose regression","authors":"Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, Dimitris N Metaxas","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhou et al., 2017] Xingyi Zhou, Qixing Huang, Xiao Sun, Xiangyang Xue, and Yichen Wei. Towards 3d human pose estimation in the wild: a weakly-supervised approach. In CVPR, pages 398-407, 2017.","title":"Towards 3d human pose estimation in the wild: a weakly-supervised approach","authors":"Zhou","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}