Semi-Dynamic Hypergraph Neural Network for 3D Pose Estimation

1 Introduction

Pose estimation aims to estimate 2D or 3D positions of human body joints from an image or video. It is an active research area in computer vision due to its wide range of potential applications. Like many other areas in computer vision, Convolutional Neural Networks (CNN) [Alex et al., 2012] have also been used for pose estimation. There are generally two kinds of CNN-based approaches. One is to directly estimate 3D human pose through a CNN [Sun et al., 2018] from an image. The other is to estimate a 2D pose through a CNN first, and then recover the 3D pose from the estimated 2D pose [Martinez et al., 2017; Moreno-Noguer, 2017; Pavlakos et al., 2017; Rayat Imtiaz Hossain and Little, 2018]. The former approach usually requires sufficient amount of annotated data and much computing resource for training where the latter may not utilize visual information in the second stage for 3D recovery. However, the human body is a typical chain-like structure constrained by human kinematics. Most of CNN-based pose estimation do not leverage this prior information.

∗Corresponding author

Graph Convolutional Network (GCNs) [Kipf andWelling, 2016] as a generalization of CNNs to deal with non-Euclidean data can learn feature representation of a graph through gradually aggregating features of multiple nodes to compute features for a node of interest. Some attempts of using GCN for 3D pose estimation are reported in [Yan et al., 2018; Zhao et al., 2019; Cai et al., 2019; Ci et al., 2019]. Although these methods have achieved promising results, they are limited by the fact that they all treat the human body as a tree structure and represent it as a simplified graph. According to human kinematics, the human body has a typical chainlike structure. The movement of body joints is not only constrained by directly neighboring joints, but also is subject to multiple non-neighboring joints. Such a complex relationship can hardly be captured by a simple graph with a set of fixed connections between joints.

To address this problem, we propose to represent the hu[Feng et al., 2019] aiming to learn local and global kinematic constrains among joints (shown in Figure 1). Unlike a simple graph, a hypergraph represents the human body and its kinematic constrains with flexible hyperedges. These hyperedges have no fixed degrees and are able to connect different body joints freely according to the dy-

namic interaction relationships among them. This characteristic conforms to the concept of the kinematic chain [Michel et al., 2015] and is able to model the kinematic constrains through adding or deleting hyperedge dynamically.

Upon the hypergraph representation of the human body, this paper develops a novel semi-dynamic hypergraph neural network (SD-HNN) for 3D human pose estimation. Specifically, based on the knowledge of human kinematic chain, we treat a series of closely related joints as a whole to construct a static hypergraph, and each kinematic chain will be regarded as a hyperedge in the hypergraph. In addition, we also construct a dynamic hypergraph of a human body, whose structure will be changed in the course of convolution operation along with the convolution kernel. The motivation is inspired that the kinematic constrains or the effects among different body joints may be different under different poses. A dynamic hypergraph is able to deal with the above situation intuitively. The static and dynamic hypergraphs are combined together and trained in an end-to-end fashion. We refer to such a scheme as being semi-dynamic.

Overall, the key contributions of this paper are:

• We propose to represent a human body as a hypergraph and, hence, to capture human kinematics using a combination of a static hypergraph and a dynamic hypergraph.
• Upon the representation, we develop a semi-dynamic hypergraph neural network (SD-HNN) for recovering 3D poses from 2D poses, which can be trained in an end-to-end way.
• The proposed representation and SD-HNN are extensively validated on Human 3.6m and MPI-INF-3DHP datasets. Their effectiveness has been demonstrated by the state-of-the-art performance.

2 RelatedWork

In the past few years, hand-crafted features, such as HOG, SIFT, DPM, are commonly used for 3D human pose regression. Recently, many state-of-the-art methods are based on deep convolution neural networks to estimate 3D human pose. As 2D human pose estimation [Cao et al., 2017; Chu et al., 2017; Insafutdinov et al., 2016] advances, a twostage framework becomes popular for 3D human pose estimation. In this framework, the 2D human pose is estimated or predicted in the first stage by a CNN, and then the 2D pose together with the intermediate layer features from the CNN are used to regress 3D pose [Chen and Ramanan, 2017; Martinez et al., 2017; Pavlakos et al., 2017; Fang et al., 2018]. The proposed SD-HNN follows the two-stage framework.

Graph Neural Networks (GNNs) have also been used in pose estimation. Construction of GCNs on a graph generally follows either spatial perspective or spectral perspective. Convolution operations directly on the graph are implemented in [Atwood and Towsley, 2016; Ci et al., 2019; Cai et al., 2019], while in [Kipf and Welling, 2016] convolution operations are in the spectral domain. Specifically, ST-GCN [Yan et al., 2018] and SEM-GCN [Zhao et al., 2019] both follow the spatial perspective. ST-GCN is probably the first representative work to adopt graph-based neural networks to model dynamic skeletons for action recognition. SEM-GCN employed GCNs to regress 3D pose from 2D by capturing both local and global relationships among joints. This paper extends GCN to a hypergraph neural network (HNN) in order to better capture human kinematics.

3 Semi-Dynamic Hypergraph Neural Network

In this section, we first briefly introduce GCN and the concept of HNN. With these preliminary knowledge, details of the proposed Semi-Dynamic Hypergraph following together with the network structure based on it will be demonstrated.

3.1 GCN and HNN

Assume that an input graph is G= {A,X}. The adjacency matrix is A ∈ Rn×n . Specifically, A = [aij] ∈ Rn×n gives the connectivity information between different nodes, while aij > 0 means there exists an edge between node i and node j. The node set is X ∈ Rn×d , which is treated as input signal on the graph. n represents the number of vertex on graph, d is the dimension of feature. Based on above terminologies, a standard convolution of GCN [Kipf and Welling, 2016] can be represented as follows:

where H(l) denotes the representation of graph nodes in the l-th layer, W (l) stands for the model parameters in the l-th layer, σ (·) represents a non-linear activation function. Additionally, ∼A represents a normalized graph adjacency matrix. Here, ∼A = A+ I, where I is the degree matrix of the image.

The concept of GCN is extended to hypergraph in [Feng et al., 2019] and a new hypergraph neural network (HNN) framework is proposed. The convolution of HNN is reformulated as follows:

where Dv, De denote the diagonal matrices of the vertex degrees and the edge degrees respectively. The filter Γ is applied over the nodes in hypergraph to extract features.

3.2 Semi-Dynamic Hypergraph

In previous work, the human body is often represented as a tree structure. There are 5 common pieces of pre-defined kinematic chains, where 4 pieces represent the limbs respectively and 1 represents the body trunk. In such fixed structural representation, the connections among different joints only represent physical relationships, which cannot cover those potential non-physical connections among other joints while the human is moving. Therefore, we involve the novel hypergraph to describe the kinematic characteristics of human body and construct a semi-dynamic HNN model to address above shortcoming.

Compared with simple graph, each edge in hypergraph can connect more than two nodes. First of all, we transform previous common pieces of pre-defined kinematic chains into 5 hyperedges, each hyperedge corresponding to the kinematic chain connects all of the joints in that kinematic chain. We call this representation as static hypergraph, since it will be used as the fundamental structure for all the human poses. However, there may be some potential connections among different non-adjacent joints under kinematic constrains, which are ignored directly by such hypergraph structure.

To solve this problem, we try to connect those joints which may have potential relation under kinematic constrains into same hyperedges in our hypergraph dynamically. For example, in movement category 'sitting', the hyperedges in our hypergraph may contain more joints from legs, which is different from the movement category such as 'greeting'. Finally, We introduce a new semi-dynamic hypergraph (Figure 2), which contains 5 common pre-defined static hyperedges and different numbers of dynamic hyperedges, as our hypergraph represention of human body.

Before explaining the convolution process on dynamic hypergraph, we introduce how to construct this kind of hypergraph. First, we define a set of hyperedges S, and then construct the hypergraph only by the hyperedges in S. The S contains two parameters, Emax represents the number of edges in dynamic hypergraph, Mmax represents the maxium number of nodes in one hyperedge. We add joint i and joint j into one hyperedge according to their distance. Inspired by some previous work, the relation between two nodes in hyperedge can be calculated according to the formula below:

where Dist is the distance between two joints in feature space, and the Θ is a weight factor matrix. When two joints i and j are very close in graph, the Θ(i, j) will be a small value, while i and j are very far from each other, the Θ(i, j) will be larger. Since the dynamic hypergraph is designed for capturing the relationship between non-adjacent joints and overcome the limitations of traditional tree structure, different joints will have more chance to be connected together. Through setting different Emax and Mmax values, various dynamic hypergraphs can be constructed as our input. In our experiment, the optimal number of hyperedges and that of nodes in a hyperedge can be seen in the ablation study part. The details can be referred to Algorithm 1.

We follow the method in [Feng et al., 2019] for hypergraph convolutions. Different from graph convolution, the hypergraph convolution is a node-edge-node transform. For the input feature X ∈ RN×C , by left multiplying HT ∈ RE×N , we can estimate the edge feature of RE×C , where H is the hyperedge incidence matrix. The edge-node transform is quite same but the order of multiplication is opposite. Since we already know the H in our hypergraph, the node-edge-node transform can be reduced to node-node transform.

Compared with the 'vanilla' HNN, the main difference is that our model involves two types of hypergraph convolutions, the fixed ones and dynamic ones, to enforce the kinematic constrains of human body as illustrated in Figure 2. The convolution on static hypergraph is quite similar to the 'vanilla' HNN. Let Xl be the output of l-th hypergraph layer, and σ denotes a nonlinear activation function. W represents a learnable weight matrix of l-th hypergraph layer and Hs represents the kernel for node-node transform which can be calculated from the input hypergraph structure. The static graph convolution can be expressed as:

To deal with different dynamic hypergraph structures, we add an adaptive weighting matrix M in SD-HNN. Combining the static part and the dynamic part together, we get a complete formula of our network convolution in Equation 5:

where F (l) is the output of l-th hypergraph layer, D is convolution kernel of dynamic hypergraph and M is an adaptive weighting matrix which can be learned during training. In summary, the information propagation in our method is a combination of static part and dynamic part, which is one kind of semi-dynamic way. For each part, the model will perform the convolution independently, exploring the information between neighborhood and global joints, and then weighting matrix will be updated iteratively.

Compared with other GCN models, we have the following two obvious differences: (1) The hypergraph is first involved in our model to estimate the 3D human pose, which is able to better formulate the complex kinematic relationship among human joints. (2) Our network can deal with different hypergraphs, and the corresponding learnable matrix are used to adapt to the dynamic changes of these hypergraph structures. These two characteristics make our network more robust for 3D pose estimation.

3.3 Framework Overview

Figure 3 shows the proposed SD-HNN, which takes a series of 2D poses as input. The input is formally defined as (ν, e), which includes a 2D joints set ν, and a hyperedge based skeleton set e. Suppose P is a set with N 2D joints, and their corresponding 3D joints under predefined camera coordinate system is J, our method aims to learn a regression function A

to minimizes the following error on a dataset of human poses that contains N samples:

where Iproj is the inverse transformation of projection and L computes the L2 distance. This regression function can be trained in an end-to-end fasion.

4 Experiments

4.1 Implementation Details

In our experiment, the Stacked Hourglass network [Newell et al., 2016] is adopted as the basic 2D pose detector and is initialized with weights pre-trained on the MPII dataset and fineturned on Human3.6M. Our model is trained with Adam optimizer for 100 epochs, learning rate of 0.001 and batch size of 256. ReLU is chosen as the nonlinear activation function. The hidden dimension of our method is 256, which means the input data with the shape of (16,2) is mapped into a 256 dimension vector.

4.2 Datasets And Protocols

We evaluate the proposed method on two datasets: Human3.6M and MPI-INF-3DHP. The Human3.6M is one of the largest datasets for 3D human pose estimation. It consists of 3.6 millions of images featuring 11 actors performing 15 daily activities, such as walking, eating, sitting and smoking with 4 camera views. Simultaneously, the dataset also provides some important basic data such as camera parameters, ground truth data of 2D and 3D pose. We use the subject 1, 5, 6, 7 and 8 in Human3.6m for training and subject 9 and 11 for testing. Similar to most of single-frame human pose estimation methods, our method down-sample the data of Human3.6m, and for the video data from original 50fps to 10fps.

The Mean Per Joint Position Error (MPJPE) is used to measure the error between the results and ground truth, that is, ground truth and estimated value are aligned to the root point and then are used to calculate the error. We call this process as Protocol 1. Besides, there are some other work calculating P-MPJPE at the same time, that is, the ground truth and the estimated pose are both aligned through rigid transformation. We call the process as Protocol 2.

MPI-INF-3DHP is a 3D human pose dataset using MoCap system. The test set contains 2929 valid frames from 6 subjects, performing 7 actions. We employ average PCK (with a threshold 150mm) and AUC as the evaluation metrics.

4.3 Quantitative Results

Human3.6M Dataset. We take the 2D pose detected by the Stacked Hourglass network(SH) and the ground truth 2D pose of Human3.6m as the input respectively. Table 1 shows the quantitative comparisons of MPJPE between the estimated pose and the ground-truth on Human3.6M under Protocol 1. As shown in this table, our method achieves competitive results with inaccurate 2D poses, and outperforms other stateof-the-art methods on most individual actions and the best average accuracy with ground truth 2D data. Table 2 shows the quantitative comparisons of MPJPE under Protocol 2. With the ground truth 2D pose as the input, our method achieves the best results.

MPI-INF-3DHP Dataset. Table 3 shows the comparison results between our proposed method with other state-of-theart methods on the MPI-INF-3DHP dataset. We train our model only with Human3.6M data. Our proposed method achieves the best result, since the semi-dynamic hypergraph and its corresponding convolution operation can enforce more kinematic constrains to enhance the generalization capability of our model.

4.4 Ablation Study

In this section, we will verify the effectiveness of different dynamic components in our method on the Human3.6M dataset under Protocol 1. For the effect of initial hypergraph structure on final results, different numbers of hyperedge are set in our initial hypergraph and the results are shown in Table 4. From this table, we can see that when the number of hyperedges is changed from 5 to 20, the results almost keep the same, which means that the ability of our proposed method in constructing dynamic hypergraphs does not depend on the input heavily. Even only 5 pieces of hyperedges are contained in the hypergrah, which only contains the static part, our SD- HNN still can well represent the human body along with its kinematic characteristics by leveraging the corresponding semi-dynamic hypergraph.

To prove the role of semi-dynamic hypergraph in our proposed method, we test SD-HNN on Human3.6M test data with or without dynamic hypergraphs. Table 5 shows the result and demonstrates the dynamic hypergraph part really has improved the performance of HNN by establishing more reasonable connections.

Figure 4 shows the changes of MPJPE in network training with or without dynamic hypergraph. From this table, we can see the introduction of dynamic hypergraph into SD-HNN accelerates the convergence speed during early stage of training, and also achieves better results eventually.

4.5 Qualitative Study

Some results on the Human3.6M are visuzlided in Figure 5. In this figure, the estimated 3D poses with ground truth are compared. From the results, we can see even the human perform complex actions or have self-occlusion, the proposed method still can generate right results.

5 Conclusion

In this paper, we propose a novel Semi-Dynamic Hypergraph Neural Network (SD-HNN) to estimate 3D human pose. SDHNN adopts hypergraph to represent human body to effectively exploit the kiematic constrains among adjacent and non-adjacent joints. A pose hypergraph in SD-HNN has a static component constructed according to the conventional tree body structure, and a dynamic component representing the dynamic kinematic constrains among different joints. These two hypergraphs are combined together to be trained in an end-to-end way. Unlike traditional Graph Convolutional Networks (GCNs) that are based on a fixed tree structure, the SD-HNN is able to deal with ambiguity in human pose estimation. The proposed method achieves state-of-the-art performance on both Human3.6M and MPI-INF-3DHP datasets. In the future, we will further explore other efficient methods to make better use of edge features in hypergraph for more challenge works, such as multiple person pose estimation.

Acknowledgments

The authors would like to thank all the anonymous reviewers. This work was supported by National Natural Science Foundation of China under Grant Number 61772474, 61822701, 61872324, Program for Science & Technology Innovation Talents in Universities of Henan Province(20HASTIT021) and Youth Talent Promotion Project in Henan Province (2019HYTP022).