PoseGTAC: Graph Transformer Encoder-Decoder with Atrous Convolution for 3D Human Pose Estimation

1 Introduction

In recent years, 3D human pose estimation is attracting intensive attention in various human-related research fields, such as action recognition [Yan et al., 2018; Ji et al., 2019], human-object interaction [Li et al., 2020] and motion prediction [Mao et al., 2019]. Its goal is to estimate 3D coordinates of human body joints from 2D poses or images. Compared with the methods [Zhou et al., 2017; Wu and Xiao, 2020] using RGB images, the 2D-to-3D methods [Zhao et al., 2019; Liu et al., 2020] using only 2D poses can avoid the influence of background noise and greatly reduce the computational complexity, thus achieving competitive performance.

In this paper, we focus on the topic of 2D-to-3D pose estimation, which aims to predict 3D poses only given the 2D

∗Corresponding author

pose data. Recently, as an extension of the standard convolutional network, graph neural network (GNN) has shown its natural superiority in capturing the irregular structures in visual data that CNN cannot handle. For human pose estimation, both the 2D and 3D pose information can be regarded as a graph and intuitively be modeled by GNN. The GNNbased methods take the body joints as the nodes and the bones physically connecting body joints as the edges to build the graph. Compared with the traditional methods, the GNNbased methods have achieved better performance. For example, ST-GCN [Yan et al., 2018] first utilized graph convolutional network (GCN) to aggregate the skeleton features and achieved impressive performance. Later, SemGCN [Zhao et al., 2019] introduced the semantic graph convolutional network to capture local and non-local information. Besides, SD-HNN [Liu et al., 2020] leveraged hypergraphs to model the dynamics of the human body for 3D pose estimation.

Though the above approaches have achieved good performance in 3D human pose estimation, they generally adopt the restricted receptive fields of filters and aggregate the single-scale joint information. As illustrated in Fig. 1, existing approaches limitedly consider the 1-hop neighbors when calculating the graph convolution, while neglecting the valuable multi-scale contextual information. Actually, the multi-scale contextual information contains rich features that are essential to facilitate the prediction performance, though it implicitly resides among the higher-order neighbors. Moreover, several recent approaches [Zhao et al., 2019; Liu et al., 2020] attempt to extract non-local information, but ignore the semantic positional information (i.e., the joint type). For instance, two joints of the same coordinates with different semantics may convey totally different information.

To overcome the limitations of existing GNN-based approaches, we propose a novel Graph Transformer EncoderDecoder with Atrous Convolution, dubbed PoseGTAC, to enhance the extraction of multi-scale context and long-range relationships in the pose. As the overall framework is shown in Fig. 2, our proposed PoseGTAC consists of stacked graph atrous convolution layers and graph transformer layers in an encoder-decoder structure which exploits multi-scale features based on human kinetics. Notably, the Graph Atrous Convolution (GAC) can effectively enlarge the receptive fields of filters and densely learn multi-scale pose context, and Graph Transformer Layer (GTL) is used to capture global longrange information. Moreover, graph pooling and graph unpooling are adopted in PoseGTAC to ensure the interaction of multi-scale information from local to global.

Our main contributions are three-fold: (1) We propose a novel PoseGTAC method that can effectively extract local multi-scale context and global long-range relationships for 3Dhuman pose estimation. (2)We design an advancedGraph Atrous Convolution (GAC) to enlarge the receptive fields of filters and learn multi-scale pose context, and a Graph Transformer Layer (GTL) to capture global long-range relationships. The two key components are flexibly combined in an encoder-decoder structure. (3) We conduct extensive experiments on two widely-used datasets Human3.6M and MPIINF-3DHP to demonstrate the superiority of our proposed PoseGTAC model comparing to the state-of-the-art methods.

2 RelatedWork

2D-to-3D Pose Estimation. With a lot of research into 3D human pose estimation, the existing methods can be grouped into three directions, 2D-to-3D pose estimation [Zhao et al., 2019; Liu et al., 2020], monocular image-based 3D pose estimation [Zhao et al., 2019; Wu and Xiao, 2020] and multiview image-based 3D pose estimation [Qiu et al., 2019]. The recent GNN-based methods have greatly improved prediction performance in 2D-to-3D pose estimation. ST-GCN [Yan et al., 2018] first applied graph convolution to aggregate features in the skeleton. SemGCN [Zhao et al., 2019] utilized semantic graph convolution to lift 2D pose to 3D pose by extracting local and non-local information. SD-HNN [Liu et al., 2020] introduced static and dynamic hypergraphs to represent a human body for 3D pose estimation. However, they do not take full advantage of multi-scale contextual information.

Graph Neural Networks. As a generalization of standard convolution and pooling, many methods of graph convolution and graph pooling have recently been proposed. Inspired by the graph Laplacian methods, [Kipf and Welling, 2017] proposed graph convolution networks by the Chebyshev approximation, which is the most widely used form of graph convolution. GraphSAGE [Hamilton et al., 2017] embedded node features by sampling and aggregating and introduced transductive graph convolution. SAGPool [Lee et al., 2019] attempted to use a learnable mask to select the node features retained. In this work, we propose a novel graph transformer encoder-decoder with atrous convolution.

Attention Mechanism. The pioneering work of [Vaswani et al., 2017] introduced a new attention based network named Transformer for the machine translation task of natural language processing (NLP). Transformer is mainly composed of multi-head attention module and feedforward network. There has also been a lot of work based on attention mechanism in computer vision recently [Wang et al., 2018; Carion et al., 2020]. Unlike non-local network based methods [Zhao et al., 2019], our PoseGTAC method introduces the transformer operation on the graph to capture long-range relationships.

3 ProposedMethod

3.1 Preliminaries

Graph Definition. The raw pose data, i.e., the joint keypoint vector, is a set of 2D coordinates. Usually, in GNN-based approaches, the vector can be represented as a graph G = (V, E), where the vertices are all joints and edges are physical connections between two joints. Here V is the set of N joints and E is characterized by the adjacency matrix A ∈ RN×N . In this way, the pose data is transformed into a graph sequence and specifically represented as a tensor X ∈ RN×C , where N and C denote the numbers of joints and channels.

Graph Convolution. Based on the above definition, the existing GNN-based methods generally use the stacked graph convolution module to extract the high-level skeleton information to regress 3D pose. The commonly used graph convolution operation can be represented as follows:

where σ(·) denotes the activation function, i is the index of the current layer and Λ denotes the normalized Laplace matrix. Λ = D- 1 2AD- 12 , where A is the adjacency matrix with the self-loop, and D denotes the degree matrix of the graph. W denotes a Cout × Cin × 1 × 1 learnable weight matrix and M denotes an N × N attention mask matrix. ⊙ denotes the element-level dot product.

In addition to using local GCN to extract skeleton features, non-local module is introduced to extract long-range information, which is generally represented as extracting the relationships between the current node and all other nodes.

3.2 Graph Atrous Convolution (GAC)

As shown in Eq. 1, the previous methods use the restricted filters to convolve only 1-hop neighbors, ignoring the multiscale contextual information from higher-order neighbors. We consider that multi-scale context is indispensable for 3D human pose estimation. To this end, we introduce a multi-scale graph convolution termed Graph Atrous Convolution (GAC) to capture the multi-scale context residing in the higher-order neighbors. Inspired by the atrous convolution [Yu and Koltun, 2016; Chen et al., 2018] in image segmentation, convolution operations with different dilation factors are used in parallel. As illustrated in Fig. 3, in our graph convolution, the dilation factor is defined as the distance to the root node, and graph atrous convolution is represented as paralleled convolutions with root node, 1-hop, 2-hop and 3hop neighbors, etc. We first formally define the k-hop matrix Ak as follows:

where d(vi, vj) denotes the distance of the shortest path between vi and vj on the skeleton graph, and Ak is the k-hop adjacency matrix with the self-loop.

where Λk denotes the normalized k-hop Laplace matrix. Λk = D- 12 k AkD- 12 k , where Dk denotes the degree matrix of the graph. Wk denotes a learnable weight matrix for node embedding and Mk denotes a N×N learnable attention mask matrix. Y(l) k ∈ RN×C denotes the output of k-hop graph atrous convolution.

Moreover, in order to facilitate global context information, skeleton features pooled globally are concatenated with the output of parallel graph atrous convolution in Eq. 3 and then fed to a multi-layer perceptron (MLP) to aggregate multiscale and global context features.

where AvgPool(·) and Cat(·) respectively denote the average pooling and concatenation operation. Y(l) pool ∈ RN×C represents the output features pooled globally, Y(l) ∈ RN×[(k+1)×C] is the features of concatenating all branches. W is a learnable weight matrix for feature aggregation and dimension reduction. X(l+1) denotes the final output features of the GAC layer.

3.3 Graph Transformer Layer (GTL)

Although the multi-scale contextual information has been well extracted by the GAC module based on the local physical connection of the human body, there is a lack of long-range information that can effectively promote pose representation learning. To determine whether there is a connection between two joints and how strong the connection is, we introduce the graph transformer layer (GTL) to better capture long-range information.

Since the joints in the pose have no order to uniquely identify their types from the input, the position encoding needs to be added to complement the position information. In particular, we follow the sine and cosine functions as the position encoding functions in [Vaswani et al., 2017]:

where pos is the position and i is the dimension of the position encoding vector. As shown in Fig. 4, in the GTL, the raw input is first added with the position encoding, and then fed to two embedding functions (e.g., θ and φ) for obtaining the high-level features. The dot product is adopted to measure the similarity of the two joints in an embedding space. Specifically, we can calculate the attention matrix representing the strength of the relationships between the joints as follows:

where Softmax(·) denotes the softmax operation used for normalization, and Matt denotes the attention map. Wθ and Wφ are the learnable weight matrices of the embedding functions θ and φ, respectively.

In addition, we add an extra N × N global attention matrix Mglobal to pay more attention on unconstrained learning. Specifically, the global attention matrix is added to the attention matrix Matt and then multiplied by the original input. As a result, the multi-head attention and feedforward network can be used to obtain long-range features with rich attention.

3.4 The PoseGTACArchitecture

To obtain multi-scale (e.g., joint-scale and part-scale) information based on human kinetics, graph pooling and graph unpooling are adopted to capture effectively the interaction of multi-scale information in the pose. Before graph pooling and unpooling, the nodes of each scale s are divided into different regions Rs according to the physical priors of the human body, such as the upper left leg, the lower left leg, the torso, etc. For the upper scale features, we implement the pooling by using average pooling to aggregate the point features that are divided into a region in the current scale to get the lower scale features. In other words, multiple points are averaged into one point. Specifically, the formula is implemented as follows:

where Rsi denotes the region feature under scale s, and Xsj denotes a joint feature element belonging to Rsi set. Xs+1 i denotes a new joint feature under scale s+1 obtained through graph pooling.

Due to the independence of the partitioned regions, the unpooling is realized by copying an upper scale feature multiple times and then concatenating them together in the corresponding lower scale region.

where the number of repetitions of Xs+1 i is determined by the size of the corresponding set Rsi .

In addition, the features obtained by unpooling are concatenated with the corresponding features on the contraction path, and then fed to the next layer. The graph atrous convolution layer and graph transformer layer are used to extract local and global information, while pooling and unpooling are used to facilitate the interaction of information from local to global. Fig. 2 illustrates the architecture of our proposed PoseGTAC, which stacks five graph atrous convolution layers and five graph transformer layers at different scales. Two graph convolution layers are used for input encoding and output decoding procedures. Each layer is followed by a BN layer and a ReLU layer. The mean squared error (MSE) between the predicted pose and the ground-truth is used as our loss function, which can be trained in an end-to-end manner.

4 Experiments

4.1 Experimental Setup

Datasets and Evaluation Protocols. Following previous studies [Liu et al., 2020], we adopt two benchmark datasets Human3.6M and MPI-INF-3DHP in our experiments.

Human3.6M [Ionescu et al., 2014] is the largest 3D human pose estimation dataset. It contains 3.6 million images, where 11 professional actors perform 15 actions such as walking, greeting, smoking and making a phone call. Both 2D and 3D ground-truth data are available for supervised 3D human pose estimation. We use five subjects (S1, S5, S6, S7, and S8) for training and two subjects (S9 and S11) for testing. In order to reduce redundancy, we follow [Zhao et al., 2019] and downsample the raw videos from 50fps to 10fps for the training and testing datasets. MPI-INF-3DHP [Mehta et al., 2017] is the dataset obtained using MoCap system for 3D human pose estimation. The test set consists of 2,929 frames from 6 subjects performing 7 actions.

Three evaluation protocols are adopted in our experiments: Protocol #1 is the mean per-joint position error (MPJPE) in millimeters which measures the error between the groundtruth and predictions. Protocol #2 is P-MPJPE which reports the error between the ground-truth and predictions through the rigid transformation including translation, rotation and scale. As an auxiliary, we also measure the mean per-joint velocity error (MPJVE) as Protocol #3, which is obtained by the MPJPE of the first derivative and represents the smoothness of predicted results. Two standard metrics of PCK (Percentage of Correct Keypoints) under 150mm radius and AUC (Area under the ROC Curve) are used for quantitative evaluation for MPI-INF-3DHP.

Implement Details. Our PoseGTAC model consists of two graph convolution layers, five graph transformer layers (GTL) with three heads and five graph atrous convolution (GAC) layers. The number of channels for all layers is 128, except for GTL, which has 32 intermediate channels for reducing the computational complexity of the model. We set up three different scales containing 16,10 and 5 joints. The Adam optimizer is adopted as our optimizer with the initial learning rate 0.001 and the decay factor 0.96 per 100K steps. We train our model for 50 epochs with the batch size 256. All experiments are conducted on PyTorch deep learning framework with a single RTX-2080Ti GPU.

PCK ↑ AUC ↑

32.5

32.0

35.3

36.0

37.5

39.3

4.2 Quantitative Results

To better evaluate the performance of our proposed PoseGTAC model, we show quantitative results and compare them to the state-of-the-art methods on 3D human pose estimation.

Table 1 and Table 2 show the experiment results on the Human3.6M dataset for Protocol #1 and Protocol #2, respectively. We can observe that the proposed PoseGTAC method outperforms all the compared models on both two protocols and obtains the best results in terms of the average and most individual actions. In particular, the MPJPE reduces by 1.7mm with an error reduction of 4.3%, and the P-MPJPE decreases by 2.7mm with an 8.4% error reduction, respectively. It proves that our PoseGTAC extracts rich multi-scale context and promotes long-range feature interaction.

Moreover, we also provide supplementary to evaluate the smoothness of the predicted pose by our PoseGTAC model as Protocol #3. Additionally, the results on Protocol #3 also reflect the effectiveness and precision of our model from the side. As reported in Table 4, the MPJVE in our PoseGTAC

model reduces the results of the previous work [Pavllo et al., 2019] by 80%, which clearly validates that the pose predicted by our PoseGTAC model is smoother and more precise.

For the MPI-INF-3DHP dataset, we train our model with Human3.6M data without post-process of fine-tuning or retraining. As shown in Table 3, comparing to the best counterpart of [Liu et al., 2020], our PoseGTAC model achieves an improvement of 1.5% PCK and 1.8% AUC, which consistently indicates the effectiveness of our PoseGTAC model.

4.3 Ablation Studies

Effectiveness of Each Module. To verify the effectiveness of different modules in our proposed model, we conduct a series of ablation studies on the Human3.6M dataset under Protocol #1. For the ablation of the overall network architecture, we take SemGCN [Zhao et al., 2019] as our baseline. We set whether to use or combine graph transformer layer (GTL) and graph atrous convolution (GAC), and finally compare our PoseGTAC model with the encoder-decoder framework. As shown in Table 6, compared to our baseline, adding only GAC or GTL is able to obtain remarkable error reduction, and adopting the encoder-decoder framework achieves the best results.

We further explore the configuration of the two main modules we designed in detail. Table 5 shows the performance comparison of GAC module with different numbers of branches and whether the pooling is used in GAC. We configure our GAC with four branches (i.e., "4s") and six branches (i.e., "6s"). By contrast, the error of our GAC with four branches is the lowest, and then is further reduced by adding pooling (i.e., "4s+pool"). We also evaluate the components of GTL, including the position encoding (PE) and global attention (GA). Based on our "vanilla" GTL, adding the position encoding or global attention can obtain 1.3% and 0.8% error reduction, showing that the position encoding can well complement the semantic information in the joint sequence.

Visualization of the Learned Matrices. Furthermore, to explore how the information is aggregated between the joints, we visualize the learned weight matrices of the first GAC and GTL of our PoseGTAC model. Specifically, we obtain the final adjacency matrix in Fig. 5(a) by adding the weight matrices of the four branches of GAC. From the adjacency matrix, i.e., the GAC filter, we can see that the receptive field of our filter is sufficiently enlarged and contains all joints from the root to the 3-hop neighbors. It is probable that our model focuses more on the wrists and feet that incorporate multi-scale information. In addition, the attention matrix is obtained by adding the attention map and global attention matrix in GTL. As shown in Fig. 5(b), long-range relationships can be effectively extracted in each joint, especially in the hip and spine. It indicates that our proposed model not only focuses on local multi-scale contextual information, but also concentrate on global long-range relationships.

Qualitative Pose Estimation Results. We finally illustrate typical visualizations of the poses predicted by our PoseGTAC model in Fig. 6, where five different actions, "greeting", "posing", "sitting", and "purchases" are included. We can observe that the poses for all sequences predicted by PoseGTAC are considerably accurate comparing to the ground-truth annotations, even in the challenging scenario with occlusion problem (e.g., "purchases").

5 Conclusion

In this work, we proposed a novel Graph Transformer Encoder-Decoder with Atrous Convolution named PoseGTAC to extract effectively multi-scale contextual information and capture accurately global long-range relationships for 3D human pose estimation. Moreover, we designed two modules, graph atrous convolution (GAC) and graph transformer layer (GTL), respectively for the extraction of multi-scale and longrange information, and combine them in the encoder-decoder structure. The extensive experiments on the Human3.6M and MPI-INF-3DHP datasets validated that our PoseGTAC model can extract abundant multi-scale and global long-range information, which is beneficial for 3D human pose estimation.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grants 61976049, 62072080 and U20B2063; the Sichuan Science and Technology Program 2018GZDZX0032, 2019ZDZX0008, 2019YFG0003, 2019YFG0533 and 2020YFS0057.