Self-Supervised Gait Encoding with Locality-Aware Attention for Person Re-Identification

1 Introduction

Fast, effective and reliable person re-identification (Re-ID) is vital for various applications such as security authentication, human tracking, and role-based activity understanding [Vezzani et al., 2013]. For person Re-ID, gait is one of the most useful human body characteristics, and it has drawn increasing attention from researchers since it can be extracted by unobtrusive methods without co-operative subjects [Connor and Ross, 2018]. Existing physiological and psychological studies [Murray et al., 1964; Cutting and Kozlowski, 1977] suggest that different human individuals have different gait characteristics, which contain numerous unique and relatively stable patterns (e.g., stride length and angles of body joints). In the field of gait analysis, gait can be described by two types of methods: (a) Appearance-based methods [Zhang et al., 2019], which exploit human silhouettes from aligned image sequences to depict gait. However, this type of methods are vulnerable to body shape changes and appearance changes. (b) Model-based methods [Liao et al., 2020], which model gait by body structure and motion of human body joints. Unlike appearance-based methods, model-based methods are scale and view invariant [Nambiar et al., 2019], thus leading to better robustness in practice. Among different models, 3D skeleton model, which describes humans by 3D coordinates of key body joints, is a highly efficient representation of human body structure and motion [Han et al., 2017]. Compared with RGB or depth data, 3D skeleton enjoys many merits like better robustness and much smaller data size, and they can be easily collected by modern devices like Kinect. Therefore, exploiting 3D skeleton data to perform gait analysis for downstream tasks like person Re-ID (illustrated in Fig. 1) has gained surging popularity [Liao et al., 2020]. Nevertheless, the way to extract or learn discriminative gait features from 3D skeleton sequence data remains open to be explored.

∗Authors contribute equally.

†Codes are available at https://github.com/Kali-Hac/SGE-LA.

To this end, most existing works like [Barbosa et al., 2012; Andersson and Araujo, 2015] rely on hand-crafted skeleton descriptors. However, it is usually complicated and tedious to design such hand-crafted descriptors, e.g., [Andersson and Araujo, 2015] defines 80 skeleton descriptors from the views of anthropometric and gait attributes for person Re-ID. Besides, they heavily rely on domain knowledge such as human anatomy [Yoo et al., 2002], thus lacking the ability to mine latent gait features beyond human cognition. To alleviate this problem, few recent works like [Haque et al., 2016] resort to deep neural networks to learn gait representations automatically. However, they all follow the classic supervised learning paradigm and require labeled data, so they cannot perform gait representation learning with unlabeled skeleton data. In this paper, we for the first time propose a self-supervised approach with locality-aware attention mechanism, which only requires unlabeled 3D skeleton data for gait encoding. First, by introducing certain self-supervision as a high-level learning goal, we not only enable our model to learn gait representations from unlabeled skeleton data, but also encourage the model to capture richer high-level semantics (e.g., sequence order, body part motion) and more discriminative gait features. Specifically, we propose a self-supervised learning objective that aims to reconstruct the input skeleton sequence in reverse order, which is implemented by an encoder-decoder architecture. Second, since the continuity of motion leads to very small pose/skeleton changes in a small time interval [Aggarwal and Cai, 1999], this endows skeleton sequences with the property of locality: For each skeleton frame in a skeleton sequence, its temporally adjacent skeleton frames within a local context enjoy higher correlations to itself. Thus, to enable better skeleton reconstruction and gait representation learning, we propose the locality-aware attention mechanism to incorporate such locality into the gait encoding process. Last, we leverage the context vectors learned by the proposed locality-aware attention mechanism to construct Attentionbased Gait Encodings (AGEs) as the final gait representations. We demonstrate that AGEs, which are learned without skeleton sequence labels, can be directly applied to person Re-ID and achieve highly competitive performance.

In summary, we make the following contributions:

• We propose a self-supervised approach based on reverse sequential skeleton reconstruction and an encoderdecoder model, which enable us to learn discriminative gait representations without skeleton sequence labels.
• We propose a locality-aware attention mechanism to exploit the locality nature in skeleton sequences for better skeleton reconstruction and gait encoding.
• We propose AGEs as novel gait representations, which are shown to be highly effective for person Re-ID.

Experiments demonstrate the effectiveness of our approach in gait encoding/person Re-ID. It is also shown that our model is readily transferable among datasets, which validates its abilities to learn transferable high-level features of skeletons.

2 RelatedWork

PersonRe-IDusing SkeletonData. For personRe-ID, most existing works extract hand-crafted features to depict certain geometric, morphological or anthropometric attributes of 3D skeleton data. [Barbosa et al., 2012] computes 7 Euclidean distances between the floor plane and joint or joint pairs to construct a distance matrix, which is learned by a quasiexhaustive strategy to perform person Re-ID. [Munaro et al., 2014b] further extends them to 13 skeleton descriptors (D13 ) and leverages support vector machine (SVM) and k-nearest neighbor (KNN) for classification. In [Pala et al., 2019], 16 Euclidean distances between body joints (D16 ) are fed to Adaboost to realize Re-ID. Since skeleton based features alone are hard to achieve satisfactory Re-ID performance, features from other modalities (e.g., 3D point clouds [Munaro et al., 2014a], 3D face descriptor [Pala et al., 2019]) are also used to enhance the performance. Meanwhile, few recent works exploit supervised deep learning models to learn gait representations from skeleton data: [Haque et al., 2016] utilizes long short-term memory (LSTM) [Hochreiter and Schmidhuber, 1997] to model temporal dynamics of body joints to perform person Re-ID; [Liao et al., 2020] proposes PoseGait, which feeds 81 hand-crafted pose features of 3D skeleton data into convolutional neural networks (CNN) for human recognition.

Depth-based and Multi-modal Person Re-ID Methods. Depth-based methods typically exploit human shapes or silhouettes from depth images to extract gait features for person Re-ID. [Sivapalan et al., 2011] extends the Gait Energy Image (GEI) [Chunli and Kejun, 2010] to 3D domain and proposes Gait Energy Volume (GEV) algorithm based on depth images to perform gait-based human recognition. 3D point clouds based on depth data are also widely used to estimate body shape and motion trajectories. [Munaro et al., 2014b] proposes point cloud matching (PCM) to compute the distances of multi-view point cloud sets, so as to discriminate different persons. [Haque et al., 2016] adopts 3D LSTM to model motion dynamics of 3D point clouds for person Re-ID. As to multi-modal methods, they usually combine skeleton-based features with extra RGB or depth information (e.g., depth shape features based on point clouds [Munaro et al., 2014a; Hasan and Babaguchi, 2016; Wu et al., 2017]) to boost Re-ID performance. In [Karianakis et al., 2018], CNN-LSTM with reinforced temporal attention (RTA) is proposed for person Re-ID based on a split-rate RGB-depth transfer approach.

3 The Proposed Approach

Suppose that an input skeleton sequence S = (S1, · · · , Sf) contains f consecutive skeleton frames, where Si ∈ RJ×3 contains 3D coordinates of J body joints. The training set Φ = {S(i) }Ni=1 contains N skeleton sequences collected from different persons. Each skeleton sequence S(i) corresponds to a label yi, where yi ∈ {1, · · · ,C} and C is the number of persons. Our goal is to learn discriminative gait features v from S without using any label. Then, the effectiveness of learned features v is validated by using them to perform person Re-ID: Learned features and labels are used to train a simple recognition network (note that the learned

Attention Mechanism

features v are fixed and NOT tuned by training at this stage). The overview of the proposed approach is given in Fig. 2, and we present the details of each technical component below.

3.1 Self-Supervised Skeleton Reconstruction

To learn gait representations without labels, we propose to introduce self-supervision by learning to reconstruct input skeleton sequences in reverse order, i.e., by taking S as inputs, we expect our model to output the sequence ˆ S = ( ˆ S1, · · · , ˆ Sf) = (Sf , · · · , S1), which gives ˆ St = Sf-t+1. Compared with na¨ıve reconstruction that learns to reconstruct exact inputs (S↦-→S), the proposed learning objective (S ↦-→ ˆ S) is endowed with high-level information (skeleton order in a sequence) that is meaningful to human perception, which requires the model to capture richer high-level semantics (e.g., body parts, motion patterns) to achieve this learning objective. In this way, our model is expected to learn better gait representations than frequently-used plain reconstruction. Formally, given an input skeleton sequence, we use the encoder to encode each skeleton frame St (t ∈ {1, · · · f}) and the previous step's latent state ht-1 (if existed), which provides context information, into current latent state ht:

where ht ∈ Rk , φE(·) denotes our Gait Encoder (GE). GE is built with an LSTM, which aims to capture long-term temporal dynamics of skeleton sequences. h1, · · · , hf are encoded gait states that contain preliminary gait encoding information. In the training phase, encoded gait states are decoded by a Gait Decoder (GD) to reconstruct the target sequence ˆ S, and the decoding process is performed below (see Fig. 2):

where φD(·) denotes the GD. GD consists of an LSTM and a fully connected (FC) layer that outputs predicted skeletons. ˆht ∈ Rk refers to the tth decoded gait state, i.e., the latent state output by GD's LSTM to predict tth skeleton St. When the decoding is initialized (t = 1), we feed an all0 skeleton placeholder Z ∈ RJ and the final encoded gait state hf into GD to decode the first skeleton. Afterwards, to predict tth skeleton St, φD(·) takes three inputs from the (t - 1)th decoding step: decoded gait state ˆht-1, the ground truth skeleton ˆ St-1 (Sf-t+2) that enables better convergence, and the attentional state vector ht-1 that fuses encoding and decoding information based on the proposed attention mechanism, which will be elaborated in Sec. 3.2. In this way, we define the objective function LR for skeleton reconstruction, which minimizes the mean square error (MSE) between a target skeleton sequence ˆ S and a predicted skeleton sequence S:

where Sij, ˆ Sij represent the j th joint position of the ith predicted or target skeleton. In the testing phase, to test the reconstruction ability of our model, we use the predicted skeleton St-1 rather than target skeleton ˆ St-1 as the input to φD in the t > 1 case, i.e., (ˆht, St) = φD (ˆht-1, St-1, ht-1).

To facilitate training, our implementation actually optimizes Eq. 3 on each individual dimension of the skeleton's 3D coordinates: S[d] ↦-→ˆ S[d] , where d ∈ {X, Y, Z} corresponds to a certain dimension of 3D space, and S[d] , ˆ S[d] ∈ Rf×J .

3.2 Locality-Aware AttentionMechanism

As learning gait features requires capturing motion patterns from skeleton sequences, it is natural to consider a straightforward property of motion-Continuity. Motion's continuity ensures that those skeletons in a small temporal interval will NOT undergo drastic changes, thus resulting in higher interskeleton correlations in this local temporal interval, which is referred as "locality". Due to such locality, when reconstructing one skeleton in a sequence, we expect our model to pay more attention to its neighboring skeletons in the local context. To this end, we propose a novel locality-aware attention mechanism, with its modules presented below:

Basic Attention Alignment. We first introduce Basic Attention (BA) alignment [Luong et al., 2015] to measure the content (i.e., latent state) based correlations between the input sequence and the predicted sequence. As shown in Fig. 2, at tth decoding step, we compute the BA Alignment Scores (BAS) between ˆht and encoded gait state hj (j ∈ {1, . . . , f}):

BAS aims to focus on those more correlative skeletons in the encoding stage, and provide preliminary attention weights for skeleton decoding. However, BA alignment only considers the content based correlations and does not explicitly take locality into consideration, which motivates us to propose locality mask and locality-aware attention alignment below.

Locality Mask. Our motivation is to incorporate locality into the gait encoding process for better skeleton reconstruction. As the goal is to decode the tth skeleton St as ˆ St, we assume those skeletons in the local temporal context of Spt to be highly correlated to St, where pt = f -t+1 (note that we use reverse reconstruction). To describe the local context centered at Spt , we define an attentional window [pt-D, pt+D], where D is a selected integer to control the attentional range. Since the locality will favor temporal positions near pt (closer positions are more correlative), a direct solution is to place a Gaussian distribution centered around pt as a locality mask:

where we empirically set σ = D2 , j is a position within the window centered at pt. We can weight BAS by this locality mask to compute Masked BA Alignment Scores (MBAS) below, which directly forces alignment scores to obtain locality:

Locality-Aware Attention Alignment. Despite that the locality mask is straightforward to yield locality, it is a very coarse solution that brutally changes the alignment scores. Therefore, instead of using MBAS (˜at(j)) directly, we propose the Locality-aware Attention (LA) alignment. Specifically, an LA alignment loss term LA is used to encourage LA alignment to learn similar locality of the ˜at(j):

By adding the loss term LA, we can obtain LA Alignment Scores (LAS). Note that in Eq. 4, the final learned at(j) is BAS. For clarity, we use at(j) to represent learned LAS here.

With the guidance of LA, our model learns to allocate more attention to the local context by itself, rather than using a locality mask. To utilize alignment scores to yield an attentionweighted encoded gait state at the tth step, we calculate the context vector ct by a sum of weighted encoded gait states:

Note that context vector ct can also be computed by BAS or MBAS. ct provides a synthesized gait encoding that is more relevant to ˆht, which facilitates the reconstruction of tth skeleton. To combine both encoding and decoding information for reconstruction, we use a concatenation layer fatt(·) that combines ct and ˆht into an attentional state vector ht:

where Watt represents the learnable weight matrix in the layer. Finally, we generate (reconstruct) the tth skeleton by the FC layer fF (·) of the GD:

whereWF is the weights to be learned in this FC layer.

Remarks: (a) To provide an intuitive illustration of the proposed locality-aware attention mechanism, we visualize the BAS and LAS attention matrices, which are formed by average alignment scores computed on dimension X as an example. As shown by Fig. 3(a), LA alignment significantly improves the locality of learned alignment scores: Relatively large alignment scores are densely distributed near the clinodiagonal line of the alignment score matrix (clinodiagonal line reflects skeletons' correlations to themselves when performing reverse reconstruction), which means adjacent skeletons are assigned with larger attention than remote skeletons. By contrast, despite that BA alignment also learns locality to some extent, BA's alignment weights show a much more even distribution, and many non-adjacent skeletons are also given large alignment scores. Similar trends are also observed for dimension Y and Z. (b) The proposed LA alignment enables us to learn better reconstruction of skeleton sequences. We visualize the reconstruction loss during training in four cases: using no attention mechanism, BAS, MBAS and LAS. As shown by Fig. 3(b), it can be observed that training with LAS converges at a faster speed with an evidently smaller reconstruction loss, which justifies our intuition that locality will facilitate training. Interestingly, we observe that using the locality mask directly in fact does not benefit training, and it verifies that learning is a better way to obtain locality.

3.3 Attention-based Gait Encodings (AGEs)

Since our ultimate goal is to learn good gait features from skeleton data to perform person Re-ID, we need to extract certain internal skeleton embedding from the proposed model as gait representations. Unlike traditional LSTM based methods that basically rely on the last hidden state to compress the temporal dynamics of a sequence [Weston et al., 2015], we recall that the dynamic context vector ct from the attention mechanism integrates the key encoded gait states of input skeletons and retains crucial spatio-temporal information to reconstruct target skeletons. Hence, we utilize them instead of the last hidden state to build our final gait representationsAGEs. A skeleton-level AGE (vt) is defined as follows:

where cdt denotes the context vector computed on dimension d ∈ {X, Y,Z} of the tth step in decoding. To perform person Re-ID, we use vt to train a simple recognition network fRN(·) that consists of a hidden layer and a softmax layer. We average the prediction of each skeleton-level AGE fRN(vt; θτ) (t ∈ {1, · · · , f}) in a skeleton sequence to be the final sequence-level prediction for person Re-ID, where θτ refers to parameters of fRN(·). Note that during training, each skeleton in one sequence shares the same skeleton sequence label yi. Besides, skeleton labels are only used to train the recognition network, i.e., AGEs are fixed during training to demonstrate the effectiveness of learned gait features.

3.4 The Entire Approach

As a summary, the computation flow of the entire approach during skeleton reconstruction is h → ˆh → a → c → h → S. To guide model training in the reconstruction process, we combine the skeleton reconstruction loss LR in Eq. 3 and the LA alignment loss LA in Eq. 7 as follows:

where Θ denotes the parameters of the model, λR, λA are the weight coefficients to trade off the importance of the reconstruction loss and LA alignment loss, ‖Θ‖22 is L2 regularization. For the person Re-ID task, we employ the cross-entropy loss to train the recognition network with AGEs (vt).

4 Experiments

4.1 Experimental Settings

Datasets: We evaluate our method on three public ReID datasets that provide 3D skeleton data: BIWI [Munaro et al., 2014b], IAS-Lab [Munaro et al., 2014c] and Kinect Gait Biometry Dataset (KGBD) [Andersson and Araujo, 2015]. They collect skeleton data from 50, 11 and 164 different individuals respectively. We follow the evaluation setup in [Haque et al., 2016], which is frequently used in the literature: For BIWI, we use the full training set and the Walking testing set that contains dynamic skeleton data; For IAS-Lab, we use the full training set and two test splits, IAS-A and IASB; For KGBD, since no training and testing splits are given, we randomly leave one skeleton video of each person for testing and use the rest of videos for training. The experiments are repeated for multiple times and the average performance is reported on KGBD. We discard the first and last 10 frames of each original skeleton sequence to avoid ineffective skeleton recording. Then, we spilt the training dataset into multiple skeleton sequences with the length f , and two consecutive sequences share f2 overlapping skeletons, which aims to obtain as many skeleton sequences as possible to train our model.

Implementation Details: The number of body joints is set to the maximum number in all datasets, namely J = 20. The sequence length f is empirically set to 6 as it achieves the best performance in average among different sequence length settings. To learn the locality-aware attention for the whole sequence, the attentional range D of LA is set to 6. We use a 2-layer LSTM with k = 256 hidden units per layer for both GE and GD. We empirically set both λR and λA to 1, while a momentum 0.9 is utilized for optimization. We use a learning rate lr = 0.0005, and we set the weight of L2 regularization β to 0.02. The batch size is set to 128 in all experiments.

Evaluation Metrics: Person Re-ID typically follows a "multi-shot" manner that leverages predictions of multiple frames or a sequence-level representation to produce a sequence label. In this paper, we compute both Rank-1 accuracy and nAUC (area under the cumulative matching curve (CMC) normalized by the number of ranks [Gray and Tao, 2008]) to evaluate multi-shot person Re-ID performance.

4.2 Performance Comparison

In Table 1, we conduct an extensive comparison with existing skeleton based person Re-ID methods (Id = 10-15) in the literature. In the meantime, we also include classic depth-based methods (Id = 1-3) and representative multi-modal methods (Id = 4-9) as a reference. We obtain observations below:

Comparison with Skeleton-based Methods: As shown by Table 1, our approach enjoys obvious advantages over existing skeleton-based methods in terms of person Re-ID performance: First, our approach evidently outperforms those methods that rely on manually-designed geometric or anthropometric skeleton descriptors (Id = 10-12). For example, D13 (Id = 11) and recent D16 (Id = 12) are two most representative hand-crafted feature based methods, and our model outperforms both of them by a large margin (17.3%-40.8% Rank-1 accuracy and 5.7%-22.2% nAUC on different datasets). Second, our approach is also superior to recent skeleton based methods that utilize deep neural networks (Id = 13-15). Our approach is the best performer on three out of four datasets (BIWI, IAS-A, IAS-B) with a 14.7%-43.3% Rank-1 accuracy and 1.8%-20.7% nAUC gain. On KGBD dataset, our approach ranks 2nd and performs slightly inferior to the latest PoseGait (Id = 15). However, despite that CNN is used, PoseGait still requires extracting 81 hand-crafted features. Besides, labeled skeleton data are indispensable for existing deep learning based methods, while our approach can learn better gait representations by unlabeled skeletons only.

Comparison with Depth-based Methods and Multimodal Methods: Despite that our approach only takes skeleton data as inputs, our approach consistently outperforms baselines of classic depth-based methods (Id = 1-3) by at least 24% Rank-1 and 3.2% nAUC gain. Considering the fact that skeleton data are of much smaller size than depth image data, our approach is both effective and efficient. As to the comparison with recent methods that exploit multi-modal inputs (Id = 4-9), the performance of our approach is still highly competitive: Although few multi-modal methods perform better on IAS-B, our skeleton based method achieves the best Rank-1 accuracy on BIWI and IAS-A. Interestingly, we note that the multi-modal approach that uses both point cloud matching (PCM) and skeletons yields the best accuray on IAS-B, but it performs markedly worse on datasets that undergo more frequent shape and appearance changes (IAS-A and BIWI). By contrast, our approach consistently achieves stable and satisfactory performance on each dataset. Thus, with 3D skeletons as the sole input, our approach can be a promising solution to person Re-ID and other potential skeleton-related tasks.

5 Discussion

Ablation Study. We perform ablation study to verify the effectiveness of each model component. As shown in Table 2, we draw the following conclusions: (a) The proposed encoder-decoder architecture (GE-GD) performs remarkably better (5.4% Rank-1 accuracy and 4.5% nAUC gain) than supervised learning paradigm that uses GE only, which verifies the necessity of encoder-decoder architecture and skeleton reconstruction mechanism. (b) Using reverse reconstruction (Rev. in Table 2) produces evident performance gain (1.4%-9.4% Rank-1 accuracy and 0.8%-1.4% nAUC) when compared with those configurations without reverse reconstruction. Such results justify our claim that reverse reconstruction enables the model to learn more discriminative gait features for person Re-ID. (c) Introducing attention mechanism (BAS) improves the model by 6.6% Rank-1 accuracy and 3.1% nAUC, while LAS further improves BAS's performance by 4.4% Rank-1 and 1.2% nAUC. Besides, it is noted that directly using locality mask (MBAS) degrades the performance. This substantiates our claim that learning locality enables better gait representation learning. (d) AGEs provide more effective gait representations: Using AGEs can consistently improve the Re-ID performance by up to 1.8% Rank1 accuracy and 0.7% nAUC, regardless of the type of used alignment scores. Other datasets report similar results.

Gait Encoding Model Transfer. We discover that the gait encoding model learned on one dataset can be readily transferred to other datasets. For example, we use the model pretrained on KGBD to directly encode skeletons from BIWI and IAS-Lab, and compare the Re-ID performance with the model trained on BIWI and IAS-Lab themselves (denoted by "Self"): As shown by Table 3, two cases achieve fairly comparable performance, while the transferred model even outperforms the original model on IAS-A (LAS) and IAS-B (BAS). Such transferability demonstrates that our approach can learn transferable high-level semantics of 3D skeleton data, which enables the pre-trained model to capture discriminative gait features from unseen skeletons of a new dataset.

6 Conclusion

In this paper, we propose a generic self-supervised approach to learn effective gait representations for person Re-ID. We introduce self-supervision by learning reverse skeleton reconstruction, which enables our model to learn high-level semantics and discriminative gait features from unlabeled skeleton data. To facilitate skeleton reconstruction and gait representation learning, a novel locality-aware attention mechanism is proposed to incorporate the locality into gait encoding process. We construct AGEs as final gait representations to perform person Re-ID. Our approach evidently outperforms existing skeleton-based Re-ID methods, and its performance is comparable or superior to depth-based/multi-modal methods.

Acknowledgements

This work was supported in part by the National Key Research and Development Program of China (Grant No. 2019YFA0706200, No. 2018YFB1003203), in part by the National Natural Science Foundation of China (Grant No. 61632014, No. 61627808, No. 61210010, No. 61773392, No. 61672528), in part by the National Basic Research Program of China (973 Program, Grant No. 2014CB744600), in part by the Program of Beijing Municipal Science & Technology Commission (Grant No. Z171100000117005), and in part by the Program for Guangdong Introducing Innovative and Enterpreneurial Teams 2017ZT07X183, Fundamental Research Funds for the Central Universities D2191240. Xiping Hu, Jun Cheng and Bin Hu are the corresponding authors of this paper.