Deep View-Aware Metric Learning for Person Re-Identification

1 Introduction

Person re-identification attracts increasing attentions due to its importance in many applications, e.g., video surveillance, pedestrian retrieval, and human-computer interaction. Although significant progress has been made in these years, there are still some challenging problems existing in person re-identification: 1) dramatic changes in visual appearance; 2) dissimilarity between two images of the same pedestrian; 3) similarity between two images of different pedestrians.

To address these issues, typical methods usually focus on extracting robust features or finding a similarity measure. For instance, semantic features from different body regions are captured through a multi-stage ROI pooling pipeline [Zhao et al., 2017]. In [Zhang et al., 2016], a semantics-aware image representation is learned to capture the intrinsic structure information of persons. In [Cheng et al., 2016], an enhanced triplet loss function is proposed to learn a distance measure between two pedestrian images.

However, these approaches are incapable to efficiently discover the intrinsic relevance between image pairs. Fortunately, we find that the intrinsic relevance between image pairs in similar views is different with that in dissimilar views. As shown in Figure 1(a) and (b), the image pairs of the same pedestrian captured from similar views can be connected by some details, such as the patterns of clothing or the carried stuff. As shown in Figure 1(c) and (d), the appearances of pedestrians change dramatically and some details are missed when the image pairs of the same pedestrian are captured from dissimilar views. Even so, the image pairs can also be connected by some view-robust features. Therefore, we can exploit the views as the supplementary information to capture the intrinsic relevance between image pairs.

∗Corresponding author

In this paper, we propose a novel approach called deep view-aware metric learning (DVAML) model, where the image pairs are projected into different feature subspaces according to their view information, which can discover the intrinsic relevance between image pairs from different aspects. Multiple metrics are exploited to supervise the learning of the different feature subspaces. Extensive experiments conducted on three datasets including CUHK01, CUHK03, and PRID2011 show the effectiveness of our method compared with state-of-the-art approaches.

2 RelatedWork

Typical person re-identification methods usually focus on feature extraction or metric learning. Widely-used features include histograms [Li and Wang, 2013; Khamis et al., 2014;

Zhao et al., 2013a], local binary patterns (LBP) [Li and Wang, 2013; Zhao et al., 2013a; Khamis et al., 2014], Gabor features [Li and Wang, 2013] and other cues [Zhang et al., 2014]. However, the changes of appearance lead to instability of these features. The basic idea of metric learning is to find a mapping function from feature space to distance space with certain merits [Cheng et al., 2016], such as Mahalanobis metric learning (KISSME) [Davis et al., 2007], Information-theoretic metric learning (ITML) [Davis et al., 2007], and large margin nearest neighbour (LMNN) [Weinberger and Saul, 2009]. Both lines of the methods regard feature extraction and metric learning processes as two separate steps, which limit the performances significantly.

Inspired by the great success of deep learning networks in computer vision and pattern recognition tasks [Yang et al., 2017b; Deng et al., 2018; Yang et al., 2018; Li et al., 2018; Yang et al., 2017a; Liu et al., 2016; Yang et al., 2017c], many researchers consider the feature and metric learning jointly in an integrated deep architecture, where feature representation can be learned under supervision of the distance metric loss. In [Li et al., 2014], deep learning is exploited to automatically learn features for the re-identification task. And the deep framework has been improved in [Ahmed et al., 2015] by incorporating neighbouring locations of other images. In [Ding et al., 2015], the relative distance between two images are captured by a triplet loss. To further constrain the distances of pairs, [Cheng et al., 2016] propose an enhance triplet loss function. A multi-task deep network (MTDnet) [Chen et al., 2017] are presented to consider the classification loss and the ranking loss simultaneously and takes advantages both of them during training. In order to jointly extract single-image and cross-image feature representations, [Wang et al., 2016] propose a unified triplet and siamese deep architecture.

3 Our Architecture

Fig. 3 shows the framework of the proposed method. The input images of the network are first embedded by the convolution neural network(CNN). Then the output feature maps are respectively projected to different feature subspaces through two full connected layers that do not share parameters. Finally, multiple metric loss functions are used to guide network optimization.

3.1 View-Aware Feature Embedding

Given a training set {xi, y1 i , y2 i }i=1,2,...,n, where xi denotes the ith pedestrian image, y1 i and y2 i are the identity label and the view label of xi respectively. The view label y2 i is set as 0 when the pedestrian is in front or back views, while as 1 when the pedestrian is in side views. Furthermore, we denote the feature maps of input images extracted by CNN as F = {f(xi)}i=1,2,...,n and combine them into four kinds of pairs as shown on Figure 2:

where 1 ≤ i, j ≤ n, the feature maps of image pairs (f(xi), f(xj)) ∈ {PS, NS} are projected to one feature subspace by the full connection layer FC1, while the (f(xk), f(xl)) ∈ {PD, ND} are projected to another one by FC2. The outputs of FC1 and FC2 are defined as follows:

where W1 and W2 are the weights of FC1 and FC2 while b1 and b2 represent the biases. ∗ refers to the convolution operation.

The CNN is built inspired by [Xiao et al., 2016; Simonyan and Zisserman, 2014] and details of the structure are listed in Table 1. Each ReLU layer is followed by a Batch Normalization (BN) layer which accelerates the convergence process and avoids manually tweaking the initialization of weights and biases. For training the CNN from scratch, we randomly dropout 50% neurons of the fc7 1 and fc7 2 layers.

Besides, We learn a view classifier to choose the corresponding feature subspace which the feature maps of images are projected into. The view classifier is optimized by minimizing the sigmoid loss. The first and second layers of the view classifier are convolutional layers with the kernel size 32×7×7 and 32×5×5 respectively. The kernels size of third convolutional layer is 64×3×3 and all the three layers are followed by a pooling layer. Then two fully connected layers that contain 512 neurons receive the output of the third convolutional layer.

3.2 Multiple Metric Loss

Different with other methods, we utilize two metric loss functions to constrain the distances between image pairs in similar views and in dissimilar views separately. Figure 4 illustrates the failure case of other methods that constrain the distances between all image pairs with one loss. In Figure 4(a), the loss decreases the distances between (x1, x2) and between

(x1, x3) by giving x1 a upwards and a downwards pulling force respectively. Simultaneously, x1 is given a left and a right pushing force to increase the distances between (x1, x4) and between (x1, x5). Therefore, x1 will still in place. In Figure 4(b), image pairs in similar views and in dissimilar views are first projected into two different feature subspaces as introduced in 3.1 section. Then L1 and L2 are used to supervise the learning of the different feature subspaces respectively. L1 decreases the distance between (x1, x2) by pulling x1 upwards and increases the distance between (x1, x5) by pushing x1 right, while L2 decreases the distance between (x1, x3) by pulling x1 downwards and increases the distance between (x1, x4) by pushing x1 left. x1 moves finally towards the correct direction in both feature subspaces.

The loss function L1 based on the lifted structured feature embedding[Song et al., 2016] is defined as:

where L1i,j is used to limit the distance between xi and xj in PS less than all pairs in NS which contain xi or xj with a margin β1.

Correspondingly, the loss function L2 is defined as:

where L2i,j is used to limit the distance between xi and xj in PD less than all pairs in ND which contain xi or xj with a margin β2. The value |PS| and |PD| are respectively equal to the number of image pairs belong to PS and PD in each batch.

Noted that the distances between images pairs in similar and dissimilar views are respectively denoted as:

in which S denotes the set of image pairs in similar views including PS and NS, while D denotes the set of image pairs with dissimilar views consisting of PD and ND.

Considering that the loss function L1 and L2 only constrains the distances between image pairs in similar views and dissimilar views separately. So we use L3 constrain the distances between image pairs of same person and between image pairs of different person overall. The loss function L3 is defined as:

in which P denotes the set of the image pairs of same identity consist of PS and PD. N constrains all the image pairs in NS and ND.

We finally unite the three loss functions to a joint loss function:

where the three hyper-parameters α1, α2 and α3 are used to balance the three loss functions.

3.3 Optimization

The network is jointly supervised and optimized by the joint loss. The back propagation gradient with respect to the distance Di,j is:

where 1 is the indicator function which outputs 1 if the expression evaluates to true and outputs 0 otherwise. The gradients of the distances between image pairs in PS consist of two components coming from L1 and L3 , while that of image pairs in PD coming from L2 and L3 .

The gradient with respect to the distance Di,k is:

in which the gradients of the three loss functions with respect to Di,k are:

Algorithm 1 BackPropagation gradient

The gradient with respect to Dj,l is:

The gradients of the distances between image pairs in similar views with respect to the feature maps of them are:

Correspondingly, the gradients of the distances between image pairs in dissimilar views with respect to the feature maps of them are:

4 Experiments

4.1 Datasets and Settings

Datasets: We evaluate our method on three datasets, CUHK03 [Li et al., 2014], CUHK01 [Li et al., 2012] and PRID2011 [Hirzer et al., 2011]. CUHK03 dataset contains 13164 images from 1360 persons. We select 1160 persons for training, 100 for validation and 100 for testing following the same setting as [Li et al., 2014] and [Ahmed et al., 2015]. CUHK01 [Li et al., 2012] dataset contains 971 persons captured from two camera views in a campus. We utilize the same protocol used in [Zhou et al., 2017] and [Wang et al., 2016], where 871 person images are used for training and the left for testing. Following the setting in [Chen et al., 2017], 100 persons in PRID2011 dataset are used for training, specially, 100 for probe and 649 (the remaining persons from camera B except the 100 for training) for gallery in test set.

For training the view classifier, we manually annotate the coarse view labels for 200 persons in CUHK03 and CUHK01. Considering the size and complexity of PRID2011 dataset, 50 persons in it are annotated. Since the view classification is treated as a simple binary classification problem, the view classifier achieves a high accuracy rate even though training samples are limited.

Considering that multiple cameras in varied views may be retrieved for a given query image in practical application, we fuse the three datasets two by two into three new datasets: Syn1 (CUHK03 and CUHK01), Syn2 (CUHK03 and PRID2011) and Syn3 (CUHK01 and PRID2011). Then we train the model and retrieve probe of the one on the two datasets for each fused. For example, the same 1260 persons in CUHK03 and 871 persons in CUHK01 are chosen as Syn1's training set. For the probe of CUHK03, there are another 100 persons from CUHK01 in the CUHK03's gallery besides the original 100. Equally, there are 200 persons in the CUHK01's gallery. Noted that 749 persons are used as the gallery of PRID2011 in Syn2 and Syn3, while only 200 persons as the CUHK01's and CUHK03's gallery.

Parameters Implementation. All the images are resized to 128 ∗ 64 before being fed to the network and the batchsize of the input is 64. For identity classifier, we first pretrain a model on CUHK03 and the learning rate is set to 0.001, then finetune this model with learning rate 0.002. We picked a set of optimal loss weights α1 = 0.4, α2 = 0.4, α3 = 0.2 experimentally. And all the margin parameters β1, β2, β3 are set to 1 [Song et al., 2016].

Evaluation Protocol: We report the single-shot results on the three datasets and the synthetic dataset of them. The datasets are separated into the training set and the testing set, and the testing set is further divided into probe set and gallery set that contain different images of the same person. The result is evaluated by cumulative matching characteristic (CMC) curve, which is an estimation of finding the corrected match in the top n match.

4.2 Results and Analysis

We compare our method with 7 common methods including KISSME, ITML, LMNN, eSDC [Zhao et al., 2013b], kLFDA [Xiong et al., 2014], FPNN [Li et al., 2014], IDLA [Ahmed et al., 2015]. And Table 2-7 show the evaluation results of these method.

Results on CUHK03. In Table 2, we also compare our method with two deep methods GatedSiamese [Varior et al., 2016] and MTDnet [Chen et al., 2017] except the 7 methods. From Table1, most of deep learning methods except FPNN obviously outperform the traditional methods. Ours model has achieved the top performances by 76.11% on Top1 accuracy (vs. 74.68% by the next best method).

Results on CUHK01. Table 3 extra lists the results of the P2S [Zhou et al., 2017] and ImpTrpLoss [Cheng et al., 2016] methods. From Table 3, we can see that our method outperforms the state-of-the-art methods by more than 80% on Top1 accuracy (80.54% vs. 78.50% by MTDnet-cross). Note that the Top 5 recognition rate of our method reaches 95.00%, meaning that the trained model has high probability of finding the correct person from other cameras.

Results on PRID2011. We can see from Table 4 that our method achieves 33% on Top 1 matching rate and outperforms the previous best accuracy on PRID2011 even though the training samples are limited for deep neural networks. It is noted that the Top 5 matching rate achieves 65% far beyond the accuracy of other methods.

Results on Syn1, Syn2 and Syn3. Table 5-7 illustrates the performance of ours method and the previous methods on synthetic datasets. And it's obvious that our method has achieved the top performance on all the three datasets. From Table 5, we can see that the matching rate of our method on CUHK03 and CUHK01 reaches 72.77% and 73.00%. It's worth noting that the accuracy on Syn1 declines contrast to each single dataset due to the gallery images from the other dataset can interfere the retrieval of the probe. The results on Syn2 are listed in Table 6 and our method's accuracy achieves 75.02% and 35.00% on CUHK03 and PRID2011 respectively. The accuracy of PRID2011 when trained on Syn2 dataset improves 2% compared with on single dataset, since the added training samples may improve the performance of model on small dataset. The accuracy of CUHK01 and PRID2011 listed in Table 7 reaches 76.00% and 31.00%. It is noted that we improve the performance on PRID2011 by more than 20% on the rank-5 accuracy in contrast to the previous methods.

Effect of deep view-aware metric. We employ an viewaware method to project image pairs into different feature subspaces. The method has positive effect on person reidentification since it captures the intrinsic relevance between image pairs. Besides, the proposed network is constrained by three metric loss functions, of which the two supervise separately the learning of the different feature subspaces and the last supervises the whole network. And our model wins over 3% compared to other methods on the three synthetic datasets, which further proves the effectiveness of it.

5 Conclusion

In this paper, we propose a deep view-aware metric learning (DVAML) model for person re-identification. We project image pairs in similar and dissimilar views into different feature subspaces, respectively. Then multiple loss functions are used to supervise the learning of the different feature subspaces. In addition, we also consider the actual situation that a given probe image can be retrieved from multiple cameras by synthesizing different datasets. Experiment results on CUHK03, CUHK01, PRID2011 and three synthetic datasets illustrate that our method outperforms the state-of-the-art approaches.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 61572388 and Grant 61703327, and in part by the Key R&D Program. The Key Industry Innovation Chain of Shaanxi under Grant 2017ZDCXL-GY-05-04-02 and Grant 2017ZDCXL-GY-0504-02.