Cross-Modality Person Re-Identification with Generative Adversarial Training

1 Introduction

Person re-identification (Re-ID) has received ever-increasing research focus recently, which aims to match pedestrian images across different cameras. The key challenge lies in that different cameras usually suffer from significant changes in different views, human body poses, illumination and backgrounds. To this end, the existing works can be roughtly categorized into either metric learning methods or feature learning methods. The former targets at learning a distance metric to handle the above robust matching problem [Zheng et al., 2011; K¨ostinger et al., 2012; Wang et al., 2013; Li et al., 2013; Xiong et al., 2014; Liao et al., 2015; Paisitkriangkrai et al., 2015; Liao and Li, 2015; Wang et al., 2016; Subramaniam et al., 2016]. The latter is to learn invariant feature directly, upon which efficient L2 or Cosine distance can be applied [Farenzena et al., 2010; Kviatkovsky et al., 2013; Zhao et al., 2013; Yang et al., 2014; Liao et al., 2015; Matsukawa et al., 2016; Ma et al., 2017].

∗Corresponding author: Rongrong Ji (rrji@xmu.edu.cn)

However, very few works have paid attention to the ReID between RGB cameras and infrared cameras, which, in our consideration, is essentially a cross-modality problem and widely encountered in real-world scenarios. In many applications, the surveillance cameras could be heterogeneous, such as near-infrared(IR), thermal and depth cameras. Especially, many new-generation surveillance cameras nowadays support automatic switching between RGB and infrared modalities, which facilitates such cameras to work at night. The key challenges lies in two folds, i.e., the lack of discriminative information to re-identify the same person between RGB and infrared modalities, and the difficulty to learn a robust metric towards for such a large-scale cross-modality retrieval. Very recently, the work in [Wu et al., 2017] introduces a crossmodality RGB-IR dataset named SYSU RGB-IR Re-ID, as shown in Fig.1. The proposed method analysizes three different network structures and uses deep zero padding for evolving domain-specific structure in one-stream network, which is suitable for RGB-IR Re-ID. However, two serious challenging problems in person Re-ID remain open. First, learning a distance metric is difficult, especially in large-scale crossmodality retrieval. The second challenge is the unbalanced data from different modalities which indicates inconsistent distribution and representation of different modalities and heterogeneous gap.

In this paper, we tackle the above two challenges by proposing a novel cross-modality generative adversarial network (termed cmGAN). To handle the issue of insufficient discriminative information, we leverage the cutting-edge generative adversarial training theory to design our own discriminator to learn discriminative feature representation from different modalities. To handle the issue of the unbalanced data from different modalities, we proposed a novel hybrid loss that integrates identification loss and cross-modality triplet loss, which minimize inter-class ambiguity while maximizing cross-modality similarity among instances . In particular, the proposed cmGAN consists of a deep convolutional neural network as generator to generate modality-invariant representation for RGB and IR images in a common subspace, as well as a modality classifier as discriminator that discriminates between different modalities. The generator is supervised and optimized by identification loss and cross-modality triplet loss, by which identification loss can separate interclass embedding while cross-modality triplet loss minimizes the gap among RGB and infrared representations. The discriminator is a modality classifier that tries to discriminate RGB-IR image representations between different modalities. In the proposed framework, the generator and discriminator beat each other as a minimax game to learn discriminative common representation. Through the joint exploitation of the above, heterogeneous data can be directly compared by such a robust distance metric for cross-modality person Re-ID. The main contributions of this paper are three-fold:

• We explore the problem of cross-modality RGB-IR person Re-ID by exploiting the cross-modality representation learning from the perspective of generative adversarial training. To the best of our knowledge, it is the first effort towards GAN based cross-modality person Re-ID.
• We design a loss function for cross-modality generative adversarial network (cmGAN) to learn discriminative common representations. With this loss function, the generator and discriminator in cmGAN beat each other with minimax game to learn cross-modality representations.
• Extensive experiments on challenging RGB-IR reidentification dataset demonstrate the advantages of our proposal over state-of-the-art methods of CMC by 12.17% in terms of mAP by 11.85%.

The rest of paper is organized as follows. Sec.2 outlines related works on person Re-ID, cross modality retrieval and generative adversarial networks. Sec.3 describes the proposed framework for learning discriminative representations for RGB-IR person Re-ID. Experimental results of our approach are demonstrated in Sec.4. Finally we conclude this paper in Sec.5.

2 RelatedWork

2.1 Person Re-ID

There are two fundamental components in Person Re-ID: distance metric learning and feature representation. For the former group of works, metric learning is to formalize the problem as a supervised metric learning where a projection matrix is sought out. Subsequently, metrics like Mahalanobis metric are well exploited. [K¨ostinger et al., 2012] proposed a largescale metric learning from equivalence constraint (KISSME), which essentially models a log likelihood ratio test between two Gaussian distributions. [Ding et al., 2015] proposed a deep image representation based on relative distance comparison for person Re-ID. Another group is to learn discriminative features that can be efficiently combined by existing L2 or Cosine distances. In particular, several deep learning schemes are exploited. [Yi et al., 2014] constructed a Siamese neural network to learn pairwise similarity and used pedestrian body parts to train their CNN model. [Ahmed et al., 2015] proposed a new patch matching layer that compares the activation of two images in neighboring pixels. However, the aforementioned works only address the person Re-ID problem from the same modality. They cannot handle multimodal data with consistent distribution, which cannot bridge the heterogeneous gap.

2.2 Cross Modality Retrieval

Cross-modality retrieval [Peng et al., 2017a]targets at search instances across different modality data, such as searching text in image database related to it semantically. Search between image and document is a representative cross modality retrieval, which has attracted extensive research focus in the past decade. Representative methods include, but not limited to, traditional statistical correlation analysis, DNNbased methods [Zhang et al., 2014], cross-media graph regularization methods, metric learning methods, and dictionary learning methods. The widely-used cross-modality retrieval datasets include Wikipedia, XMedia, NUS-WIDE and Pascal VOC 2007. Among all the methods, the implementation of deep neural network has a significant influence on retrieval performance because of its effectiveness and efficiency. And the recent endeavors also facilitate the learning of cross-modality binary codes, such as [Srivastava and Salakhutdinov, 2014]. A deep semantic hashing with generative adversarial networks is proposed by [Qiu et al., 2017], which explores semi-supervised GAN to generate synthetic training data for hashing. Note that, our scenario is related to the above works, which however differs in person Re-ID. Therefore such works cannot be directly applied.

2.3 Generative Adversarial Network

After first proposed by [Goodfellow et al., 2014], GAN has received ever-increasing research focus in computer vision and artificial intelligence research [Arjovsky et al., 2017; Isola et al., 2017; Ledig et al., 2017]. The GAN consists of a generator and a discriminator, the former is to capture the distribution of real data, while the latter reveals whether a sample is fake or real.

To the best of our knowledge, the existing GAN methods typically handle intra-modality sample generation. As among few pioneering works, [Zhao et al., 2017] propose a deep modal that turns cross-view image hashing into singleview GAN for cross modality image retrieval. [Peng et al., 2017b] proposed a method to effectively correlate large-scale heterogeneous data by GAN. [Wang et al., 2017] presented an adversarial cross-modal retrieval method to seek an effective common subspace based on adversarial learning.

3 The Proposed cmGAN

3.1 ProblemFormulation

Formally speaking, let V be the RGB images and I be the IR images or thermal images. The multi-modal dataset is represented as D = {Dtr, Dte}, where Dtr = {Vtr, Itr} denotes the training data and Dte denotes the testing data , in which Dte = {Vte, Ite}. Here, vj ∈ Rd is the feature vector extracted from the RGB image and ij ∈ Rd is the feature extracted from the IR image, where d is the feature dimension. Assume Dtr contains a set of n training images with the corresponding identity labels as Y = {yi}n1 .

3.2 The ProposedModel

Fig.2 shows the framework of the proposed cmGAN model, which contains generator and discriminator components. Features V and I extracted from RGB and IR modalities respectively are fed into the fully-connected layers to generate a 2,048 dimension feature. Then, the generator is trained with both the identification loss and cross-modality triplet loss (detailed in Generator). Finally, the modality classifier plays a role as a discriminator to discriminate between different modalities (detailed in Discriminator). The model components are introduced in details as below:

Generator We construct the generative model with two kinds of loss functions i.e., the intra-class feature embedding loss and the inter-modality loss. The first loss is to ensure the intra-modal discrimination. In order to project feature representation for each modality in the common subspace, several fully-connected layers are built with the identification loss. The feed-forward network is activated by softmax and added after the global average pooling, which outputs a probability distribution of person identifications. This intra-class feature embedding loss by uses the following probability distributions as:

Here, θide denotes the parameters of the feed-forward network and pj is the probability distribution of each ID. The second loss, i.e. the cross-modality loss is to impose triplet constraints to minimize the gap among features of identical persons from different modalities. Formally speaking, let Di = {va, ip, in} or Di = {ia, vp, vn} be the input feature of different modalities. Such Di forms the i-th triple, where (va, ip) and (ia, vp) are from the same person in different modalities, while in and vn have different identification labels respectively. We then compute the inter-modality loss respectively:

Here, [z]+ = max(z, 0), the threshold ξ is a margin that is enforced between positive and negative pairs. The overall cross-modality triplet loss can be formulated as a combination of ℓtripletV(θV ) and ℓtripletI(θI).

By combining the intra class embedding loss with the intermodality loss, we come up with the final objective function as:

where α and β are hyper-parameters.

Discriminator We construct a modality classifier D with parameters θD to act as discriminator. This modality classifier is to learn to judge whether a learned vector of representation is within the same modality or between different modalities, which acts as an adversary. The modality classifier is consisted of a 3-layer feed-forward neural network, upon which the discriminator (adversarial) loss is defined by the cross-entropy loss of the modality classifier as follows:

where mi is the modality label, and D(·; θD) is the modality probability of each input image.

3.3 The Training Algorithm

The model learning is conducted end-to-end by jointly minimaxing generator and the discriminator losses , which is similar to [Peng et al., 2017b] and [Wang et al., 2017]. Since the generator and the discriminator run as a minimax game that targets at beating each other, the optimization carries out two sub-processes:

Here, γ is a hyper-parameter. The generative model is trained for K steps in each iteration in the training stage. The overall training procedure is shown in Algorithm.1.

4 Experiments

In this section, we conduct extensive experiments to evaluate the efficacy of the proposed method. In the experiments reported blow, we compare our proposed cmGAN method

Algorithm 1 The Learning of the Proposed cmGAN model

Require:

Training dataset {x1,y1,s1},{x2,y2,s2},... (x is raw image, y is Id label, s is domain label), A per-trained CNN model v = f(x; θ1) as generator, A CNN model p = d(v; θ2) as discriminator, parameter: α, β, γ, η, K, ξ.

Repeat until convergence:

• 1: for each i ∈ labels do
• 2: randomly create the pairs:({xa,yi,s},{xb,yi,s})
• 3: end for Calculate the generator loss
• 4: V = f(X; θ1)
• 5: ℓtriplet = ∑a,p,n [D2a,p -D2a,n + ξ]+ (ya = yp, ya ̸= yn, sa ̸= sp, sa ̸= sn)
• 6: ℓide = CrossEntropy Loss(V, Y)
• 7: ℓgen = α ∗ ℓtriplet +β ∗ ℓide Calculate the discriminator loss
• 8: P = d(V; θ2)
• 9: ℓdis = CrossEntropy Loss(P, S) Train the parameters θ1 and θ2
• 10: if e %K ̸= 0 then
• 11: ℓTotal = ℓgen -γ ∗ ℓdis;
• 12: else
• 13: ℓTotal = η ∗ (γ ∗ ℓdis - ℓgen);
• 14: end if
• 15: Update ℓTotal
• 16: return learned representation in common space f(x; θ1)

with the state-of-the-art methods on SYSU RGB-IR Re-ID dataset [Wu et al., 2017] to verify its effectiveness. Then we conduct further analysis to investigate the performance of cmGAN in more details.

4.1 Datasets and Settings

The SYSU RGB-IR Re-ID dataset 1 is the first benchmark for cross-modality (RGB-IR) Re-ID, which is captured by 6 cameras, including two IR cameras and four RGB ones. This dataset contains 491 persons with total 287,628 RGB images and 15,792 IR images from four RGB cameras and two IR cameras. The dataset is separated into the training set and the test set, where images of the same person can only appear in either set. And the training set consists of total 32,451 images including 19,659 RGB images and 12,792 IR images. It is a very challenging dataset due to the great differences between two modalities.

4.2 Evaluation Protocol

Our experiments follow the evaluation protocol in [Wu et al., 2017]. When we evaluate our model, we set RGB images for gallery images, and those IR images are for probe set. There are two modes, all-search mode and indoor-search mode. For all-search mode, RGB cameras 1, 2, 4 and 5 are for gallery set and IR cameras 3 and 6 are for probe set. For indoor-search mode, RGB cameras 1 and 2 (excluding outdoor cameras 4 and 5) are for gallery set and IR cameras 3 and 6 are for probe set which is easier than former. For both modes, single-shot and multi-shot settings are used. we randomly choose one image and ten images of the identity to form the gallery set for single-shot and multi-shot setting in RGB images respectively. Given a probe image from camera 3 or 6, we compute similarities between the probe image and gallery images. It is important to note the probe image and gallery image are from different locations, that is, camera 2 and camera 3 are in the same location, so probe images of camera 3 skip the gallery images of camera 2. After computing similarities, we can get a ranking list according to descending order of similarities.

1 http://isee.sysu.edu.cn/project/RGBIRReID.htm

4.3 Implementation Details

We use NVIDIA GeForce 1080Ti graphics cards for our entire experiments. And we extract 2,048d features for RGB and IR images from the global average pooling layer in ResNet-50. Three fully-connected layers are used in the modality classifier. Furthermore, We stick to the two fullyconnected layers for ID predictions.

The batch size is set to 20. Empirically after testing several groups of parameter combinations, the generative model training step K is set to be 5. The adaptive parameter γ controls the weight of the discriminator loss and is fixed to be 0.05. We set margin ξ in cross-modality triplet loss in the range [0.7, 0.9, 1.2, 1.4, 1.5, 1.6] and conduct experiments by fixing other hyper-parameters to evaluate our model. The impact of the ξ is shown in Fig.3. We observe that our method achieves better performance when the margin ξ raising. The best performance reported result of our method are obtained when ξ is set to be 1.4.

Considering the different convergence rate of generator and discriminator, the learning rate (lr) of them are set being different respectively. We set the learning rate of generator range in [0.0001, 0.001, 0.01] and discriminator learning rate also range in [0.0001,0.001,0.01]. After testing several groups of parameter combinations empirically, We get the best result when generator lr is set to be 0.0001 with discriminator lr is set to be 0.001 respectively. Therefore, we set the training epoch to 2000 and more than it.

Empirically, we set α and β by 1:1 proportion because we consider cross-modality triplet loss has an equal effect on the retrieval results as well as identification loss.

4.4 Quantitative Evaluations

We shows the comparative results of cmGAN against 12 existing methods [Dalal and Triggs, 2005; Liao et al., 2015; Wu et al., 2017] on SYSU RGB-IR Re-ID in Tab.1, including the rank-1, 10, 20 accuracies of Cumulative Match Characteristic curve (CMC) and mean average precision(mAP).

From Tab.1, it is clear that our cmGAN method significantly outperforms all existing methods in all categories, which is in terms of all search and indoor search mode or one-shot and multi-shot settings. Specifically, the proposed cmGAN method outperforms the 2nd best method of [Wu et al., 2017] by using one-stream network (deep zero-padding) on all search and indoor search tasks in terms of the rank1 metric by 12.17% (26.97-14.80) and 11.05% (31.63-20.58) under one-shot respectively.

In Tab.1, the results of three rows on the bottom show the performance of cmGAN and its two variations. It is shown that both the cross-modality triplet loss constraint and identification loss terms contribute to the final retrieval results, which demonstrates that to optimize the proposed model with two loss function together is better than only one of them. We can also find that the triplet loss constraint contributes more to the performance than the identification term. That means the identification loss do not include cross-modality information, so the cross-modality triplet loss can do a better job in projecting two different modalities features into the same feature subspace.

4.5 Further Analysis on cmGAN

In the proposed cmGAN, we target at optimizing the proposed method with the generative loss and discriminative loss in adversarial training process. We investigate the values of the generative loss and the discriminative loss from epoch 1 to 2,500, as shown in Fig.4. The figure demonstrates that the generative loss decreases almost monotonously and converges smoothly, when the discriminative loss changes suddenly at first and then stabilizes at about 0.7.

The results in Fig.5 demonstrate that the mAP score keeps increasing and holds until the generative loss converges. We can see the loss of cmGAN with the discriminator decreases more slowly than without the discriminator. It means that the generator and the discriminator form an adversary training process to project the feature representations from different modalities into common subspace. We set a small learning rate in order to achieve better performance with the triplet loss convergence. When the triplet loss is lower than 0.1, we consider that triplet loss has converged. Correspondingly, we increase the learning rate to speed up the convergence of the identification loss. Depending on the discriminator, the cmGAN has a more smooth convergence process. The results in Fig.5 also validate that the discriminative loss affects the cmGAN as a directional guide during the feature embedding into the cross-modality common subspace.

As the results shown in Fig.6, we randomly select some IR images as queries to search from RGB images. The images in the first column are the query images. The retrieved images are sorted from left to right according to descending order of the similarity score. The first two rows are results under allsearch mode and the last row are under indoor-search mode. The correct matches are in the green rectangles, and the false matching images are in the red rectangles. We can see our method gets great shots under most situations.

Figure 6: Some examples of retrieval results on SYSU RGB-IR ReID using the proposed method.

5 Conclusions

In this paper, we propose a novel cross-modality generative adversarial network (termed cmGAN) to learn discriminative common representations for RGB-IR person re-identification problem, which is formulated as cross-modality problem. In our approach, the cross-modality generative adversarial networks consist of a deep convolutional neural network as generator for learning image representations and a modality classifier as discriminator which tries to discriminate between RGB and infrared image modalities. We use identification loss and cross-modality triplet loss together for generator in our cmGAN to handle the large-scale cross-modality metric learning problem. Specifically, The identification loss can separate inter-class embedding meanwhile the cross-modality triplet constraints minimize the gap among the representation from different modality. Both generator and discriminator beat each other with a minimax game and the discriminative common representation is generated by the adversarial learning process for person re-identification. Comprehensive experimental results on the challenging cross-modality person re-identification dataset, SYSU RGB-IR Re-ID, have demonstrated our approach outperforms state-of-the-art methods.

Acknowledgments

This work is supported by the National Key R&D Program (No.2017YFC0113000, and No.2016YFB1001503), Nature Science Foundation of China (No.U1705262, No.61772443, and No.61572410), Post Doctoral Innovative Talent Support Program under Grant BX201600094, China Post-Doctoral Science Foundation under Grant 2017M612134, Scientific Research Project of National Language Committee of China (Grant No. YB135-49), and Nature Science Foundation of Fujian Province, China (No. 2017J01125 and No. 2018J01106).