Modality-aware Style Adaptation for RGB-Infrared Person Re-Identification

1 Introduction

Person re-identification (Re-ID) aims to retrieve a target person across disjoint camera views [Ye et al., 2020c; Ye et al., 2020a]. Given probe images of a person-of-interest, Re-ID searches the gallery set to match images with the same identity. However, Re-ID based on RGB cameras usually fails to capture valid appearance information under poor lighting conditions. With the introduction of infrared cameras, the cross-modality RGB-Infrared (RGB-IR) person Re-ID is proposed to leverage the discrepancy of modalities originated from different imaging processes of RGB and IR cameras, where the probe consists of RGB images and the gallery consists of IR images or vice versa.

∗Corresponding author.

One critical challenge for RGB-IR person Re-ID is the large cross-modality gap between RGB and IR images. To reduce the modality gap, existing methods for RGB-IR Re-ID can be generally divided into two types. Some methods work in the feature space by designing some modality-specific and modality-shared embedding networks or classification layers [Dai et al., 2018; Ye et al., 2019b; Feng et al., 2019]. However, these methods usually ignore the large gap existed in image space. The other methods apply GAN-based networks to achieve dual image style conversion while bridging the two modalities in image space [Wang et al., 2019a; Wang et al., 2019b], using high-similarity images exchanged with complex image generation structures, as shown in Fig. 1a.

To alleviate the modality gap, we propose a modalityaware style adaptation (MSA) framework to facilitate the mining of potential relations between RGB and IR modalities by introducing new related modalities. Different from existing methods, MSA enlightens image-to-image communication between RGB and IR modalities and adopts the knowledge to generate new modalities that are compatible with both RGB and IR modalities. As illustrated in Fig. 1b, MSA adapts RGB and IR images to new R2I and I2R styles in parallel under the guidance from the counterpart, filling the large modality gap utilizing connections among the four modalities. Firstly, a compact image generation framework is designed without deep CNN structures since MSA does not require high-similarity generated images. In detail, MSA shuffles knowledge of patches in an original image and reconstructs constrained by two image-level losses, which are calculated by two devised image style quantification metrics from global and patch views. Subsequently, the two original and two generated modalities are fed into a common embedding network and features of the four modalities are constrained by identification loss and triplet loss. Above all, the trainable image generation structure and feature embedding network are optimized simultaneously by two imagelevel losses and two feature-level losses during an end-to-end training. The main contributions of this work are as follows:

• We propose a compact modality-aware style adaptation (MSA) framework, which introduces two new related modalities to help explore modality-invariant features.
• We design two image-level losses and combine them with feature-level losses to form a dual-space (image and feature space) total objective function.
• MSA outperforms the state-of-the-arts on two publicly available datasets. Specially, over 6% improvements of rank-1 and mAP on the RegDB dataset.

2 RelatedWorks

2.1 RGB-Infrared Person Re-identification

For RGB-IR Re-ID, various methods are proposed to reduce the modality gap, and most of them can be divided into two types. The first type contains methods focusing on the feature space. For instance, [Wu et al., 2017] researched three kinds of frameworks and used zero-padding to learn the common space for two different modalities. [Dai et al., 2018] designed a cross-modality generative adversarial network (cmGAN) to reduce the modality gap by learning a highly integrated feature space. [Ye et al., 2019a] designed two separate identity classifiers and a modality-sharable classifier to learn the discriminative information of different modalities. The second type aims at image generation, which reduces the modality gap by converting images from RGB modality to IR style or vice versa, while keeping the identity information. [Wang et al., 2019b] reduced the image-level discrepancy through a dual image transformation. [Wang et al., 2019a] generated fake IR images to learn more discriminative information of identities from different modalities.

Methods focusing on feature space are usually limited by the large image-level gap. However, existing imagelevel methods usually bridge the two modalities through deep CNN-based style conversion with a lot of extra parameters. MSA works in image space and design a compact two-layer image generation structure.

2.2 Image SimilarityMetric

The GAN-based Re-ID methods use the feature-level discriminator to constrain the similarity of images. However, some image-level metrics comparing the images pixel by pixel for evaluation are still effective. For instance, [Wang et al., 2004] proposed a structural similarity (SSIM) to compare two images from the view of luminance, contrast and structure. More generally, the peak signal-to-noise ratio (PSNR), which measures the image similarity by calculating the mean square error (MSE) of two images and the image-level L1 loss are widely used to measure the similarity of two images [Zhang et al., 2019; Liu et al., 2020]. Hardly existing RGB-IR Re-ID methods focus on style conversion in image space, therefore ignore to construct image-level distance metrics for style discrimination. In MSA, we fill this vacancy by designing two style similarity metrics used in the image space and further introduce two image-level losses.

3 The ProposedMethod

MSA achieves style adaptation by reconstructing the source images while siphoning off knowledge from the target style, where the results are compatible with both source and target modalities. Based on that, we denote a triplet of images with form {Xs, Xa, Xt} referring to images from the source, generated, and target modalities during style adaptation. And it can be adapted as {Xrgb, Xr2i, Xir} or {Xir, Xi2r, Xrgb} in MSA. The proposed framework is shown in Fig. 2a. In image space, MSA aims to generate two new modalities through dual style adaptation. In feature space, it focuses more on constructing a high integrated feature space where the crossmodality variations are alleviated.

3.1 Image Style Adaptation

The image style adaptation (ISA) module achieves the image reconstruction stage in style adaptation and designs a structure with concise operations. As shown in Fig. 3, firstly, based on the patch-level receptive field, the convolutional layer combines local pixel information of the input image into one pixel of the intermediate image. Secondly, the deconvolutional layer takes the intermediate image as input and redistributes the pixels with a learnable form in the output image. During training, two ISA modules are applied with the same structure but optimized separately. Above all, the process of ISA is defined as:

where C and D represent the convolution and deconvolution operations separately. Xs and Xa confirm the definition above. What worth noticed is there is no padding operation existed in ISA to avoid the loss of information during deconvolution. Moreover, the kernel sizes of C and D are both set as 3 × 3, with neglected influence on the image identity (the label) due to the small covering range.

3.2 Style Similarity Constraint

The style similarity constraint (SSC) module siphons off knowledge from the target style in style adaptation. As shown in Fig. 2c, SSC consists of an image style quantification (ISQ) block and two style similarity losses (SSL+ and SSL). Specifically, it models the relations among modalities by discriminating the source, generated and target styles in image space (ISQ) and guides the style adaptation by calculating image-level style distances of them (SSL+ and SSL).

Image style quantification. Firstly, we discriminate different styles by quantifying image style. Considering the patchbased ISA module, the image style quantification (ISQ) module is also proposed from a patch view using patch luminance and contrast of an image, which are described by the mean value µ and standard deviation σ [Wang et al., 2004]. We denote an image x ∈ RC×H×W , where C represents the channel dimension, H and W represent the image size. As shown in Fig. 4, an N × N window conforming to the Gaussian distribution is designed to separately calculate the µ and σ within the sliding window in each step. Subsequently, results in each step are concatenated keeping the topological relationship invariant. Based on that, the quantified image style ISx ∈ RC×H×W contains two parallel outputs:

where

here, G represents a Gaussian filter and ⊙ represents the element-wise product of matrixes. Γ(·) is a truncation function which takes the input pixel as center and cuts out a new N ×N image. x2 ∈ RC×H×W is calculated by x ⊙ x. And ⌈⌋ is the concatenation operation. Especially, the ISx is calculated parallel using Eq. 2 in each channel, and that of the whole image is the concatenation of C channels.

Style similarity loss. Secondly, after discriminating different styles using ISQ, two style similarity losses are designed to restrict the distances between the generated style and the source (target) style during style adaptation. As illustrated in Fig. 2c, the style distances between {Xt,Xa} and {Xs,Xa} are calculated differently. For the first pair, LSSL is defined using the MSE loss:

where

and

here, c is the index of channel. Then, for {Xs,Xa}, which are highly similar to each other in terms of content, a more powerful global view constraint may work better [Zhang et al., 2019]. Directly, the style of images is affected by the value of each pixel, thus a reinforced global-view image style IS+ x ∈ RC×H×W of image x is quantified as:

and corresponding LSSL+ is defined as:

where ||·|| represents the l1-norm used to further sharpen the generated Xa compared with MSE [Zhang et al., 2019]. More details about the function of LSSL+ and LSSL are discussed in Section 4.4.

Overall image-level loss. The style adaptation is mainly achieved in image space using the aforementioned modules. To propagate knowledge from both RGB and IR modalities to the new modalities in image space, MSA restricts the style distance of {Xt,Xa} and {Xs,Xa} simultaneously. Firstly, LSSL is used to reduce the style distance between {Xt,Xa} to force Xa to stay more closer to Xt in style. However, the effect of LSSL is limited by the different content information of {Xt,Xa}. Moreover, due to the simple structure of ISA, the generated Xa is actually highly similar to Xs, which puts MSA in a dilemma where the similarity between {Xs,Xa} is too high while that between {Xt,Xa} is not high enough. Accordingly, the more powerful LSSL+ is introduced to enlarge the style distance between {Xs,Xa} on the one hand, and assist LSSL to close {Xt,Xa} on the other hand. As a result, the overall image-level loss is defined as:

3.3 Feature Representation Learning

We design a common embedding network F for the four modalities as shown in Fig. 2a, to ensure that knowledge from one specific style can be broadcast to others through the backpropagation during end-to-end training. And the output embeddings are constrained by two feature-level losses.

3.4 Feature-level Loss

Firstly, identity (ID) classification loss [Liu et al., 2018] is applied by optimizing the cross-entropy loss between labels and predicted possibilities. Secondly, an improved multimodality triplet loss is developed as shown in Fig. 2c. Compared with the conventional one, it ignores the modality attribute of images and only leaves restrictions on the same identity for (a, p) and different identities for (a, n). Moreover, it can handles features from four modalities simultaneously. Above all, the multi-modality triplet loss with hard sample mining is defined as:

where λ is a margin parameter and [z]+ = max(z, 0). D(·) is the Euclidean distance. (a, p, n) = (xima , ximp , xjmn ), where i,j are unequal identities of persons. (ma,mp,mn) are corresponding modalities of the anchor and its hard samples, allowing any combination of modalities.

3.5 Multi-modality Collaborative Learning

The aforementioned modules are integrated by a multimodality collaborative learning strategy and optimized by both image-level and feature-level losses. The overall loss function is formulated as:

Algorithm 1 shows the overall training process of MSA. In conclusion, the trainable ISA is introduced to help search two 'new styles', with which the feature embedding network is optimal. Therefore, end-to-end training is necessary and it enables stronger losses from image and feature spaces.

Algorithm 1 Multi-modality collaborative learning of MSA

Input: RGB images:Xrgb, IR images: Xir

Parameter: α, β, γ

Output: Configurations of MSA

• 1: Initialize the parameters of F, D and C
• 2: for each xrgb, xir do
• 3: Generate xr2i, xi2r using Eq. (1)
• 4: for each triplet of {xrgb, xr2i, xir} do
• 5: Calculate LSSL1 , LSSL+1 using Eq. (4), (8).
• 6: end for
• 7: for each triplet of {xir, xi2r, xrgb} do
• 8: Calculate LSSL2 , LSSL+2 using Eq. (4), (8).
• 9: end for
• 10: Limg = α(LSSL1 + LSSL2 )- β(LSSL+1 + LSSL+2 )
• 11: for each {xrgb, xir, xr2i, xi2r} do
• 12: Calculte F(rgb), F(ir), F(r2i), F(i2r)
• 13: Calculte Lid
• 14: Calculte Lmmtri using Eq. (10).
• 15: end for
• 16: Ltotal = Limg +γLmmtri +Lid
• 17: end for

4 Experiments

4.1 Datasets and Settings

Datasets. (1) SYSU-MM01 [Wu et al., 2017] is a largescale and challenging dataset, containing 491 identities with 30,071 RGB images and 15,792 IR images. Among them, 395 persons with 22,580 RGB images and 11,909 IR images are divided into the training set, and 96 persons are divided into the testing set. During testing, we apply the more challenging single-shot mode. (2) RegDB [Nguyen et al., 2017] contains 412 identities, which has 10 RGB images and 10 thermal images. The dataset is randomly divided into two halves, training and testing set following [Ye et al., 2020c]. During testing, we adopt the Visible2Thermal mode which takes the RGB images as probe and IR images as gallery.

Implementation details. The ResNet-50 [He et al., 2016] with pre-trained parameters on ImageNet [Krizhevsky et al., 2012] is taken as our backbone, where the stride of the last convolutional layer is changed to one. During training, we randomly select six identities with four RGB images and four IR images sampled for each identity. We also adopt the random erasing [Lu et al., 2020] for data augmentation. The Adam optimizer is used to guide the training process in 60 epochs. The λ in Lmmtri is set to 0.5. And the trade-off hyperparameters α, β and γ are set to 1:1:1 and 13:10:7 separately on SYSU-MM01 and RegDB datasets. During testing, only the RGB and IR images are taken as input of the feature embedding network.

4.2 Comparison with State-of-the-Art Methods

The proposed method is compared with the state-of-the-arts on two datasets, including Zero-Padding [Wu et al., 2017], HCML [Ye et al., 2018], cmGAN [Dai et al., 2018], DHSME [Hao et al., 2019], D2RL [Wang et al., 2019b], AlignGAN [Wang et al., 2019a], AT [Ye et al., 2020a], HiCMD [Choi et al., 2020], JSIA [Wang et al., 2020], XIV [Li et al., 2020], DDAG [Ye et al., 2020b], SIM [Jia et al., 2020] and cm-SSFT[Lu et al., 2020]. Table 1 and Table 2 show the results over the RegDB and SYSU-MM01 datasets.

With a compact model, MSA outperforms the SOTA by 10.3% and 6.9% in terms of rank 1 and mAP on the RegDB dataset. On the SYSU-MM01 dataset, we reach the leading level from the above view, but outperform the SOTA in rank 10 and rank 20 accuracies which are not listed in the table, with improvements of 3.76% and 3.08% under all-search mode and 1.69% and 1.51% under indoor-search mode. Moreover, Compared with methods concerning the generation of new images, MSA does not rely on GAN-based components [Wang et al., 2019a; Wang et al., 2019b] and achieves not only channel-wise [Li et al., 2020] but also pixelwise image reconstruction. In addition, we align the settings with AGW [Ye et al., 2020c] and DDAG [Ye et al., 2020b] for a fair comparison as in Table 3. Results show that MSA is still effective without the improvement of backbone, data and network augmentation.

4.3 Ablation Study

Extensive ablation experiments are set to evaluate each component of MSA as shown in Table 4. To ensure the robustness of results, all experiments are performed using two backbones. Settings for each group are: 1) B: Baseline model trained with Lid and Lmmtri of original RGB and IR modalities like [Ye et al., 2020c]. 2) B + ISA: Model trained with Lid and Lmmtri of at least three modalities, which is optimized only by feature-level losses. 3) B + ISA + SSC: Model trained with Ltotal of three modalities. 4) MSA: Model trained with Ltotal of four modalities.

Evaluation of R2I and I2R (ISA module). As shown in Table 4, the baseline does not perform well with the original modalities due to the large modality gap. MSA boosts the performance by introducing two new modalities, hence we evaluate if either of the R2I and I2R modalities works by comparing the 2nd group with the baseline. Results support the effectiveness of both new modalities generated by the designed ISA module and further verify the positive significance of generating compatible modalities with new styles for RGB-IR Re-ID. Besides, more new modalities perform better with stronger promotion to explore potential information.

Evaluation of SSC. We evaluate the designed image-level losses using experiments with the same option of modalities in the last three groups. Results demonstrate SSC compensates for the limitation of ISA we mentioned in Section 3.2 and benefits to exploit more related information in image space. However, another observation is the improvement seems slim with only the R2I or I2R modality. One reasonable analysis is there are fewer parameters (fewer ISA modules) to optimize when only one new modality is generated, which limits the effect of SSC.

Evaluation of SSL and SSL+. In this paper, we propose two different image quantification metrics along with two losses separately for {Xt,Xa} and {Xs,Xa}. To research how the two losses work with different pairs of images, we evaluate LSSL and LSSL+ in detail as shown in Table 5. Actually, performances of different options are almost the same on the RegDB dataset but vary on the larger and more challenging SYSU-MM01 dataset. Results show that both SSL

and SSL+ are valid to guide the style adaptation though SSL+ provides a more powerful constraint. Moreover, SSL fits {Xt, Xa} better due to the content differences between them, which weaken the pixel-based SSL+ designed in a global view as described in Section 3.2. In a word, the patch-view SSL fits images with large content difference and the SSL+ is more suitable for images with the same content information to strengthen the constraint from a global view.

4.4 Discussions

Image-level visualization. In this part, we show images of the genearted R2I and I2R modalities. As shown in Fig. 5, they inherit content information from source images, meanwhile the luminance, contrast as well as the 'tone' of color are changed to a new form. Specially, the I2R images are more colorful than the source IR images, profiting from the knowledge siphoned off from RGB images by the designed SSC module. Moreover, although they look not perfect enough, the improvements in Section 4.2 support our initial motivation: the high-similarity fake images are not necessary.

Feature-level visualization. One key point of MSA is we achieve the style adaptation throughout in image space. To explore how the image-level operations affect the feature space after a deep CNN backbone, we visualize the feature space using t-SNE [Maaten and Hinton, 2008] as shown in Fig. 6. Compared with the baseline, MSA introduces R2I and I2R, and features from initial RGB and IR modalities are clustered mainly based on identity. In Fig. 6b, the generated Xa and its corresponding Xs are almost overlapped because of the simple structure of ISA. In Fig. 6c, Xa can be distinguished from Xs as shown in the dashed boxes and can extract more potential information about Xs. Accordingly, the feature space is redistributed due to the addition of new modalities, during which more potential connections and differences between RGB and IR modalities are explored because of the similarity among the four modalities. As a result, the variations between RGB and IR modalities are effectively reduced in feature space.

Parameters analysis. During training, three key parameters α, β and γ in Ltotal will guide the optimization of the whole network. Fig. 7 shows the performances of MSA when the three parameters are set as different values. Experiments show that MSA performs optimally when the proportions of them are 13:10:7 on the RegDB dataset.

Model analysis. MSA aims to achieve a highly compact framework with no need for high-quality generated images. To this end, we evaluate the processing FLOPs and model parameters of MSA. For a 288 × 144 image, the FLOPs of MSA are 5.1689G, only 0.0072G more than the baseline. Moreover, the extra model parameters are less than 0.0001Mwith a total of 24.32M. Additionally, the model parameters of the backbone of cm-SSFT [Lu et al., 2020] (SOTA) are more than 70M. As a result, MSA achieves 20.30% and 17.25% improvements compared with the baseline concerning rank-1 and mAP on the SYSU-MM01 dataset.

5 Conclusion

A modality-aware style adaptation (MSA) framework is proposed for the RGB-IR Re-ID, which introduces two new modalities to assist the exploration of modality-invariant features. We design a highly compact framework, containing a trainable feature-free image reconstruction structure to generate images, two image-based style quantification metrics and two image-level style distance losses to assist optimization of the whole network. As a result, the generated images significantly boost the performance of RGB-IR Re-ID.

Acknowledgments

This work is supported by National Key R&D Program of China (No.2020AAA0108904), Science and Technology Plan of Shenzhen (No.JCYJ20190808182209321).