Adversarial Feature Disentanglement for Long-Term Person Re-identification

1 Introduction

Person re-identification (Re-ID), the process of matching person images across different camera views, is widely used in person search [Shi et al., 2020] and multi-object tracking [Ke et al., 2019]. In recent years, many person Re-ID methods have made great success in short-term scenarios. These methods often rely heavily on the appearance of the people, which makes them cannot be easily applied to address the clotheschanging problem in long-term scenarios. From Fig. 1, we find that the clothing features that are effective in short-term task become ineffective in long-term task, while identity features are discriminative to tackle the clothes-changing problem. Based on this observation, we aim to find how to split the identity and clothing information of person images and make them both independent and representative, so that our method can be adapted to both short and long-term scenarios.

∗Contact Author

Recently, great efforts have been devoted to extracting the robust identity representations of the human body for longterm person Re-ID task. Some works [Yang et al., 2020; Wan et al., 2020] proposed to use contour sketch or facial information as identity-related features for person reidentification. These methods only extract a certain type of explicit representation, but cannot get the complete identityrelated information of the human body. Other works [Li et al., 2020] proposed to use generated adversarial learning strategy to learn implicit shape-based features. These methods only pay attention to the adversarial learning between intra-class samples but ignore the important inter-class information.

Considering the above problems, we propose an end-toend adversarial feature disentanglement network (AFD-Net) to disentangle identity-related and identity-unrelated (clothing) features. The AFD-Net combines intra-class reconstruction and inter-class adversary to make the two disentangled features independent and representative to adapt to different scenarios. At the same time, the inter-class network can recombine the identity and clothing information across different identities to obtain new adversarial clothes-changing images. Since there is no ground truth for generated images, a re-feeding strategy is designed to supervise the adversarial image generation. To be specific, the generated images are fed back to the AFD-Net to re-disentangle and reconstruct for image-level self-supervision in the original image space and feature-level soft-supervision in the disentangled feature space. Moreover, we collect two long-term datasets for evaluation. The Market-Clothes dataset is a generated challenging dataset in which each identity has 2 to 71 sets of clothes. The PKU-Market-Reid dataset is a collected real-world dataset, which has complex background and occlusion in a supermarket. It is worth mentioning that our method does not need to be trained on the clothes-changing dataset like many longtermmethods. It can be used for long or short-term tasks only by training on short-term datasets such as Market-1501.

The main contributions can be summarized as follows:

• We propose an adversarial feature disentanglement network (AFD-Net) which contains intra-class reconstruction and inter-class adversary to disentangle the identityrelated and identity-unrelated (clothing) features. The inter-class network can recombine two features across identities to obtain adversarial clothes-changing images.
• To supervise the adversarial image generation, a refeeding strategy is designed to re-disentangle and reconstruct new generated images for image-level selfsupervision in the original image space and feature-level soft-supervision in the disentangled feature space.
• Two challenging datasets (Market-Clothes and PKUMarket-Reid) are collected for evaluation. The performance on one short-term dataset (Market-1501) and five long-term datasets (PRCC, Celeb-reID, Celeb-reIDlight, Market-Clothes, PKU-Market-Reid) confirm the effectiveness and generalizability of our method.

2 RelatedWork

Short-term person Re-ID. Short-term person Re-ID task is carried out based on the assumption that people do not change clothes. Most short-term methods focus on tackling the challenges of matching images under posture, viewpoint, background, illumination, or occlusion variations. We classify these methods into six categories: representation learning based methods [Geng et al., 2016], metric learning based methods [Varior et al., 2016], global features based methods [Zheng et al., 2017], local features based methods [Sun et al., 2018], video-based methods [Liu et al., 2018], and GAN-based methods [Zheng et al., 2019]. These methods can achieve high performance on many short-term datasets, but due to their appearance dependencies, they cannot be easily applied to address the clothes-changing problem.

Long-term person Re-ID. People may change their clothes in long-term person Re-ID task, which makes longterm task face a huge challenge of appearance variations. Most of the current long-term person Re-ID methods aim at finding human representations that are independent of appearance information. These representations include global features fusing facial information [Wan et al., 2020], clothing features fusing biometric information [Yu et al., 2020], local features combining with attention mechanisms [Huang et al., 2020], and adversarial learning features [Li et al., 2020]. The first three representations belong to explicit feature learning, and the last one belongs to implicit feature learning. We believe that explicit learning can obtain exact body representations, while implicit learning is more likely to obtain discriminative features autonomously. Since the identity information in the long-term person Re-ID is critical and ambiguous, we aim to use the implicit learning method. [Zheng et al., 2019] proposed a joint learning method to couple Re-ID learning and data generation end-to-end. They think that the appearance information indicates the identity of the person, which is effective for short-term task. However, in our long-term task, it is more reasonable to regard identity as information independent of appearance. [Li et al., 2020] proposed a novel representation learning method that can generate a shape-based feature representation that is invariant to clothing. They only pay attention to the adversarial learning between intra-class samples but ignore the inter-class case. In our method, we simultaneously obtain intra-class and inter-class information to enhance the independence of identity and clothing features.

Long-term datasets. Due to the difficulty of collection, there are fewer long-term datasets. Huang et al. [Huang et al., 2020] built two large-scale long-term datasets called "CelebreID-light" and "Celeb-reID", which were acquired from the Internet using street snap-shots of celebrities. Yu et al. [Yu et al., 2020] constructed a novel large-scale Re-ID benchmark named clothes changing person set (COCAS), which combined a new Re-ID setting with clothes template. Yang et al. [Yang et al., 2020] and Qian et al. [Qian et al., 2020] proposed two real-world long-term datasets PRCC and LTCC respectively. Wan et al. [Wan et al., 2020] and Li et al. [Li et al., 2020] used generation or rendering methods to get two "fake" clothes-changing datasets. We choose the three largest datasets mentioned above to evaluate our method. And taking into account the high efficiency of the generation method and the practicability of the real scene, we also collect two longterm datasets: Market-Clothes and PKU-Market-Reid. The comparison of various long-term datasets is shown in Table 1.

3 ProposedMethod

In this section, the proposed adversarial feature disentanglement network (AFD-Net) is illustrated in Fig. 2. The inputs of the AFD-Net are four images including two pairs from two identities. The intra-class image pair is disentangled and reconstructed into original images to reduce intra-class feature variations. The inter-class image pair is disentangled and recombined to generate new adversarial clothes-changing images. Moreover, a re-feeding strategy is designed to supervise the adversarial image generation by re-disentangling and reconstructing these new images, which contains image-level self-supervision in the original image space and feature-level soft-supervision in the disentangled feature space.

3.1 Intra-Class Reconstruction

The goal of this part is to encourage the identity and clothing encoders to pull their codes of the same identity together so that intra-class feature variations are reduced. Specifically, the encoders Ei, Ec are used to separately extract identity and clothing features s, f of two different images x, x′ from the same person. Then the generator G combines these two features to reconstruct the original image x that provides identity features before. The image-level reconstruction loss Limg recon for xa, xb is defined as:

where || · ||1 denotes the L1 norm. L1 norm is widely adopted in reconstruction loss since it can preserve image sharpness.

In the meantime, to utilize labeled information of training data for person re-ID, we employ classification loss on the identity and clothing features of person images. That can force these two features of different identities to stay apart. The classification loss for identity and clothing features are:

where p(ˆ ya|xsa ) and p(ˆ ya|xf ′a ) are the predicted probabilities that xa and x′a belong to the ground-truth class ˆ ya separately based on their identity and clothing codes. The probabilities p(ˆ yb|xsb ) and p(ˆ yb|xf ′b ) have the same definition.

3.2 Inter-Class Adversary

After the intra-class feature variations are reduced, the interclass adversary is carried out across different identities, which can force the identity and clothing encoders to learn their respective relevant information. To be more precise, first, two encoders Ei, Ec are used to separately extract identity and clothing features of two different identities. Second, the identity and clothing features are recombined to generate new adversarial clothes-changing images, like xab = G(sa, fb) and xba = G(sb, fa). Since there are no clothes-changing images in the same pose as the original images in the dataset, we cannot get the ground truth of the generated images. It means that the adversarial image generation has no pixel-level groundtruth supervision, so we introduce a re-feeding strategy for supervision later. To make features representational as intraclass, the classification loss is also employed on clothing features of xa and xb:

where p(ˆ ya|xfa ), p(ˆ yb|xfb ) are the predicted probabilities that xa, xb belong to the class ˆ ya, ˆ yb based on clothing code. The total classification loss of raw features Lraw id is:

To further enforce G to perform content recovery, we produce perceptually realistic outputs by using the discriminator D to discriminate the real images xa, xb with the generated images xab, xba. The adversarial loss Lab adv , Lba adv are:

The total adversarial loss Ladv is:

3.3 Image Re-feeding

Since there is no ground truth for clothes-changing images, a re-feeding strategy is designed to supervise the adversarial image generation. The re-feeding strategy contains imagelevel self-supervision in the original image space and featurelevel soft-supervision in the disentangled feature space. For image-level self-supervision, the generated images are fed back to AFD-Net to re-disentangle and reconstruct into the original images. Like xab, xba corresponding to xa, xb, we can get xaba, xbab from xab, xba, which is formulated as:

where the features sab rec , f ba rec derive from xa, and the features sba rec , f ab rec derive from xb. We find that the above-mentioned reconstructed images xaba, xbab are exactly the original images xa, xb. Thus, we can achieve self-supervision by supervising the original images. The image-level reconstruction loss Limg re-recon in re-feeding part can be expressed as:

We add Limg re-recon and Limg recon calculated before to obtain the total image-level reconstruction loss Limg rec :

For feature-level soft-supervision, each generated image is softly supervised by two corresponding original images in the disentangled feature space. For example, the identity and clothing features of xab are supervised by xa and xb respectively. The feature-level reconstruction loss Lsrec , Lfrec are:

The total feature-level reconstruction loss Lfea rec is:

Market-Clothes

The classification loss is also employed on srec and frec. Since the two reconstructed images xab and xba are generated from two images with different ID, how to determine their labels is an important problem. We introduce a soft label strategy to tackle this problem, and prove its effectiveness in ablation experiments. For example, because the image xab is generated by features sa and fb, we set the ground truth of sab rec as ya, of f ab rec as yb to be consistent with sa and fb. This can further force the two encoders to output mutually independent features. The classification loss Lres id , Lref id are:

where p(ˆ ya|xsab ) and p(ˆ yb|xfab ) are the predicted probabilities that xab belongs to the ground-truth class ˆ ya and ˆ yb separately based on its identity and clothing codes. The probabilities p(ˆ yb|xsba ) and p(ˆ ya|xfba ) have the same definition. The total classification loss of reconstructed features Lrec id is:

In summary, through the image reconstruction of xaba and xbab in image-level self-supervision, the feature reconstruction and classification of srec and frec in feature-level softsupervision, the adversarial image generation is supervised and new reliable clothes-changing images are generated.

3.4 Optimization

We train the AFD-Net end-to-end, and the sum of the abovementioned losses in the entire network is shown as:

Following [Huang et al., 2018], we use a large weight λimg rec = 5 for the image reconstruction loss. The feature reconstruction loss is inaccurate in the initial stage, so we gradually increase λfea rec from 0 to 1 during the training process. Similarly, the identification loss Lrec id may make the training unstable due to the low quality of adversarial generated images at the beginning, so we set a small weight λrec id = 0.5. The remaining weights λraw id and λadv are set to 1.

4 Experiments

4.1 Datasets

We evaluate our method on six person Re-ID datasets, including three large-scale long-term datasets: Celeb-reID, Celeb-reID-light, PRCC, one benchmark short-term dataset: Market-1501, and two collected long-term datasets: MarketClothes, PKU-Market-Reid which are shown in Fig. 3.

Celeb-reID / Celeb-reID-light. Celeb-reID [Huang et al., 2020] is composed of 34,186 images of 1,052 identities that crawled from websites. It has 20,208 images from 632 identities for training and 13,978 images from 420 identities for testing. Celeb-reID-light is a light version of Celeb-reID, which has 490 and 100 identities for training and testing.

PRCC. PRCC [Yang et al., 2020] consists of 33,698 images from 221 identities. Each person in Cameras A and B is wearing the same clothes, and in A and C is wearing different clothes. The dataset has 150 and 71 identities for training and testing, and also provides contour sketch images for use.

Market-1501. Market-1501 [Zheng et al., 2015] is composed of 32,668 images of 1,501 identities collected from 6 camera views. The dataset is split into two non-overlapping fixed parts: 12,936 images from 751 identities for training and 19,732 images from 750 identities for testing.

Market-Clothes. Market-Clothes is our generated longterm dataset which is shown in Fig. 3. We change the clothes of person in Market-1501. It consists of 16,483 images from 751 identities, and each identity has 2 to 71 sets of clothes. It is a challenging dataset only used for the test phase.

PKU-Market-Reid. PKU-Market-Reid is our collected dataset which is shown in Fig. 3. It contains 6,028 images of 25 identities collected from 4 camera views, and each identity has 3 to 10 sets of clothes. This dataset contains different clothes, pose, illumination, shielding, and other complex conditions in supermarket. It is only used for the test phase too.

4.2 Implementation Details

We implement the AFD-Net using PyTorch with only one NVIDIA GTX 1080Ti GPU. All images are resized to 256 × 128 for input. The SGD optimizer is used to train Ei, Ec with lr = 0.002, momentum = 0.9. Encoders of the same type in AFD-Net share parameters. The Adam optimizer is applied to optimize G, D with lr = 0.0001, (β1, β2) = (0, 0.999). We use ResNet50 [He et al., 2016] pre-trained on ImageNet as our backbone of Ei and Ec, and convert the input of Ei to grayscale for color filtering. In the test phase, Ei outputs the identity code in batch × 2048, Ec outputs the clothing code in batch × 1024. We use the concatenation of two features for long-term task, and clothing features for short-term task.

4.3 Comparison with State-of-the-Arts

In this part, the common methods and long-term methods are split into two big rows, and the symbol R is short for Rank.

Celeb-reID / Celeb-reID-light. Experimental results on two high-resolution long-term datasets are shown in Table 2. Our AFD-Net outperforms all the compared methods on Rank-1 and Rank-5 but underperforms the mAP of method MGN. The reason is that MGN utilizes multi-granularity features for Re-ID, which not only increases the mAP but also increases the complexity. However, our method can achieve comparable mAP by only using two disentangled features.

PRCC. As shown in Table 3, the Rank-1 / mAP of AFDNet achieves 42.8% / 89.0% in clothes-changing task, and

95.7% / 99.8% in clothes-unchanging task, which outperforms all the other methods by a large margin. This is because our approach can disentangle identity and clothing features, make themmore independent to adapt to different scenarios.

Market-1501. To further prove that our method can also guarantee high performance in short-term task, we also evaluate our method on a large-scale short-term person Re-ID dataset: Market-1501. As shown in Table 4, our approach achieves comparable performance with short-term methods and far exceeds other long-term methods on Market-1501, which indicates the good generalization of our method.

Market-Clothes / PKU-Market-Reid. We use the model trained on the Market-1501 dataset to test on our two collected datasets. The experimental results are shown in the Table 5. On the challenging Market-Clothes dataset, other methods suffer significant performance declines, while ours remains stable and superior. On the real PKU-Market-Reid dataset, our method can still adapt to the complexity of realworld scenarios and achieve the best performance. These results can prove the domain adaptability of our method. The identity features clustering scatter plots are shown in Fig. 4.

4.4 Ablation Study

To investigate the effect of each component in our AFD-Net, we perform ablation studies on the Market-Clothes dataset.

Identity vs. clothing features. The best performance appears when identity and clothing features are concatenated as shown in Table 6. To further show the independence of the two features, the feature visualization is performed on the Market-1501 dataset in Fig. 5. It shows that our AFD-Net can effectively disentangle the local clothing information and the global context-aware identity information of human body.

Intra-class vs. inter-class networks. We also compare the intra-class reconstruction and inter-class adversary networks as shown in Table 7. It shows that the "intra+inter" surpasses "intra" and "inter" by a large margin, which indicates that the inter-class information is more important than the intra-class, and training the two networks end-to-end is effective.

Effect of re-feeding strategy. To evaluate the impact of the re-feeding strategy, we conduct an ablation study as shown in Table 8. Compared to "Non-Refeed", "Refeed" surpasses it by +7.34% Rank-1 accuracy and +2.51% mAP. It proves that the re-feeding part can effectively supervise the adversarial image generation, and further disentangle the identity and clothing features to make them more independent.

Effect of soft label strategy. We investigate how to set the supervised labels for disentangled features of generated images. As xab for example, the comparisons are shown in Table 9. "LaLa" and "LbLb" denote that both features are set to the same label, whereas "LbLa" and "LaLb" denote different labels. Our soft label strategy "LaLb" gets the excellent performance. This is because sab rec and f ab rec are derive from xa and xb, so setting their labels to ya and yb is effective.

5 Conclusion

In this paper, we propose an adversarial feature disentanglement network (AFD-Net), intending to address the clotheschanging problem in long-term person Re-ID. First, the AFDNet combines the intra-class reconstruction and inter-class adversary to disentangle the identity and clothing features, which can make these features more independent and discriminative. Second, AFD-Net can recombine the identity and clothing features across different identities to generate new reliable clothes-changing images for further disentangling. Third, a re-feeding strategy can supervise the adversarial image generation by image-level self-supervision in the original image space and feature-level soft-supervision in the disentangled feature space. Finally, to facilitate the research about the clothes-changing problem, we collect two longterm Re-ID datasets: Market-Clothes and PKU-Market-Reid.

Acknowledgments

This work is supported by National Natural Science Foundation of China (No.62073004), Science and Technology Plan of Shenzhen (No.JCYJ20190808182209321).