Beyond Intra-modality: A Survey of Heterogeneous Person Re-identification

1 Introduction

Person re-identification (ReID) is a human-centric AI technology that finds person of interest in a large amount of videos. It facilitates various applications that require painful and boring video watching, including searching for video shots related to an actor of interest from TV series, finding a lost child in a shopping mall from camera videos, and reidentifying suspects in video surveillance systems. Its efficiency and effectiveness accelerate the process of video analysis. In recent years, we have made significant advances in general ReID. The performance is remarkably high on some public datasets [Luo et al., 2019; Wang et al., 2019b; Wang et al., 2019a; Zheng et al., 2019; Zhou et al., 2019], for example, 96.1% Rank-1 accuracy on the Market-1501

†Equal contribution

dataset [Zheng et al., 2015a], which even outperforms human [Zhang et al., 2017]. However, the general ReID assumes an ideal scenario for brevity where all person images are captured in the daytime with the visible spectrum, and have sufficient details to represent a person. Considering all these images are in the desired modality, we name person ReID under this scenario Homogeneous Person Reidentification (Homo-ReID). It just counts the challenges of intra-modality discrepancies, such as pose and viewpoint changes [Zheng et al., 2019; Zhang et al., 2019], background and illumination variations [Zeng et al., 2020], and occlusions [Zheng et al., 2015b; Huang et al., 2015].

In real-world applications, however, it is impractical to assume the data source is always in such a desired modality. For example, to find a criminal, the system have to check person images in low-resolution (LR), or captured by infrared (IR) cameras when the illumination condition is not sufficient. Moreover, witnesses' descriptions (in terms of text) and sketches drawn by artists shall also be used as the cues. These scenarios face challenges beyond the intra-modality discrepancy investigated in Homo-ReID as shown in Figure 1. Algorithms developed under the desired modality lose their effectiveness when applied to these scenarios.

To improve the practical capability of the ReID system, we should make more efforts to bridge the gap between different camera specifications and settings (e.g., low- vs. highresolution data), different sensory devices (e.g., infrared vs. visible light devices), and reproduction of human memory and direct recording by a camera (e.g., sketch/text description vs. digital images). We define this practical person retrieval Heterogeneous Person Re-identification (Hetero-ReID).

Encouragingly, in the past few years, we have already seen quite a number of remarkable signs of progress on HeteroReID, which are primarily assisted by a growing variety of Hetero-ReID benchmark datasets allowing direct comparison of different methodologies. While Homo-ReID attracts much attention in the research community, we feel that it is the right time to call for more attention and research efforts to the important and still largely unexplored area of Hetero-ReID by providing a quality survey. This paper offers a comprehensive and up-to-date review of the diverse and growing array of Hetero-ReID techniques.

The differences between this survey and the previous ones. To our knowledge, there are few reviews in the ReID field. [Nambiar et al., 2019] explored the ReID applications built on gait sequences, which is a special and different focus. [Vezzani et al., 2013; Zheng et al., 2016; Gou et al., 2018] focused on Homo-ReID. [Vezzani et al., 2013] made a multidimensional overview of Homo-ReID. [Zheng et al., 2016] described critical future directions and briefed some important yet under-developed issues. [Gou et al., 2018] conducted a systematic evaluation with different features and metrics. These works well position the current progress of Homo-ReID. However, they have no investigation on Hetero-ReID and confine to address the intra-modality discrepancy. [Leng et al., 2019; Ye et al., 2020] started to look into some inter-modality challenges, but provided a limited summary of current efforts or problems present in HeteroReID. Note that in other areas, face recognition [Ouyang et al., 2016] and image retrieval [Wang et al., 2016a] also raise the urgent requirements of addressing the inter-modality discrepancy. This survey seeks to provide a summary of current research from different perspectives.

Goals of this survey. We aim to (i) present the idea of Hetero-ReID and raise awareness of this problem; (ii) thoroughly review the literature on Hetero-ReID and provide a panorama for researchers in other fields, with which they can quickly understand and step into the new area; (iii) give comprehensive guidance of future directions to peers in our community.

Contributions. (i) We highlight the difference between Hetero-ReID and Homo-ReID, and conduct a systematic review of Hetero-ReID beyond intra-modality, where the intermodality discrepancy is the main challenge. (ii) We categorize related works into different modalities on which they operate and summarize the state-of-the-arts. As Figure 1 shows, we consider four cross-modality application scenarios: Lowresolution (LR), Infrared (IR), Sketch, and Text. (iii) We analyze representative methods and identify future directions in this research field to share the vision and expand the horizons of Hetero-ReID.

2 Person Re-identification

2.1 ReIDDiagram

ReID system always consists of a feature extraction part and a descriptor matching part. At first, feature vectors of a gallery G = {fg1 , fg2 , ..., fgN } are extracted by the pre-trained feature extraction part. When we input a probe into the system, the probe's feature vector fp is obtained by the feature extraction part. After that, the feature vector fp matches against feature vectors G by the descriptor matching part. For Homo-ReID, both probe samples and gallery samples are from the desired modality (daytime, visible spectrum, and high-resolution images). While for Hetero-ReID, the probe sample or gallery samples are from another modality (see Figure 2).

2.2 Hetero-ReID vs. Homo-ReID

Homo-ReID only faces the challenges of persons' appearance changes, and person images are captured in the desired modality with only intra-modality discrepancy. Following the diagram of ReID, researches tried to design robust and discriminative features or learn effective distance metrics. For more details, the readers can refer to HomoReID surveys [Gou et al., 2018]. Hetero-ReID should face more challenges beyond intra-modality discrepancies, such as HR-to-LR discrepancy, RGB-to-IR discrepancy, Photo-toSketch discrepancy, and Image-to-Text discrepancy. These extremely large discrepancies are generated from different modalities. Hence, the approaches of Hetero-ReID should pay more attention to remove the inter-modality discrepancy. Note that we don't attribute the RGB-depth based ReID to our Hetero-ReID, although there are a couple of researches investigated this kind of task [Barbosa et al., 2012; Mogelmose et al., 2013; Haque et al., 2016; Wu et al., 2017a; Munaro et al., 2014]. In realistic applications, the depth channel works as a complement for the RGB channels, and a depth image of a person will not be used to retrieval in an RGB image gallery.

We make a comparison between Homo-ReID and HeteroReID, as Table 1 shows. For the media type, Homo-ReID only exploits desired RGB images, while Hetero-ReID takes additional LR/IR/Sketch images or Text description into account. For the participant, Homo-ReID only employs the resources from machine intelligence, while Hetero-ReID also brings in the input from human intelligence, such as sketch images drawn by painters or text descriptions of the suspect by witnesses. For the main challenge, Homo-ReID only needs to deal with the intra-modality discrepancy, such as viewpoint variations, image misaligned or occlusion and illumination changes, while Hetero-ReID should bridge the gaps deriving from both intra- and inter-modality discrepancies. Since additional inter-modality discrepancies are larger than intra-modality discrepancies, Hetero-ReID is more challenging. As intra- and inter-modality discrepancies are essentially different, the methods designed for Homo-ReID cannot be directly used in Hetero-ReID.

There is also a big performance gap between Homo-ReID and Hetero-ReID. The state-of-the-art performances (CMC1) are reported in Table 1. They are respectively CAR [Zhou et al., 2019] on Market-1501 [Zheng et al., 2015a] for HomoReID, RAIN [Chen et al., 2019] on MLR-VIPER [Jiao et al., 2018] for Hetero-ReID LR, D2RL [Wang et al., 2019c] on SYSU-MM01 [Wu et al., 2017b] for Hetero-ReID IR, CDAFL [Pang et al., 2018] on PKU-Sketch [Pang et al., 2018] for Hetero-ReID Sketch, and A-GANet [Liu et al., 2019] on CUHK-PEDES [Li et al., 2017b] for Hetero-ReID Text. The performances of Hetero-ReID are still not so satisfied as that of Homo-ReID. However, compared with HomoReID, there are not so many publications related to HeteroReID. Considering the realistic value and research significance, we feel that it is the right time to call for more attention and research efforts to the area of Hetero-ReID.

3 Available Datasets and EvaluationMetrics

3.1 Datasets

We summarize available datasets for Hetero-ReID in Table 2, including the application scenario, the total number of cameras, identities, and samples. To make a comparison, we also list two typical Homo-ReID datasets, including Market1501 and MSMT17. The Hetero-ReID datasets are categorized into four types, and we also take notice of the data construction manner, i.e., simulated or really collected. Figure 1 shows some samples for typical datasets. We have the following summaries: (i) Although many Hetero-ReID LR datasets were constructed, most of them are simulated w.r.t.

from Homo-ReID datasets. (ii) Only two Hetero-ReID IR datasets were released. SYSU-MM01 is an active IR dataset, while RegDB is a passive one. Only one Hetero-ReID Sketch dataset and one Hetero-ReID Text dataset were constructed. (iii) Compared with Homo-ReID, there is still a lot of room to construct more available datasets. For Hetero-ReID LR, it requires to construct practical datasets instead of the simulated ones. For the other three kinds of application scenarios, datasets under different conditions/styles or with larger scales should be constructed.

3.2 EvaluationMetrics

Evaluation metrics are the same for Homo-ReID and Hetero-ReID. When evaluating Hetero-ReID methods, we often choose the Cumulative Matching Characteristics (CMC) [Wang et al., 2007] curve and the mean average precision (mAP) [Zheng et al., 2015a] as the evaluation metrics. In practice, when only one ground truth exists in the gallery, or we are caring about the true matches ranking in top positions of the returning list, the CMC curve is an acceptable selection. On the other hand, if multiple true matches exist in the gallery and we would like to retrieval all the results, mAP is a good selection. Since mAP cares more about the ability of retrieval recall, while CMC does not. Hence, CMC and mAP are always used together for Hetero-ReID.

4 Hetero-ReIDMethods

In this section, we first provide the milestones of existing Hetero-ReID studies, and then make a survey of Hetero-ReID methods from two perspectives. From the perspective of application scenario, we classify all related methods into four kinds of Hetero-ReID application scenarios. From the perspective of learning pipeline, we categorize typical methods into three types.

4.1 Milestones of Existing Hetero-ReID Studies

Through the unremitting efforts of AI researchers, HeteroReID has achieved remarkable success in different aspects. We draw a timeline to introduce important milestones for Hetero-ReID and present them in Figure 3. In particular, 'Low-resolution', 'Infrared-Visible', 'Text-Image', and 'Sketch-Photo' indicate milestones of application scenarios. Their typical methods are introduced in § 4.2. 'Metric Learning', 'Representation Learning', and 'Modality Unification' indicate milestones of learning pipelines, which are discussed in § 4.3. Besides, milestones of new technical trends, such as deep learning, adversarial learning, and GAN, are pointed out as well.

4.2 From the Perspective of Application Scenario

Hetero-ReID LR. Commonly, the resolution of person image changes a lot, due to the variations in the person-camera distance and camera deployment settings. Hetero-ReID LR application scenario attempts to compare images with different resolutions, where low-resolution leads to an extreme loss of appearance information and a large increase of discrepancy. [Jing et al., 2015] was the first to investigate the Hetero-ReID LR task. They designed a semi-coupled lowrank discriminant dictionary learning method, constructing a relationship mapping from the features of normal person images to that of LR person images. [Li et al., 2015] proposed a joint multi-scale learning framework by learning metrics on feature domains of two different image scales simultaneously. [Wang et al., 2016b] changed the problem to be low-resolution with different scales. They observed that scale-distance functions could be classified, and then learned a surface in function space to separate functions to identify persons. Jiao et al. [Jiao et al., 2018] developed a superresolution and identity joint learning method to improve the Hetero-ReID LR performance. Similar to [Jing et al., 2015],

[Li et al., 2018] designed a semi-coupled projective dictionary learning model to bridge the gap across different resolutions. [Chen et al., 2019] combined a resolution adaptation network and a re-identification network together to solve Hetero-ReID LR problem. [Wang et al., 2018] first cascaded multiple SR-GANs in series to promote the ability of scale-adaptive super-resolution, then plugged-in a reidentification network to enhance the ability of person representation. [Mao et al., 2019] jointly trained a ForegroundFocus Super-Resolution module and a Resolution-Invariant Feature Extractor, and then obtained a strong resolution invariant representation. [Cheng et al., 2020] discovered the underlying association knowledge between image SR and ReID, and leveraged it as an extra learning constraint for enhancing the compatibility of SR and ReID models.

Hetero-ReID IR. The visible cameras are not able to capture clear and valid appearance information under poor illumination environments (e.g. during the nighttime), which limits the applicability of ReID in practical surveillance applications. Hetero-ReID IR application scenario provides a good supplement for nighttime surveillance applications, which also introduces a large intra-modality discrepancy. [Wu et al., 2017b] was the first to investigate Hetero-ReID task in the Hetero-ReID IR task. They proposed a deep zero-padding module to improve the one-stream network, making the implicit network structure more suitable. [Ye et al., 2018a] jointly optimized the modality-specific and modality-shared metrics and designed a hierarchical cross-modality matching model. [Ye et al., 2018b; Ye et al., 2019] attempted to learn discriminative features with a bi-directional dual-constrained top-ranking loss function. [Dai et al., 2018] proposed to learn discriminative common representations with a generator for learning image representations and a discriminator for discriminating the modalities of RGB and IR images. [Wang et al., 2019c] presented a new framework, taking advantage of CycleGAN to reduce modality discrepancy from imagelevel and advantage of sophisticated Homo-ReID models to reduce appearance discrepancy from feature-level. [Wang et al., 2020] jointly exploited pixel alignment and feature alignment by generators. [Li et al., 2020] introduced an auxiliary X modality as an assistant and an X-Infrared-Visible three-mode learning framework. [Kansal et al., 2020] designed a network with disentanglement loss to distill identity features and dispel spectrum features. [Choi et al., 2020] also attempted to disentangle ID-discriminative factors and ID-excluded factors from images, and they designed an IDpreserving person image generation network to implement the idea. [Jin et al., 2020] also proposed to distill identityrelevant feature from the removed spectrum style information and restitute it to the network to ensure high discrimination. [Lu et al., 2020] explored the modality-shared information and the modality-specific characteristics to boost the performance. [Yang et al., 2020] proposed a bi-directional random walk scheme to mining more reliable relationships between images by traversing heterogeneous manifolds in the feature space of each modality. Note that the method [Yang et al., 2020] performs as a re-ranking method thus obtains relative high performances.

Hetero-ReID Sketch. When someone reports a lost child case while the child was not identified by the surveillance camera, we call for an algorithm to automatically pick out the potential candidates according to a sketch by an artist's drawing. Consequently, investigators will be able to narrow down and hence expedite their search. [Pang et al., 2018] was the first and only one to investigate Hetero-ReID task in the Sketch application scenario. They not only raised a dataset, but also proposed to learn identity features and modalityinvariant features by exploiting a cross-modal adversarial feature learning framework.

Hetero-ReID Text. Searching a person with free-form natural language descriptions can be widely applied. In the early years, some researches started to investigate the ReID with attribute-based queries [Feris et al., 2014; Ye et al., 2015; Wang et al., 2015; Yin et al., 2018]. Hetero-ReID Text application scenario denotes a situation that the queries are natural language descriptions. [Li et al., 2017b] evaluated and compared a wide range of possible models, and proposed an RNN with Gated Neural Attention mechanism. [Li et al., 2017a] learned to embed cross-modal features, and then refined the matching results with a co-attention mechanism. [Zhang and Lu, 2018] designed a cross-modal projection matching loss and a cross-modal projection classification loss for learning discriminative image-text embeddings. [Chen et al., 2018b] proposed a patch-word matching model and designed an adaptive threshold mechanism into the model. [Chen et al., 2018a] attempted to build global and local image-language associations, which enforce semantic consistencies between local visual and linguistic features. [Liu et al., 2019] proposed a deep adversarial graph attention convolution network.

Summary. From the perspective of application scenario, we make the following summaries:

• Most of the methods selected a deep learning framework. It is probably because all the heterogeneous application scenarios are raised in recent years, and deep learning methods are in their high-speed development period. In general, deep learning methods are superior in shared feature learning and classification tasks when a significant amount of training samples are available.
• Different methods have different focuses. To fill the gaps between the desired modality and other heterogeneous modalities, some methods (e.g. [Li et al., 2015]) try to learn a metric, others (like [Ye et al., 2018a]) attempt to learn shared features, and there are also methods (for example [Wang et al., 2019c]) unifying the modality on the data level. In § 4.3, we will make an analysis and discussion.

• The existing researches in each application scenario still have many limitations. For the Hetero-ReID LR application scenario, recent researches are mainly evaluated on the simulated datasets. For the Hetero-ReID Sketch application scenario, only one research touched this direction, due to the hardness of building a benchmark.

4.3 From the Perspective of Learning Pipeline

Representative methods employed in Hetero-ReID are compared in Table 3. We list some key and valuable information, including the reference paper, conference ('Conf.'), application scenario ('Appl.'), technology ('Techn.'), used datasets, and Learning pipeline. We can find that all representative works are published in the recent five years. It means that Hetero-ReID is a relatively new research topic. Among all the application scenarios, Hetero-ReID Sketch is less investigated. As described in the sections above, we consider that if more datasets are available, more works could be conducted and published. Particularly, to address the inter-modality dis-

crepancy, in our opinion, representative methods can be categorized into three kinds of learning pipelines.

Figure 4 shows the diagrams of the learning pipelines. Pipeline (a) employs the metric learning method to learn how to match the features from separate representation learning models, and the representation learning models are trained separately with single modality data samples (represented as 'ML'). Pipeline (b) focuses on learning shared feature models of different modalities, and training data come from both modalities (represented as 'RL'). Pipeline (c) pays attention to generating the unified-modality samples (represented as 'MU'), for example, using a super-resolution method to generate high-resolution images from low-resolution images, or using some image generation method to generate infrared images from RGB images and RGB images from infrared images.

As shown in Table 3, for each kind of application scenario, from top to bottom, the methods become more effective. Generally, pipeline 'MU' performs better than pipeline 'RL', and better than pipeline 'ML'. Hence, unifying the modalities of samples is an effective way to fill the modality gap. For the Hetero-ReID LR application scenario, the superresolution methods can be used, such as [Wang et al., 2018; Mao et al., 2019]. Thanks to the developments of GAN, unifying the modalities is considerable as [Wang et al., 2019c].

5 Conclusion and Future Directions

This paper overviews recent developments of Hetero-ReID. It covers most of the literature on Hetero-ReID. We summarized available datasets, the widely employed methods, and compared the existing techniques. We believe that Hetero-ReID will continue to be an active and promising research area with broad potential applications. Many issues in Hetero-ReID, however, still remain.

• Dataset Construction. For Hetero-ReID LR application scenario, the researchers can only get simulated datasets. For the other types of applications, although the researchers can achieve practical datasets, the choice is limited. In addition, large scale Hetero-ReID datasets are

also required. It is somehow urgent for us to built new datasets, and push forward this research area. On the other hand, synthesizing more realistic datasets (like [Zeng et al., 2020]) is also a good way to make one step forward.

• Taking Advantages of Homo-ReID Datasets and Methods. There are sufficient datasets and methods in Homo-ReID. It is easy to address the intra-modality discrepancy in the desired modality. Obviously, the feature space of the desired modality can be well-constructed. Since the heterogeneous modality samples are not enough, it is hard to build a perfect heterogeneous feature space. However, it is reasonable to project the Hetero-ReID samples into the desired modality (like [Yang et al., 2020]).
• Human Interaction and Crowd-sourcing. For Hetero-ReID Sketch and Text application scenarios, human intelligence joins into the process of ReID. Human intelligence is sometimes subjective and incomplete. So we should consider how to mine and integrate useful information to help search the target in the surveillance system. On the other hand, a lot of witnesses will provide their cues. Each person may have a different view so that the crowd-sourcing cues may be diverse to each other. We should design a strategy to remove conflict and filter valuable information.
• Investigation on Unifying the Modality. For Hetero-ReID LR application scenario, some methods [Wang et al., 2018; Mao et al., 2019] attempted to use super-resolution technologies to unify the modality, where they transfer the heterogeneous modality LR image to the desired modality. For Hetero-ReID IR application scenario, with the help of CycleGAN, one work [Wang et al., 2019c] not only transferred the image from heterogeneous (IR) modality to desired (RGB) modality but also transferred the image from desired modality to heterogeneous modality. However, for the Hetero-ReID Text and Sketch application scenarios, no work has investigated to unify the data modality. On the other hand, we can also investigate to unify the modality to a latent modality as Figure 5 shows, for example, a middlelevel resolution for the Hetero-ReID LR application scenario and a hyperspectral image for the Hetero-ReID IR application scenario.

• Integrating Multiple Hetero-ReID Application Scenarios. For a practical system, it may require different kinds of inputs to search out the target. The Hetero-ReID application scenarios can be equipped in different stages. If we can integrate different kinds of inputs, more valuable information could be used for retrieval, since different inputs have different attentions and views for the target. It would raise a novel multiple cross-modality research task.
• Considering the Privacy Issue. It is fine to conduct academic research on public datasets. But with the leakage of personal image data, privacy concerns are raising nowadays when algorithms need to be applied to practical applications. Recently, pioneer works [Mirjalili and Ross, 2017] have explored to hide some of private information presented in the images. Further research on the principles of visual cryptography, signal mixing and image perturbation to protect users' privacy on person templates are essential for addressing public concern on privacy. In particular, for Hetero-ReID, since it contains multi-modality data, person templates of different modalities or a latent modality should be investigated to protect.

Acknowledgements

This work was supported partly by the JST CREST Grant JPMJCR1686, partly by the Grant-in-Aid for JSPS Fellows 18F18378, and partly by the Microsoft Collaborative Research Grant.