HorNet: A Hierarchical Offshoot Recurrent Network for Improving Person Re-ID via Image Captioning

1 Introduction

Person re-identification (re-ID) has become increasingly popular in the modern computer vision community due to its great significance in the research and applications of visual surveillance. It aims at recognizing a person-of-interest (query) across different cameras. The most challenging problem in re-ID is how to accurately match persons under intensive variance of appearances, such as human poses, camera viewpoints, and illumination conditions. Encouraged by the remarkable success in deep learning algorithms and the emergence of large-scale datasets, many advanced methods have been developed to relieve these vision-based difficulties and made significant improvements in the community [Li et al., 2017a; Su et al., 2017; Chen et al., 2018].

Recent years witness that the application of the various axillary information, such as human poses [Su et al., 2017], person attributes [Schumann, 2017] and language descriptions [Chen et al., 2018], can significantly boost the performance of person re-ID. These serve as the augmented feature representations for improving person re-ID. Notably, the image captions could provide a comprehensive and detailed footprint of a specific person. It is semantically richer than visual attributes. More importantly, language descriptions of a particular person are often more consistent across different cameras (or views), which could alleviate the difficulty of the appearance variance in person re-ID task.

∗Contact Author

Two significant barriers exist in applying the image captions for person re-ID. The first one is the increasing complexity to handle image captions. It is certain that language descriptions contain many redundant and fuzzy information, which could be a great challenge if not handled properly. Thus an effective learning approach for constructing a compact representation of language descriptions is of vital importance. Another one is the lack of description annotations for person re-ID task. Recently, [Li et al., 2017b] proposed the CUHK-PEDES, which provides person images with annotated captions. The images from this dataset are collected from various person re-ID benchmark datasets such as CUHK01 [Li et al., 2012], CUHK03 [Li et al., 2014], Market-1501 [Zheng et al., 2015a], and et al. However, the annotations are usually restricted to these datasets. In realworld applications, the person images normally do not have paired language descriptions. Thus, a method for automatically generating the high-quality semantic image captions to various real-world datasets is also urgently needed.

In this paper, we propose a novel hierarchical offshoot recurrent network (HorNet) for improving person re-ID via image captioning. Figure 1 illustrates the schematic illustration of our framework for person re-ID task. We first use the similarity preserving generative adversarial network (SPGAN) [Deng et al., 2018] to transfer the real-world images into a unified domain, which can significantly enhance the quality of the generated descriptions via the following image captioner [Aneja et al., 2018]. Then both of the images and generated captions are used as the input to the HorNet. The HorNet has two sub-networks to handle the input images and captions, respectively. For images, we utilize mainstream CNNs (i.e., Resnet50) to extract the visual features. For captions, we develop a two-layer LSTMs module with a discrete binary gate in each time step. The gradient of the separate gates is estimated using Gumbel sigmoid [Jang et al., 2016]. This module dynamically controls the information flow from the lower layer to the upper layer via these gates. It selects the most relevant vocabularies (i.e., the correct or meaningful words), which are consistent with the input visual features. Consequently, HorNet can learn the visual representations from the given images and the language descriptions from the generated image captions jointly, and thus significantly enhance the performance of person re-ID. Finally, we verify the performance of our proposed method in two scenarios, i.e., person re-ID datasets with and without image captions. Experimental results on several widely used benchmark datasets, i.e., CUHK03, Market-1501, and Duke-MTMC, demonstrate the superiority of the proposed method. Our method can simultaneously learn the visual and language representation from both the images and captions while achieving a state-of-the-art recognition performance.

In a nutshell, our main contributions in the present work can be summarized as threefold:

1. We develop a new captioning module via image domain transfer and captioner in person re-ID system. It can generate high-quality language captions for given visual images.
2. We propose a novel hierarchical offshoot recurrent network (HorNet) based on the generated images captions, which learns the visual and language representation jointly.
3. We verify the superiority of our proposed method on person re-ID task. State-of-the-art empirical results are achieved on the three commonly used benchmark datasets.

2 RelatedWork

The early research works on person re-ID mainly focus on the visual feature extraction. For instance, [Yi et al., 2014] split a pedestrian image into three horizontal parts and train threepart CNNs to extract features. Then the similarity between two images is calculated based on the cosine distance metric of their features. [Chen et al., 2018] use triplet samples for training the network, considering not only the samples of the same person but also the samples of different people. [Liu et al., 2016] proposes a multi-scale triplet CNN for person reID. Due to recently released large-scale benchmark dataset, e.g., CUHK03 [Li et al., 2014], Market-1501 [Zheng et al., 2015a], many researchers try to learn a deep model based on the identity loss for person re-ID. [Zheng et al., 2016] directly uses a conventional fine-tuning approach and outperforms many previous results. Also, recent research [Zheng et al., 2017a] proves that a discriminative loss, combined with the verification loss objective, is superior.

Several recent research has endeavored to use auxiliary information to aid the feature representation for the person reID. Some research [Su et al., 2017; Zhao et al., 2017] relies on the extra information of the person's poses for person reID. They leverage the human parts cues to alleviate the pose variations and learn robust feature representations from both the global and local image regions. Another type of auxiliary information, attributes of a person, has been used in person re-ID [Lin et al., 2017]. However, these methods all rely on the attribute annotations, which are normally hard to collect in real-world applications. [Schumann, 2017] uses automatically detected attributes and visual features for person re-ID. The attribute detector is trained on another dataset which contains the attribute annotations.

The relationship between visual representations and language descriptions has long been investigated. It has attracted high attention in tasks such as image captioning [Yan et al., 2018b], visual question answering. Associating person images and their corresponding language descriptions for the person searching has been proposed in [Li et al., 2017b]. Several research works employ the language descriptions as complementary information, together with visual representations, for person re-ID. [Chen et al., 2018] exploit natural language descriptions as additional training supervision for effective visual features. [Yan et al., 2018a] propose to combine the language descriptions and image features and fuse them for the person re-ID task. Previous language models encode the sentences using either Recurrent Neural Networks (RNNs) language encoder or Convolutional Neural Networks (CNNs) encoder. Recent research [Bahdanau et al., 2014] employ the attention mechanism for these language models by looking over the entire sentence and assigning weights to each word independently. Especially, RNNs with attention have been widely applied in machine translation, image captioning, speech recognition and Question and Answering (QA). The attention mechanism allows the model to look over the entire sequence and pick up the most relevant information. Most of the previous attention mechanism employs a similar approach to [Bahdanau et al., 2014], where the neural model assigns soft weights on the input tokens. Recently, [Ke et al., 2018] proposes a Focused Hierarchical Encoder (FHE) for the Question Answering (QA), which consists of multi-layer LSTMs with the discrete binary gates between each layer. Our HorNet also utilizes the discrete gate but with a very different mechanism and purpose. We aim to eliminate redundant information or incorrect language tokens, while they tried to answer the question.

3 Our Method

3.1 Improving Person Re-ID via Image Captioning

The image caption of a specific person is semantically rich and can provide complementary information for the visual representations. However, the handcrafted descriptions of a person image are hard to collect due to the annotation difficulties in real-world person re-ID applications. We propose a method to generalize the language descriptions accurately from a dataset with image captions to others without such captions. The whole scheme of our approach is illustrated in Figure 2. Given images with captions, i.e., the CUHKPEDES dataset, we use SPGAN to transfer arbitrary image to the CUHK-PEDES style. The SPGAN is proposed to improve image-to-image domain adaptation by preserving both the self-similarity and domain-dissimilarity for person re-ID. We utilize it in our case as in [Deng et al., 2018] to transfer the image domain (or style) of the un-annotated datasets. Then we train an image captioner [Aneja et al., 2018] to generate image descriptions automatically on the transferred datasets. The visualization of the domain transfer process and corresponding generated captions are illustrated in Figure 4. It is clear that the transferred images have more accurate language descriptions. However, the generated sentences, which are based on the domain-translated images, still contain some incorrect keywords and redundant information. The proposed HorNet, which contains the discrete binary gates, can select the most relevant language tokens with the visual features, and thus provide a good solution for the issue.

3.2 The Proposed HorNet Model

To facilitate the visual representations of a person in the person re-ID task, we propose the HorNet to learn the visual features and the corresponding language description jointly. The HorNet adds a branch to the CNNs (i.e., Resnet-50) with two-layer LSTMs and a discrete or continuous gate between each layer at every time step. The lower-layer LSTM handles the input languages while the upper-layer LSTM selects the relevant language features via the gates. Finally, the last hidden state of the upper-layer LSTM is concatenated with the visual features extracted via Resnet-50 to generate a compact representation. The objective function of the HorNet consists of two parts: the identification loss and the triplet loss, which are trained jointly to optimize the person re-ID model.

We present the pipeline of our proposed method in Figure 3. The input to the HorNet consists of two parts, i.e., the image and the corresponding language descriptions. Let the language descriptions be processed by a two-layer LSTM model. The bottom layer is a normal LSTM network, which reasons on the sequential input of the language descriptions. More formally, let the D = (d1, d2, ..., dn) be the input of description, ht be the hidden state, ct be the LSTM cell state at time t. The term E = (e1, e2, ..., en), where et = Word Embedding(dt), t = 1, 2, ..., n, denotes the word embedding of the input description. In our research work, we use a linear embedding for the input language tokens. Hence, the bottom LSTM layer can be expressed in Equation (1).

where function LSTM denotes the compact form of the forward pass of an LSTM unit with a forget gate as:

where et ∈ Rd denotes input vector, f l t ∈ Rh is forget gate's activation, ilt ∈ Rh is input gate's activation, olt ∈ Rh is output gate's activation, clt ∈ Rh is cell state vector, and hlt ∈ Rh is hidden state of the LSTM unit l. W ∈ Rh×d ,U ∈ Rh×h and b ∈ Rh are weight matrices and bias vector parameters which need to be learned during training. The activation σ is sigmoid function and the operator ∗ denotes the Hadamard product (i.e., element-wise product).

The boundary gate controls the information from the lower layer to the upper layer. The boundary gate zt is estimated with Gumbel sigmoid, which is derived directly from the Gumbel softmax proposed in [Jang et al., 2016].

The Gumbel softmax replaces the argmax in the Gumbel-Max Trick with the following Softmax function:

where g1, ..., gk are i.i.d. sampled from the distribution Gumbel(0, 1), and τ is the temperature parameter. K indicates the dimension of the generated Softmax vector (i.e., the number of categories).

To derive the Gumbel sigmoid, we firstly re-write the Sigmoid function as a Softmax of two values: πi and 0, as in the following Equation (3).

Hence, the Gumbel sigmoid can be written as in the following Equation (4).

where gi and g′ are independently sampled from the distribution Gumbel(0, 1).

Thus, the upper-layer LSTM inputs are the gated hidden units of the lower-layer, which can be expressed as the following Equation (5) and Equation (6). In our experiments, all the soft gates zt are estimated using the Gumbel sigmoid with a constant τ of 0.3.

where F denotes the deep visual features of the images extracted via CNNs (i.e., Resnet-50) and the Concat indicates the features concatenation operation.

To obtain a discrete value (i.e., language tokens selection), we also set the hard gates zt = ˜ yi in Equation (5).

Finally, we forward the last hidden unit of the language branch to form a compact representation by using a concatenation operation with the corresponding visual features F as,

Our loss function is the combination of the identification loss and triplet loss objectives, which can be expressed in the following Equation (9) as,

where LID is a K-class cross entropy loss parametered by θ. It treats each ID number of person as an individual class as,

and LTriplet denotes the Triplet loss as,

where xa is the anchor, xp indicates the positive example, and xn is the negative sample. The α means margin.

4 Experiments

We evaluated the proposed methods on person re-ID datasets such as CUHK03 [Li et al., 2014], Market-1501 [Zheng et al., 2015a] and Duke-MTMC [Ristani et al., 2016]. There are two types of experiments: with and without description annotations. For CUHK03 and Market-1501, the description annotations can be directly retrieved from the CUHK-PEDES dataset [Li et al., 2017b]. Hence, we evaluated the proposed method for person re-ID by using these annotations. However, Duke-MTMC lacks the language annotations. We used an image captioner to generate language descriptions, which are used to jointly optimize the proposed HorNet.

4.1 Implementation Details

For the HorNet, the embedding dimension of word vectors is 512. The dimension of the hidden unit of the LSTM is also 512. Since we used a fully connected layer to process the last hidden unit of the upper-layer LSTM and consequently reduced its dimension to 256, the dimension of the final representation for vision and language is 2304 (i.e., 256+2048), in which the dimension of the visual features is 2048.

4.2 Experiments with CUHK-PEDES Annotations

To first verify the effect of the language descriptions in person re-ID, we augmented two standard person re-ID datasets, which are CUHK03 and Market-1501, by using annotations from CUHK-PEDES dataset. The annotations of CUHK-PEDES are initially developed for cross-modal textbased person search. Since the persons in Market-1501 and CUHK03 have many similar samples and only four images of each person in these two datasets have language descriptions, we annotated the unannotated images in those datasets with the language information from the same ID, which is the same as the protocol in [Chen et al., 2018].

We first evaluated the proposed method on CUHK03 dataset, using the classical evaluation protocols [Li et al., 2014]. We tested the baseline method which uses the identification loss based on the Resnet-50 model, with 72.4 CMC top-1 accuracy in the CUHK03 detected images. The CMC top-1 result is raised to 80.0 by augmenting the language descriptions. A similar phenomenon can be seen in the CUHK03 labeled images. We performed an ablation study to verify the effect of the different part in HorNet. The experimental results are presented in Table 1. The proposed HorNet with reranking technique can achieve the best performance on Market-1501, CUHK03 detected images, and CUHK03 labeled images datasets.

The comparison with other state-of-the-art methods are listed in Table 2. Specifically, we compared the proposed HorNet with other methods which employ auxiliary information, which include Deep Semantic Embedding (DSE) [Chang et al., 2018], ACRN [Schumann, 2017], ACRN [Schumann, 2017], Vision and Language (VL) [Yan et al., 2018a], Image-language Association (ILA) [Chen et al., 2018]. ACRN applies axillary attribute information to aid the person re-ID. DSE, ACRN, VL and ILA all employ the external language descriptions for person re-ID. Among them, VL uses a vanilla LSTM or CNN language encoding model, which is discriminatively poorer than our HorNet, since the proposed HorNet uses discrete gates to select useful information for person re-ID. ILA uses the same training and testing protocol but with a more complex model. Our model can also be combined with various metric learning techniques, including the Rerank proposed in [Zhong et al., 2017]. We also employed the Rerank to post-process our features, with improved results. Overall, our HorNet performs much better than the ILA on the CUHK03 classic evaluation protocol, achieved 97.1% CMC top-1 accuracy, with a 4.6% raise over the ILA. We also conducted experiments on the Market-1501 dataset, the results are presented in Table 1 and Table 3. A similar phenomenon to those of CUHK03 can be seen in Table 3, with 85.8% mAP result.

4.3 Experiments on the Duke-MTMCDataset (Without Captions)

In a realistic person re-ID system, language annotations are rare and hard to get. Hence, we want to see if the automatically generated language descriptions can boost the performance of a person re-ID system. We chose a more challenging and realistic dataset, i.e., Duke-MTMC [Ristani et al., 2016] to verify this assumption. Firstly, we trained an image captioning model based on the CUHK-PEDES dataset by using the convolutional image captioning model, which has released code and good performance [Aneja et al., 2018]. We split the CUHK-PEDES images into two splits: 95% for training and 5% for validation. We used the early stopping technique to train the image captioning model and achieved 35.4 BLEU-1, 22.4 BLEU-2, 15.0 BLEU-3, 9.9 BLEU-4, 22.3 METEOR, 34.2 ROUGE L and 22.1 CIDEr results on the validation set. Subsequently, we used the trained image captioning model to generate language descriptions for the Duke-MTMC dataset. However, we found that the generated descriptions are not discriminative enough, as shown in Figure 4. There are many incorrent or imprecise keywords in the language descriptions. Also, we tested the performance by augmenting the Duke-MTMC with the generated descriptions and the results turned out to be poor, even worse than the baselines, only with 52.5% mAP result, as shown in Table 4. The cause of this phenomenon is the poor generalization capability of the image captioning model, especially when there is a domain difference between two diverse datasets. To alleviate this problem, we used the SPGAN [Deng et al., 2018] to transfer the image style of the Duke-MTMC to the CUHKPEDES. The generated language descriptions are with much better quality, as presented in Figure 4. The results from the augmentation with the generated language descriptions on the transferred Duke-MTMC images are much better than that provided by the simple visual features, with 60.4% mAP result on Duke-MTMC. To prove the superiority of the HorNet, we also use BERT [Devlin et al., 2018] to replace HorNet, but with a poorer performance. Furthermore, we also implement a Rerank [Zhong et al., 2017] to boost the final recognition performance and achieved 79.2% mAP result.

5 Conclusions

In this paper, we developed a language captioning module via image domain transfer and captioner techniques in person reID system. It can generate high-quality language descriptions for visual images, which can significantly compensate for the visual variance in person re-ID. Then we proposed a novel hierarchical offshoot recurrent network (HorNet) for improving person re-ID via such an automatical image captioning module. It can learn the visual and language representation from both images and the generated captions, and thus enhance the performance. The experiments demonstrate promising results of our model on CUHK03, Market-1501 and Duke-MTMC datasets. Future research includes a more robust language captioning module and advanced metric learning methods.