Label-Attended Hashing for Multi-Label Image Retrieval

1 Introduction

Similarity hash code has been widely used in large-scale image retrieval owing to its lightweight storage (compact binary code) and efficient comparison (exclusive-OR) [Liu et al., 2019; 2020]. For classic image hashing, correctly recognizing the object from an image is an important factor to promote the retrieval precision. However, it becomes more challenging for multi-label image retrieval where each image contains more objects. Several studies [Lai et al., 2016; Song et al., 2017; Huang et al., 2018] specifically designed for multi-label image retrieval recognize each object in isolation and then fuse these features to learn a hash model, which is essentially limited by ignoring the complex topology structure between objects and thus fails to further improve the precision of retrieval. For example, a basketball will appear in a court with a higher probability than a football. The existing hashing methods have not benefited from this prior information.

∗Corresponding Author

To address this problem, there are two key challenges: (1) how to construct an ideal topology structure and what kind of dependency should be expressed, (2) how to use the topology information to learn image representations in the hash task via an end-to-end manner.

Based on this, we first explore the correlation between objects according to their co-occurrence probability [Wang et al., 2017] which is collected from labels via attention mechanism. As shown in Figure 1, we leverage the directed graph structure to capture and explore the label dependency between objects. The reason why we model a directed graph is that the mutual conditional probabilities between two objects (labels) are asymmetrical. Take the three images in Figure 1 as example, when person appears in the image, book will oc- Image Representation Learning cur with probability 33.33%. However, in the condition of book appearing, person will certainly occur with probability 100%.

For the second challenge, we first combine Graph Convolutional Network (GCN) [Zhou et al., 2018] and Word2Vector method [Pennington et al., 2014] to learn the label cooccurrence embeddings, where these embeddings can well express the features of co-occurrence correlation. Although these features come from the label word vectors, they function as the classifiers in the fusion of image features and help learn the image features that embody the co-occurrence correlation. ML-GCN [Chen et al., 2019] adopts a dot product (DP) in this fusion process, which achieves good results in terms of image classification. For the feature extraction of hash tasks that require more effective and more attention information, we use our improved MFB instead of DP. On the one hand, the MFB component improves fusion efficiency by adding fully connected (fc) layers with more flexible fitting space. On the other hand, it generates more fusion parameters by group sum-pooling to both promote the fusion precision and extend the attention representations.

Based on the above work, we propose a label-attended hash (LAH) model which can capture multi-label image features and fast achieve hash mapping according to the label dependency in the end-to-end pipeline. For the label co-occurrence embedding, we take both the word vectors and co-occurrence correlation as input, then use GCN to calculate the label embeddings which implicitly represent the co-occurrence relationship. Meanwhile, we use ResNet-101 [He et al., 2016] to generate an image representation by means of global maxpooling. In the following, we fuse co-occurrence embeddings and the image representation via MFB to learn an end-to-end hash model. As for the hash function, we introduce the stateof-the-art hash function consisting of a quantization loss and a pairwise cross-entropy loss based on the Cauchy distribution to obtain the optimal result, both of which conduce to concentrate relevant images to be within a small Hamming ball. The overall deep architecture is shown in Figure 2. The proposed LAH model can be trained end-to-end along red line by backpropagation. Extensive experiments on public multilabel datasets demonstrate LAH can generate highly concentrated and compact hash codes and achieve the state-of-the-art retrieval results. LAH also has higher representation learning efficiency compared with ML-GCN.

2 Label-Attended Hashing

In the implementation, we assume that there are N samples {xi|i = 1, 2, · · · ,N} in the training set, where L(xi) denotes the label set of the i-th sample and sij denotes the similarity between xi and xj (sij=1 if xi is similar to xj while sij=0 if they are dissimilar). Specifically, in the multilabel scenario, we stipulate sij = 1 if L(xi) ∩ L(xj) ̸= ∅, otherwise sij = 0. In addition, we assume that there are R objects {rg|g = 1, 2, · · · ,R} in the whole label set and ER= {E(rg)|g = 1, 2, · · · , R} where E(rg) denotes the cooccurrence embedding of the g-th object. The architecture for LAH is shown in Figure 2, where the pipeline along the red arrow is the backbone of our algorithm which can be trained by backpropagation. When training our model in this backbone, we input pairs of {(xi, xj, sij)} and ER, where D(x) and D(E(r))×R respectively denote the dimension of x and ER. After the image representation learning stage, x is transformed into a vector ˜x with D(˜x) dimension which will be sent to MFB along with E(rg) to generate a 1 × R vector Z after completing R times fusion. In the last hash learning stage, LAH transforms Z into K-dimensional continuous code ˜ Z ∈ RK for each image x, and then transforms ˜ Z into K-dimensional hash code by h = sgn(tanh( ˜ Z)) ∈ {-1, 1}K in the fc hash (fch) layer. Finally, with Cauchy cross-entropy loss and Cauchy quantization loss, LAH learns a non-linear hash function F(x) : x → h ∈ {-1, 1}K , which encodes each sample x into compact K-bit hash code. The details of each component are described below.

2.1 Image Representation Learning

Any CNN based models can complete the feature extraction of an image in our architecture. In our experiments, following ML-GCN, we choose ResNet-101 as the model. Based on this, for an image x with the resolution D(x) = 448×448, we can obtain a 2048×14×14-dimensional feature vector from the "conv5_x" layer. At last, we adopt global max-pooling Fgmp to generate the image-level feature ˜x:

where θ denotes the CNN parameters and D(˜x) = 2048.

2.2 GCN for Label Co-occurrence Embedding Learning

The motivation of introducing GCN is that it can learn the relationship representation from the feature descriptions according to the input of the relationship. Essentially, it learns a propagation function Fgcn on the graph via weight propagation to complete feature extraction. We use the GloVe [Pennington et al., 2014] model to convert the object (word description in the label set) into vector V (r), where V c ∈ RR×D(V (r)) represents the input in c-th layer and D(V (r)) represents the dimension of V (r). The input of relationship is the correlation matrix A ∈ RR×R, and the updated node features are denoted as V c+1 ∈ RR×D(V (r))′ . Each GCN layer propagation function is described as:

where ˆ A = ˜D- 12 (A + IR) ˜D- 1 2 with ˜Dii = ∑j ˜ Aij and ˜ A = A + IR.

In our experiments, we use two GCN layers, i.e., E(r) = V c+2 . Referring to ML-GCN, we adopt the data-driven method to construct matrix A which is the key of cooccurrence learning. Specifically, in order to determine each element of matrix A, we have to first collect the occurrence times as well as co-occurrence times of each object according to the label set. Let Ti, Tj respectively denote the occurrence times of ri, rj in the label set and Tij (which equals Tji) denote the co-occurrence times of these two objects. Then, we model the label correlation dependency in the form of conditional probability, i.e.,

which denotes the probability of occurrence of ri when rj appears. Note that, as shown in Figure 1, Ti ̸= Tj , Pij ̸= Pji. Based on this, Aij = Pji where Aij denotes the i-th row and j-th column element of A.

However, in the implementation, we find if directly using this simple correlation, the overall data distribution may exhibit a long-tail effect owing to the rare co-occurrence objects which become noise and affect the model convergence. To promote the efficiency and prevent over-fitting, we use the threshold τ to binarize the matrix A, the operation can be written as:

where ¯ A is the binary correlation matrix. In addition, there exactly exists the over-smoothing problem [Chen et al., 2019] when using the above binary correlation matrix to train the GCN model, which may result in that the generated hash codes become indistinguishable. Therefore, similar to MLGCN, we adopt the weighted scheme and draw the final correlation matrix A as:

where q ∈ (0, 1). In this way, we can update a node feature by choosing a fixed weight q. For example, a node feature will rely more on itself if using a smaller q. Otherwise, its feature will be determined by other neighbor nodes.

2.3 MFB for Fusion

The MFB is one of the keys to achieve high precision with fast learning efficiency for LAH. The motivation of improving this component to build the LAH fusion mechanism lies in that both the multi-label image representation learning and the label co-occurrence embedding learning implicitly adopt the attention mechanism, while MFB can achieve the highest efficiency and best performance when conducting multimodal co-attention fusion. Specifically, on the one hand, we utilize Hadmard product and group sum-pooling instead of the DP to increase the interaction of vector elements among different modalities, which promotes the precision. On the other hand, by means of sum-pooling, it reduces over-fitting and parameter explosion caused by the increasing interaction and thus accelerates the model convergence.

Given two feature vectors in different modalities, i.e., image representation ˜x ∈ RD(˜x) and co-occurrence embedding E(r) ∈ RD(E(r)) , we also draw lessons from tricks for unimodal data [Li et al., 2017] to reduce high computational cost and a risk of over-fitting. For the feature of the i-th object, the multi-modal bilinear model with two low-rank matrices is defined as follows:

where k is the latent dimensionality of the factorized matrices U = [u1, · · · , uk] ∈ RD(˜x)×k and V = [v1, · · · , vk] ∈ RD(E(r))×k , ◦ is the Hadmard product, i.e., the element-wise multiplication of two vectors, 1 ∈ Rk is an all-one vector.

In practice, as shown in Figure 2, we adopt two parallel kdimensional fc layers (red and blue fc layers shown in MFB framework) to complete this transform respectively, where M1 = U T i ˜x andM2 = V T i E(r).

According to the total number of object categories, we need to get R vector z as input to match the subsequent hash mapping. Another purpose of this design is to provide better explanation for the representation of each object. Therefore, to obtain the output Z = [z1, · · · , zR], the weights to be learned are two three-order tensors ˜U = [U1, · · · ,UR] ∈ RD(˜x)×k×R and ˜ V = [V1, · · · , VR] ∈ RD(E(r))×k×R. Without loss of generality, we can reformulate ˜U and ˜ V as 2-D matrices ˜U ∈ RD(˜x)×(k×R) and ˜ V ∈ RD(E(r))×(k×R) respectively. At last, we obtain the result of group sum-pooling as below:

where kG ∈ [1, k]Z+ and the function F G sum-pooling (∗, k) means using a one-dimensional non-overlapped window with the size kG to perform sum-pooling over ∗. Different from [Yu et al., 2017], we improve this work where all the R identical vectors in ˜U are generated from ˜x while the vectors in ˜ V correspond to the elements of ER transformed by fc layer.

2.4 Cauchy Distribution for Hash Learning

Due to the diversity of retrieval target for the multi-label image, it becomes a tricky task to generate hash codes with small distance between similar data and large distance between dissimilar data in the Hamming space. Luckily, Cauchy distribution can adjust the location of data peak to embody the focus of attention. According to the verification of DCH [Cao et al., 2018], the hash function based on Cauchy distribution can solve the problem that the conventional sigmoid function brings low aggregation degree of similar samples within short Hamming distance, and it can achieve better results when Hamming radius ≤ 2 in terms of Mean Average Precision (MAP), Precision and Recall. Therefore, we adopt Cauchy distribution based function as our LAH hash function to promote the aggregation degree of similar data within small Hamming radius by jointly preserving similarity of pairwise images and controlling the quantization error.

Assume that hi, hj ∈ {-1, 1}K respectively denote the output of xi and xj after the fch layer while {(xi, xj, sij) : sij ∈ S} denotes the input. Based on the Cauchy distribution, we design the probability function:

where σ(∗) is well-defined probability function, δ(hi, hj) denotes the Hamming distance between hi and hj, γ is the scale parameter of the Cauchy distribution. Note that the smaller γ, the higher aggregation degree within short Hamming radius. We repeatedly conduct extensive experiments and choose γ = 0.15.

To learn high-quality hash codes and control the quantization error ‖ hi - sgn(hi) ‖ resulting from continuous relaxation, we combine parameter γ (Equation (2)) and Cauchy distribution to design prior for each hash code as follows:

where 1 ∈ RK . In order to cooperate with the continuous relaxation based on Cauchy distribution, we adopt δ(hi, hj) = K2 (1 - cos(hi, hj)) to normalize Euclidean distance as the approximation of Hamming distance to ease the optimization. The normalized Euclidean distance compares each pair of continuous codes on a unit sphere, while the Hamming distance compares each pair of hash codes on a unit hyper-cube.

According to the deduction of Bayesian learning in DCH and Equation (2), the Cauchy cross-entropy loss Lcce is derived as:

where

where Ss = {sij ∈ S : sij = 1} is the set of similar pairs and Sd = {sij ∈ S : sij = 0} is the set of dissimilar pairs.

In addition, according to equation (3), the Cauchy quantization loss Lcq is

Based on above equations, the completed hash function

where λ is a hyper-parameter to trade-off two loss functions. Note that our sign function is designed as

where hi is a element in vector h ∈ RK . This binarization will not severely affect the retrieval result because it have been adjusted by the quantization loss.

3 Experiments

3.1 Datasets

VOC2007. [Everingham et al., 2010] consists of 9,963 multi-label images and 20 object classes. On average, each image is annotated with 1.5 labels. Note that we transform 0 to 1 in label dataset.

MS-COCO. [Lin et al., 2014] is a popular multiple object dataset for image recognition, segmentation and captioning, which contains 118,287 training images, 40,504 validation images and 40,775 test images, where each image is labeled by some of the 80 semantic concepts.

FLICKR25K. [Huiskes and Lew, 2008] is a collection of 25,000 multi-label images belonging to 24 unique provided labels, and each image is annotated by 4.7 labels on average. We randomly select 2,000 images as the test set. The remaining images are used as the retrieval images, where we randomly select 10,000 images as the training set.

3.2 EvaluationMetrics

Following classic settings [Cao et al., 2018], we report the three standard evaluation metrics to measure the quality of hash codes within Hamming radius 2: Mean Average Precision within Hamming Radius 2 (MAP@H≤2), Precision curves within Hamming Radius 2 (P@H≤2), and Recall curves within Hamming Radius 2 (R@H≤2).

3.3 Implementation Details

For label co-occurrence embedding learning, our LAH consists of two GCN layers with output dimensionality of 1024 and 2048 (D(E(r)) = 2048), where the initial label vector (D(V (r)) = 300) is generated by GloVe trained on the Wikipedia dataset. For the label expressed by multiple words, we adopt the average of embeddings for all words to present the initial label vector. As for the correlation matrix, we set τ = 0.4 and q = 0.2. In the part of image representation learning, we adopt ResNet-101 pre-trained on ImageNet [Deng et al., 2009] to function as the backbone before MFB component, where mini-batch size is fixed as 256 and the raw images (input) are random resized into 448×448 using random horizontal flips. In the part of MFB, without otherwise stated, we set k = 350 for all datasets. For fair comparisons with other algorithms, we set G = 350.

In the part of fch, we set the model parameters (γ = 1 and λ = 0.55) of LAH by cross-validation. We fine-tune ResNet-101 as well as train GCN, MFB and fch all through backpropagation. The processing is implemented using PyTorch 1 . For network optimization, Stochastic Gradient Descent (SGD) [Amari, 1993] is used as the optimizer with 0.9 momentum and 10-4 weight decay. Note that all the results are obtained within 20 epochs.

3.4 Results

Learning Efficiency

To show the efficiency of MFB, we compare the classification precision of LAH (using the improved feature extraction part based on ML-GCN) with that of ML-GCN in each epoch. For fair comparisons, we use the same training parameters, loss function and datasets (i.e., VOC2007 and MS-COCO) as ML-GCN. As shown in Figure 3, LAH has already converged on the 15-th and 17-th epoch, and obtained the best MAP of 93.43% and 82.40% respectively. However, at the same epoch, ML-GCN never converges and its MAP scores are respectively 72.91% and 38.39% lower than LAH on VOC2007 and MS-COCO.

1 https://pytorch.org/

Comparisons with State-of-the-Arts

We compare the retrieval performance of LAH with existing multi-label hashing methods including DSRH [Zhao et al., 2015], IAH [Lai et al., 2016], OLAH [Huang et al., 2018], RCDH [Ma et al., 2018], T-MLZSH [Zou et al., 2019], and the state-of-the-art hashing methods including DCH [Cao et al., 2018], GCNH [Zhou et al., 2018] and DistillHash [Yang et al., 2019].

The Mean Average Precision within Hamming Radius 2 (MAP@H≤2) of all comparison methods are listed in Table 1. Experimental results show LAH has stable advantages over other algorithms at different code lengths and averagely outperforms the runner-up by 1.37%, 0.77% and 1.21% in terms of MAP@H≤2 on VOC2007, MS-COCO and FLICKR25K respectively.

Note that, we find both the MAP@H≤2 values of DCH and DistillHash will greatly decline at 64 bits and 128 bits, but this phenomenon becomes not obvious on those hash algorithms specially designed for multi-label datasets. This shows that the well-designed multi-label hash algorithms can carry more semantic information with longer hash codes. Fortunately, our proposed LAH owns this ability. At the same time, it can be seen that GCNH, whose difference from others is to use GCN to complete the image feature extraction, also owns this ability. We believe GCN has played a vital role in this process and our LAH also benefits from it. In addition, LAH outperforms GCNH owing that we introduce the co-occurrence relationship into GCN.

The performance of Precision within Hamming Radius 2 (P@H≤2) reflects the proportion of retrieved images related to the query image. As shown in Figure 4(a), (b) and (c), LAH achieves the highest P@H≤2 results on all the three benchmark datasets at different code lengths, and averagely exceeds the runner-up by 1.12%, 0.94% and 0.22% on VOC2007, MS-COCO and FLICKR25K respectively. This result verifies the superiority of LAH in feature extraction, because preciser classification features can bring preciser hash codes.

Note that R@H≤2 reflects the aggregation degree of similar images in the Hamming space. As shown in Figure 4(d), (e) and (f), LAH also achieves the highest R@H≤2 results on all the three datasets, and averagely exceeds the runner-up by 6.45%, 4.16% and 1.36% respectively. This result verifies the superiority of LAH in aggregating similar data, which benefits from the design of hash function. The visualization results are displayed in Section 3.6.

3.5 Ablation Study

We present the influence on the performance of LAH using different components. Word embedding method (WEM) is a LAH component used to acquire initial word vectors for labels. We compare the performance under GloVe [Pennington et al., 2014] with other popular counterparts including GoogleNews [Mikolov et al., 2013] and FastText [Joulin et al., 2016]. Fusion method (FM) is a LAH component used to fuse the label information and image feature. We compare the performance under MFB and DP which is used in MLGCN. As shown in Table 2, the precision results under MFB at different code lengths are respectively 12.03%, 2.40% and 6.32% higher than that under DP on VOC2007, MS-COCO and FLICKR25K. In addition, LAH produces better performance using GloVe than other WEMs, and respectively outperforms them by 3.69% and 4.06% over GoogleNew and FastText. In addition, LAH averagely outperforms the approach that uses neither WEM nor FM by 5.67% in all cases. From these results, it can been seen that the combination of components we designed is effective and optimal.

3.6 Visualization Study

Figure 5 shows the t-SNE visualization of the hash codes learnt by LAH and our baseline RCDH on VOC2007 dataset. LAH can not only better distinguish different categories but also gather the same categories more compactly. Although RCDH is equally excellent, its effect is not as good as LAH.

4 Conclusion

In this paper, we propose LAH, a label-attended hashing algorithm for multi-label image retrieval. LAH adopts cooccurrence probability over the label set to explore the interdependency between objects and uses GCN to learn the cooccurrence embeddings. In the following, LAH utilizes MFB to fuse the image representations and label co-occurrence embeddings in an end-to-end manner, which integrates the multi-modal co-attention as well as promotes the learning efficiency. Extensive experiments on VOC2007, MS-COCO and FLICKR25K demonstrate LAH has a high convergence rate and can generate high-quality hash codes and achieve better retrieval results than the state-of-the-art hashing methods.

Acknowledgments

This work is supported by the National Natural Science Foundation of China No.61902135 and the Innovation Group Project of the National Natural Science Foundation of China No.61821003. Wuhan is a great city whose residents are all heroes. We will surely overcome the COVID-19 and harvest freedom in the end.