Object Detection based Deep Unsupervised Hashing

1 Introduction

With the rapid growth of image data, approximate nearest neighbour (ANN) search have attracted more and more attention from researchers in the large-scale image search area. Among the existing ANN search techniques, similaritypreserving hashing methods are advantageous due to their high retrieval efficiency and low storage cost. The main idea of hashing methods is to transform high dimensional data points into a set of compact binary codes, meanwhile, maintain similarity of the original data points. Since the original data points are represented by binary codes instead of real valued features, the searching time and storage cost can be dramatically reduced.

In general, data-dependent hashing can be divided into unsupervised [Gong and Lazebnik, 2011; Huang et al., 2017; Ghasedi Dizaji et al., 2018; Lin et al., 2016] and supervised [Wang et al., 2018; Qiu et al., 2017] methods. The unsupervised hashing methods mainly utilize the features of images to generate similarity-preserving binary codes without any supervised information. Compared with unsupervised hashing, supervised hashing methods incorporate semantic labels of training data into training process, thus they can perform more remarkably in generating similarity-preserving binary code. However, in many real applications, there are no semantic labels for images, and supervised hashing methods cannot be used in these cases Intuitively, if we can detect the objects in images and use their corresponding classes as the pseudo-labels of the images, then we can use the pseudo-labels as 'supervised information' to guide hash codes learning to obtain better performance. An illustrative example is shown in Figure 1, the block (i) is the procedure of existing unsupervised hashing methods. They use the hand-crafted or learnt features as inputs, and then they directly use the Euclidean distance between images or the similarity between one image and its rotated image to guide the hash training, which will make the existing unsupervised hashing models to judge that the images Ia and Ib are dissimilar with high possibility. Actually, the images Ia and Ib are similar, because the two images belong to the class 'car'. On the contrary, in the block (ii), if we first use an object detection model to get the pseudolabels 'car'of the two images by detecting objects inside an image. By utilizing the pseudo-labels, we can construct pairwise similarity to guide hashing learning and make the hashing models to judge that the two images Ia and Ib are similar with high possibility.

∗The author is corresponding author

Inspired by this idea, we propose a novel Object Detection based Deep Unsupervised Hashing model, called ODDUH. In particular, an object detection model is first pre-trained on a large database. Then, we utilize the object detection model to mine latent semantic 'label information' ( i.e., pseudolabels) from images. Note that some of the pseudo-labels obtained by object detection are wrong because the precision of object detection is not 100%, and the noisy label will damage the final hashing performance. In order to decrease the harm of noisy pseudo-labels, we define a novel similarity criterion called pair-wise percentage similarity. Furthermore, a shared CNN is introduced to capture the feature representations of images. Finally, the pair-wise percentage similarity and the learnt feature representations of images are used to learn hash functions which can generate high-quality similarity-preserving hash codes. Extensive experiments on two real-world public datasets illustrate that our method outperforms the state-of-the-art unsupervised hash methods in image retrieval task.

2 RelatedWork

Generally, existing hashing methods can be divided into two categories: data-independent hashing and data-dependent hashing. For data-independent hashing methods, the hashing functions depend on random projection without any training data. The representative data-independent methods include Locality Sensitivity Hashing (LSH) [Gionis et al., 1999] and its variants [Datar et al., 2004; Kulis et al., 2009]. For datadependent hashing methods, they can achieve better accuracy with shorter codes by learning hash functions from training data. Furthermore, data-dependent methods can be mainly classified into two categories: supervised [Wang et al., 2018; Qiu et al., 2017] and unsupervised [Gionis et al., 1999; Jin et al., 2014; Ghasedi Dizaji et al., 2018] methods.

The supervised data-dependent methods can achieve remarkable performance by utilizing labeled data to learn hashing functions. The representative methods in this category include Deep Supervised Hashing with Pairwise (DPSH) [Li et al., 2015], HashNet [Cao et al., 2017] and Deep Semantic Hashing with GANs (DSH-GANs) [Qiu et al., 2017], etc. DPSH utilizes pair-wise similarity to guide hash functions learning in an end-to-end deep network. HashNet alleviates data imbalance that the number of similar pairs is much smaller than the number of dissimilar pairs by adjusting the weights of semantic similarity matrix to learn discriminative hashing codes. DSH-GANs introduces generative adversarial network (GAN) to generative fake images and construct triplet with the fake images and real images to supervise hash functions learning process.

The unsupervised data-dependent methods can be divided as traditional unsupervised hashing methods and deep unsupervised hashing methods. The traditional unsupervised hashing methods use hand-crafted features and shallow hash functions to obtain binary hash code. Lots of algorithms in this category have been proposed, including Spectral Hashing (SH) [Weiss et al., 2009], and Iterative Quantization (ITQ)

[Gong and Lazebnik, 2011]. However, limited by the handcrafted features and shallow hash functions, it is hard for them to deal with complex and high dimensional real-world data and keep the semantic similarity between original data in the binary hash codes. The deep unsupervised hashing methods utilize deep architecture to extract image features to learn hash code. For example, Deepbit [Lin et al., 2016] get rotation invariant and balanced binary hash codes by defined a quantization loss. Unsupervised triplet hashing (UTH) [Huang et al., 2017] employs an unsupervised triplet loss to get balanced hash codes. HashGAN [Ghasedi Dizaji et al., 2018] generate compact hash codes by a generative adversarial hashing network.

However, few existing unsupervised hashing methods try to mine the latent semantic 'label information' in the images. Thus, in this paper, we propose a novel deep unsupervised hashing model based on object detection to generate highquality hash codes by incorporating the latent semantic 'label information' mined by object detection into the training process.

3 Object Detection based Deep Unsupervised Hashing Method

In this section, we will present the proposed Object Detection based Deep Unsupervised Hashing Network (ODDUH) in detail. We first give a description of model notation in section 3.1. The whole architecture of ODDUH will be introduced in section 3.2. Then we discuss the detail of similarity definition and the object function in section 3.3 and section 3.4, respectively. Finally, we will introduce the learning of parameters in section 3.5.

3.1 Notation

Suppose a dataset has n images X = {xi}ni=1 , and the ith image is xi. The goal of similarity-preserving hashing is to learn a mapping H : xi → bi ∈ {-1, 1}k , where k is the length of hashing codes, such that an input image xi will be encoded into a k-bit binary code bi.

3.2 The Architecture of ODDUH

As shown in Figure 2, our architecture consists of three parts: latent semantic 'label information' mining, feature learning and hash function learning.

In the latent semantic 'label information' mining part, the ODDUH uses a pre-trained object detection model named YOLOv2 [Redmon and Farhadi, 2017] to mining the latent semantic 'label information' in images. Note that other stateof-the-art object detection models can also be used here, such as SSD [Liu et al., 2016] and Mask R-CNN [He et al., 2017].

The feature learning part includes a convolutional neural network (CNN) component and two fully-connected layers. The CNN component contains five convolutional layers. Particularly, the first convolutional layer filters the input images with 96 kernels of size 11 × 11 with a stride of four pixels. The output of the first convolutional layer will be responsenormalized and maxpooled (size 2 × 2) to be the input of the second convolutional layer. The second layer has 256 kernels of size 5 × 5 with a stride of one pixel and a pad of size

2 pixels. Its output will be response-normalized and maxpooled (size 2 × 2) to be the input of the third convolutional layer. The third, fourth and fifth layers have 384 kernels of size 3 × 3 with a stride of one pixel and a pad of size one pixel. The fifth layer has a max-pooling layer with filter of size 2 × 2. After the CNN component, the architecture holds two fully-connected layers which have 4,096 hidden units. The activation function used in this part is Rectified Linear Units (ReLu) [Krizhevsky et al., 2012].

The hash function learning part is a hashing layer which has k units. Furthermore, k is the length of hash code, and the hashing functions are learnt by the hashing layer. Eventually, we use element-wise sign function sgn(·), which returns 1 if the element is positive and returns -1 otherwise, to process the outputs of the hashing layer and get the binary code b.

3.3 Pair-wise Percentage Similarity

In ODDUH, we use an object detection model to mine the objects in images and get their classes (i.e., pseudo-labels). Note that some of the pseudo-labels obtained by object detection are wrong because the precision of object detection is not 100%. If two images share at least one pseudo-label, then we let the similarity of them equal to 1, and equal to 0 otherwise(i.e., two-value similarity). It will easily make two dissimilar images as a similar pair which will damage the final hashing performance. Thus, in order to decrease the harm of noisy pseudo-labels, a novel pair-wise percentage similarity is defined as:

where 〈li, lj〉 calculate the inner product and li ∈ {0, 1}c is the pseudo-label vector of xi, where c is the total number of classes that pseudo-labels belong to. If ith image xi has the

By incorporating the pair-wise percentage similarity into the training process, the learnt binary codes B = {bi}ni=1 can preserve the similarity in S = {sij |i, j ∈ {1, 2, . . . , n}, sij ∈ [0, 1]}. More specifically, if sij = 0, the binary codes bi and bj should have large Hamming distance; If sij = 1, the binary codes bi and bj should have a small Hamming distance; Otherwise, the binary codes bi and bj should have a suitable Hamming distance complying with the similarity sij .

3.4 Objective Function

Given the binary codes B = {bi}ni=1 for all the images, we can define the likelihood of the pair-wise percentage similarity sij as:

where tij = 2sij - 1, Ψij = 1 2 bTi bj, and σ(x) = 1 1+e-x . When sij = 0 or 1, we take the negative log-likelihood of the observed pair-wise labels in S to measure the pair-wise similarity loss; When 0 < sij < 1, we take mean square error to measure the pair-wise similarity loss. Thus, the pair-wise similarity loss function can be defined as:

where α is a hyper-parameter. Iij is used to denote two conditions, which is defined as:

By minimizing Eq. (3), we can make the hamming distance between two completely similar points as small as possible, and simultaneously make the hamming distance between two dissimilar points as large as possible. Meanwhile, we can make the partly similar images xi and xj have the suitable hamming distance complying with the similarity sij .

However, Eq. (3) is a discrete optimization problem, which is difficult to solve. Following the idea of previous work [Li et al., 2015], we relax b from discrete to continuous, and then reformulate Eq. (3) as:

where Θij = 12 uTi uj . ui ∈ Rk is the outputs of hashing layer: ui = WT F(xi; θ) + v, where the mapping F : Rd → R4096 is parameterized by θ, and θ represents the parameters of the seven layers of network in the feature learning part. W ∈ R4096×k is the weight matrix to be learnt at the hashing layer, v ∈ Rk is the bias. Due to the ui is not the binary codes, we used a quantization loss to make ui to be close to binary codes. The quantization loss is defined as:

Then, by connecting the pseudo-label pair-wise similarity loss and quantization loss, the final objective function can be defined as:

where β is a hyper-parameter.

3.5 Learning

In our method, there are four types of parameters B,W, θ, v need to be learnt, during the training phase. A mini-batch gradient descent method is used as optimization algorithm. Moreover, we use an alternating method to learn the parameters. More specifically, we optimize one parameter with other parameters fixed.

With parameters W, θ, v fixed, the bi can be directly optimized as follows:

With bi fixed, the other parameters W, θ, v are learned by standard back-propagation algorithm. Especially, we are able to compute the derivatives of the loss function about ui as follows:

Then, we can use the standard back-propagation algorithm to update W, θ and v with Eq. (9):

4 Experiments

4.1 Datasets and Baselines

We conduct experiments on two public benchmark datasets: Pascal VOC 2007 1 [Everingham et al., 2010] and BMVC 2009 2 [Allan and Verbeek, 2009]. Pascal VOC 2007 consists of 9,963 multi-label images. There are 20 object classes in this dataset. On average, each image is annotated with 1.5 labels. BMVC 2009 contains 96,378 images collected from Flickr. Each image in the dataset is associated with one or multiple labels in 20 semantic concepts.

Our proposed method is an unsupervised method, thus we compare our method with eight classical and state-of-the-art unsupervised hashing methods including: LSH [Gionis et al., 1999], ITQ [Gong and Lazebnik, 2011], SH [Weiss et al., 2009], PCAH [Wang et al., 2010], SGH [Jiang and Li, 2015], UH BDNN [Do et al., 2016], UTH [Huang et al., 2017], and HashGAN [Ghasedi Dizaji et al., 2018], where LSH, SH, ITQ, PCAH and SGH are traditional unsupervised methods and the other three are deep unsupervised methods. Note that the five traditional unsupervised hashing methods use handcrafted features as inputs, i.e., each image in Pascal VOC 2007 and BMVC 2009 is represented by a 512-dimensional GIST vector. For the deep unsupervised hashing method UH BDNN, it use the outputs of the fc7 layer in AlexNet as image representation. For the other two deep unsupervised hashing methods and our proposed method, we resize all the images to be 224 × 224 pixels and then directly use the raw image pixels as input. When carrying out experiments on the two datasets respectively, we randomly select 2,000 images as test set and the left images as training dataset. Moreover, in order to proves that the learnt representations by deep network from raw images are more superior than hand-crafted features in hash learning procedure, we also conduct the experiments by using the outputs of the fc7 layer in AlexNet [Krizhevsky et al., 2012] as image representation in the five hand-crafted feature based hashing approaches and denote them as LSH+CNN, SH+CNN, ITQ+CNN and PCAH+CNN, respectively.

4.2 Implementation Details

For the object detection component, we choose YOLOv2 [Redmon and Farhadi, 2017]. It is pre-trained in COCO 2014 dataset which contains 81 object classes. Please note that all the object classes contained in Pascal VOC 2007 and BMVC

1 http://host.robots.ox.ac.uk/pascal/VOC/voc2007/

2 http://pascal.inrialpes.fr/data2/flickr-bmvc2009/

2009 are subnet of the 81 object classes. All the weights and bias in the feature learning part and hash function learning part are learned via back-propagation algorithm. Furthermore, the weights and bias in the feature learning part are initialized as the values pre-trained in Alexnet [Krizhevsky et al., 2012]. We adopt SGD with a mini-batch size of 128 as our optimization algorithm. The learning rate is initialized as 0.01. The hyper-parameters α, β in ODDUH are empirically set as 2 and 100, respectively, and will be discussed in section 4.5. And the learning rate is adjusted to one tenth of the current learning rate every one third of epoches.

4.3 Evaluation Criterions

To verify the effectiveness of learned hash codes, we evaluate the image retrieval quality for different methods by Average Cumulative Gains (ACG) [J¨arvelin and Kek¨al¨ainen, 2000], Normalized Discounted Cumulative Gains (NDCG) [J¨arvelin and Kek¨al¨ainen, 2002], Mean Average Precision (MAP) [Baeza-Yates et al., 2011], Weighted Mean Average Precision (W-MAP) [Zhao et al., 2015] and Precision at top n retrieved images (Precision@k).

4.4 Experimental Results

Table 1, Table 2 and Figure 3 summarize the comparative results of different hashing methods by MAP, WMAP, NDCG ACG and Precison@n over Pascal VOC 2007 and BMVC 2009, respectively. In general, from the two tabels and Figure

3, we have two observations: (1) Our proposed method substantially outperforms the other unsupervised hashing methods for different length of hash code. For example, on Pascal VOC 2007, comparing with the best traditional competitor ITQ+CNN on 48-bits, the results of ODDUH have a relative increase of 26.8% on MAP, 17.7% on WMAP, 16.7% on NDCG, 16.9% on ACG. Moreover, on BMVC 2009, comparing with the competitor UH BDNN on 48-bits, the results of ODDUH have a relative increase of 8.3% on MAP, 11.4% on WMAP, 14.2% on NDCG, 17.1% on ACG. In Figure 3, the precision@n curves show ODDUH can get the best performance. All these results obviously indicate that pseudo-labels mined from images are useful to improve the performance of hashing models; (2) The performance of the methods with deep features is better than the one of the methods with handcrafted features. For example, the MAP@1000 of ITQ+CNN over Pascal VOC 2007 is 0.6070 in 48 bits, and the corresponding value of ITQ is 0.3223.

In order to verify the denoisy ability of pair-wise percentage similarity on hash codes learning, an experiment is carried out over Pascal VOC 2007. As shown in Figure 4, it can be found that ODDUH with percentage similarity can get a better performance than ODDUH with two-value similarity. It verifies that pair-wise percentage similarity can decrease the harm of noisy pseudo-labels in hashing code learning procedure.

4.5 Sensitivity to Hyper-Parameters

Figure 5(a) shows the effect of the hyper-parameter α on 48 bits over Pascal VOC 2007 and BMVC 2009. It can be found that ODDUH is not sensitive to α on both datasets. For instance, ODDUH can achieve good performance on both datasets with 1 ≤ α ≤ 5. Figure 5(b) shows the effect of the hyper-parameter β on 48 bits over Pascal VOC 2007 and BMVC 2009. Also, ODDUH is not sensitive to β in a large range. For example, ODDUH can achieve good performance on both datasets with 80 ≤ β ≤ 150. And we can also obtain similar conclusion on other length of hash codes for both hyper-Parameters α and β.

5 Conclusion

In this paper, we have proposed a novel Object Detection based Deep Unsupervised Hashing method for unlabeled data, called ODDUH. To the best of our knowledge, ODDUH is the first method which try to mine semantic 'label information' from images to guide hashing function learning. By incorporating the semantic 'label information' into the training process, the learnt hashing functions can generate highquality similarity-preserving hash codes. Extensive experiments on two real-world public datasets have shown that the proposed ODDUH method can outperform the state-of-theart baselines in image retrieval application.

Acknowledgments

The work is supported by SFSMBRP(2018YFB1005100), BIGKE(No. 20160754021), NSFC (No. 61772076 and 61751201), NSFB (No. Z181100008918002), Major Project of Zhijiang Lab (No. 2019DH0ZX01), and CETC(No. w2018018).