TUCH: Turning Cross-view Hashing into Single-view Hashing via Generative Adversarial Nets

1 Introduction

While multimedia big data of massive volumes and high dimensions are pervasive in search engines and social networks, it has attracted increasing attention to approximate nearest neighbors search across different media modalities that brings both computation efficiency and search quality. Since correspondence data from different modalities may endow semantic correlations, it has a tendency to support cross-view retrieval that returns relevant search results from one view as response to query of another view, e.g. retrieval of images of text query. An effective solution to cross-view retrieval is hashing method that learns compact binary codes with similar binary codes for similar objects from high-dimensional data. This paper focuses on cross-view hashing that builds isomorphic hash codes for efficient cross-view retrieval. So far, effective and efficient cross-view hashing remains a big challenge, because of the heterogeneity across different views [Wei et al., 2014], and the semantic gap between low-level features and high-level semantics [Smeulders et al., 2000].

∗This research was supported by the National Natural Science Foundation of China (No. 61571269) and the Royal Society Newton Mobility Grant (IE150997). Corresponding author: Guiguang Ding.

Several recent models for cross-view hashing [Bronstein et al., 2010; Zhen and Yeung, 2012; Lin et al., 2015; 2016; Wu et al., 2015; Cao et al., 2016a; Ding et al., 2016; Guo et al., 2016] have followed the same multi-view embedding framework, which integrates both image data and text data into an independent semantic embedding space. The multi-view embedding framework basically learns projection matrixes from image space and text space to the embedding space as Figure 1 (a). Due to this unnecessary intermediate transformation, the information leak is unavoidable. Intuitively, a single-view problem may be easier to solve than a cross-view problem. And if we can convert a cross-view hashing problem into a single-view hashing problem on a specific domain, at least the information loss on this domain will be minimized. Generative Adversarial Networks (GANs) approach is originally used for text to image synthesis [Reed et al., 2016], which provides a ready-to-use solution to convert text data into image domain, making it possible to solve cross-view hashing problem with single-view method.

In this work, we strive for the goal of efficient cross-modal retrieval of images in response to natural sentence queries or vice versa. Inspired by the idea of GANs, this paper presents a new model which turns cross-view hashing into single-view hashing on image domain with NO multi-view embedding required, whose overview is illustrated in Figure 1 (b). It will keep the information of image as much as possible and result in good aggregation characteristics of image features so that we can achieve a better retrieval result in cross-view retrieval. TUCH architecture constitutes a language model network T for text feature extraction, a generator network G to generate fake images from text feature and a discriminate hashing network H for learning hashing functions, eventually generating compact binary codes, while distinguishing correlation of image and text. Net G and net H constitute an end-to-end multi-task deep learning model to train generative adversarial model and hashing function simultaneously. Most of the deep hashing models for image retrieval tried to add a regularization term to minimize the quantization error into loss function [Lai et al., 2015; Zhu et al., 2016; Cao et al., 2016b] or insert a gradient snapping layer into the network to achieve it [Liu and Lu, 2016]. But the quantization error cannot be avoided in those frameworks. We add a discrete layer in our TUCH framework which directly outputs binary codes for loss computation and adopt a simple backpropagation algorithm for optimization. In summary, we make the following contributions in this paper:

Figure 1: Multi-view hashing framework embeds both image data and text data into an independent semantic hamming space while TUCH framework generates fake image with text to image synthesis method and applies single-view hashing method on the image domain (with generated fake images).

• We put forward a novel end-to-end deep learning approach termed as TUCH for cross-view hashing, which is the first to transfer multi-view problem into a singleview hashing problem in image domain with GANs.
• A multi-task architecture is adopted considering the tag information from the retrieval similarity and the description information for sentence to image generation.
• We add a discrete layer in TUCH, which directly outputs binary codes via loss computation. By doing so, the information loss during the feature quantization is avoided, which is unachievable in previous works. A simple and efficient algorithm is adopted for loss backpropagation at the discrete layer.

2 RelatedWork

2.1 Hashing for Cross-View Retrieval

Hashing is a widely used indexing technique for image retrieval [Wang et al., 2016]. Locality Sensitive Hashing [Gionis et al., 1999] can be regarded as the seminal hashing work which adopts random splits in the feature space to generate binary codes. Thereafter, a number of learning based hashing approaches are proposed, which fall into two categories: unsupervised hashing and supervised hashing. No supervision but only statistics information of data is taken into consideration in unsupervised hashing approaches, such as manifold structure [Weiss et al., 2008; Liu et al., 2014; Shen et al., 2015b], variance of features [Gong et al., 2013; Xu et al., 2013; Kong and Li, 2012] and cluster property [He et al., 2013]. On the contrary, supervised hashing approaches take advantage of the supervised knowledge so as to better capture the intrinsic semantic property of data, in which the representatives include Supervised Hashing with Kernels [Liu et al., 2012], Supervised Discrete Hashing [Shen et al., 2015a] and Latent Factor Hashing [Zhang et al., 2014].

Several recent models for cross-view hashing [Bronstein et al., 2010; Zhen and Yeung, 2012; Lin et al., 2015; Wu et al., 2015] have followed the multi-view embedding framework, which embeds both image data and text data into an independent semantic embedding space. The multi-view embedding framework basically learns projection matrixes from image space and text space to the embedding space.

Recent years, researchers are trying to apply deep neural networks in hashing problem. A two-stage training strategy was proposed that generates hash codes via a combination of disjoint CNN network and discrete code learning [Xia et al., 2014]. Alternatively, some end-to-end deep hashing models for image retrieval were proposed by minimizing the quantization error [Lai et al., 2015; Zhu et al., 2016] or inserting a gradient snapping layer into the network [Liu and Lu, 2016] while simultaneously minimizing the sample similarity loss. Benefiting from the power of CNN, these approaches achieve significant improvement over the traditional approaches, but they can only be applied to single-view retrieval. DVSH model [Cao et al., 2016a] is the first end-to-end deep learning approach for cross-view hashing that enables efficient crossview retrieval of images in response to sentence queries and vice versa. Following the multi-view embedding framework, it applies deep hashing method in cross-view retrieval problem and achieves the state-of-the-art results. However, the projection from image view and text view to the semantic embedding space will cause the information loss in both views and poor cross-view retrieval result. That is to say, both deep and non-deep multi-view embedding based cross-view hashing methods still have room for improvement.

2.2 Generative Adversarial Networks (GANs)

Two challenges in multi-modal learning include: 1) learning a shared representation across modalities, and 2) estimating missing data (e.g. by retrieval or synthesis) in one modality conditioned on another. Generative adversarial networks (GANs) [Goodfellow et al., 2014] have benefited from convolutional decoder networks for the generator network module. A Laplacian pyramid of adversarial generator and discriminators is used to synthesize images at multiple resolutions [Denton et al., 2015]. A standard convolutional decoder incorporating batch normalization is adopted [Radford et al., 2015], which developed a highly effective and stable architecture to achieve striking image synthesis results. An endto-end differentiable architecture from the character level to pixel level is proposed using model conditions on text descriptions to achieve sentence text to corresponding image synthesis [Reed et al., 2016], which provide a good way to convert sentence text into fake image, making it possible to transfer a cross-view sentence-image retrieval problem into a single-view problem. In this work we adopt a multi-task GAN architecture which aims to generate fake image with tag features from sentence text, so that we can embed text domain into image domain for a single-view hashing method.

3 Background

3.1 Generative Adversarial Networks (GANs)

Generative adversarial networks (GANs) consist of a generator G and a discriminator D that compete in a two-player minimax game: The discriminator tries to distinguish real training data from synthetic images, while the generator tries to fool the discriminator. Specifically, D and G play the following game on V(D,G):

where x represents image sample from dataset, z represents random noise vetor. It is proved that this minimax game has a global optimum precisely when pg(z) = pdata, and that under mild conditions (e.g. G and D have enough capacity) pg(z) converges to pdata.

3.2 Conditional Generative Adversarial Networks for Text to Image Synthesis

Conditional Generative Adversarial Networks have been used for text to image synthesis by small modification over GANs [Reed et al., 2016] as below:

where t represents the feature of sentence text, x refers to true image sample relevant to t, ˆx represents image sample inrelevant. In brief, generator generates a fake image with random noise and text feature to fool the discriminator, and the discriminator tries to distinguish real text-relevant training data from synthetic images and irrelevant ones.

4 Turning Cross-viewHashing into Single-view Hashing via GANs

4.1 ProblemDefinition

The definition of cross-view hashing between image and sentence based text in this paper follows [Cao et al., 2016a]. We are given n training images{I1,. . . ,In} and each image Im has ktm visual concept tags and a set of kdm sentence descriptions Dm = {D1, . . . ,Dkdm }. Each visual concept tag belongs to set T = {T1, . . . , TK}, where K is the size of T . For each image there is a label vector ym ∈ {0, 1}K where ymj = 1 if Im has tag Tj and ymj = 0 otherwise. The set of all descriptions is D = ⋃ni=1 Di. We aim to learn hashing models hI : I → {-1, 1}KH and hD : D → {-1, 1}KH to map image and sentence to KH-bit codes. It is desired that hI and hD can preserve the semantic similarity in visual and text domains in the same Hamming space {-1, 1}KH .

4.2 Network Architecture

We use the following notation. The generator network is denoted G: RZ × RKt → RI . The discriminative hashing network as H: RI × RKt → {0, 1} × RK , where Kt is the dimension of the text description encoding. I is the dimension of the image, and Z is the dimension of the noise input to G. We illustrate our network architecture in Figure 2.

In generator G, first we sample from the noise prior z ∈ RZ ∼ N(0, 1) and we concatenate the text query encoding t with z. Following this, inference proceeds as in a normal deconvolutional network: we feed-forward it through the generator G; a synthetic image ˙x is generated via ˙x ←G(z, t). Image generation corresponds to feed-forward inference in the generator G conditioned on query text and a noise sample.

In the discriminative hashing network H, we perform a CNN network with multi-task objective functions. There are two input branches,, where the first one is an image branch with a standard 18 layers Deep Residual Network (Resnet) [He et al., 2016], while the other is a fully-connected text branch. The discrete layer which outputs binary hash codes follows right after the last fully-connected layer (FC-I) of the image branch. The semantic embedding loss is calculated by the discrete layer. We concatenate the last layer of text branch with FC-I and put a discriminative loss after it. The semantic embedding loss is to preserve the similarity of both image domain and text domain. The discriminative loss is to judge if an image (real or fake) is relevant to the given text encoding.

4.3 Loss Function

Semantic Embedding Loss. Given an input image Im, suppose the output of the hashing layer hm ∈ {-1, 1}KH is the KH-bit hash codes. Then the hash codes are projected to the final semantic vector space by the parameter Wh and the output is denoted as φm. The semantic embedding loss mostly focuses on improving the generalize ability of the hashing model. Meanwhile, it is also desired that the samples with common concept have very similar hash codes whereas samples without common concept have dissimilar hash codes. The image branch of H can be regarded as a single-view hashing network. The objective function of G and H is the same. That is to say, G aims to generate similarity-preserving fake image and H aims to extract similarity-preserving hash codes. We take Wh as a projection matrix from Semantic Hamming space {-1, 1}KH to the tag space RK . The semantic embedding loss is as follow:

where ym refers to the label vector of Im, φm = Whhm ∈ RK , hm is the KH bit hash code of Im.

Discriminative Loss. Given a text encoding t, suppose the image relevant to t is x, the image irrelevant to t is ˆ x. The discriminative branch of H is referred as HD : RI × RKt → {0, 1}.

The generator G tries to generate fake images to fool the discriminative branch of H. The discriminative branch of H

tries to distinguish real relevant images from synthetic images and real irrelevant images. According to Equ (2), we can get LH and LG from V (HD,G), which donate the discriminative loss to be minimized for generator G and discriminative hashing network H.

The combined multi-task loss is as below:

4.4 Optimization

With the overall loss function, we adopt the backpropagation algorithm with mini-batch stochastic gradient descent method to train the network. The optimization is a 2-step procedure, in which we optimize G and H in turn. Algorithm 1 summarizes the training procedure.

There is no other problem during backpropagation except that we adopt a discrete hashing layer in the network whose discretion operation by sign function is non-differentiable at 0 and the derivative at the other part is also zero such that the gradient vanishes when propagated through this layer. To address this issue, we adopt the straight-through estimator to compute the gradients. Specifically, suppose the input of the hashing layer is r ∈ RKH , the output of the layer is b where bi = sign(ri). According to the chain rule we can obtain the gradient ∂L ∂r = ∂L ∂b ∂b ∂r = 0 , because ∂b ∂r = 0. We adopt the following straight-through estimator to propagate the loss through the hashing layer instead.

Algorithm 1 Training algorithm with step size α.

Input: minibatch text encoding t, matching images x, mismatching images ˆx, number of training batch steps S

Output: generator G and discriminative hashing network H

• 1: procedure TRAIN(α, t, x, ˆx, S)
• 2: Init(G);Init(H);
• 3: for i = 1 to S do
• 4: z ∼ N(0, 1)Z {Draw sample of random noise}
• 5: ˙x ← G(z, t) {Forward through generator}
• 6: Calculate LH with x, ˙x, ˆx, t, according to Equ (6)
• 7: H ← H - ∂LH ∂H {Update H}
• 8: Calculate LG with ˙x, t, according to Equ (7)
• 9: G ← G - α ∂LG ∂G {Update G}
• 10: end for
• 11: return G, H;
• 12: end procedure

Moreover, when |ri| > 1 , we set the gradient to 0. To be clear, we consider |ri| as the confidence coefficient of bi = sign(ri). If the confidence coefficient is large enough while training, we don't try to change this bit. Obviously, when we take output of the discrete hashing layer directly as the hash codes, the quantization error is 0. And the loss is computed directly by the binary codes, so that the loss can reflect fitting the degree of hashing codes on the training set well.

5 Experiments

We conduct extensive experiments to evaluate the efficiency of the proposed TUCH model with several state-of-the-art hashing methods on two widely-used benchmark datasets.

5.1 Data Preparation

The evaluation is conducted on two benchmark cross-view datasets: Microsoft COCO [Lin et al., 2014] and IAPR TC12 [Grubinger et al., 2006].

Microsoft COCO The current release of the recently proposed dataset contains 82783 training images and 40137 validation images. For each image, it provides at least five sentences annotations, belonging to 80 most frequent categories as ground truth labels. After pruning images with no category information, we get 82081 training images and randomly sample 5000 query images from the validation set along with their meta data.

IAPR TC-12 This dataset consists of 20000 images collected from a wide variety of domains, such as sports and actions, people, animals, cities, landscapes and so on. There are at least 1 sentence annotations for each image. Besides, it provides category annotations generated from segmentation tasks with 275 concepts. We prune the original IAPR TC-12 to form a new dataset, which consists of 18673 images with 22 most frequent concept tags.

5.2 Baseline and Evaluation Setup

We compare the cross-view retrieval performance of our approach with nine state-of-the-art cross-view hashing methods including three unsupervised methods IMH [Song et al., 2013], CVH [Kumar and Udupa, 2011] and CorrAE [Feng et al., 2014], and six supervised methods CMSSH [Bronstein et al., 2010], CM-NN [Masci et al., 2014], SCM [Zhang and Li, 2014], QCH [Wu et al., 2015], SePH [Lin et al., 2015] and DVSH [Cao et al., 2016a], in which CorrAE,CM-NN and DVSH are deep methods.

We follow [Wu et al., 2015; Zhang and Li, 2014; Cao et al., 2016a] to evaluate the retrieval performance with Mean Average Precision (MAP), precision-recall curves, and precision@top-R curves. We adopt MAP@R=500 following the baseline method [Wu et al., 2015; Cao et al., 2016a]. And the sampled training set includes 5000 images along with all labels and sentences annotations for both Microsoft COCO and IAPR TC-12.

For Microsoft COCO, we randomly select 5000 images with annotations as training set for hashing, 1000 images with annotations as validation set and 1000 images with annotations as query set. For IAPR TC-12, we randomly select 5000 images with annotations as training set for hashing, 1000 images with annotations as validation set and 100 images with annotations per class as query set. The pairwise similarity labels for training are randomly constructed using semantic labels or concepts, and each pair is considered similar (dissimilar) if they share at least one (none) semantic label.

5.3 Training Details

We implement the TUCH model in the open-source Torch7 framework. We take the text encoding model in [Reed et al., 2016] to extract text encoding t. For training network, the generator G is a normal deconvolutional network as stated in [Reed et al., 2016], and we employ the ResNet architecture as the image branch of the discriminative hashing network. We take pre-trained 18 layers ResNet as the initial value of the image branch to fine-tune.

For both datasets, we first train the unsupervised text encoding model with all the sentences descriptions except for query set. And then we train an unsupervised GAN model with G and discriminative sub-network of H. The training data is all the sentence-image pairs without tag except for query set. Finally, we add the hashing layer and the semantic embedding loss into H along with the discriminative loss, and perform a supervised training on small labeled training set. The TUCH approach involves 4 penalty parameters λ1, λ2, λH and λG for trading off the relative importance of irrelevant image, fake image, discriminative loss for H and discriminative loss for G in Equ (6) and Equ (7). And we can achieve good results with λ1 = λ2 = 0.5 and λH = λG = 0.01.

5.4 Results and Discussions

We compare our approach TUCH with the nine baseline methods on the two datasets in term of MAP, precision-recall curve and precision@top-R curves of two cross-view retrieval tasks: image query on sentence database (I → T), and sentence query on image database (T → I).

We evaluate all methods with 16, 32, 64 and 128 bits hash codes. The results are reported in Table 1. From the experimental results, we can observe that TUCH nearly outperforms all baseline methods for most cross-view tasks on the benchmark datasets. Specifically, compared to the state-of-theart sentence-image hashing retrieval baseline DVSH, TUCH achieves absolute increases of 1.1%/3.9% and 0.4%/1.9% in average MAP of different code length for two cross-view retrieval tasks I → T and T → I on Microsoft COCO and IAPR TC-12 datasets. The performance of unsupervised baseline methods differs on the two benchmark datasets. Compared to the best unsupervised method IMH on Microsoft COCO, TUCH get absolute increases of 5.6% and 6.3% on I → T and T → I tasks, while the data on IAPR TC-12 is 11.8% and 14.4% which is compared with CorrAE. We can observe that TUCH gains more increases in the T → I task on both benchmark datasets. It is because the hashing function is directly over image domain, and the information of image is reserved as much as possible, so that hash codes of images shows good aggregation characteristics. Once the hash codes of texts are place in the right semantic position, the relevant images can be found easily.

The precision-recall curves with 32 bits for the two crossview tasks I → T and T → I on two datasets Microsoft COCO and IAPR TC-12 are shown in Figure 3, respectively. TUCH shows the best cross-view retrieval performance at all recall levels on T → I task, and almost best performance on I → T task. Figure 4 shows the precision@top-R curves of all comparison methods with 32 bits on the two datasets, which shows how the precision changes with the number R of top-retrieved results. TUCH outperforms other baseline method again and shows effectiveness and stability on different levels of R value.

6 Conclusion

This paper presents a novel deep model that is able to turn cross-view hashing into single-view hashing on image domain via GANs (TUCH) for sentence-image cross-view hashing, thus enabling the information of image to be preserved as much as possible. Our TUCH model converts sentences into images and then solves the multi-view hashing problem with a multi-task and end-to-end generative adversarial learning architecture. Besides, we add a discrete hashing layer in TUCH which directly outputs binary codes via loss computation, which avoids the information leak during the feature quantization. Comprehensive empirical evidence shows that our TUCH approach achieves state-of-the-art results, especially on text to image retrieval task, on imagesentences datasets, i.e. standard IAPRTC-12 and large-scale Microsoft COCO. In the future, we plan to extend TUCH on mobile computing, which relies on good model compression approach and parallel computing algorithm with high speed on mobile device.