Deep Unified Cross-Modality Hashing by Pairwise Data Alignment

1 Introduction

With the increasing multimedia data, approximate nearest neighbor (ANN) search has been a fundamental problem in the information retrieval area. Among several ANN search methods, hashing [Wang et al., 2018; Chen et al., 2019; Wang et al., 2020] has attracted extensive attention. It maps data points to binary codes with hash functions by preserving the similarity in the original space of data points. Due to binary codes, the storage cost can be dramatically reduced with sublinear search complexity. In many applications, such as search engines, systems have to handle data of multi-modalities. It requires researchers to support cross-modality retrieval that can return relevant results of one modality when querying another modality, e.g., retrieving images with text queries. Due to its low storage cost and search time, cross-modality hashing (CMH) area receives more and more attention recently.

∗Lijun Zhang is the corresponding author.

Existing CMH methods can be roughly divided into shallow CMH methods [Zhai et al., 2013; Wu et al., 2015; Xie et al., 2016; Ye and Peng, 2018] and deep CMH methods [Jiang and Li, 2017; Yan et al., 2017; Li et al., 2018; Shi et al., 2019; Sun et al., 2019; Xu et al., 2019]. Most shallow methods capture the semantic relevance in a common Hamming space and learn hash functions that map hand-crafted features to hash codes, which means the feature extraction procedure is independent of the hash code learning procedure. That may block the way to achieve satisfying performance, as the hand-crafted features might not be optimal for the hash code learning procedure. Recently, deep learning [Zagoruyko and Komodakis, 2016] has shown its superiority of representation learning in various applications, such as image recognition [He et al., 2016]. Deep CMH methods leverage the power of deep learning by integrating feature learning and hash code learning into a single framework. As a result, deep methods capture non-linear correlations among cross-modality instances and achieve better performance than shallow methods. The key to CMH is to bridge the modality gap, as it makes original data distributions and feature representations of modalities different. Thus, it is necessary to explore semantic relevance between different modalities in sufficient details to reduce that gap and further improve search performance. Recently, some supervised deep methods [Li et al., 2018; Zhan et al., 2020] exploit semantic labels or relevance information, thereby better cross-modality correlations.

However, previous work [Jiang and Li, 2017; Li et al., 2018; Sun et al., 2019] mainly aligns different modalities in the level of representation using pre-defined loss functions and derives independent hash functions and codes for different modalities. Representation (level) alignment may reduce the modality gap in some intermediate procedures of independent hash functions, but inputs, other layers and final outputs (hash codes) of deep hash functions are usually of different distributions. Actually, high-dimensional modality-specific original data contain abundant information that enables us to bridge the modality gap by the data (level) alignment. As we have paired data of different modalities, e.g., image-text paired data, leveraging that data to learn mappings between

1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 0 modalities (data alignment) is possible and may bridge the modality gap better than representation alignment. Besides, with the mappings between modalities, we can derive unified hash functions and learn unified hash codes for correlated data for different modalities at the same time, while existing work cannot. The difference between data alignment (unified hash codes and functions) and representation alignment (separated hash codes and functions) is presented in Figure 1.

In this paper, to better bridge the modality gap and derive unified hash functions and hash codes for different modalities, we propose a novel Deep Unified Cross-Modality Hashing (DUCMH) method. Specifically, DUCMH alternatively learns unified hash functions and unified hash codes, while aligning different modalities by data alignment under the supervision of image-text pairs at hand. Our unified hash function maps texts to hash codes, while with an auxiliary image to text mapping trained under the supervision of paired data, the unified hash function is also able to map images to hash codes. The auxiliary image to text mapping maximizes the semantic relevance and distribution consistency between different modalities, while the unified hash function is employed for unifying hash codes of different modalities and further reducing modality gap. To empirically evaluate our DUCMH, we conduct extensive experiments on three commonly used image-text datasets, showing the superiority of our method over several state-of-the-art methods.

The main contributions of DUCMH are summarized below:

• To the best of our knowledge, DUCMH is the first deep method that derives unified hash codes for database instances and unified hashing functions for unseen query points at the same time.
• DUCMH is the first deep method to apply data alignment and learn the mapping between modalities. Data alignment can carefully preserve semantic relevance and distribution consistency across modalities, while effectively bridging the modality gap.
• Extensive experiments conducted on three image-text benchmarks show the superiority of our DUCMH over several state-of-the-art cross-modality hashing methods, including both shallow and deep methods.

2 RelatedWork

Existing cross-modality hashing work can be divided into unsupervised and supervised methods. For unsupervised methods, the intra- and inter-modality relations are exploited to generate hash codes without any supervised information [Hotelling, 1992]. On the other side, with more semantic information, supervised methods are able to capture correlations better and thus achieve superior performance [Ding et al., 2014; Zhang and Li, 2014; Wang et al., 2015; Luo et al., 2018]. Semantic Correlation Maximization (SCM) [Zhang and Li, 2014] reduces the modality gap by utilizing label information to learn a modality-specific transformation, and preserves the maximal correlation between modalities. Collective Matrix Factorization Hashing (CMFH) [Ding et al., 2014] employs the latent factor model to learn hash codes by collective matrix factorization from different modalities. Semantic Topic Multimodal Hashing (STMH) [Wang et al., 2015] utilizes latent semantic information among different modalities to learn hash codes. Supervised Discrete Manifold-Embedded Cross-Modality Hashing (SDMCH) [Luo et al., 2018] generates binary hash codes by exploiting the non-linear manifold structure of data and constructs the correlations among heterogeneous multiple modalities with semantic information.

The above methods utilize hand-crafted features to generate hash codes, which might not be optimally compatible with the hash-code learning procedure. With the advances of deep learning, more and more deep supervised CMH work has been proposed. Deep methods [Jiang and Li, 2017; Li et al., 2018; Shi et al., 2019; Zhan et al., 2020] integrate feature learning and hash-code learning into a unified architecture with promising performance. Deep Cross-modality Hashing (DCMH) [Jiang and Li, 2017] is the first deep framework that performs feature learning and hash-code learning simultaneously. Self-Supervised Adversarial Hashing (SSAH) [Li et al., 2018] incorporates a self-supervised semantic network coupled with multi-label information, and carries out adversarial learning to maximize the semantic relevance and feature distribution consistency between different modalities. EquallyGuided Discriminative Hashing (EGDH) [Shi et al., 2019] takes semantic structure and discriminability into consideration to learn better hash functions and hash codes. Supervised Hierarchical Deep Hashing (SHDCH) [Zhan et al., 2020] explores the power of hierarchical labels to further boost the learning procedure.

While previous methods achieve success in performance by representation alignment, they ignore image-text pairs at hand, and never leverage that information to reduce the modality gap by data alignment or derive unified hash functions and codes simultaneously. To the best of our knowledge, DUCMH is the first deep method to employ data alignment by fully utilizing the image-text paired data for better reducing the modality gap and deriving unified hash functions and codes simultaneously.

3 DUCMH

In this section, we present the details about our deep unified cross-modality hashing (DUCMH) method, including model formulation and learning algorithm. DUCMH, shown in Figure 2, is an end-to-end learning framework consisting two parts: the feature learning part and the hash-code learning part. We employ asymmetric learning to alternatively update each part and data alignment to bridge the modality gap.

3.1 Problem and Notation

Scalars are defined by lowercase letters, such as w. Vectors and matrices are denoted by boldface lowercase letters and uppercase letters, e.g., w and W. The i-th element of w and i-th row, j-column element of W are represented by subscripts as wi andWij. Besides, W∗j andWi∗ represent the j-th column and i-th row of matrix W, while ˜W∗j and ˜Wi∗ represent the matrices of matrix W without j-th column and i-th row.

Without loss of generality, we focus on CMH using imagetext paired data. Assume that we have n training data points, in which each instance has two modalities of features, i.e., image and text. O = {oi}ni=1 is a cross-modality dataset with n instance, while for i-th instance oi = (xi, yi, li), xi, yi, and li = [li,1, . . . , li,clsnum] are the corresponding image, text and label, and clsnum is the number of classes. If oi belongs to the j-th class, li,j = 1, otherwise li,j = 0. Besides, we denote L = [l1, . . . , ln] ∈ {0, 1}n×clsnum. The similarity matrix S ∈ {-1,+1}n×n is generated by labels. We let Sij = 1 if oi and oj are semantically similar, in another word, share at least one label, otherwise Sij = -1.

Given the above training information O and S, the goal of cross-modality hashing is to learn two hash functions hx(·) ∈ {-1,+1}c for the image modality and hy(·) ∈ {-1,+1}c for the text modality, where c is code length. Meanwhile, the similarity-preserving hash codes B = [b⊤1 , . . . , b⊤n ]⊤ ∈ {-1, +1}n×c should also be derived. Two hash functions are used for generating hash codes of the unseen data points while hash codes should preserve the cross-modality similarity in S. Specifically, when Sij = 1, the Hamming distance between the binary codes bi and bj should be small. Otherwise, the distance should be large.

3.2 Our Model - Learning by Paired Data Feature Learning - Unified Hash Functions

As aforementioned, previous work [Li et al., 2018; Shi et al., 2019] usually employs representation alignment, which only reduces the modality gap in some intermediate procedure of algorithms. On the other side, as the image-text pairs of database are at hand, it is possible to employ data alignment to directly bridge the modality gap by learning a mapping from image to text or from text to image.

DUCMH chooses to learn the mapping fi2t(·) from image to text. There are two reasons: first, it is easier to learn the mapping from image to text than another, as the image always contains more information, such as spatial information and styles, which are hard to be pictured by a sentence. Second, images might be influenced a lot by the lights, angles, and other conditions, while texts are more steady and the light, angle, spatial information of an image can be easily represented by a simple word. That mapping fi2t(·) can be learned under the supervision of image-text pairs.

With the mapping fi2t(·), we can represent an image x as a text fi2t(x). By employing an alignment loss presented in the following section, the modality gap between image and text can be better reduced than the representation alignment previous work employs.

For the real text y and predicted text fi2t(x), we use a unified hash function hy(·) mapping them to hash codes as,

To generate discrete binary hash codes, the sign function sign(·) is attached after the last layer of our unified hash function hy(·). However, as the gradients of sign(·) are 0 at most of points, we use tanh(·) instead to better train the unified hash function hy(·) and the image to text mapping fi2t(·).

Hash Code Learning - Unified Hash Codes

In this section, we present our objective function. As unified hash codes B should preserve semantic information and also be capable of classification, we design three corresponding losses shown below. Besides, to bridge the modality gap, we also employ an alignment loss to align different modalities for better search performance.

First, DUCMH leverages image-text pairs {(xi, yi)}ni=1 to learn a precise image to text mapping fi2t(·) for data alignment. Thus, to fully bridge the modality gap on the database points, we use alignment loss to model the modality gap, which is shown below,

where ϵ is a hyper-parameter.

Second, as the learned database hash codes and the hash codes generated by the hashing function should preserve semantic similarity, we employ the similarity loss [Zhang and Li, 2014]. Similarity loss requires that for any two hash codes, if they are semantically similar, the Hamming distance between them should be 0. Otherwise the Hamming distance should be c. Therefore, this loss we minimize is as:

where α is a hyper-parameter, bi and bj are the hash codes of i-th and j-th data point learned by the alternative learning algorithm shown in the following section.

Third, as the hash space is small, i.e., 2c points, we expect hash codes are able to preserve discriminative semantic information and recover labels with simple transformations. We use h∗(·)W and bW to represent the predicted label recovered from hash codes, where W ∈ Rc×clsnum. To simplify the formulation of this loss, we employ Mean Squared Error to measure the label information contained in hash codes. Now, the classification loss is defined as follows:

where α is a hyper-parameter defined above.

Last but not the least, as we replace sign(·) with tanh(·) after the last layer of our unified hash function hy(·) for better performance, hash functions is still expected to generate discrete hash codes. Intuitively, the quantization loss [Jiang and Li, 2017] is as:

where ρ is a hyper-parameter.

Now, by minimizing four losses, our objective function can be formulated as:

where Θ is the parameter of the unified hash function hy(·) and the image to text mapping fi2t(·). Hyper-parameters ϵ, α and ρ are empirically set to 5000, 50 and 200 for scaling the order of each loss.

3.3 Optimization

In this part, we present an alternative learning algorithm to learn Θ, B and W in Eq. (5), which means we update one parameter with others fixed. This algorithm is also summarized in Algorithm 1, while details are presented below.

Normally, we are only given database points O and the pairwise supervised information S between them. To accelerate the training procedure, we can learn hash-codes and hash functions by sampling a subset of O as the query set Otrain for training, i.e., Otrain ⊆ O in each step. Thus, the training complexity can be reduced to O(nm) from O(n2 ), where O hides variables irrelevant to the size of dataset, m and n represent the size of datasets O and Otrain.

Learn Θ with B and W fixed

When B and W are fixed, we learn and update the parameter Θ by back-propagation algorithm.

Learn B with Θ and W fixed

To simplify the calculation, when fixing Θ and W, we rewrite Eq. (5) in the matrix form as:

where const is the constant independent of B, V = [hx(x1)⊤, . . . , hx(xn)⊤]⊤ ∈ Rn×c , T = [hy(y1)⊤, . . . , hy(yn)⊤]⊤ ∈ Rn×c . With little algebra, we can derive,

where Q = V ⊤BV ⊤ + T ⊤BT ⊤ + αWW ⊤B⊤ and R = -2cV ⊤S - 2cT ⊤S - 2αWL⊤ - 2ρ(T + V )⊤, and tr(·) is

the trace of a matrix.

Algorithm 1 The alternative learning algorithm

Input: O = {oi}ni=1 : n data points. S ∈ {-1,+1}n×n is the similarity matrix. c is the target binary code length. Minibatch size batchsize, sample size m and iteration number Tout, Tin.

Output: Θ is the parameter of the networks hy(·) and fi2t(·). B is the binary hash codes for data points.

Initialization: initialize Θ, B, andW.

• 1: for to = 1 → Tout do
• 2: Generate training set Otrian ⊆ O and Strian by randomly indexing.
• 3: for ti = 1 → Tin do
• 4: for batch = 1 → m/batchsize do
• 5: Construct themini-batchObatch trian by randomly sample batchsize points from Otrian.
• 6: Calculate the binary hash codes of mini-batch Obatch trian by inferencing.
• 7: Calculate the gradient by the chain rule and update Θ←Θ- η ∂ℓ ∂Θ by back propagation.
• 8: end for
• 9: end for
• 10: for k = 1 → c do
• 11: Update B∗k as Eq. (7).
• 12: end for
• 13: Update W as Eq. (8).
• 14: end for
• 15: return Θ and B.

As this problem is NP-hard, to ease this problem, inspired by SDH [Shen et al., 2015], we update one bit a time using the discrete cyclic coordinate descent method. We alternatively update a column of B with other columns fixed. Hence, this bit by bit problem at bit k becomes:

where Qk∗ = V ⊤ ∗k ˜B∗k ˜ V ⊤ ∗k + T ⊤ ∗k ˜B∗k ˜ T ⊤ ∗k + αWk∗ ˜W ⊤ k∗ ˜B⊤ ∗k , Rk∗ = -2cV ⊤ ∗k S - 2cT ⊤ ∗k S - 2αWk∗L⊤ - 2ρ(T∗k + V∗k)⊤. Next, the optimal solution of problem (6) is as follows:

Learn W with Θ and B fixed

When fixing Θ and B, we rewrite Eq. (5) as:

where const is the constant independent of W. It is easy to solve W by the regularized least squares problem, while the closed-form solution is as:

where P = (αV ⊤V + αT ⊤T + αB⊤B + I) and I is the diagonal with elements being 1.

4 Experiment

In this section, we carry out experiments to empirically evaluate the performance of DUCMH on three image-text benchmarks, and then compare it to state-of-the-art approaches.

4.1 Datasets and Evaluation Protocols

Three datasets, MIRFLICKR-25K [Huiskes and Lew, 2008], IAPR TC-12 [Escalante et al., 2010], and NUS-WIDE [Chua et al., 2009] are used for evaluation. MIRFLICKR-25K consists of 25,000 images collected from Flickr website. Each image is associated with several textual tags and annotated by one of the 24 unique labels. We select the points which have at least 20 textual tags for experiments. The text is represented by a 1386-dimensional bag-of-words (BOW) vector. IAPR TC-12 consists of 20,000 image-text (images-sentences) pairs which are annotated using 255 labels. The text for each point is represented as a 2912-dimensional BOW vector from sentences following DCMH [Jiang and Li, 2017]. NUS-WIDE contains 260,648 web images with textual tags. It is a multilabel dataset where each point is annotated with one or multiple labels from 81 concept labels. We select 195,834 imagetext pairs that belong to the 21 most frequent concepts. The text for each point is represented as a 1000-dimensional BOW vector. For MIRFLICKR-25K and IAPR TC-12, 2,000 data points are randomly sampled as the test (query) set, while for NUS-WIDE, 2,100 data points are selected. The remaining points as the retrieval set (database).

The retrieval performance is evaluated by one of the most widely used criteria, Mean Average Precision (mAP), which is the average of average precision for all the queries. All the data are reported with average values running five times.

4.2 Implementation Details and Comparison Methods

Our DUCMH method is implemented based on PyTorch [Paszke et al., 2019] with eight NVIDIA V100 GPUs and optimized by the mini-batch SGD with the size of 64 and weight decay. The learning rate is initialized as 0.0001 for the image to text mapping fi2t(·) and 0.004 for the unified hash function hy(·). For two neural network architectures, we use the following model: Image to text mapping fi2t(·): It is based on CNN-F neural networks [Chatfield et al., 2014]. We reserve the first seven layers, which are the same as those in CNN-F. Following this, a middle full-connected layer with 512 nodes and final full-connected layer which has nodes equal to the number of tags are framed. The unified hash function hy(·): It is built by a two convolutional layers followed by the tanh(·) activation. The first convolutional layer, followed by the ReLu activation and dropout with 0.5, has the input channel, output channel, kernel size and stride of 1, 10240, (the number of tags, 1) and (1, 1). The second convolutional layer has the input channel, output channel, kernel size and stride of 10240, c, (1, 1) and (1, 1).

We compare our method with several state-of-the-art (SOTA) hashing methods, including shallow methods, i.e., CCA [Hotelling, 1992], CMFH [Ding et al., 2014], SCM [Zhang and Li, 2014], STMH [Wang et al., 2015], SDMCH [Luo et al., 2018], and deep supervised methods, i.e., DCMH [Jiang and Li, 2017], SSAH [Li et al., 2018], EGDH [Shi et al., 2019], SHDCH [Zhan et al., 2020]. For our method and other deep hashing methods, the raw image is resized to 224 × 224 pixels as inputs, and the text inputs are BoW vectors. For traditional shallow methods, we extract 4096-dimensional deep features by the CNN-F model pre-trained on ImageNet to conduct fair comparisons. Besides, for all the SOTA methods, we employ the hyper-parameters introduced in their papers.

4.3 Results on Image and Text (Tags)

Following the experiment settings of previous work [Ding et al., 2014; Jiang and Li, 2017; Li et al., 2018; Shi et al., 2019], we first test the power of data alignment between text (tags) and image modalities.

The mAP searching results are presented in Table 1. In all datasets and two tasks, i.e., text to image (T → I) and image to text (I → T), our proposed DUCMH outperforms all the methods with the power of data alignment and unified hash functions and codes. Specifically, in MIRFLICKR-25K, DUCMH outperforms SOTAs by 0.0461, 0.0423, 0.0325 of 16, 32, and 64 bits on the task of T → I, respectively, while on the task of I → T, our DUCMH outperforms SOTAs by 0.0895, 0.0844, 0.0770 of 16, 32, and 64 bits. Similar results can be found in other datasets on two tasks.

4.4 Results on Image and Text (Sentences)

As the original data of text modality in IAPR TC-12 are sentences, we conduct experiments on IAPR TC-12 to test the power of data alignment between text (sentences) and image modalities. We replace the image to text mapping fi2t(·) with a classical image caption method, i.e., show and tell [Vinyals et al., 2015], and the unified hash function hy(·) with a classical sentence classification method, i.e., TextCNN [Kim, 2014] (represented by "DUCMH with stcs" in Table 2). We still use the hyper-parameters introduced in the original papers, while other hyper-parameters ϵ, α and ρ are still the same.

The results are shown in Table 2. As tags are pure semantic information without noises extracted from sentences and learning to predict conceptual tags is much easier than learning to caption images, it is reasonable that the model using tags ("DUCMH with tags" in Table 2) outperforms the model using sentences ("DUCMH with stcs" in Table 2).

5 Conclusion

In this paper, we presented a novel cross-modality hashing method named DUCMH. DUCMH was the first deep method that learned unified hash functions and hash codes simultaneously. The key contribution was leveraging the power of data alignment and image-text data pairs at hand to better reduce modality gap than the representation alignment previous work employed. Extensive experiments showed the superiority of our method over several state-of-the-art hashing methods. In the future, we would like to explore the potential of data alignment on the task of paired unsupervised and unpaired supervised cross-modality hashing.

Acknowledgments

This work was partially supported by the NSFC (61976112), JiangsuSF (BK20200064), and Open Research Projects of Zhejiang Lab (NO. 2021KB0AB02).