--- title: "Contrastive Transformer Masked Image Hashing for Degraded Image Retrieval" authors: - "Xiaobo Shen" - "Haoyu Cai" - "Xiuwen Gong" - "Yuhui Zheng" doi: "10.24963/ijcai.2024/135" url: "https://ijcai.org/proceedings/2024/135" pdf: "https://ijcai.org/proceedings/2024/0135.pdf" venue: "Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence" year: 2024 pages: "1218-1226" track: "Main Track" keywords: - "Computer Vision: CV: Image and video retrieval" - "Machine Learning: ML: Unsupervised learning" --- # Contrastive Transformer Masked Image Hashing for Degraded Image Retrieval Xiaobo Shen, Haoyu Cai, Xiuwen Gong, Yuhui Zheng *Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence*, pages 1218-1226. DOI: [10.24963/ijcai.2024/135](https://doi.org/10.24963/ijcai.2024/135) ## Abstract Hashing utilizes hash code as a compact image representation, offering excellent performance in large-scale image retrieval due to its computational and storage advantages. However, the prevalence of degraded images on social media platforms, resulting from imperfections in the image capture process, poses new challenges for conventional image retrieval methods. To address this issue, we propose Contrastive Transformer Masked Image Hashing (CTMIH), a novel deep unsupervised hashing method specifically designed for degraded image retrieval, which is challenging yet relatively less studied. CTMIH addresses the problem by training on transformed and masked images, aiming to learn transform-invariant hash code in an unsupervised manner to mitigate performance degradation caused by image deterioration. CTMIH utilizes Vision Transformer (ViT) architecture applied to image patches to capture distant semantic relevance. CTMIH introduces cross-view debiased contrastive loss to align hash tokens from augmented views of the same image and presents semantic mask reconstruction loss at the patch level to recover masked patch tokens. Extensive empirical studies conducted on three benchmark datasets demonstrate the superiority of the proposed CTMIH over the state-of-the-art in both degraded and normal image retrieval. ## Full text ## 1 Introduction Hashing [Wang et al., 2017] has been widely applied for efficient retrieval from large-scale image databases due to its superiority of low computation and storage costs. It aims to convert high-dimensional image features into low-dimensional compact hashing codes while preserving similarity structure among images. Many learning-based hashing methods [Weiss et al., 2008; Gong et al., 2012; Wang et al., 2017] have been proposed by employing machine learning on hash code generation, and are still being actively studied to support fast and accurate large-scale image retrieval. ∗Corresponding author. Figure 1: The illustration of degraded image retrieval. The four example images with different degradations, i.e., flip, color jitter, grayscale, and blur, are shown. In degraded image retrieval, the goal is to retrieve similar images from a normal image database given degraded image queries. To accomplish this, hash codes are utilized to represent both the degraded image queries (represented by solid shapes) and the normal image database (represented by hollow shapes). The retrieval process is performed in Hamming space. Image degradation often leads to discrepancies between the hash codes of degraded images and their original normal images. These discrepancies can significantly reduce retrieval performance. ![Image](https://ijcai.org/proceedings/2024/fulltext/135/image_000000_7a312f3b72887f5bb691ad0dc5b3c57eeda82c8d7493d6584fdfb393aab2bf66.png) There has been a recent research focus on deep hashing [Luo et al., 2023] that introduces deep learning [LeCun et al., 2015] into hashing, and learns image features and hash code in an end-to-end manner. With powerful learning capability of deep architectures, e.g., CNN [Krizhevsky et al., 2012], ViT [Dosovitskiy et al., 2021], deep hashing has shown its significant superiority over conventional shallow hashing [Weiss et al., 2008; Gong et al., 2012; Shen et al., 2015]. Supervised deep hashing [Cao et al., 2017; Li et al., 2017; Liu et al., 2019; Fan et al., 2020] employs pairwise semantics or class label information to supervise training, and has achieved promising performance in image retrieval. However, supervised deep hashing heavily requires manually annotated labels, which are expensive to collect. Unsupervised deep hashing without need of semantic labels is valuable yet challenging in real applications. Until now some efforts [Shen et al., 2020; Luo et al., 2021; Qiu et al., 2021; Ma et al., 2022; Yu et al., 2023] have been made towards unsupervised deep hashing. Degraded images [Wang et al., 2020; Yang et al., 2022; Park et al., 2023] are commonly encountered in various scenarios due to imperfections in the image capture process. These imperfections can result in serious degradations such as flipping, color jitter, grayscale, and blur, as illustrated in Figure 1. In practical applications such as pedestrian monitoring using surveillance cameras, it is often necessary to use the degraded pedestrian images as queries to retrieve similar high-quality images from a large-scale face image database [Li et al., 2019]. The need for large-scale degraded image retrieval is evident, however this task is challenging and has received relatively less attention. Image degradation significantly affects semantic similarity of images, and may mislead training of conventional hashing, leading to reduced retrieval performance. Therefore, it is imperative to develop new deep hashing for degraded image retrieval, which remains relatively unexplored and presents substantial challenges. Figure 2: Illustration of the proposed CTMIH. CTMIH first generates two augmented views of an image by applying image transformation as image degradations, based on which two masked views are then obtained via the block masking strategy. The architecture of CTMIH consists of a weight-sharing Siamese network structure, where two branches receive patches from augmented views and masked views. Each branch employs a Vision Transformer (ViT) encoder and a hash layer to extract patch tokens and hash token, and quantizes hash token to obtain hash code. CTMIH is trained using the idea of self-distillation, minimizing three losses. The cross-view constrastive loss JC aims to preserve the cross-view similarity between masked and augmented hash tokens, ensuring consistency across different views. The semantic mask reconstruction loss JR aims to recover masked patch tokens to their corresponding augmented patch tokens, enhancing the quality of reconstructed images. The quantization loss JQ reduces quantization error by treating it as a binary classification problem. ![Image](https://ijcai.org/proceedings/2024/fulltext/135/image_000001_fc8aee78752986a0be813a19eb531551378051c95cee0c48d514eea793b03f0e.png) To address this issue, we propose Contrastive Transformer Masked Image Hashing (CTMIH) that is specifically designed for degraded image retrieval. The key idea of CTMIH is to train hashing model on both transformed images and masked images to learn transform-invariant hash codes in an unsupervised manner, such that retrieval performance degradation caused by image degradation can be mitigated. The overview of the proposed CTMIH is illustrated in Figure 2. The main contributions of this work can be summarised as follows: - We address the challenging task of degraded image retrieval, which has received relatively less attention. To tackle this, we propose a new deep unsupervised hashing method called Contrastive Transformer Masked Image Hashing (CTMIH). - CTMIH utilizes ViT to encode image patches and learns transform-invariant hash code by aligning augmented - and masked hash tokens while recovering masked patch tokens. - Extensive empirical evaluations conducted on three benchmark image datasets demonstrate the superior performance of the proposed method over the state-of-theart in both degraded and normal image retrieval. ## 2 RelatedWork Deep Hashing Due to the strong learning capability of advanced deep architectures, e.g., CNN [Krizhevsky et al., 2012], ViT [Dosovitskiy et al., 2021], deep hashing [Luo et al., 2023] has achieved promising performance in large-scale image retrieval. According to whether semantic supervision is used or not, deep hashing methods can be roughly divided into supervised deep hashing and unsupervised deep hashing. Supervised deep hashing[Cao et al., 2017; Li et al., 2017; Liu et al., 2019; Fan et al., 2020] typically outperforms unsupervised deep hashing by incorporating additional semantic labels. However, manual label annotation is time consuming and expensive, which is not often available in real applications. Recently unsupervised deep hashing has been a hot research focus, as it overcomes the limitation of manual labeling. Some works [Dai et al., 2017; Shen et al., 2020] learn hash code by performing reconstruction tasks using some advanced deep architectures, e.g., variational autoencoders (VAE) [Kingma and Welling, 2013] and generative adversarial network (GAN) [Goodfellow et al., 2014]. For instance, Semantic Structure based unsupervised Deep Hashing (SSDH) [Yang et al., 2018] leverages two half Gaussian distributions to construct semantic structure, and further designs a pairwise similarity preservation loss. Bi-half Net [Li and van Gemert, 2021] proposes to learn hash code by maximizing its bit entropy, and designs a parameter-free layer to force continuous image features to approximate the optimal half-half bit distribution. In addition, contrastive learning [Chen et al., 2020; He et al., 2020], as a powerful self-supervised method, has been introduced into deep unsupervised hashing, based on which some hashing methods have been proposed [Luo et al., 2021; Qiu et al., 2021; Ma et al., 2022]. Contrastive Quantization with Code Memory (MeCoQ) [Wang et al., 2022] proposes to use contrastive loss to better capture discriminative visual semantics and further use quantization code memory to enhance contrastive learning with lower feature drift. While many deep hashing methods employ CNN as a backbone, there have been a few recent works [Qiu et al., 2021; Yu et al., 2022; Ng et al., 2023] that consider ViT as a backbone to encode images. Most existing deep unsupervised hashing methods have been developed for mainly conventional normal image retrieval, and may not perform well in degraded image retrieval, where image queries are degraded. Degraded Image Analysis There have been a few attempts [Wang et al., 2020; Yang et al., 2022] on degraded image analysis, mainly on degraded image classification. Based on the observation that the distributions of corresponding patches in high- and low-quality images have uniform margins, feature de-drifting module (FDM) [Wang et al., 2020] is proposed to learn the mapping relationship between deep representations of high- and low-quality images, and is further leveraged as a deep degradation prior to degraded image classification. In addition, a self-feature distillation method with uncertainty modeling [Yang et al., 2022] is proposed for degraded image classification. It employs the high-quality features to distill the corresponding degraded ones and conduct uncertainty modeling, increasing importance of feature regions that are difficult to recover. However, to our knowledge, until now there have been few attempts of learning hash code for degraded image retrieval. ## 3 The ProposedMethod ## 3.1 Problem Setup Generally speaking, the goal of deep hashing for image retrieval is to map an ideal image x to hash code b, which is used to support fast image retrieval in Hamming space. However, in real-world applications, due to various sources of degradation, e.g., noise, motion blur, and ColorJitter, we only observe the degraded image ˜x instead of the ideal one. This work develops deep hashing for a challenging yet less studied task, i.e., degraded image retrieval, where image queries are degraded, while images in database are normal. In degraded image retrieval, given a degraded image query ˜x, the aim of the proposed method is to learn a deep hash function H : ˜x -→ b ∈ {-1, 1}L, where b is the hash code, L is code length. The learned hash code is then employed to achieve fast and accurate retrieval of the degraded image query from a normal image database. As the degraded images have different inherent appearances from normal images, it can result in obvious retrieval performance degradation using the conventional hash function. To address this issue, the ## Algorithm 1 Image Transformation T Input: Image X, hyper-parameter δ; Output: Transformed Image. - 1: Crop X with size of 256 × 256 randomly; - 2: Resize X to size of 224 × 224; - 3: Flip X horizontally with a probability of 0.5δ; - 4: Add colorjitter to X with a probability of 0.8δ; - 5: Convert X to grayscale image with a probability of 0.4δ; - 6: Apply Gaussian blur to X with a probability of 0.5δ. key of the proposed method is to learn the transform-invariant hash function that is robust to image degradation. As such, the degraded image has similar hash code to its normal image, making it possible to quickly and accurately retrieve images that are visually or semantically similar to the degraded image query. ## 3.2 Formulation Architecture The most common and natural solution for mitigating retrieval performance degradation caused by image degradation is to train a deep hashing model with augmented images having various image transformations. For a given image X, the two random augmentations are applied, yielding two distorted views: U = T (X, δu) and V = T (X, δv), where T denotes the transformation function that performs image degradation, hyper-parameter δu and δv control the degrees of the two transformation. In this work, the transformations include random crop, resize, flip, color jitter, and blur, and the transformation procedure is illustrated in Algorithm 1. It is clear that large δ leads to heavy image degradation. In addition, motivated by the success of masked image modeling, the blockwise masking is applied to two views U and V to obtain masked views ˜U and ˜V respectively. Specifically, for instance, given U, we first divide it into K non-overlapping patches uk, where k = 1, . . . ,K, and then, with a masking ratio r , perform masking on a random subset of patches, which are replaced by e. As illustrated in Figure 2, Siamese network structure with two branches sharing network weights is employed in CTMIH, and the patches of augmented views and masked views are fed into the two branches. Existing deep hashing methods mainly employ CNN as the backbone that performs convolution operations on a small neighborhood of an image, and struggles to relate concepts spatially apart. Vision Transformer (ViT) [Dosovitskiy et al., 2021] can effectively capture distant semantic relevance in an image by applying selfattention to a series of patches in an image. Inspired by the powerful capability of ViT on image modeling, we propose to employ ViT as an encoder in each branch, which receives image patches as sequential data and generates their latent representations. Following ViT encoder, a hash layer consists of a fully-connected layer and the sign function. For instance, given augmented view U and its masked view ˜U, hash layer outputs their hash tokens hu and h˜u , and patch tokens puk and p˜uk , where k = 1, . . . ,K, and finally binarizes hu to obtain hash code bu = sign(hu ). Cross-view Debiased Contrastive Learning For conventional hashing methods, a degraded image tends to produce different hash code compared to its original counterpart due to changes in appearance, leading to unsatisfactory performance in degraded image retrieval. To mitigate this issue, a possible solution is to leverage contrastive learning (CL) [Chen et al., 2020; He et al., 2020] that aims to distinguish between semantically similar and dissimilar pairs of samples. Based on a triplet set that consists of anchor, positive, and negative samples, CL encourages the anchor and positive sample to be pulled closer together while pushing away the anchor from the negative samples. This work considers the benefit of CL that can well align the hash tokens of the two views augmented from one image, and push away the hash tokens of different images. Specifically, given the i-th image Xi, the hash tokens of its two augmented views are denoted as hui and hvi , and its two masked views are denoted as h˜ui , h˜ vi . Taking hui as anchor, we regard another masked view h˜ vi as positive sample, and {h˜ vj }nj=1,j̸=i as negative samples, where n is batch size. In this work, we contrast augmented and masked views with the purpose of distilling knowledge of unmasked views to help learn the hash token of masked views. Our idea is to enable the positive pairs to be close and the negative pairs to be apart. Treating hui as anchor, we minimize the following conventional CL loss: where function o (hui , h˜ vi ) = exp (hui⊤h˜ vi/τ) measures the similarity between two embeddings, τ is a temperature coefficient that controls dynamic range of product. The ℓ2 normalization is applied on input embeddings before computing the inner product, such that the inner product is equivalent to cosine similarity. As can be seen in (1), in conventional CL, negative points are randomly sampled from the whole set as true labels are unavailable. However, some of these negative samples may actually belong to the same class as anchor, referred as sampling bias, and this bias has been shown to degrade performance. To mitigate this gap, debiased CL is employed to correct sampling bias of negative samples, and its loss is defined as follows: where the negative similarity term can be modified as: where ϱ- and ϱ+ represent the class probability that two images belong to same and different classes respectively, and we have ϱ- + ϱ+ = 1. Similarly, we can further define JC(hvi , h˜ui ). To this end, by considering all the n images, we have the following debiased CL loss: Semantic Mask Reconstruction Learning Masked image patch reconstruction is a popular self-supervised pretext task with the idea of auto-encoding [Zhou et al., 2021], and has been previously achieved by predicting raw pixels. This work leverages the benefits of masked image modeling to explore internal local structures in an image and better train ViT. Specifically, for instance, given one augmented view Ui and the corresponding masked view ˜Ui of the i-th image, their patch token sequences are denoted as {puik }Kk=1 and {p˜uik }Kk=1 . In this work, we employ the idea of selfdistillation, and instead of recovering raw pixels, propose to semantically recover the masked patch token using original patch token, i.e., enable the masked and its original tokens to be close. Specifically, given the i-th image, we recover its k-th image patch by minimizing the following cross-entropy (CE) loss: where mik is set to 1 if the k-th patch of the i-th image is masked, and set to 0 otherwise. It simplifies masked recovering by turning it into a classification problem that is optimized by CE loss, and preserves more semantic information of patch token. By considering all the image patches of all the n images, we have the following semantic mask reconstruction loss: ## 3.3 Training and Inference Training To obtain high-quality hash code, quantization loss is introduced to reduce quantization error. Motivated by the observation that hashing aims to predict the sign of each bit, this problem can be naturally regarded as the binary classification. Specifically, we employ a pre-defined Gaussian distribution estimator g(h) = exp(- (h-µ)2 2σ2 ) to evaluate binary likelihood of each hash bit, where µ and σ denote mean and standard deviation respectively. We then define G(·) = {g+, g-}, where with the same σ, g+ and g- are defined with µ = 1 and µ = -1 respectively. The quantization loss is calculated as binary cross-entropy classification loss (BCE). Taking hui for example, its quantization loss is defined as follows: where BCE loss is defined as Q(y, g) = -y log g + (1 - y) log(1 - g), g+ l = g+(huil ) and gl = g-(huil ) denote the two estimated likelihoods of l-th hash bit of hui , y+ l = 12 (sign(huil ) + 1) and yl = 1 - y+ l denote the likelihood labels, L is the code length. In this way, the quantization error is reduced by minimizing the above binary classification loss. Considering the hash tokens of all the n images, we have the following quantization loss: Table 1: mAPs of all the hashing methods for degraded image retrieval on the three benchmark datasets. The bold and underlined indicate best and second best, respectively. | Method | Refernce | Backbone | MS COCO - 16 bits | MS COCO - 32 bits | MS COCO - 64 bits | NUS-WIDE - 16 bits | NUS-WIDE - 32 bits | NUS-WIDE - 64 bits | ImageNet - 16 bits | ImageNet - 32 bits | ImageNet - 64 bits | |-------------|------------|------------|---------------------|---------------------|---------------------|----------------------|----------------------|----------------------|----------------------|----------------------|----------------------| | SH | NeurIPS 08 | VGG | 0.524 | 0.529 | 0.544 | 0.528 | 0.523 | 0.548 | 0.214 | 0.325 | 0.406 | | ITQ | TPAMI 13 | VGG | 0.598 | 0.646 | 0.667 | 0.612 | 0.618 | 0.646 | 0.327 | 0.461 | 0.549 | | SSDH | IJCAI 18 | VGG | 0.540 | 0.564 | 0.592 | 0.673 | 0.692 | 0.707 | 0.176 | 0.217 | 0.248 | | GreedyHash | NeurIPS 18 | VGG | 0.549 | 0.570 | 0.617 | 0.637 | 0.646 | 0.688 | 0.147 | 0.307 | 0.434 | | Bi-half Net | AAAI 21 | VGG | 0.638 | 0.711 | 0.734 | 0.736 | 0.756 | 0.766 | 0.439 | 0.537 | 0.591 | | MeCoQ | AAAI 22 | VGG | 0.692 | 0.738 | 0.753 | 0.744 | 0.776 | 0.788 | 0.599 | 0.646 | 0.710 | | OH | ACMMM 23 | VGG | 0.656 | 0.667 | 0.692 | 0.711 | 0.742 | 0.771 | 0.654 | 0.674 | 0.705 | | CIBHash | IJCAI 21 | ViT | 0.767 | 0.796 | 0.813 | 0.781 | 0.801 | 0.813 | 0.732 | 0.768 | 0.795 | | WCH | ACCV 22 | ViT | 0.741 | 0.775 | 0.794 | 0.781 | 0.804 | 0.819 | 0.733 | 0.775 | 0.796 | | SDC | BMVC 23 | ViT | 0.765 | 0.801 | 0.823 | 0.764 | 0.796 | 0.809 | 0.734 | 0.779 | 0.814 | | CTMIH | Ours | ViT | 0.809 | 0.834 | 0.846 | 0.795 | 0.816 | 0.826 | 0.820 | 0.860 | 0.869 | To this end, by summarizing the three losses, i.e., JC, JR, JQ, we have the following objective function of the proposed method: where α and β are two non-negative parameters to balance each term. Inference Once the proposed CTMIH is trained, and given an image query, we first divide it into several patches and further feed it into the ViT encoder and hash layer without masking to generate its hash code. For degraded image retrieval, the image query is required to be degraded. We perform transformations on a normal image to obtain a degraded image query, which is detailed in the experiment setup. ## 4 Experiments ## 4.1 Experimental Setup Datasets The experiments are conducted on three benchmark image datasets, which are detailed as follows: MSCOCO [Lin et al., 2014] is a large-scale image dataset for object detection, segmentation, and captioning. Following previous works, a subset of 122,218 images from 80 categories is used, where the 5,000 images are randomly selected as the query set and the remaining images are used as the database. The 10,000 images are randomly selected from the database for training. NUS-WIDE [Chua et al., 2009] is a multi-label dataset consisting of 269,648 images from 81 categories. A subset with images from the 21 most frequent categories is used. The 100 images are randomly sampled from each category as the query set and the remaining images are used as the database. The 500 images for each category are randomly sampled from the database for training. ImageNet [Russakovsky et al., 2014] is a single-label image dataset, where each image is labeled by one of 1, 000 categories. A subset with 143,495 images in 100 categories is used, where 100 images from each category are randomly sampled for training, 5,000 images are sampled as the query set, and the remaining images are used as the database. Baselines We compare the proposed method with various state-of-the-art unsupervised hashing baselines, including two shallow hashing methods, i.e., SH [Weiss et al., 2008] and ITQ [Gong et al., 2012], five CNN based deep hashing methods, i.e., SSDH [Yang et al., 2018], GreedyHash [Su et al., 2018], Bi-half Net [Li and van Gemert, 2021], MeCoQ [Wang et al., 2022], OH [Yu et al., 2023], three ViT based deep hashing methods, i.e., CIBHash [Qiu et al., 2021], WCH [Yu et al., 2022], SDC [Ng et al., 2023]. Experimental Setting For all the methods, the images are resized to 224 × 224 × 3. For shallow hashing methods, the 4,096-dimensional feature extracted by the VGG-F model pre-trained on ImageNet is used for training. For deep hashing methods, the raw image is directly used as for training. For the proposed method, we apply Algorithm 1 on each image in the training set to generate two augmented views, where δu and δv are set to 0.5 and 1 respectively. To generate the degraded image query, we apply Algorithm 1 on each image in the query set by setting δ to 0.5 by default. The standard ViT-Base is used as the backbone, and the size and number of patches are set to 16 and 196 respectively. The masking ratio r is set to 0.3, class probability ϱ+ is set to 0.05, and temperature τ is set to 0.5. The two hyper-parameters α and β are set to 0.1 and 0.1 respectively. The batch size is set to 32, the number of epochs is set to 100, and the learning rates of ViT and hash layer are set to 10-5 and 10-3 respectively. The proposed method is trained using Adam optimizer. Evaluation Metrics Following [Zhang et al., 2017], we consider the widely-used mean Average Precision (mAP@K) and Precision curve as evaluation metrics, and K is set to 5000 for NUS-WIDE and MSCOCO, and 1000 for ImageNet. ## 4.2 Comparisons with State-of-the-art Results on Degraded Image Retrieval This section evaluates the performance of degraded image retrieval. The mAPs of the proposed CTMIH and ten state-of-the-art hashing baselines on three benchmark datasets are reported in Table 1. In addition, precision curves of all the methods with respect to Figure 3: Precision curves of all the hashing methods with respect to 32 bits on three benchmark datasets. ![Image](https://ijcai.org/proceedings/2024/fulltext/135/image_000002_3496f3dc74e37e67f8d64ce8eefa829c3771f56064da9326db4c4378cb1e20d2.png) Table 2: mAPs of all the hashing methods for normal image retrieval on the three benchmark datasets. The mAP drops on degraded image retrieval compared to normal image retrieval are also reported. | Method | MS COCO | NUS-WIDE | ImageNet | |-------------|-------------|-------------|-------------| | SH | 0.585↘0.061 | 0.638↘0.110 | 0.338↘0.124 | | ITQ | 0.658↘0.060 | 0.718↘0.102 | 0.439↘0.112 | | SSDH | 0.556↘0.016 | 0.696↘0.023 | 0.229↘0.053 | | GreedyHash | 0.586↘0.037 | 0.675↘0.038 | 0.231↘0.084 | | Bi-half Net | 0.645↘0.007 | 0.750↘0.014 | 0.540↘0.101 | | MeCoQ | 0.755↘0.063 | 0.770↘0.026 | 0.705↘0.051 | | OH | 0.748↘0.092 | 0.796↘0.085 | 0.672↘0.018 | | CIBHash | 0.793↘0.026 | 0.790↘0.009 | 0.783↘0.051 | | WCH | 0.760↘0.019 | 0.788↘0.007 | 0.761↘0.028 | | SDC | 0.810↘0.045 | 0.787↘0.023 | 0.764↘0.030 | | CTMIH | 0.818↘0.009 | 0.799↘0.004 | 0.832↘0.012 | 32 bits are shown in Figure 3. From Table 1 and Figure 3, we can clearly observe that 1) the proposed CTMIH outperforms all the baselines in all the 10 cases. Specifically, it outperforms the baselines averagely by 3.3%, 1.1%, 7.4% on MS COCO, NUS-WIDE, ImageNet respectively. In addition, the precision curves of CTMIH are generally above those of the baselines. 2) among all the baselines, CIBHash, WCH, SDC with ViT backbone outperform the other deep hashing baselines with VGG backbone by a margin, followed by shallow hashing baselines. The empirical results clearly demonstrate the superiority of the proposed CTMIH for degraded image retrieval. Results on Normal Image Retrieval This section evaluates the performance of conventional normal image retrieval, where the image queries are normal. The mAPs of all the hashing methods with respect to 16 bits are reported in Table 2. From this table, we see that the proposed CTMIH outperforms all the baselines, indicating that the proposed method also works well on conventional normal image retrieval. In addition, compared to normal image retrieval, the mAP drops of the proposed method on degraded image retrieval are lower than those of the baselines. This suggests that the proposed CTMIH is superior in mitigating the performance degradation caused by image degradation. ## 4.3 Further Analysis Evaluation on Varying Degrees of Image Degradation The section evaluates the sensitivity of the deep hashing methods with respect to varying degrees of image degradation. We vary the parameter δ from the range of [0, 0.2, 0.4, 0.6, 0.8, 1.0] to generate image queries with varying degrees of degradation, where the sample image query is shown in Figure 4. Figure 4 reports the mAPs of all the deep hashing methods with varying degrees of degradation on ImageNet. From this figure, we can clearly observe that mAP of the proposed CTMIH is not relatively insensitive to the change of δ, compared to the deep hashing baselines. Specifically, as the degree of image degradation increases, the mAP of the proposed CTMIH decreases by 1.3%, while the mAPs of the baselines have shown a decrease ranging from 5.2% to 20.1%. The above results clearly demonstrate that the proposed CTMIH is robust to image degradation, and performs well on degraded image retrieval. Figure 4: mAPs of all the deep hashing methods with respect to varying degrees of image degradation on ImageNet. ![Image](https://ijcai.org/proceedings/2024/fulltext/135/image_000003_a695c764dce07218ae3a64ff85b489404e9da1a87a41aed9a257cc9fee287790.png) Table 3: The mAPs of the proposed method and four variants on three datasets. | Method | MS COCO | NUS-WIDE | ImageNet | |----------|-----------|------------|------------| | Naive CL | 0.719 | 0.698 | 0.674 | | w/o JQ | 0.765 | 0.731 | 0.757 | | w/o JR | 0.792 | 0.779 | 0.806 | | w/o JC | 0.428 | 0.535 | 0.413 | | CTMIH | 0.809 | 0.795 | 0.820 | Ablation Study This section empirically evaluates the effectiveness of each loss in the proposed CTMIH. We compare the proposed method with a baseline that is trained with conventional contrastive loss without masking strategy, and its three variants without each of three losses (w/o). The mAPs of these methods with respect to 16 bits on the three datasets are reported in Table 3. From Table 3, we can clearly observe that removing JQ, JR, JC results in an average decrease of 5.7%, 1.6%, 34.9% in mAP respectively on the three datasets, where the debiased contrastive loss has the greatest impact on performance. In addition, CTMIH obviously outperforms naive CL baseline by 11.10% on average. Figure 5: The t-SNE visualization of ImageNet using ten hashing methods. The different colors indicate different labels and a total of 10 classes with 1,000 samples from each class are randomly sampled for visualization. ![Image](https://ijcai.org/proceedings/2024/fulltext/135/image_000004_4fcaba0cb7cf905e94c8f32199fe3fe2740204386da293e63a98798299536344.png) Figure 6: The visualization of attention maps of the proposed CTMIH on 5 randomly selected degraded images from ImageNet. ![Image](https://ijcai.org/proceedings/2024/fulltext/135/image_000005_f4bcb25c709d9e1a09b0d3068b8918a9244b633d4cf93a440fed1ee516c027d6.png) Figure 7: Top-10 retrieved results of the four deep hashing methods on a normal image and its degraded image queries from ImageNet. ![Image](https://ijcai.org/proceedings/2024/fulltext/135/image_000006_c932ce9d40b21ef4854e10e07c8cba53976c01e1484bbb79b2d527bc64f2224b.png) Embedding Visualization This section qualitatively compares different hashing methods by visualizing learned hash codes of degraded images. We conduct experiment on ImageNet, randomly selecting 10,000 samples that belong to 10 classes, and setting code length to 32. The hash codes learned by ten methods are visualized into a 2-dimensional space with t-SNE [van der Maaten and Hinton, 2008], as illustrated in Figure 5. From Figure 5, we see that the visualization results are generally consistent with the quantitative empirical results. Attention Map Visualization This section qualitatively evaluates the effectiveness of the proposed CTMIH by visualizing its attention maps. Figure 6 illustrates the attention maps generated by CTMIH for 5 randomly selected degraded images from ImageNet. From the figure, we can observe that the generated attention maps focus on the key components of images through masking reconstruction task. It indicates that the proposed CTMIH can effectively identify and prioritize important regions in the image. Case Study This section presents a case study comparing the performance of the proposed CTMIH with three baselines, namely CIBHash, WCH, and SDC, for both degraded and normal image retrieval tasks. Figure 7 illustrates the top10 retrieved images of one randomly selected normal image and its degraded image queries from ImageNet. A retrieved sample is marked in green if its label matches that of the query, and it is marked in red otherwise. From the results shown, we can observe that CTMIH outperforms the three baselines in terms of retrieving correct images in both tasks. These findings further confirm the effectiveness of the proposed CTMIH for both degraded and normal image retrieval tasks. ## 5 Conclusions This work presents a preliminary attempt to learn hash code from degraded images, and proposes Contrastive Transformer Masked Image Hashing (CTMIH) for this challenging yet less studied degraded image retrieval. The proposed CTMIH aims to learn transform-invariant hash code through unsupervised training on transformed and masked images, mitigating performance degradation caused by image degradation. CTMIH aligns hash tokens of augmented views from the same image by performing constrastive learning, and recovers masked patch tokens with the idea of masked image prediction. The extensive empirical studies demonstrate the proposed CTMIH performs effectively on both degraded and normal image retrieval. ## Acknowledgments This work was supported by the National Natural Science Foundation of China under Grant No. 62176126, U20B2065, U22B2056, 62272468, the Natural Science Foundation of Jiangsu Province, China under Grant No. BK20230095, BK20211539. ## References 1. [Cao et al., 2017] Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Philip S Yu. Hashnet: Deep learning to hash by continuation. In Proceedings of IEEE International Con- ference on Computer Vision, pages 5608-5617, 2017. 2. [Chen et al., 2020] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey E. Hinton. A simple frame- work for contrastive learning of visual representations. In Proceedings of International Conference on Machine Learning, volume 119, pages 1597-1607, 2020. 3. [Chua et al., 2009] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. Nus- wide: a real-world web image database from national uni- versity of singapore. In Proceedings of ACM Interna- tional Conference on Image and Video Retrieval, pages 1-9, 2009. 4. [Dai et al., 2017] Bo Dai, Ruiqi Guo, Sanjiv Kumar, Niao He, and Le Song. Stochastic generative hashing. In Pro- ceedings of International Conference on Machine Learn- ing, pages 913-922, 2017. 5. [Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. In Proceedings of International Conference on Learning Representations, 2021. 6. [Fan et al., 2020] Lixin Fan, Kam Woh Ng, Ce Ju, Tianyu Zhang, and Chee Seng Chan. Deep polarized network for supervised learning of accurate binary hashing codes. In Proceedings of International Joint Conference on Artifi- cial Intelligence, pages 825-831, 2020. 7. [Gong et al., 2012] Yunchao Gong, Svetlana Lazebnik, Al- bert Gordo, and Florent Perronnin. Iterative quantization: A procrustean approach to learning binary codes for large- scale image retrieval. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 35(12):2916-2929, 2012. 8. [Goodfellow et al., 2014] Ian J. Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63:139 -144, 2014. 9. [He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Sain- ing Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceed- ings of IEEE Conference on Computer Vision and Pattern Recognition, pages 9726-9735, 2020. 10. [Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of International Conference on Learning Representations, 2013. 11. [Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Proceedings of Ad- vances in Neural Information Processing Systems, pages 1106-1114, 2012. 12. [LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Ge- offrey Hinton. Deep learning. Nature, 521(7553):436- 444, 2015. 13. [Li and van Gemert, 2021] Yunqiang Li and Jan van Gemert. Deep unsupervised image hashing by maximizing bit en- tropy. In Proceedings of AAAI Conference on Artificial Intelligence, pages 2002-2010, 2021. 14. [Li et al., 2017] Qi Li, Zhenan Sun, Ran He, and Tieniu Tan. Deep supervised discrete hashing. In Proceedings of Ad- vances in Neural Information Processing Systems, pages 2482-2491, 2017. 15. [Li et al., 2019] Dangwei Li, Zhang Zhang, Xiaotang Chen, and Kaiqi Huang. A richly annotated pedestrian dataset for person retrieval in real surveillance scenarios. IEEE Trans- actions on Image Processing, 28(4):1575-1590, 2019. 16. [Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In Proceedings of European Con- ference on Computer Vision, pages 740-755, 2014. 17. [Liu et al., 2019] Haomiao Liu, Ruiping Wang, Shiguang Shan, and Xilin Chen. Deep supervised hashing for fast image retrieval. Proceedings of IEEE Conference on Com- puter Vision and Pattern Recognition, 127(9):1217-1234, 2019. 18. [Luo et al., 2021] Xiao Luo, Daqing Wu, Zeyu Ma, Chong Chen, Minghua Deng, Jinwen Ma, Zhongming Jin, Jian- qiang Huang, and Xian-Sheng Hua. Cimon: Towards high-quality hash codes. In Proceedings of International Joint Conference on Artificial Intelligence, pages 902- 908, 2021. 19. [Luo et al., 2023] Xiao Luo, Haixin Wang, Daqing Wu, Chong Chen, Minghua Deng, Jianqiang Huang, and Xian-Sheng Hua. A survey on deep hashing methods. ACM Transactions on Knowledge Discovery from Data, 17(1):15:1-15:50, 2023. 20. [Ma et al., 2022] Zeyu Ma, Xiao Luo, Yingjie Chen, Mix- iao Hou, Jinxing Li, Minghua Deng, and Guangming Lu. Improved deep unsupervised hashing with fine-grained se- mantic similarity mining for multi-label image retrieval. In Proceedings of International Joint Conference on Arti- ficial Intelligence, pages 1254-1260, 2022. 21. [Ng et al., 2023] Kam Woh Ng, Xiatian Zhu, Jiun Tian Hoe, Chee Seng Chan, Tianyu Zhang, Yi-Zhe Song, Tao Xi- ang, Universiti Malaya CISiP, and GACRD Center. Un- supervised hashing with similarity distribution calibration. 2023. 22. [Park et al., 2023] Dongwon Park, Byung Hyun Lee, and Se Young Chun. All-in-one image restoration for unknown degradations using adaptive discriminative filters for spe- cific degradations. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 5815- 5824, 2023. 23. [Qiu et al., 2021] Zexuan Qiu, Qinliang Su, Zijing Ou, Jianxing Yu, and Changyou Chen. Unsupervised hashing with contrastive information bottleneck. In Proceedings of International Joint Conference on Artificial Intelligence, pages 959-965, 2021. 24. [Russakovsky et al., 2014] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115, 09 2014. 25. [Shen et al., 2015] Fumin Shen, Chunhua Shen, Wei Liu, and Heng Tao Shen. Supervised discrete hashing. In Pro- ceedings of IEEE Conference on Computer Vision and Pat- tern Recognition, pages 37-45, 2015. 26. [Shen et al., 2020] Yuming Shen, Jie Qin, Jiaxin Chen, Mengyang Yu, Li Liu, Fan Zhu, Fumin Shen, and Ling Shao. Auto-encoding twin-bottleneck hashing. In Pro- ceedings of IEEE Conference on Computer Vision and Pat- tern Recognition, pages 2818-2827, 2020. 27. [Su et al., 2018] Shupeng Su, Chao Zhang, Kai Han, and Yonghong Tian. Greedy hash: Towards fast optimization for accurate hash coding in cnn. Proceedings of Advances in Neural Information Processing Systems, 31, 2018. 28. [van der Maaten and Hinton, 2008] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. The Journal of Machine Learning Research, 9(86):2579-2605, 2008. 29. [Wang et al., 2017] Jingdong Wang, Ting Zhang, Nicu Sebe, Heng Tao Shen, et al. A survey on learning to hash. IEEE Transactions on Pattern Analysis and Machine In- telligence, 40(4):769-790, 2017. 30. [Wang et al., 2020] Yang Wang, Yang Cao, Zheng-Jun Zha, Jing Zhang, and Zhiwei Xiong. Deep degradation prior for low-quality image classification. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 11049-11058, 2020. 31. [Wang et al., 2022] Jinpeng Wang, Ziyun Zeng, Bin Chen, Tao Dai, and Shu-Tao Xia. Contrastive quantization with code memory for unsupervised image retrieval. In Pro- ceedings of AAAI Conference on Artificial Intelligence, pages 2468-2476, 2022. 32. [Weiss et al., 2008] Yair Weiss, Antonio Torralba, and Rob Fergus. Spectral hashing. Proceedings of Advances in Neural Information Processing Systems, 21, 2008. 33. [Yang et al., 2018] Erkun Yang, Cheng Deng, Tongliang Liu, Wei Liu, and Dacheng Tao. Semantic structure-based unsupervised deep hashing. In Proceedings of Interna- tional Joint Conference on Artificial Intelligence, pages 1064-1070, 2018. 34. [Yang et al., 2022] Zhou Yang, Weisheng Dong, Xin Li, Jin- jian Wu, Leida Li, and Guangming Shi. Self-feature dis- tillation with uncertainty modeling for degraded image recognition. In Proceedings of European Conference on Computer Vision, pages 552-569, 2022. 35. [Yu et al., 2022] Jiaguo Yu, Huming Qiu, Dubing Chen, and Haofeng Zhang. Weighted contrative hashing. In Proceed- ings of the Asian Conference on Computer Vision, pages 3861-3876, 2022. 36. [Yu et al., 2023] Jiaguo Yu, Yuming Shen, and Haofeng Zhang. Hashing one with all. In Proceedings of ACM In- ternational Conference on Multimedia, pages 6420-6431, 2023. 37. [Zhang et al., 2017] Haofeng Zhang, Li Liu, Yang Long, and Ling Shao. Unsupervised deep hashing with pseudo labels for scalable image retrieval. IEEE Transactions on Image Processing, 27(4):1626-1638, 2017. 38. [Zhou et al., 2021] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. Proceedings of International Conference on Learning Representations, 2021.