Domain-Smoothing Network for Zero-Shot Sketch-Based Image Retrieval

1 Introduction

Sketch-based image retrieval (SBIR) [Sangkloy et al., 2016; Bhunia et al., 2020] is an interesting and practical task aiming to retrieve similar images based on the queries of the handdrawn sketches. With the help of large-scale labeled datasets, SBIR can achieve astonishing performance. However, collecting and annotating large-scale datasets is time-consuming and labor-intensive. To this end, zero-shot SBIR (ZS-SBIR) [Shen et al., 2018] is proposed, which aims to retrieve categories or samples unseen at the training stage. However, solving ZS-SBIR is significantly challenging since it simultaneously deals with the inherent domain gap and limited knowledge about the unseen categories [Deng et al., 2020].

Previous ZS-SBIR methods [Xu et al., 2020] attempt to learn a common representation of sketches and images with cross-entropy loss or triplet-based loss, which neglect the impact of the inherent domain gap. Specifically, cross-entropy loss equally treats sketches and images for classification and category-specific memory bank neglects the relations between sketches and images of the same category. Besides, triplet-based loss is usually utilized to measure the relations between samples. However, due to the limited positive and negative samples, triplet-based loss may not mine the relations sufficiently. Furthermore, sketches within the same category might be extremely diverse in appearance due to the drawing styles of different people, which is ignored by most methods and degrades retrieval performance.

∗Contact Author

Figure 1: The illustration of our DSN method. (Top) To smooth the domain gap between sketches and images, we propose a cross-modal contrastive method to mine the relations between samples with sufficiently positive and negative samples. (Bottom) Besides, a categoryspecific memory bank is built to address the large intra-class diversity in the sketch domain, where prototypes of each sketch category are learned to reduce the variability in appearance.

To alleviate the domain gap and the intra-class diversity in sketch domain, we propose a Domain-Smoothing Network (DSN) for ZS-SBIR (seen Fig. 1), mainly including a crossmodal contrastive method and a category-specific memory bank. Specifically, a pre-trained CNN backbone is first applied to extract visual features of the images and sketches. To smooth the domain gap between the sketches and images, a cross-modal contrastive method is then conducted after a projection head for visual features. Meanwhile, a categoryspecific memory bank is proposed, where sketch category prototypes are learned to reduce the intra-class diversity in the sketch domain. Besides, to enhance the discrimination of obtained representations, our model is trained with the knowledge of the teacher model and label information inspired by previous work on teacher signal [Liu et al., 2019]. To validate the superiority of our DSN method, extensive experiments are conducted and the results demonstrate our method remarkably outperforms the state-of-the-art methods on two popular datasets.

The main contributions of this work are summarized as:

• We propose the DSN model for ZS-SBIR task, where a cross-modal contrastive method is utilized to mine the relations between sketches and images with sufficient positive and negative samples to significantly smooth the domain gap.
• To address the large intra-class diversity in the sketch domain, we elaborate a category-specific memory bank to learn prototypes for each sketch category to reduce the variability in appearance.
• Extensive experiments conducted on two popular datasets, i.e., Sketchy and TU-Berlin, demonstrate that our approach outperforms the state-of-the-art methods by a large margin.

2 RelatedWork

Since ZS-SBIR is an extremely challenging task integrated with SBIR and zero-shot learning (ZSL), we will review both of them in this section.

2.1 Sketch-Based Image Retrieval

The main target of SBIR task is to alleviate the domain gap between image and sketch by aligning their features in a common semantic space. Early works utilize hand-crafted features to represent sketches and match them with the edge maps of images by various Bag-of-Words model. However, it is difficult to align the edge map with hand-crafted features directly, which makes the problem of domain gap unresolved. With the advent of deep neural networks, researchers tend to employ CNN to extract deep features from images and sketches to learn transferrable representation. Among them, Siamese architecture based methods [Qi et al., 2016] play an important role by handling sketches and images separately with ranking loss. Compared with these methods, our DSN model proposes a cross-modal contrastive method to address the domain gap effectively. By pulling together samples with the same label and pushing apart those from different categories in different modalities and views, the cross-modal contrastive method can mine the relations between sketches and images sufficiently to smooth the inherent domain gap.

2.2 Zero-Shot Learning

Zero-shot learning (ZSL) aims to recognize objects whose instances are unseen during the training period [Xian et al., 2018a; Wei et al., 2019; Wei et al., 2020a]. Existing ZSL methods can be classified into two categories: embedding-based [Wei et al., 2020b] and generative-based [Li et al., 2021] approaches. For the embedding-based method, [Kodirov et al., 2017] learned an auto-encoder to map visual features to semantic embeddings, where the nearest neighbor search is conducted. For the generative-based method, [Xian et al., 2018b] proposed a feature generating network to synthesize unseen visual features to mitigate the data imbalance problem in ZSL. Besides, various semantic representation methods have been proposed, including attributes annotations [Akata et al., 2015] and word vectors [Mikolov et al., 2013] to bridge the seen and unseen categories. Due to the space limitation, more details of ZSL can refer to [Xian et al., 2018a].

2.3 Zero-Shot Sketch-Based Image Retrieval

ZS-SBIR is a novel task that conducts SBIR under zero-shot setting and is more practical in realistic applications. [Shen et al., 2018] first combined ZSL with SBIR, and proposed a cross-modal hashing method for ZS-SBIR. [Yelamarthi et al., 2018] built an auto-encoder to generate image features by sketches for alignment, while [Dutta and Akata, 2019] proposed a paired cycle-consistent generative model with adversarial training to align the sketch and image in the same semantic space. [Dutta and Biswas, 2019] presented a feature disentanglement method, where the images generated by sketches and corresponding styles can be conducted in retrieval as queries. [Liu et al., 2019] treated ZS-SBIR as a catastrophic forgetting problem and employed a CNN model pre-trained on ImageNet as the teacher model to preserve the knowledge. However, all these methods ignore the large intra-class diversity in the sketch domain. In this work, we propose a category-specific memory bank to learn prototypes for each sketch category to reduce the intra-class diversity in the sketch domain.

3 Methodology

3.1 ProblemDefinition

In ZS-SBIR, the dataset can be divided into two parts: the training set and the testing set. The training set, denoted as Dseen = {I seen , S seen }, is utilized for training the retrieval model, where I seen and Sseen represent images and sketches from seen categories, respectively. Similarly, the testing set is utilized for validating the performance of the retrieval model, which can be denoted as Dunseen = {I unseen , S unseen }. Mathematically, let I seen = {(Ij, yj)|yj ∈ C seen }N1 j=1 and S seen = {(Si, yi)|yi ∈ C seen }N2 i=1 , where y represents the category label. C seen denotes the seen category set. N1 and N2 denote the number of images and sketches in training set, respectively.

During the training period, we sample batches of images and sketches from the training set, where corresponding labels are utilized to guide the model to learn discriminative features. And at testing stage, a sketch query S′ ∈ S unseen with its label y ∈ C unseen , where C unseen denotes the unseen category set, is given to match the similar image I ′ ∈ I unseen . If the label y′ of the retrieved image is the same as that of the given sketch, i.e., y = y′ , we name this as a successful retrieval. In zero-shot setting, there is no testing category appears in training period, i.e., C seen ⋂ C unseen = ∅.

3.2 Domain-Smoothing Network

The architecture of our DSN model is illustrated in Figure 2. Specifically, we employ a CSE-ResNet-50 backbone (CSE) [Lu et al., 2018] to extract feature embeddings for images and sketches. To smooth the domain gap between sketch and image, we propose a cross-modal contrastive method to align these two domains. Besides, a category-specific memory bank is built to address the large intra-class diversity in sketch domain. The details of these modules will be discussed in this section.

Cross-modal Contrastive Method

The domain gap is an inherent challenge in SBIR task. Previous methods usually use a unified framework to learn a common representation of sketches and images by cross-entropy loss or triplet-based loss. However, cross-entropy loss may neglect the relations between sketches and images as mentioned above. Besides, triplet-based loss is usually utilized to measure the relations between samples. However, due to the limited positive and negative samples, triplet-based loss may not mine the relations sufficiently.

To effectively smooth the domain gap, we propose a crossmodal contrastive method. In detail, for each image I and sketch S from Dseen , two random augmentations are generated for contrastive learning by augmentation method [Khosla et al., 2020], denoted as {ˆ I, ˜ I, ˆ S, ˜ S}. Then CSE is applied to extract feature embeddings for each image, sketch and these augmentations as:

After obtaining feature embeddings f , a projection head with two fully-connected layers, named as P, is utilized for mapping f to vector v = P(f). In this way, we can obtain the vectors for two augmentations of sketch and image, which can be written as {ˆ vimg, ˜ vimg, ˆ vske, ˜ vske} ⊆ V . To illustrate clearly, we denote ˆ vimg as the anchor. It should be noted that even though ˆ vimg and ˜ vimg are in different views, they share the same label. So ˜ vimg is an intra-modal positive sample for ˆ vimg . Similarly, ˆ vske and ˜ vske are cross-modal samples for ˆ vimg. Their roles of positive or negative depends on whether their labels are the same as ˆ vimg. Then contrastive method is conducted to pull together the anchor with the positive samples and push apart the anchor from the negative samples in a cross-modal scenario.

Our cross-modal contrastive method (CMCM) loss can be summarized as:

where for each vector v ∈ V , there is a corresponding label y. |V | represents the cardinality of the vector set V , and Ni =

|{yj|yj = yi, j ̸= i, vi ∈ V, vj ∈ V }|. 1condition ∈ {0, 1} is an indicator function evaluating to 1 iff condition is true. τ is the temperature parameter for contrastive learning.

Category-specific Memory Bank

To reduce the large intra-class diversity in the sketch domain, we propose a category-specific memory bank to learn a prototype representation for sketches from the same category, where the image is enforced to be closer to the prototype of top-k similar sketches at the feature level.

Specifically, for each training batch, we can obtain a batch of feature embeddings {Fimg,Fske}, where Fimg = {fimg|yimg ∈ Cseen} and Fske = {fske|yske ∈ Cseen}. In order to update the memory bank and leverage the history knowledge at the same time, we maintain a stack filled with k sketch features for each category adaptively. First, for fske and its label yske, we search for the corresponding image features F ′ img with the same label in a batch, where F ′ img = {fimg|yimg = yske}. The mean value of corresponding image features f ′ img can be obtained as:

Next, we calculate the cosine similarity between f ′ img and F ′ske ∪ {fske} to select new top-k similar sketch features and store them in the memory bank, where F ′ske is the previous top-k sketch features.

After updating the memory bank, the prototype of new topk sketch features can be obtained as:

whereMske denotes the new top-k sketch features. Then the memory loss can be formulated as:

where | · | indicates the norm of the feature vector.

Discriminative Learning

To learn discriminative representations, we use cross-entropy loss to align the learned feature with its label:

where α and β is the weight and bias of the classifier, and N denotes the number of samples in a training batch.

To bridge the seen and unseen categories for zero-shot setting, auxiliary semantic knowledge should be introduced to transfer the knowledge learned from the seen category. Inspired by [Liu et al., 2019], we employ a teacher model pre-trained on ImageNet to help our DSN model learn transferrable representations.

In detail, we utilize the prediction of the teacher model p ∈ R|CT | as the label information instead of one-hot vector used in cross-entropy loss, where C T denotes the label space of the teacher model and C T ∩C unseen = ∅ in our zero-shot setting. Finally, the auxiliary semantic knowledge (ASK) loss can be formulated as:

where p′ denotes the teacher signal p with semantic constraints proposed in [Liu et al., 2019] and γ, δ are weight and bias of the another classifier, respectively.

The total discriminative learning loss can be formulated as:

3.3 Objective and Optimization

In summary, the full objective function of our DSN model is:

where λ1, λ2 and λ3 are the hyperparameters to balance the performance of our model.

4 Experiments

4.1 Datasets

To verify the effectiveness of our method, We evaluate our DSN method on two popular SBIR datasets, i.e., Sketchy [Sangkloy et al., 2016] and TU-Berlin [Eitz et al., 2012].

Sketchy. It consists of 75,471 sketches and 12,500 images from 125 categories. [Liu et al., 2017] extended the image gallery with additional 60,502 photos so there are 73,002 samples in extended Sketchy dataset totally.

TU-Berlin. It consists of 20,000 sketches uniformly distributed over 250 categories. [Liu et al., 2017] extended this dataset with 204,489 natural images.

For comparison, we follow the split of [Shen et al., 2018] to randomly choose 25 categories and the rest 100 categories for testing and training in Sketchy. For TU-Berlin, 30 categories and the rest 220 categories are chosen for testing and training. Also, we follow [Shen et al., 2018] to select at least 400 images for each category in the testing set.

4.2 Implementation Details

We implement our method with PyTorch on two TITAN RTX GPUs. In our model, we utilize a CSE-ResNet-50 pre-trained on ImageNet [Lu et al., 2018] to initialize the backbone. Inspired by [Liu et al., 2019], we use the same pre-trained backbone to provide teacher signal as auxiliary semantic knowledge. In the training stage, we train our model with Adam optimizer. The initial learning rate is set to 0.0001 and exponentially decays to 1e-7 in training. The batch size in our original setting is 96 and the training epochs is 10. Usually, we set hyperparameters λ1 = 0.1, λ2 = 1 and λ3 = 1 unless stated. For cross-modal contrastive method, we set the temperature τ as 0.07 and the dimension of contrastive vectors is 128 in all the experiments. For category-specific memory bank, we set k = 10.

4.3 Comparing with ExistingMethods

We compare our DSN model with various methods related to SBIR, ZSL and ZS-SBIR. Especially for ZS-SBIR, we employ all the experiments under the same setting as others for fairness. In addition, to generate binary hash codes for comparison, we use iterative quantization (ITQ) [Gong et al., 2012]. Then the hamming distance is computed to conduct retrieval while cosine distance is utilized for real value feature retrieval. The results of our model for ZS-SBIR are shown in Table 1.

As we can see, our DSN method outperforms other existing methods especially in Sketchy dataset with 64-d features. Specifically, in 64-d real value feature retrieval, our DSN outperforms the state-of-the-art (SOTA) method [Dutta and Biswas, 2019; Dutta and Akata, 2019] by 29% in Sketchy and 49% in TU-Berlin dataset, respectively. In 512-d real value feature retrieval, our DSN outperforms the SOTA [Liu et al., 2019] by 7% in Sketchy and 1% in TU-Berlin. For binary hash retrieval, we conduct ITQ [Gong et al., 2012] following the same setting with [Liu et al., 2019] and outperform SOTA by 16% and 7% for Sketchy and TU-Berlin respectively in 64-d setting. Also, we provide 512-d binary hash code retrieval results in the Table 1, which is similar to 512-d real value feature results. It demonstrates that our DSN method can not only generate accurate real value features but also perform stably even in binary hash setting for ZS-SBIR.

All of these results show that our DSN method can alleviate the domain gap between sketch and domain effectively, and reduce the large intra-class diversity in sketch domain which is ignored by most existing methods.

4.4 Experiment Analysis

Quantitative Analysis

Inspired by [Liu et al., 2019], we employ a pre-trained teacher model to provide teacher signal as auxiliary semantic knowledge. So in our experiments, we set λ3 to 1 and explore different λ1 and λ2 for our proposed method.

In Figure 3, we show mAP@all results with different hyperparameters. As we can see, setting λ1 = 0.1 and λ2 = 1 can achieve the best performance with 64-d binary hash codes in Sketchy. It should be noted that we adopt this setting in all the experiments unless stated otherwise.

Qualitative Analysis

Visulization of Similarity. To demonstrate the superiority of our method, we also visualize the similarity between sketch and image in retrieval feature level (seen Fig. 4). Here, with 64-d retrieval features in the testing set of Sketchy dataset. The horizontal and vertical axes represent image and sketch categories respectively. Best view in color.

we calculate the cosine similarity using the mean sketch feature and mean image feature for each category in the testing set of Sketchy. The darkness of matrix color represents the similarity. As shown in Figure 4, our DSN method smooths the domain gap to learn discriminative features, which can boost the performance of retrieval. However, negative cases still exist. For example, the camel and horse are similar in appearance, which leads to a higher similarity value in the matrix compared with other pairs of categories.

Examples of Retrieval. We show top 10 ZS-SBIR results using our DSN model with 512-d real value features in Figure 5. The mis-retrieved results are in the red border. As we can see from these examples, our DSN model performs well in most cases except for some visual similarly images. For example, in the retrieval results of pineapple sketch (the 4th row in Figure 5), a tree image is mis-retrieved, which is similar to the pineapple query in appearance.

4.5 Ablation Study

We conduct four ablation studies to validate the performance of our DSN model: 1) A baseline that learns a common representation for sketches and images with a pre-trained model as teacher model inspired by [Liu et al., 2019]; 2) Adding our cross-modal contrastive method Lcmcm to the baseline that smooths the domain gap by mining relations between samples; 3) Adding our category-specific memory bank Lml to the baseline that reduces the large intra-class diversity; 4) Full DSN model. The experiments are conducted with 64-d binary hash codes for retrieval in Sketchy dataset.

As shown in Table 2, Lcmcm can improve baseline performance by 16% and 5% in Sketchy and TU-berlin respectively, while Lml can improve baseline performance by 14% and 5%. However, combining Lcmcm with Lml improves the baseline only by 21% and 8% for each dataset. It should be noted that the improvement with Lcmcm+Lml isn't a simple addition operation with Lcmcm and Lml. In our knowledge, the reason may be that our cross-modal contrastive method has already reduced the intra-class diversity in sketch domain to some extent, which pulls the sketches with a same label together in a training batch. These two novel methods have intersection in reducing intra-class diversity, which results in the phenomenon mentioned above. Besides, the experiment results demonstrating Imp(Lcmcm) > Imp(Lml), where Imp denotes the improvement from baseline, confirms our conjecture from another perspective.

5 Conclusion

In this paper, we present a novel yet effective DomainSmoothing Network for ZS-SBIR. First, a cross-modal contrastive method is proposed to smooth the domain gap by mining relations between sketches and images with sufficient positive and negative samples. Second, we build a categoryspecific memory bank to address the large intra-class diversity in the sketch domain, where prototypes of each sketch category are learned to reduce the variability in appearance. We conduct extensive experiments on two popular datasets, i.e., Sketchy and TU-Berlin, which demonstrate our DSN method outperforms existing methods by a large margin and validate its superiority.

Acknowledgments

Our work was supported in part by the National Natural Science Foundation of China under Grant 62071361, and in part by the Fundamental Research Funds for the Central Universities ZDRC2102.