Progressive Domain-Independent Feature Decomposition Network for Zero-Shot Sketch-Based Image Retrieval

1 Introduction

With the explosive growth of image contents in our real world, image retrieval has been playing an important role in many fields, such as e-commerce, medical diagnosis and remote sensing. Conventional image retrieval requires providing textual descriptions, which are difficult to be obtained in many real-world cases. On mobile devices, image retrieval with free-hand sketches, for delivering targeted candidates visually and concisely, has attracted increasing attention and formed the area of Sketch-Based Image Retrieval (SBIR).

∗Contact Author

Since it is difficult to guarantee that all categories can be trained in realistic scenarios, unsatisfactory performance is often yielded when testing on unseen categories. In view of this, a more realistic setting is emerged, namely ZS-SBIR, which combines Zero-Shot Learning (ZSL) and SBIR for real-world applications. The ZS-SBIR is a retrieval task under zero-shot scenario, and people can quickly draw sketches on the mobile for retrieval, so it attracted increasing attention in real world. However, ZS-SBIR is extremely challenging since it simultaneously needs to deal with cross-modal matching, significant domain gap, as well as limited knowledge of unseen classes. The traditional SBIR methods [Liu et al., 2017a; Zhang et al., 2018] cannot directly address these problems effectively since they over-fit the source domain and meanwhile neglect the unseen categories. On the contrary, traditional ZSL [Kodirov et al., 2017; Zhang and Saligrama, 2016] methods often focus on solving single-modal problems. Therefore, ZS-SBIR tries to solve these problems sufficiently by combining the advantages of ZSL and SBIR.

As shown in Figure 1(a) and Figure 1(b), previous work attempted to overcome these challenges through simultaneously projecting sketch/image features and label/semantic supervision to a low-dimensional common space. However, these strategies deteriorate the original semantic knowledge, because the low-dimensional projection maps complete semantic embedding from original semantic space to semantically incomplete low-dimensional space. Hence, as shown in Figure 1(c), we present progressive projection strategy that first learns semantic features with original semantic supervision, and then projects them into a common retrieval space, which is beneficial to knowledge transfer due to the strong semantic supervision can be maintained. Another issue is that, the domain information and semantic information are entangled in visual features, and the distribution of two different domains are highly heterogeneous, which seriously hinders the reduction of domain gap between sketch and image, making cross-modal matching difficult. Since semantic information remains the same expression in different domains, we argue that only semantic information is crucial for cross-modal matching. To this end, we decompose the visual features to attain domain-independent retrieval features that only contain semantic information.

In this paper, we have proposed a Progressive DomainIndependent Feature Decomposition (PDFD) network for ZSSBIR task. First, PDFD decomposes visual features into semantic features and domain features, where the semantic features are adversarially learned with the supervision of original semantic embedding, while the domain features are learned with a modality classifier. Subsequently, the learned semantic features are projected into a common space as retrieval features under the category and modality supervision. Besides, in order to reduce the domain interference for cross-modal matching, we introduce cross-reconstruction loss to encourage the retrieval features capture clean and complete semantic information. It is expected that such retrieval features can reconstruct the sketch or image visual features by combining with sketch-domain features or image-domain features. In this network, the parameters of decoders and encoders for sketches and images are shared.

The main contributions of this work are summarized:

• We propose a feature decomposition model to effectively reduce the domain gap by generating domainindependent retrieval features with a novel crossreconstruction loss.
• The proposed progressive projection strategy preserves the strong semantic supervision when generating retrieval features, which is beneficial to knowledge transfer under the zero-shot scenario.
• Extensive experiments conducted on two popular largescale datasets demonstrate that our proposed PDFD significantly outperforms state-of-the-art methods.

2 RelatedWork

In this section, we briefly review the prior literature in the fields of SBIR, ZSL and ZS-SBIR.

2.1 Sketch-Based Image Retrieval

The existing SBIR approaches can be mainly divided into two categories: hand-crafted features based methods and deep learning based ones. The hand-crafted features based methods attempt to bridge the domain gap by using edge-maps extracted from images, such as gradient field HOG descriptor [Hu and Collomosse, 2013] and Learned Key Shapes (LKS) [Saavedra et al., 2015]. As for the deep learning based methods, Yu et al. [2017] first adopted CNN to learn better feature representation for sketches and images. Besides, siamese architecture [Qi et al., 2016] achieves a better metric of retrieval by minimizing the loss function for samples from the same category and maximizing the loss function for samples from different categories.

2.2 Zero-Shot Learning

Existing zero-shot approaches can be classified into two categories: embedding-based and generative-based approaches. In the first category, some approaches learn non-linear multimodal embedding [Akata et al., 2015; Xian et al., 2016]. As for generative-based approaches, Wei et al. [2019] utilize a generator to synthesize unseen features and learn the semantic relations in a common space. It's worth noting that, most of ZSL methods utilize auxiliary information, e.g., a textbased embedding [Mikolov et al., 2013] or a hierarchical embedding [Miller, 1995] for label embedding, which is beneficial to knowledge transfer.

2.3 Zero-Shot Sketch-Based Image Retrieval

The first work [Shen et al., 2018] of ZS-SBIR utilizes a multi-modal learning network to mitigate heterogeneity between two different modalities. The recent work SEMPCYC [Dutta and Akata, 2019] proposes a paired cycleconsistent generative model based on semantically alignment, which maintains a cycle consistency that only requires supervision at category level. Besides, SAKE [Liu et al., 2019; Kodirov et al., 2017] proposes a teacher-student network to maximally preserving previously acquired knowledge to reduce the domain gap between the seen source domain and unseen target domain.

3 Methodology

3.1 ProblemDefinition

We first provide a formal definition of the ZS-SBIR task. Let Dtr = {X seen , S seen , Y seen } be a training set that contains sketch-image pairs X seen = {xske i , ximg i }Ns i=1 , semantic embedding S seen = {sseen i }Ns i=1 , and category labels Y seen = {yseen i }Ns i=1 with Ns samples. The test set is denoted as Dte = {X un , Y un } that contains sketches X un = {xun i }Nu i=1 and category labels Y un = {yun i }Nu i=1 with Nu samples, which satisfies the zero-shot setting Y seen ∩ Y un = ∅. During the test, given an unseen sketch xun i in X un , the objective is to retrieve corresponding natural images from the image gallery which contain all images including training images.

The architecture of our proposed PDFD is illustrated in Figure 2, which contains two branches for sketch and image, respectively. Each branch first decomposes visual features into domain features and semantic features. Subsequently, decoders with shared parameters are trained to project semantic features into retrieval features for subsequent ZS-SBIR task.

3.2 Visual Features Decomposition

In zero-shot learning, it is important to provide knowledge supervision when learning semantic features. Our proposed PDFD utilizes text-based embedding and hierarchical embedding to provide such supervision.

Semantic Knowledge Embedding

In PDFD, we adopt two widely-used text-based embedding, i.e., Word2Vec [Mikolov et al., 2013] and GloVe [Pennington et al., 2014] to obtain text representations. As for hierarchical embedding in PDFD, the similarity between words is calculated in WordNet 1 with path similarity and JiangConrath [Jiang and Conrath, 1997] similarity.

Semantic Features

As illustrated in Figure 2, each branch has a semantic encoder Eθsem , common discriminator Dθdis , and semantic embedding that combine text-based embedding and hierarchical embedding. Given a training sketch-image pair, their visual features are extracted from VGG16 [Simonyan and Zisserman, 2014] network pre-trained on ImageNet [Deng et al., 2009] dataset (before the last pooling layer). Then the semantic features are learned in an adversarial fashion, which means that the learned semantic features are expected to be as similar as the semantic embedding by 'fooling' the discriminator Dθdis . Specifically, the objective can be formulated as:

where xske , ximg , sseen , Eθsem (·) and Dθdis (·) denote the sketch features, image features, semantic embedding, semantic generation function and discriminator function, respectively. Besides, the semantic generation function and discriminator function are parameterized by θsem and θdis. Here, Eθsem minimizes the objective against an opponent Dθdis that tries to maximize it.

Domain Features

Since semantic features and domain features are separated, we argue that semantic, image-domain features and sketchdomain features should be distinguished from each other. Thus, we categorize these three kinds of features into three different modalities. Here, the domain encoder Eθdom is adopted to attain the domain features with the constraint of the modality classifier. The modality classification loss can be

where yske and yimg are labels indicating whether the corresponding features belong to sketch and image. Moreover, xdom ske , and xdom img denote the domain features from sketch and image branch respectively. They can be formulated as:

It is worth noting that the domain features also have been constrained to cross-reconstruction loss, which will be introduced in Section 3.3.

1 https://wordnet.princeton.edu/

3.3 Retrieval Features Generation

After learning semantic features and domain features, PDFD generates retrieval features under two kinds of constraints.

Classification Constraint

It should be noted that the semantic features learned from two branches are only constrained by adversarial loss, which only ensures that the semantic features possess semantic knowledge. However, it can not guarantee the features to be classdiscriminative. Therefore, category classifier is introduced after the two branches. The category classification loss can be formulated as:

where y is the category label of xske and ximg . Moreover, xsem ske , xsem img , xret ske and xret img denote the semantic features and retrieval features generated from sketch and image branch respectively. The generation of retrieval features can be formulated as:

where Dθret (·) is the generation function of retrieval features. On the other hand, retrieval features should be domainindependent, such that they ought to be classified the semantic modality. We adopt the same modality classifier used in generation of domain features to ensure that, where the modality classification loss is written as:

where ysem is the label indicating whether the corresponding features belong to semantic modality.

Cross Reconstruction Constraint

To ensure learning clean and complete retrieval features, we argue that such features should reconstruct the original sketch/image visual features by combined with sketch/imagedomain features. To this end, the cross-reconstruction loss is introduced to ensure the reconstructed features are similar to the original features. These reconstructed features are formulated as:

where Rθ′ske (·) and Rθ′img (·) denote the reconstruction function on the sketch branch and image branch, respectively. Besides, ˜xske 1 and ˜xske 2 denote the reconstructed sketch features; ˜ximg 1 and ˜ximg 2 denote the reconstructed image features. Furthermore, the cross-reconstruction losses in sketch and image branch are written as:

The total cross-reconstruction loss can be formulated as:

3.4 Objective and Optimization

Since the modality loss is constrained both on domain features and semantic features, we can formulate the total modality loss as:

Finally, the full objective of our proposed PDFD is:

where λadv, λccls, λrec and λmcls are coefficients for balancing the overall performance.

4 Experiment

4.1 Datasets and Setup

There are two widely-used large-scale sketch datasets Sketchy [Sangkloy et al., 2016] and TU-Berlin [Eitz et al., 2012] for ZS-SBIR.

Sketchy originally consists of 75,479 sketches and 12,500 images from 125 categories. Liu et al. [2017a] extended the image gallery by collecting extra 60,502 images from ImageNet [Deng et al., 2009] dataset, such that the total number of images is 73,002 in the extended version.

TU-Berlin originally consists of 20,000 unique free-hand sketches evenly distributed over 250 object categories. Compared to Sketchy, TU-Berlin only has category-level matches rather than instance-level matches.

Following the same zero-shot data partitioning in SEMPCYC [Dutta and Akata, 2019], we also follow the same evaluation criterion in most previous works [Dutta and Akata, 2019; Shen et al., 2018] in terms of mean average precision (mAP@all) and precision considering the top 100 (Prec@100) retrievals.

4.2 Implementation Details

PDFD is trained with Adam [Kingma and Ba, 2014] optimizer on PyTorch with an initial learning rate lr = 0.0001, β1 = 0.5, β2 = 0.99. The input size of the image/sketch is 224×224. We use the grid search method to select the best coefficients, which are λadv = 1.0, λrec = 1.0, λmcls = 1.0, λccls = 0.01 when training on Sketch and λadv = 1.0, λrec = 0.5, λmcls = 0.4, λccls = 0.4 when training on TUBerlin. VGG16 [Simonyan and Zisserman, 2014] pre-trained on ImageNet is adopted as a feature extractor. The word textbased embedding [Mikolov et al., 2013] is adopted to extract 300-dimensional word vectors. Under the zero-shot setting, we only consider the seen classes when constructing the hierarchy embedding [Miller, 1995] for obtaining the class embedding. Therefore, the hierarchical embedding for Sketchy and TU-Berlin respectively contain 354 and 664 nodes.

Table 1: ZS-SBIR performance of our proposed PDFD compared with existing SBIR, ZSL and ZS-SBIR approaches.

(a) Retrieval results on Sketchy.

(b) Retrieval results on TU-Berlin.

Figure 3: The top 10 images retrieved by our PDFD on the two datasets. The red borders indicate mis-retrieved images.

4.3 Comparison with Peer Methods

Apart from ZS-SBIR methods, some existing SBIR and ZSL approaches are also involved in retrieval comparison. The performances of all the comparisons are shown in Table 1, where we can observe that most ZS-SBIR methods outperform SBIR and ZSL methods while GN Triplet [Sangkloy et al., 2016] and SAE [Kodirov et al., 2017] reach the best performance in SBIR and ZSL, respectively. The main reason is that SBIR and ZSL methods are unable to take both domain gap and knowledge transfer into consideration. Therefore, the ZS-SBIR methods have better performance as they possess both the ability of reducing the domain gap and transferring the semantic knowledge. Due to the larger number of classes in TU-Berlin, all involved methods perform relatively worse on this dataset compared with Sketchy.

Most of the ZS-SBIR methods are conducted to retrieve based on 64-dimensional features, so PDFD generates 64dimensional retrieval features for retrieval and outperforms the best competitor [Dutta and Biswas, 2019] by more than 13% on Sketchy and 10% on TU-Berlin. However, the SAKE [Liu et al., 2019] adopts the 512-dimensional features for retrieval, and then applies iterative quantization algorithm [Liu et al., 2017b] to obtain the 64-dimensional binary codes. For a fair comparison, PDFD also obtains 512-dimensional retrieval features and 64-dimensional binary codes. The result shows that our model significantly outperforms SAKE by around 11% on Sketchy and 1% on TU-Berlin when adopting 512-dimensional retrieval features.

All of these demonstrate the effectiveness of PDFD for domain gap reduction and semantic knowledge transfer. The retrieved images for sketches are shown in Figure 3. The red borders indicate the incorrectly retrieved images.

4.4 Effect of Semantic Knowledge Embedding

Since different types of semantic embedding have different impact on performance, we analyze the effects of different semantic embedding as well as different combinations of them on retrieval performance based on 64-dimensional retrieval features. Table 2 shows the quantitative results. As we can see, the combination of Word2Vec and Jiang-Conrath [Jiang and Conrath, 1997] hierarchical similarity reaches the highest mAP of 62.3% on Sketchy, while on TU Berlin dataset, the combination of Glove and Jiang-Conrath reaches the highest mAP of 46.0%. Note that we adopt the same embedding setting for all ablation studies.

4.5 Ablation Study

Five ablation studies are conducted to validate the effectiveness of our proposed PDFD as exhibited in Table 3, which are: 1) A baseline that simultaneously projects visual features and semantic supervision into a low-dimensional common space; 2) Adding our progressive projection strategy to the baseline that first learns semantic features with the original semantic supervision, and then projects them as retrieval features under the category classification loss Lccls; 3) Further adding our proposed feature decomposition to attain domainindependent retrieval features under the modality classification loss Lmcls; 4) Replacing the modality classification loss Lmcls in "3)" with Lrec to validate the effectiveness of crossreconstruction; 5) Full PDFD model. Moreover, all the retrieval features in ablation studies are 64-dimensional.

As shown in Table 3, our full model outperforms all baselines. The progressive projection strategy in "2)" improves baseline performance by around 10% on Sketchy and 4% on TU-Berlin, as this strategy benefits semantic knowledge transfer. By further decomposing visual features into semantic features and domain features under the modality classification loss Lmcls, we can derive domain-independent retrieval features and improve the cross-modal retrieval performance. Moreover, the proposed cross-reconstruction loss Lrec encourages learning retrieval features with clean and complete semantic information, which improves the performance by a large margin. Finally, with all proposed modules, the full model reaches the highest mAP@all of 62.3% on Sketchy and mAP@all of 46.0% on TU-Berlin.

5 Conclusion

We have presented a novel progressive domain-independent feature decomposition network to address the problem of ZSSBIR more effectively. On one hand, a progressive projection strategy is exploited to preserve the semantic information with the strong supervision of original semantic knowledge for learning semantic features. On the other hand, the cross-reconstruction loss is imposed to reduce the domain gap by ensuring that the retrieval features capture clean and complete semantic information. Experiments on two large-scale datasets show that our proposed PDFD significantly outperforms existing state-of-the-art methods in ZS-SBIR task.

Acknowledgments

This work is supported in part by the Key R&D ProgramThe Key Industry Innovation Chain of Shaanxi under Grant 2019ZDLGY03-02-01, and in part by the National Key R&D Program of China under Grant 2017YFE0104100 and 2016YFE0200400.