Dual Adversarial Networks for Zero-shot Cross-media Retrieval

1 Introduction

Multimedia data has been an important part of our life including image, text, video, audio and other types, and human cognition of outside world is through the fusion of multiple sensory channels, such as vision and audition. Thus, it is quite important for artificial intelligence to effectively process and utilize multimedia data. In that case, many multimedia analysis tasks have been widely studied [Peng et al., 2017;

∗ corresponding author.

Query data of unseen categories

Figure 1: The brief illustration of zero-shot cross-media retrieval.

Xian et al., 2017; Li et al., 2018], and cross-media retrieval becomes a research hotspot, which retrieves data of different media types using a query of any media type [Peng et al., 2017]. Different from single-media retrieval, the key challenge of cross-media retrieval is that the distributions and representations of different media types are inconsistent, which makes it hard to measure the similarity among cross-media data. Moreover, the collecting and labeling of cross-media data are heavily labor-consuming and time-consuming for numerous and dynamically increasing categories in the real world. Cross-media retrieval methods usually require training data of all the retrieval categories to train the retrieval models, which lack of extensibility to retrieve data of new categories.

Zero-shot learning [Larochelle et al., 2008] is a promising solution for recognizing new categories with limited training categories. Inspired by this, we propose zero-shot crossmedia retrieval by extending zero-shot learning to the crossmedia scenario, which aims to retrieve data of new categories across different media types. Different from existing zero-shot learning for single-media analysis, zero-shot crossmedia retrieval intends to achieve retrieval across multiple media types in zero-shot scenario where there are no overlaps between categories of training and testing data. A brief illustration of zero-shot cross-media retrieval is shown in Figure 1, in which categories of query data are never seen in the training. We denote known categories in the training as seen categories, and new categories not included in the training as unseen categories. It is very challenging to handle not only the inconsistent semantics across seen and unseen categories but also the heterogeneous distributions of cross-media data.

For addressing the above problems, this paper proposes a Dual Adversarial Networks for Zero-shot Cross-media Retrieval (DANZCR) approach, which exploits word embeddings as semantic space, and transforms data of different media types into semantic space via an adversarial learning framework. More specifically, the proposed approach establishes generative adversarial networks (GANs) in a dual structure, where the forward GAN learns from the input image and text to generate common representations in semantic space; and the reverse GAN uses the generated common representations to reconstruct the input image and text for preserving the original data structure. Our DANZCR approach performs zero-shot learning and correlation learning at the same time, which can generate common representations to conduct zero-shot cross-media retrieval.

The main contributions of this paper can be concluded as follows:

• A dual GANs architecture is proposed for zero-shot crossmedia retrieval, which consists of forward GAN and reverse GAN. The dual GANs collaboratively promote each other, which capture the underlying data structures, as well as strengthen relations between input data and semantic space to generalize across seen and unseen categories.
• An adversarial training method is proposed for zero-shot cross-media retrieval, which learns common representations by discriminating the generated common representations from which media types and categories to preserve the inherent cross-media correlation. Word embeddings are exploited for generating common representations to model semantic information via the adversarial training process, which enhances the knowledge transfer to unseen categories.

Comprehensive experimental results show the effectiveness of our proposed approach for zero-shot cross-media retrieval on three widely-used cross-media retrieval datasets: Wikipedia, Pascal Sentence and NUS-WIDE.

2 RelatedWork

2.1 Cross-media Retrieval

Cross-media retrieval aims to measure similarities among data of different media types, which are of different feature spaces. Most existing methods focus on common representation learning, which can be divided into traditional methods and Deep Neural Networks (DNN) based methods. Traditional methods mainly learn linear projections for different media types. Canonical correlation analysis (CCA) [Rasiwasia et al., 2010] is proposed to learn cross-media common representation by maximizing the pairwise correlation. Joint representation learning (JRL) [Zhai et al., 2014] is proposed to make use of semi-supervised regularization and semantic information, which can jointly learn common representation projections for up to five media types. Kang et al. [Kang et al., 2015] use a local group based priori to exploit popular block based features and jointly learn basis matrices for different media types. Recently, DNN-based cross-media retrieval has become an active research topic. Wang et al. [Wang et al., 2017] propose an adversarial learning method to learn common subspace, which is implemented as an interplay between feature projector and modality classifier. Wei et al. [Wei et al., 2017] propose to use CNN model pre-trained on ImageNet as the feature extractor for images, and show the effectiveness of CNN feature in cross-media retrieval. However, existing methods usually require that testing categories are the same as training categories, which cannot satisfy the requirement of extensibility in real-world applications.

2.2 Zero-shot Learning

Zero-shot learning aims to address the increasing difficulty caused by collecting data for the large number of categories. Most existing methods take advantage of external knowledge, such as encyclopedias and attributes, to improve the scalability for recognizing the categories beyond training set. Textual descriptions in Wikipedia articles are used as external knowledge in [Elhoseiny et al., 2014] to learn visual classifiers for images. Wu et al. [Wu et al., 2014] make use of information from search engine to address zero-shot event detection. Farhadi et al. [Farhadi et al., 2009] describe objects with manually annotated attributes to classify unseen objects. In [Ba et al., 2016], the attributes are extracted for unseen image classification from textual descriptions. However, these methods are mostly designed for single-media scenario, which cannot support cross-media retrieval. Our DANZCR approach exploits word embeddings from trained Natural Language Processing model as external knowledge, and performs zero-shot learning and correlation learning at the same time to conduct zero-shot cross-media retrieval. To the best of our knowledge, our proposed DANZCR is the first approach to address the problem of zero-shot cross-media retrieval.

2.3 Generative Adversarial Networks (GANs)

Generative adversarial networks (GANs) have achieved impressive results after the work by Goodfellow et al [Goodfellow et al., 2014]. Most of the existing GANs-based works focus on the generative problem to generate new data. Radford et al. [Radford et al., 2015] propose deep convolutional generative adversarial networks (DCGANs) to generate images from a noise input by using deconvolutions. Mehdi et al. [Mirza and Osindero, 2014] propose conditional GANs (cGANs) to add a condition on both the generator and discriminator to generate data instead of uncontrollable noise input. Ledig et al. [Ledig et al., 2017] propose super-resolution generative adversarial networks (SRGANs) with a perceptual loss function, which consists of an adversarial loss and a content loss. Recent works on image translation, such as DualGAN [Yi et al., 2017] and CycleGAN [Zhu et al., 2017], explore GANs in a dual task to transform image style between two image domains. However, the aforementioned works only deal with the generative problem from one media type to another through one pathway network structure, which cannot model the distribution over the cross-media input. Inspired by works on image translation, we consider zero-shot cross-media retrieval as a domain transform problem between input space and semantic space. We utilize GANs as basic modules in a dual structure for each media type, and generate common semantic representations in a multi-pathway network for cross-media data.

3 Our DANZCRApproach

3.1 ProblemDefinition

We take image and text media types as examples to give the formulation of zero-shot cross-media retrieval. The cross-media dataset is represented as D = {DUD,DUQ,DSD,DSQ}, where DUD, DUQ, DSD and DSQ denote unseen category data set, unseen category query set, seen category data set, and seen category query set respectively. For each subset, DUD = {in, tn, ln}NUD n=1 , where in, tn and ln denote n-th instance of image, text and corresponding category label respectively, and NUD is the total number of the instances in DUD. Similarly, DUQ = {in, tn, ln}NUQ n=1 , DSD = {in, tn, ln}NSD n=1 , and DSQ = {in, tn, ln}NSQ n=1 . Notably, there is no overlap categories between seen category set {DSD,DSQ} and unseen category set {DUD, DUQ}. Our goal is to retrieve relevant instances of different media types. Specifically, during the training stage, only the seen category data set DSD can be used to train the method. In the testing stage, to distinguish between zeroshot scenario and conventional scenario, two retrieval tasks are introduced: zero-shot cross-media retrieval takes queries fromDUQ to retrieve relevant cross-media data in DUD, and conventional cross-media retrieval use queries fromDSQ and search their relevant instances in DSD.

3.2 Architecture of DANZCR

The overview illustration of our DANZCR approach is shown as Figure 2. A pair of GANs: forward and reverse GANs in a dual structure are adopted to form two parallels for both image and text. For each media type, the data original representations are first extracted and the forward GAN generates common representations from the original representations, and then the reverse GAN transforms the generated representations to data original representations as a dual process. Each GAN consists of a generative model and a discriminative model. We introduce the detailed network architectures of the forward and reverse GAN respectively as follows:

1) Forward GAN: The forward GAN is designed to generate semantic common representations with category word embeddings as supervised information, in order to model correlation across different medias and categories. GF I and DF I denote forward generative model and discriminative model of forward GAN for image, and GFT and DFT are for text.

The forward generative models generate common representations from image and text original representations, which are built with several fully-connected layers. For image instance in, the image original representation is defined as f in and common representation is sin , f tn and stn are for text instance tn. So there are equations sin = GFI(f in ) and stn = GF T (f tn ).

The forward discriminative models are also made of several fully-connected layers for image and text. Both of them attempt to discriminate which media types and categories the generated common representations belong to. The forward discriminative models take concatenation of generated common representation and data original representation as input. The output is a single value to predict the generated common representation is real or not, as well as discriminate the semantic correlation between common representation and data representation. DFI tries to determine the word embedding for corresponding category sln as the real data, while generated common representation sin , generated common representation of corresponding text stn and word embedding for irrelevant category sl′n as the fake data. DFT is similar to DF I .

The objective function of forward GAN can be defined as:

2) Reverse GAN: The reverse GAN is designed to reconstruct the original representations of input data, which makes common representations to preserve the data original structure, as well as strengthens relations between input data and semantic space. GRI and DRI denote reverse generative model and discriminative model of reverse GAN for image, and GRT and DRT for text.

The reverse generative models are built with several fullyconnected layers , which learn from common representations of the forward GAN to generate the reconstruction representations of input data original representations. The reconstruction representations are defined as rin and rtn for image and text respectively. So there are equations rin = GRI(GFI(f in )) = GRI(sin ) and rtn = GRT (GFT (f tn )) = GRT (stn ).

The reverse discriminate models of image and text are also constructed by several fully-connected layers. They take the data representation as input and output a single value to predict the data representation is real or not. DRI attempts to distinguish image representation f in as real with the generated reconstruction representation rin as fake. DRT is similar for text.

The objective function of reverse GAN is:

However, for zero-shot cross-media retrieval, there are usually a few data of limited known categories for training. If there are large amounts of data, the model has the capacity to learn transform mode from input data to common representation in semantic space, which is unrealistic and meaningless.

So to further enforce the correlation learning, a recovery objective function via the l2 norm is proposed as follows:

According to the above definitions, our DANZCR can be trained by jointly learning of two dual GANs for both image and text. The full objective function of DANZCR approach is:

3.3 Training Procedure

With the defined objective functions, the generative model and discriminative model can play a minimax game with each other, which are trained iteratively in an adversarial process. The parameters of generative model are fixed during the discriminative model training stage and vice versa. The optimization procedure of our approach are presented as follows:

1. Common representation generation: The image and text original representations are extracted and then the common representations are generated from the forward generative models.
2. Optimizing forward discriminative models: The forward discriminative models are trained by ascending stochastic gradient for image and text respectively as follows:

where N is the number of an instance batch.

1. Representation reconstruction: The reconstruction representations are generated by the reverse generative models from generated common representations.
2. Optimizing reverse discriminative models: The stochastic gradient equations can be defined to optimize the reverse discriminative models for image and text respectively as:

1. Optimizing forward generative models: The equations to optimize the forward generative models for image and text are as follows:

1. Optimizing forward and reverse generative models: The forward and reverse generative models are updated by descending the stochastic gradient as follows:

where the parameter λF and λR control the relative contribution of the recovery objective terms.

3.4 Implementation

We adopt TensorFlow 1 to implement our model with a base learning rate 1-4 and dropout probability 0.9. The parameters λF and λR are set to 1-2 . The word embeddings for categories are the 300-dimensional features extracted by Word2Vec model [Mikolov et al., 2013], which are pretrained on Google News. The image representation model has the same configuration with 19-layer VGGNet [Simonyan and Zisserman, 2014] and 4,096 dimensional feature vector from fc7 layer is extracted as the image original representation. The 300-dimensional text original representation is extracted by Doc2Vec model [Le and Mikolov, 2014],which is also pre-trained on Google News. Then the forward generative models with three fully-connected layers are adopted for both image and text to generate common representations with each layer following a ReLU layer and a dropout layer except the last. The number of hidden units are 4,096, 4,096 and 300. The reverse generative models of both image and text are composed of three fully-connected layers to reconstruct image and text representations, with the 4,096 hidden units for the first two layers. The forward and reverse discriminative model have similar structure of three fully-connected layers with 4,096, 2,048 and 1 hidden units, which project the input data into a single value as a predict score to distinguish real or fake.

4 Experiments

4.1 Dataset

Wikipedia dataset [Rasiwasia et al., 2010] is widely used for cross-media retrieval evaluation. It is based on "featured articles" in Wikipedia, which contains 2,866 image/text pairs with 10 high-level semantic categories. The dataset is randomly split into two parts: 2,173 pairs are selected as training set and 693 pairs are selected as testing set.

Pascal Sentence dataset [Farhadi et al., 2010] is selected from 2008 PASCAL development kit, which contains 1,000 image/text pairs organized into 20 categories. Each image instance has 5 sentences as description. There are 800 image/text pairs selected as training set and 200 image/text pairs for testing set.

NUS-WIDE dataset [Chua et al., 2009] consists of about 270,000 images with their tags categorized into 81 categories. Only the images exclusively belonging to one of the 10 largest categories in NUS-WIDE dataset are selected for experiments following [Zhuang et al., 2013], and each image along with its corresponding tags is viewed together as an image/text pair with unique category label. Finally, there are about 70,000 image/text pairs, where training set consists of 42,941 pairs and testing set consists of 28,661 pairs.

For all three datasets, we conduct two cross-media retrieval tasks to evaluate our approach comprehensively: zeroshot retrieval and conventional retrieval according to the definition in Section 3.1. Inspired by [Xian et al., 2017;

1 www.tensorflow.org

Liu et al., 2017], we employ a new dataset split setting based on the original split settings, which is shown in Figure 3. The training set and testing set in the original split setting are further divided into two sets: seen category set and unseen category set respectively, and each set includes 50% categories.

It is noted that input features and evaluation protocol in this paper are different with the compared methods. For the original results of the compared methods, the features are mostly obtained with fine-tuning models, and evaluation protocol is designed to test only on seen categories. While in this paper, we use input features without fine-tuning and the evaluation protocol is designed to test on both seen and unseen categories. For fair and objectively comparison, we directly re-run the source codes of all compared methods provided by their authors.

4.2 EvaluationMetrics

Taking Image→Text as an example, we take each image as a query, and measure the cosine distance between the common representations of the query image and all texts. Finally, we get a ranking list according to the distances and then compute the mean average precision (MAP) to evaluate the retrieval results. The MAP scores are computed as all queries' mean of average precision (AP), and AP is computed as:

where R denotes relevant item number in testing set (according to the category label in our experiments), Rk denotes the relevant item number in top k results, n denotes the testing set size, and relk = 1 means the k-th result is relevant, and 0 otherwise.

4.3 Experimental Analysis

Among the compared methods, LGCFL, JRL and CCA are traditional methods, while Deep-SM and ACMR are DNNbased methods. The MAP scores for zero-shot cross-media retrieval are shown in Table 1, in which our DANZCR approach achieves the best results on the three datasets. We can observe that for the compared methods there is no obvious gap between results of DNN-based methods and traditional methods, and some traditional methods are even higher than DNN-based methods. This is because testing categories of zero-shot retrieval are not included in the training, which may limit the learning ability of DNN-based methods. The MAP scores for conventional cross-media retrieval are shown in Table 2, and our proposed DANZCR approach also achieves the best retrieval accuracy among all the compared methods. For conventional cross-media retrieval, the DNN-based methods are all higher than traditional methods.

Results of conventional cross-media retrieval are obviously higher than zero-shot cross-media retrieval for all methods. Because zero-shot cross-media retrieval is more challenging in which categories of testing data are never seen during the training. Our DANZCR achieves the best accuracy compared with other methods for both zero-shot and conventional crossmedia retrieval, for exploring the external knowledge from the word embeddings of different categories via a dual GANs structure, which can learn more discriminative common representations and ensure the generalization ability across seen and unseen categories to support zero-shot cross-media retrieval.

To further present the effectiveness for each component in our proposed approach, three kinds of baseline experiments are conducted on Pascal Sentences dataset, which are shown in Table 3 and 4. "DANZCR-Single" means only the forward GAN is used without the reverse GAN. "DANZCR-NoDis" means discriminating terms are not used in the adversarial training method, which refers to removing the items log(1 - DFI(stn , f in )), log(1-DFT (sin , f tn )), log(1-DFI(sl′n , f in )) and log(1 - DFT (sl′n , f tn )) in Equation 1. We can observe that our DANZCR approach obtains higher accuracy than "DANZCR-Single", which demonstrates that reverse GAN model can effectively preserve semantic consistency for each media and collaboratively promote forward GAN to generalize across seen and unseen categories. Results of "DANZCRNoDis" decline compared with DANZCR, which indicates that discrimination information across different media types and categories are both beneficial to learn more discriminative semantic common representations.

5 Conclusion

This paper has proposed a Dual Adversarial Networks for Zero-shot Cross-media Retrieval (DANZCR) approach, which performs zero-shot learning and correlation learning at the same time to generate common representations for zeroshot cross-media retrieval. The dual GANs architecture consists of forward GAN and reverse GAN to collaboratively promote each other, which captures the underlying data structures, as well as strengthens relations between input data and semantic space to generalize across seen and unseen categories. Besides, the adversarial training method discriminates the generated common representations from which media types and categories to preserve the inherent cross-media correlation, which makes the generated common representations model semantic information and enhances the knowledge transfer to unseen categories.

The future work lies in the following aspects: Firstly, external knowledge will be effectively utilized to build connections between seen and unseen categories. Secondly, a large number of unlabeled data can be exploited in the future to form an unsupervised model.

Acknowledgments

This work was supported by National Natural Science Foundation of China under Grant 61771025 and Grant 61532005.