Cross-modal Bidirectional Translation via Reinforcement Learning

1 Introduction

Heterogeneous data of different modalities, such as image and text, have been widely available with huge quantity on the Internet, and they commonly coexist. For example, an image often co-occurs with its corresponding text description on a web page to describe the same semantics such as objects or events. While cross-modal correlation naturally exists between image and text data to describe specific kinds of statistical dependencies. However, the inconsistent representations of different modalities make it very challenging to establish correlation and measure the semantical relevance between them. For addressing the above issue, some works have been done for bridging the gap between heterogeneous data, such as cross-modal retrieval [Rasiwasia et al., 2010; Peng et al., 2017a], where the data of different modalities can be retrieved flexibly by giving a query of any modality at the same time, which is extremely useful for users to retrieve whatever they want across multiple media types.

∗Corresponding author.

Most of existing methods [Feng et al., 2014; Peng et al., 2016; Rasiwasia et al., 2010] follow an intuitive idea to map the data of different modalities into one common space to learn the common representation, so that the general distance metrics can be directly adopted to calculate the similarities between them, and further establish correlation among the heterogeneous data. Traditional methods [Hardoon et al., 2004; Rasiwasia et al., 2010] learn mapping matrices by optimizing the statistical values to project the data of different modalities into the common space. Recently, with the great progress of deep learning, many methods [Feng et al., 2014; Peng et al., 2017b] utilize the strong learning ability of deep neural network to perform cross-modal correlation learning. However, the aforementioned methods mainly project the feature of each modality into one common space, which cannot fully capture the complex cross-modal correlation with such unidirectional projections. Thus, we attempt to conduct bidirectional transformation between image and text, which not only transforms from image to text, but also transforms back to text from image, so as to mine the intrinsic characteristic in each modality and further enhance the cross-modal correlation through the bidirectional learning process.

Inspired by the recent progress of neural machine translation [Cho et al., 2014; He et al., 2016], whose key problem is to establish a corresponding relationship and make arbitrary conversion between two or more entirely different languages, we can intuitively treat images as a special kind of language, where each pixel or each region in one image can be taken as a visual word, and all of them weave together to provide rich visual descriptions. Therefore, we can conduct bidirectional translation between the bilingual pair of image and text to exploit the intrinsic characteristic in each modality and further learn the cross-modal correlation. Besides, deep reinforcement learning has recently attracted much attention. However, most of them mainly focus on video or board games [Mnih et al., 2015], it is still a challenging problem to apply it into other scenarios with no pre-defined rules and explicit reward signals. While the bidirectional cross-modal translation process can be treated as a bidirectional game between image and text, which is a promising way to obtain reward signals for reinforcement learning. Thus we can utilize the power of reinforcement learning to promote cross-modal correlation modeling. Overall, in this paper, we propose Cross-modal Bidirectional Translation (CBT) approach with the following contributions:

• Cross-modal translation mechanism. We treat image and text as bilingual sequence pairs, and utilize recurrent neural network to exploit both fine-grained local and context information within each modality. Furthermore, bidirectional translation training strategy is adopted to translate from one modality to another and also translate back, which can fully capture the intrinsic characteristic in each modality and further enhance the cross-modal correlation through the bidirectional learning process.
• Cross-modal reinforcement learning. We construct a bidirectional game between image and text, which can be played as a round to promote the cross-modal bidirectional translation process through reinforcement procedure. Two kinds of reward signals are designed to extract from inter-modality correlation learning error and intra-modality reconstruction error in both two feature spaces of image and text, which can mutually boost for the cross-modal correlation learning.

To verify the performance of cross-modal correlation learning, we conduct extensive experiments on the cross-modal retrieval paradigm, and our proposed approach achieves the best retrieval accuracy compared with totally 11 state-of-theart methods on 3 cross-modal datasets.

2 RelatedWorks

2.1 Cross-modal Correlation Learning

Traditional cross-modal correlation learning methods [Rasiwasia et al., 2010; Li et al., 2003; Zhai et al., 2014] mainly learn linear projections to map the features of different modalities into one common space, where the similarity of heterogeneous data can be directly calculated by general distance metric on the learned common representations. A class of representative methods utilize canonical correlation analysis (CCA) to optimize the statistical values for the crossmodal correlation learning [Rasiwasia et al., 2010]. Besides, another kind of methods construct graphs to correlate the heterogeneous data in the common space, such as joint representation learning (JRL) proposed by Zhai et al. [Zhai et al., 2014] to adopt graph regularization as well as utilize semisupervised information.

Recently, deep learning based methods [Feng et al., 2014; Peng et al., 2017b] have become mainstream for cross-modal correlation learning. Correspondence autoencoder (Corr-AE) [Feng et al., 2014] is proposed to jointly model the crossmodal correlation and reconstruction information. Andrew et al. [Andrew et al., 2013] integrate CCA with deep network to propose deep canonical correlation analysis (DCCA). Wei et al. [Wei et al., 2017] utilize convolutional neural network to learn strong representation for image and perform deep semantic matching (Deep-SM). Besides, Peng et al. [Peng et al., 2016; 2017b] propose cross-modal multiple deep networks (CMDN) and cross-modal correlation learning (CCL) methods to fully exploit inter-modality and intramodality correlation and further model fine-grained information for better performance. Inspired by the recent progress of generative adversarial networks, there are some attempts [Wang et al., 2017] to adopt adversarial learning for crossmodal correlation modeling.

2.2 Neural Machine Translation

As a classical research topic in natural language process, machine translation aims to establish a corresponding relationship between different languages with both structural and vocabulary differences. Most of the recent works adopt deep neural network to achieve promising results in neural machine translation. Cho et al. [Cho et al., 2014] propose recurrent neural network (RNN) based encoder-decoder architecture, where one RNN encodes a sequence of symbols into a intermediate representation, and the other decodes it into another sequence of symbols. Similarly, Sutskever et al. [Sutskever et al., 2014] propose sequence to sequence learning with neural networks with a general end-to-end method that makes minimal assumptions on the sequence structure. Bahdanau et al. [Bahdanau et al., 2015] improve the basic encoder-decoder architecture by joint learning to align and translate. He et al. [He et al., 2016] adopt dual learning mechanism with reinforcement learning process to automatically learn from unlabeled data. Inspired by the recent progress in neural machine translation, we take images as a special kind of language to conduct bidirectional translation between image and text for cross-modal correlation learning.

2.3 Reinforcement Learning

Reinforcement learning generally address the problem of how the agents learn to optimize their control that maximizes cumulative reward through interactions with the environment. Mnih et al. [Mnih et al., 2015] integrate traditional Qlearning algorithm with multi-layer network to propose deep Q network (DQN). However, most of the existing methods mainly focus on video or board games [Mnih et al., 2013]. It is still quite challenging to apply it into other scenarios. There are some attempts to perform object detection [Caicedo and Lazebnik, 2015] or image caption [Ren et al., 2017]. Inspired by these, we treat the cross-modal bidirectional translation process as a bidirectional game between image and text to obtain the reward signals, and utilize policy gradient methods for reward maximization, which is widely used in reinforcement learning tasks [Sutton et al., 1999].

3 Our CBT Approach

As shown in Figure 1, we propose cross-modal translation mechanism to effectively model cross-modal correlation by bidirectional translation training, taking the translation process as a bidirectional game between image and text, and inter-modality and intra-modality reward signals can be extracted from correlation learning error and reconstruction error to utilize the power of reinforcement learning. We first introduce the formal definition on the multimodal dataset, denoted as D = {I, T} to conduct correlation learning, where I = {ip}np=1 and text T = {tp}np=1 with totally n instances in each modality. ip and tp are the p-th instance of image and text with the semantic category label cp.

3.1 Cross-modal Translation Framework

Inspired by the sequence to sequence model in neural machine translation [Cho et al., 2014], we construct cross-modal sequence model with convolutional recurrent network for both image and text, which can fully exploit the fine-grained local and spatial context information simultaneously.

For the image data, each image ip is fed into 19-layer VGGNet [Simonyan and Zisserman, 2014] to generate separate feature vectors for different regions that contain fine-grained local information, which are obtained from the response of each filter over the a 7 × 7 mapping in the last pooling layer (pool5). These regions are composed as a sequence, which can be regarded as the eye movement result when glancing at the image, and denoted as {vi1 , ..., vin } with totally n regions. Then, long short term memory (LSTM) network is adopted on these image sequence to model the fine-grained spatial context information of image. The LSTM is updated recursively with the following equations:

where i, f, c and o denote the activation vectors of input, forget, memory cell and output respectively. x is the input image sequences, and h is the output from the hidden units. W and U are the weight matrices and b is the bias term. ⊙ denotes the element-wise multiplication. And σ is the sigmoid nonlinearity to activate the gate. Then, the output sequence can be obtained from LSTM and averaged as Hi = 1/n ∑nk=1 hik . Besides, we also generate image feature representation Gi from the last fully-connected layer of VGGNet to exploit the global information of image. The final encoded image representation is the averaged outputs Hi concatenated with global image representation Gi, denoted as Si = {sip }.

For the text data, each input text tp is represented as an n × k matrix, where n is the number of words in tp, and each word has a k-dimensional vector extracted by Word2Vec model, which is pre-trained on billions of words in Google News. Then Word CNN is adopted on the input matrix following [Kim, 2014], which is similar with the CNN for image except the 2D convolution and spatial max-pooling are replaced by temporal (1D) convolution and temporal maxpooling. We still generate the features of text fragments from the activation of last pooling layer, and split as a sequence denoted as {vt1 , ..., vtn }, which contains rich fine-grained local information of text. To further exploit the context information, we also adopt LSTM to model the temporal dependency along the input text sequence, which is updated following the equations (1) to (3), where x denotes the text sequence {vt1 , ..., vtn }. Similarly, the output sequence from LSTM is averaged as Ht = 1/n ∑nk=1 htk , and we also extract the global text representation Gt from the last fully-connected layer of the above Word CNN. They are concatenated as the final encoded text representation, denoted as St = {stp }.

To conduct cross-modal translation between image and text, we construct two-pathway networks, which consist of several fully-connected layers on each pathway. Specifically, image-to-text pathway translates image representation sip to its corresponding text representation stp , which aims to make the translated representation from image as far as possible to be similar with stp . While text-to-image pathway tends to translate text representation stp back to image, which generates the translated representation close to sip . Besides, the translated representations of image and text are also translated back to their original feature space through the other pathway. Furthermore, we also connect the two-pathway networks at the middle layer, which have the shared weights and are followed by a softmax loss layer as semantic constraints, aiming to make image-to-text pathway and text-to-image pathway fully interact and keep semantic consistency.

Algorithm 1 Reinforcement training process of CBT

Require: Image training data Itr, text training data Ttr, batchsize N, hyper-parameter α, learning rate γ.

• 1: repeat
• 2: Sample N encoded image representations from the CNN-RNN based network.
• 3: Generate N translated representations for each image sip with P (·|s; θIT ) as simid,1 , ..., simid,N , and translate back with P (·|s; θT I ) as siori,1 , ..., siori,N .
• 4: for k = 1, ..., N do
• 5: Set inter-modality reward rinter p for the k-th sample with equation (7).
• 6: Set intra-modality reward rintra p for the k-th sample with equation (8).
• 7: Set the total reward of the k-th sample rp.
• 8: end for
• 9: Compute stochastic gradient of θIT by equation (9)
• 10: Compute stochastic gradient of θTI by equation (10)
• 11: Model updates:

• 12: Go through the above process from step 2 to 11 symmetrically for the game beginning from text stp .
• 13: until CBT converges
• 14: return Optimized CBT model.

3.2 Reinforcement Learning Procedure

We design a bidirectional game between image and text, which is played as a round to realize the bidirectional translation process with reinforcement learning. Specifically, the two-pathway networks mentioned in Section 3.1 for crossmodal translation are denoted as P(·|s; θIT ) and P(·|s; θTI) respectively, where θIT and θTI are their parameters.

For the game begins with one image sip in Si , the first state is to translate it into text feature space to get the translated representation simid,p , and we can extract the inter-modality reward rinter p , which can be obtained from correlation error that indicates the similarity between simid,p and a sampled text instance stp . Then, the second state is to conduct translation from simid,p back to original image space as siori,p , which can generate intra-modality reward rintra p from reconstruction error. Thus, the total reward is calculated as follows:

where α is the parameter to balance the two rewards. The two-pathway networks are trained through policy gradient methods for maximizing reward, which is widely used in reinforcement learning. The reinforcement learning process is defined as follows:

Note that we translate image to text and generate N candidate translated representations from Si to form a mini-batch, and each of them is sampled with one text representation. The probability P (simid,p |Si ; θIT ) indicates the relevance between the p-th candidate and its sampled target text, which means those candidate pairs consisting relevant image and text in same category can get higher rewards compared with other irrelevant pairs. Similarly, we conduct the game beginning with text stp with two states, namely to translate into image first, and then translate back to text. Thus the reinforcement learning process is symmetric with equation (5).

Then, taking the game beginning from image as an example, the details of objective function are introduced in the following parts. First, two kinds of rewards rinter p and rintra p in the above equation are defined as:

where ‖·‖2 denotes the 2-norm, and "norm" means to normalize the similarity score to [0, 1], which can be represented as the probability that indicates how similar the translated representation is with its corresponding sample, and those relevant candidate pairs would have larger similarities than others. With the objective function defined in equation (5), we calculate the stochastic gradient of θIT and θTI according to policy gradient theorem as follows:

Finally, we summarize the reinforcement training process of proposed CBT in Algorithm 1. Besides, the gradient from semantic constraint is summed with policy gradient to preserve semantic consistency during reinforcement learning process. Specifically, we obtain the intermediate representation from the middle shared layer to establish correlation between image and text, which can preserve the semantic constraints and get benefit from both inter-modality and intra-modality rewards from the cross-modal reinforcement learning process.

3.3 Implementation Details

Our proposed CBT approach is implemented by TensorFlow. The Word CNN contains 3 convolution layers, followed by ReLU activation and max-pooling. Their parameters are (384,15)→(512,9)→(256,7), where the first means the number of convolution kernels and the second is the kernel width. For image, the pre-trained CNN of 19-layer VGGNet is adopted to obtain the for image representations. The LSTM for image and text have two units in series, whose output has the same dimension with input as 300. Besides, the global image representation has 4,096 dimensions from VGGNet and global text representation has 300 dimensions from Word CNN respectively. Each of them are concatenated with the output from LSTM. Finally, two-pathway network consists of 5 fully-connected layers (4,396→3,000→2,000→1,000→600) from image to text on each pathway. The two pathways are tied at middle layer followed by softmax function for semantic constraints.

4 Experiments

4.1 Datasets

The brief introduction of 3 cross-modal datasets adopted in the experiments is given in the following paragraphs.

Wikipedia dataset [Rasiwasia et al., 2010] has 10 categories with 2,866 image/text pairs. We follow [Peng et al., 2016; Feng et al., 2014] to split it into 3 subsets, namely 2,173 pairs for training, 231 for validation and 462 for testing.

Pascal Sentence dataset [Rashtchian et al., 2010] contains 1,000 images with totally 20 categories, and each image has 5 independent sentences. Following [Peng et al., 2016; Feng et al., 2014], 800 image/text pairs are selected for training, while 100 pairs for testing and 100 pairs for validation.

XMediaNet dataset [Peng et al., 2017a] is a large-scale cross-modal dataset with 200 categories, and has 40,000 image/text pairs, which are divided into 3 subsets, 32,000 pairs for training, 4,000 for testing and 4,000 for validation.

4.2 EvaluationMetric and ComparedMethods

To comprehensively evaluate the performance of cross-modal correlation, we perform two cross-modal retrieval tasks as: retrieving text by image query (Image→Text) and retrieving image by text query (Text→Image). We adopt mean average precision (MAP) as the evaluation metric, which is calculated on all returned results for comprehensive evaluation. It should be noted that not only top 50 returned results are calculated in Corr-AE [Feng et al., 2014] and ACMR [Wang et al., 2017], while the rest returned results are not considered.

The proposed CBT approach is compared with 11 state-ofthe-art cross-modal retrieval methods to fully verify its effectiveness, including 5 traditional cross-modal retrieval methods, namely CCA [Rasiwasia et al., 2010], CFA [Li et al., 2003], KCCA [Hardoon et al., 2004], JRL [Zhai et al., 2014] and LGCFL [Kang et al., 2015], and 6 deep learning based methods, namely Corr-AE [Feng et al., 2014], DCCA [Andrew et al., 2013], CMDN [Peng et al., 2016], Deep-SM [Wei et al., 2017], CCL [Peng et al., 2017b] and ACMR [Wang et al., 2017]. For fair comparison, all the compared methods adopt the same CNN features for both image and text, which are extracted from the CNN architectures used in our approach. Specifically, the CNN feature for image is extracted from the fc7 layer in 19-layer VGGNet [Simonyan and Zisserman, 2014] with 4,096 dimensions. While the CNN feature for text is extracted fromWord CNN with the same configuration of [Kim, 2014] with 300 dimensions.

4.3 Comparisons with State-of-the-art Methods

The experimental results are shown in Tables 1, 2 and 3, which include the MAP scores of two retrieval tasks and their average results on 3 datasets. Obviously, our proposed CBT approach achieves the best retrieval accuracies. Among all the compared methods, we can draw the follow observations: First, most deep learning based methods achieve better retrieval accuracies than the traditional methods, where CCL has the best performance, which verifies the effectiveness of deep network. Second, traditional methods benefit from the CNN feature to get better performance than their original works with hand-crafted features, and even some of them have close accuracy with deep learning based methods, such as JRL and LGCFL. Compared with the state-of-theart methods, our proposed CBT approach achieves promising improvement with following 2 reasons: (1) Cross-modal translation strategy to conduct bidirectional transformation between image and text to capture the cross-modal correlation in the original feature space of each modality, while the compared methods mainly model the cross-modal correlation through unidirectional projections that limit their performance. (2) Cross-modal reinforcement learning is adopted to extract intra-modality and inter-modality rewards in a bidirectional game, which can model correlation and reconstruction information in both two feature spaces of image and text simultaneously, while the compared methods only model them in single intermediate transformation.

4.4 Parameter Analysis and Baseline Comparisons

We conduct parameter experiment on the effect of key parameter α in reward function 4. The value of α ranges from 0.1 to 0.9, and results are shown in Figure 2. We further evaluate the performance with only single reward in Table 4, where "CBT-inter" means that only inter-modality reward extracted from correlation learning error is adopted, while "CBT-intra" means only intra-modality reward from reconstruction error.

From the above results, we have the following observations: (1) Compared with "CBT-baseline" which only adopts semantic constraint between two-pathway network, the two rewards can further promote the cross-modal correlation learning. (2) The retrieval accuracy becomes highest when α is larger than 0.5, and "CBT-inter" also has better performance than "CBT-intra", which indicates that correlation learning plays a more important role than modeling reconstruction information. (3) Compared with "CBT-inter" that only considers the unidirectional translation from one to another, CBT outperforms it with bidirectional translation to fully capture the cross-modal correlation. (4) CBT outperforms all the baseline methods, which verifies the effectiveness on the integration of two rewards to further promote the accuracy of cross-modal retrieval.

5 Conclusion

In this paper, we have proposed Cross-modal Bidirectional Translation (CBT) approach to conduct bidirectional translation between image and text. First, a cross-modal translation mechanism is designed to model the cross-modal correlation as well as exploit the fine-grained local and context information in original feature space of each modality. Second, crossmodal reinforcement learning is proposed to jointly model the correlation and reconstruction information as two kinds of rewards in the bidirectional game played as a round between image and text. Extensive experiments verify the effectiveness of our proposed CBT approach. In the future work, we attempt to perform unsupervised learning to exploit unlabeled data for practical applications.

Acknowledgments

This work was supported by National Natural Science Foundation of China under Grant 61771025 and Grant 61532005.