Knowledge Aware Semantic Concept Expansion for Image-Text Matching

1 Introduction

Image and text matching is an important vision-language cross-modality task for many real-world applications including image retrieval [Xie et al., 2016; Huang et al., 2018] and caption [Karpathy and Fei-Fei, 2015; Lin and Parikh, 2016]. Before calculating the similarity between images and text, a matching model needs to obtain a rich representation of the images first. Inspired by the achievements of computer vision tasks, most of the current image-text matching models utilize pre-trained neural networks to extract mid-level feature embeddings as the representations of images. Although these representations can obtain global visual information of the images, they fail to extract high-level semantic information. So the semantic gap between images and language is not well addressed and it leads to the limited performance when matching an image with text.

∗ This work was done during the first author's intership in MSR Asia

† Corresponding Author

Recently, some works have tried to learn semantic enhanced image representations for the image-text matching task. For example, [Karpathy and Fei-Fei, 2015] proposed to align image region with text words for image caption. [Huang et al., 2018] exploited a multi-label CNN detection model to extract semantic concepts and then fused these concepts with global context of images, which has made significant improvement in image-text matching. However, these methods are limited to small size of detected concepts, since they depend on existing detection models normally dealing with common concept candidates. Therefore, these methods perform poorly if the concepts mentioned in the text cannot be detected via pre-trained detection models.

We argue that it is necessary to detect more accurate concepts from an image to help obtain a richer image representation. Take the example of Figure 1 as an example, current detection models (e.g. [Wang et al., 2017]) could only detect related concepts cat and laptop from the given image. If the concept candidates are expanded to include more specific concepts like calico and keyboard, we can get higher semantic similarity score between the image and text. However, it is challenging to detect long-tail, even absent concepts based on existing detection models. These models are usually trained on a small-size concept vocabulary to avoid a sharp decrease of detection performance with a extended vocabulary. As shown in the Figure 1, the concept calico in the query text is a long-tail concept which is absent in the ldetection concept vocabulary and thus cannot be detected.

To address this issue, we incorporate external scene knowledge to privilege the imagination to the model. In this work, we mainly focus on utilizing co-occurrence common-sense knowledge which can be extracted from a large number of images. E.g. laptop and keyboard; sky and cloud; boat and water; tree and leaves are frequently appear in the same image. We define this common-sense knowledge as Scene Concept Graph (SCG). It provides rich prior scene information to expand more semantic concepts of images which are often noticed and described in query text. We directly use human annotated scene graphs from Visual Genome [Krishna et al., 2017]. Scene graph of an image is a graph consisting of concepts and relationships between them. It can be represented as a set of triplets of <subject, relation, object>, where subject and object are concepts, and relations are the interactions and relationships between them. Then we aggregate co-occurred <subject, object> pairs from scene graphs of all images. After that, we clean the co-occurrence pairs by considering the appearance times and construct the SCG. After that, we can utilize this knowledge to infer more concepts. Figure 1 is an intuitive example of image-retrieval by SCG. In this case, the Scene Concept Graph provides common-sense information to expand the occluded concepts keyboard in the image. Since laptop and keyboard frequently appeared together, although the keyboard is occluded by the cat, it still can be inferred by co-occurrence pair in SCG. Furthermore, in order to avoid noisy concepts expanding, we used a neural network to select the semantic concepts related to the image. By introducing the SCG, the performance of our model outperforms state-ofthe-art models.

Our contributions are summarized as below:

1. We build a Scene Concept Graph by considering cooccurrence pairs of semantic concepts in scene graph of images, which incorporate common-sense information.
2. We propose a novel model to expand more semantic concepts by the Scene Concept Graph and selectively fuse them to enhance image representation semantically.
3. We conduct extensive experiments and the results show that our model achieves state-of-the-art performance.

2 RelatedWork

Embedding based methods. DeViSE [Frome et al., 2013] is the first embedding based method which projected image features and skip-gram word features by a linear mapping and calculated similarity accordingly. [Faghri et al., 2017] introduced a simple modification on the loss function. [Kiros et al., 2014] adopted a sequence to sequence pipeline to learn joint embedding space with a language model. [Wang et al., 2016] proposed DSPE, a structured preserving network match image and text embeddings. [Zheng et al., 2017] proposed using a dual task to embed image and text to a shared visual-textual space discriminatively. [Wang et al., 2018] introduced two-branch embeddings and considered the novel neighborhood constraints. Cross-modality joint embedding is a baseline model but lack of semantic understanding.

Semantic knowledge based methods. [Karpathy and FeiFei, 2015] took advantage of R-CNN to detect local regions from an image and aligned them to each word. [Huang et al., 2018] proposed learning semantic concepts and orders to improve image representation. This work is most similar to ours. The main difference is that we incorporate commonsense scene information to expand more contextual semantic concepts precisely. Many recent state-of-the-art models are attention based models. [Lee et al., 2018b] proposed SCAN to use stacked cross attention to align image and text in a finer-grained model. Different from previous models, our work incorporated external common-sense scene information to enhance the representation of image and the experiment results showed a significant improvement.

Knowledge for image/text tasks. Combining a knowledge graph with deep learning for image-text tasks is related to our work. Knowledge can be used for image tasks like image classification [Marino et al., 2017], zero shot classification [Lee et al., 2018a] and for text tasks [Zhou et al., 2018]. There are also several image and text tasks incorporating external knowledge like image captioning [Mogadala et al., 2017]and VQA [Wu et al., 2016]. For cross modality image and text match, [Wang et al., 2006] used multi-modality ontology to retrieve images. [Belilovsky et al., 2016] aligned scene graphs with images by the algorithms of bag-of-words, subpath representations and neural network. The difference is that we utilized the Scene Concept Graph which captures the co-occurred object pairs in the same scene as common-sense knowledge to expand more occluded and long tail concepts.

Our work starts from embedding based methods and utilized semantic concepts by incorporating Scene Concept Graph to enhance the representation of image/text embedding.

3 Scene Concept Graph Based Image-Text Matching

We formulate the image-text matching problem as a ranking model similar to [Wang et al., 2018] and our model is depicted in Figure 2. Given the input image and the query text, the output is the similarity score of matching two modalities. We generate the pre-trained query text encoding and image encoding separately and jointly embed them into the same space by maximizing the margin of positive and negative image and text pairs. As shown in Figure 2, in order to incorporate contextual concepts to enhance image representation, we first adopt a Concept Detection Module to extract accurate concepts on a small vocabulary. Then we use the assembled Scene concept Graph to expand more contextual concepts by Concept Expansion Module and learn to predict more accurate semantic concepts by the Concept Prediction Module. Next we fuse whole-image encoding generated by Vision Feature Module with learned concepts to generate an enhanced image representation by the Image-Concept Fusion Module.

Finally we calculate the cosine similarity score with text encoding by Text Encoding Module and optimize the triplet loss function [Schroff et al., 2015].

In the rest of this section, we first introduce the input text encoding. And then we describe the pre-trained whole-image encoding and the semantic-image feature encoding in details. The final loss function will be presented in the last part of this section.

3.1 Text Feature Encoding

We adopt LSTM [Hochreiter and Schmidhuber, 1997] as our text encoder to acquire the representation of text. For each word wi in query text with N words, we train an embedding of xi from scratch and input a token in text S = {xi|xi ∈ Rk }i=1...N at each time step t to the LSTM model. We use the hidden state after feeding the last token as the representation of text ft.

3.2 Image Feature Encoding

Visual Feature Module

For the whole-image representation (denoted as fi ∈ Ri ), we directly use a ImageNet [Deng et al., 2009] pre-trained VGG19 [Simonyan and Zisserman, 2015] network and take the output of the last convolutional layer (16 th layer) as the image feature: we first resize each image to 256 × 256 and conduct a 5-cropping 1 . A left-right mirror flipping on each image is also applied to get 10 different pre-processed copies. Then, we feed this 10×224×224×3 (RGB 3-channel) image data matrix into the feature extractor and obtain a 10 × 4096dimensional output of the model's first FC layer. After that, we average these feature vectors as the whole-image representation: fi ∈ R4096 .

1 Top-left, top-right, bottom-left, bottom-right and center.

Concept Detection Module

In order to bridge the semantic gap of text and image, we need to detect semantic concepts first. We use a multi-label images classification model [Wang et al., 2017] to detect whether a concept appears in an image. The detection model aims to produce a multi-hot vector gd ∈ [0, 1]|Vd| as the detected concepts of the image, where Vd is the vocabulary of detectable concepts and Vc is the vocabulary of all semantic concepts; Vd ⊆ Vc (mostly, |Vd| ≪ |Vc|). For each i-th concept, we used 1 or 0 on i-th of vector to indicate existence or absence respectively. E.g. if the Vd = {people, cat, laptop, cake}, and gd of Figure 1 would be [0, 1, 1, 0]. After that, we can directly fuse gd with its whole-image feature encoding fi to enhance image representation.

According to our experiment, although the detection model using the small concept vocabulary can guarantee the accuracy of concept detection, it will neglect long-tail but informative concepts. However, the performance of the detection model will drop significantly when using a large concept vocabulary. This motivates us to come up with a method to discover as many concepts as possible without accuracy loss. Therefore we introduce the Scene Concept Graph to expand more frequently queried concepts based on images and their description.

Concept Expansion Module

To learn to expand the concepts, we first build a commonsense Scene Concept Graph using Visual Genome [Krishna et al., 2017], which is a dataset consisting of human-labeled scene graphs. We aggregate all co-occurring concept pairs in scene graphs of images to build a Scene Concept Graph: KSCG = {t|t =< cs, co >; cs, co ∈ Vc} where Vc is the concept vocabulary including all concepts. Figure 3 illustrates the construction of SCG. The SCG can provide strong contextual signals for semantic image understanding. For example, House can be found frequently in the pair <House, Window> and Tree always accompanies with Leaves in the pair <Tree, Leaves>. After we detect House and Tree from an image, expansion model will predict neglected concepts Window and Leaves by the common-sense scene information.

We formulate the concept expansion model as the algorithm 1. In this algorithm, we iterate all concepts gd detected by the concept detection module as a seed to retrieve from the SCG. For each concept c, we extract all co-occurred concepts and aggregate all these concepts as expanded semantic concepts ga ∈ [0, 1]|Vc| . This procedure is depicted in figure 4.

Concept Prediction Module

Although we can expand a lot of undetected semantic concepts by SCG, it is obvious that some noisy concepts irrelevant to the image will also be expanded, leading to even worse performance. How to selectively learn relevant concepts of the image is a challenge to us. For example, with a high probability, laptop co-occurred with mouse, which is a noisy for the example in Figure 1. So we propose a mechanism to selectively learn and predict whether a concept is relevant to the image or not.

To learn relevant concepts, we construct a prediction model which takes the whole-image encoding fi; the detected con-

Algorithm 1 Concept Expansion

Require: detected concepts of an image Ld ⊆ {c|c ∈ Vd}

Require: Scene Concept Graph KSCG

Ensure: expanded concept vector ga ∈ [0, 1]|Vc|

• 1: La ←Ld
• 2: for c in Ld do
• 3: Lsubject ← {cs| < cs, co >∈ KSCG, co = c}
• 4: Lobject ← {co| < cs, co >∈ KSCG, cs = c}
• 5: La ←La ∪Lsubject ∪Lobjects
• 6: end for
• 7: convert concept list La to a multi-hop concept vector ga ∈ [0, 1]|Vc| .

cept vector gd and the expanded concept vector ga as input and output the relevant concept vector ˆ gc ∈ [0, 1]|Vc| , where 1 indicates the corresponding concept is in the image and 0 otherwise.

In order to predict whether concepts are relevant or not, we first fuse expanded concepts ga and whole-image representation fi by:

Then, we construct a model to predict gc directly:

where Mi ∈ Ri×k ; Mg ∈ R|Vc|×k ; Mc ∈ Rk×|Vc| ; bi, bg ∈ Rk ; bc, bm ∈ R|Vc| , σ is the sigmoid function and ◦ is the element-wise product. ˆ gc ∈ R|Vc| is the prediction of ground-truth gc ∈ [0, 1]|Vc| . ELU is Exponential Linear Units [Clevert et al., 2016].

To train this prediction model, we use description text of images to build a pseudo labels by matching the concepts with text literally. We heuristically label concept vector using 0, 1 as gc ∈ [0, 1]|Vc| , where 1 means the concept appeared in the description text literally and 0 means the concept is in the text.

During training procedure, First, we use the text to label the expanded concepts ga to generate the expected concepts gc. Then we train the prediction model to make the learned concept vector ˆ gc similar to expected concept vector gc. We use the log-exponential objective function to train this prediction model:

Our overall model is trained by an end-to-end pipeline, and we add this prediction loss Lcp to the final loss function. After training, we fix the prediction model and directly run the model on inference phase.

Image-Concept Fusion Module

In this section, we will describe how we conduct imageconcept fusion.

We fuse the predicted concept vector ˆ gc ∈ R|Vc| with whole-image encoding fi ∈ R4096 to get a final image representation fci ∈ Re where e is the dimension of embedding vector. In this paper, we use two different fusion methods.

The baseline fusion method named element-wise product. First we map each of these two modality inputs into the same embedding space by projecting with Wg ∈ R|Vc|×e and Wf ∈ Ri×e respectively. Then, we normalize these two mapped embeddings by L2-normalization and combine them by element-wise product and f (mul) ci ∈ Re is the enhanced image representation.

The second fusion method is called gated fusion [Huang et al., 2018], which selectively fuses concepts and efficiently parameterizes interactions between two modalities. We use a gated mechanism to learn to fuse those two vectors by the following equations:

where Wg, Ug ∈ R|Vc|×e ; Wf , Uf ∈ Ri×e and the f (sco) ci ∈ Re is the concept-enhanced image feature.

3.3 Loss Function of Joint Learning

To learn image and text matching as well as image-relevant semantic concepts jointly, our loss function consists of two parts: prediction loss Lcp and image-text matching loss Lm so as to train the model in an end-to-end style. As for the loss Lm, we use cosine similarity to calculate the score. After generating the representation of image and text, we embed them jointly into the same space by maximizing the margin of positive and negative samples. We use triplet loss [Schroff et al., 2015] for two modality encodings:

where x and y are encodings of two modality, N(x) is the set of negative samples of x, d(·) is the similarity function and [x]+ = max(0, x).

In order to train a discriminative model, we also adopt a neighborhood constraint [Wang et al., 2018; Kiela et al., 2018]:

where P(xi) is the set of positive samples for xi (e.g. all descriptions of the image that described by xi).

Finally, we merge these ranking constraints into one loss function with the semantic concept prediction loss Lcp as we discussed before:

where I is the image set and D is the descriptions of all images.

4 Experiments

We conducted extensive experiments to evaluate the performance of our model and compared different settings of our method to state-of-the-art algorithms. We also conducted ablation studies to analyze the effectiveness of incorporating different vocabulary size of semantic concepts for the final image-text matching task.

4.1 Datasets

Visual Genome [Krishna et al., 2017] is a structured dataset that contains hundreds of thousands of dense annotated images. It has 2.3M human annotated scene graphs for 108,077 images, which is represented as a set of triplets including concepts and relationships. After the normalization by synsets of each object and the clearing of tailed triplets, we collected 7,699 different concepts as our vocabulary to construct a dataset for training the concept detection module. We aggregated the triplets, which appear at least 1000 times, to build the Scene Concept Graph which includes 121,307 concept pairs.

MSCOCO [Lin et al., 2014] is a large-scale dataset which contains 123,287 images (the combination of train2014 and val2014), each accompanied with 5 captions which are the text used to query the image. We follow [Karpathy and Fei-Fei, 2015] to prepare the training, validation and test dataset by splitting all images to 113,287 (for training), 5,000 (for validation) and 5,000 (for test). For the evaluation on MSCOCO 5K setting, we used all these 5K testing images and their captions (25K). We used 1/5 of these testing dataset (1K images, 5K captions) for MSCOCO 1K evaluation in order to compare with some algorithm which report their result only on MSCOCO 1K dataset.

Flickr30K [Young et al., 2014] is a dataset that consists of 31,783 images of events, activities and scenes and 158,915 captions which are used to query the image. We followed the split in [Karpathy and Fei-Fei, 2015] and [Faghri et al., 2017] that used 1,000 images for testing and 1,000 images for validation and the rest of them (28,783 images) for training.

4.2 EvaluationMetrics

For the evaluation metric, we adopt the widely used measurement recall at K (R@K) for both sentence retrieval and image retrieval task. To be specific, each image in MSCOCO and Flickr30K dataset has 5 sentences as ground-truth, and each sentence has 1 corresponding image. Take image retrieval as an example, we rank the similarity scores of all images and select top K candidates. We regard the query as a "successful query" if the target image presents in these candidates. And the R@K is the proportion of success in all queries.

4.3 Implementation Details

In our two-modality joint embedding network, we used LSTM with 1024 hidden units to encode text of images and a VGG19 [Simonyan and Zisserman, 2015] pre-trained by ImageNet as our image feature extractor. The dimension of whole-image representation is i = 4096 (fi ∈ R4096 ). The dimension of concept-enhanced image representation and text representation is e = 512 (ft, fci ∈ R512 ).

In order to study the effectiveness of the concept detection model, we tried several different size of vocabularies by picking up the most frequent concepts in all images. The vocabulary sizes various from 256, 512 to 1024. Experiments indicated that we can get the best results by setting vocabulary size to |Vd| = 512. Extensive experiments were conducted and will be explained in the ablation study section later. For the expansion and prediction module, we used all 7,699 concepts as Vc, i.e. |Vc| = 7699.

We used λ1 = 5.0, λ2 = 1.0, λ3 = 1.5 and λ4 = 0.05 as the hyper-parameters of loss function. An Adam Optimizer was adopted to optimize model's parameters. We also utilized batch normalization before non-linear activation and the dropout mechanism after non-linear activation in order to promote speed of training.

4.4 Comparison with State-of-the-Art Models

Table 1 lists the experiment results on MSCOCO dataset and a comparison with other methods. We tried two different evaluation datasets that using 1,000 and 5,000 test images respectively. Prod (Element-wise production) and Gated Fusion are the two different fusion methods. From the table we can see that our algorithm outperforms all of the previous state-ofthe-art methods on both 1K and 5K splits and Gated setting except R@5 result on image retrieval for the 5K dataset. SCO [Huang et al., 2018] is our baseline which also learned the semantic concepts from an image to facilitate this task. From the result, we can see our method significantly improved the matching performance on all datasets and settings over SCO

which demonstrates Scene Concept Graph provides effective semantic information to understand the image. SCAN [Lee et al., 2018b] is the most recent report on this task which is using stacked co-attention on a finer-grained level. Although our work and SCAN follow different research directions, our model can still get better results in most cases. Another conclusion is that Gated Fusion consistently outperforms the Production Fusion, which proved that the carefully designed mechanisms fuse better.

Table 2 shows the experiment results on the Flickr30K dataset. Our model with Gated Fusion can achieved the best results on most cases which is similar to the experiment conclusion of MSCOCO.

4.5 Ablation Studies

We also investigated the effectiveness of detection, expansion, and prediction models by several ablation experiments. Table 3 shows the results of matching. First, we tried to explore the impact of different vocabulary sizes of the detection model. The vocabulary sizes are set to 129, 256, 512, 768 and 1024 separately. From the result, we can see that either small the size or the large size got worse results than the middle size 512, which demonstrated the existing shortcoming of the detection model. We set the vocabulary size to 512 in the rest of the experiments.

We conducted another experiment on semantic concepts generated using different methods. Since Gated Fusion always achieved the best results, we used all Gated Fusion mechanisms in the rest of the experiments. First, SCO [Huang et al., 2018] in Table 1 shows our baseline result without expansion. Other results are shown in Table 3. Next we used the detected concepts gd ∈ R512 to expand more concepts ga ∈ R7699 by Scene Concept Graph without prediction

(Expansion Only) which got a worse result than our baseline. This is caused by introducing many noisy concepts. Then our best model (Gated Fusion) achieved a much better result. We also conducted an experiment on a ground-truth dataset, which we used to train our prediction model. This is the upper bound of our algorithm and shown that there is still a gap to predict the semantic concept well.

4.6 Statistics of Concept Expansion

Table 4 presents a statistics of concepts of the dataset. Detected Concepts indicates the numbers of semantic concepts that extracted by multi-label image classification model. The second row is the numbers of concepts that expanded by Scene Concept Graph. The third row is the numbers of semantic concepts expanded by SCG and filtered by caption tokens, which is the ground-truth label for training prediction model.

4.7 Case Study and Analysis

We ran our model on the real case in Figure 1 and inspected all intermediate results of semantic concept detection, expansion, and prediction models. The results are shown in Figure 5. Although some noisy concepts were introduced, our model can discover the concepts mentioned in text. Besides, we also presented several real showcases of retrieval using different methods. Because of the limited space, please check out this link: https://goo.gl/izcSN9.

5 Conclusion and FutureWork

In this paper, we proposed learning semantic concepts using co-occurred common-sense knowledge for image-text matching. Our main contribution is to utilize the Scene Concept Graph we extracted by Scene Graph of Visual Genome to expand more semantic concepts, which can deal with the challenges of detecting occluded and long-tailed concepts. Experiment results demonstrated that our method can achieve state-of-the-art results. We also conducted intensive ablation studies which showed the effectiveness of incorporating the Scene Concept Graph.

In the future, we will consider using triplets in scene graphs by taking the relationship between concepts into account. We would also like to extend our scene knowledge aware model for more cross-modality tasks.

Acknowledgements

We thank the reviewers for their carefully reading and suggestions. This work was supported by the National Natural Science Foundation of China (No. 61370137), the National Basic Research Program of China (No.2012CB7207002), the Ministry of Education - China Mobile Research Foundation Project (2016/2-7)