Multi-modal Sentence Summarization with Modality Attention and Image Filtering

1 Introduction

Sentence summarization is a well-studied task that creates a condensed version of a long sentence. Sequence-to-sequence (seq2seq) model that encodes a source sequence into a latent representation and outputs another sequence is the dominating framework for sentence summarization [Rush et al., 2015; Takase et al., 2016; Zhou et al., 2017; Li et al., 2017b]. Intuitively, readers can easier grasp the gist of the event by scanning the image than by reading long sentences, and thus we believe that the image will also reduce the difficulty for machine to understand a news event [Li et al., 2017a]. As shown in Figure 1, the two source sentences contain complex details and multiple event objects, leading to unsatisfactory summaries using the text-only model. However, it is easy to see that the paired images visualize the event highlights which can help to produce better summaries. To explore the effectiveness of images, we introduce in this work a new multimodal sentence summarization (MMSS) task that generates a short summary based on a pair of sentence and image.

Multi-modal approaches are explored in machine translation (MT) recently and outperform text-only models [Calixto et al., 2017; Helcl and Libovick´y, 2017; Caglayan et al., ]. The multi-modal MT requires that the paired image should describe the whole information covered by the source sentence and this requirement is hard to meet when the sentence is very long. In contrast to multi-modal MT, images which usually indicate event highlights are more suitable for summarization that aims at abstracting the important information from the original text.

To tackle the MMSS task, we design a novel seq2seq model with hierarchical attention mechanisms. The bottom attention layer focuses on the inner parts of image and sentence. The top attention layer balances the distribution of different modalities. Since some abstract concepts, such as guilty and freedom, can only be well encoded in textual modality [Paivio, 1990; Hill et al., 2013; Kiela et al., 2014], we design image filter modules to block out visual noises. In addition, we incorporate different visual features into our model to initialize target language decoder and present a visual coverage mechanism to handle the word repetition problem.

Our main contributions are as follows:

• We introduce an MMSS task that can generate a textual summary from a pair of sentence and image.
• We propose an inner-modality and inter-modality atten-

tion model to focus on image patches or text units for summary generation.

• We propose image filters to selectively use visual information to benefit summarization.
• We construct a multi-modal sentence summarization corpus that consists of 66,000 summary triples (sentence, image, summary). The experimental results on this dataset demonstrate that our proposed system can take advantage of multi-modal information and outperform other baseline methods.

2 Background: Seq2seq Learning

In this section, we describe the basic seq2seq learning framework. Given a source sequence x, the seq2seq model maximizes the conditional probability of a target sequence y: p(y|x). The encoder and the decoder are the two basic components in the seq2seq model. Recurrent neural networks (RNN) encoder [Cho et al., 2014] converts x into a context representation c as follows:

where ht ∈ Rn is a hidden state at time t, and ct is a context vector generated from the sequence of the hidden states. fenc and fc are nonlinear activation functions.

The decoder generates word yt given the context vector ct and the previously generated words {y1, · · · , yt-1}:

where st is the hidden state of the decoder and fdec is a nonlinear activation function that computes the probability vector for output words at time t. The loss function Lt for each time t is the negative log likelihood of the target word yt:

3 Our ProposedModel

3.1 Overview

We begin by defining the multi-modal sentence summarization task. The input of the task is a pair of sentence and image, and the output is a condensed summary. As shown in Figure 2, our proposed model consists of four modules: sentence encoder, image encoder, summary decoder and image filter. The sentence encoder is a bidirectional GRU (BiGRU) [Chung et al., 2014], and pre-trained CNN (Oxford VGGnet [Simonyan and Zisserman, 2014]) is used to encode images. Our summary decoder is a uni-directional GRU with a hierarchical attention mechanism and a softmax layer over the target vocabulary to generate words. Specifically, beyond the attention over regions of the image and words of the input sentence, we explore the modality-based attention to navigate our model to pay different attention to textual and visual information when generating different words. We design two types of image filters, namely, image attention filter and image context filter, to remove visual noises and enrich the semantic meaning of the corresponding sentence.

Multi-modal Input

Figure 2: The framework of our model.

3.2 Sentence Encoder

Given a source sentence x = (x1, · · · , xn), we employ a BiGRU to build its hidden representation (h1, · · · , hn).

The BiGRU encodes source sentence forwardly and backwardly to generate two sequences of the hidden states: (-→h 1, · · · , -→h n) and (←- h 1, · · · , ←- h n), respectively, where:

E[xi] is the embedding for word xi.

The final sentence representation hi is the concatenation of the forward and backward vectors: hi = [-→h i; ←- h i].

3.3 Image Encoder

Given an image, we apply a pre-trained VGG (VGG19) model to extract visual features. We extract a 7×7×512 feature map of the last pooling layer (pool5 layer) of the network as the local feature, and we extract the 4096-dimensional fullyconnected layer (fc7 layer) as the global feature q. We flatten the local patch feature into matrix A = (a1, · · · , aL) (L = 49) and al ∈ R512 corresponds to a patch of the image.

3.4 Summary Decoder

At each time step t, the state of the decoder st is calculated as follows:

In text-only seq2seq model, the decoder's hidden state s0 is initialized as follow:

Intuitively, s0 is important because the first fewwords to be decoded significantly influence the remainder of the generated summary. We incorporate three different visual features into our model to initialize target language decoder: image global features (Equation 9), the average of image local features (Equation 10) and weighted image local features (Equation 11).

where αi is the weight scalar for ai:

where σ denotes sigmoid activation function. The underlying intuition for Equation 11 is that different image patches do not make equal contributions aligning to the source sentence; for instance, background regions often contain trivial information, which would have less contribution to the model than foreground regions. To do so, the image patch and the sentence are matched to determine informative patches.

Context vector ct in Equation 7 is obtained by a modalitybased attention mechanism as follows:

where βtxt t is attention weight for text context vector ctxt and βimg t is attention weight for image context vector cimg, respectively, calculated as follows:

We compute the text context vector ctxt as a weighted sum of the source annotations as follows:

where each vector is weighted by the attention weight αtxt t,i , as calculated in Equations 17 and 18 as follows:

We compute the image context vector cimg as a weighted sum of the source image patch representation A using the "soft" attention strategy [Xu et al., 2015]. This process is very similar to the text attention strategy assuming that decoder needs to attend to different image patches to generate different words.

where each visual vector al is weighted by the attention weight αimg t,i , as calculated in Equations 20 and 21 as follows:

The probability for the next target word yt is computed using the multi-modal hidden state st, and the previously emitted word yt-1 as follows:

p(yt|{y1, · · · , yt-1}) ∝ exp(Lytanh(Lsst +LeE[yt-1]) (22) where Ly, Ls and Le are model parameters.

In addition, to solve the problem of repetition, we adopt the textual coverage mechanism [See et al., 2017] which aims to avoid generating repetitive target words by penalizing repetitive attention to the same position of the source sequence. For coverage model, the attention distributions over all previous decoder time are maintained, which record the attention history for all source words. Then the decoder is not expected to pay current attention to the words which have received enough attention before. For our task, we apply a multi-modal coverage mechanism including textual and visual coverage.

In our multi-modal coverage model, we maintain two coverage vector ctxt t,i and cimg t,i , which is the sum of attention distributions over all previous decoder steps:

Intuitively, ctxt t,i and cimg t,i denote the degree of coverage that the source words and the image patches have been covered by the target before time t, respectively. Note that ctxt 0,i and cimg 0,i are zero vectors, because at that step, none of the source word or the image patch is covered by the target.

The coverage vectors are used to calculate the attention mechanism, changing Equation 17 and 20 to:

The coverage losses for text and image at time t are defined as follows respectively:

Finally, we add the multi-modal coverage loss to the original loss function in Equation 4 to yield new loss functions as follows:

3.5 Image Filter

Compared to text summarization task, the key point for multimodal summarization is effectively making use of image. Meanwhile, we should avoid introducing noises in the case that (1) the image fails to represent some semantic meanings of words, such as abstract concepts, or (2) the image is insufficiently exact to capture the keypoint of the source sentence. To this end, we introduce an image filtering mechanism and propose two types of image filters, i.e., an image attention filter and an image context filter.

Image Attention Filter

The image attention filter is a scalar applied to the image attention βimg t (in Equation 15), which aims to re-balance the attention of the image and sentence based on the following aspects: (1) The correlation between the source image and sentence, which is measured by the image global feature q and the initial state of summary decoder s0. (2) The relation to the next target word, which relates to the state of the decoder st-1. The image attention filter Ia ∈ [0, 1] is calculated as follows:

Then, the image attention βimg t in Equation 15 is updated as follows:

Xu et al. [2015] and Calixto et al. [2017] adopt gating scalars to image context. The difference is that our scalar not only relates to the current decoder state for generating the next target word but also associate different modalities, which aims to evaluate the effect of the image.

Image Context Filter

The image context filter is an element-wise filtering vector applied to the image context cimg (in Equation 19) that is expected to filter out the image noises inside the image context. The image context filter is calculated as follows:

The image context in Equation 19 is updated as follows:

Image context filter is partially inspired by gating mechanism which has gained great popularity in neural network models [Hochreiter and Schmidhuber, 1997; Srivastava et al., 2015], and this filter is expected to give fine tune for each dimension, and the information can only flow on the dimensions where the gate is "open". In conclusion, image attention filter is directly applied to change the attention scale between image and text, and image context filter is designed to select the most salient visual features.

4 Related work

4.1 Seq2seq SummarizationModels

Rush et al. [2015] are the first to apply the seq2seq model to abstractive sentence summarization. They propose an attentive CNN encoder and a neural network language model [Bengio et al., 2003] decoder. Chopra et al. [2016] use RNN as the decoder and achieve better performance. Nallapati et al. [2016] further replace the encoder with an RNN, forming a full RNN seq2seq model. Gu et al. [2016] and Zeng et al. [2016] incorporate a copying mechanism into seq2seq learning and Gulcehre et al. [2016] propose a switch gate to control whether to copy from the source or generate a word by the decoder. See et al. [See et al., 2017] present a seq2seq model with coverage mechanism to eliminate repetition. Current seq2seq summarization methods only focus on text, while the opportunity to optimize the summary quality with the aid of visually grounded representations is ignored.

4.2 Multi-modal Seq2seqModels

Multi-modal MT has recently attracted much attention from researchers. Libovick´y and Helcl [2017] propose multisource seq2seq learning with hierarchical attention combination. Calixto and Liu [2017] use images as words in the source sentences to improve translation quality. They use visual representation to initialize the encoder or decoder. Our work is partially inspired by the models of Libovick´y and Helcl [2017] and Calixto et al. [2017] with some differences: (1) Libovick´y et al. [2017] use the hierarchical attention but do not pay attention to image patches. In addition, they do not apply image filters. (2) Calixto et al. [2017] propose two separate attention mechanisms to decode target words, while our model can hierarchically pay attention to source words, image patches and different modalities. (3) Calixto et al. [2017] apply a gating scalar which only relates to decoder state, but our image filters not only relate to the current decoder state for generating the next target word but also consider the correlations between different modalities, which is peculiar for sentence summarization tasks. Our motivation is that (1) Different image patches relate to different words and image filters enable our model to selectively use the image. (2) Hierarchical attention mechanism can decide how much attention the decoder pay to different modalities when generate different words. (3) Our image filters explore association between textual and global visual information, which can measure the extent to which the image can capture the gist of the source sentence.

5 Dataset Construction

To the best of our knowledge, there is no benchmark dataset for the MMSS task, and we construct a corpus. Each sample in our corpus is a triple (sentence, image, headline) in which the sentence-headline pair is from the annotated Gigaword corpus [Rush et al., 2015] 1 and the image is crawled from Yahoo! Image Search. The Gigaword corpus provides 3.8 million first-sentence-headline pairs. For each first-sentence, we search Yahoo! Image Search, and crawl the top-5 ranked images. Next, we delete the explicit trivial images such as portraits, thumbnails and advertisements.

As a result, we collect 123,839 sentences paired with five images each. Then we employ 10 graduate students to select the best-match image for each sentence. If there is no matching image for the sentence, the annotators label '0'. Each five-image-sentence sample is annotated by two students and the samples with accordance annotations are preserved. Finally, we collect 66,000 samples. Following the settings of the Gigaword summarization corpus, we randomly split our corpus as a training set with 62,000 samples, a test set with 2,000 samples and a development set with 2,000 samples.

1 https://github.com/harvardnlp/sent-summary

6 Experiment

6.1 Comparative Methods

We compare a set of sentence summarization and multimodal MT baselines. Lead baseline uses the first 8 words as the summary. Compress model [Clarke and Lapata, 2008] produces a compressed result based on the syntactic structure of the original sentence. ABS [Rush et al., 2015] uses an attentive CNN encoder and a neural network language model decoder to summarize the sentence. SEASS is a state-of-theart sentence summarization systems, which employs a selective encoding model to control the information flow from the encoder to the decoder. Multi-Source [Libovick´y and Helcl, 2017] and Doubly-Attentive [Calixto et al., 2017] are multimodal MT baselines (introduced in Section 4). Text-only model is a seq2seq model with attention only using a sentence to generate summary. We report performances of our multi-modal (MM) models with different decoder initializations, image attention filter, image context filter, and coverage for different modalities.

6.2 Experimental Settings

We set word embedding size to 300 and GRU hidden state sizes to 512. We use the full source and target vocabularies collected from the training data, which have 36,916 and 26,168 words, respectively. We use dropout [Srivastava et al., 2014] with probability of 0.2. We set the initial learning rate for Adam [Kingma and Ba, 2015] to 5 × 10-4 . At training time, we test ROUGE-2 [Lin, 2004] F1 score on the development set for every 2,000 batches, and we halve the learning rate if model performance drops. Our models typically converge within 50 epochs using an early stopping strategy for our MMmodel without coverage. To obtain our final coverage model, we add the coverage loss to the objective function for further training with the initial learning 5 × 10-5 . At test time, we use beam search with beam size 10 to generate the summary. We report ROUGE F1 score including ROUGE-1 (R-1), ROUGE-2 (R-2) and ROUGE-L (R-L).

6.3 Experimental Results

Table 1 shows the results of the comparative methods and our proposed methods. The state-of-the-art sentence summarization systems SEASS performs better than our Text-only model. For our MM models, the model without an image filter performs only slightly better than the text-only model, and the models with image filters outperform the text-only model and other baseline models. This proves that image filter plays a significant role in the MMSS task. Initializing decoder with image features is effective for our task. Compared to global and average of local visual features, the models with weighted average of local visual feature achieve greater improvements over the text-only model, which suggests that different image patches do not make equal contributions for our task. Coverage mechanism leads to further improvement, which achieves +2.70% R-1, +2.17% R-2, +2.57% R-L improvement over the Text-only model) for our best model. In addition, we have tested an image captioning model [Vinyals et al., 2015] and get poor results (R-1 is lower than 10%), which indicates that image captioning model is not suitable for this task.

6.4 Analysis

In this section, we discuss the effectiveness of visual information for sentence summarization task (results are reported on the test set).

1. Effectiveness of multi-modal initialization of decoder. From Table 1 we observe that incorporating images to initialize hidden state of decoder improves the performance. We conjecture that image features are effective as these features can capture the highlights of the source sentences. Table 2 shows that our image attention filter model with multimodal decoder initialization can predict the first words that appear in the reference summary in much higher accuracies than that with text-only initialization. Example summaries can be found in Figure 1. In part at least, this behavior can be ascribed to better initialization of decoder. Furthermore, we observe the model which predicts the first word correctly achieves 0.235 higher ROUGE-1 score on average than the model which predicts wrong in our experiment.

2. Are images helpful to produce more abstractive sum-

• mary? Abstractive summarization can produce novel words that do not appear in source sentence. This phenomenon is common for human-written summaries, but is challenging for a machine. We believe that attention mechanism allows a model to focus on the specific positions of source representation that will help to predict the next word, but it adds a constraint on the context vectors for summarization system, which tends to predict a word obtaining the greatest attention in the source. Table 3 shows that our multi-modal model is more abstractive than text-only model. This is expected because the attention to the image can provide abundant information beyond source text. An example is shown in Figure 3.
• Which kind of images are more useful? To answer this question, we ask 3 graduate students to annotate the matching scores for 300 sentence-image pairs in the test set. The score ranges from 1 to 3: 1 denotes "partially match", 2 denotes "basically match" and 3 denotes "completely match". The results in Table 4 show that the samples with better matched images achieve higher ROUGE score and also attract higher attention for image, indicating that better matched images are more useful for the MMSS task.
• Effectiveness of multi-modal coverage. To explicitly demonstrate how the repetition problem is eliminated by coverage mechanism, we calculate the average count of repeated words for each summary with coverage for different modalities. Figure 4 shows the results for (MM Decoderwconv) model with different image filters, which indicates that textual and visual coverage model can reduce repeated words.

7 Conclusions

This paper addresses a multi-modal sentence summarization task, namely, how to transform a sentence-image pair into a short-length summary. Our proposed model can simultaneously focus on image patches and text units to generate summaries with a modality-based attention mechanism. We design image filters to selectively use visual information to enrich the semantics for the source sentence. We introduce a visual coverage mechanism which prove that multi-modal coverage is effective for our task. We provide a publicly available multi-modal sentence summarization corpus and the experiments on this dataset show our proposed model significantly outperforms the baseline models. Our dataset is released to the public 2 .

Acknowledgments

The research work descried in this paper has been supported by the National Key Research and Development Program of China under Grant No. 2016QY02D0303 and the Natural Science Foundation of China under Grant No. 61333018.