{"type":"ScholarlyArticle","title":"Multi-modal Sentence Summarization with Modality Attention and Image Filtering","authors":[{"name":"Haoran Li","slug":"haoran-li","given":"Haoran","family":"Li","affiliation":"National Laboratory of Pattern Recognition , CASIA , Beijing , China; University of Chinese Academy of Sciences , Beijing , China"},{"name":"Junnan Zhu","slug":"junnan-zhu","given":"Junnan","family":"Zhu","affiliation":"National Laboratory of Pattern Recognition , CASIA , Beijing , China; University of Chinese Academy of Sciences , Beijing , China"},{"name":"Tianshang Liu","slug":"tianshang-liu","given":"Tianshang","family":"Liu","affiliation":"National Laboratory of Pattern Recognition , CASIA , Beijing , China; University of Chinese Academy of Sciences , Beijing , China"},{"name":"Jiajun Zhang","slug":"jiajun-zhang","given":"Jiajun","family":"Zhang","affiliation":"National Laboratory of Pattern Recognition , CASIA , Beijing , China; University of Chinese Academy of Sciences , Beijing , China"},{"name":"Chengqing Zong","slug":"chengqing-zong","given":"Chengqing","family":"Zong","affiliation":"National Laboratory of Pattern Recognition , CASIA , Beijing , China; University of Chinese Academy of Sciences , Beijing , China; CAS Center for Excellence in Brain Science and Intelligence Technology , Shanghai , China"}],"abstract":"In this paper, we introduce a multi-modal sentence summarization task that produces a short summary from a pair of sentence and image. This task is more challenging than sentence summarization. It not only needs to effectively incorporate visual features into standard text summarization framework, but also requires to avoid noise of image. To this end, we propose a modality-based attention mechanism to pay different attention to image patches and text units, and we design image filters to selectively use visual information to enhance the semantics of the input sentence. We construct a multimodal sentence summarization dataset and extensive experiments on this dataset demonstrate that our models significantly outperform conventional models which only employ text as input. Further analyses suggest that sentence summarization task can benefit from visually grounded representations from a variety of aspects.","doi":"10.24963/ijcai.2018/577","url":"https://ijcai.org/proceedings/2018/577","pdf":"https://ijcai.org/proceedings/2018/0577.pdf","markdown":"https://ijcai.org/proceedings/2018/577.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/577","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":577,"pages":{"first":4152,"last":4158},"keywords":["Natural Language Processing: Natural Language Summarization","Natural Language Processing: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Bengio et al., 2003] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilis- tic language model. JMLR, pages 1137-1155, 2003.","title":"A neural probabilistic language model","authors":"Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin","year":"2003","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"Caglayan et al., ] Ozan Caglayan, Walid Aransa, Adrien Bardet, Mercedes García-Martínez, Fethi Bougares, Loïc Barrault, Marc Masana, Luis Herranz, and Joost van de Weijer. Lium-cvc submissions for wmt17 multimodal translation task. In Proceedings of the Second Conference on Machine Translation.","title":"Lium-cvc submissions for wmt17 multimodal translation task","authors":"Caglayan","year":null,"venue":"Proceedings of the Second Conference on Machine Translation","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Calixto and Liu, 2017] Iacer Calixto and Qun Liu. Incor- porating global visual features into attention-based neural machine translation. In EMNLP, pages 992-1003, 2017.","title":"Incorporating global visual features into attention-based neural machine translation","authors":"Liu Calixto, Iacer Calixto, Qun Liu","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Calixto et al., 2017] Iacer Calixto, Qun Liu, and Nick Campbell. Doubly-attentive decoder for multi-modal neu- ral machine translation. In ACL, pages 1913-1924, 2017.","title":"Doubly-attentive decoder for multi-modal neural machine translation","authors":"Calixto","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, pages 1724-1734, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chopra et al., 2016] Sumit Chopra, Michael Auli, and Alexander M Rush. Abstractive sentence summarization with attentive recurrent neural networks. In NAACL, pages 93-98, 2016.","title":"Abstractive sentence summarization with attentive recurrent neural networks","authors":"Sumit Chopra, Michael Auli, Alexander M Rush","year":"2016","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chung et al., 2014] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evalua- tion of gated recurrent neural networks on sequence mod- eling. arXiv:1412.3555, 2014.","title":"Empirical evaluation of gated recurrent neural networks on sequence modeling","authors":"Chung","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Clarke and Lapata, 2008] James Clarke and Mirella Lapata. Global inference for sentence compression an integer lin- ear programming approach. AI Access Foundation, 2008.","title":"Global inference for sentence compression an integer linear programming approach","authors":"Clarke, Lapata, James Clarke, Mirella Lapata","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gu et al., 2016] Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. Incorporating copying mechanism in sequence-to-sequence learning. In ACL, pages 1631-1640, 2016.","title":"Incorporating copying mechanism in sequence-to-sequence learning","authors":"Gu","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gulcehre et al., 2016] Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. Pointing the unknown words. In ACL, pages 140-149, 2016.","title":"Pointing the unknown words","authors":"Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, Yoshua Bengio","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Helcl and Libovický, 2017] Jindřich Helcl and Jindřich Li- bovický. Cuni system for the wmt17 multimodal trans- lation task. In Proceedings of the Second Conference on Machine Translation, pages 450-457, 2017.","title":"Jindřich Helcl and Jindřich Libovický. Cuni system for the wmt17 multimodal translation task","authors":"Libovický Helcl","year":"2017","venue":"Proceedings of the Second Conference on Machine Translation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hill et al., 2013] Felix Hill, Douwe Kiela, and Anna Korho- nen. Concreteness and corpora: A theoretical and practical analysis. In Workshop on Cognitive Modeling and Com- putational Linguistics, pages 75-83, 2013.","title":"Concreteness and corpora: A theoretical and practical analysis","authors":"Felix Hill, Douwe Kiela, Anna Korhonen","year":"2013","venue":"Workshop on Cognitive Modeling and Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, pages 1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kiela et al., 2014] Douwe Kiela, Felix Hill, Anna Korho- nen, and Stephen Clark. Improving multi-modal repre- sentations using image dispersion: Why less is sometimes more. In ACL, pages 835-841, 2014.","title":"Improving multi-modal representations using image dispersion: Why less is sometimes more","authors":"Douwe Kiela, Felix Hill, Anna Korhonen, Stephen Clark","year":"2014","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2017a] Haoran Li, Junnan Zhu, Cong Ma, Jiajun Zhang, and Chengqing Zong. Multi-modal summariza- tion for asynchronous collection of text, image, audio and video. In EMNLP, pages 1092-1102, 2017.","title":"Multi-modal summarization for asynchronous collection of text, image, audio and video","authors":"Li","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2017b] Piji Li, Wai Lam, Lidong Bing, and Zihao Wang. Deep recurrent generative decoder for abstractive text summarization. In EMNLP, pages 2091-2100, 2017.","title":"Deep recurrent generative decoder for abstractive text summarization","authors":"Li","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Libovický and Helcl, 2017] Jindřich Libovický and Jindřich Helcl. Attention strategies for multi-source sequence-to-sequence learning. In ACL, pages 196-202, 2017.","title":"Jindřich Libovický and Jindřich Helcl. Attention strategies for multi-source sequence-to-sequence learning","authors":"Helcl Libovický","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lin, 2004] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out, 2004.","title":"Rouge: A package for automatic evaluation of summaries","authors":"Chin-Yew Lin","year":"2004","venue":"Text Summarization Branches Out","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nallapati et al., 2016] Ramesh Nallapati, Bowen Zhou, Ci- cero dos Santos, Caglar Gulcehre, and Bing Xiang. Ab- stractive text summarization using sequence-to-sequence rnns and beyond. In CoNLL, pages 280-290, 2016.","title":"Abstractive text summarization using sequence-to-sequence rnns and beyond","authors":"Ramesh Nallapati, Bowen Zhou, Caglar Cicero Dos Santos, Bing Gulcehre, Xiang","year":"2016","venue":"CoNLL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Paivio, 1990] Allan Paivio. Mental representations: A dual coding approach. Oxford University Press, 1990.","title":"Mental representations: A dual coding approach","authors":"Allan Paivio","year":"1990","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rush et al., 2015] Alexander M. Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In EMNLP, pages 379-389, 2015.","title":"A neural attention model for abstractive sentence summarization","authors":"Alexander M Rush, Sumit Chopra, Jason Weston","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[See et al., 2017] Abigail See, Peter J. Liu, and Christo- pher D. Manning. Get to the point: Summarization with pointer-generator networks. In ACL, pages 1073-1083, 2017.","title":"Get to the point: Summarization with pointer-generator networks","authors":"Abigail See, Peter J Liu, Christopher D Manning","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 15(1):1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Srivastava et al., 2015] Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arX- iv:1505.00387, 2015.","title":"Highway networks","authors":"Rupesh Kumar Srivastava, Klaus Greff, Jürgen Schmidhuber","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Takase et al., 2016] Sho Takase, Jun Suzuki, Naoak- i Okazaki, Tsutomu Hirao, and Masaaki Nagata. Neural headline generation on abstract meaning representation. In EMNLP, pages 1054-1059, 2016.","title":"Neural headline generation on abstract meaning representation","authors":"Takase","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neu- ral image caption generator. In CVPR, 2015.","title":"Show and tell: A neural image caption generator","authors":"Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, pages 2048-2057, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, Yoshua Bengio","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zeng et al., 2016] Wenyuan Zeng, Wenjie Luo, Sanja Fi- dler, and Raquel Urtasun. Efficient summarization with read-again and copy mechanism. arXiv:1611.03382, 2016.","title":"Efficient summarization with read-again and copy mechanism","authors":"Wenyuan Zeng, Wenjie Luo, Sanja Fidler, Raquel Urtasun","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhou et al., 2017] Qingyu Zhou, Nan Yang, Furu Wei, and Ming Zhou. Selective encoding for abstractive sentence summarization. In ACL, pages 1095-1104, 2017.","title":"Selective encoding for abstractive sentence summarization","authors":"Qingyu Zhou, Nan Yang, Furu Wei, Ming Zhou","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}