{"type":"ScholarlyArticle","title":"Exploring and Distilling Cross-Modal Information for Image Captioning","authors":[{"name":"Fenglin Liu","slug":"fenglin-liu","given":"Fenglin","family":"Liu","affiliation":"Shenzhen Key Lab for Information Centric Networking & Blockchain Technology (ICNLAB) , School of Electronics and Computer Engineering (SECE) , Peking University"},{"name":"Xuancheng Ren","slug":"xuancheng-ren","given":"Xuancheng","family":"Ren","affiliation":"MOE Key Laboratory of Computational Linguistics , School of EECS , Peking University"},{"name":"Yuanxin Liu","slug":"yuanxin-liu","given":"Yuanxin","family":"Liu","affiliation":"School of ICE , Beijing University of Posts and Telecommunications"},{"name":"Kai Lei","slug":"kai-lei","given":"Kai","family":"Lei","affiliation":"Shenzhen Key Lab for Information Centric Networking & Blockchain Technology (ICNLAB) , School of Electronics and Computer Engineering (SECE) , Peking University"},{"name":"Xu Sun","slug":"xu-sun","given":"Xu","family":"Sun","affiliation":"MOE Key Laboratory of Computational Linguistics , School of EECS , Peking University; sign , building , standing, street , man , brick , walking , tower , train"}],"abstract":"Recently, attention-based encoder-decoder models have been used extensively in image captioning. Yet there is still great difficulty for the current methods to achieve deep image understanding. In this work, we argue that such understanding requires visual attention to correlated image regions and semantic attention to coherent attributes of interest. To perform effective attention, we explore image captioning from a cross-modal perspective and propose the Global-and-Local Information Exploring-and-Distilling approach that explores and distills the source information in vision and language. It globally provides the aspect vector, a spatial and relational representation of images based on caption contexts, through the extraction of salient region groupings and attribute collocations, and locally extracts the fine-grained regions and attributes in reference to the aspect vector for word selection. Our fully-attentive model achieves a CIDEr score of 129.3 in offline COCO evaluation with remarkable efficiency in terms of accuracy, speed, and parameter budget.","doi":"10.24963/ijcai.2019/708","url":"https://ijcai.org/proceedings/2019/708","pdf":"https://ijcai.org/proceedings/2019/0708.pdf","markdown":"https://ijcai.org/proceedings/2019/708.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/708","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":708,"pages":{"first":5095,"last":5101},"keywords":["Computer Vision: Language and Vision","Natural Language Processing: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Anderson et al., 2016] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE: Semantic propositional image caption evaluation. In ECCV, 2016.","title":"SPICE: Semantic propositional image caption evaluation","authors":"Anderson","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anderson et al., 2018] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for im- age captioning and VQA. In CVPR, 2018.","title":"Bottom-up and top-down attention for image captioning and VQA","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Banerjee and Lavie, 2005] Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evalua- tion with improved correlation with human judgments. In ACL, 2005.","title":"METEOR: An automatic metric for MT evaluation with improved correlation with human judgments","authors":"Lavie Banerjee, Satanjeev Banerjee, Alon Lavie","year":"2005","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brendel and Bethge, 2019] Wieland Brendel and Matthias Bethge. Approximating CNNs with bag-of-local-features models works surprisingly well on imageNet. In ICLR, 2019.","title":"Approximating CNNs with bag-of-local-features models works surprisingly well on imageNet","authors":"Wieland Brendel, Matthias Bethge","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2015] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015.","title":"Microsoft COCO captions: Data collection and evaluation server","authors":"Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, C Lawrence Zitnick","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fang et al., 2015] Hao Fang, Saurabh Gupta, Forrest N. Ian- dola, Rupesh Kumar Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C. Platt, C. Lawrence Zitnick, and Geoffrey Zweig. From captions to visual concepts and back. In CVPR, 2015.","title":"From captions to visual concepts and back","authors":"Saurabh Hao Fang, Forrest N Gupta, Rupesh Kumar Iandola, Li Srivastava, Piotr Deng, Jianfeng Dollár, Xiaodong Gao, Margaret He, John C Mitchell, C Lawrence Platt, Geoffrey Zitnick, Zweig","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Geirhos et al., 2019] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR, 2019.","title":"ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness","authors":"Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, Wieland Brendel","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jiang et al., 2018a] Wenhao Jiang, Lin Ma, Xinpeng Chen, Hanwang Zhang, and Wei Liu. Learning to guide decoding for image captioning. In AAAI, 2018.","title":"Learning to guide decoding for image captioning","authors":"Jiang","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jiang et al., 2018b] Wenhao Jiang, Lin Ma, Yu-Gang Jiang, Wei Liu, and Tong Zhang. Recurrent fusion network for image captioning. In ECCV, 2018.","title":"Recurrent fusion network for image captioning","authors":"Jiang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Karpathy and Li, 2015] Andrej Karpathy and Fei-Fei Li. Deep visual-semantic alignments for generating image de- scriptions. In CVPR, 2015.","title":"Deep visual-semantic alignments for generating image descriptions","authors":"Li ; Karpathy, Andrej Karpathy, Fei-Fei Li","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Ba, 2014] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin, 2004] Chin-Yew Lin. ROUGE: A package for auto- matic evaluation of summaries. In ACL, 2004.","title":"ROUGE: A package for automatic evaluation of summaries","authors":"Chin-Yew Lin","year":"2004","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2018a] Daqing Liu, Zheng-Jun Zha, Hanwang Zhang, Yongdong Zhang, and Feng Wu. Context-aware visual policy network for sequence-level image caption- ing. In ACM Multimedia Conference, 2018.","title":"Context-aware visual policy network for sequence-level image captioning","authors":"Liu","year":"2018","venue":"ACM Multimedia Conference","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2018b] Fenglin Liu, Xuancheng Ren, Yuanxin Liu, Houfeng Wang, and Xu Sun. simNet: Stepwise image-topic merging network for generating detailed and comprehensive image captions. In EMNLP, 2018.","title":"simNet: Stepwise image-topic merging network for generating detailed and comprehensive image captions","authors":"Liu","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lu et al., 2017] Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In CVPR, 2017.","title":"Knowing when to look: Adaptive attention via a visual sentinel for image captioning","authors":"Lu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a Method for auto- matic evaluation of machine translation. In ACL, 2002.","title":"BLEU: a Method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rennie et al., 2017] Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. Self- critical sequence training for image captioning. In CVPR, 2017.","title":"Selfcritical sequence training for image captioning","authors":"Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, Vaibhava Goel","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vedantam et al., 2015] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In CVPR, 2015.","title":"Cider: Consensusbased image description evaluation","authors":"C Lawrence Ramakrishna Vedantam, Devi Zitnick, Parikh","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neu- ral image caption generator. In CVPR, 2015.","title":"Show and tell: A neural image caption generator","authors":"Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2016] Qi Wu, Chunhua Shen, Lingqiao Liu, An- thony R. Dick, and Anton van den Hengel. What value do explicit high level concepts have in vision to language problems? In CVPR, 2016.","title":"What value do explicit high level concepts have in vision to language problems?","authors":"Qi Wu, Chunhua Shen, Lingqiao Liu, Anthony R Dick, Anton Van Den, Hengel","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, Yoshua Bengio","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yao et al., 2017] Ting Yao, Yingwei Pan, Yehao Li, Zhao- fan Qiu, and Tao Mei. Boosting image captioning with attributes. In ICCV, 2017.","title":"Boosting image captioning with attributes","authors":"Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, Tao Mei","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yao et al., 2018] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Exploring visual relationship for image captioning. In ECCV, 2018.","title":"Exploring visual relationship for image captioning","authors":"Ting Yao, Yingwei Pan, Yehao Li, Tao Mei","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[You et al., 2016] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In CVPR, 2016.","title":"Image captioning with semantic attention","authors":"You","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2006] Cha Zhang, John C. Platt, and Paul A. Viola. Multiple instance boosting for object detection. In NIPS, 2006.","title":"Multiple instance boosting for object detection","authors":"Cha Zhang, John C Platt, Paul A Viola","year":"2006","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhu et al., 2018] Xinxin Zhu, Lixiang Li, Jing Liu, Haipeng Peng, and Xinxin Niu. Captioning transformer with stacked attention modules. Applied Sciences, 2018.","title":"Captioning transformer with stacked attention modules","authors":"Xinxin Zhu, Lixiang Li, Jing Liu, Haipeng Peng, Xinxin Niu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}