{"type":"ScholarlyArticle","title":"Show, Observe and Tell: Attribute-driven Attention Model for Image Captioning","authors":[{"name":"Hui Chen","slug":"hui-chen","given":"Hui","family":"Chen","affiliation":"† Beijing National Research Center for Information Science and Technology(BNRist) £ School of Software , Tsinghua University , Beijing , China"},{"name":"Guiguang Ding","slug":"guiguang-ding","given":"Guiguang","family":"Ding","affiliation":"† Beijing National Research Center for Information Science and Technology(BNRist) £ School of Software , Tsinghua University , Beijing , China"},{"name":"Zijia Lin","slug":"zijia-lin","given":"Zijia","family":"Lin","affiliation":"‡ Microsoft Research , Beijing , China"},{"name":"Sicheng Zhao","slug":"sicheng-zhao","given":"Sicheng","family":"Zhao","affiliation":"† Beijing National Research Center for Information Science and Technology(BNRist) £ School of Software , Tsinghua University , Beijing , China"},{"name":"Jungong Han","slug":"jungong-han","given":"Jungong","family":"Han","affiliation":"§ School of Computing & Communications , Lancaster University , UK"}],"abstract":"Despite the fact that attribute-based approaches and attention-based approaches have been proven to be effective in image captioning, most attribute-based approaches simply predict attributes independently without taking the co-occurrence dependencies among attributes into account. Besides, most attention-based captioning models directly leverage the feature map extracted from CNN, in which many features may be redundant in relation to the image content. In this paper, we focus on training a good attribute-inference model via the recurrent neural network (RNN) for image captioning, where the co-occurrence dependencies among attributes can be maintained. The uniqueness of our inference model lies in the usage of a RNN with the visual attention mechanism to \\textit{observe} the image before generating captions. Additionally, it is noticed that compact and attribute-driven features will be more useful for the attention-based captioning model. To this end, we extract the context feature for each attribute, and guide the captioning model adaptively attend to these context features. We verify the effectiveness and superiority of the proposed approach over the other captioning approaches by conducting massive experiments and comparisons on MS COCO image captioning dataset.","doi":"10.24963/ijcai.2018/84","url":"https://ijcai.org/proceedings/2018/84","pdf":"https://ijcai.org/proceedings/2018/0084.pdf","markdown":"https://ijcai.org/proceedings/2018/84.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/84","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":84,"pages":{"first":606,"last":612},"keywords":["Computer Vision: Language and Vision","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Anderson et al., 2017] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and vqa. arXiv preprint arXiv:1707.07998, 2017.","title":"Bottom-up and top-down attention for image captioning and vqa","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2017a] Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, and Tat-Seng Chua. Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning. CVPR, 2017.","title":"Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning","authors":"Chen","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2017b] Minghai Chen, Guiguang Ding, Sicheng Zhao, Hui Chen, Qiang Liu, and Jungong Han. Reference based lstm for image captioning. AAAI, 2017.","title":"Reference based lstm for image captioning","authors":"Chen","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fang et al., 2015] Hao Fang, Saurabh Gupta, Forrest Iandola, Ru- pesh K Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xi- aodong He, Margaret Mitchell, John C Platt, et al. From captions to visual concepts and back. In CVPR, pages 1473-1482, 2015.","title":"From captions to visual concepts and back","authors":"Saurabh Hao Fang, Forrest Gupta, Iandola, K Rupesh, Li Srivastava, Piotr Deng, Jianfeng Dollár, Xiaodong Gao, Margaret He, John C Mitchell, Platt","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gan et al., 2017] Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, and Li Deng. Semantic compositional networks for visual captioning. In CVPR, 2017.","title":"Semantic compositional networks for visual captioning","authors":"Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, Li Deng","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 00:770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jia et al., 2015] Xu Jia, Efstratios Gavves, Basura Fernando, and Tinne Tuytelaars. Guiding the long-short term memory model for image caption generation. In IEEE International Conference on Computer Vision, pages 2407-2415, 2015.","title":"Guiding the long-short term memory model for image caption generation","authors":"Xu Jia, Efstratios Gavves, Basura Fernando, Tinne Tuytelaars","year":"2015","venue":"IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Karpathy and Li, 2015] Andrej Karpathy and Fei-Fei Li. Deep visual-semantic alignments for generating image descriptions. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3128-3137, 2015.","title":"Deep visual-semantic alignments for generating image descriptions","authors":"Li ; Karpathy, Andrej Karpathy, Fei-Fei Li","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lin and Hovy, 2003] Chin-Yew Lin and Eduard Hovy. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Hu- man Language Technology-Volume 1, pages 71-78. Association for Computational Linguistics, 2003.","title":"Automatic evaluation of summaries using n-gram co-occurrence statistics","authors":"Lin, Hovy ; ] Chin-Yew Lin, Eduard Hovy","year":"2003","venue":"Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin et al., 2014] Tsung Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll¡ §¡ér, and C. Lawrence Zitnick. Microsoft coco: Common objects in con- text. In European Conference on Computer Vision, pages 740- 755, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung Yi, Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll¡, C Lawrence Zitnick","year":"2014","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2017] F. Liu, T. Xiang, Timothy Hospedales, W. Yang, and C. Sun. Semantic regularisation for recurrent image annota- tion. In Computer Vision and Pattern Recognition (CVPR 2017), 2 2017.","title":"Semantic regularisation for recurrent image annotation","authors":"F Liu, T Xiang, Timothy Hospedales, W Yang, C Sun","year":"2017","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lu et al., 2017] Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. CVPR, 2017.","title":"Knowing when to look: Adaptive attention via a visual sentinel for image captioning","authors":"Lu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mao et al., 2015] Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, and Alan L. Yuille. Deep captioning with multimodal recurrent neu- ral networks (m-rnn). In International Conference on Learning Representations, 2015.","title":"Deep captioning with multimodal recurrent neural networks (m-rnn)","authors":"Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Alan L Yuille","year":"2015","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311- 318. Association for Computational Linguistics, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"Proceedings of the 40th annual meeting on association for computational linguistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rennie et al., 2017] Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. CVPR, 2017.","title":"Self-critical sequence training for image captioning","authors":"Etienne Steven J Rennie, Youssef Marcheret, Jarret Mroueh, Vaibhava Ross, Goel","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Vedantam et al., 2015] Ramakrishna Vedantam, C Lawrence Zit- nick, and Devi Parikh. Cider: Consensus-based image descrip- tion evaluation. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 4566-4575, 2015.","title":"Cider: Consensus-based image description evaluation","authors":"Ramakrishna Vedantam, Lawrence Zitnick, Devi Parikh","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Alexander Toshev, Samy Ben- gio, and Dumitru Erhan. Show and tell: A neural image caption generator. In CVPR, pages 3156-3164, 2015.","title":"Show and tell: A neural image caption generator","authors":"Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2016] Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang, Chang Huang, and Wei Xu. Cnn-rnn: A unified frame- work for multi-label image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2285-2294, 2016.","title":"Cnn-rnn: A unified framework for multi-label image classification","authors":"Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang, Chang Huang, Wei Xu","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wu and Cohen, 2016] Zhilin Yang Ye Yuan Yuexin Wu and Ruslan Salakhutdinov William W Cohen. Encode, review, and decode: Reviewer module for caption generation. NIPS, 2016.","title":"Encode, review, and decode: Reviewer module for caption generation","authors":"Zhilin Yang, Ye Yuan, Yuexin Wu, Ruslan Salakhutdinov, William W Cohen","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wu et al., 2016] Qi Wu, Chunhua Shen, Lingqiao Liu, Anthony Dick, and Anton van den Hengel. What value do explicit high level concepts have in vision to language problems? In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 203-212, 2016.","title":"What value do explicit high level concepts have in vision to language problems?","authors":"Qi Wu, Chunhua Shen, Lingqiao Liu, Anthony Dick, Anton Van Den, Hengel","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, pages 2048-2057, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, Yoshua Bengio","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yang et al., 2016] Zhilin Yang, Ye Yuan, Yuexin Wu, Ruslan Salakhutdinov, and William W. Cohen. Encode, review, and de- code: Reviewer module for caption generation. NIPS, 2016.","title":"Encode, review, and decode: Reviewer module for caption generation","authors":"Zhilin Yang, Ye Yuan, Yuexin Wu, Ruslan Salakhutdinov, William W Cohen","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yao et al., 2017] Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qi- u, and Tao Mei. Boosting image captioning with attributes. In ICCV, 2017.","title":"Boosting image captioning with attributes","authors":"Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, Tao Mei","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[You et al., 2016] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 4651-4659, 2016.","title":"Image captioning with semantic attention","authors":"You","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}