{"type":"ScholarlyArticle","title":"Image Cationing with Visual-Semantic LSTM","authors":[{"name":"Nannan Li","slug":"nannan-li","given":"Nannan","family":"Li","affiliation":"School of Remote Sensing and Information Engineering , Wuhan University , Wuhan , P.R. China"},{"name":"Zhenzhong Chen","slug":"zhenzhong-chen","given":"Zhenzhong","family":"Chen","affiliation":"School of Remote Sensing and Information Engineering , Wuhan University , Wuhan , P.R. China"}],"abstract":"In this paper, a novel image captioning approach is proposed to describe the content of images. Inspired by the visual processing of our cognitive system, we propose a visual-semantic LSTM model to locate the attention objects with their low-level features in the visual cell, and then successively extract high-level semantic features in the semantic cell. In addition, a state perturbation term is introduced to the word sampling strategy in the REINFORCE based method to explore proper vocabularies in the training process. Experimental results on MS COCO and Flickr30K validate the effectiveness of our approach when compared to the state-of-the-art methods.","doi":"10.24963/ijcai.2018/110","url":"https://ijcai.org/proceedings/2018/110","pdf":"https://ijcai.org/proceedings/2018/0110.pdf","markdown":"https://ijcai.org/proceedings/2018/110.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/110","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":110,"pages":{"first":793,"last":799},"keywords":["Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Computer Vision: Language and Vision"],"videos":[],"references":[{"position":1,"raw":"[Aman et al., 2018] Sinha Aman, Namkoong Hongseok, and John Duchi. Certifiable distributional robustness with prin- cipled adversarial training. In ICLR, 2018.","title":"Certifiable distributional robustness with principled adversarial training","authors":"Namkoong Sinha Aman, John Hongseok, Duchi","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anderson et al., 2017] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for im- age captioning and visual question answering. In arXiv preprint arXiv:1707.07998, 2017.","title":"Bottom-up and top-down attention for image captioning and visual question answering","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio et al., 2015] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for se- quence prediction with recurrent neural networks. In NIPS, 2015.","title":"Scheduled sampling for sequence prediction with recurrent neural networks","authors":"Samy Bengio, Oriol Vinyals, Navdeep Jaitly, Noam Shazeer","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2017] Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, and Tat-Seng Chua. SCA-CNN: Spatial and channel-wise attention in convo- lutional networks for image captioning. In CVPR, 2017.","title":"SCA-CNN: Spatial and channel-wise attention in convolutional networks for image captioning","authors":"Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, Tat-Seng Chua","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Choi et al., 2008] Myung Jin Choi, Antonio Torralba, and Alan S. Willsky. Context models and out-of-context ob- jects. Pattern Recognition Letters, 2008.","title":"Context models and out-of-context objects","authors":"Jin Myung, Antonio Choi, Alan S Torralba, Willsky","year":"2008","venue":"Pattern Recognition Letters","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fang et al., 2015] Hao Fang, John C. Platt, C. Lawrence Zit- nick, Geoffrey Zweig, Saurabh Gupta, Forrest Iandola, Rupesh K. Srivastava, Li Deng, Piotr Dollar, and Jianfeng Gao. From captions to visual concepts and back. In CVPR, 2015.","title":"From captions to visual concepts and back","authors":"Hao Fang, John C Platt, C Lawrence Zitnick, Geoffrey Zweig, Saurabh Gupta, Forrest Iandola, K Rupesh, Li Srivastava, Piotr Deng, Jianfeng Dollar, Gao","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gan et al., 2017] Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, and Li Deng. Semantic compositional networks for visual captioning. In CVPR, 2017.","title":"Semantic compositional networks for visual captioning","authors":"Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, Li Deng","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gu et al., 2016] Jiuxiang Gu, Gang Wang, Jianfei Cai, and Tsuhan Chen. An empirical study of language CNN for image captioning. In ICCV, 2016.","title":"An empirical study of language CNN for image captioning","authors":"Gu","year":"2016","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In ICML, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Karpathy and Fei-Fei, 2015] Andrej Karpathy and Li Fei- Fei. Deep visual-semantic alignments for generating im- age descriptions. In CVPR, 2015.","title":"Deep visual-semantic alignments for generating image descriptions","authors":"Fei-Fei ; Karpathy, Andrej Karpathy, Li Fei-Fei","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin et al., 2014] Tsung Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Com- mon objects in context. In ECCV, 2014.","title":"Microsoft COCO: Common objects in context","authors":"Tsung Yi, Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2017] Chang Liu, Fuchun Sun, Changhu Wang, Feng Wang, and Alan Yuille. Mat: A multimodal attentive translator for image captioning. In IJCAI, 2017.","title":"Mat: A multimodal attentive translator for image captioning","authors":"Chang Liu, Fuchun Sun, Changhu Wang, Feng Wang, Alan Yuille","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/563"},{"position":14,"raw":"[Lu et al., 2017] Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In CVPR, 2017.","title":"Knowing when to look: Adaptive attention via a visual sentinel for image captioning","authors":"Lu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mao et al., 2015] Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan Yuille. Deep caption- ing with multimodal recurrent neural networks (m-RNN). In ICLR, 2015.","title":"Deep captioning with multimodal recurrent neural networks (m-RNN)","authors":"Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, Alan Yuille","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mun et al., 2017] Jonghwan Mun, Minsu Cho, and Bo- hyung Han. Text-guided attention model for image cap- tioning. In AAAI, 2017.","title":"Text-guided attention model for image captioning","authors":"Mun","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS, 2015.","title":"Faster R-CNN: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ren et al., 2017] Zhou Ren, Xiaoyu Wang, Ning Zhang, Xutao Lv, and Li-Jia Li. Deep reinforcement learning- based image captioning with embedding reward. In CVPR, 2017.","title":"Deep reinforcement learningbased image captioning with embedding reward","authors":"Xiaoyu Zhou Ren, Ning Wang, Xutao Zhang, Li-Jia Lv, Li","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rennie et al., 2017] Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self- critical sequence training for image captioning. In CVPR, 2017.","title":"Selfcritical sequence training for image captioning","authors":"Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, Vaibhava Goel","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press Cambridge, 1998.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2017a] Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. Gated self-matching net- works for reading comprehension and question answering. In ACL, 2017.","title":"Gated self-matching networks for reading comprehension and question answering","authors":"Wang","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2017b] Yufei Wang, Zhe Lin, Xiaohui Shen, Scott Cohen, and Garrison W. Cottrell. Skeleton key: Im- age captioning by skeleton-attribute decomposition. In CVPR, 2017.","title":"Skeleton key: Image captioning by skeleton-attribute decomposition","authors":"Wang","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2016] Qi Wu, Chunhua Shen, Lingqiao Liu, An- thony Dick, and Anton van den Hengel. What value do ex- plicit high level concepts have in vision to language prob- lems? In CVPR, 2016.","title":"What value do explicit high level concepts have in vision to language problems?","authors":"Qi Wu, Chunhua Shen, Lingqiao Liu, Anthony Dick, Anton Van Den, Hengel","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, Yoshua Bengio","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yao et al., 2017] Ting Yao, Yingwei Pan, Yehao Li, Zhao- fan Qiu, and Tao Mei. Boosting image captioning with attributes. In ICCV, 2017.","title":"Boosting image captioning with attributes","authors":"Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, Tao Mei","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[You et al., 2016] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In CVPR, 2016.","title":"Image captioning with semantic attention","authors":"You","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Young et al., 2014] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. In ACL, 2014.","title":"From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions","authors":"Peter Young, Alice Lai, Micah Hodosh, Julia Hockenmaier","year":"2014","venue":"ACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}