{"type":"ScholarlyArticle","title":"Multi-Level Policy and Reward Reinforcement Learning for Image Captioning","authors":[{"name":"Anan Liu","slug":"anan-liu","given":"Anan","family":"Liu","affiliation":"School of Electrical and Information Engineering , Tianjin University , Tianjin , China"},{"name":"Ning Xu","slug":"ning-xu","given":"Ning","family":"Xu","affiliation":"School of Electrical and Information Engineering , Tianjin University , Tianjin , China"},{"name":"Hanwang Zhang","slug":"hanwang-zhang","given":"Hanwang","family":"Zhang","affiliation":"School of Computer Science and Engineering , Nanyang Technological University , Singapore"},{"name":"Weizhi Nie","slug":"weizhi-nie","given":"Weizhi","family":"Nie","affiliation":"School of Electrical and Information Engineering , Tianjin University , Tianjin , China"},{"name":"Yuting Su","slug":"yuting-su","given":"Yuting","family":"Su","affiliation":"School of Electrical and Information Engineering , Tianjin University , Tianjin , China"},{"name":"Yongdong Zhang","slug":"yongdong-zhang","given":"Yongdong","family":"Zhang","affiliation":"University of Science and Technology of China , Hefei , China"}],"abstract":"Image captioning is one of the most challenging\r\nhallmark of AI, due to its complexity in visual and\r\nnatural language understanding. As it is essentially\r\na sequential prediction task, recent advances in image\r\ncaptioning use Reinforcement Learning (RL) to\r\nbetter explore the dynamics of word-by-word generation.\r\nHowever, existing RL-based image captioning\r\nmethods mainly rely on a single policy network\r\nand reward function that does not well fit the\r\nmulti-level (word and sentence) and multi-modal\r\n(vision and language) nature of the task. To this\r\nend, we propose a novel multi-level policy and reward\r\nRL framework for image captioning. It contains\r\ntwo modules: 1) Multi-Level Policy Network\r\nthat can adaptively fuse the word-level policy and\r\nthe sentence-level policy for the word generation;\r\nand 2) Multi-Level Reward Function that collaboratively\r\nleverages both vision-language reward and\r\nlanguage-language reward to guide the policy. Further,\r\nwe propose a guidance term to bridge the policy\r\nand the reward for RL optimization. Extensive\r\nexperiments and analysis on MSCOCO and Flickr30k\r\nshow that the proposed framework can achieve\r\ncompeting performances with respect to different\r\nevaluation metrics.","doi":"10.24963/ijcai.2018/114","url":"https://ijcai.org/proceedings/2018/114","pdf":"https://ijcai.org/proceedings/2018/0114.pdf","markdown":"https://ijcai.org/proceedings/2018/114.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/114","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":114,"pages":{"first":821,"last":827},"keywords":["Computer Vision: Language and Vision","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation"],"videos":[],"references":[{"position":1,"raw":"[Anderson et al., 2017] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for im- age captioning and visual question answering. In arXiv preprint arXiv:1707.07998, 2017.","title":"Bottom-up and top-down attention for image captioning and visual question answering","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cheng et al., 2018] Zhiyong Cheng, Ying Ding, Lei Zhu, and Mohan S. Kankanhalli. Aspect-aware latent factor model: Rating prediction with ratings and reviews. In WWW, pages 639-648, 2018.","title":"Kankanhalli. Aspect-aware latent factor model: Rating prediction with ratings and reviews","authors":"Zhiyong Cheng, Ying Ding, Lei Zhu, Mohan S","year":"2018","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dai et al., 2017] Bo Dai, Sanja Fidler, Raquel Urtasun, and Dahua Lin. Towards diverse and natural image descrip- tions via a conditional GAN. In ICCV, pages 2989-2998, 2017.","title":"Towards diverse and natural image descrip-tions via a conditional GAN","authors":"Bo Dai, Sanja Fidler, Raquel Urtasun, Dahua Lin","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Das et al., 2017] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M. F. Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In CVPR, pages 1080-1089, 2017.","title":"Visual dialog","authors":"Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, M F José, Devi Moura, Dhruv Parikh, Batra","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Farhadi et al., 2010] Ali Farhadi, Seyyed Moham- mad Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, and David A. Forsyth. Every picture tells a story: Generating sentences from images. In ECCV, pages 15-29, 2010.","title":"Every picture tells a story: Generating sentences from images","authors":"Ali Farhadi, Seyyed Mohammad Mohsen, Mohammad Amin Hejrati, Peter Sadeghi, Cyrus Young, Julia Rashtchian, David A Hockenmaier, Forsyth","year":"2010","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Karpathy and Fei-Fei, 2017] Andrej Karpathy and Li Fei- Fei. Deep visual-semantic alignments for generating im- age descriptions. IEEE Trans. Pattern Anal. Mach. Intell., 39(4):664-676, 2017.","title":"Deep visual-semantic alignments for generating image descriptions","authors":"Fei-Fei ; Karpathy, Andrej Karpathy, Li Fei-Fei","year":"2017","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Karpathy et al., 2014] Andrej Karpathy, Armand Joulin, and Fei-Fei Li. Deep fragment embeddings for bidirec- tional image sentence mapping. In NIPS, pages 1889- 1897, 2014.","title":"Deep fragment embeddings for bidirectional image sentence mapping","authors":"Andrej Karpathy, Armand Joulin, Fei-Fei Li","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kiros et al., 2015] Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel. Unifying visual-semantic embeddings with multimodal neural language models. 2015.","title":"Unifying visual-semantic embeddings with multimodal neural language models","authors":"Ryan Kiros, Ruslan Salakhutdinov, Richard S Zemel","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2017] Linghui Li, Sheng Tang, Lixi Deng, Yong- dong Zhang, and Qi Tian. Image caption with global-local attention. In AAAI, pages 4133-4139, 2017.","title":"Image caption with global-local attention","authors":"Linghui Li, Sheng Tang, Lixi Deng, Yongdong Zhang, Qi Tian","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2017a] An-An Liu, Ning Xu, Yongkang Wong, Junnan Li, Yuting Su, and Mohan S. Kankanhalli. Hier- archical & multimodal video captioning: Discovering and transferring multimodal knowledge for vision to language. CVIU, 163:113-125, 2017.","title":"Kankanhalli. Hierarchical & multimodal video captioning: Discovering and transferring multimodal knowledge for vision to language","authors":"Liu","year":"2017","venue":"CVIU","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2017b] Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. Improved image cap- tioning via policy gradient optimization of spider. In IC- CV, pages 873-881, 2017.","title":"Improved image captioning via policy gradient optimization of spider","authors":"Liu","year":"2017","venue":"IC-CV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lu et al., 2017] Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In CVPR, pages 3242-3250, 2017.","title":"Knowing when to look: Adaptive attention via a visual sentinel for image captioning","authors":"Lu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mao et al., 2015] Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, and Alan L. Yuille. Deep captioning with multi- modal recurrent neural networks (m-rnn). In ICLR, 2015.","title":"Deep captioning with multimodal recurrent neural networks (m-rnn)","authors":"Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Alan L Yuille","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Nie et al., 2013] Liqiang Nie, Meng Wang, Yue Gao, Zheng-Jun Zha, and Tat-Seng Chua. Beyond text QA: multimedia answer generation by harvesting web informa- tion. IEEE Trans. Multimedia, 15(2):426-441, 2013.","title":"Beyond text QA: multimedia answer generation by harvesting web information","authors":"Liqiang Nie, Meng Wang, Yue Gao, Zheng-Jun Zha, Tat-Seng Chua","year":"2013","venue":"IEEE Trans. Multimedia","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pan et al., 2016] Yingwei Pan, Tao Mei, Ting Yao, Houqiang Li, and Yong Rui. Jointly modeling embedding and translation to bridge video and language. In CVPR, pages 4594-4602, 2016.","title":"Jointly modeling embedding and translation to bridge video and language","authors":"Pan","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ranzato et al., 2016] Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In ICLR, 2016.","title":"Sequence level training with recurrent neural networks","authors":"Ranzato","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ren et al., 2017] Zhou Ren, Xiaoyu Wang, Ning Zhang, X- utao Lv, and Li-Jia Li. Deep reinforcement learning-based image captioning with embedding reward. In CVPR, pages 1151-1159, 2017.","title":"Deep reinforcement learning-based image captioning with embedding reward","authors":"Xiaoyu Zhou Ren, Ning Wang, Xutao Zhang, Li-Jia Lv, Li","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rennie et al., 2017] Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. Self- critical sequence training for image captioning. In CVPR, pages 1179-1195, 2017.","title":"Selfcritical sequence training for image captioning","authors":"Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, Vaibhava Goel","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J. Mad- dison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Ve- davyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mas- tering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, Demis Hassabis","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sutton et al., 1999] Richard S. Sutton, David A. M- cAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NIPS, pages 1057-1063, 1999.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"Richard S Sutton, David A M-Callester, P Satinder, Yishay Singh, Mansour","year":"1999","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vedantam et al., 2015] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In CVPR, pages 4566-4575, 2015.","title":"Cider: Consensusbased image description evaluation","authors":"C Lawrence Ramakrishna Vedantam, Devi Zitnick, Parikh","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neu- ral image caption generator. In CVPR, pages 3156-3164, 2015.","title":"Show and tell: A neural image caption generator","authors":"Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2016] Qi Wu, Chunhua Shen, Lingqiao Liu, An- thony R. Dick, and Anton van den Hengel. What value do explicit high level concepts have in vision to language problems? In CVPR, pages 203-212, 2016.","title":"What value do explicit high level concepts have in vision to language problems?","authors":"Qi Wu, Chunhua Shen, Lingqiao Liu, Anthony R Dick, Anton Van Den, Hengel","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdi- nov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual at- tention. In ICML, pages 2048-2057, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdinov, Richard S Zemel, Yoshua Bengio","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2016] Zhilin Yang, Ye Yuan, Yuexin Wu, Rus- lan Salakhutdinov, and William W. Cohen. Encode, re- view, and decode: Reviewer module for caption genera- tion. In NIPS, 2016.","title":"Encode, review, and decode: Reviewer module for caption generation","authors":"Zhilin Yang, Ye Yuan, Yuexin Wu, Ruslan Salakhutdinov, William W Cohen","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yao et al., 2017] Ting Yao, Yingwei Pan, Yehao Li, Zhao- fan Qiu, and Tao Mei. Boosting image captioning with attributes. In ICCV, 2017.","title":"Boosting image captioning with attributes","authors":"Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, Tao Mei","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[You et al., 2016] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In CVPR, pages 4651-4659, 2016.","title":"Image captioning with semantic attention","authors":"You","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yun et al., 2017] Sangdoo Yun, Jongwon Choi, Youngjoon Yoo, Kimin Yun, and Jin Young Choi. Action-decision net- works for visual tracking with deep reinforcement learn- ing. In CVPR, pages 1349-1358, 2017.","title":"Action-decision networks for visual tracking with deep reinforcement learning","authors":"Yun","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2017] Hanwang Zhang, Zawlin Kyaw, Shih- Fu Chang, and Tat-Seng Chua. Visual translation embed- ding network for visual relation detection. In CVPR, pages 3107-3115, 2017.","title":"Visual translation embedding network for visual relation detection","authors":"Hanwang Zhang, Zawlin Kyaw, Shih-Fu Chang, Tat-Seng Chua","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}