{"type":"ScholarlyArticle","title":"Show and Tell More: Topic-Oriented Multi-Sentence Image Captioning","authors":[{"name":"Yuzhao Mao","slug":"yuzhao-mao","given":"Yuzhao","family":"Mao","affiliation":"Center for Intelligence Science and Technology , School of Computer Science , Beijing University of Posts and Telecommunications"},{"name":"Chang Zhou","slug":"chang-zhou","given":"Chang","family":"Zhou","affiliation":"Center for Intelligence Science and Technology , School of Computer Science , Beijing University of Posts and Telecommunications"},{"name":"Xiaojie Wang","slug":"xiaojie-wang","given":"Xiaojie","family":"Wang","affiliation":"Center for Intelligence Science and Technology , School of Computer Science , Beijing University of Posts and Telecommunications"},{"name":"Ruifan Li","slug":"ruifan-li","given":"Ruifan","family":"Li","affiliation":"Center for Intelligence Science and Technology , School of Computer Science , Beijing University of Posts and Telecommunications"}],"abstract":"Image captioning aims to generate textual descriptions for images. Most previous work generates a single-sentence description for each image. However, a picture is worth a thousand words. Single-sentence can hardly give a complete view of an image even by humans. In this paper, we propose a novel Topic-Oriented Multi-Sentence (\\emph{TOMS}) captioning model, which can generate multiple topic-oriented sentences to describe an image. Different from object instances or attributes, topics mined by the latent Dirichlet allocation reflect hidden thematic structures in reference sentences of an image. In our model, each topic is integrated to a caption generator with a Fusion Gate Unit (FGU) to guide the generation of a sentence towards a certain topic perspective. With multiple sentences from different topics, our \\emph{TOMS} provides a complete description of an image. Experimental results on both sentence and paragraph datasets demonstrate the effectiveness of our \\emph{TOMS} in terms of topical consistency and descriptive completeness.","doi":"10.24963/ijcai.2018/592","url":"https://ijcai.org/proceedings/2018/592","pdf":"https://ijcai.org/proceedings/2018/0592.pdf","markdown":"https://ijcai.org/proceedings/2018/592.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/592","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":592,"pages":{"first":4258,"last":4264},"keywords":["Natural Language Processing: Natural Language Generation","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Computer Vision: Language and Vision"],"videos":[],"references":[{"position":1,"raw":"[Dai et al., 2017] Bo Dai, Dahua Lin, Raquel Urtasun, and Sanja Fidler. Towards diverse and natural image descrip- tions via a conditional gan. CVPR, 2017.","title":"Towards diverse and natural image descriptions via a conditional gan","authors":"Bo Dai, Dahua Lin, Raquel Urtasun, Sanja Fidler","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dong et al., 2017] Yinpeng Dong, Hang Su, Jun Zhu, and Bo Zhang. Improving interpretability of deep neural net- works with semantic information. CVPR, 2017.","title":"Improving interpretability of deep neural networks with semantic information","authors":"Dong","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Gan et al., 2017] Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, and Li Deng. Semantic compositional networks for visual captioning. In CVPR, volume 2, 2017.","title":"Semantic compositional networks for visual captioning","authors":"Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, Li Deng","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Griffiths and Steyvers, 2004] Thomas L Griffiths and Mark Steyvers. Finding scientific topics. Proceedings of the National academy of Sciences, 101(suppl 1):5228-5235, 2004.","title":"Finding scientific topics","authors":"L Thomas, Mark Griffiths, Steyvers","year":"2004","venue":"Proceedings of the National academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hodosh et al., 2013] Micah Hodosh, Peter Young, and Ju- lia Hockenmaier. Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Ar- tificial Intelligence Research, 47:853-899, 2013.","title":"Framing image description as a ranking task: Data, models and evaluation metrics","authors":"Hodosh","year":"2013","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jia et al., 2015] Xu Jia, Efstratios Gavves, Basura Fer- nando, and Tinne Tuytelaars. Guiding long-short term memory for image caption generation. ICCV, 2015.","title":"Guiding long-short term memory for image caption generation","authors":"Xu Jia, Efstratios Gavves, Basura Fernando, Tinne Tuytelaars","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Johnson et al., 2016] Justin Johnson, Andrej Karpathy, and Li Fei-Fei. Densecap: Fully convolutional localization networks for dense captioning. 2016.","title":"Densecap: Fully convolutional localization networks for dense captioning","authors":"Justin Johnson, Andrej Karpathy, Li Fei-Fei","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Karpathy and Fei-Fei, 2015] Andrej Karpathy and Li Fei- Fei. Deep visual-semantic alignments for generating im- age descriptions. In CVPR, pages 3128-3137, 2015.","title":"Deep visual-semantic alignments for generating image descriptions","authors":"Fei-Fei ; Karpathy, Andrej Karpathy, Li Fei-Fei","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kiros et al., 2014] Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Multimodal neural language models. In ICML, volume 14, pages 595-603, 2014.","title":"Multimodal neural language models","authors":"Ryan Kiros, Ruslan Salakhutdinov, Richard S Zemel","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krause et al., 2017] Jonathan Krause, Justin Johnson, Ran- jay Krishna, and Li Fei-Fei. A hierarchical approach for generating descriptive image paragraphs. CVPR, 2017.","title":"A hierarchical approach for generating descriptive image paragraphs","authors":"Jonathan Krause, Justin Johnson, Ranjay Krishna, Li Fei-Fei","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kulkarni et al., 2013] Girish Kulkarni, Visruth Premraj, Vi- cente Ordonez, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C Berg, and Tamara L Berg. Babytalk: Under- standing and generating simple image descriptions. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 35(12):2891-2903, 2013.","title":"Babytalk: Understanding and generating simple image descriptions","authors":"Girish Kulkarni, Visruth Premraj, Vicente Ordonez, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C Berg, Tamara L Berg","year":"2013","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kuznetsova et al., 2014] Polina Kuznetsova, Vicente Or- donez, Tamara L Berg, and Yejin Choi. Treetalk: Com- position and compression of trees for image descriptions. TACL, 2(10):351-362, 2014.","title":"Treetalk: Composition and compression of trees for image descriptions","authors":"Polina Kuznetsova, Vicente Ordonez, Tamara L Berg, Yejin Choi","year":"2014","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lavie and Agarwal, 2007] Alon Lavie and Abhaya Agar- wal. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Pro- ceedings of the Second Workshop on Statistical Machine Translation, pages 228-231. Association for Computa- tional Linguistics, 2007.","title":"Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments","authors":"Alon Lavie, Abhaya Agarwal","year":"2007","venue":"Proceedings of the Second Workshop on Statistical Machine Translation","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liang et al., 2017] Xiaodan Liang, Zhiting Hu, Hao Zhang, Chuang Gan, and Eric P Xing. Recurrent topic-transition gan for visual paragraph generation. ICCV, 2017.","title":"Recurrent topic-transition gan for visual paragraph generation","authors":"Xiaodan Liang, Zhiting Hu, Hao Zhang, Chuang Gan, Eric P Xing","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. pages 740-755, 2014.","title":"Microsoft coco: Common objects in context","authors":"Lin","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin, 2010] Lin. Rouge: A package for automatic evaluation of summaries. 2010.","title":"Rouge: A package for automatic evaluation of summaries","authors":"Lin","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2017] Feng Liu, Tao Xiang, Timothy M Hospedales, Wankou Yang, and Changyin Sun. Semantic regularisation for recurrent image annotation. CVPR, 2017.","title":"Semantic regularisation for recurrent image annotation","authors":"Feng Liu, Tao Xiang, Timothy M Hospedales, Wankou Yang, Changyin Sun","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mao et al., 2016] Junhua Mao, Huang Jonathan, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object de- scriptions. CVPR, 2016.","title":"Generation and comprehension of unambiguous object descriptions","authors":"Junhua Mao, Huang Jonathan, Alexander Toshev, Oana Camburu, Alan Yuille, Kevin Murphy","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311-318. Association for Computational Linguis- tics, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vedantam et al., 2014] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. Computer Science, pages 4566-4575, 2014.","title":"Cider: Consensusbased image description evaluation","authors":"C Lawrence Ramakrishna Vedantam, Devi Zitnick, Parikh","year":"2014","venue":"Computer Science","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neu- ral image caption generator. In CVPR, pages 3156-3164, 2015.","title":"Show and tell: A neural image caption generator","authors":"Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2017] Liwei Wang, Alexander Schwing, and Svetlana Lazebnik. Diverse and accurate image descrip- tion using a variational auto-encoder with an additive gaus- sian encoding space. In NIPS, pages 5758-5768, 2017.","title":"Diverse and accurate image description using a variational auto-encoder with an additive gaussian encoding space","authors":"Liwei Wang, Alexander Schwing, Svetlana Lazebnik","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning, pages 2048-2057, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, Yoshua Bengio","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[You et al., 2016] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. CVPR, 2016.","title":"Image captioning with semantic attention","authors":"You","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Young et al., 2014] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.","title":"From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions","authors":"Peter Young, Alice Lai, Micah Hodosh, Julia Hockenmaier","year":"2014","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yu et al., 2016] Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, and Wei Xu. Video paragraph captioning using hierarchical recurrent neural networks. In CVPR, pages 4584-4593, 2016.","title":"Video paragraph captioning using hierarchical recurrent neural networks","authors":"Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, Wei Xu","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}