{"type":"ScholarlyArticle","title":"Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures (Extended Abstract)","authors":[{"name":"Raffaella Bernardi","slug":"raffaella-bernardi","given":"Raffaella","family":"Bernardi","affiliation":"University of Trento"},{"name":"Ruket Cakici","slug":"ruket-cakici","given":"Ruket","family":"Cakici","affiliation":"Middle East Technical University"},{"name":"Desmond Elliott","slug":"desmond-elliott","given":"Desmond","family":"Elliott","affiliation":"University of Amsterdam"},{"name":"Aykut Erdem","slug":"aykut-erdem","given":"Aykut","family":"Erdem","affiliation":"Hacettepe University"},{"name":"Erkut Erdem","slug":"erkut-erdem","given":"Erkut","family":"Erdem","affiliation":"Hacettepe University"},{"name":"Nazli Ikizler-Cinbis","slug":"nazli-ikizler-cinbis","given":"Nazli","family":"Ikizler-Cinbis","affiliation":"Hacettepe University"},{"name":"Frank Keller","slug":"frank-keller","given":"Frank","family":"Keller","affiliation":"University of Edinburgh"},{"name":"Adrian Muscat","slug":"adrian-muscat","given":"Adrian","family":"Muscat","affiliation":"University of Malta"},{"name":"Barbara Plank","slug":"barbara-plank","given":"Barbara","family":"Plank","affiliation":"University of Groningen"}],"abstract":"Automatic image description generation is a  challenging problem that has recently received a large amount of  interest from the computer vision and natural language processing  communities. In this survey, we classify the known approaches  based on how they conceptualise this problem and provide a review of existing models, highlighting their advantages  and disadvantages. Moreover, we give an overview of the benchmark  image-text datasets and the evaluation measures that have been developed  to assess the quality of machine-generated descriptions. Finally we explore future directions in the area  of automatic image description.","doi":"10.24963/ijcai.2017/704","url":"https://ijcai.org/proceedings/2017/704","pdf":"https://ijcai.org/proceedings/2017/0704.pdf","markdown":"https://ijcai.org/proceedings/2017/704.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/704","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Journal track","track":"Journal track","number":704,"pages":{"first":4970,"last":4974},"keywords":["Robotics and Vision: Vision and Perception","Natural Language Processing: Natural Language Generation"],"videos":[],"references":[{"position":1,"raw":"[Anderson et al., 2016] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Com- puter Vision -ECCV 2016: Lecture Notes in Computer Science, Springer, Cham, 2016.","title":"Spice: Semantic propositional image caption evaluation","authors":"Anderson","year":"2016","venue":"Computer Vision -ECCV 2016","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bernardi et al., 2016] Raffaella Bernardi, Ruket Cakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, and Barbara Plank. Automatic description generation from images: A survey of models, datasets, and evaluation measures. Jour- nal of Artificial Intelligence Research, 55:409-442, 2016.","title":"Automatic description generation from images: A survey of models, datasets, and evaluation measures","authors":"Raffaella Bernardi, Ruket Cakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, Barbara Plank","year":"2016","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Denkowski and Lavie, 2014] Michael Denkowski and Alon Lavie. Meteor Universal: Language Specific Translation Evaluation for Any Target Language. In Conference of the European Chapter of the Association for Computational Linguistics Workshop on Statistical Machine Translation, 2014.","title":"Meteor Universal: Language Specific Translation Evaluation for Any Target Language","authors":"Lavie Denkowski, Michael Denkowski, Alon Lavie","year":"2014","venue":"Conference of the European Chapter of the Association for Computational Linguistics Workshop on Statistical Machine Translation","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Devlin et al., 2015] Jacob Devlin, Hao Cheng, Hao Fang, Saurabh Gupta, Li Deng, Xiaodong He, Geoffrey Zweig, and Margaret Mitchell. Language Models for Image Cap- tioning: The Quirks and What Works. In Annual Meeting of the Association for Computational Linguistics, 2015.","title":"Language Models for Image Captioning: The Quirks and What Works","authors":"Jacob Devlin, Hao Cheng, Hao Fang, Saurabh Gupta, Li Deng, Xiaodong He, Geoffrey Zweig, Margaret Mitchell","year":"2015","venue":"Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Elliott and Keller, 2013] Desmond Elliott and Frank Keller. Image Description using Visual Dependency Representa- tions. In Conference on Empirical Methods in Natural Language Processing, 2013.","title":"Image Description using Visual Dependency Representations","authors":"Elliott, Keller, Desmond Elliott, Frank Keller","year":"2013","venue":"Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Elliott and Keller, 2014] Desmond Elliott and Frank Keller. Comparing Automatic Evaluation Measures for Image De- scription. In Annual Meeting of the Association for Com- putational Linguistics, 2014.","title":"Comparing Automatic Evaluation Measures for Image Description","authors":"Desmond Elliott, Frank Keller","year":"2014","venue":"Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Elliott et al., 2016] Desmond Elliott, Stella Frank, Khalil Sima'an, and Lucia Specia. Multi30K: Multilingual English-German Image Descriptions. In Sixth Workshop on Vision and Language, 2016.","title":"Multi30K: Multilingual English-German Image Descriptions","authors":"Desmond Elliott, Stella Frank, Khalil Sima'an, Lucia Specia","year":"2016","venue":"Sixth Workshop on Vision and Language","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fang et al., 2015] Hao Fang, Saurabh Gupta, Forrest Ian- dola, Rupesh Srivastava, Li Deng, Piotr Dollár, Jian- feng Gao, Xiaodong He, Margaret Mitchell, John Platt, C. Lawrence Zitnick, and Geoffrey Zweig. From captions to visual concepts and back. In IEEE Conference on Com- puter Vision and Pattern Recognition, 2015.","title":"From captions to visual concepts and back","authors":"Saurabh Hao Fang, Forrest Gupta, Rupesh Iandola, Li Srivastava, Piotr Deng, Jianfeng Dollár, Xiaodong Gao, Margaret He, John Mitchell, C Lawrence Platt, Geoffrey Zitnick, Zweig","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Feng and Lapata, 2008] Yansong Feng and Mirella Lapata. Automatic Image Annotation Using Auxiliary Text Infor- mation. In Annual Meeting of the Association for Compu- tational Linguistics, 2008.","title":"Automatic Image Annotation Using Auxiliary Text Information","authors":"Lapata Feng, Yansong Feng, Mirella Lapata","year":"2008","venue":"Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hodosh et al., 2013] Micah Hodosh, Peter Young, and Julia Hockenmaier. Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics. Journal of Artificial Intelligence Research, 47:853-899, 2013.","title":"Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics","authors":"Hodosh","year":"2013","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Karpathy and Fei-Fei, 2015] Andrej Karpathy and Li Fei- Fei. Deep visual-semantic alignments for generating im- age descriptions. In IEEE Conference on Computer Vision and Pattern Recognition, 2015.","title":"Deep visual-semantic alignments for generating image descriptions","authors":"Fei-Fei ; Karpathy, Andrej Karpathy, Li Fei-Fei","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Karpathy et al., 2014] Andrej Karpathy, Armand Joulin, and Li Fei-Fei. Deep Fragment Embeddings for Bidirec- tional Image Sentence Mapping. In Advances in Neural Information Processing Systems, 2014.","title":"Deep Fragment Embeddings for Bidirectional Image Sentence Mapping","authors":"Andrej Karpathy, Armand Joulin, Li Fei-Fei","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kiros et al., 2015] Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel. Unifying visual-semantic embeddings with multimodal neural language models. In Advances in Neural Information Processing Systems Deep Learning Workshop, 2015.","title":"Unifying visual-semantic embeddings with multimodal neural language models","authors":"Ryan Kiros, Ruslan Salakhutdinov, Richard S Zemel","year":"2015","venue":"Advances in Neural Information Processing Systems Deep Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kulkarni et al., 2011] Girish Kulkarni, Visruth Premraj, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C. Berg, and Tamara L. Berg. Baby talk: Understanding and gen- erating simple image descriptions. In IEEE Conference on Computer Vision and Pattern Recognition, 2011.","title":"Baby talk: Understanding and generating simple image descriptions","authors":"Girish Kulkarni, Visruth Premraj, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C Berg, Tamara L Berg","year":"2011","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kuznetsova et al., 2012] Polina Kuznetsova, Vicente Or- donez, Alexander C. Berg, Tamara L. Berg, and Yejin Choi. Collective Generation of Natural Image Descrip- tions. In Annual Meeting of the Association for Computa- tional Linguistics, 2012.","title":"Collective Generation of Natural Image Descriptions","authors":"Polina Kuznetsova, Vicente Ordonez, Alexander C Berg, Tamara L Berg, Yejin Choi","year":"2012","venue":"Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Com- mon objects in context. In European Conference on Com- puter Vision, 2014.","title":"Microsoft COCO: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mason and Charniak, 2014] Rebecca Mason and Eugene Charniak. Nonparametric Method for Data-driven Im- age Captioning. In Annual Meeting of the Association for Computational Linguistics, 2014.","title":"Nonparametric Method for Data-driven Image Captioning","authors":"Charniak ; Mason, Rebecca Mason, Eugene Charniak","year":"2014","venue":"Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mitchell et al., 2012] Margaret Mitchell, Xufeng Han, Jesse Dodge, Alyssa Mensch, Amit Goyal, A C Berg, Kota Ya- maguchi, T L Berg, Karl Stratos, Hal Daume, III, and III. Midge: generating image descriptions from computer vi- sion detections. In Conference of the European Chapter of the Association for Computational Linguistics, 2012.","title":"Midge: generating image descriptions from computer vision detections","authors":"Margaret Mitchell, Xufeng Han, Jesse Dodge, Alyssa Mensch, Amit Goyal, A C Berg, Kota Yamaguchi, T L Berg, Karl Stratos, Hal Daume, Iii Iii","year":"2012","venue":"Conference of the European Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ordonez et al., 2011] Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. Im2text: Describing images using 1 million captioned photographs. In Advances in Neural Information Processing Systems, 2011.","title":"Im2text: Describing images using 1 million captioned photographs","authors":"Vicente Ordonez, Girish Kulkarni, Tamara L Berg","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ortiz et al., 2015] Luis M. G. Ortiz, Clemens Wolff, and Mirella Lapata. Learning to Interpret and Describe Ab- stract Scenes. In Conference of the North American Chap- ter of the Association of Computational Linguistics, 2015.","title":"Learning to Interpret and Describe Abstract Scenes","authors":"M G Luis, Clemens Ortiz, Mirella Wolff, Lapata","year":"2015","venue":"Conference of the North American Chapter of the Association of Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics, 2002.","title":"BLEU: A method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Socher et al., 2014] Richard Socher, Andrej Karpathy, Quoc V. Le, Christopher D. Manning, and Andrew Ng. Grounded Compositional Semantics for Finding and De- scribing Images with Sentences. Transactions of the Asso- ciation for Computational Linguistics, 2:207-218, 2014.","title":"Grounded Compositional Semantics for Finding and Describing Images with Sentences","authors":"Richard Socher, Andrej Karpathy, Quoc V Le, Christopher D Manning, Andrew Ng","year":"2014","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Specia et al., 2016] Lucia Specia, Stella Frank, Khalil Sima'an, and Desmond Elliott. A Shared Task on Mul- timodal Machine Translation and Crosslingual Image De- scription. In Conference on Machine Translation, 2016.","title":"A Shared Task on Multimodal Machine Translation and Crosslingual Image Description","authors":"Specia","year":"2016","venue":"Conference on Machine Translation","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vedantam et al., 2015] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In IEEE Conference on Computer Vision and Pattern Recognition, 2015.","title":"Cider: Consensusbased image description evaluation","authors":"C Lawrence Ramakrishna Vedantam, Devi Zitnick, Parikh","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neu- ral image caption generator. In IEEE Conference on Com- puter Vision and Pattern Recognition, 2015.","title":"Show and tell: A neural image caption generator","authors":"Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Young et al., 2014] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.","title":"From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions","authors":"Peter Young, Alice Lai, Micah Hodosh, Julia Hockenmaier","year":"2014","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}