{"type":"ScholarlyArticle","title":"Human Consensus-Oriented Image Captioning","authors":[{"name":"Ziwei Wang","slug":"ziwei-wang","given":"Ziwei","family":"Wang","affiliation":"School of Information Technology and Electrical Engineering The University of Queensland , Australia"},{"name":"Zi Huang","slug":"zi-huang","given":"Zi","family":"Huang","affiliation":"School of Information Technology and Electrical Engineering The University of Queensland , Australia"},{"name":"Yadan Luo","slug":"yadan-luo","given":"Yadan","family":"Luo","affiliation":"School of Information Technology and Electrical Engineering The University of Queensland , Australia"}],"abstract":"Image captioning aims to describe an image with a concise, accurate, and interesting sentence. To build such an automatic neural captioner, the traditional models align the generated words with a number of human-annotated sentences to mimic human-like captions. However, the crowd-sourced annotations inevitably come with data quality issues such as grammatical errors, wrong identification of visual objects and sub-optimal sentence focus. During the model training, existing methods treat all the annotations equally regardless of the data quality. In this work, we explicitly engage human consensus to measure the quality of ground truth captions in advance, and directly encourage the model to learn high quality captions with high priority. Therefore, the proposed consensus-oriented method can accelerate the training process and achieve superior performance with only supervised objective without time-consuming reinforcement learning. The novel consensus loss can be implemented into most of the existing state-of-the-art methods, boosting the BLEU-4 performance by maximum relative 12.47% comparing to the conventional cross-entropy loss. Extensive experiments are conducted on MS-COCO Image Captioning dataset demonstrating the proposed human consensus-oriented training method can significantly improve the training efficiency and model effectiveness.","doi":"10.24963/ijcai.2020/92","url":"https://ijcai.org/proceedings/2020/92","pdf":"https://ijcai.org/proceedings/2020/0092.pdf","markdown":"https://ijcai.org/proceedings/2020/92.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/92","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":92,"pages":{"first":659,"last":665},"keywords":["Natural Language Processing: Machine Translation","Machine Learning: Deep Learning","Computer Vision: Language and Vision"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25294/file_25294.m3u8","length_sec":291,"provider":"hls"}],"references":[{"position":1,"raw":"[Anderson et al., 2018] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for im- age captioning and visual question answering. In CVPR, pages 6077-6086, 2018.","title":"Bottom-up and top-down attention for image captioning and visual question answering","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Banerjee and Lavie, 2005] Satanjeev Banerjee and Alon Lavie. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In ACL Workshop, pages 65-72, 2005.","title":"METEOR: an automatic metric for MT evaluation with improved correlation with human judgments","authors":"Lavie Banerjee, Satanjeev Banerjee, Alon Lavie","year":"2005","venue":"ACL Workshop","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bolukbasi et al., 2016] Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Tauman Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In NIPS, 2016.","title":"Man is to computer programmer as woman is to homemaker? debiasing word embeddings","authors":"Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, Adam Tauman, Kalai","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gao et al., 2019] L. Gao, X. Li, J. Song, and H. T. Shen. Hierarchical lstms with adaptive attention for visual cap- tioning. TPAMI, 2019.","title":"Hierarchical lstms with adaptive attention for visual captioning","authors":"L Gao, X Li, J Song, H T Shen","year":"2019","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hendricks et al., 2018] Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. Women also snowboard: Overcoming bias in captioning models. In ECCV, pages 793-811, 2018.","title":"Women also snowboard: Overcoming bias in captioning models","authors":"Anne Lisa, Kaylee Hendricks, Kate Burns, Trevor Saenko, Anna Darrell, Rohrbach","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Herdade et al., 2019] Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares. Image captioning: Trans- forming objects into words. In NeurIPS, 2019.","title":"Image captioning: Transforming objects into words","authors":"Simao Herdade, Armin Kappeler, Kofi Boakye, Joao Soares","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Karpathy and Fei-Fei, 2017] Andrej Karpathy and Li Fei- Fei. Deep visual-semantic alignments for generating im- age descriptions. TPAMI, pages 664-676, 2017.","title":"Deep visual-semantic alignments for generating image descriptions","authors":"Fei-Fei ; Karpathy, Andrej Karpathy, Li Fei-Fei","year":"2017","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Krause et al., 2017] Jonathan Krause, Justin Johnson, Ran- jay Krishna, and Li Fei-Fei. A hierarchical approach for generating descriptive image paragraphs. In CVPR, pages 3337-3345, 2017.","title":"A hierarchical approach for generating descriptive image paragraphs","authors":"Jonathan Krause, Justin Johnson, Ranjay Krishna, Li Fei-Fei","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2020] Yang Li, Yadan Luo, and Zi Huang. Graph- based relation-aware representation learning for clothing matching. In ADC, pages 189-197, 2020.","title":"Graphbased relation-aware representation learning for clothing matching","authors":"Yang Li, Yadan Luo, Zi Huang","year":"2020","venue":"ADC","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge J. Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: com- mon objects in context. In ECCV, pages 740-755, 2014.","title":"Microsoft COCO: common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge J Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 2018] Tsung-Yi Lin, Priyal Goyal, Ross Gir- shick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection. TPAMI, 2018.","title":"Focal loss for dense object detection","authors":"Tsung-Yi Lin, Priyal Goyal, Ross Girshick, Kaiming He, Piotr Dollar","year":"2018","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin, 2004] Chin-Yew Lin. ROUGE: A package for auto- matic evaluation of summaries. In Text Summarization Branches Out, pages 74-81. ACL, 2004.","title":"ROUGE: A package for automatic evaluation of summaries","authors":"Chin-Yew Lin","year":"2004","venue":"Text Summarization Branches Out","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lu et al., 2017] Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In CVPR, 2017.","title":"Knowing when to look: Adaptive attention via a visual sentinel for image captioning","authors":"Lu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Luo et al., 2019] Yadan Luo, Zi Huang, Zheng Zhang, Zi- wei Wang, Jingjing Li, and Yang Yang. Curiosity-driven reinforcement learning for diverse visual paragraph gener- ation. ACM MM, pages 2341-2350, 2019.","title":"Curiosity-driven reinforcement learning for diverse visual paragraph generation","authors":"Yadan Luo, Zi Huang, Zheng Zhang, Ziwei Wang, Jingjing Li, Yang Yang","year":"2019","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In ACL, pages 311-318, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rennie et al., 2017] Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. Self- critical sequence training for image captioning. In CVPR, pages 1179-1195, 2017.","title":"Selfcritical sequence training for image captioning","authors":"Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, Vaibhava Goel","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Song et al., 2018] Jingkuan Song, Yuyu Guo, Lianli Gao, Xuelong Li, Alan Hanjalic, and Heng Tao Shen. From de- terministic to generative: Multimodal stochastic rnns for video captioning. TNNLS, pages 3047-3058, 2018.","title":"From deterministic to generative: Multimodal stochastic rnns for video captioning","authors":"Song","year":"2018","venue":"TNNLS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 6000-6010, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vedantam et al., 2015] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In CVPR, pages 4566-4575, 2015.","title":"Cider: Consensusbased image description evaluation","authors":"C Lawrence Ramakrishna Vedantam, Devi Zitnick, Parikh","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neu- ral image caption generator. In CVPR, pages 3156-3164, 2015.","title":"Show and tell: A neural image caption generator","authors":"Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2018] Ziwei Wang, Yadan Luo, Yang Li, Zi Huang, and Hongzhi Yin. Look deeper see richer: Depth-aware image paragraph captioning. In ACM MM, pages 672-680, 2018.","title":"Look deeper see richer: Depth-aware image paragraph captioning","authors":"Ziwei Wang, Yadan Luo, Yang Li, Zi Huang, Hongzhi Yin","year":"2018","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2020] Ziwei Wang, Zi Huang, and Yadan Luo. PAIC: Parallelised attentive image captioning. In ADC, pages 16-28, 2020.","title":"PAIC: Parallelised attentive image captioning","authors":"Ziwei Wang, Zi Huang, Yadan Luo","year":"2020","venue":"ADC","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, pages 229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdi- nov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual at- tention. In ICML, pages 2048-2057, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdinov, Richard S Zemel, Yoshua Bengio","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2018] Yang Yang, Jie Zhou, Jiangbo Ai, Yi Bin, Alan Hanjalic, H. T. Shen, and Yanli Ji. Video captioning by adversarial lstm. TIP, pages 5600-5611, 2018.","title":"Video captioning by adversarial lstm","authors":"Yang Yang, Jie Zhou, Jiangbo Ai, Yi Bin, Alan Hanjalic, H T Shen, Yanli Ji","year":"2018","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yao et al., 2017] Ting Yao, Yingwei Pan, Yehao Li, Zhao- fan Qiu, and Tao Mei. Boosting image captioning with attributes. In ICCV, pages 4904-4912, 2017.","title":"Boosting image captioning with attributes","authors":"Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, Tao Mei","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[You et al., 2016] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In CVPR, pages 4651-4659, 2016.","title":"Image captioning with semantic attention","authors":"You","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zha et al., 2019] Zheng-Jun Zha, Daqing Liu, Hanwang Zhang, Yongdong Zhang, and Feng Wu. Context-aware visual policy network for fine-grained image captioning. TPAMI, 2019.","title":"Context-aware visual policy network for fine-grained image captioning","authors":"Zheng-Jun Zha, Daqing Liu, Hanwang Zhang, Yongdong Zhang, Feng Wu","year":"2019","venue":"TPAMI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}