{"type":"ScholarlyArticle","title":"ER-SAN: Enhanced-Adaptive Relation Self-Attention Network for Image Captioning","authors":[{"name":"Jingyu Li","slug":"jingyu-li","given":"Jingyu","family":"Li","affiliation":"University of Science and Technology of China , Hefei , China"},{"name":"Zhendong Mao","slug":"zhendong-mao","given":"Zhendong","family":"Mao","affiliation":"University of Science and Technology of China , Hefei , China"},{"name":"Shancheng Fang","slug":"shancheng-fang","given":"Shancheng","family":"Fang","affiliation":"University of Science and Technology of China , Hefei , China"},{"name":"Hao Li","slug":"hao-li","given":"Hao","family":"Li","affiliation":"Huazhong University of Science and Technology , Wuhan , China"}],"abstract":"Image captioning (IC), bringing vision to language, has drawn extensive attention. Precisely describing visual relations between image objects is a key challenge in IC. We argue that the visual relations, that is geometric positions (i.e., distance and size) and semantic interactions (i.e., actions and possessives), indicate the mutual correlations between objects. Existing Transformer-based methods typically resort to geometric positions to enhance the representation of visual relations, yet only using the shallow geometric is unable to precisely cover the complex and actional correlations. In this paper, we propose to enhance the correlations between objects from a comprehensive view that jointly considers explicit semantic and geometric relations, generating plausible captions with accurate relationship predictions. Specifically, we propose a novel Enhanced-Adaptive Relation Self-Attention Network (ER-SAN). We design the direction-sensitive semantic-enhanced attention, which considers content objects to semantic relations and semantic relations to content objects attention to learn explicit semantic-aware relations. Further, we devise an adaptive re-weight relation module that determines how much semantic and geometric attention should be activated to each relation feature. Extensive experiments on MS-COCO dataset demonstrate the effectiveness of our ER-SAN,  with improvements of CIDEr from 128.6% to 135.3%, achieving state-of-the-art performance. Codes will be released \\url{https://github.com/CrossmodalGroup/ER-SAN}.","doi":"10.24963/ijcai.2022/151","url":"https://ijcai.org/proceedings/2022/151","pdf":"https://ijcai.org/proceedings/2022/0151.pdf","markdown":"https://ijcai.org/proceedings/2022/151.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/151","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":151,"pages":{"first":1081,"last":1087},"keywords":["Natural Language Processing: Language Generation","Computer Vision: Vision and language"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986979","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985122","length_sec":683,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Anderson et al., 2016] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In European con- ference on computer vision, pages 382-398. Springer, 2016.","title":"Spice: Semantic propositional image caption evaluation","authors":"Anderson","year":"2016","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anderson et al., 2018] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for im- age captioning and visual question answering. In Proceed- ings of the IEEE conference on computer vision and pat- tern recognition, pages 6077-6086, 2018.","title":"Bottom-up and top-down attention for image captioning and visual question answering","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Banerjee and Lavie, 2005] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceed- ings of the acl workshop on intrinsic and extrinsic evalua- tion measures for machine translation and/or summariza- tion, pages 65-72, 2005.","title":"Meteor: An automatic metric for mt evaluation with improved correlation with human judgments","authors":"Lavie Banerjee, Satanjeev Banerjee, Alon Lavie","year":"2005","venue":"Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cornia et al., 2020] Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed-memory transformer for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10578-10587, 2020.","title":"Meshed-memory transformer for image captioning","authors":"Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, Rita Cucchiara","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fan et al., 2021] Zhihao Fan, Zhongyu Wei, Siyuan Wang, Ruize Wang, Zejun Li, Haijun Shan, and Xuanjing Huang. Tcic: Theme concepts learning cross language and vision for image captioning. arXiv preprint arXiv:2106.10936, 2021.","title":"Tcic: Theme concepts learning cross language and vision for image captioning","authors":"Zhihao Fan, Zhongyu Wei, Siyuan Wang, Ruize Wang, Zejun Li, Haijun Shan, Xuanjing Huang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2021/91"},{"position":6,"raw":"[Guo et al., 2020] Longteng Guo, Jing Liu, Xinxin Zhu, Peng Yao, Shichen Lu, and Hanqing Lu. Normalized and geometry-aware self-attention network for image cap- tioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10327- 10336, 2020.","title":"Normalized and geometry-aware self-attention network for image captioning","authors":"Longteng Guo, Jing Liu, Xinxin Zhu, Peng Yao, Shichen Lu, Hanqing Lu","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Herdade et al., 2019] Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares. Image caption- ing: Transforming objects into words. arXiv preprint arXiv:1906.05963, 2019.","title":"Image captioning: Transforming objects into words","authors":"Simao Herdade, Armin Kappeler, Kofi Boakye, Joao Soares","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2019] Lun Huang, Wenmin Wang, Jie Chen, and Xiao-Yong Wei. Attention on attention for image cap- tioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4634-4643, 2019.","title":"Attention on attention for image captioning","authors":"Lun Huang, Wenmin Wang, Jie Chen, Xiao-Yong Wei","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Karpathy and Fei-Fei, 2015] Andrej Karpathy and Li Fei- Fei. Deep visual-semantic alignments for generating im- age descriptions. In CVPR, pages 3128-3137, 2015.","title":"Deep visual-semantic alignments for generating image descriptions","authors":"Fei-Fei ; Karpathy, Andrej Karpathy, Li Fei-Fei","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kipf and Welling, 2016] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.","title":"Semi-supervised classification with graph convolutional networks","authors":"N Thomas, Max Kipf, Welling","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Pi- otr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Luo et al., 2021] Yunpeng Luo, Jiayi Ji, Xiaoshuai Sun, Li- ujuan Cao, Yongjian Wu, Feiyue Huang, Chia-Wen Lin, and Rongrong Ji. Dual-level collaborative transformer for image captioning. arXiv preprint arXiv:2101.06462, 2021.","title":"Dual-level collaborative transformer for image captioning","authors":"Yunpeng Luo, Jiayi Ji, Xiaoshuai Sun, Liujuan Cao, Yongjian Wu, Feiyue Huang, Chia-Wen Lin, Rongrong Ji","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pan et al., 2020] Yingwei Pan, Ting Yao, Yehao Li, and Tao Mei. X-linear attention networks for image caption- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10971- 10980, 2020.","title":"X-linear attention networks for image captioning","authors":"Pan","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Compu- tational Linguistics, pages 311-318, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"Proceedings of the 40th annual meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time ob- ject detection with region proposal networks. Advances in neural information processing systems, 28:91-99, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[ROUGE, 2004] Lin CY ROUGE. A package for automatic evaluation of summaries. In Proceedings of Workshop on Text Summarization of ACL, Spain, 2004.","title":"A package for automatic evaluation of summaries","authors":"C Y Lin, Rouge","year":"2004","venue":"Proceedings of Workshop on Text Summarization of ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shi et al., 2020] Zhan Shi, Xu Zhou, Xipeng Qiu, and Xi- aodan Zhu. Improving image captioning with better use of captions. arXiv preprint arXiv:2006.11807, 2020.","title":"Improving image captioning with better use of captions","authors":"Zhan Shi, Xu Zhou, Xipeng Qiu, Xiaodan Zhu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Song et al., 2021] Zeliang Song, Xiaofei Zhou, Linhua Dong, Jianlong Tan, and Li Guo. Direction relation transformer for image captioning. In Proceedings of the 29th ACM International Conference on Multimedia, pages 5056-5064, 2021.","title":"Direction relation transformer for image captioning","authors":"Song","year":"2021","venue":"Proceedings of the 29th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vedantam et al., 2015] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.","title":"Cider: Consensusbased image description evaluation","authors":"Ramakrishna Vedantam, Lawrence Zitnick, Devi Parikh","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2019] Xu Yang, Hanwang Zhang, and Jianfei Cai. Learning to collocate neural modules for image cap- tioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.","title":"Learning to collocate neural modules for image captioning","authors":"Xu Yang, Hanwang Zhang, Jianfei Cai","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yao et al., 2018] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Exploring visual relationship for image captioning. In Proceedings of the European conference on computer vision (ECCV), pages 684-699, 2018.","title":"Exploring visual relationship for image captioning","authors":"Ting Yao, Yingwei Pan, Yehao Li, Tao Mei","year":"2018","venue":"Proceedings of the European conference on computer vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2021] Xuying Zhang, Xiaoshuai Sun, Yun- peng Luo, Jiayi Ji, Yiyi Zhou, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Rstnet: Captioning with adaptive attention on visual and non-visual words. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 15465-15474, 2021.","title":"Rstnet: Captioning with adaptive attention on visual and non-visual words","authors":"Zhang","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}