{"type":"ScholarlyArticle","title":"S2 Transformer for Image Captioning","authors":[{"name":"Pengpeng Zeng","slug":"pengpeng-zeng","given":"Pengpeng","family":"Zeng","affiliation":"School of Computer Science and Engineering and Shenzhen Institute for Advanced Study , University of Electronic Science and Technology of China , Chengdu , China."},{"name":"Haonan Zhang","slug":"haonan-zhang","given":"Haonan","family":"Zhang","affiliation":"School of Computer Science and Engineering and Shenzhen Institute for Advanced Study , University of Electronic Science and Technology of China , Chengdu , China."},{"name":"Jingkuan Song","slug":"jingkuan-song","given":"Jingkuan","family":"Song","affiliation":"School of Computer Science and Engineering and Shenzhen Institute for Advanced Study , University of Electronic Science and Technology of China , Chengdu , China."},{"name":"Lianli Gao","slug":"lianli-gao","given":"Lianli","family":"Gao","affiliation":"School of Computer Science and Engineering and Shenzhen Institute for Advanced Study , University of Electronic Science and Technology of China , Chengdu , China."}],"abstract":"Transformer-based architectures with grid features represent the state-of-the-art in visual and language reasoning tasks, such as visual question answering and image-text matching. However, directly applying them to image captioning may result in spatial and fine-grained semantic information loss. Their applicability to image captioning is still largely under-explored. Towards this goal, we propose a simple yet effective method, Spatial- and Scale-aware Transformer (S2 Transformer) for image captioning. Specifically, we firstly propose a Spatial-aware Pseudo-supervised (SP) module, which resorts to feature clustering to help preserve spatial information for grid features. Next, to maintain the model size and produce superior results, we build a simple weighted residual connection, named Scale-wise Reinforcement (SR) module, to simultaneously explore both low- and high-level encoded features with rich semantics. Extensive experiments on the MSCOCO benchmark demonstrate that our method achieves new state-of-art performance without bringing excessive parameters compared with the vanilla transformer. The source code is available at https://github.com/zchoi/S2-Transformer","doi":"10.24963/ijcai.2022/224","url":"https://ijcai.org/proceedings/2022/224","pdf":"https://ijcai.org/proceedings/2022/0224.pdf","markdown":"https://ijcai.org/proceedings/2022/224.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/224","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":224,"pages":{"first":1608,"last":1614},"keywords":["Computer Vision: Vision and language","Computer Vision: Visual reasoning and symbolic representation"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986451","length_sec":88,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984594","length_sec":343,"provider":"slideslive"}],"references":[{"position":1,"raw":"GT1: A cat that is sitting next to a laptop. Trans: An animal sitting on a desk. Ours: An orange cat sitting on a desk next to a laptop. GT2: Cat sitting right next to keyboard on laptop. GT3: The orange cat is sitting on the laptop. GT1: A bundle of fruit on a wooden table. Trans: A bunch of bananas and pineapple. Ours: A bunch of bananas on a table. GT2: A big bunch of green bananas is on a table. GT3: Green plantain sitting on a dining table. GT1: A man takes a picture of food in a restaurant. Trans: A man is eating and taking a picture of food. Ours: A man taking a picture of a pizza with a camera. GT2: A man taking a picture of his meal at diner table. GT3: A man taking a photo of food on a table. GT1: A close up of a stop sign in a small town Trans: A stop sign sitting on road. Ours: A red stop sign on the side of a street. GT2: A red stop sign sitting next to country road. GT3: A close up of a stop sign in a small town. GT1: Four men are posing for picture at an event. Trans: A group of men standing next to each other. Ours: Four older men posing for picture at event. GT2: A group of man standing next to each other. GT3: A group of four older men posing for a photo. GT1: A long line of pans that are hanging on the wall. Trans: A lot of pots and pans on the counter. Ours: A kitchen with pots and pans hanging on the wall. GT2: A kitchen with pots hanging over the stove. GT3: Hanging frying pans in commercial kitchen.","title":"food. Ours: A man taking a picture of a pizza with a camera. GT2: A man taking a picture of his meal at diner table. GT3: A man taking a photo of food on a table. GT1: A close up of a stop sign in a small town Trans: A stop sign sitting on road","authors":null,"year":null,"venue":"Trans: A bunch of bananas and pineapple. Ours: A bunch of bananas on a table. GT2: A big bunch of green bananas is on a table. GT3: Green plantain sitting on a dining table. GT1: A man takes a picture of food in a restaurant","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anderson et al., 2016] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In ECCV, 2016.","title":"Spice: Semantic propositional image caption evaluation","authors":"Peter Anderson, Basura Fernando, Mark Johnson, Stephen Gould","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Anderson et al., 2018] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for im- age captioning and visual question answering. In CVPR, 2018.","title":"Bottom-up and top-down attention for image captioning and visual question answering","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Arandjelovic et al., 2016] Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic. Netvlad: Cnn architecture for weakly supervised place recognition. In CVPR, 2016.","title":"Netvlad: Cnn architecture for weakly supervised place recognition","authors":"Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, Josef Sivic","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Banerjee and Lavie, 2005] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In ACL workshop, 2005.","title":"Meteor: An automatic metric for mt evaluation with improved correlation with human judgments","authors":"Lavie Banerjee, Satanjeev Banerjee, Alon Lavie","year":"2005","venue":"ACL workshop","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2021] Wenqing Chen, Jidong Tian, Caoyun Fan, Hao He, and Yaohui Jin. Dependent multi-task learn- ing with causal intervention for image captioning. In IJ- CAI, 2021.","title":"Dependent multi-task learning with causal intervention for image captioning","authors":"Wenqing Chen, Jidong Tian, Caoyun Fan, Yaohui Hao He, Jin","year":"2021","venue":"IJ-CAI","doi":null,"url":null,"internal_url":"/proceedings/2021/312"},{"position":7,"raw":"[Cornia et al., 2019] Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Show, control and tell: A framework for generating controllable and grounded captions. In CVPR, 2019.","title":"Show, control and tell: A framework for generating controllable and grounded captions","authors":"Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cornia et al., 2020] Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed-memory transformer for image captioning. In CVPR, 2020.","title":"Meshed-memory transformer for image captioning","authors":"Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, Rita Cucchiara","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fan et al., 2021] Zhihao Fan, Zhongyu Wei, Siyuan Wang, Ruize Wang, Zejun Li, Haijun Shan, and Xuanjing Huang. TCIC: theme concepts learning cross language and vision for image captioning. In IJCAI, 2021.","title":"TCIC: theme concepts learning cross language and vision for image captioning","authors":"Zhihao Fan, Zhongyu Wei, Siyuan Wang, Ruize Wang, Zejun Li, Haijun Shan, Xuanjing Huang","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/91"},{"position":10,"raw":"[Herdade et al., 2019] Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares. Image captioning: Trans- forming objects into words. In NeurIPS, 2019.","title":"Image captioning: Transforming objects into words","authors":"Simao Herdade, Armin Kappeler, Kofi Boakye, Joao Soares","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang et al., 2019] Lun Huang, Wenmin Wang, Jie Chen, and Xiao-Yong Wei. Attention on attention for image cap- tioning. In ICCV, 2019.","title":"Attention on attention for image captioning","authors":"Lun Huang, Wenmin Wang, Jie Chen, Xiao-Yong Wei","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jiang et al., 2018] Wenhao Jiang, Lin Ma, Yu-Gang Jiang, Wei Liu, and Tong Zhang. Recurrent fusion network for image captioning. In ECCV, 2018.","title":"Recurrent fusion network for image captioning","authors":"Wenhao Jiang, Lin Ma, Yu-Gang Jiang, Wei Liu, Tong Zhang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jiang et al., 2020] Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Learned-Miller, and Xinlei Chen. In de- fense of grid features for visual question answering. In CVPR, 2020.","title":"In defense of grid features for visual question answering","authors":"Jiang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Karpathy and Fei-Fei, 2015] Andrej Karpathy and Li Fei- Fei. Deep visual-semantic alignments for generating im- age descriptions. In CVPR, 2015.","title":"Deep visual-semantic alignments for generating image descriptions","authors":"Fei-Fei ; Karpathy, Andrej Karpathy, Li Fei-Fei","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krishna et al., 2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowd- sourced dense image annotations. IJCV, 2017.","title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","authors":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael S Bernstein, Li Fei-Fei","year":"2017","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2019] Guang Li, Linchao Zhu, Ping Liu, and Yi Yang. Entangled transformer for image captioning. In ICCV, 2019.","title":"Entangled transformer for image captioning","authors":"Guang Li, Linchao Zhu, Ping Liu, Yi Yang","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin, 2004] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In ACL, 2004.","title":"Rouge: A package for automatic evaluation of summaries","authors":"Chin-Yew Lin","year":"2004","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pan et al., 2020] Yingwei Pan, Ting Yao, Yehao Li, and Tao Mei. X-linear attention networks for image captioning. In CVPR, 2020.","title":"X-linear attention networks for image captioning","authors":"Pan","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In ACL, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross B. Gir- shick, and Jian Sun. Faster R-CNN: towards real-time ob- ject detection with region proposal networks. In NeurIPS, 2015.","title":"Faster R-CNN: towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rennie et al., 2017] Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self- critical sequence training for image captioning. In CVPR, 2017.","title":"Selfcritical sequence training for image captioning","authors":"Etienne Steven J Rennie, Youssef Marcheret, Jerret Mroueh, Vaibhava Ross, Goel","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NeurIPS, 2014.","title":"Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vedantam et al., 2015] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In CVPR, 2015.","title":"Cider: Consensusbased image description evaluation","authors":"Ramakrishna Vedantam, Lawrence Zitnick, Devi Parikh","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vinyals et al., 2016] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: Lessons learned from the 2015 mscoco image captioning chal- lenge. TPAMI, 2016.","title":"Show and tell: Lessons learned from the 2015 mscoco image captioning challenge","authors":"Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan","year":"2016","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2020] Ziwei Wang, Zi Huang, and Yadan Luo. Human consensus-oriented image captioning. In IJCAI, 2020.","title":"Human consensus-oriented image captioning","authors":"Ziwei Wang, Zi Huang, Yadan Luo","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2020/92"},{"position":26,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, Yoshua Bengio","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yang et al., 2019] Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. Auto-encoding scene graphs for image captioning. In CVPR, 2019.","title":"Auto-encoding scene graphs for image captioning","authors":"Xu Yang, Kaihua Tang, Hanwang Zhang, Jianfei Cai","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yao et al., 2018] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Exploring visual relationship for image captioning. In ECCV, 2018.","title":"Exploring visual relationship for image captioning","authors":"Ting Yao, Yingwei Pan, Yehao Li, Tao Mei","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2021] Xuying Zhang, Xiaoshuai Sun, Yun- peng Luo, Jiayi Ji, Yiyi Zhou, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Rstnet: Captioning with adap- tive attention on visual and non-visual words. In CVPR, 2021.","title":"Rstnet: Captioning with adaptive attention on visual and non-visual words","authors":"Zhang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}