{"type":"ScholarlyArticle","title":"Recurrent Relational Memory Network for Unsupervised Image Captioning","authors":[{"name":"Dan Guo","slug":"dan-guo","given":"Dan","family":"Guo","affiliation":"Key Laboratory of Knowledge Engineering with Big Data (HFUT) , Ministry of Education School of Computer Science and Information Engineering , Hefei University of Technology (HFUT)"},{"name":"Yang Wang","slug":"yang-wang","given":"Yang","family":"Wang","affiliation":"Key Laboratory of Knowledge Engineering with Big Data (HFUT) , Ministry of Education School of Computer Science and Information Engineering , Hefei University of Technology (HFUT)"},{"name":"Peipei Song","slug":"peipei-song","given":"Peipei","family":"Song","affiliation":"Key Laboratory of Knowledge Engineering with Big Data (HFUT) , Ministry of Education School of Computer Science and Information Engineering , Hefei University of Technology (HFUT)"},{"name":"Meng Wang","slug":"meng-wang","given":"Meng","family":"Wang","affiliation":"Key Laboratory of Knowledge Engineering with Big Data (HFUT) , Ministry of Education School of Computer Science and Information Engineering , Hefei University of Technology (HFUT)"}],"abstract":"Unsupervised image captioning with no annotations is an emerging challenge in computer vision, where the existing arts usually adopt GAN (Generative\r\nAdversarial Networks) models. In this paper, we propose a novel memory-based network rather than GAN, named Recurrent Relational Memory\r\nNetwork (R2M). Unlike complicated and sensitive adversarial learning that non-ideally performs for long sentence generation, R2M implements\r\na concepts-to-sentence memory translator through two-stage memory mechanisms: fusion and recurrent memories, correlating the relational reasoning\r\nbetween common visual concepts and the generated words for long periods. R2M encodes visual context through unsupervised training on images,\r\nwhile enabling the memory to learn from irrelevant textual corpus via supervised fashion. Our solution enjoys less learnable parameters and\r\nhigher computational efficiency than GAN-based methods, which heavily bear parameter sensitivity. We experimentally validate the superiority of R2M\r\nthan state-of-the-arts on all benchmark datasets.","doi":"10.24963/ijcai.2020/128","url":"https://ijcai.org/proceedings/2020/128","pdf":"https://ijcai.org/proceedings/2020/0128.pdf","markdown":"https://ijcai.org/proceedings/2020/128.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/128","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":128,"pages":{"first":920,"last":926},"keywords":["Computer Vision: Language and Vision","Machine Learning: Deep Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24503/file_24503.m3u8","length_sec":282,"provider":"hls"}],"references":[{"position":1,"raw":"[Anderson et al., 2017] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Guided open vocab- ulary image captioning with constrained beam search. In EMNLP, page 936-945, 2017.","title":"Guided open vocabulary image captioning with constrained beam search","authors":"Anderson","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In ICCV, pages 2425-2433, 2015.","title":"Vqa: Visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Donahue and Simonyan, 2019] Jeff Donahue and Karen Si- monyan. Large scale adversarial representation learning. In NeurIPS, pages 10541-10551, 2019.","title":"Large scale adversarial representation learning","authors":"Simonyan Donahue","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fan et al., 2019] Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang. Heteroge- neous memory enhanced multimodal attention model for video question answering. In CVPR, pages 1999-2007, 2019.","title":"Heterogeneous memory enhanced multimodal attention model for video question answering","authors":"Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, Heng Huang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Feng et al., 2019] Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. Unsupervised image captioning. In CVPR, pages 4125-4134, 2019.","title":"Unsupervised image captioning","authors":"Yang Feng, Lin Ma, Wei Liu, Jiebo Luo","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fu et al., 2019] Canmiao Fu, Wenjie Pei, Qiong Cao, Chaopeng Zhang, Yong Zhao, Xiaoyong Shen, and Yu- Wing Tai. Non-local recurrent neural memory for super- vised sequence modeling. In ICCV, pages 6311-6320, 2019.","title":"Non-local recurrent neural memory for supervised sequence modeling","authors":"Canmiao Fu, Wenjie Pei, Qiong Cao, Chaopeng Zhang, Yong Zhao, Xiaoyong Shen, Yu-Wing Tai","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gu et al., 2019] Jiuxiang Gu, Shafiq Joty, Jianfei Cai, Han- dong Zhao, Xu Yang, and Gang Wang. Unpaired image captioning via scene graph alignments. In ICCV, pages 10323-10332, 2019.","title":"Unpaired image captioning via scene graph alignments","authors":"Jiuxiang Gu, Shafiq Joty, Jianfei Cai, Handong Zhao, Xu Yang, Gang Wang","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Guo et al., 2019] Longteng Guo, Jing Liu, Peng Yao, Jiang- wei Li, and Hanqing Lu. Mscap: Multi-style image cap- tioning with unpaired stylized text. In CVPR, pages 4204- 4213, 2019.","title":"Mscap: Multi-style image captioning with unpaired stylized text","authors":"Longteng Guo, Jing Liu, Peng Yao, Jiangwei Li, Hanqing Lu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang and Wang, 2019] Yan Huang and Liang Wang. Acmm: Aligned cross-modal memory for few-shot image and sentence matching. In ICCV, pages 5774-5783, 2019.","title":"Acmm: Aligned cross-modal memory for few-shot image and sentence matching","authors":"Yan Huang, Liang Wang","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2017] Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, et al. Speed/accuracy trade-offs for modern convolutional object detectors. In CVPR, pages 7310-7311, 2017.","title":"Speed/accuracy trade-offs for modern convolutional object detectors","authors":"Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang et al., 2019] Lun Huang, Wenmin Wang, Jie Chen, and Xiao-Yong Wei. Attention on attention for image cap- tioning. In ICCV, pages 4634-4643, 2019.","title":"Attention on attention for image captioning","authors":"Lun Huang, Wenmin Wang, Jie Chen, Xiao-Yong Wei","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krasin et al., 2017] Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, et al. Openimages: A public dataset for large-scale multi- label and multi-class image classification. Dataset avail- able from https://github. com/openimages, 2:3, 2017.","title":"Openimages: A public dataset for large-scale multilabel and multi-class image classification","authors":"Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit","year":"2017","venue":null,"doi":null,"url":"https://github.com/openimages,2:3","internal_url":null},{"position":13,"raw":"[Kuznetsova et al., 2018] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont- Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship de- tection at scale. arXiv preprint arXiv:1811.00982, 2018.","title":"The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale","authors":"Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Laina et al., 2019] Iro Laina, Christian Rupprecht, and Nas- sir Navab. Towards unsupervised image captioning with shared multimodal embeddings. In ICCV, pages 7414- 7424, 2019.","title":"Towards unsupervised image captioning with shared multimodal embeddings","authors":"Iro Laina, Christian Rupprecht, Nassir Navab","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lample et al., 2018] Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. Unsuper- vised machine translation using monolingual corpora only. In ICLR, 2018.","title":"Unsupervised machine translation using monolingual corpora only","authors":"Guillaume Lample, Alexis Conneau, Ludovic Denoyer, Marc'aurelio Ranzato","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In ECCV, pages 740-755, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mathews et al., 2016] Alexander Patrick Mathews, Lexing Xie, and Xuming He. Senticap: Generating image descrip- tions with sentiments. In AAAI, pages 3574-3580, 2016.","title":"Senticap: Generating image descriptions with sentiments","authors":"Alexander Patrick Mathews, Lexing Xie, Xuming He","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nie et al., 2019] Weili Nie, Nina Narodytska, and Ankit Pa- tel. Relgan: Relational generative adversarial networks for text generation. In ICLR, 2019.","title":"Relgan: Relational generative adversarial networks for text generation","authors":"Weili Nie, Nina Narodytska, Ankit Patel","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pei et al., 2019] Wenjie Pei, Jiyuan Zhang, Xiangrong Wang, Lei Ke, Xiaoyong Shen, and Yu-Wing Tai. Memory-attended recurrent network for video captioning. In CVPR, pages 8347-8356, 2019.","title":"Memory-attended recurrent network for video captioning","authors":"Wenjie Pei, Jiyuan Zhang, Xiangrong Wang, Lei Ke, Xiaoyong Shen, Yu-Wing Tai","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Santoro et al., 2018] Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Tim- othy Lillicrap. Relational recurrent neural networks. In NeurIPS, pages 7299-7310, 2018.","title":"Relational recurrent neural networks","authors":"Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, Timothy Lillicrap","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sharma et al., 2018] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, pages 2556-2565, 2018.","title":"Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning","authors":"Piyush Sharma, Nan Ding, Sebastian Goodman, Radu Soricut","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Szegedy et al., 2017] Christian Szegedy, Sergey Ioffe, Vin- cent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In AAAI, pages 4278-4284, 2017.","title":"Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning","authors":"Szegedy","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang et al., 2018] Zhen Yang, Wei Chen, Feng Wang, and Bo Xu. Unsupervised neural machine translation with weight sharing. In ACL, pages 46-55, 2018.","title":"Unsupervised neural machine translation with weight sharing","authors":"Zhen Yang, Wei Chen, Feng Wang, Bo Xu","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yao et al., 2019] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Hierarchy parsing for image captioning. In ICCV, pages 2621-2629, 2019.","title":"Hierarchy parsing for image captioning","authors":"Ting Yao, Yingwei Pan, Yehao Li, Tao Mei","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Young et al., 2014] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67-78, 2014.","title":"From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions","authors":"Peter Young, Alice Lai, Micah Hodosh, Julia Hockenmaier","year":"2014","venue":"TACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}