{"type":"ScholarlyArticle","title":"Non-Autoregressive Image Captioning with Counterfactuals-Critical Multi-Agent Learning","authors":[{"name":"Longteng Guo","slug":"longteng-guo","given":"Longteng","family":"Guo","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation , Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences"},{"name":"Jing Liu","slug":"jing-liu","given":"Jing","family":"Liu","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation , Chinese Academy of Sciences"},{"name":"Xinxin Zhu","slug":"xinxin-zhu","given":"Xinxin","family":"Zhu","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation , Chinese Academy of Sciences"},{"name":"Xingjian He","slug":"xingjian-he","given":"Xingjian","family":"He","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation , Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences"},{"name":"Jie Jiang","slug":"jie-jiang","given":"Jie","family":"Jiang","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation , Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences"},{"name":"Hanqing Lu","slug":"hanqing-lu","given":"Hanqing","family":"Lu","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation , Chinese Academy of Sciences"}],"abstract":"Most image captioning models are autoregressive, i.e. they generate each word by conditioning on previously generated words, which leads to heavy latency during inference. Recently, non-autoregressive decoding has been proposed in machine translation to speed up the inference time by generating all words in parallel. Typically, these models use the word-level cross-entropy loss to optimize each word independently. However, such a learning process fails to consider the sentence-level consistency, thus resulting in inferior generation quality of these non-autoregressive models. In this paper, we propose a Non-Autoregressive Image Captioning (NAIC) model with a novel training paradigm: Counterfactuals-critical Multi-Agent Learning (CMAL). CMAL formulates NAIC as a multi-agent reinforcement learning system where positions in the target sequence are viewed as agents that learn to cooperatively maximize a sentence-level reward. Besides, we propose to utilize massive unlabeled images to boost captioning performance. Extensive experiments on MSCOCO image captioning benchmark show that our NAIC model achieves a performance comparable to state-of-the-art autoregressive models, while brings 13.9x decoding speedup.","doi":"10.24963/ijcai.2020/107","url":"https://ijcai.org/proceedings/2020/107","pdf":"https://ijcai.org/proceedings/2020/0107.pdf","markdown":"https://ijcai.org/proceedings/2020/107.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/107","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":107,"pages":{"first":767,"last":773},"keywords":["Computer Vision: Language and Vision","Natural Language Processing: Natural Language Generation"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24825/file_24825.m3u8","length_sec":283,"provider":"hls"}],"references":[{"position":1,"raw":"[Anderson et al., 2017] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for im- age captioning and vqa. arXiv preprint arXiv:1707.07998, 2017.","title":"Bottom-up and top-down attention for image captioning and vqa","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bus ¸oniu et al., 2010] Lucian Bus ¸oniu, Robert Babuška, and Bart De Schutter. Multi-agent reinforcement learning: An overview. In Innovations in multi-agent systems and applications-1, pages 183-221. Springer, 2010.","title":"Lucian Bus ¸oniu, Robert Babuška, and Bart De Schutter. Multi-agent reinforcement learning: An overview","authors":"Bus ¸oniu","year":"2010","venue":"Innovations in multi-agent systems and applications-1","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chang et al., 2004] Yu-Han Chang, Tracey Ho, and Leslie P Kaelbling. All learning is local: Multi-agent learning in global reward games. In Advances in neural information processing systems, pages 807-814, 2004.","title":"All learning is local: Multi-agent learning in global reward games","authors":"Yu-Han Chang, Tracey Ho, Leslie P Kaelbling","year":"2004","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2015] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arX- iv:1504.00325, 2015.","title":"Microsoft coco captions: Data collection and evaluation server","authors":"Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, C Lawrence, Zitnick","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2019] Long Chen, Hanwang Zhang, Jun Xiao, Xiangnan He, Shiliang Pu, and Shih-Fu Chang. Counter- factual critic multi-agent training for scene graph genera- tion. In Proceedings of the IEEE International Conference on Computer Vision, pages 4613-4623, 2019.","title":"Counterfactual critic multi-agent training for scene graph generation","authors":"Long Chen, Hanwang Zhang, Jun Xiao, Xiangnan He, Shiliang Pu, Shih-Fu Chang","year":"2019","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fei, 2019] Zheng-cong Fei. Fast image caption gener- ation with position alignment. arXiv preprint arX- iv:1912.06365, 2019.","title":"Fast image caption generation with position alignment","authors":"Zheng-Cong Fei","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Foerster et al., 2018] Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.","title":"Counterfactual multi-agent policy gradients","authors":"Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, Shimon Whiteson","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gao et al., 2019] Junlong Gao, Xi Meng, Shiqi Wang, Xi- a Li, Shanshe Wang, Siwei Ma, and Wen Gao. Masked non-autoregressive image captioning. arXiv preprint arX- iv:1906.00717, 2019.","title":"Masked non-autoregressive image captioning","authors":"Junlong Gao, Xi Meng, Shiqi Wang, Xia Li, Shanshe Wang, Siwei Ma, Wen Gao","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gu et al., 2017] Jiatao Gu, James Bradbury, Caiming X- iong, Victor OK Li, and Richard Socher. Non- autoregressive neural machine translation. arXiv preprint arXiv:1711.02281, 2017.","title":"Nonautoregressive neural machine translation","authors":"Gu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guo et al., 2019a] Junliang Guo, Xu Tan, Di He, Tao Qin, Linli Xu, and Tie-Yan Liu. Non-autoregressive neural ma- chine translation with enhanced decoder input. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3723-3730, 2019.","title":"Non-autoregressive neural machine translation with enhanced decoder input","authors":"Guo","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Guo et al., 2019b] Longteng Guo, Jing Liu, Jinhui Tang, Jiangwei Li, Wei Luo, and Lu Hanqing. Aligning linguis- tic words and visual semantic units for image captioning. In ACM MM, 2019.","title":"Aligning linguistic words and visual semantic units for image captioning","authors":"Guo","year":"2019","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Guo et al., 2019c] Longteng Guo, Jing Liu, Peng Yao, Jiangwei Li, and Hanqing Lu. Mscap: Multi-style im- age captioning with unpaired stylized text. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4204-4213, 2019.","title":"Mscap: Multi-style image captioning with unpaired stylized text","authors":"Guo","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Herdade et al., 2019] Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares. Image captioning: Transforming objects into words. arXiv preprint arX- iv:1906.05963, 2019.","title":"Image captioning: Transforming objects into words","authors":"Simao Herdade, Armin Kappeler, Kofi Boakye, Joao Soares","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Karpathy and Feifei, 2015] Andrej Karpathy and Li Feifei. Deep visual-semantic alignments for generating image de- scriptions. computer vision and pattern recognition, pages 3128-3137, 2015.","title":"Deep visual-semantic alignments for generating image descriptions. computer vision and pattern recognition","authors":"Feifei Karpathy, Andrej Karpathy, Li Feifei","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kim and Rush, 2016] Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. arXiv preprint arXiv:1606.07947, 2016.","title":"Sequence-level knowledge distillation","authors":"Yoon Kim, Alexander M Rush","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lee et al., 2018] Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neural sequence modeling by iterative refinement. arXiv preprint arXiv:1802.06901, 2018.","title":"Deterministic non-autoregressive neural sequence modeling by iterative refinement","authors":"Jason Lee, Elman Mansimov, Kyunghyun Cho","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2019] Guang Li, Linchao Zhu, Ping Liu, and Y- i Yang. Entangled transformer for image captioning. In Proceedings of the IEEE International Conference on Computer Vision, pages 8928-8937, 2019.","title":"Entangled transformer for image captioning","authors":"Guang Li, Linchao Zhu, Ping Liu, Yi Yang","year":"2019","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rennie et al., 2017] Steven J Rennie, Etienne Marcheret, Y- oussef Mroueh, Jarret Ross, and Vaibhava Goel. Self- critical sequence training for image captioning. computer vision and pattern recognition, 2017.","title":"Selfcritical sequence training for image captioning. computer vision and pattern recognition","authors":"Etienne Steven J Rennie, Youssef Marcheret, Jarret Mroueh, Vaibhava Ross, Goel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, vol- ume 1. MIT press Cambridge, 1998.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems, pages 6000-6010, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vinyals et al., 2017] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: Lesson- s learned from the 2015 mscoco image captioning chal- lenge. IEEE transactions on pattern analysis and machine intelligence, 39(4):652-663, 2017.","title":"Show and tell: Lessons learned from the 2015 mscoco image captioning challenge","authors":"Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan","year":"2017","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2019] Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. Non-autoregressive machine translation with auxiliary regularization. arXiv preprint arXiv:1902.10245, 2019.","title":"Non-autoregressive machine translation with auxiliary regularization","authors":"Yiren Wang, Fei Tian, Di He, Tao Qin, Chengxiang Zhai, Tie-Yan Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Weaver and Tao, 2001] Lex Weaver and Nigel Tao. The optimal reward baseline for gradient-based reinforcemen- t learning. In Proceedings of the Seventeenth conference on Uncertainty in artificial intelligence, pages 538-545.","title":"The optimal reward baseline for gradient-based reinforcement learning","authors":"Tao Weaver, Lex Weaver, Nigel Tao","year":"2001","venue":"Proceedings of the Seventeenth conference on Uncertainty in artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Williams, 1992] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine Learning, 8(3-4):229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Ronald J Williams","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}