{"type":"ScholarlyArticle","title":"Learning to Discretely Compose Reasoning Module Networks for Video Captioning","authors":[{"name":"Ganchao Tan","slug":"ganchao-tan","given":"Ganchao","family":"Tan","affiliation":"University of Science and Technology of China"},{"name":"Daqing Liu","slug":"daqing-liu","given":"Daqing","family":"Liu","affiliation":"University of Science and Technology of China"},{"name":"Meng Wang","slug":"meng-wang","given":"Meng","family":"Wang","affiliation":"Hefei University of Technology"},{"name":"Zheng-Jun Zha","slug":"zheng-jun-zha","given":"Zheng-Jun","family":"Zha","affiliation":"University of Science and Technology of China"}],"abstract":"Generating natural language descriptions for videos, i.e., video captioning, essentially requires step-by-step reasoning along the generation process. For example, to generate the sentence “a man is shooting a basketball”, we need to first locate and describe the subject “man”, next reason out the man is “shooting”, then describe the object “basketball” of shooting. However, existing visual reasoning methods designed for visual question answering are not appropriate to video captioning, for it requires more complex visual reasoning on videos over both space and time, and dynamic module composition along the generation process. In this paper, we propose a novel visual reasoning approach for video captioning, named Reasoning Module Networks (RMN), to equip the existing encoder-decoder framework with the above reasoning capacity. Specifically, our RMN employs 1) three sophisticated spatio-temporal reasoning modules, and 2) a dynamic and discrete module selector trained by a linguistic loss with a Gumbel approximation. Extensive experiments on MSVD and MSR-VTT datasets demonstrate the proposed RMN outperforms the state-of-the-art methods while providing an explicit and explainable generation process. Our code is available at https://github.com/tgc1997/RMN.","doi":"10.24963/ijcai.2020/104","url":"https://ijcai.org/proceedings/2020/104","pdf":"https://ijcai.org/proceedings/2020/0104.pdf","markdown":"https://ijcai.org/proceedings/2020/104.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/104","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":104,"pages":{"first":745,"last":752},"keywords":["Computer Vision: Language and Vision"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25081/file_25081.m3u8","length_sec":286,"provider":"hls"}],"references":[{"position":1,"raw":"[Anderson et al., 2018] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, 2018.","title":"Bottom-up and top-down attention for image captioning and visual question answering","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Andreas et al., 2016] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In CVPR, 2016.","title":"Neural module networks","authors":"Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Dan Klein","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Banerjee and Lavie, 2005] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for ma- chine translation and/or summarization, pages 65-72, 2005.","title":"Meteor: An automatic metric for mt evaluation with improved correlation with human judgments","authors":"Lavie Banerjee, Satanjeev Banerjee, Alon Lavie","year":"2005","venue":"Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Carreira and Zisserman, 2017] Joao Carreira and Andrew Zisser- man. Quo vadis, action recognition? a new model and the kinet- ics dataset. In CVPR, 2017.","title":"Quo vadis, action recognition? a new model and the kinetics dataset","authors":"Joao Carreira, Andrew Zisserman","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen and Dolan, 2011] David L Chen and William B Dolan. Col- lecting highly parallel data for paraphrase evaluation. In ACL, 2011.","title":"Collecting highly parallel data for paraphrase evaluation","authors":"Chen, Dolan ; ] David, L Chen, William B Dolan","year":"2011","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen and Jiang, 2019] Shaoxiang Chen and Yu-Gang Jiang. Mo- tion guided spatial attention for video captioning. In AAAI, 2019.","title":"Motion guided spatial attention for video captioning","authors":"Chen, Jiang, Shaoxiang Chen, Yu-Gang Jiang","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cirik et al., 2018] Volkan Cirik, Taylor Berg-Kirkpatrick, and Louis-Philippe Morency. Using syntax to ground referring ex- pressions in natural images. In AAAI, 2018.","title":"Using syntax to ground referring expressions in natural images","authors":"Volkan Cirik, Taylor Berg-Kirkpatrick, Louis-Philippe Morency","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Donahue et al., 2015] Jeffrey Donahue, Lisa Anne Hendricks, Ser- gio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolu- tional networks for visual recognition and description. In CVPR, 2015.","title":"Long-term recurrent convolutional networks for visual recognition and description","authors":"Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, Trevor Darrell","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guadarrama et al., 2013] Sergio Guadarrama, Niveda Krish- namoorthy, Girish Malkarnenkar, Subhashini Venugopalan, Ray- mond Mooney, Trevor Darrell, and Kate Saenko. Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition. In ICCV, 2013.","title":"Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition","authors":"Sergio Guadarrama, Niveda Krishnamoorthy, Girish Malkarnenkar, Subhashini Venugopalan, Raymond Mooney, Trevor Darrell, Kate Saenko","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hong et al., 2019] Richang Hong, Daqing Liu, Xiaoyu Mo, Xi- angnan He, and Hanwang Zhang. Learning to compose and rea- son with language tree structures for visual grounding. T-PAMI, 2019.","title":"Learning to compose and reason with language tree structures for visual grounding","authors":"Hong","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hou et al., 2019] Jingyi Hou, Xinxiao Wu, Wentian Zhao, Jiebo Luo, and Yunde Jia. Joint syntax representation learning and vi- sual cue translation for video captioning. In ICCV, 2019.","title":"Joint syntax representation learning and visual cue translation for video captioning","authors":"Jingyi Hou, Xinxiao Wu, Wentian Zhao, Jiebo Luo, Yunde Jia","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hu et al., 2017] Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In ICCV, 2017.","title":"Learning to reason: End-to-end module networks for visual question answering","authors":"Hu","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hu et al., 2018] Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. Explainable neural computation via stack neural module networks. In ECCV, 2018.","title":"Explainable neural computation via stack neural module networks","authors":"Hu","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jang et al., 2017] Eric Jang, Shixiang Gu, and Ben Poole. Cate- gorical reparameterization with gumbel-softmax. In ICLR, 2017.","title":"Categorical reparameterization with gumbel-softmax","authors":"Eric Jang, Shixiang Gu, Ben Poole","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kay et al., 2017] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Vi- ola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. arXiv:1705.06950, 2017.","title":"The kinetics human action video dataset","authors":"Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, Andrew Zisserman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kojima et al., 2002] Atsuhiro Kojima, Takeshi Tamura, and Ku- nio Fukunaga. Natural language description of human activities from video images based on concept hierarchy of actions. IJCV, 50(2):171-184, 2002.","title":"Natural language description of human activities from video images based on concept hierarchy of actions","authors":"Atsuhiro Kojima, Takeshi Tamura, Kunio Fukunaga","year":"2002","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2017] Xuelong Li, Bin Zhao, Xiaoqiang Lu, et al. Mam- rnn: Multi-level attention model based rnn for video captioning. In IJCAI, 2017.","title":"Mamrnn: Multi-level attention model based rnn for video captioning","authors":"Xuelong Li, Bin Zhao, Xiaoqiang Lu","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lin, 2004] Chin-Yew Lin. ROUGE: A package for automatic eval- uation of summaries. In Text Summarization Branches Out, July 2004.","title":"ROUGE: A package for automatic evaluation of summaries","authors":"Chin-Yew Lin","year":"2004","venue":"Text Summarization Branches Out","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2018] Daqing Liu, Zheng-Jun Zha, Hanwang Zhang, Yongdong Zhang, and Feng Wu. Context-aware visual policy net- work for sequence-level image captioning. In ACM MM, 2018.","title":"Context-aware visual policy network for sequence-level image captioning","authors":"Daqing Liu, Zheng-Jun Zha, Hanwang Zhang, Yongdong Zhang, Feng Wu","year":"2018","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2019] Daqing Liu, Hanwang Zhang, Feng Wu, and Zheng-Jun Zha. Learning to assemble neural module tree net- works for visual grounding. In ICCV, 2019.","title":"Learning to assemble neural module tree networks for visual grounding","authors":"Daqing Liu, Hanwang Zhang, Feng Wu, Zheng-Jun Zha","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pan et al., 2017] Yingwei Pan, Ting Yao, Houqiang Li, and Tao Mei. Video captioning with transferred semantic attributes. In CVPR, 2017.","title":"Video captioning with transferred semantic attributes","authors":"Pan","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evalu- ation of machine translation. In ACL, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Pei et al., 2019] Wenjie Pei, Jiyuan Zhang, Xiangrong Wang, Lei Ke, Xiaoyong Shen, and Yu-Wing Tai. Memory-attended recur- rent network for video captioning. In CVPR, 2019.","title":"Memory-attended recurrent network for video captioning","authors":"Wenjie Pei, Jiyuan Zhang, Xiangrong Wang, Lei Ke, Xiaoyong Shen, Yu-Wing Tai","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ren et al., 2016] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2016.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexan- der C. Berg, and Fei-Fei Li. Imagenet large scale visual recogni- tion challenge. IJCV, 115:211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S Bernstein, Alexander C Berg, Fei-Fei Li","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Szegedy et al., 2017] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception- resnet and the impact of residual connections on learning. In AAAI, 2017.","title":"Inception-v4, inceptionresnet and the impact of residual connections on learning","authors":"Szegedy","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tian and Oh, 2019] Junjiao Tian and Jean Oh. Image captioning with compositional neural module networks. In AAAI, 2019.","title":"Image captioning with compositional neural module networks","authors":"Oh Tian, Junjiao Tian, Jean Oh","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":"/proceedings/2019/496"},{"position":31,"raw":"[Vedantam et al., 2015] Ramakrishna Vedantam, C Lawrence Zit- nick, and Devi Parikh. Cider: Consensus-based image descrip- tion evaluation. In CVPR, 2015.","title":"Cider: Consensus-based image description evaluation","authors":"Ramakrishna Vedantam, Lawrence Zitnick, Devi Parikh","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Venugopalan et al., 2015] Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence-video to text. In ICCV, 2015.","title":"Sequence to sequence-video to text","authors":"Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, Kate Saenko","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wang et al., 2018] Bairui Wang, Lin Ma, Wei Zhang, and Wei Liu. Reconstruction network for video captioning. In CVPR, 2018.","title":"Reconstruction network for video captioning","authors":"Bairui Wang, Lin Ma, Wei Zhang, Wei Liu","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wang et al., 2019] Bairui Wang, Lin Ma, Wei Zhang, Wenhao Jiang, Jingwen Wang, and Wei Liu. Controllable video caption- ing with pos sequence guidance based on gated fusion network. In CVPR, 2019.","title":"Controllable video captioning with pos sequence guidance based on gated fusion network","authors":"Bairui Wang, Lin Ma, Wei Zhang, Wenhao Jiang, Jingwen Wang, Wei Liu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Xu et al., 2016] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr- vtt: A large video description dataset for bridging video and lan- guage. In CVPR, 2016.","title":"Msrvtt: A large video description dataset for bridging video and language","authors":"Jun Xu, Tao Mei, Ting Yao, Yong Rui","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Yang et al., 2019a] T. Yang, Z. Zha, and H. Zhang. Making history matter: History-advantage sequence training for visual dialog. In ICCV, 2019.","title":"Making history matter: History-advantage sequence training for visual dialog","authors":"Yang","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Yang et al., 2019b] Xu Yang, Hanwang Zhang, and Jianfei Cai. Learning to collocate neural modules for image captioning. In ICCV, 2019.","title":"Learning to collocate neural modules for image captioning","authors":"Yang","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Yao et al., 2015] Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, and Aaron Courville. Describing videos by exploiting temporal structure. In ICCV, 2015.","title":"Describing videos by exploiting temporal structure","authors":"Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, Aaron Courville","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zha et al., 2019] Z. Zha, D. Liu, H. Zhang, Y. Zhang, and F. Wu. Context-aware visual policy network for fine-grained image cap- tioning. T-PAMI, 2019.","title":"Context-aware visual policy network for fine-grained image captioning","authors":"Z Zha, D Liu, H Zhang, Y Zhang, F Wu","year":"2019","venue":"T-PAMI","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhang and Peng, 2019] Junchao Zhang and Yuxin Peng. Object- aware aggregation with bidirectional temporal graph for video captioning. In CVPR, 2019.","title":"Objectaware aggregation with bidirectional temporal graph for video captioning","authors":"Peng Zhang, Junchao Zhang, Yuxin Peng","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}