{"type":"ScholarlyArticle","title":"GL-RG: Global-Local Representation Granularity for Video Captioning","authors":[{"name":"Liqi Yan","slug":"liqi-yan","given":"Liqi","family":"Yan","affiliation":"Fudan University; Westlake University; Rochester Institute of Technology"},{"name":"Qifan Wang","slug":"qifan-wang","given":"Qifan","family":"Wang","affiliation":""},{"name":"Yiming Cui","slug":"yiming-cui","given":"Yiming","family":"Cui","affiliation":"University of Florida"},{"name":"Fuli Feng","slug":"fuli-feng","given":"Fuli","family":"Feng","affiliation":"University of Science and Technology of China"},{"name":"Xiaojun Quan","slug":"xiaojun-quan","given":"Xiaojun","family":"Quan","affiliation":"Sun Yat-sen University"},{"name":"Xiangyu Zhang","slug":"xiangyu-zhang","given":"Xiangyu","family":"Zhang","affiliation":"Purdue University"},{"name":"Dongfang Liu","slug":"dongfang-liu","given":"Dongfang","family":"Liu","affiliation":"Rochester Institute of Technology"}],"abstract":"Video captioning is a challenging task as it needs to  accurately  transform  visual  understanding  into natural language description.  To date, state-of-the-art  methods  inadequately  model  global-local representation across video frames for caption generation, leaving plenty of room for improvement.  In this work, we approach the video captioning task from a new perspective and propose  a GL-RG framework for video captioning, namely a Global-Local Representation Granularity. Our GL-RG demonstrates three advantages over the prior efforts: 1) we explicitly exploit extensive visual representations from different video ranges to improve linguistic expression; 2) we devise a novel global-local encoder to produce rich semantic vocabulary to obtain a descriptive granularity of video contents across frames; 3) we develop an incremental training strategy which organizes model learning in an incremental fashion to incur an optimal captioning behavior. Experimental results on the challenging MSR-VTT and MSVD datasets show that our DL-RG outperforms recent state-of-the-art methods by a significant margin. Code is available at https://github.com/ylqi/GL-RG.","doi":"10.24963/ijcai.2022/384","url":"https://ijcai.org/proceedings/2022/384","pdf":"https://ijcai.org/proceedings/2022/0384.pdf","markdown":"https://ijcai.org/proceedings/2022/384.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/384","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Knowledge Representation and Reasoning","track":"Main Track","number":384,"pages":{"first":2769,"last":2775},"keywords":["Knowledge Representation and Reasoning: Applications","Computer Vision: Video analysis and understanding"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986578","length_sec":81,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984721","length_sec":351,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Carreira and Zisserman, 2017] João Carreira and Andrew Zisser- man. Quo vadis, action recognition? A new model and the kinet- ics dataset. In CVPR, 2017.","title":"Quo vadis, action recognition? A new model and the kinetics dataset","authors":"João Carreira, Andrew Zisserman","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen and Dolan, 2011] David L. Chen and W. Dolan. Collecting highly parallel data for paraphrase evaluation. In ACL, 2011.","title":"Collecting highly parallel data for paraphrase evaluation","authors":"David L Chen, W Dolan","year":"2011","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018] Yangyu Chen, Shuhui Wang, Weigang Zhang, and Qingming Huang. Less is more: Picking informative frames for video captioning. In ECCV, 2018.","title":"Less is more: Picking informative frames for video captioning","authors":"Yangyu Chen, Shuhui Wang, Weigang Zhang, Qingming Huang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deng et al., 2021] Chaorui Deng, Shizhe Chen, Da Chen, Yuan He, and Qi Wu. Sketch, ground, and refine: Top-down dense video captioning. In CVPR, 2021.","title":"Sketch, ground, and refine: Top-down dense video captioning","authors":"Chaorui Deng, Shizhe Chen, Da Chen, Yuan He, Qi Wu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ghosh et al., 2020] Pallabi Ghosh, Yi Yao, L. Davis, and Ajay Di- vakaran. Stacked spatio-temporal graph convolutional networks for action segmentation. In WACV, 2020.","title":"Stacked spatio-temporal graph convolutional networks for action segmentation","authors":"Pallabi Ghosh, Yi Yao, L Davis, Ajay Divakaran","year":"2020","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hou et al., 2019] Jingyi Hou, X. Wu, Wentian Zhao, Jiebo Luo, and Y. Jia. Joint syntax representation learning and visual cue translation for video captioning. In ICCV, 2019.","title":"Joint syntax representation learning and visual cue translation for video captioning","authors":"X Jingyi Hou, Wentian Wu, Jiebo Zhao, Y Luo, Jia","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2019] Yaosi Hu, Zhenzhong Chen, Zheng-Jun Zha, and Feng Wu. Hierarchical global-local temporal modeling for video captioning. In ACM MM, 2019.","title":"Hierarchical global-local temporal modeling for video captioning","authors":"Yaosi Hu, Zhenzhong Chen, Zheng-Jun Zha, Feng Wu","year":"2019","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Liu et al., 2020] Dongfang Liu, Yiming Cui, Yingjie Chen, Jiyong Zhang, and Bin Fan. Video object detection for autonomous driv- ing: Motion-aid feature calibration. Neurocomputing, 409:1-11, 2020.","title":"Video object detection for autonomous driving: Motion-aid feature calibration","authors":"Dongfang Liu, Yiming Cui, Yingjie Chen, Jiyong Zhang, Bin Fan","year":"2020","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Liu et al., 2021a] Dongfang Liu, Yiming Cui, Wenbo Tan, and Yingjie Chen. Sg-net: Spatial granularity network for one-stage video instance segmentation. In CVPR, 2021.","title":"Sg-net: Spatial granularity network for one-stage video instance segmentation","authors":"Liu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2021b] Fenglin Liu, Xuancheng Ren, Xian Wu, Bang Yang, Shen Ge, and Xu Sun. O2NA: an object-oriented non- autoregressive approach for controllable video captioning. In ACL/IJCNLP, 2021.","title":"O2NA: an object-oriented nonautoregressive approach for controllable video captioning","authors":"Liu","year":"2021","venue":"ACL/IJCNLP","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Pan et al., 2020] Boxiao Pan, Haoye Cai, Dean Huang, Kuan-Hui Lee, Adrien Gaidon, E. Adeli, and Juan Carlos Niebles. Spatio- temporal graph for video captioning with knowledge distillation. In CVPR, 2020.","title":"Spatiotemporal graph for video captioning with knowledge distillation","authors":"Pan","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Pasunuru and Bansal, 2017] Ramakanth Pasunuru and Mohit Bansal. Reinforced video captioning with entailment rewards. In EMNLP, 2017.","title":"Ramakanth Pasunuru and Mohit Bansal. Reinforced video captioning with entailment rewards","authors":"Bansal Pasunuru","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pei et al., 2019a] W. Pei, Jiyuan Zhang, Xiangrong Wang, Lei Ke, Xiaoyong Shen, and Yu-Wing Tai. Memory-attended recurrent network for video captioning. In CVPR, 2019.","title":"Memory-attended recurrent network for video captioning","authors":"Pei","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pei et al., 2019b] Wenjie Pei, Jiyuan Zhang, Xiangrong Wang, Lei Ke, Xiaoyong Shen, and Yu-Wing Tai. Memory-attended recur- rent network for video captioning. In CVPR, 2019.","title":"Memory-attended recurrent network for video captioning","authors":"Pei","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rahman et al., 2019] Tanzila Rahman, Bicheng Xu, and Leonid Sigal. Watch, listen and tell: Multi-modal weakly supervised dense event captioning. In ICCV, 2019.","title":"Watch, listen and tell: Multi-modal weakly supervised dense event captioning","authors":"Rahman","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rennie et al., 2017] Steven Rennie, E. Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In CVPR, 2017.","title":"Self-critical sequence training for image captioning","authors":"E Steven Rennie, Youssef Marcheret, Jarret Mroueh, Vaibhava Ross, Goel","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ryu et al., 2021] Hobin Ryu, Sunghun Kang, Haeyong Kang, and C. Yoo. Semantic grouping network for video captioning. In AAAI, 2021.","title":"Semantic grouping network for video captioning","authors":"Sunghun Hobin Ryu, Haeyong Kang, C Kang, Yoo","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sharma et al., 2017] Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language genera- tion. CoRR, abs/1706.09799, 2017.","title":"Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation","authors":"Shikhar Sharma, Layla El Asri, Hannes Schulz, Jeremie Zumer","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shi et al., 2019] Xiangxi Shi, Jianfei Cai, Shafiq R. Joty, and Jiux- iang Gu. Watch it twice: Video captioning with a refocused video encoder. In ACM MM, 2019.","title":"Watch it twice: Video captioning with a refocused video encoder","authors":"Shi","year":"2019","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tran et al., 2017] Du Tran, Jamie Ray, Zheng Shou, S. Chang, and Manohar Paluri. Convnet architecture search for spatiotemporal feature learning. ArXiv, abs/1708.05038, 2017.","title":"Convnet architecture search for spatiotemporal feature learning","authors":"Du Tran, Jamie Ray, Zheng Shou, S Chang, Manohar Paluri","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tsai et al., 2019] Yao-Hung Hubert Tsai, Santosh Kumar Divvala, Louis-Philippe Morency, Ruslan Salakhutdinov, and Ali Farhadi. Video relationship reasoning using gated spatio-temporal energy graph. In CVPR, 2019.","title":"Video relationship reasoning using gated spatio-temporal energy graph","authors":"Yao-Hung Hubert Tsai, Santosh Kumar Divvala, Louis-Philippe Morency, Ruslan Salakhutdinov, Ali Farhadi","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vedantam et al., 2015] Ramakrishna Vedantam, C. Zitnick, and Devi Parikh. Cider: Consensus-based image description evalu- ation. In CVPR, 2015.","title":"Cider: Consensus-based image description evaluation","authors":"C Ramakrishna Vedantam, Devi Zitnick, Parikh","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Venugopalan et al., 2015] Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence-video to text. In ICCV, 2015.","title":"Sequence to sequence-video to text","authors":"Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, Kate Saenko","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2018a] Bairui Wang, Lin Ma, Wei Zhang, and Wei Liu. Reconstruction network for video captioning. In CVPR, 2018.","title":"Reconstruction network for video captioning","authors":"Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2018b] Xin Wang, Wenhu Chen, Jiawei Wu, Yuan- Fang Wang, and William Yang Wang. Video captioning via hier- archical reinforcement learning. In CVPR, 2018.","title":"Video captioning via hierarchical reinforcement learning","authors":"Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2019] Bairui Wang, L. Ma, W. Zhang, Wenhao Jiang, Junling Wang, and W. Liu. Controllable video captioning with pos sequence guidance based on gated fusion network. In ICCV, 2019.","title":"Controllable video captioning with pos sequence guidance based on gated fusion network","authors":"L Bairui Wang, W Ma, Wenhao Zhang, Junling Jiang, W Wang, Liu","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Williams and Zipser, 1989] Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neu- ral networks. Neural computation, 1(2):270-280, 1989.","title":"A learning algorithm for continually running fully recurrent neural networks","authors":"J Ronald, David Williams, Zipser","year":"1989","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xie et al., 2017] Saining Xie, Ross Girshick, Piotr Dollar, Z. Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, 2017.","title":"Aggregated residual transformations for deep neural networks","authors":"Saining Xie, Ross Girshick, Piotr Dollar, Z Tu, Kaiming He","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xu et al., 2016] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr- vtt: A large video description dataset for bridging video and lan- guage. In CVPR, 2016.","title":"Msrvtt: A large video description dataset for bridging video and language","authors":"Jun Xu, Tao Mei, Ting Yao, Yong Rui","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2017] Ziwei Yang, Yahong Han, and Zheng Wang. Catching the temporal regions-of-interest for video captioning. In ACM MM, 2017.","title":"Catching the temporal regions-of-interest for video captioning","authors":"Ziwei Yang, Yahong Han, Zheng Wang","year":"2017","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yao et al., 2015] Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, and Aaron Courville. Describing videos by exploiting temporal structure. In CVPR, 2015.","title":"Describing videos by exploiting temporal structure","authors":"Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, Aaron Courville","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang and Peng, 2019] Junchao Zhang and Y. Peng. Object-aware aggregation with bidirectional temporal graph for video caption- ing. In CVPR, 2019.","title":"Object-aware aggregation with bidirectional temporal graph for video captioning","authors":"Peng ; ] Junchao Zhang, Y Zhang, Peng","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2020a] Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. Bridging the gap between training and inference for neural machine translation. In IJCAI, 2020.","title":"Bridging the gap between training and inference for neural machine translation","authors":"Zhang","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2020b] Z. Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, and Z. Zha. Object relational graph with teacher-recommended learning for video captioning. In CVPR, 2020.","title":"Object relational graph with teacher-recommended learning for video captioning","authors":"Zhang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang et al., 2021] Ziqi Zhang, Zhongang Qi, Chunfeng Yuan, Ying Shan, Bing Li, Ying Deng, and Weiming Hu. Open-book video captioning with retrieve-copy-generate network. In CVPR, 2021.","title":"Open-book video captioning with retrieve-copy-generate network","authors":"Ziqi Zhang, Zhongang Qi, Chunfeng Yuan, Ying Shan, Bing Li, Ying Deng, Weiming Hu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zheng et al., 2020] Qi Zheng, Chaoyue Wang, and D. Tao. Syntax- aware action targeting for video captioning. In CVPR, 2020.","title":"Syntaxaware action targeting for video captioning","authors":"Qi Zheng, Chaoyue Wang, D Tao","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}