{"type":"ScholarlyArticle","title":"Dual Video Summarization: From Frames to Captions","authors":[{"name":"Zhenzhen Hu","slug":"zhenzhen-hu","given":"Zhenzhen","family":"Hu","affiliation":"Hefei University of Technology; Institute of Artificial Intelligence Hefei Comprehensive National Science Center"},{"name":"Zhenshan Wang","slug":"zhenshan-wang","given":"Zhenshan","family":"Wang","affiliation":"Hefei University of Technology"},{"name":"Zijie Song","slug":"zijie-song","given":"Zijie","family":"Song","affiliation":"Hefei University of Technology"},{"name":"Richang Hong","slug":"richang-hong","given":"Richang","family":"Hong","affiliation":"Hefei University of Technology"}],"abstract":"Video summarization and video captioning both condense the video content from the perspective of visual and text modes, i.e. the keyframe selection and language description generation. Existing video-and-language learning models commonly sample multiple frames for training instead of observing all. These sampled deputies greatly improve computational efficiency, but do they represent the original video content enough with no more redundancy? In this work, we propose a dual video summarization framework and verify it in the context of video captioning. Given the video frames, we firstly extract the visual representation based on the ViT model fine-tuned on the video-text domain. Then we summarize the keyframes according to the frame-lever score.  To compress the number of keyframes as much as possible while ensuring the quality of captioning, we learn a cross-modal video summarizer to select the most semantically consistent frames according to the pseudo score label. Top K frames ( K is no more than 3% of the entire video.) are chosen to form the video representation. Moreover, to evaluate the static appearance and temporal information of video, we design the ranking scheme of video representation from two aspects: feature-oriented and sequence-oriented. Finally, we generate the descriptions with a lightweight LSTM decoder. The experiment results on the MSR-VTT and MSVD dataset reveal that, for the generative task as video captioning, a small number of keyframes can convey the same semantic information to perform well on captioning, or even better than the original sampling.","doi":"10.24963/ijcai.2023/94","url":"https://ijcai.org/proceedings/2023/94","pdf":"https://ijcai.org/proceedings/2023/0094.pdf","markdown":"https://ijcai.org/proceedings/2023/94.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/94","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":94,"pages":{"first":846,"last":854},"keywords":["Computer Vision: CV: Video analysis and understanding","Computer Vision: CV: Vision and language"],"videos":[],"references":[{"position":1,"raw":"[Anne Hendricks et al., 2017] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natu- ral language. In Proceedings of the IEEE ICCV, pages 5803-5812, 2017.","title":"Localizing moments in video with natural language","authors":"Anne Lisa, Oliver Hendricks, Eli Wang, Josef Shechtman, Trevor Sivic, Bryan Darrell, Russell","year":"2017","venue":"Proceedings of the IEEE ICCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bain et al., 2021] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF ICCV, pages 1728-1738, 2021.","title":"Frozen in time: A joint video and image encoder for end-to-end retrieval","authors":"Bain","year":"2021","venue":"Proceedings of the IEEE/CVF ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baraldi et al., 2017] Lorenzo Baraldi, Costantino Grana, and Rita Cucchiara. Hierarchical boundary-aware neural encoder for video captioning. In Proceedings of the IEEE Conference on CVPR, pages 1657-1666, 2017.","title":"Hierarchical boundary-aware neural encoder for video captioning","authors":"Lorenzo Baraldi, Costantino Grana, Rita Cucchiara","year":"2017","venue":"Proceedings of the IEEE Conference on CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bengio et al., 2015] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for se- quence prediction with recurrent neural networks. Ad- vances in neural information processing systems, 28, 2015.","title":"Scheduled sampling for sequence prediction with recurrent neural networks","authors":"Samy Bengio, Oriol Vinyals, Navdeep Jaitly, Noam Shazeer","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Borji and Itti, 2012] Ali Borji and Laurent Itti. State-of-the- art in visual attention modeling. IEEE transactions on PAMI, 35(1):185-207, 2012.","title":"Ali Borji and Laurent Itti. State-of-theart in visual attention modeling","authors":"Itti Borji","year":"2012","venue":"IEEE transactions on PAMI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Buch et al., 2022] Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the\" video\" in video-language under- standing. In Proceedings of the IEEE/CVF Conference on CVPR, pages 2917-2927, 2022.","title":"Revisiting the\" video\" in video-language understanding","authors":"Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, Juan Carlos Niebles","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen and Dolan, 2011] David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the associa- tion for computational linguistics: human language tech- nologies, pages 190-200, 2011.","title":"Collecting highly parallel data for paraphrase evaluation","authors":"David Chen, William B Dolan","year":"2011","venue":"Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen and Jiang, 2019] Shaoxiang Chen and Yu-Gang Jiang. Motion guided spatial attention for video captioning. In Proceedings of the AAAI conference on artificial intelli- gence, volume 33, pages 8191-8198, 2019.","title":"Motion guided spatial attention for video captioning","authors":"Chen, Jiang, Shaoxiang Chen, Yu-Gang Jiang","year":"2019","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2018] Yangyu Chen, Shuhui Wang, Weigang Zhang, and Qingming Huang. Less is more: Picking in- formative frames for video captioning. In Proceedings of the ECCV, pages 358-373, 2018.","title":"Less is more: Picking informative frames for video captioning","authors":"Yangyu Chen, Shuhui Wang, Weigang Zhang, Qingming Huang","year":"2018","venue":"Proceedings of the ECCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen et al., 2020] Shaoxiang Chen, Wenhao Jiang, Wei Liu, and Yu-Gang Jiang. Learning modality interaction for temporal sentence localization and event captioning in videos. In European Conference on Computer Vision, pages 333-351, 2020.","title":"Learning modality interaction for temporal sentence localization and event captioning in videos","authors":"Shaoxiang Chen, Wenhao Jiang, Wei Liu, Yu-Gang Jiang","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Escorcia et al., 2019] Victor Escorcia, Mattia Soldan, Josef Sivic, Bernard Ghanem, and Bryan Russell. Temporal lo- calization of moments in video collections with natural language. arXiv preprint arXiv:1907.12763, 2019.","title":"Temporal localization of moments in video collections with natural language","authors":"Victor Escorcia, Mattia Soldan, Josef Sivic, Bernard Ghanem, Bryan Russell","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gabeur et al., 2020] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In European Conference on Computer Vision, pages 214-229, 2020.","title":"Multi-modal transformer for video retrieval","authors":"Valentin Gabeur, Chen Sun, Karteek Alahari, Cordelia Schmid","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gao et al., 2022] Yiqi Gao, Xinglin Hou, Wei Suo, Mengyang Sun, Tiezheng Ge, Yuning Jiang, and Peng Wang. Dual-level decoupled transformer for video cap- tioning. arXiv preprint arXiv:2205.03039, 2022.","title":"Dual-level decoupled transformer for video captioning","authors":"Yiqi Gao, Xinglin Hou, Wei Suo, Mengyang Sun, Tiezheng Ge, Yuning Jiang, Peng Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gygli et al., 2013] Michael Gygli, Helmut Grabner, Hayko Riemenschneider, Fabian Nater, and Luc Van Gool. The interestingness of images. In Proceedings of the IEEE ICCV, pages 1633-1640, 2013.","title":"The interestingness of images","authors":"Michael Gygli, Helmut Grabner, Hayko Riemenschneider, Fabian Nater, Luc Van Gool","year":"2013","venue":"Proceedings of the IEEE ICCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Gygli et al., 2014] Michael Gygli, Helmut Grabner, Hayko Riemenschneider, and Luc Van Gool. Creating summaries from user videos. In European conference on computer vision, pages 505-520, 2014.","title":"Creating summaries from user videos","authors":"Michael Gygli, Helmut Grabner, Hayko Riemenschneider, Luc Van Gool","year":"2014","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hessel et al., 2021] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021.","title":"Clipscore: A reference-free evaluation metric for image captioning","authors":"Hessel","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lei et al., 2021] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In Proceedings of the IEEE/CVF Conference on CVPR, pages 7331-7341, 2021.","title":"Less is more: Clipbert for video-and-language learning via sparse sampling","authors":"Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, Jingjing Liu","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lei et al., 2022] Jie Lei, Tamara L Berg, and Mohit Bansal. Revealing single frame bias for video-and-language learn- ing. arXiv preprint arXiv:2206.03428, 2022.","title":"Revealing single frame bias for video-and-language learning","authors":"Jie Lei, Tamara L Berg, Mohit Bansal","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lin et al., 2021] Xudong Lin, Gedas Bertasius, Jue Wang, Shih-Fu Chang, Devi Parikh, and Lorenzo Torresani. Vx2text: End-to-end learning of video-based text gen- eration from multimodal inputs. In Proceedings of the IEEE/CVF Conference on CVPR, pages 7005-7015, 2021.","title":"Vx2text: End-to-end learning of video-based text generation from multimodal inputs","authors":"Xudong Lin, Gedas Bertasius, Jue Wang, Shih-Fu Chang, Devi Parikh, Lorenzo Torresani","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2021] Fenglin Liu, Xuancheng Ren, Xian Wu, Bang Yang, Shen Ge, and Xu Sun. O2na: An object-oriented non-autoregressive approach for control- lable video captioning. arXiv preprint arXiv:2108.02359, 2021.","title":"O2na: An object-oriented non-autoregressive approach for controllable video captioning","authors":"Fenglin Liu, Xuancheng Ren, Xian Wu, Bang Yang, Shen Ge, Xu Sun","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Luo et al., 2021] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval. arXiv preprint arXiv:2104.08860, 2021.","title":"Clip4clip: An empirical study of clip for end to end video clip retrieval","authors":"Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, Tianrui Li","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Narasimhan et al., 2021] Medhini Narasimhan, Anna Rohrbach, and Trevor Darrell. Clip-it! language-guided video summarization. Advances in Neural Information Processing Systems, 34:13988-14000, 2021.","title":"Clip-it! language-guided video summarization","authors":"Narasimhan","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Pan et al., 2016] Pingbo Pan, Zhongwen Xu, Yi Yang, Fei Wu, and Yueting Zhuang. Hierarchical recurrent neural encoder for video representation with application to cap- tioning. In Proceedings of the IEEE Conference on CVPR, pages 1029-1038, 2016.","title":"Hierarchical recurrent neural encoder for video representation with application to captioning","authors":"Pan","year":"2016","venue":"Proceedings of the IEEE Conference on CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Pan et al., 2020] Boxiao Pan, Haoye Cai, De-An Huang, Kuan-Hui Lee, Adrien Gaidon, Ehsan Adeli, and Juan Car- los Niebles. Spatio-temporal graph for video caption- ing with knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10870-10879, 2020.","title":"Spatio-temporal graph for video captioning with knowledge distillation","authors":"Pan","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Pei et al., 2019] Wenjie Pei, Jiyuan Zhang, Xiangrong Wang, Lei Ke, Xiaoyong Shen, and Yu-Wing Tai. Memory-attended recurrent network for video captioning. In Proceedings of the IEEE/CVF Conference on CVPR, pages 8347-8356, 2019.","title":"Memory-attended recurrent network for video captioning","authors":"Wenjie Pei, Jiyuan Zhang, Xiangrong Wang, Lei Ke, Xiaoyong Shen, Yu-Wing Tai","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Qiu et al., 2022] Jielin Qiu, Jiacheng Zhu, Mengdi Xu, Franck Dernoncourt, Trung Bui, Zhaowen Wang, Bo Li, Ding Zhao, and Hailin Jin. Mhms: Multimodal hi- erarchical multimedia summarization. arXiv preprint arXiv:2204.03734, 2022.","title":"Mhms: Multimodal hierarchical multimedia summarization","authors":"Qiu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In International Conference on Machine Learning, pages 8748-8763, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Ryu et al., 2021] Hobin Ryu, Sunghun Kang, Haeyong Kang, and Chang D Yoo. Semantic grouping network for video captioning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2514-2522, 2021.","title":"Semantic grouping network for video captioning","authors":"Sunghun Hobin Ryu, Haeyong Kang, Chang D Kang, Yoo","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sanabria et al., 2018] Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loïc Barrault, Lucia Specia, and Florian Metze. How2: A large-scale dataset for multimodal language understanding. In NeurIPS, 2018.","title":"How2: A large-scale dataset for multimodal language understanding","authors":"Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loïc Barrault, Lucia Specia, Florian Metze","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Shang et al., 2021] Xindi Shang, Zehuan Yuan, Anran Wang, and Changhu Wang. Multimodal video summariza- tion via time-aware transformers. In Proceedings of the 29th ACM International Conference on Multimedia, pages 1756-1765, 2021.","title":"Multimodal video summarization via time-aware transformers","authors":"Shang","year":"2021","venue":"Proceedings of the 29th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Song et al., 2015] Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes. Tvsum: Summarizing web videos using titles. In Proceedings of the IEEE Confer- ence on CVPR, pages 5179-5187, 2015.","title":"Tvsum: Summarizing web videos using titles","authors":"Song","year":"2015","venue":"Proceedings of the IEEE Conference on CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Song et al., 2017] Jingkuan Song, Zhao Guo, Lianli Gao, Wu Liu, Dongxiang Zhang, and Heng Tao Shen. Hier- archical lstm with adjusted temporal attention for video captioning. arXiv preprint arXiv:1706.01231, 2017.","title":"Hierarchical lstm with adjusted temporal attention for video captioning","authors":"Jingkuan Song, Zhao Guo, Lianli Gao, Wu Liu, Dongxiang Zhang, Heng Tao Shen","year":"2017","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2017/381"},{"position":35,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Vedantam et al., 2015] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In Proceedings of the IEEE Conference on CVPR, pages 4566-4575, 2015.","title":"Cider: Consensusbased image description evaluation","authors":"Ramakrishna Vedantam, Lawrence Zitnick, Devi Parikh","year":"2015","venue":"Proceedings of the IEEE Conference on CVPR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Venugopalan et al., 2015] Subhashini Venugopalan, Mar- cus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence- video to text. In Proceedings of the IEEE ICCV, pages 4534-4542, 2015.","title":"Sequence to sequencevideo to text","authors":"Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, Kate Saenko","year":"2015","venue":"Proceedings of the IEEE ICCV","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wang et al., 2019] Bairui Wang, Lin Ma, Wei Zhang, Wen- hao Jiang, Jingwen Wang, and Wei Liu. Controllable video captioning with pos sequence guidance based on gated fu- sion network. In Proceedings of the IEEE/CVF ICCV, pages 2641-2650, 2019.","title":"Controllable video captioning with pos sequence guidance based on gated fusion network","authors":"Bairui Wang, Lin Ma, Wei Zhang, Wenhao Jiang, Jingwen Wang, Wei Liu","year":"2019","venue":"Proceedings of the IEEE/CVF ICCV","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Xiao et al., 2020] Shuwen Xiao, Zhou Zhao, Zijian Zhang, Xiaohui Yan, and Min Yang. Convolutional hierarchical attention network for query-focused video summarization. In Proceedings of the AAAI Conference on Artificial Intel- ligence, volume 34, pages 12426-12433, 2020.","title":"Convolutional hierarchical attention network for query-focused video summarization","authors":"Shuwen Xiao, Zhou Zhao, Zijian Zhang, Xiaohui Yan, Min Yang","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Xu et al., 2016] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE Confer- ence on CVPR, pages 5288-5296, 2016.","title":"Msr-vtt: A large video description dataset for bridging video and language","authors":"Jun Xu, Tao Mei, Ting Yao, Yong Rui","year":"2016","venue":"Proceedings of the IEEE Conference on CVPR","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Yan et al., 2022] Liqi Yan, Qifan Wang, Yiming Cui, Fuli Feng, Xiaojun Quan, Xiangyu Zhang, and Dongfang Liu. Gl-rg: Global-local representation granularity for video captioning. arXiv preprint arXiv:2205.10706, 2022.","title":"Gl-rg: Global-local representation granularity for video captioning","authors":"Yan","year":"2022","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2022/384"},{"position":42,"raw":"[Yang et al., 2021] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learn- ing to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF ICCV, pages 1686-1697, 2021.","title":"Just ask: Learning to answer questions from millions of narrated videos","authors":"Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, Cordelia Schmid","year":"2021","venue":"Proceedings of the IEEE/CVF ICCV","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Zhang and Peng, 2020] Junchao Zhang and Yuxin Peng. Video captioning with object-aware spatio-temporal corre- lation and aggregation. IEEE Transactions on Image Pro- cessing, 29:6209-6222, 2020.","title":"Video captioning with object-aware spatio-temporal correlation and aggregation","authors":"Peng Zhang, Junchao Zhang, Yuxin Peng","year":"2020","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Zhang et al., 2020] Ziqi Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, and Zheng-Jun Zha. Object relational graph with teacher-recommended learn- ing for video captioning. In Proceedings of the IEEE/CVF Conference on CVPR, pages 13278-13288, 2020.","title":"Object relational graph with teacher-recommended learning for video captioning","authors":"Ziqi Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, Zheng-Jun Zha","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on CVPR","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Zhang et al., 2021] Ziqi Zhang, Zhongang Qi, Chunfeng Yuan, Ying Shan, Bing Li, Ying Deng, and Weiming Hu. Open-book video captioning with retrieve-copy-generate network. In Proceedings of the IEEE/CVF Conference on CVPR, pages 9837-9846, 2021.","title":"Open-book video captioning with retrieve-copy-generate network","authors":"Ziqi Zhang, Zhongang Qi, Chunfeng Yuan, Ying Shan, Bing Li, Ying Deng, Weiming Hu","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on CVPR","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Zhao and Xing, 2014] Bin Zhao and Eric P Xing. Quasi real-time summarization for consumer videos. In Proceed- ings of the IEEE Conference on CVPR, pages 2513-2520, 2014.","title":"Quasi real-time summarization for consumer videos","authors":"Xing ; ] Bin Zhao, Eric P Zhao, Xing","year":"2014","venue":"Proceedings of the IEEE Conference on CVPR","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Zheng et al., 2020] Qi Zheng, Chaoyue Wang, and Dacheng Tao. Syntax-aware action targeting for video captioning. In Proceedings of the IEEE/CVF Conference on CVPR, pages 13096-13105, 2020.","title":"Syntax-aware action targeting for video captioning","authors":"Zheng","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on CVPR","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Zhou et al., 2018] Kaiyang Zhou, Yu Qiao, and Tao Xiang. Deep reinforcement learning for unsupervised video sum- marization with diversity-representativeness reward. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 32, 2018.","title":"Deep reinforcement learning for unsupervised video summarization with diversity-representativeness reward","authors":"Kaiyang Zhou, Yu Qiao, Tao Xiang","year":"2018","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}