{"type":"ScholarlyArticle","title":"Prompt Learns Prompt: Exploring Knowledge-Aware Generative Prompt Collaboration For Video Captioning","authors":[{"name":"Liqi Yan","slug":"liqi-yan","given":"Liqi","family":"Yan","affiliation":"Westlake Institute for Advanced Study , Fudan University"},{"name":"Cheng Han","slug":"cheng-han","given":"Cheng","family":"Han","affiliation":"Chester"},{"name":"Zenglin Xu","slug":"zenglin-xu","given":"Zenglin","family":"Xu","affiliation":"School of Computer Science and Technology , Harbin Institute of Technology; F . Carlson Center for Imaging Science , Rochester Institute of Technology"},{"name":"Dongfang Liu","slug":"dongfang-liu","given":"Dongfang","family":"Liu","affiliation":"Chester"},{"name":"Qifan Wang","slug":"qifan-wang","given":"Qifan","family":"Wang","affiliation":"Meta AI , Meta"}],"abstract":"Fine-tuning large vision-language models is a challenging task. Prompt tuning approaches have been introduced to learn fixed textual or visual prompts while freezing the pre-trained model in downstream tasks. Despite the effectiveness of prompt tuning, what do those learnable prompts learn remains unexplained. In this work, we explore whether prompts in the fine-tuning can learn knowledge-aware prompts from the pre-training, by designing two different sets of prompts in pre-training and fine-tuning phases respectively. Specifically, we present a Video-Language Prompt tuning (VL-Prompt) approach for video captioning, which first efficiently pre-train a video-language model to extract key information (e.g., actions and objects) with flexibly generated Knowledge-Aware Prompt (KAP). Then, we design a Video-Language Prompt (VLP) to transfer the knowledge from the knowledge-aware prompts and fine-tune the model to generate full captions. Experimental results show the superior performance of our approach over several state-of-the-art baselines. We further demonstrate that the video-language prompts are well learned from the knowledge-aware prompts.","doi":"10.24963/ijcai.2023/180","url":"https://ijcai.org/proceedings/2023/180","pdf":"https://ijcai.org/proceedings/2023/0180.pdf","markdown":"https://ijcai.org/proceedings/2023/180.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/180","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":180,"pages":{"first":1622,"last":1630},"keywords":["Computer Vision: CV: Vision and language","Computer Vision: CV: Video analysis and understanding"],"videos":[],"references":[{"position":1,"raw":"[Akbik et al., 2019] Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Voll- graf. FLAIR: An easy-to-use framework for state-of-the- art NLP. In NAACL, pages 54-59, 2019.","title":"FLAIR: An easy-to-use framework for state-of-theart NLP","authors":"Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, Roland Vollgraf","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alec and Karthik, 2018] Radford Alec and Narasimhan Karthik. Improving language understanding by generative pre-training. OpenAI, 2018.","title":"Improving language understanding by generative pre-training","authors":"Alec Radford, Narasimhan Karthik","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Arnab et al., 2021] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In ICCV, 2021.","title":"Vivit: A video vision transformer","authors":"Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, Cordelia Schmid","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Banerjee and Lavie, 2005] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In ACL workshop, pages 65-72, 2005.","title":"Meteor: An automatic metric for mt evaluation with improved correlation with human judgments","authors":"Lavie Banerjee, Satanjeev Banerjee, Alon Lavie","year":"2005","venue":"ACL workshop","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Beltagy et al., 2020] Iz Beltagy, Matthew E Peters, and Ar- man Cohan. Longformer: The long-document trans- former. arXiv preprint arXiv:2004.05150, 2020.","title":"Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer","authors":"Beltagy","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bertasius et al., 2021] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, volume 2, page 4, 2021.","title":"Is space-time attention all you need for video understanding?","authors":"Gedas Bertasius, Heng Wang, Lorenzo Torresani","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2018] Yangyu Chen, Shuhui Wang, Weigang Zhang, and Qingming Huang. Less is more: Picking in- formative frames for video captioning. In ECCV, 2018.","title":"Less is more: Picking informative frames for video captioning","authors":"Yangyu Chen, Shuhui Wang, Weigang Zhang, Qingming Huang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2020] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020.","title":"Generative pretraining from pixels","authors":"Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, Ilya Sutskever","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2022] Long Chen, Yuhang Zheng, and Jun Xiao. Rethinking data augmentation for robust visual question answering. In ECCV, pages 95-112, 2022.","title":"Rethinking data augmentation for robust visual question answering","authors":"Chen","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Du et al., 2022] Yu Du, Fangyun Wei, Zihe Zhang, Miao- jing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vision-language model. In CVPR, pages 14084-14093, 2022.","title":"Learning to prompt for open-vocabulary object detection with vision-language model","authors":"Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, Guoqi Li","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Huang et al., 2022] Siteng Huang, Biao Gong, Yulin Pan, Jianwen Jiang, Yiliang Lv, Yuyuan Li, and Donglin Wang. Vop: Text-video co-operative prompt tuning for cross- modal retrieval. arXiv preprint arXiv:2211.12764, 2022.","title":"Vop: Text-video co-operative prompt tuning for crossmodal retrieval","authors":"Siteng Huang, Biao Gong, Yulin Pan, Jianwen Jiang, Yiliang Lv, Yuyuan Li, Donglin Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jia et al., 2022] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariha- ran, and Ser-Nam Lim. Visual prompt tuning. In ECCV, pages 709-727, 2022.","title":"Visual prompt tuning","authors":"Jia","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ju et al., 2022] Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. In ECCV, 2022.","title":"Prompting visual-language models for efficient video understanding","authors":"Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, Weidi Xie","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lester et al., 2021] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP, pages 3045-3059, 2021.","title":"The power of scale for parameter-efficient prompt tuning","authors":"Brian Lester, Rami Al-Rfou, Noah Constant","year":"2021","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li and Liang, 2021] Xiang Lisa Li and Percy Liang. Prefix- tuning: Optimizing continuous prompts for generation. In ACL-IJCNLP, pages 4582-4597, 2021.","title":"Prefixtuning: Optimizing continuous prompts for generation","authors":"Lisa Xiang, Percy Li, Liang","year":"2021","venue":"ACL-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2021] Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven C. H. Hoi. Align and prompt: Video-and-language pre-training with entity prompts. CVPR, pages 4943-4953, 2021.","title":"Align and prompt: Video-and-language pre-training with entity prompts","authors":"Li","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lin and Och, 2004] Chin-Yew Lin and Franz Josef Och. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In ACL, pages 605-612, 2004.","title":"Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics","authors":"Lin, Och ; Chin-Yew Lin, Franz Josef, Och","year":"2004","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lin et al., 2022] Kevin Lin, Linjie Li, Chung-Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang. Swinbert: End-to-end transformers with sparse attention for video captioning. In CVPR, 2022.","title":"Swinbert: End-to-end transformers with sparse attention for video captioning","authors":"Kevin Lin, Linjie Li, Chung-Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, Lijuan Wang","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2021a] Fenglin Liu, Xuancheng Ren, Xian Wu, Bang Yang, Shen Ge, and Xu Sun. O2NA: an object- oriented non-autoregressive approach for controllable video captioning. In ACL/IJCNLP, 2021.","title":"O2NA: an objectoriented non-autoregressive approach for controllable video captioning","authors":"Liu","year":"2021","venue":"ACL/IJCNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2021b] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 2021.","title":"Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing","authors":"Liu","year":"2021","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liu et al., 2021c] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. ArXiv, abs/2103.10385, 2021.","title":"Gpt understands, too","authors":"Liu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Liu et al., 2021d] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Liu","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Liu et al., 2022] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin trans- former. In CVPR, pages 3202-3211, 2022.","title":"Video swin transformer","authors":"Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, Han Hu","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Nag et al., 2022] Sauradip Nag, Xiatian Zhu, Yi-Zhe Song, and Tao Xiang. Zero-shot temporal action detection via vision-language prompting. In ECCV, 2022.","title":"Zero-shot temporal action detection via vision-language prompting","authors":"Nag","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Pan et al., 2020] Boxiao Pan, Haoye Cai, De-An Huang, Kuan-Hui Lee, Adrien Gaidon, Ehsan Adeli, and Juan Car- los Niebles. Spatio-temporal graph for video captioning with knowledge distillation. In CVPR, 2020.","title":"Spatio-temporal graph for video captioning with knowledge distillation","authors":"Pan","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In ACL, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Qin et al., 2023] Zheyun Qin, Xiankai Lu, Xiushan Nie, Dongfang Liu, Yilong Yin, and Wenguan Wang. Coarse- to-fine video instance segmentation with factorized con- ditional appearance flows. Journal of Automatica Sinica, 10(5):1-17, 2023.","title":"Coarseto-fine video instance segmentation with factorized conditional appearance flows","authors":"Qin","year":"2023","venue":"Journal of Automatica Sinica","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In ICML, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Ryu et al., 2021] Hobin Ryu, Sunghun Kang, Haeyong Kang, and C. Yoo. Semantic grouping network for video captioning. In AAAI, 2021.","title":"Semantic grouping network for video captioning","authors":"Sunghun Hobin Ryu, Haeyong Kang, C Kang, Yoo","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Shannon, 1948] Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical jour- nal, 27(3):379-423, 1948.","title":"A mathematical theory of communication","authors":"Claude Elwood, Shannon","year":"1948","venue":"The Bell system technical journal","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Shen et al., 2022] Sheng Shen, Shijia Yang, Tianjun Zhang, Bohan Zhai, Joseph E Gonzalez, Kurt Keutzer, and Trevor Darrell. Multitask vision-language prompt tuning. arXiv preprint arXiv:2211.11720, 2022.","title":"Multitask vision-language prompt tuning","authors":"Sheng Shen, Shijia Yang, Tianjun Zhang, Bohan Zhai, Joseph E Gonzalez, Kurt Keutzer, Trevor Darrell","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Shin et al., 2020] Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automat- ically generated prompts. In EMNLP, 2020.","title":"Autoprompt: Eliciting knowledge from language models with automatically generated prompts","authors":"Taylor Shin, Yasaman Razeghi, Robert L Logan, I V, Eric Wallace, Sameer Singh","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Shu et al., 2022] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot gen- eralization in vision-language models. In NeurIPS, 2022.","title":"Test-time prompt tuning for zero-shot generalization in vision-language models","authors":"Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, Chaowei Xiao","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Tay et al., 2020] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys, 2020.","title":"Efficient transformers: A survey","authors":"Yi Tay, Mostafa Dehghani, Dara Bahri, Donald Metzler","year":"2020","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Thomas and Joy, 2006] MTCAJ Thomas and A Thomas Joy. Elements of information theory. Wiley-Interscience, 2006.","title":"Elements of information theory","authors":"Joy Thomas","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Uzair Khattak et al., 2022] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. arXiv e-prints, pages arXiv-2210, 2022.","title":"Maple: Multi-modal prompt learning","authors":"Uzair Khattak","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Vedantam et al., 2015] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In CVPR, 2015.","title":"Cider: Consensusbased image description evaluation","authors":"Ramakrishna Vedantam, Lawrence Zitnick, Devi Parikh","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Venugopalan et al., 2015] Subhashini Venugopalan, Mar- cus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence- video to text. In CVPR, pages 4534-4542, 2015.","title":"Sequence to sequencevideo to text","authors":"Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, Kate Saenko","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Wang et al., 2018a] Bairui Wang, Lin Ma, Wei Zhang, and Wei Liu. Reconstruction network for video captioning. In CVPR, 2018.","title":"Reconstruction network for video captioning","authors":"Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Wang et al., 2018b] Xin Wang, Wenhu Chen, Jiawei Wu, Yuan-Fang Wang, and William Yang Wang. Video cap- tioning via hierarchical reinforcement learning. In CVPR, 2018.","title":"Video captioning via hierarchical reinforcement learning","authors":"Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Wang et al., 2019] Bairui Wang, L. Ma, W. Zhang, Wenhao Jiang, Junling Wang, and W. Liu. Controllable video cap- tioning with pos sequence guidance based on gated fusion network. In ICCV, 2019.","title":"Controllable video captioning with pos sequence guidance based on gated fusion network","authors":"L Bairui Wang, W Ma, Wenhao Zhang, Junling Jiang, W Wang, Liu","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Wang et al., 2022a] Qifan Wang, Yi Fang, Anirudh Ravula, Fuli Feng, Xiaojun Quan, and Dongfang Liu. Webformer: The web-page transformer for structure information ex- traction. In ACM Web Conference, 2022.","title":"Webformer: The web-page transformer for structure information extraction","authors":"Wang","year":"2022","venue":"ACM Web Conference","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Wang et al., 2022b] Wenguan Wang, James Liang, and Dongfang Liu. Learning equivariant segmentation with instance-unique querying. In NeurIPS, 2022.","title":"Learning equivariant segmentation with instance-unique querying","authors":"Wang","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Wang et al., 2023] Wenguan Wang, Cheng Han, Tianfei Zhou, and Dongfang Liu. Visual recognition with deep nearest centroids. In ICLR, 2023.","title":"Visual recognition with deep nearest centroids","authors":"Wang","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Xu et al., 2016] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016.","title":"Msr-vtt: A large video description dataset for bridging video and language","authors":"Jun Xu, Tao Mei, Ting Yao, Yong Rui","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Yan et al., 2022a] Liqi Yan, Siqi Ma, Qifan Wang, Yingjie Chen, Xiangyu Zhang, Andreas Savakis, and Dongfang Liu. Video captioning using global-local representation. TCSVT, 32(10):6642-6656, 2022.","title":"Video captioning using global-local representation","authors":"Yan","year":"2022","venue":"TCSVT","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Yan et al., 2022b] Liqi Yan, Qifan Wang, Yiming Cui, Fuli Feng, Xiaojun Quan, Xiangyu Zhang, and Dongfang Liu. Gl-rg: Global-local representation granularity for video captioning. In IJCAI, 2022.","title":"Gl-rg: Global-local representation granularity for video captioning","authors":"Yan","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/384"},{"position":51,"raw":"[Ye et al., 2022] Hanhua Ye, Guorong Li, Yuankai Qi, Shuhui Wang, Qingming Huang, and Ming-Hsuan Yang. Hierarchical modular network for video captioning. In CVPR, pages 17939-17948, 2022.","title":"Hierarchical modular network for video captioning","authors":"Ye","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Zaheer et al., 2020] Manzil Zaheer, Guru Guruganesh, Ku- mar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santi- ago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer se- quences. NeurIPS, 33:17283-17297, 2020.","title":"Big bird: Transformers for longer sequences","authors":"Manzil Zaheer, Guru Guruganesh, Avinava Kumar, Joshua Dubey, Chris Ainslie, Santiago Alberti, Philip Ontanon, Anirudh Pham, Qifan Ravula, Li Wang, Yang","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Zang et al., 2022] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vi- sion and language prompt learning. arXiv preprint arXiv:2210.07225, 2022.","title":"Unified vision and language prompt learning","authors":"Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, Chen Change Loy","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Zhang and Peng, 2019] Junchao Zhang and Yuxin Peng. Object-aware aggregation with bidirectional temporal graph for video captioning. In CVPR, 2019.","title":"Object-aware aggregation with bidirectional temporal graph for video captioning","authors":"Peng Zhang, Junchao Zhang, Yuxin Peng","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Zhang et al., 2020] Ziqi Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, and Zheng-Jun Zha. Object relational graph with teacher-recommended learn- ing for video captioning. In CVPR, 2020.","title":"Object relational graph with teacher-recommended learning for video captioning","authors":"Ziqi Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, Zheng-Jun Zha","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Zhang et al., 2021] Ziqi Zhang, Zhongang Qi, Chunfeng Yuan, Ying Shan, Bing Li, Ying Deng, and Weiming Hu. Open-book video captioning with retrieve-copy-generate network. In CVPR, 2021.","title":"Open-book video captioning with retrieve-copy-generate network","authors":"Ziqi Zhang, Zhongang Qi, Chunfeng Yuan, Ying Shan, Bing Li, Ying Deng, Weiming Hu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":57,"raw":"[Zheng et al., 2020] Qi Zheng, Chaoyue Wang, and D. Tao. Syntax-aware action targeting for video captioning. In CVPR, 2020.","title":"Syntax-aware action targeting for video captioning","authors":"Qi Zheng, Chaoyue Wang, D Tao","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":58,"raw":"[Zhengbao et al., 2019] Jiang Zhengbao, Xu Frank, F., Araki J., and Neubig Graham. How can we know what language models know? TACL, 8:423-438, 2019.","title":"How can we know what language models know?","authors":"Jiang Zhengbao, Xu Frank, F Araki, J, Neubig Graham","year":"2019","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":59,"raw":"[Zhou et al., 2022a] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022.","title":"Conditional prompt learning for vision-language models","authors":"Zhou","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":60,"raw":"[Zhou et al., 2022b] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 130(9):2337-2348, 2022.","title":"Learning to prompt for vision-language models","authors":"Zhou","year":"2022","venue":"IJCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}