{"type":"ScholarlyArticle","title":"GUIDE: A Guideline-Guided Dataset for Instructional Video Comprehension","authors":[{"name":"Jiafeng Liang","slug":"jiafeng-liang","given":"Jiafeng","family":"Liang","affiliation":"Harbin Institute of Technology , Harbin , China"},{"name":"Shixin Jiang","slug":"shixin-jiang","given":"Shixin","family":"Jiang","affiliation":"Harbin Institute of Technology , Harbin , China"},{"name":"Zekun Wang","slug":"zekun-wang","given":"Zekun","family":"Wang","affiliation":"Harbin Institute of Technology , Harbin , China"},{"name":"Haojie Pan","slug":"haojie-pan","given":"Haojie","family":"Pan","affiliation":"Kuaishou Technology , Beijing , China"},{"name":"Zerui Chen","slug":"zerui-chen","given":"Zerui","family":"Chen","affiliation":"Harbin Institute of Technology , Harbin , China"},{"name":"Zheng Chu","slug":"zheng-chu","given":"Zheng","family":"Chu","affiliation":"Harbin Institute of Technology , Harbin , China"},{"name":"Ming Liu","slug":"ming-liu","given":"Ming","family":"Liu","affiliation":"Harbin Institute of Technology , Harbin , China; Peng Cheng Laboratory , Shenzhen , China"},{"name":"Ruiji Fu","slug":"ruiji-fu","given":"Ruiji","family":"Fu","affiliation":"Kuaishou Technology , Beijing , China"},{"name":"Zhongyuan Wang","slug":"zhongyuan-wang","given":"Zhongyuan","family":"Wang","affiliation":"Kuaishou Technology , Beijing , China"},{"name":"Bing Qin","slug":"bing-qin","given":"Bing","family":"Qin","affiliation":"Harbin Institute of Technology , Harbin , China; Peng Cheng Laboratory , Shenzhen , China"}],"abstract":"There are substantial instructional videos on the Internet, which provide us tutorials for completing various tasks. Existing instructional video datasets only focus on specific steps at the video level, lacking experiential guidelines at the task level, which can lead to beginners struggling to learn new tasks due to the lack of relevant experience. Moreover, the specific steps without guidelines are trivial and unsystematic, making it difficult to provide a clear tutorial. To address these problems, we present the Guide (Guideline-Guided) dataset, which contains 3.5K videos of 560 instructional tasks in 8 domains related to our daily life. Specifically, we annotate each instructional task with a guideline, representing a common pattern shared by all task-related videos. On this basis, we annotate systematic specific steps, including their associated guideline steps, specific step descriptions and timestamps. Our proposed benchmark consists of three sub-tasks to evaluate comprehension ability of models: (1) Step Captioning: models have to generate captions for specific steps from videos. (2) Guideline Summarization: models have to mine the common pattern in task-related videos and summarize a guideline from them. (3) Guideline-Guided Captioning: models have to generate captions for specific steps under the guide of guideline. We evaluate plenty of foundation models with Guide and perform in-depth analysis. Given the diversity and practicality of Guide, we believe that it can be used as a better benchmark for instructional video comprehension.","doi":"10.24963/ijcai.2024/118","url":"https://ijcai.org/proceedings/2024/118","pdf":"https://ijcai.org/proceedings/2024/0118.pdf","markdown":"https://ijcai.org/proceedings/2024/118.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/118","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":118,"pages":{"first":1065,"last":1073},"keywords":["Natural Language Processing: NLP: Resources and evaluation","Computer Vision: CV: Video analysis and understanding"],"videos":[],"references":[{"position":1,"raw":"[Anderson et al., 2016] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE: semantic propo- sitional image caption evaluation. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision -ECCV 2016 -14th European Conference, Amster- dam, The Netherlands, October 11-14, 2016, Proceedings, Part V, volume 9909 of Lecture Notes in Computer Science, pages 382-398. Springer, 2016.","title":"SPICE: semantic propositional image caption evaluation","authors":"Anderson","year":"2016","venue":"Computer Vision -ECCV 2016 -14th European Conference","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Banerjee and Lavie, 2005] Satanjeev Banerjee and Alon Lavie. METEOR: an automatic metric for MT evalua- tion with improved correlation with human judgments. In Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare R. Voss, editors, Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL 2005, Ann Arbor, Michigan, USA, June 29, 2005, pages 65-72. Association for Compu- tational Linguistics, 2005.","title":"METEOR: an automatic metric for MT evaluation with improved correlation with human judgments","authors":"Lavie Banerjee, Satanjeev Banerjee, Alon Lavie","year":"2005","venue":"Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL 2005","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bolkan et al., 2020] San Bolkan, Goodboy, and Alan K. In- struction, example order, and student learning: Reducing extraneous cognitive load by providing structure for elabo- rated examples. Communication Education, 69(3):300-316, 2020.","title":"Instruction, example order, and student learning: Reducing extraneous cognitive load by providing structure for elaborated examples","authors":"Bolkan","year":"2020","venue":"Communication Education","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chiang et al., 2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.","title":"Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality","authors":"Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, Ion Stoica, Eric P Xing","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chung et al., 2022] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro- Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yan- ping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetuned language models, 2022.","title":"Scaling instruction-finetuned language models","authors":"Chung Hyung Won, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shane Shixiang, Zhuyun Gu, Mirac Dai, Xinyun Suzgun, Aakanksha Chen, Alex Chowdhery, Marie Castro-Ros, Kevin Pellat, Dasha Robinson, Sharan Valter, Gaurav Narang, Adams Mishra, Vincent Yu, Yanping Zhao, Andrew Huang, Hongkun Dai, Slav Yu, Ed H Petrov, Jeff Chi, Jacob Dean, Adam Devlin, Denny Roberts, Quoc V Zhou, Jason Le, Wei","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Damen et al., 2018] Dima Damen, Hazel Doughty, Gio- vanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocen- tric vision: The dataset. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision -ECCV 2018 -15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part IV, vol- ume 11208 of Lecture Notes in Computer Science, pages 753-771. Springer, 2018.","title":"Scaling egocentric vision: The dataset","authors":"Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, Michael Wray","year":"2018","venue":"Computer Vision -ECCV 2018 -15th European Conference","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Vir- tual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. Dvornik et al., 2023]","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":"9th International Conference on Learning Representations, ICLR 2021, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"Nikita Dvornik, Isma Hadji, Ran Zhang, Konstantinos G. Derpanis, Richard P. Wildes, and Allan D. Jepson. Stepformer: Self-supervised step discov- ery and localization in instructional videos. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 18952-18961. IEEE, 2023.","title":"Stepformer: Self-supervised step discovery and localization in instructional videos","authors":"Nikita Dvornik, Isma Hadji, Ran Zhang, Konstantinos G Derpanis, Richard P Wildes, Allan D Jepson","year":"2023","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Girdhar et al., 2023] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Ar- mand Joulin, and Ishan Misra. Imagebind one embedding space to bind them all. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2023, Vancou- ver, BC, Canada, June 17-24, 2023, pages 15180-15190. IEEE, 2023.","title":"Imagebind one embedding space to bind them all","authors":"Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, Ishan Misra","year":"2023","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gu et al., 2023] Xin Gu, Guang Chen, Yufei Wang, Libo Zhang, Tiejian Luo, and Longyin Wen. Text with knowl- edge graph augmented transformer for video captioning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 18941-18951. IEEE, 2023.","title":"Text with knowledge graph augmented transformer for video captioning","authors":"Xin Gu, Guang Chen, Yufei Wang, Libo Zhang, Tiejian Luo, Longyin Wen","year":"2023","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Han et al., 2020] Tengda Han, Weidi Xie, and Andrew Zis- serman. Self-supervised co-training for video represen- tation learning. In Hugo Larochelle, Marc'Aurelio Ran- zato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.","title":"Self-supervised co-training for video representation learning","authors":"Han","year":"2020","venue":"Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2023] Tianyao He, Huabin Liu, Yuxi Li, Xiao Ma, Cheng Zhong, Yang Zhang, and Weiyao Lin. Collaborative weakly supervised video correlation learning for procedure- aware instructional video analysis, 2023.","title":"Collaborative weakly supervised video correlation learning for procedureaware instructional video analysis","authors":"He","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang et al., 2018] De-An Huang, Shyamal Buch, Lucio M. Dery, Animesh Garg, Li Fei-Fei, and Juan Carlos Niebles. Finding \"it\": Weakly-supervised reference-aware visual grounding in instructional videos. In 2018 IEEE Confer- ence on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 5948-5957. Computer Vision Foundation / IEEE Computer Society, 2018.","title":"Finding \"it\": Weakly-supervised reference-aware visual grounding in instructional videos","authors":"De-An, Shyamal Huang, Lucio M Buch, Animesh Dery, Li Garg, Juan Carlos Fei-Fei, Niebles","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Iashin and Rahtu, 2020] Vladimir Iashin and Esa Rahtu. A better use of audio-visual cues: Dense video captioning with bi-modal transformer. In 31st British Machine Vision Conference 2020, BMVC 2020, Virtual Event, UK, Septem- ber 7-10, 2020. BMVA Press, 2020.","title":"Vladimir Iashin and Esa Rahtu. A better use of audio-visual cues: Dense video captioning with bi-modal transformer","authors":"Rahtu Iashin","year":"2020","venue":"31st British Machine Vision Conference 2020, BMVC 2020, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kuehne et al., 2014] Hilde Kuehne, Ali Bilgin Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, pages 780-787. IEEE Computer Society, 2014.","title":"The language of actions: Recovering the syntax and semantics of goal-directed human activities","authors":"Kuehne","year":"2014","venue":"2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2023a] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scar- lett, editors, International Conference on Machine Learn- ing, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 19730-19742.","title":"BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models","authors":"Li","year":"2023","venue":"International Conference on Machine Learning, ICML 2023","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"PMLR, 2023.","title":null,"authors":"Pmlr","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2023b] Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. CoRR, abs/2305.06355, 2023.","title":"Videochat: Chat-centric video understanding","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liang et al., 2023] Jiafeng Liang, Yuxin Wang, Zekun Wang, Ming Liu, Ruiji Fu, Zhongyuan Wang, and Bing Qin. GTR: A grafting-then-reassembling framework for dynamic scene graph generation. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, China, pages 1177-1185. ijcai.org, 2023.","title":"GTR: A grafting-then-reassembling framework for dynamic scene graph generation","authors":"Liang","year":"2023","venue":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023","doi":null,"url":null,"internal_url":"/proceedings/2023/131"},{"position":20,"raw":"[OpenAI, 2022] OpenAI. Introducing chatgpt, 2022. [OpenAI, 2023] OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023.","title":"OpenAI. Introducing chatgpt","authors":"Openai","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Radford et al., 2023] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Ma- chine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 28492-28518. PMLR, 2023.","title":"Robust speech recognition via large-scale weak supervision","authors":"Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, Ilya Sutskever","year":"2023","venue":"International Conference on Machine Learning, ICML 2023","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tang et al., 2019] Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. COIN: A large-scale dataset for compre- hensive instructional video analysis. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 1207-1216. Computer Vision Foundation / IEEE, 2019.","title":"COIN: A large-scale dataset for comprehensive instructional video analysis","authors":"Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, Jie Zhou","year":"2019","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Touvron et al., 2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023.","title":"Llama: Open and efficient foundation language models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vedantam et al., 2015] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.","title":"Cider: Consensusbased image description evaluation","authors":"Ramakrishna Vedantam, Lawrence Zitnick, Devi Parikh","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2022] Yi Wang, Kunchang Li, Yizhuo Li, Yi- nan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Intern- video: General video foundation models via generative and discriminative learning. CoRR, abs/2212.03191, 2022.","title":"Internvideo: General video foundation models via generative and discriminative learning","authors":"Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, Yu Qiao","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2023] Antoine Yang, Arsha Nagrani, Paul Hong- suck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a visual language model for dense video cap- tioning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 10714-10726. IEEE, 2023.","title":"Vid2seq: Large-scale pretraining of a visual language model for dense video captioning","authors":"Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, Cordelia Schmid","year":"2023","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ye et al., 2023] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large lan- guage models with multimodality. CoRR, abs/2304.14178, 2023.","title":"mplug-owl: Modularization empowers large language models with multimodality","authors":"Ye","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zala et al., 2023] Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Barlas Oguz, Yashar Mehdad, and Mohit Bansal. Hierarchical video-moment retrieval and step- captioning. In IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 23056-23065. IEEE, 2023.","title":"Hierarchical video-moment retrieval and stepcaptioning","authors":"Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Barlas Oguz, Yashar Mehdad, Mohit Bansal","year":"2023","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2023] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. In Yansong Feng and Els Lefever, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 -System Demonstrations, Singapore, Decem- ber 6-10, 2023, pages 543-553. Association for Computa- tional Linguistics, 2023.","title":"Video-llama: An instruction-tuned audio-visual language model for video understanding","authors":"Hang Zhang, Xin Li, Lidong Bing","year":"2023","venue":"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 -System Demonstrations","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhou et al., 2018] Luowei Zhou, Chenliang Xu, and Jason J. Corso. Towards automatic learning of procedures from web instructional videos. In Sheila A. McIlraith and Kil- ian Q. Weinberger, editors, Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence, (AAAI- 18), the 30th innovative Applications of Artificial Intelli- gence (IAAI-18), and the 8th AAAI Symposium on Educa- tional Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 7590- 7598. AAAI Press, 2018.","title":"Towards automatic learning of procedures from web instructional videos","authors":"Zhou","year":"2018","venue":"AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhukov et al., 2019] Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David F. Fouhey, Ivan Laptev, and Josef Sivic. Cross-task weakly supervised learning from instructional videos. In IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 3537-3545. Computer Vision Foundation / IEEE, 2019.","title":"Cross-task weakly supervised learning from instructional videos","authors":"Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David F Fouhey, Ivan Laptev, Josef Sivic","year":"2019","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}