{"type":"ScholarlyArticle","title":"Hallucination-Aware Prompt Optimization for Text-to-Video Synthesis","authors":[{"name":"Jiapeng Wang","slug":"jiapeng-wang","given":"Jiapeng","family":"Wang","affiliation":"South China University of Technology , Guangzhou , China"},{"name":"Chengyu Wang","slug":"chengyu-wang","given":"Chengyu","family":"Wang","affiliation":"Alibaba Cloud Computing , Hangzhou , China"},{"name":"Jun Huang","slug":"jun-huang","given":"Jun","family":"Huang","affiliation":"Alibaba Cloud Computing , Hangzhou , China"},{"name":"Lianwen Jin","slug":"lianwen-jin","given":"Lianwen","family":"Jin","affiliation":"South China University of Technology , Guangzhou , China"}],"abstract":"The rapid advancements in AI-generated content (AIGC) have led to extensive research and application of deep text-to-video (T2V) synthesis models, such as OpenAI's Sora. These models typically rely on high-quality prompt-video pairs and detailed text prompts for model training in order to produce high-quality videos. To boost the effectiveness of Sora-like T2V models, we introduce VidPrompter, an innovative large multi-modal model supporting T2V applications with three key functionalities: (1) generating detailed prompts from raw videos, (2) enhancing prompts from videos grounded with short descriptions, and (3) refining simple user-provided prompts to elevate T2V video quality. We train VidPrompter using a hybrid multi-task paradigm and propose the hallucination-aware direct preference optimization (HDPO) technique to improve the multi-modal, multi-task prompt optimization process. Experiments on various tasks show our method surpasses strong baselines and other competitors.","doi":"10.24963/ijcai.2025/1133","url":"https://ijcai.org/proceedings/2025/1133","pdf":"https://ijcai.org/proceedings/2025/1133.pdf","markdown":"https://ijcai.org/proceedings/2025/1133.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/1133","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"AI, Arts & Creativity","track":"AI, Arts & Creativity","number":1133,"pages":{"first":10198,"last":10206},"keywords":["Methods and resources: AI systems for collaboration and co-creation","Methods and resources: Machine learning, deep learning, neural models, reinforcement learning","Application domains: Images, movies and visual arts"],"videos":[],"references":[{"position":1,"raw":"[Azar et al., 2024] Mohammad Gheshlaghi Azar, Zhao- han Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theo- retical paradigm to understand learning from human pref- erences. In AISTATS, pages 4447-4455, 2024.","title":"A general theoretical paradigm to understand learning from human preferences","authors":"Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, Daniele Calandriello","year":"2024","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bird et al., 2009] Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: Ana- lyzing text with the natural language toolkit. \"O'Reilly Media, Inc.\", 2009.","title":"Natural language processing with Python: Analyzing text with the natural language toolkit","authors":"Steven Bird, Ewan Klein, Edward Loper","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bradley and Terry, 1952] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324- 345, 1952.","title":"Rank analysis of incomplete block designs: I. the method of paired comparisons","authors":"Terry ; Bradley, Allan Ralph, Milton E Bradley, Terry","year":"1952","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brooks et al., 2024] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation mod- els as world simulators. 2024.","title":"Video generation models as world simulators","authors":"Tim Brooks, Bill Peebles, Connor Holmes, Will Depue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, Aditya Ramesh","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2024a] Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. VideoCrafter2: Overcoming data limitations for high-quality video diffusion models. In CVPR, pages 7310-7320, 2024.","title":"VideoCrafter2: Overcoming data limitations for high-quality video diffusion models","authors":"Chen","year":"2024","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2024b] Huayu Chen, Guande He, Hang Su, and Jun Zhu. Noise contrastive alignment of language models with explicit rewards. NeurIPS, 2024.","title":"Noise contrastive alignment of language models with explicit rewards","authors":"Chen","year":"2024","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2024c] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Zhenyu Tang, Li Yuan, et al. ShareGPT4Video: Improving video understanding and generation with better captions. NeurIPS, 37:19472-19495, 2024.","title":"ShareGPT4Video: Improving video understanding and generation with better captions","authors":"Chen","year":"2024","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2024d] Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70M: Captioning 70M videos with multiple cross-modality teachers. In CVPR, pages 13320-13331, 2024.","title":"Panda-70M: Captioning 70M videos with multiple cross-modality teachers","authors":"Chen","year":"2024","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Christiano et al., 2017] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. NeurIPS, 30, 2017.","title":"Deep reinforcement learning from human preferences","authors":"Jan Paul F Christiano, Tom Leike, Miljan Brown, Shane Martic, Dario Legg, Amodei","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Deng et al., 2024] Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, James Zou, Kai-Wei Chang, and Wei Wang. Enhancing large vision language models with self-training on image comprehension. NeurIPS, 37:131369-131397, 2024.","title":"Enhancing large vision language models with self-training on image comprehension","authors":"Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, James Zou, Kai-Wei Chang, Wei Wang","year":"2024","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ethayarajh et al., 2024] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment as prospect theoretic optimization. In ICML, 2024.","title":"Model alignment as prospect theoretic optimization","authors":"Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, Douwe Kiela","year":"2024","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Fu et al., 2024] Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video- MME: The first-ever comprehensive evaluation bench- mark of multi-modal LLMs in video analysis. arXiv preprint arXiv:2405.21075, 2024.","title":"Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis","authors":"Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Guan et al., 2024] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusion- Bench: An advanced diagnostic suite for entangled lan- guage hallucination and visual illusion in large vision- language models. In CVPR, pages 14375-14385, 2024.","title":"Hallusion-Bench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large visionlanguage models","authors":"Fuxiao Tianrui Guan, Xiyang Liu, Ruiqi Wu, Zongxia Xian, Xiaoyu Li, Xijun Liu, Lichang Wang, Furong Chen, Yaser Huang, Yacoob","year":"2024","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Guo et al., 2024] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. AnimateDiff: Animate your personalized text-to-image diffusion models without specific tuning. In ICLR, 2024.","title":"AnimateDiff: Animate your personalized text-to-image diffusion models without specific tuning","authors":"Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, Bo Dai","year":"2024","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Honnibal et al., 2020] Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial-strength natural language processing in Python. 2020.","title":"spaCy: Industrial-strength natural language processing in Python","authors":"Matthew Honnibal, Ines Montani, Sofie Van Landeghem, Adriane Boyd","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Huang et al., 2024a] Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. VTimeLLM: Empower LLM to grasp video moments. In CVPR, pages 14271- 14280, 2024.","title":"VTimeLLM: Empower LLM to grasp video moments","authors":"Huang","year":"2024","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Huang et al., 2024b] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. OPERA: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In CVPR, pages 13418-13427, 2024.","title":"OPERA: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation","authors":"Huang","year":"2024","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Huang et al., 2024c] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. VBench: Comprehensive benchmark suite for video gen- erative models. In CVPR, pages 21807-21818, 2024.","title":"VBench: Comprehensive benchmark suite for video generative models","authors":"Huang","year":"2024","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Jiang et al., 2024] Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large lan- guage model. In CVPR, pages 27036-27046, 2024.","title":"Hallucination augmented contrastive learning for multimodal large language model","authors":"Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, Shikun Zhang","year":"2024","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2023a] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large lan- guage models. In ICML, 2023.","title":"BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models","authors":"Li","year":"2023","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li et al., 2023b] Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distil- lation for large visual language models. arXiv preprint arXiv:2312.10665, 2023.","title":"Silkie: Preference distillation for large visual language models","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Li et al., 2023c] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In EMNLP, pages 292-305, 2023.","title":"Evaluating object hallucination in large vision-language models","authors":"Li","year":"2023","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Li et al., 2024] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. MVBench: A comprehensive multi-modal video understanding benchmark. In CVPR, pages 22195- 22206, 2024.","title":"MVBench: A comprehensive multi-modal video understanding benchmark","authors":"Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo","year":"2024","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Lin et al., 2024] Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. VILA: On pre-training for visual language models. In CVPR, pages 26689-26699, 2024.","title":"VILA: On pre-training for visual language models","authors":"Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, Song Han","year":"2024","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Liu et al., 2023] Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference op- timization. In ICLR, 2023.","title":"Statistical rejection sampling improves preference optimization","authors":"Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, Jialu Liu","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Meng et al., 2024] Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with a reference-free reward. NeurIPS, 37:124198-124235, 2024.","title":"SimPO: Simple preference optimization with a reference-free reward","authors":"Yu Meng, Mengzhou Xia, Danqi Chen","year":"2024","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ouyang et al., 2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instruc- tions with human feedback. NeurIPS, 35:27730-27744, 2022.","title":"Training language models to follow instructions with human feedback","authors":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervi- sion. In ICML, pages 8748-8763, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Rafailov et al., 2023] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your lan- guage model is secretly a reward model. NeurIPS, 36, 2023.","title":"Direct preference optimization: Your language model is secretly a reward model","authors":"Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, Chelsea Finn","year":"2023","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Ren et al., 2024] Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. TimeChat: A time-sensitive multi- modal large language model for long video understanding. In CVPR, pages 14313-14323, 2024.","title":"TimeChat: A time-sensitive multimodal large language model for long video understanding","authors":"Ren","year":"2024","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Sun et al., 2024] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang Yan Gui, Yu Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually aug- mented RLHF. In Findings of ACL, pages 13088-13110, 2024.","title":"Aligning large multimodal models with factually augmented RLHF","authors":"Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Yan Liang, Yu Xiong Gui, Yiming Wang, Yang","year":"2024","venue":"Findings of ACL","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Touvron et al., 2023a] Hugo Touvron, Thibaut Lavril, Gau- tier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Ham- bro, Faisal Azhar, et al. Llama: Open and efficient founda- tion language models. arXiv preprint arXiv:2302.13971, 2023.","title":"Llama: Open and efficient foundation language models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Touvron et al., 2023b] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.","title":"Llama 2: Open foundation and fine-tuned chat models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wang et al., 2024a] Fei Wang, Wenxuan Zhou, James Y Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. mDPO: Conditional preference optimization for multimodal large language models. In EMNLP, pages 8078-8088, 2024.","title":"mDPO: Conditional preference optimization for multimodal large language models","authors":"Wang","year":"2024","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wang et al., 2024b] Jiapeng Wang, Chengyu Wang, Kunzhe Huang, Jun Huang, and Lianwen Jin. VideoCLIP-XL: Advancing long description understanding for video CLIP models. In EMNLP, pages 16061-16075, 2024.","title":"VideoCLIP-XL: Advancing long description understanding for video CLIP models","authors":"Wang","year":"2024","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wu et al., 2024] Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. In ICML Workshops, 2024.","title":"Self-play preference optimization for language model alignment","authors":"Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, Quanquan Gu","year":"2024","venue":"ICML Workshops","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Xu et al., 2023] Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682, 2023.","title":"Some things are more cringe than others: Preference optimization with the pairwise cringe loss","authors":"Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, Jason Weston","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Xu et al., 2024] Jiaqi Xu, Xinyi Zou, Kunzhe Huang, Yunkuo Chen, Bo Liu, MengLi Cheng, Xing Shi, and Jun Huang. EasyAnimate: A high-performance long video generation method based on Transformer architec- ture. arXiv preprint arXiv:2405.18991, 2024.","title":"EasyAnimate: A high-performance long video generation method based on Transformer architecture","authors":"Jiaqi Xu, Xinyi Zou, Kunzhe Huang, Yunkuo Chen, Bo Liu, Mengli Cheng, Xing Shi, Jun Huang","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Yu et al., 2024a] Tianyu Yu, Yuan Yao, Haoye Zhang, Tai- wen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. RLHF-V: Towards trustworthy MLLMs via behavior alignment from fine- grained correctional human feedback. In CVPR, pages 13807-13816, 2024.","title":"RLHF-V: Towards trustworthy MLLMs via behavior alignment from finegrained correctional human feedback","authors":"Yu","year":"2024","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Yu et al., 2024b] Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Tai- wen He, Zhiyuan Liu, Tat-Seng Chua, et al. RLAIF- V: Aligning MLLMs through open-source AI feed- back for super GPT-4V trustworthiness. arXiv preprint arXiv:2405.17220, 2024.","title":"RLAIF-V: Aligning MLLMs through open-source AI feedback for super GPT-4V trustworthiness","authors":"Yu","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Yuan et al., 2024] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Ja- son Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024.","title":"Self-rewarding language models","authors":"Yuan","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Zhang et al., 2023] Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An instruction-tuned audio-visual lan- guage model for video understanding. In EMNLP, pages 543-553, 2023.","title":"Video-LLaMA: An instruction-tuned audio-visual language model for video understanding","authors":"Hang Zhang, Xin Li, Lidong Bing","year":"2023","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Zhang et al., 2024] Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-CLIP: Un- locking the long-text capability of CLIP. In ECCV, pages 310-325, 2024.","title":"Long-CLIP: Unlocking the long-text capability of CLIP","authors":"Zhang","year":"2024","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Zheng et al., 2024] Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-Sora: Democratiz- ing efficient video production for all, 2024.","title":"Open-Sora: Democratizing efficient video production for all","authors":"Zheng","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Zhou et al., 2024] Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modal- ities in vision large language models via preference fine- tuning. In ICLR Workshops, 2024.","title":"Aligning modalities in vision large language models via preference finetuning","authors":"Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, Huaxiu Yao","year":"2024","venue":"ICLR Workshops","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}