{"type":"ScholarlyArticle","title":"FasterVD: On Acceleration of Video Diffusion Models","authors":[{"name":"Pinrui Yu","slug":"pinrui-yu","given":"Pinrui","family":"Yu","affiliation":"Northeastern University"},{"name":"Dan Luo","slug":"dan-luo","given":"Dan","family":"Luo","affiliation":"Northeastern University"},{"name":"Timothy Rupprecht","slug":"timothy-rupprecht","given":"Timothy","family":"Rupprecht","affiliation":"Northeastern University"},{"name":"Lei Lu","slug":"lei-lu","given":"Lei","family":"Lu","affiliation":"Northeastern University"},{"name":"Zhenglun Kong","slug":"zhenglun-kong","given":"Zhenglun","family":"Kong","affiliation":"Northeastern University"},{"name":"Pu Zhao","slug":"pu-zhao","given":"Pu","family":"Zhao","affiliation":"Northeastern University"},{"name":"Yanyu Li","slug":"yanyu-li","given":"Yanyu","family":"Li","affiliation":"Northeastern University"},{"name":"Octavia Camps","slug":"octavia-camps","given":"Octavia","family":"Camps","affiliation":"Northeastern University"},{"name":"Xue Lin","slug":"xue-lin","given":"Xue","family":"Lin","affiliation":"Northeastern University"},{"name":"Yanzhi Wang","slug":"yanzhi-wang","given":"Yanzhi","family":"Wang","affiliation":"Northeastern University"}],"abstract":"Equipped with Denoising Diffusion Probabilistic Models, video content generation has gained significant research interest recently. However, diffusion pipelines call for intensive computation and model storage, which poses challenges for their wide and efficient deployment. In this work, we address this issue by integrating LCM-LoRA to reduce the denoising steps and escalating the video generation process by frame skipping and interpolation. Our framework achieves an approximately 10× inference acceleration for high-quality realistic video generation on commonly available GPUs.","doi":"10.24963/ijcai.2024/1044","url":"https://ijcai.org/proceedings/2024/1044","pdf":"https://ijcai.org/proceedings/2024/1044.pdf","markdown":"https://ijcai.org/proceedings/2024/1044.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/1044","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Demo Track","track":"Demo Track","number":1044,"pages":{"first":8838,"last":8842},"keywords":["Machine Learning: ML: Optimization","Machine Learning: ML: Applications"],"videos":[],"references":[{"position":1,"raw":"[Bai et al., 2019] Qiyu Bai, Qi Dan, Zhe Mu, and Maokun Yang. A systematic review of emoji: Current research and future perspectives. Frontiers in psychology, 10:2221, 2019.","title":"A systematic review of emoji: Current research and future perspectives","authors":"Bai","year":"2019","venue":"Frontiers in psychology","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blattmann et al., 2023a] Andreas Blattmann, Tim Dock- horn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023.","title":"Stable video diffusion: Scaling latent video diffusion models to large datasets","authors":"Blattmann","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blattmann et al., 2023b] Andreas Blattmann, Robin Rom- bach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563- 22575, 2023.","title":"Align your latents: High-resolution video synthesis with latent diffusion models","authors":"Blattmann","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Girdhar et al., 2023] Rohit Girdhar, Mannat Singh, An- drew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Is- han Misra. Emu video: Factorizing text-to-video gen- eration by explicit image conditioning. arXiv preprint arXiv:2311.10709, 2023.","title":"Emu video: Factorizing text-to-video generation by explicit image conditioning","authors":"Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, Ishan Misra","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Guo et al., 2023] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. An- imatediff: Animate your personalized text-to-image dif- fusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023.","title":"Animatediff: Animate your personalized text-to-image diffusion models without specific tuning","authors":"Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, Bo Dai","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2022] Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion mod- els for high-fidelity long video generation. 2022.","title":"Latent video diffusion models for high-fidelity long video generation","authors":"He","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ho et al., 2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.","title":"Denoising diffusion probabilistic models","authors":"Jonathan Ho, Ajay Jain, Pieter Abbeel","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2023] Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character anima- tion. arXiv preprint arXiv:2311.17117, 2023.","title":"Animate anyone: Consistent and controllable image-to-video synthesis for character animation","authors":"Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, Liefeng Bo","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Karras et al., 2022] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364, 2022.","title":"Elucidating the design space of diffusion-based generative models","authors":"Tero Karras, Miika Aittala, Timo Aila, Samuli Laine","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kim et al., 2023] Bo-Kyeong Kim, Hyoung-Kyu Song, Thibault Castells, and Shinkook Choi. On architectural compression of text-to-image diffusion models. arXiv preprint arXiv:2305.15798, 2023.","title":"On architectural compression of text-to-image diffusion models","authors":"Kim","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2024] Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffusion model on mobile devices within two seconds. Advances in Neural Information Processing Systems, 36, 2024.","title":"Snapfusion: Text-to-image diffusion model on mobile devices within two seconds","authors":"Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, Jian Ren","year":"2024","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin et al., 2022] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. arXiv preprint arXiv:2211.10440, 2022.","title":"Magic3d: High-resolution text-to-3d content creation","authors":"Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, Tsung-Yi Lin","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Luo et al., 2023a] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Syn- thesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023.","title":"Latent consistency models: Synthesizing high-resolution images with few-step inference","authors":"Luo","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Luo et al., 2023b] Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao. Lcm-lora: A univer- sal stable-diffusion acceleration module. arXiv preprint arXiv:2311.05556, 2023.","title":"Lcm-lora: A universal stable-diffusion acceleration module","authors":"Luo","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Luo et al., 2023c] Z. Luo, D. Chen, Y. Zhang, Y. Huang, L. Wang, Y. Shen, D. Zhao, J. Zhou, and T. Tan. No- tice of removal: Videofusion: Decomposed diffusion mod- els for high-quality video generation. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10209-10218, Los Alamitos, CA, USA, jun 2023. IEEE Computer Society.","title":"Notice of removal: Videofusion: Decomposed diffusion models for high-quality video generation","authors":"Luo","year":"2023","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Reda et al., 2022] Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Cur- less. Film: Frame interpolation for large motion. In Eu- ropean Conference on Computer Vision, pages 250-266. Springer, 2022.","title":"Film: Frame interpolation for large motion","authors":"Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, Brian Curless","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rombach et al., 2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.","title":"High-resolution image synthesis with latent diffusion models","authors":"Rombach","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICAI, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"MICAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Salimans and Ho, 2022] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion mod- els. arXiv preprint arXiv:2202.00512, 2022.","title":"Progressive distillation for fast sampling of diffusion models","authors":"Ho Salimans, Tim Salimans, Jonathan Ho","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sohl-Dickstein et al., 2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermody- namics. In International Conference on Machine Learn- ing, pages 2256-2265. PMLR, 2015.","title":"Deep unsupervised learning using nonequilibrium thermodynamics","authors":"Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, Surya Ganguli","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Song et al., 2020a] Jiaming Song, Chenlin Meng, and Ste- fano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.","title":"Jiaming Song, Chenlin Meng, and Stefano Ermon","authors":"Song","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Song et al., 2020b] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Er- mon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.","title":"Score-based generative modeling through stochastic differential equations","authors":"Song","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu et al., 2023a] Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. arXiv preprint arXiv:2311.16498, 2023.","title":"Magicanimate: Temporally consistent human image animation using diffusion model","authors":"Xu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2023b] Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. 2023.","title":"Magicanimate: Temporally consistent human image animation using diffusion model","authors":"Xu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhao et al., 2023] Yang Zhao, Yanwu Xu, Zhisheng Xiao, and Tingbo Hou. Mobilediffusion: Subsecond text-to- image generation on mobile devices. arXiv preprint arXiv:2311.16567, 2023.","title":"Mobilediffusion: Subsecond text-toimage generation on mobile devices","authors":"Yang Zhao, Yanwu Xu, Zhisheng Xiao, Tingbo Hou","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}