{"type":"ScholarlyArticle","title":"ToDo: Token Downsampling for Efficient Generation of High-Resolution Images","authors":[{"name":"Ethan Smith","slug":"ethan-smith","given":"Ethan","family":"Smith","affiliation":"Leonardo AI Research Lab , North Sydney , NSW , Australia"},{"name":"Nayan Saxena","slug":"nayan-saxena","given":"Nayan","family":"Saxena","affiliation":"Leonardo AI Research Lab , North Sydney , NSW , Australia"},{"name":"Aninda Saha","slug":"aninda-saha","given":"Aninda","family":"Saha","affiliation":"Leonardo AI Research Lab , North Sydney , NSW , Australia"}],"abstract":"Attention has been a crucial component in the success of image diffusion models, however, their quadratic computational complexity limits the sizes of images we can process within reasonable time and memory constraints. This paper investigates the importance of dense attention in generative image models, which often contain redundant features, making them suitable for sparser attention mechanisms. We propose a novel training-free method ToDo that relies on token downsampling of key and value tokens to accelerate Stable Diffusion inference by up to 2x for common sizes and up to 4.5x or more for high resolutions like 2048 × 2048. We demonstrate that our approach outperforms previous methods in balancing efficient throughput and fidelity.","doi":"10.24963/ijcai.2024/1036","url":"https://ijcai.org/proceedings/2024/1036","pdf":"https://ijcai.org/proceedings/2024/1036.pdf","markdown":"https://ijcai.org/proceedings/2024/1036.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/1036","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Demo Track","track":"Demo Track","number":1036,"pages":{"first":8801,"last":8804},"keywords":["Machine Learning: ML: Attention models","Computer Vision: CV: Neural generative models, auto encoders, GANs"],"videos":[],"references":[{"position":1,"raw":"[Bankman, 2008] Isaac Bankman. Handbook of medical im- age processing and analysis. Elsevier, 2008.","title":"Isaac Bankman. Handbook of medical image processing and analysis","authors":"Bankman","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bolya and Hoffman, 2023] Daniel Bolya and Judy Hoff- man. Token merging for fast stable diffusion. CVPR Workshop on Efficient Deep Learning for Computer Vi- sion, 2023.","title":"Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion","authors":"Hoffman Bolya","year":"2023","venue":"CVPR Workshop on Efficient Deep Learning for Computer Vision","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bolya et al., 2023] Daniel Bolya, Cheng-Yang Fu, Xiao- liang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. In International Conference on Learning Representations, 2023.","title":"Token merging: Your ViT but faster","authors":"Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, Judy Hoffman","year":"2023","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2020] Dengsheng Chen, Jun Li, and Kai Xu. Arelu: Attention-based rectified linear unit, 2020.","title":"Arelu: Attention-based rectified linear unit","authors":"Chen","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Choromanski et al., 2022] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Col- well, and Adrian Weller. Rethinking attention with per- formers, 2022.","title":"Rethinking attention with performers","authors":"Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, Adrian Weller","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dao et al., 2022] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, 2022.","title":"FlashAttention: Fast and memory-efficient exact attention with IO-awareness","authors":"Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, Christopher Ré","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gonzalez, 2009] Rafael C Gonzalez. Digital image process- ing. Pearson education india, 2009.","title":"Digital image processing","authors":"C Rafael, Gonzalez","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hassani et al., 2023] Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer, 2023. [Katharopoulos et al., 2020]","title":"Neighborhood attention transformer","authors":"Ali Hassani, Steven Walton, Jiachen Li, Shen Li, Humphrey Shi","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention, 2020.","title":"Transformers are rnns: Fast autoregressive transformers with linear attention","authors":"Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, Franc ¸ois, Fleuret","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Khan et al., 2022] Salman Khan, Muzammal Naseer, Mu- nawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM Computing Surveys, 54(10s):1-41, January 2022.","title":"Transformers in vision: A survey","authors":"Salman Khan, Muzammal Naseer, Munawar Hayat, Fahad Syed Waqas Zamir, Mubarak Shahbaz Khan, Shah","year":"2022","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Luo et al., 2023] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Syn- thesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023.","title":"Latent consistency models: Synthesizing high-resolution images with few-step inference","authors":"Luo","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Rombach et al., 2021] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021. [Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234- 241. Springer, 2015.","title":"U-net: Convolutional networks for biomedical image segmentation","authors":"Rombach","year":"2015","venue":"Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Song and Ermon, 2019] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.","title":"Generative modeling by estimating gradients of the data distribution","authors":"Ermon ; Song, Yang Song, Stefano Ermon","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Song et al., 2020] Jiaming Song, Chenlin Meng, and Ste- fano Ermon. Denoising diffusion implicit models. arXiv:2010.02502, October 2020.","title":"Denoising diffusion implicit models","authors":"Jiaming Song, Chenlin Meng, Stefano Ermon","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Team, 2024] Huggingface Diffusers Team. Speed up infer- ence, 2024.","title":"Huggingface Diffusers Team. Speed up inference","authors":null,"year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Vaswani et al., 2023] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Yang et al., 2021] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers. CoRR, abs/2107.00641, 2021.","title":"Focal self-attention for local-global interactions in vision transformers","authors":"Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, Jianfeng Gao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Zhuang et al., 2023] Bohan Zhuang, Jing Liu, Zizheng Pan, Haoyu He, Yuetian Weng, and Chunhua Shen. A sur- vey on efficient training of transformers. arXiv preprint arXiv:2302.01107, 2023.","title":"A survey on efficient training of transformers","authors":"Bohan Zhuang, Jing Liu, Zizheng Pan, Haoyu He, Yuetian Weng, Chunhua Shen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2023/764"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}