{"type":"ScholarlyArticle","title":"Boosting Diffusion Models with an Adaptive Momentum Sampler","authors":[{"name":"Xiyu Wang","slug":"xiyu-wang","given":"Xiyu","family":"Wang","affiliation":"School of Computer Science , Faculty of Engineering , The University of Sydney , Australia"},{"name":"Anh-Dung Dinh","slug":"anh-dung-dinh","given":"Anh-Dung","family":"Dinh","affiliation":"School of Computer Science , Faculty of Engineering , The University of Sydney , Australia"},{"name":"Daochang Liu","slug":"daochang-liu","given":"Daochang","family":"Liu","affiliation":"School of Computer Science , Faculty of Engineering , The University of Sydney , Australia"},{"name":"Chang Xu","slug":"chang-xu","given":"Chang","family":"Xu","affiliation":"School of Computer Science , Faculty of Engineering , The University of Sydney , Australia"}],"abstract":"Diffusion probabilistic models (DPMs) have been shown to generate high-quality images without the need for delicate adversarial training. The sampling process of DPMs is mathematically similar to Stochastic Gradient Descent (SGD), with both being iteratively updated with a function increment. Building on this, we present a novel reverse sampler for DPMs in this paper, drawing inspiration from the widely-used Adam optimizer. Our proposed sampler can be readily applied to a pre-trained diffusion model, utilizing momentum mechanisms and adaptive updating to enhance the generated image's quality. By effectively reusing update directions from early steps, our proposed sampler achieves a better balance between high-level semantics and low-level details. Additionally, this sampler is flexible and can be easily integrated into pre-trained DPMs regardless of the sampler used during training. Our experimental results on multiple benchmarks demonstrate that our proposed reverse sampler yields remarkable improvements over different baselines.","doi":"10.24963/ijcai.2024/157","url":"https://ijcai.org/proceedings/2024/157","pdf":"https://ijcai.org/proceedings/2024/0157.pdf","markdown":"https://ijcai.org/proceedings/2024/157.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/157","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":157,"pages":{"first":1416,"last":1424},"keywords":["Computer Vision: CV: Image and video synthesis and generation"],"videos":[],"references":[{"position":1,"raw":"[Bao et al., 2022] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-DPM: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. arXiv preprint arXiv:2201.06503, 2022.","title":"Analytic-DPM: an analytic estimate of the optimal reverse variance in diffusion probabilistic models","authors":"Fan Bao, Chongxuan Li, Jun Zhu, Bo Zhang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Borji, 2019] Ali Borji. Pros and Cons of GAN evaluation measures. Computer Vision and Image Understanding, 179:41-65, 2019.","title":"Pros and Cons of GAN evaluation measures","authors":"Ali Borji","year":"2019","venue":"Computer Vision and Image Understanding","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brock et al., 2018] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018.","title":"Large scale gan training for high fidelity natural image synthesis","authors":"Andrew Brock, Jeff Donahue, Karen Simonyan","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bruce et al., 2024] Jake Bruce, Michael Dennis, Ashley Ed- wards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. arXiv preprint arXiv:2402.15391, 2024.","title":"Genie: Generative interactive environments","authors":"Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dhariwal and Nichol, 2021] Prafulla Dhariwal and Alexan- der Nichol. Diffusion models beat gans on image synthe- sis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.","title":"Diffusion models beat gans on image synthesis","authors":"Nichol Dhariwal","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dinh et al., 2023] Anh-Dung Dinh, Daochang Liu, and Chang Xu. Pixelasparam: A gradient view on diffusion sampling with guidance. In International Conference on Machine Learning, pages 8120-8137. PMLR, 2023.","title":"Pixelasparam: A gradient view on diffusion sampling with guidance","authors":"Anh-Dung Dinh, Daochang Liu, Chang Xu","year":"2023","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dinh et al., 2024] Anh-Dung Dinh, Daochang Liu, and Chang Xu. Rethinking conditional diffusion sampling with progressive guidance. Advances in Neural Informa- tion Processing Systems, 36, 2024.","title":"Rethinking conditional diffusion sampling with progressive guidance","authors":"Anh-Dung Dinh, Daochang Liu, Chang Xu","year":"2024","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dung and Binh, 2022] Dinh Anh Dung and Huynh Thi Thanh Binh. Gdegan: Graphical discriminative embedding gan for tabular data. In 2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA), pages 1-11. IEEE, 2022.","title":"Gdegan: Graphical discriminative embedding gan for tabular data","authors":"Anh Dinh, Dung, Huynh Thi, Thanh Binh","year":"2022","venue":"IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gong et al., 2019] Mingming Gong, Yanwu Xu, Chunyuan Li, Kun Zhang, and Kayhan Batmanghelich. Twin auxilary classifiers gan. Advances in neural information processing systems, 32, 2019.","title":"Twin auxilary classifiers gan","authors":"Mingming Gong, Yanwu Xu, Chunyuan Li, Kun Zhang, Kayhan Batmanghelich","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, vol- ume 27.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Curran Associates, Inc., 2014.","title":null,"authors":"Curran Associates, Inc","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Guo et al., 2020] Tianyu Guo, Chang Xu, Jiajun Huang, Yunhe Wang, Boxin Shi, Chao Xu, and Dacheng Tao. On positive-unlabeled classification in gan. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 8385-8393, 2020.","title":"On positive-unlabeled classification in gan","authors":"Tianyu Guo, Chang Xu, Jiajun Huang, Yunhe Wang, Boxin Shi, Chao Xu, Dacheng Tao","year":"2020","venue":"Proceedings of the ieee/cvf conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hinton and Salakhutdinov, 2006] Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504-507, 2006.","title":"Reducing the dimensionality of data with neural networks","authors":"Geoffrey E Hinton, Ruslan R Salakhutdinov","year":"2006","venue":"science","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ho et al., 2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.","title":"Denoising diffusion probabilistic models","authors":"Jonathan Ho, Ajay Jain, Pieter Abbeel","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Karras et al., 2017] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.","title":"Progressive growing of gans for improved quality, stability, and variation","authors":"Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Karras et al., 2020] Tero Karras, Samuli Laine, Miika Ait- tala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. An- alyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 8110-8119, 2020.","title":"Analyzing and improving the image quality of stylegan","authors":"Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, Timo Aila","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Karras et al., 2022] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364, 2022.","title":"Elucidating the design space of diffusion-based generative models","authors":"Tero Karras, Miika Aittala, Timo Aila, Samuli Laine","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny im- ages. 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2018] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(2018):11, 2018.","title":"Large-scale celebfaces attributes (celeba) dataset","authors":"Ziwei Liu, Ping Luo, Xiaogang Wang, Xiaoou Tang","year":"2018","venue":"Retrieved","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2023] Daochang Liu, Qiyue Li, Anh-Dung Dinh, Tingting Jiang, Mubarak Shah, and Chang Xu. Diffusion action segmentation. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 10139- 10149, 2023.","title":"Diffusion action segmentation","authors":"Liu","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Lucic et al., 2018] Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are GANs created equal? a large-scale study. Advances in neural information processing systems, 31, 2018.","title":"Are GANs created equal? a large-scale study","authors":"Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, Olivier Bousquet","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Mukkamala and Hein, 2017] Mahesh Chandra Mukkamala and Matthias Hein. Variants of RMSProp and Adagrad with logarithmic regret bounds. In International con- ference on machine learning, pages 2545-2553.","title":"Variants of RMSProp and Adagrad with logarithmic regret bounds","authors":"Hein Mukkamala, Mahesh Chandra Mukkamala, Matthias Hein","year":"2017","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"PMLR, 2017.","title":null,"authors":"Pmlr","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Nichol and Dhariwal, 2021] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion proba- bilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.","title":"Improved denoising diffusion probabilistic models","authors":"Nichol, Dhariwal, Alexander Quinn, Nichol, Prafulla Dhariwal","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Odena et al., 2017] Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier gans. In International conference on ma- chine learning, pages 2642-2651. PMLR, 2017.","title":"Conditional image synthesis with auxiliary classifier gans","authors":"Odena","year":"2017","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Rombach et al., 2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.","title":"High-resolution image synthesis with latent diffusion models","authors":"Rombach","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234- 241. Springer, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Salimans et al., 2016] Tim Salimans, Ian Goodfellow, Woj- ciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. Advances in neu- ral information processing systems, 29, 2016.","title":"Improved techniques for training GANs","authors":"Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Shao et al., 2021] Huajie Shao, Zhisheng Xiao, Shuochao Yao, Dachun Sun, Aston Zhang, Shengzhong Liu, Tianshi Wang, Jinyang Li, and Tarek Abdelzaher. ControlVAE: Tuning, analytical properties, and performance analysis. IEEE transactions on pattern analysis and machine intel- ligence, 2021.","title":"ControlVAE: Tuning, analytical properties, and performance analysis","authors":"Huajie Shao, Zhisheng Xiao, Shuochao Yao, Dachun Sun, Aston Zhang, Shengzhong Liu, Tianshi Wang, Jinyang Li, Tarek Abdelzaher","year":"2021","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Song and Ermon, 2019] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.","title":"Generative modeling by estimating gradients of the data distribution","authors":"Yang Song, Stefano Ermon","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Song and Ermon, 2020] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing sys- tems, 33:12438-12448, 2020.","title":"Improved techniques for training score-based generative models","authors":"Ermon ; Song, Yang Song, Stefano Ermon","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Song et al., 2020a] Jiaming Song, Chenlin Meng, and Ste- fano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.","title":"Jiaming Song, Chenlin Meng, and Stefano Ermon","authors":"Song","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Song et al., 2020b] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Er- mon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.","title":"Score-based generative modeling through stochastic differential equations","authors":"Song","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wang et al., 2023] Yunke Wang, Xiyu Wang, Anh-Dung Dinh, Bo Du, and Charles Xu. Learning to schedule in diffusion probabilistic models. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2478-2488, 2023.","title":"Learning to schedule in diffusion probabilistic models","authors":"Wang","year":"2023","venue":"Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wu et al., 2023] Zike Wu, Pan Zhou, Kenji Kawaguchi, and Hanwang Zhang. Momentum-accelerated diffusion pro- cess for faster training and sampling. 2023.","title":"Momentum-accelerated diffusion process for faster training and sampling","authors":"Zike Wu, Pan Zhou, Kenji Kawaguchi, Hanwang Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Yu et al., 2015] Fisher Yu, Ari Seff, Yinda Zhang, Shu- ran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.","title":"LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop","authors":"Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, Jianxiong Xiao","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Yu et al., 2022] Yuncong Yu, Dylan Kruyff, Jiao Jiao, Tim Becker, and Michael Behrisch. Pseudo: Interactive pattern search in multivariate time series with locality-sensitive hashing and relevance feedback. IEEE Transactions on Visualization and Computer Graphics, 29(1):33-42, 2022.","title":"Pseudo: Interactive pattern search in multivariate time series with locality-sensitive hashing and relevance feedback","authors":"Yu","year":"2022","venue":"IEEE Transactions on Visualization and Computer Graphics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}