{"type":"ScholarlyArticle","title":"Interactive Visual Learning for Stable Diffusion","authors":[{"name":"Seongmin Lee","slug":"seongmin-lee","given":"Seongmin","family":"Lee","affiliation":"Georgia Tech"},{"name":"Benjamin Hoover","slug":"benjamin-hoover","given":"Benjamin","family":"Hoover","affiliation":"Georgia Tech; IBM Research"},{"name":"Hendrik Strobelt","slug":"hendrik-strobelt","given":"Hendrik","family":"Strobelt","affiliation":"IBM Research"},{"name":"Zijie J. Wang","slug":"zijie-j-wang","given":"Zijie J.","family":"Wang","affiliation":"Georgia Tech"},{"name":"ShengYun Peng","slug":"shengyun-peng","given":"ShengYun","family":"Peng","affiliation":"Georgia Tech"},{"name":"Austin Wright","slug":"austin-wright","given":"Austin","family":"Wright","affiliation":"Georgia Tech"},{"name":"Kevin Li","slug":"kevin-li","given":"Kevin","family":"Li","affiliation":"Georgia Tech"},{"name":"Haekyu Park","slug":"haekyu-park","given":"Haekyu","family":"Park","affiliation":"Georgia Tech"},{"name":"Haoyang Yang","slug":"haoyang-yang","given":"Haoyang","family":"Yang","affiliation":"Georgia Tech"},{"name":"Duen Horng Chau","slug":"duen-horng-chau","given":"Duen Horng","family":"Chau","affiliation":"Georgia Tech"}],"abstract":"Diffusion-based generative models’ impressive ability to create convincing images has garnered global attention. However, their complex internal structures and operations often pose challenges for non-experts to grasp. We introduce Diffusion Explainer, the first interactive visualization tool designed to elucidate how Stable Diffusion transforms text prompts into images. It tightly integrates a visual overview of Stable Diffusion’s complex components with detailed explanations of their underlying operations. This integration enables users to fluidly transition between multiple levels of abstraction through animations and interactive elements. Offering real-time hands-on experience, Diffusion Explainer allows users to adjust Stable Diffusion’s hyperparameters and prompts without the need for installation or specialized hardware. Accessible via users’ web browsers, Diffusion Explainer is making significant strides in democratizing AI education, fostering broader public access. More than 7,200 users spanning 113 countries have used our open-sourced tool at https://poloclub.github.io/diffusion-explainer/. A video demo is available at https://youtu.be/MbkIADZjPnA.","doi":"10.24963/ijcai.2024/1017","url":"https://ijcai.org/proceedings/2024/1017","pdf":"https://ijcai.org/proceedings/2024/1017.pdf","markdown":"https://ijcai.org/proceedings/2024/1017.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/1017","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Demo Track","track":"Demo Track","number":1017,"pages":{"first":8721,"last":8724},"keywords":["Computer Vision: CV: Neural generative models, auto encoders, GANs","Humans and AI: HAI: Human-computer interaction","Humans and AI: HAI: Intelligent user interfaces"],"videos":[],"references":[{"position":1,"raw":"[Alammar, 2022] Jay Alammar. The illustrated Stable Diffusion. https://jalammar.github.io/illustrated-stable- diffusion/, 2022. Accessed on: 2023-04-30.","title":"The illustrated Stable Diffusion","authors":"Jay Alammar","year":"2022","venue":null,"doi":null,"url":"https://jalammar.github.io/illustrated-stable-diffusion/","internal_url":null},{"position":2,"raw":"[AMELION, 2023] AMELION. https://twitter.com/ amelion /status/1651193228677218304, 2023. Accessed on: 2023-04-26.","title":"AMELION","authors":"Amelion","year":"2023","venue":null,"doi":null,"url":"https://twitter.com/amelion/status/1651193228677218304","internal_url":null},{"position":3,"raw":"[anton, 2023] anton. Announcing Stable Attribution -A tool which lets anyone find the human creators behind AI generated images. https://twitter.com/atroyn/status/ 1622355473193381888, 2023. Accessed on: 2023-04-30.","title":"anton. Announcing Stable Attribution -A tool which lets anyone find the human creators behind AI generated images","authors":"Anton","year":"2023","venue":null,"doi":null,"url":"https://twitter.com/atroyn/status/1622355473193381888,2023","internal_url":null},{"position":4,"raw":"[Bostock et al., 2011] Michael Bostock, Vadim Ogievetsky, and Jeffrey Heer. D 3 Data-driven Documents. IEEE transactions on visualization and computer graphics, 17(12):2301-2309, 2011.","title":"D 3 Data-driven Documents","authors":"Michael Bostock, Vadim Ogievetsky, Jeffrey Heer","year":"2011","venue":"IEEE transactions on visualization and computer graphics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brusseau, 2022] James Brusseau. Acceleration AI Ethics, the Debate between Innovation and Safety, and Stability AI's Diffusion versus OpenAI's Dall-E. arXiv preprint arXiv:2212.01834, 2022.","title":"Acceleration AI Ethics, the Debate between Innovation and Safety, and Stability AI's Diffusion versus OpenAI's Dall-E","authors":"James Brusseau","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Choudhary, 2022] Lokesh Choudhary. Stable Dif- fusion is Now Accused of 'Stealing' Artwork. https://analyticsindiamag.com/stable-diffusion-is-now- accused-of-stealing-artwork/, 2022. Accessed on: 2023-04-30.","title":"Stable Diffusion is Now Accused of 'Stealing' Artwork","authors":"Lokesh Choudhary","year":"2022","venue":null,"doi":null,"url":"https://analyticsindiamag.com/stable-diffusion-is-now-accused-of-stealing-artwork/","internal_url":null},{"position":7,"raw":"[Dixit, 2023] Pranav Dixit. Meet The Three Artists Be- hind A Landmark Lawsuit Against AI Art Generators. https://www.buzzfeednews.com/article/pranavdixit/ai-art- generators-lawsuit-stable-diffusion-midjourney, 2023. Accessed on: 2023-04-30.","title":"Meet The Three Artists Behind A Landmark Lawsuit Against AI Art Generators","authors":"Pranav Dixit","year":"2023","venue":null,"doi":null,"url":"https://www.buzzfeednews.com/article/pranavdixit/ai-art-generators-lawsuit-stable-diffusion-midjourney,2023","internal_url":null},{"position":8,"raw":"[Engler, 2023] Alex Engler. Early thoughts on regulating generative AI like ChatGPT. Brookings Institution, 2023. Accessed on: 2023-04-30.","title":"Early thoughts on regulating generative AI like ChatGPT","authors":"Alex Engler","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hendrix, 2023] Justin Hendrix. Generative AI, Section 230 and Liability: Assessing the Questions. Tech Policy Press, 2023. Accessed on: 2023-04-30.","title":"Generative AI, Section 230 and Liability: Assessing the Questions","authors":"Justin Hendrix","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Howard, 2023] Jeremy Howard. From Deep Learning Foundations to Stable Diffusion. https://www.fast.ai/posts/ part2-2023.html, 2023. Accessed on: 2023-04-30.","title":"From Deep Learning Foundations to Stable Diffusion","authors":"Jeremy Howard","year":"2023","venue":null,"doi":null,"url":"https://www.fast.ai/posts/part2-2023.html,2023","internal_url":null},{"position":11,"raw":"[Huber and Troynikov, 2023] Jeff Huber and An- ton Troynikov. Stable Attribution. https: //www.stableattribution.com, 2023. Accessed on: 2023-04-30.","title":"Stable Attribution","authors":"Jeff Huber","year":"2023","venue":null,"doi":null,"url":"https://www.stableattribution.com","internal_url":null},{"position":12,"raw":"[Karras et al., 2022] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35:26565-26577, 2022.","title":"Elucidating the design space of diffusion-based generative models","authors":"Tero Karras, Miika Aittala, Timo Aila, Samuli Laine","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Nichol and Dhariwal, 2021] Alexander Quinn Nichol and Prafulla Dhariwal. Improved Denoising Diffusion Prob- abilistic Models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.","title":"Improved Denoising Diffusion Probabilistic Models","authors":"Nichol, Dhariwal, Alexander Quinn, Nichol, Prafulla Dhariwal","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models from Natural Language Supervision. In International con- ference on machine learning, pages 8748-8763. PMLR, 2021.","title":"Learning Transferable Visual Models from Natural Language Supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rombach et al., 2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution Image Synthesis with Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.","title":"High-resolution Image Synthesis with Latent Diffusion Models","authors":"Rombach","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-net: Convolutional Networks for Biomedical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234- 241. Springer, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional Networks for Biomedical Image Segmentation","authors":"Ronneberger","year":"2015","venue":"Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ryan-Mosley, 2023] Tate Ryan-Mosley. An early guide to policymaking on generative AI. MIT Technology Review, 2023. Accessed on: 2023-04-30.","title":"An early guide to policymaking on generative AI","authors":"Tate Ryan-Mosley","year":"2023","venue":"MIT Technology Review","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Smith, 2022] Ethan Smith. A Traveler's Guide to the Latent Space. https://sweet-hall-e72.","title":"A Traveler's Guide to the Latent Space","authors":"Ethan Smith, Smith","year":"2022","venue":null,"doi":null,"url":"https://sweet-hall-e72","internal_url":null},{"position":19,"raw":"notion.site/A-Traveler-s-Guide-to-the-Latent-Space- 85efba7e5e6a40e5bd3cae980f30235f#976ba690a0904431 aac693d59830a92c, 2022. Accessed on: 2023-04-29.","title":"site/A-Traveler-s-Guide-to-the-Latent-Space-85efba7e5e6a40e5bd3cae980f30235f#976","authors":null,"year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Stability AI, 2022] Stability AI. Stable Diffusion Public Release. https://stability.ai/blog/stable-diffusion-public- release, 2022. Accessed on: 2022-08-22.","title":"Stability AI. Stable Diffusion Public Release","authors":null,"year":"2022","venue":null,"doi":null,"url":"https://stability.ai/blog/stable-diffusion-public-release,2022","internal_url":null},{"position":21,"raw":"[Sung, 2022] Morgan Sung. Lensa, the AI portrait app, has soared in popularity. But many artists question the ethics of AI art. NBC News, 2022. Accessed on: 2023-04-30.","title":"the AI portrait app, has soared in popularity. But many artists question the ethics of AI art","authors":"Morgan Sung, Lensa","year":"2022","venue":"NBC News","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Turner, 2022] Kevin Turner. Decoding latents to RGB with- out upscaling. https://discuss.huggingface.co/t/decoding- latents-to-rgb-without-upscaling/23204/2, 2022. Ac- cessed on: 2023-04-30.","title":"Decoding latents to RGB without upscaling","authors":"Kevin Turner","year":"2022","venue":null,"doi":null,"url":"https://discuss.huggingface.co/t/decoding-latents-to-rgb-without-upscaling/23204/2","internal_url":null},{"position":23,"raw":"U.S. Copyright Office. Copy- right Office Launches New Artificial Intelligence Initia- tive. https://www.copyright.gov/newsnet/2023/1004.html, 2023. Accessed on: 2023-04-30.","title":"Copyright Office Launches New Artificial Intelligence Initiative","authors":null,"year":"2023","venue":null,"doi":null,"url":"https://www.copyright.gov/newsnet/2023/1004.html","internal_url":null},{"position":24,"raw":"[von Platen, 2022] Patrick von Platen. Testing Stable Dif- fusion is hard. https://github.com/huggingface/diffusers/ issues/937, 2022. Accessed on: 2023-04-30.","title":"Testing Stable Diffusion is hard","authors":"Von Platen","year":"2022","venue":null,"doi":null,"url":"https://github.com/huggingface/diffusers/issues/937","internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}