{"type":"ScholarlyArticle","title":"Art Creation with Multi-Conditional StyleGANs","authors":[{"name":"Konstantin Dobler","slug":"konstantin-dobler","given":"Konstantin","family":"Dobler","affiliation":"Hasso Plattner Institute / University of Potsdam"},{"name":"Florian Hübscher","slug":"florian-hubscher","given":"Florian","family":"Hübscher","affiliation":"Hasso Plattner Institute / University of Potsdam"},{"name":"Jan Westphal","slug":"jan-westphal","given":"Jan","family":"Westphal","affiliation":"Hasso Plattner Institute / University of Potsdam"},{"name":"Alejandro Sierra-Múnera","slug":"alejandro-sierra-munera","given":"Alejandro","family":"Sierra-Múnera","affiliation":"Hasso Plattner Institute / University of Potsdam"},{"name":"Gerard de Melo","slug":"gerard-de-melo","given":"Gerard","family":"de Melo","affiliation":"Hasso Plattner Institute / University of Potsdam"},{"name":"Ralf Krestel","slug":"ralf-krestel","given":"Ralf","family":"Krestel","affiliation":"ZBW -Leibniz Information Centre for Economics and Kiel University"}],"abstract":"Creating art is often viewed as a uniquely human endeavor. In this paper, we introduce a multi-conditional Generative Adversarial Network (GAN) approach trained on large amounts of human paintings to synthesize realistic-looking paintings that emulate human art. Our approach is based on the StyleGAN neural network architecture, but incorporates a custom multi-conditional control mechanism that provides fine-granular control over characteristics of the generated paintings, e.g., with regard to the perceived emotion evoked in a spectator. We also investigate several evaluation techniques tailored to multi-conditional generation.","doi":"10.24963/ijcai.2022/684","url":"https://ijcai.org/proceedings/2022/684","pdf":"https://ijcai.org/proceedings/2022/0684.pdf","markdown":"https://ijcai.org/proceedings/2022/684.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/684","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Research Papers","track":"AI and Arts","number":684,"pages":{"first":4936,"last":4942},"keywords":["Application domains: Images and visual arts","Methods and resources: Machine learning, deep learning, neural models, reinforcement learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987172","length_sec":82,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985315","length_sec":643,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Achlioptas et al., 2021] Panos Achlioptas, Maks Ovs- janikov, Kilichbek Haydarov, Mohamed Elhoseiny, and Leonidas J. Guibas. ArtEmis: Affective language for visual art. In CVPR, pages 11564-11574, 2021.","title":"ArtEmis: Affective language for visual art","authors":"Panos Achlioptas, Maks Ovsjanikov, Kilichbek Haydarov, Mohamed Elhoseiny, Leonidas J Guibas","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baluja et al., 1994] Shumeet Baluja, Dean Pomerleau, and Todd Jochem. Towards automated artificial evolution for computer-generated images. Connection Science, 6(2- 3):325-354, 1994.","title":"Towards automated artificial evolution for computer-generated images","authors":"Baluja","year":"1994","venue":"Connection Science","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bińkowski et al., 2018] Mikołaj Bińkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demysti- fying MMD GANs. In ICLR, 2018.","title":"Demystifying MMD GANs","authors":"Mikołaj Bińkowski, Dougal J Sutherland, Michael Arbel, Arthur Gretton","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bohanec et al., 1992] Marko Bohanec, Bozo Urh, and Vladislav Rajkovič. Evaluating options by combined qualitative and quantitative methods. Acta Psychologica, 80(1):67-89, 1992.","title":"Evaluating options by combined qualitative and quantitative methods","authors":"Marko Bohanec, Bozo Urh, Vladislav Rajkovič","year":"1992","venue":"Acta Psychologica","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brock et al., 2019] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fi- delity natural image synthesis. ArXiv, abs/1809.11096, 2019.","title":"Large scale GAN training for high fidelity natural image synthesis","authors":"Andrew Brock, Jeff Donahue, Karen Simonyan","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[de Vries et al., 2017] Harm de Vries, Florian Strub, Jeremie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by lan- guage. In NeurIPS, volume 30, pages 6594-6604, 2017.","title":"Modulating early visual processing by language","authors":"De Vries","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Devries et al., 2019] Terrance Devries, Adriana Romero, L. Pineda, Graham W. Taylor, and Michal Drozdzal. On the evaluation of conditional GANs. ArXiv, abs/1907.08175, 2019.","title":"On the evaluation of conditional GANs","authors":"Terrance Devries, Adriana Romero, L Pineda, Graham W Taylor, Michal Drozdzal","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dorin and Korb, 2009] Alan Dorin and Kevin B Korb. Im- probable creativity. In Computational Creativity: An Inter- disciplinary Approach, number 09291 in Dagstuhl Semi- nar Proceedings. Schloss Dagstuhl -Leibniz-Zentrum fuer Informatik, Germany, 2009.","title":"Improbable creativity","authors":"Alan Dorin, Kevin B Korb","year":"2009","venue":"Computational Creativity: An Interdisciplinary Approach, number 09291","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Elgammal et al., 2017] Ahmed Elgammal, Bingchen Liu, Mohamed Elhoseiny, and Marian Mazzone. CAN: Cre- ative adversarial networks, generating \"art\" by learning about styles and deviating from style norms. In ICCC, 2017.","title":"CAN: Creative adversarial networks, generating \"art\" by learning about styles and deviating from style norms","authors":"Ahmed Elgammal, Bingchen Liu, Mohamed Elhoseiny, Marian Mazzone","year":"2017","venue":"ICCC","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In NeurIPS, volume 27, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jiao et al., 2020] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for natural language un- derstanding. In EMNLP, pages 4163-4174, 2020.","title":"TinyBERT: Distilling BERT for natural language understanding","authors":"Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, Qun Liu","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Karras et al., 2018] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In ICLR, 2018.","title":"Progressive growing of GANs for improved quality, stability, and variation","authors":"Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Karras et al., 2019] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, pages 4396-4405, 2019.","title":"A style-based generator architecture for generative adversarial networks","authors":"Tero Karras, Samuli Laine, Timo Aila","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Karras et al., 2020a] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In NeurIPS, volume 33, pages 12104-12114, 2020.","title":"Training generative adversarial networks with limited data","authors":"Karras","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Karras et al., 2020b] Tero Karras, Samuli Laine, Miika Ait- tala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. An- alyzing and improving the image quality of StyleGAN. In CVPR, pages 8107-8116, 2020.","title":"Analyzing and improving the image quality of StyleGAN","authors":"Karras","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2021] Bingchen Liu, Kunpeng Song, Yizhe Zhu, and Ahmed Elgammal. Sketch-to-art: Synthesizing styl- ized art images from sketches. In Computer Vision -ACCV 2020, pages 207-222, 2021.","title":"Sketch-to-art: Synthesizing stylized art images from sketches","authors":"Liu","year":"2021","venue":"Computer Vision -ACCV 2020","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[McCormack et al., 2019] Jon McCormack, Toby Gifford, and Patrick Hutchings. Autonomy, authenticity, author- ship and intention in computer generated art. In Computa- tional Intelligence in Music, Sound, Art and Design, pages 35-50, 2019.","title":"Autonomy, authenticity, authorship and intention in computer generated art","authors":"Mccormack","year":"2019","venue":"Computational Intelligence in Music, Sound, Art and Design","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mirza and Osindero, 2014] Mehdi Mirza and Simon Osin- dero. Conditional generative adversarial nets. ArXiv, abs/1411.1784, 2014.","title":"Conditional generative adversarial nets","authors":"Mehdi Mirza, Simon Osindero","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Miyato and Koyama, 2018] Takeru Miyato and Masanori Koyama. cGANs with projection discriminator. In ICLR, 2018.","title":"cGANs with projection discriminator","authors":"Koyama ; ] Takeru Miyato, Masanori Miyato, Koyama","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mohammad and Kiritchenko, 2018] Saif M. Mohammad and Svetlana Kiritchenko. An annotated dataset of emo- tions evoked by art. In LREC, volume 11, 2018.","title":"An annotated dataset of emotions evoked by art","authors":"M Saif, Svetlana Mohammad, Kiritchenko","year":"2018","venue":"LREC","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Park et al., 2018] Hyojin Park, Youngjoon Yoo, and Nojun Kwak. MC-GAN: Multi-conditional generative adversar- ial network for image synthesis. In The British Machine Vision Conference (BMVC), 2018.","title":"MC-GAN: Multi-conditional generative adversarial network for image synthesis","authors":"Hyojin Park, Youngjoon Yoo, Nojun Kwak","year":"2018","venue":"The British Machine Vision Conference (BMVC)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervi- sion. In ICML, volume 139, pages 8748-8763, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Szegedy et al., 2016] Christian Szegedy, Vincent Van- houcke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, pages 2818-2826, 2016.","title":"Rethinking the inception architecture for computer vision","authors":"Szegedy","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yildirim et al., 2018] Gökhan Yildirim, Calvin Seward, and Urs M. Bergmann. Disentangling multiple conditional in- puts in GANs. ArXiv, abs/1806.07819, 2018.","title":"Disentangling multiple conditional inputs in GANs","authors":"Gökhan Yildirim, Calvin Seward, Urs M Bergmann","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2018] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, pages 586-595, 2018.","title":"The unreasonable effectiveness of deep features as a perceptual metric","authors":"Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, Oliver Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhu et al., 2020] Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. Improved StyleGAN embedding: Where are the good latents? ArXiv, abs/2012.09036, 2020.","title":"Improved StyleGAN embedding: Where are the good latents?","authors":"Zhu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}