{"type":"ScholarlyArticle","title":"StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Translation","authors":[{"name":"Peter Schaldenbrand","slug":"peter-schaldenbrand","given":"Peter","family":"Schaldenbrand","affiliation":"The Robotics Institute , Carnegie Mellon University"},{"name":"Zhixuan Liu","slug":"zhixuan-liu","given":"Zhixuan","family":"Liu","affiliation":"School of Data Science , The Chinese University of Hong Kong"},{"name":"Jean Oh","slug":"jean-oh","given":"Jean","family":"Oh","affiliation":"The Robotics Institute , Carnegie Mellon University"}],"abstract":"Generating images that fit a given text description using machine learning has improved greatly with the release of technologies such as the CLIP image-text encoder model; however, current methods lack artistic control of the style of image to be generated. We present an approach for generating styled drawings for a given text description where a user can specify a desired drawing style using a sample image. Inspired by a theory in art that style and content are generally inseparable during the creative process, we propose a coupled approach, known here as StyleCLIPDraw, whereby the drawing is generated by optimizing for style and content simultaneously throughout the process as opposed to applying style transfer after creating content in a sequence. Based on human evaluation, the styles of images generated by StyleCLIPDraw are strongly preferred to those by the sequential approach. Although the quality of content generation degrades for certain styles, overall considering both content and style, StyleCLIPDraw is found far more preferred, indicating the importance of style, look, and feel of machine generated images to people as well as indicating that style is coupled in the drawing process itself. Our code, a demonstration, and style evaluation data are publicly available.","doi":"10.24963/ijcai.2022/688","url":"https://ijcai.org/proceedings/2022/688","pdf":"https://ijcai.org/proceedings/2022/0688.pdf","markdown":"https://ijcai.org/proceedings/2022/688.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/688","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Research Papers","track":"AI and Arts","number":688,"pages":{"first":4966,"last":4972},"keywords":["Application domains: Images and visual arts","Theory and philosophy of arts and creativity in AI systems: Autonomous creative or artistic AI","Methods and resources: Machine learning, deep learning, neural models, reinforcement learning","Theory and philosophy of arts and creativity in AI systems: Evaluation of artistic or creative outputs, or of systems"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987171","length_sec":79,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985314","length_sec":650,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Alsmadi, 2020] Mutasem K Alsmadi. Content-based im- age retrieval using color, shape and texture descriptors and features. Arabian Journal for Science and Engineering, 45(4):3317-3330, 2020.","title":"Content-based image retrieval using color, shape and texture descriptors and features","authors":"K Mutasem, Alsmadi","year":"2020","venue":"Arabian Journal for Science and Engineering","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Banerji et al., 2013] Sugata Banerji, Atreyee Sinha, and Chengjun Liu. New image descriptors based on color, tex- ture, shape, and wavelets for object and scene image clas- sification. Neurocomputing, 117:173-185, 2013.","title":"New image descriptors based on color, texture, shape, and wavelets for object and scene image classification","authors":"Sugata Banerji, Atreyee Sinha, Chengjun Liu","year":"2013","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bateman, 2021] Cole Bateman. Creating for creatives: A humanistic approach to designing ai tools targeted at pro- fessional animators. Bachelor's thesis, Harvard College, 2021.","title":"Creating for creatives: A humanistic approach to designing ai tools targeted at professional animators","authors":"Cole Bateman","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Beck, 2014] Paula D Beck. Fourth-Grade Students' Subjec- tive Interactions with the Seven Elements of Art: An Ex- ploratory Case Study Using Q-methodology. Long Island University, CW Post Center, 2014.","title":"Fourth-Grade Students' Subjective Interactions with the Seven Elements of Art: An Exploratory Case Study Using Q-methodology","authors":"Paula D Beck","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bretan et al., 2016] Mason Bretan, Deepak Gopinath, Philip Mullins, and Gil Weinberg. A robotic prosthesis for an amputee drummer. arXiv preprint arXiv:1612.04391, 2016.","title":"A robotic prosthesis for an amputee drummer","authors":"Mason Bretan, Deepak Gopinath, Philip Mullins, Gil Weinberg","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chandrarathne et al., 2020] Gayani Chandrarathne, Kokul Thanikasalam, and Amalka Pinidiyaarachchi. A com- prehensive study on deep image classification with small datasets. In Advances in Electronics Engineering, pages 93-106. Springer, 2020.","title":"Gayani Chandrarathne, Kokul Thanikasalam, and Amalka Pinidiyaarachchi. A comprehensive study on deep image classification with small datasets","authors":"Chandrarathne","year":"2020","venue":"Advances in Electronics Engineering","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Frans et al., 2021] Kevin Frans, LB Soros, and Olaf Witkowski. Clipdraw: Exploring text-to-drawing syn- thesis through language-image encoders. arXiv preprint arXiv:2106.14843, 2021.","title":"Clipdraw: Exploring text-to-drawing synthesis through language-image encoders","authors":"Kevin Frans, Olaf Soros, Witkowski","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Galatolo. et al., 2021] Federico Galatolo., Mario Cimino., and Gigliola Vaglini. Generating images from caption and vice versa via clip-guided generative latent space search. Proceedings of the International Conference on Image Processing and Vision Engineering, 2021.","title":"Generating images from caption and vice versa via clip-guided generative latent space search","authors":"Federico Galatolo, Mario Cimino, Gigliola Vaglini","year":"2021","venue":"Proceedings of the International Conference on Image Processing and Vision Engineering","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gatys et al., 2016] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414- 2423, 2016.","title":"Image style transfer using convolutional neural networks","authors":"Leon A Gatys, Alexander S Ecker, Matthias Bethge","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2019] Zhewei Huang, Wen Heng, and Shuchang Zhou. Learning to paint with model-based deep reinforcement learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8709-8718, 2019.","title":"Learning to paint with model-based deep reinforcement learning","authors":"Huang","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim et al., 2022] Diana Kim, Ahmed Elgammal, and Mar- ian Mazzone. Formal analysis of art: Proxy learning of vi- sual concepts from style through language models. arXiv preprint arXiv:2201.01819, 2022.","title":"Formal analysis of art: Proxy learning of visual concepts from style through language models","authors":"Diana Kim, Ahmed Elgammal, Marian Mazzone","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim, 2010] Seong-in Kim. A computer system for the anal- ysis of color-related elements in art therapy assessment: Computer color-related elements art therapy evaluation system (c creates). The Arts in psychotherapy, 37(5):378- 386, 2010.","title":"A computer system for the analysis of color-related elements in art therapy assessment: Computer color-related elements art therapy evaluation system (c creates)","authors":"Seong-In Kim","year":"2010","venue":"The Arts in psychotherapy","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kolkin et al., 2019] Nicholas Kolkin, Jason Salavon, and Gregory Shakhnarovich. Style transfer by relaxed opti- mal transport and self-similarity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10051-10060, 2019.","title":"Style transfer by relaxed optimal transport and self-similarity","authors":"Nicholas Kolkin, Jason Salavon, Gregory Shakhnarovich","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2020] Tzu-Mao Li, Michal Lukáč, Michaël Gharbi, and Jonathan Ragan-Kelley. Differentiable vec- tor graphics rasterization for editing and learning. ACM Transactions on Graphics (TOG), 39(6):1-15, 2020.","title":"Differentiable vector graphics rasterization for editing and learning","authors":"Li","year":"2020","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2021] Songhua Liu, Tianwei Lin, Dongliang He, Fu Li, Ruifeng Deng, Xin Li, Errui Ding, and Hao Wang. Paint transformer: Feed forward neural painting with stroke prediction. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 6598- 6607, 2021.","title":"Paint transformer: Feed forward neural painting with stroke prediction","authors":"Songhua Liu, Tianwei Lin, Dongliang He, Fu Li, Ruifeng Deng, Xin Li, Errui Ding, Hao Wang","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nichol et al., 2021] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photore- alistic image generation and editing with text-guided dif- fusion models. arXiv preprint arXiv:2112.10741, 2021.","title":"Glide: Towards photorealistic image generation and editing with text-guided diffusion models","authors":"Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, Mark Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Plummer et al., 2015] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collect- ing region-to-phrase correspondences for richer image-to- sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649, 2015.","title":"Flickr30k entities: Collecting region-to-phrase correspondences for richer image-tosentence models","authors":"Liwei Bryan A Plummer, Chris M Wang, Juan C Cervantes, Julia Caicedo, Svetlana Hockenmaier, Lazebnik","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervi- sion. CoRR, abs/2103.00020, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ramesh et al., 2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092, 2021.","title":"Zero-shot text-to-image generation","authors":"Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, Ilya Sutskever","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rebuffi et al., 2021] Sylvestre-Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy A Mann. Data augmentation can improve robust- ness. Advances in Neural Information Processing Systems, 34, 2021.","title":"Data augmentation can improve robustness","authors":"Sylvestre-Alvise, Sven Rebuffi, Dan Gowal, Florian Andrei Calian, Olivia Stimberg, Timothy A Wiles, Mann","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Robertson, 1967] Duncan Robertson. The dichotomy of form and content. College English, 28(4):273-279, 1967.","title":"The dichotomy of form and content","authors":"Duncan Robertson","year":"1967","venue":"College English","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schaldenbrand and Oh, 2020] Peter Schaldenbrand and Jean Oh. Content masked loss: Human-like brush stroke planning in a reinforcement learning painting agent. arXiv preprint arXiv:2012.10043, 2020.","title":"Content masked loss: Human-like brush stroke planning in a reinforcement learning painting agent","authors":"Oh Schaldenbrand, Peter Schaldenbrand, Jean Oh","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Selvaggio et al., 2021] Mario Selvaggio, Marco Cognetti, Stefanos Nikolaidis, Serena Ivaldi, and Bruno Siciliano. Autonomy in physical human-robot interaction: A brief survey. IEEE Robotics and Automation Letters, 2021.","title":"Autonomy in physical human-robot interaction: A brief survey","authors":"Mario Selvaggio, Marco Cognetti, Stefanos Nikolaidis, Serena Ivaldi, Bruno Siciliano","year":"2021","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Smith and Colton, 2021] Amy Smith and Simon Colton. Clip-guided gan image generation: An artistic exploration. Evo* 2021, page 17, 2021.","title":"Clip-guided gan image generation: An artistic exploration","authors":"Amy Smith, Simon Colton","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Thorson, 2020] Mark Thorson. Formal elements of art. En- counters With the Arts: Readings for ARTC150, 2020.","title":"Formal elements of art. Encounters With the Arts: Readings for ARTC","authors":"Mark Thorson","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tian and Ha, 2021] Yingtao Tian and David Ha. Modern evolution strategies for creativity: Fitting concrete images and abstract concepts. arXiv preprint arXiv:2109.08857, 2021.","title":"Modern evolution strategies for creativity: Fitting concrete images and abstract concepts","authors":"Ha ; Tian, Yingtao Tian, David Ha","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}