{"type":"ScholarlyArticle","title":"Scribble-to-Painting Transformation with Multi-Task Generative Adversarial Networks","authors":[{"name":"Jinning Li","slug":"jinning-li","given":"Jinning","family":"Li","affiliation":"Shanghai Jiao Tong University"},{"name":"Yexiang Xue","slug":"yexiang-xue","given":"Yexiang","family":"Xue","affiliation":"Purdue University"}],"abstract":"We propose the Dual Scribble-to-Painting Network (DSP-Net), which is able to produce artistic paintings based on user-generated scribbles. In scribble-to-painting transformation, a neural net has to infer additional details of the image, given relatively sparse information contained in the outlines of the scribble. Therefore, it is more challenging than classical image style transfer, in which the information content is reduced from photos to paintings. Inspired by the human cognitive process, we propose a multi-task generative adversarial network, which consists of two jointly trained neural nets -- one for generating artistic images and the other one for semantic segmentation. We demonstrate that joint training on these two tasks brings in additional benefit. Experimental result shows that DSP-Net outperforms state-of-the-art models both visually and quantitatively. In addition, we publish a large dataset for scribble-to-painting transformation.","doi":"10.24963/ijcai.2019/820","url":"https://ijcai.org/proceedings/2019/820","pdf":"https://ijcai.org/proceedings/2019/0820.pdf","markdown":"https://ijcai.org/proceedings/2019/820.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/820","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"AI for Improving Human Well-being","track":"AI for Improving Human Well-being","number":820,"pages":{"first":5916,"last":5922},"keywords":["Special Track on AI for Improving Human-Well Being: Entertainment;games applications (Special Track on AI and Human Wellbeing)"],"videos":[],"references":[{"position":1,"raw":"[Canny, 1986] John Canny. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, pages 679-698, 1986.","title":"A computational approach to edge detection","authors":"John Canny","year":"1986","venue":"IEEE Transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen and Hays, 2018] Wengling Chen and James Hays. Sketchygan: Towards diverse and realistic sketch to image synthesis. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pages 9416-9425, 2018.","title":"Sketchygan: Towards diverse and realistic sketch to image synthesis","authors":"Chen, Hays, Wengling Chen, James Hays","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Crowley and Zisserman, 2014] Elliot J Crowley and An- drew Zisserman. In search of art. In Workshop at the European Conference on Computer Vision, pages 54-70. Springer, 2014.","title":"Elliot J Crowley and Andrew Zisserman. In search of art","authors":"Zisserman Crowley","year":"2014","venue":"Workshop at the European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gatys et al., 2015] Leon Gatys, Alexander S Ecker, and Matthias Bethge. Texture synthesis using convolutional neural networks. In Advances in Neural Information Pro- cessing Systems, pages 262-270, 2015.","title":"Texture synthesis using convolutional neural networks","authors":"Leon Gatys, Alexander S Ecker, Matthias Bethge","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gatys et al., 2016] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2414- 2423, 2016.","title":"Image style transfer using convolutional neural networks","authors":"Leon A Gatys, Alexander S Ecker, Matthias Bethge","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Girshick, 2015] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440-1448, 2015.","title":"Fast r-cnn","authors":"Ross Girshick","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Isola et al., 2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with con- ditional adversarial networks. arXiv preprint, 2017.","title":"Image-to-image translation with conditional adversarial networks","authors":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A Efros","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Johnson et al., 2016] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Com- puter Vision, pages 694-711. Springer, 2016.","title":"Perceptual losses for real-time style transfer and super-resolution","authors":"Justin Johnson, Alexandre Alahi, Li Fei-Fei","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Khan et al., 2014] Fahad Shahbaz Khan, Shida Beigpour, Joost Van de Weijer, and Michael Felsberg. Painting-91: a large scale database for computational painting categoriza- tion. Machine vision and applications, 25(6):1385-1397, 2014.","title":"Painting-91: a large scale database for computational painting categorization","authors":"Fahad Shahbaz Khan, Shida Beigpour, Joost Van De Weijer, Michael Felsberg","year":"2014","venue":"Machine vision and applications","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li and Wand, 2016] Chuan Li and Michael Wand. Precom- puted real-time texture synthesis with markovian gener- ative adversarial networks. In European Conference on Computer Vision, pages 702-716. Springer, 2016.","title":"Precomputed real-time texture synthesis with markovian generative adversarial networks","authors":"Wand ; ] Chuan Li, Michael Li, Wand","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lim et al., 2013] Joseph J Lim, C Lawrence Zitnick, and Pi- otr Dollár. Sketch tokens: A learned mid-level represen- tation for contour and object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3158-3165, 2013.","title":"Sketch tokens: A learned mid-level representation for contour and object detection","authors":"J Joseph, C Lim, Piotr Lawrence Zitnick, Dollár","year":"2013","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In European conference on com- puter vision, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu and Tuzel, 2016] Ming-Yu Liu and Oncel Tuzel. Cou- pled generative adversarial networks. In Advances in neu- ral information processing systems, pages 469-477, 2016.","title":"Coupled generative adversarial networks","authors":"Ming-Yu Liu, Oncel Tuzel","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lu et al., 2012] Cewu Lu, Li Xu, and Jiaya Jia. Combining sketch and tone for pencil drawing production. In Proceed- ings of the Symposium on Non-Photorealistic Animation and Rendering, pages 65-73. Eurographics Association, 2012.","title":"Combining sketch and tone for pencil drawing production","authors":"Lu","year":"2012","venue":"Proceedings of the Symposium on Non-Photorealistic Animation and Rendering","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Radford et al., 2015] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.","title":"Unsupervised representation learning with deep convolutional generative adversarial networks","authors":"Radford","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sangkloy et al., 2016a] Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. The sketchy database: learning to retrieve badly drawn bunnies. ACM Transac- tions on Graphics (TOG), 35(4):119, 2016.","title":"The sketchy database: learning to retrieve badly drawn bunnies","authors":"Sangkloy","year":"2016","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sangkloy et al., 2016b] Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, and James Hays. Scribbler: Con- trolling deep image synthesis with sketch and color. arXiv preprint arXiv:1612.00835, 2016.","title":"Scribbler: Controlling deep image synthesis with sketch and color","authors":"Sangkloy","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tanno et al., 2017] Ryosuke Tanno, Shin Matsuo, Wataru Shimoda, and Keiji Yanai. Deepstylecam: A real-time style transfer app on ios. In International Conference on Multimedia Modeling, pages 446-449. Springer, 2017.","title":"Deepstylecam: A real-time style transfer app on ios","authors":"Ryosuke Tanno, Shin Matsuo, Wataru Shimoda, Keiji Yanai","year":"2017","venue":"International Conference on Multimedia Modeling","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vyshedskiy and Dunn, 2015] Andrey Vyshedskiy and Rita Dunn. Mental synthesis involves the synchronization of independent neuronal ensembles. Research Ideas and Out- comes, 1:e7642, 2015.","title":"Mental synthesis involves the synchronization of independent neuronal ensembles","authors":"Andrey Vyshedskiy, Rita Dunn","year":"2015","venue":"Research Ideas and Outcomes","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang and Gupta, 2016] Xiaolong Wang and Abhinav Gupta. Generative image modeling using style and structure adversarial networks. In European Conference on Computer Vision, pages 318-335. Springer, 2016.","title":"Generative image modeling using style and structure adversarial networks","authors":"Wang, Gupta, Xiaolong Wang, Abhinav Gupta","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhu et al., 2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image transla- tion using cycle-consistent adversarial networks. arXiv preprint, 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Zhu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}