{"type":"ScholarlyArticle","title":"From Reality to Perception: Genre-Based Neural Image Style Transfer","authors":[{"name":"Zhuoqi Ma","slug":"zhuoqi-ma","given":"Zhuoqi","family":"Ma","affiliation":"† State Key Laboratory of Integrated Services Networks , School of Electronic Engineering , Xidian University , Xi'an 710071 , China"},{"name":"Nannan Wang","slug":"nannan-wang","given":"Nannan","family":"Wang","affiliation":"‡ State Key Laboratory of Integrated Services Networks , School of Telecommunications , Xidian University , Xi'an 710071 , China"},{"name":"Xinbo Gao","slug":"xinbo-gao","given":"Xinbo","family":"Gao","affiliation":"† State Key Laboratory of Integrated Services Networks , School of Electronic Engineering , Xidian University , Xi'an 710071 , China"},{"name":"Jie Li","slug":"jie-li","given":"Jie","family":"Li","affiliation":"† State Key Laboratory of Integrated Services Networks , School of Electronic Engineering , Xidian University , Xi'an 710071 , China"}],"abstract":"We introduce a novel thought for integrating artists’ perceptions on the real world into neural image style transfer process. Conventional approaches commonly migrate color or texture patterns from style image to content image, but the underlying design aspect of the artist always get overlooked. We want to address the in-depth genre style, that how artists perceive the real world and express their perceptions in the artwork. We collect a set of Van Gogh’s paintings and cubist artworks, and their semantically corresponding real world photos. We present a novel genre style transfer framework modeled after the mechanism of actual artwork production. The target style representation is reconstructed based on the semantic correspondence between real world photo and painting, which enable the perception guidance in style transfer. The experimental results demonstrate that our method can capture the overall style of a genre or an artist. We hope that this work provides new insight for including artists’ perceptions into neural style transfer process, and helps people to understand the underlying characters of the artist or the genre.","doi":"10.24963/ijcai.2018/485","url":"https://ijcai.org/proceedings/2018/485","pdf":"https://ijcai.org/proceedings/2018/0485.pdf","markdown":"https://ijcai.org/proceedings/2018/485.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/485","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning Applications","track":"Main track","number":485,"pages":{"first":3491,"last":3497},"keywords":["Machine Learning Applications: Other Applications","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Barnes et al., 2009] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics-TOG, 28(3):24, 2009.","title":"Patchmatch: A randomized correspondence algorithm for structural image editing","authors":"Connelly Barnes, Eli Shechtman, Adam Finkelstein, Dan B Goldman","year":"2009","venue":"ACM Transactions on Graphics-TOG","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brox et al., 2004] Thomas Brox, Andrés Bruhn, Nils Pa- penberg, and Joachim Weickert. High accuracy opti- cal flow estimation based on a theory for warping. In European conference on computer vision, pages 25-36. Springer, 2004.","title":"High accuracy optical flow estimation based on a theory for warping","authors":"Thomas Brox, Andrés Bruhn, Nils Papenberg, Joachim Weickert","year":"2004","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Champandard, 2016] Alex J Champandard. Semantic style transfer and turning two-bit doodles into fine artworks. arXiv preprint arXiv:1603.01768, 2016.","title":"Semantic style transfer and turning two-bit doodles into fine artworks","authors":"Alex J Champandard","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2017] Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, and Gang Hua. Stylebank: An explicit repre- sentation for neural image style transfer. In Proc. CVPR, 2017.","title":"Stylebank: An explicit representation for neural image style transfer","authors":"Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, Gang Hua","year":"2017","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dumoulin et al., 2016] Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. In International Conference on Learning Representations, 2016.","title":"A learned representation for artistic style","authors":"Jonathon Vincent Dumoulin, Manjunath Shlens, Kudlur","year":"2016","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gatys et al., 2016] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Computer Vision and Pattern Recog- nition (CVPR), 2016 IEEE Conference on, pages 2414- 2423. IEEE, 2016.","title":"Image style transfer using convolutional neural networks","authors":"Leon A Gatys, Alexander S Ecker, Matthias Bethge","year":"2016","venue":"Computer Vision and Pattern Recognition (CVPR), 2016 IEEE Conference on","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hertzmann et al., 2001] Aaron Hertzmann, Charles E Ja- cobs, Nuria Oliver, Brian Curless, and David H Salesin. Image analogies. In Proceedings of the 28th annual con- ference on Computer graphics and interactive techniques, pages 327-340. ACM, 2001.","title":"Image analogies","authors":"Aaron Hertzmann, Charles E Jacobs, Nuria Oliver, Brian Curless, David H Salesin","year":"2001","venue":"Proceedings of the 28th annual conference on Computer graphics and interactive techniques","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Johnson et al., 2016] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Com- puter Vision, pages 694-711. Springer, 2016.","title":"Perceptual losses for real-time style transfer and super-resolution","authors":"Justin Johnson, Alexandre Alahi, Li Fei-Fei","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li and Wand, 2016] Chuan Li and Michael Wand. Combin- ing markov random fields and convolutional neural net- works for image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2479-2486, 2016.","title":"Combining markov random fields and convolutional neural networks for image synthesis","authors":"Wand ; ] Chuan Li, Michael Li, Wand","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liao et al., 2017] Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang. Visual attribute transfer through deep image analogy. ACM Transactions on Graphics-TOG, 36(4), 2017.","title":"Visual attribute transfer through deep image analogy","authors":"Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, Bing Sing, Kang","year":"2017","venue":"ACM Transactions on Graphics-TOG","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2016] Ce Liu, Jenny Yuen, and Antonio Torralba. Sift flow: Dense correspondence across scenes and its ap- plications. In Dense Image Correspondences for Com- puter Vision, pages 15-49. Springer, 2016.","title":"Sift flow: Dense correspondence across scenes and its applications","authors":"Ce Liu, Jenny Yuen, Antonio Torralba","year":"2016","venue":"Dense Image Correspondences for Computer Vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lowe, 2004] David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91-110, 2004.","title":"Distinctive image features from scale-invariant keypoints","authors":"David G Lowe","year":"2004","venue":"International journal of computer vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mordvintsev et al., 2015] Alexander Mordvintsev, Christo- pher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks. Google Research Blog. Retrieved June, 20(14):5, 2015.","title":"Inceptionism: Going deeper into neural networks","authors":"Alexander Mordvintsev, Christopher Olah, Mike Tyka","year":"2015","venue":"Google Research Blog","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Selim et al., 2016] Ahmed Selim, Mohamed Elgharib, and Linda Doyle. Painting style transfer for head portraits us- ing convolutional neural networks. ACM Transactions on Graphics (ToG), 35(4):129, 2016.","title":"Painting style transfer for head portraits using convolutional neural networks","authors":"Ahmed Selim, Mohamed Elgharib, Linda Doyle","year":"2016","venue":"ACM Transactions on Graphics (ToG)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Semmo et al., 2015] Amir Semmo, Daniel Limberger, Jan Eric Kyprianidis, and Jürgen Döllner. Image styl- ization by oil paint filtering using color palettes. In Proceedings of the workshop on Computational Aesthet- ics, pages 149-158. Eurographics Association, 2015.","title":"Image stylization by oil paint filtering using color palettes","authors":"Amir Semmo, Daniel Limberger, Jan Eric Kyprianidis, Jürgen Döllner","year":"2015","venue":"Proceedings of the workshop on Computational Aesthetics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Semmo et al., 2017] Amir Semmo, Tobias Isenberg, and Jürgen Döllner. Neural style transfer: a paradigm shift for image-based artistic rendering? In Proceedings of the Symposium on Non-Photorealistic Animation and Render- ing, page 5. ACM, 2017.","title":"Neural style transfer: a paradigm shift for image-based artistic rendering?","authors":"Amir Semmo, Tobias Isenberg, Jürgen Döllner","year":"2017","venue":"Proceedings of the Symposium on Non-Photorealistic Animation and Rendering","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Confer- ence on Learning Representations, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ulyanov et al., 2016] Dmitry Ulyanov, Andrea Vedaldi, and Victor S Lempitsky. Instance normalization: The missing ingredient for fast stylization. CoRR, 2016.","title":"Instance normalization: The missing ingredient for fast stylization","authors":"Dmitry Ulyanov, Andrea Vedaldi, Victor S Lempitsky","year":"2016","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2017] Nannan Wang, Xinbo Gao, Leiyu Sun, and Jie Li. Bayesian face sketch synthesis. IEEE Transac- tions on Image Processing, 26(3):1264-1274, 2017.","title":"Bayesian face sketch synthesis","authors":"Nannan Wang, Xinbo Gao, Leiyu Sun, Jie Li","year":"2017","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhu et al., 1997] Ciyou Zhu, Richard H Byrd, Peihuang Lu, and Jorge Nocedal. Algorithm 778: L-bfgs-b: Fortran sub- routines for large-scale bound-constrained optimization. ACM Transactions on Mathematical Software (TOMS), 23(4):550-560, 1997.","title":"Algorithm 778: L-bfgs-b: Fortran subroutines for large-scale bound-constrained optimization","authors":"Ciyou Zhu, Richard H Byrd, Peihuang Lu, Jorge Nocedal","year":"1997","venue":"ACM Transactions on Mathematical Software (TOMS)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhu et al., 2017a] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In IEEE In- ternational Conference on Computer Vision. ICCV, 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Zhu","year":"2017","venue":"IEEE International Conference on Computer Vision. ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhu et al., 2017b] Mingrui Zhu, Nannan Wang, Xinbo Gao, and Jie Li. Deep graphical feature learning for face sketch synthesis. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 3574-3580. AAAI Press, 2017.","title":"Deep graphical feature learning for face sketch synthesis","authors":"Zhu","year":"2017","venue":"Proceedings of the 26th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2017/500"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}