{"type":"ScholarlyArticle","title":"Demystifying Neural Style Transfer","authors":[{"name":"Yanghao Li","slug":"yanghao-li","given":"Yanghao","family":"Li","affiliation":"† Institute of Computer Science and Technology , Peking University"},{"name":"Naiyan Wang","slug":"naiyan-wang","given":"Naiyan","family":"Wang","affiliation":"† Institute of Computer Science and Technology , Peking University"},{"name":"Jiaying Liu","slug":"jiaying-liu","given":"Jiaying","family":"Liu","affiliation":"† Institute of Computer Science and Technology , Peking University"},{"name":"Xiaodi Hou","slug":"xiaodi-hou","given":"Xiaodi","family":"Hou","affiliation":"† Institute of Computer Science and Technology , Peking University"}],"abstract":"Neural Style Transfer has recently demonstrated very exciting results which catches eyes in both academia and industry. Despite the amazing results, the principle of neural style transfer, especially why the Gram matrices could represent style remains unclear. In this paper, we propose a novel interpretation of neural style transfer by treating it as a domain adaptation problem. Specifically, we theoretically show that matching the Gram matrices of feature maps is equivalent to minimize the Maximum Mean Discrepancy (MMD) with the second order polynomial kernel. Thus, we argue that the essence of neural style transfer is to match the feature distributions between the style images and the generated images. To further support our standpoint, we experiment with several other distribution alignment methods, and achieve appealing results. We believe this novel interpretation connects these two important research fields, and could enlighten future researches.","doi":"10.24963/ijcai.2017/310","url":"https://ijcai.org/proceedings/2017/310","pdf":"https://ijcai.org/proceedings/2017/0310.pdf","markdown":"https://ijcai.org/proceedings/2017/310.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/310","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":310,"pages":{"first":2230,"last":2236},"keywords":["Machine Learning: Deep Learning","Machine Learning: Transfer, Adaptation, Multi-task Learning","Multidisciplinary Topics and Applications: Art and Music"],"videos":[],"references":[{"position":1,"raw":"[Beijbom, 2012] Oscar Beijbom. Domain adaptations for computer vision applications. arXiv preprint arXiv:1211.4860, 2012.","title":"Oscar Beijbom. Domain adaptations for computer vision applications","authors":"Beijbom","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Champandard, 2016] Alex J Champandard. Semantic style transfer and turning two-bit doodles into fine artworks. arXiv preprint arXiv:1603.01768, 2016.","title":"Semantic style transfer and turning two-bit doodles into fine artworks","authors":"Alex J Champandard","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2016] Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. MXNet: A flexible and efficient machine learning library for heterogeneous distributed systems. NIPS Workshop on Machine Learn- ing Systems, 2016.","title":"MXNet: A flexible and efficient machine learning library for heterogeneous distributed systems","authors":"Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, Zheng Zhang","year":"2016","venue":"NIPS Workshop on Machine Learning Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Efros and Freeman, 2001] Alexei A Efros and William T Freeman. Image quilting for texture synthesis and transfer. In SIGGRAPH, 2001.","title":"Image quilting for texture synthesis and transfer","authors":"Alexei A Efros, William T Freeman","year":"2001","venue":"SIGGRAPH","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Efros and Leung, 1999] Alexei A Efros and Thomas K Le- ung. Texture synthesis by non-parametric sampling. In ICCV, 1999.","title":"Texture synthesis by non-parametric sampling","authors":"Efros, Leung ; Efros, Thomas K Leung","year":"1999","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Frigo et al., 2016] Oriel Frigo, Neus Sabater, Julie Delon, and Pierre Hellier. Split and match: Example-based adap- tive patch sampling for unsupervised style transfer. In CVPR, 2016.","title":"Split and match: Example-based adaptive patch sampling for unsupervised style transfer","authors":"Oriel Frigo, Neus Sabater, Julie Delon, Pierre Hellier","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gatys et al., 2016] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In CVPR, 2016.","title":"Image style transfer using convolutional neural networks","authors":"Leon A Gatys, Alexander S Ecker, Matthias Bethge","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gretton et al., 2012a] Arthur Gretton, Karsten M Borg- wardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.","title":"A kernel two-sample test","authors":"Gretton","year":"2012","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gretton et al., 2012b] Arthur Gretton, Dino Sejdinovic, Heiko Strathmann, Sivaraman Balakrishnan, Massimil- iano Pontil, Kenji Fukumizu, and Bharath K Sriperum- budur. Optimal kernel choice for large-scale two-sample tests. In NIPS, 2012.","title":"Optimal kernel choice for large-scale two-sample tests","authors":"Gretton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hertzmann et al., 2001] Aaron Hertzmann, Charles E Ja- cobs, Nuria Oliver, Brian Curless, and David H Salesin. Image analogies. In SIGGRAPH, 2001.","title":"Image analogies","authors":"Aaron Hertzmann, Charles E Jacobs, Nuria Oliver, Brian Curless, David H Salesin","year":"2001","venue":"SIGGRAPH","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Johnson et al., 2016] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.","title":"Perceptual losses for real-time style transfer and super-resolution","authors":"Justin Johnson, Alexandre Alahi, Li Fei-Fei","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kwatra et al., 2005] Vivek Kwatra, Irfan Essa, Aaron Bo- bick, and Nipun Kwatra. Texture optimization for example-based synthesis. ACM Transactions on Graph- ics, 24(3):795-802, 2005.","title":"Texture optimization for example-based synthesis","authors":"Vivek Kwatra, Irfan Essa, Aaron Bobick, Nipun Kwatra","year":"2005","venue":"ACM Transactions on Graphics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ledig et al., 2016] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single im- age super-resolution using a generative adversarial net- work. arXiv preprint arXiv:1609.04802, 2016.","title":"Photo-realistic single image super-resolution using a generative adversarial network","authors":"Ledig","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li and Wand, 2016] Chuan Li and Michael Wand. Combin- ing Markov random fields and convolutional neural net- works for image synthesis. In CVPR, 2016.","title":"Combining Markov random fields and convolutional neural networks for image synthesis","authors":"Wand ; Li, Chuan Li, Michael Wand","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2017] Yanghao Li, Naiyan Wang, Jianping Shi, Ji- aying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. ICLRW, 2017.","title":"Revisiting batch normalization for practical domain adaptation","authors":"Li","year":"2017","venue":"ICLRW","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liang et al., 2001] Lin Liang, Ce Liu, Ying-Qing Xu, Bain- ing Guo, and Heung-Yeung Shum. Real-time texture syn- thesis by patch-based sampling. ACM Transactions on Graphics, 20(3):127-150, 2001.","title":"Real-time texture synthesis by patch-based sampling","authors":"Lin Liang, Ce Liu, Ying-Qing Xu, Baining Guo, Heung-Yeung Shum","year":"2001","venue":"ACM Transactions on Graphics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Long et al., 2015] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I Jordan. Learning transferable fea- tures with deep adaptation networks. In ICML, 2015.","title":"Learning transferable features with deep adaptation networks","authors":"Mingsheng Long, Yue Cao, Jianmin Wang, Michael I Jordan","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Long et al., 2016] Mingsheng Long, Jianmin Wang, and Michael I Jordan. Unsupervised domain adaptation with residual transfer networks. In NIPS, 2016.","title":"Unsupervised domain adaptation with residual transfer networks","authors":"Mingsheng Long, Jianmin Wang, Michael I Jordan","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pan and Yang, 2010] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowl- edge and Data Engineering, 22(10):1345-1359, 2010.","title":"A survey on transfer learning","authors":"Jialin Sinno, Qiang Pan, Yang","year":"2010","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Patel et al., 2015] Vishal M Patel, Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. Visual domain adapta- tion: A survey of recent advances. IEEE Signal Processing Magazine, 32(3):53-69, 2015.","title":"Visual domain adaptation: A survey of recent advances","authors":"M Vishal, Raghuraman Patel, Ruonan Gopalan, Rama Li, Chellappa","year":"2015","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ruder et al., 2016] Manuel Ruder, Alexey Dosovitskiy, and Thomas Brox. Artistic style transfer for videos. In GCPR, 2016.","title":"Artistic style transfer for videos","authors":"Manuel Ruder, Alexey Dosovitskiy, Thomas Brox","year":"2016","venue":"GCPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Selim et al., 2016] Ahmed Selim, Mohamed Elgharib, and Linda Doyle. Painting style transfer for head portraits us- ing convolutional neural networks. ACM Transactions on Graphics, 35(4):129, 2016.","title":"Painting style transfer for head portraits using convolutional neural networks","authors":"Ahmed Selim, Mohamed Elgharib, Linda Doyle","year":"2016","venue":"ACM Transactions on Graphics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shih et al., 2014] YiChang Shih, Sylvain Paris, Connelly Barnes, William T Freeman, and Frédo Durand. Style transfer for headshot portraits. ACM Transactions on Graphics, 33(4):148, 2014.","title":"Style transfer for headshot portraits","authors":"Shih","year":"2014","venue":"ACM Transactions on Graphics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sun et al., 2016] Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. AAAI, 2016.","title":"Return of frustratingly easy domain adaptation","authors":"Sun","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tzeng et al., 2014] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confu- sion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474, 2014.","title":"Deep domain confusion: Maximizing for domain invariance","authors":"Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, Trevor Darrell","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ulyanov et al., 2016] Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor Lempitsky. Texture networks: Feed-forward synthesis of textures and stylized images. In ICML, 2016.","title":"Texture networks: Feed-forward synthesis of textures and stylized images","authors":"Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, Victor Lempitsky","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}