{"type":"ScholarlyArticle","title":"DivSwapper: Towards Diversified Patch-based Arbitrary Style Transfer","authors":[{"name":"Zhizhong Wang","slug":"zhizhong-wang","given":"Zhizhong","family":"Wang","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Lei Zhao","slug":"lei-zhao","given":"Lei","family":"Zhao","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Haibo Chen","slug":"haibo-chen","given":"Haibo","family":"Chen","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Zhiwen Zuo","slug":"zhiwen-zuo","given":"Zhiwen","family":"Zuo","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Ailin Li","slug":"ailin-li","given":"Ailin","family":"Li","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Wei Xing","slug":"wei-xing","given":"Wei","family":"Xing","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Dongming Lu","slug":"dongming-lu","given":"Dongming","family":"Lu","affiliation":"College of Computer Science and Technology , Zhejiang University"}],"abstract":"Gram-based and patch-based approaches are two important research lines of style transfer. Recent diversified Gram-based methods have been able to produce multiple and diverse stylized outputs for the same content and style images. However, as another widespread research interest, the diversity of patch-based methods remains challenging due to the stereotyped style swapping process based on nearest patch matching. To resolve this dilemma, in this paper, we dive into the crux of existing patch-based methods and propose a universal and efficient module, termed DivSwapper, for diversified patch-based arbitrary style transfer. The key insight is to use an essential intuition that neural patches with higher activation values could contribute more to diversity. Our DivSwapper is plug-and-play and can be easily integrated into existing patch-based and Gram-based methods to generate diverse results for arbitrary styles. We conduct theoretical analyses and extensive experiments to demonstrate the effectiveness of our method, and compared with state-of-the-art algorithms, it shows superiority in diversity, quality, and efficiency.","doi":"10.24963/ijcai.2022/690","url":"https://ijcai.org/proceedings/2022/690","pdf":"https://ijcai.org/proceedings/2022/0690.pdf","markdown":"https://ijcai.org/proceedings/2022/690.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/690","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Research Papers","track":"AI and Arts","number":690,"pages":{"first":4980,"last":4987},"keywords":["Theory and philosophy of arts and creativity in AI systems: Autonomous creative or artistic AI","Methods and resources: Machine learning, deep learning, neural models, reinforcement learning","Application domains: Images and visual arts"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987160","length_sec":166,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985303","length_sec":314,"provider":"slideslive"}],"references":[{"position":1,"raw":"Kfir Aberman, Jing Liao, Mingyi Shi, Dani Lischinski, Baoquan Chen, and Daniel Cohen-Or. Neural best- buddies: Sparse cross-domain correspondence. TOG, 37(4):1- 14, 2018.","title":"Neural bestbuddies: Sparse cross-domain correspondence","authors":"Jing Kfir Aberman, Mingyi Liao, Dani Shi, Baoquan Lischinski, Daniel Chen, Cohen-Or","year":"2018","venue":"TOG","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[An et al., 2021] Jie An, Siyu Huang, Yibing Song, Dejing Dou, Wei Liu, and Jiebo Luo. Artflow: Unbiased image style transfer via reversible neural flows. In CVPR, pages 862-871, 2021.","title":"Artflow: Unbiased image style transfer via reversible neural flows","authors":"Jie An, Siyu Huang, Yibing Song, Dejing Dou, Wei Liu, Jiebo Luo","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Champandard, 2016] Alex J Champandard. Semantic style trans- fer and turning two-bit doodles into fine artworks. arXiv preprint arXiv:1603.01768, 2016.","title":"Semantic style transfer and turning two-bit doodles into fine artworks","authors":"Alex J Champandard","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chandran et al., 2021] Prashanth Chandran, Gaspard Zoss, Paulo Gotardo, Markus Gross, and Derek Bradley. Adaptive convolu- tions for structure-aware style transfer. In CVPR, pages 7972- 7981, 2021.","title":"Adaptive convolutions for structure-aware style transfer","authors":"Prashanth Chandran, Gaspard Zoss, Paulo Gotardo, Markus Gross, Derek Bradley","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen and Schmidt, 2016] Tian Qi Chen and Mark Schmidt. Fast patch-based style transfer of arbitrary style. arXiv preprint arXiv:1612.04337, 2016.","title":"Fast patch-based style transfer of arbitrary style","authors":"Tian Qi, Chen, Mark Schmidt","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2017] Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, and Gang Hua. Stylebank: An explicit representation for neural image style transfer. In CVPR, pages 1897-1906, 2017.","title":"Stylebank: An explicit representation for neural image style transfer","authors":"Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, Gang Hua","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2021a] Haibo Chen, Lei Zhao, Zhizhong Wang, Huiming Zhang, Zhiwen Zuo, Ailin Li, Wei Xing, and Dong- ming Lu. Artistic style transfer with internal-external learning and contrastive learning. In NeurIPS, 2021.","title":"Artistic style transfer with internal-external learning and contrastive learning","authors":"Chen","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2021b] Haibo Chen, Lei Zhao, Zhizhong Wang, Huiming Zhang, Zhiwen Zuo, Ailin Li, Wei Xing, and Dong- ming Lu. Dualast: Dual style-learning networks for artistic style transfer. In CVPR, pages 872-881, 2021.","title":"Dualast: Dual style-learning networks for artistic style transfer","authors":"Chen","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cheng et al., 2021] Jiaxin Cheng, Ayush Jaiswal, Yue Wu, Pradeep Natarajan, and Prem Natarajan. Style-aware normalized loss for improving arbitrary style transfer. In CVPR, pages 134- 143, 2021.","title":"Style-aware normalized loss for improving arbitrary style transfer","authors":"Jiaxin Cheng, Ayush Jaiswal, Yue Wu, Pradeep Natarajan, Prem Natarajan","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Deng et al., 2020] Yingying Deng, Fan Tang, Weiming Dong, Wen Sun, Feiyue Huang, and Changsheng Xu. Arbitrary style transfer via multi-adaptation network. In ACM MM, pages 2719-2727, 2020.","title":"Arbitrary style transfer via multi-adaptation network","authors":"Yingying Deng, Fan Tang, Weiming Dong, Wen Sun, Feiyue Huang, Changsheng Xu","year":"2020","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gatys et al., 2016] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In CVPR, pages 2414-2423, 2016.","title":"Image style transfer using convolutional neural networks","authors":"Leon A Gatys, Alexander S Ecker, Matthias Bethge","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gu et al., 2018] Shuyang Gu, Congliang Chen, Jing Liao, and Lu Yuan. Arbitrary style transfer with deep feature reshuffle. In CVPR, pages 8222-8231, 2018.","title":"Arbitrary style transfer with deep feature reshuffle","authors":"Shuyang Gu, Congliang Chen, Jing Liao, Lu Yuan","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang and Belongie, 2017] Xun Huang and Serge Belongie. Ar- bitrary style transfer in real-time with adaptive instance normal- ization. In ICCV, pages 1501-1510, 2017.","title":"Arbitrary style transfer in real-time with adaptive instance normalization","authors":"Belongie Huang, Xun Huang, Serge Belongie","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jing et al., 2019] Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Yizhou Yu, and Mingli Song. Neural style trans- fer: A review. TVCG, 26(11):3365-3385, 2019.","title":"Neural style transfer: A review","authors":"Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Yizhou Yu, Mingli Song","year":"2019","venue":"TVCG","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jing et al., 2020] Yongcheng Jing, Xiao Liu, Yukang Ding, Xin- chao Wang, Errui Ding, Mingli Song, and Shilei Wen. Dynamic instance normalization for arbitrary style transfer. In AAAI, vol- ume 34, pages 4369-4376, 2020.","title":"Dynamic instance normalization for arbitrary style transfer","authors":"Yongcheng Jing, Xiao Liu, Yukang Ding, Xinchao Wang, Errui Ding, Mingli Song, Shilei Wen","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Johnson et al., 2016] Justin Johnson, Alexandre Alahi, and Li Fei- Fei. Perceptual losses for real-time style transfer and super- resolution. In ECCV, pages 694-711. Springer, 2016.","title":"Perceptual losses for real-time style transfer and superresolution","authors":"Justin Johnson, Alexandre Alahi, Li Fei-Fei","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kolkin et al., 2019] Nicholas Kolkin, Jason Salavon, and Gregory Shakhnarovich. Style transfer by relaxed optimal transport and self-similarity. In CVPR, pages 10051-10060, 2019.","title":"Style transfer by relaxed optimal transport and self-similarity","authors":"Nicholas Kolkin, Jason Salavon, Gregory Shakhnarovich","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li and Wand, 2016a] Chuan Li and Michael Wand. Combining markov random fields and convolutional neural networks for im- age synthesis. In CVPR, pages 2479-2486, 2016.","title":"Combining markov random fields and convolutional neural networks for image synthesis","authors":"Wand ; Li, Chuan Li, Michael Wand","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li and Wand, 2016b] Chuan Li and Michael Wand. Precomputed real-time texture synthesis with markovian generative adversarial networks. In ECCV, pages 702-716. Springer, 2016.","title":"Precomputed real-time texture synthesis with markovian generative adversarial networks","authors":"Wand ; Li, Chuan Li, Michael Wand","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2017a] Yanghao Li, Naiyan Wang, Jiaying Liu, and Xi- aodi Hou. Demystifying neural style transfer. In IJCAI, pages 2230-2236, 2017.","title":"Demystifying neural style transfer","authors":"Li","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/310"},{"position":21,"raw":"[Li et al., 2017b] Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Diversified texture syn- thesis with feed-forward networks. In CVPR, 2017.","title":"Diversified texture synthesis with feed-forward networks","authors":"Li","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li et al., 2017c] Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Universal style transfer via feature transforms. In NeurIPS, pages 386-396, 2017.","title":"Universal style transfer via feature transforms","authors":"Li","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Li et al., 2019] Xueting Li, Sifei Liu, Jan Kautz, and Ming-Hsuan Yang. Learning linear transformations for fast image and video style transfer. In CVPR, pages 3809-3817, 2019.","title":"Learning linear transformations for fast image and video style transfer","authors":"Li","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liao et al., 2017] Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang. Visual attribute transfer through deep image analogy. TOG, 2017.","title":"Visual attribute transfer through deep image analogy","authors":"Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, Bing Sing, Kang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Lin et al., 2021] Tianwei Lin, Zhuoqi Ma, Fu Li, Dongliang He, Xin Li, Errui Ding, Nannan Wang, Jie Li, and Xinbo Gao. Draft- ing and revision: Laplacian pyramid network for fast high-quality artistic style transfer. In CVPR, pages 5141-5150, 2021.","title":"Drafting and revision: Laplacian pyramid network for fast high-quality artistic style transfer","authors":"Tianwei Lin, Zhuoqi Ma, Fu Li, Dongliang He, Xin Li, Errui Ding, Nannan Wang, Jie Li, Xinbo Gao","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Liu et al., 2021] Songhua Liu, Tianwei Lin, Dongliang He, Fu Li, Meiling Wang, Xin Li, Zhengxing Sun, Qian Li, and Errui Ding. Adaattn: Revisit attention mechanism in arbitrary neural style transfer. In ICCV, pages 6649-6658, 2021.","title":"Adaattn: Revisit attention mechanism in arbitrary neural style transfer","authors":"Songhua Liu, Tianwei Lin, Dongliang He, Fu Li, Meiling Wang, Xin Li, Zhengxing Sun, Qian Li, Errui Ding","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Lu et al., 2019] Ming Lu, Hao Zhao, Anbang Yao, Yurong Chen, Feng Xu, and Li Zhang. A closed-form solution to universal style transfer. In ICCV, pages 5952-5961, 2019.","title":"A closed-form solution to universal style transfer","authors":"Ming Lu, Hao Zhao, Anbang Yao, Yurong Chen, Feng Xu, Li Zhang","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Park and Lee, 2019] Dae Young Park and Kwang Hee Lee. Ar- bitrary style transfer with style-attentional networks. In CVPR, pages 5880-5888, 2019.","title":"Arbitrary style transfer with style-attentional networks","authors":"Young Dae, Kwang Park, Lee Hee","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sheng et al., 2018] Lu Sheng, Ziyi Lin, Jing Shao, and Xiaogang Wang. Avatar-net: Multi-scale zero-shot style transfer by feature decoration. In CVPR, pages 8242-8250, 2018.","title":"Avatar-net: Multi-scale zero-shot style transfer by feature decoration","authors":"Lu Sheng, Ziyi Lin, Jing Shao, Xiaogang Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale im- age recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Ulyanov et al., 2016] Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor S Lempitsky. Texture networks: Feed- forward synthesis of textures and stylized images. In ICML, pages 1349-1357, 2016.","title":"Texture networks: Feedforward synthesis of textures and stylized images","authors":"Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, Victor S Lempitsky","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Ulyanov et al., 2017] Dmitry Ulyanov, Andrea Vedaldi, and Vic- tor Lempitsky. Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In CVPR, pages 6924-6932, 2017.","title":"Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis","authors":"Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wang et al., 2020a] Zhizhong Wang, Lei Zhao, Haibo Chen, Li- hong Qiu, Qihang Mo, Sihuan Lin, Wei Xing, and Dongming Lu. Diversified arbitrary style transfer via deep feature perturbation. In CVPR, pages 7789-7798, 2020.","title":"Diversified arbitrary style transfer via deep feature perturbation","authors":"Wang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wang et al., 2020b] Zhizhong Wang, Lei Zhao, Sihuan Lin, Qi- hang Mo, Huiming Zhang, Wei Xing, and Dongming Lu. Glstylenet: exquisite style transfer combining global and local pyramid features. IET Computer Vision, 14(8):575-586, 2020.","title":"Glstylenet: exquisite style transfer combining global and local pyramid features","authors":"Wang","year":"2020","venue":"IET Computer Vision","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wang et al., 2021] Zhizhong Wang, Lei Zhao, Haibo Chen, Zhi- wen Zuo, Ailin Li, Wei Xing, and Dongming Lu. Evaluate and improve the quality of neural style transfer. CVIU, 207:103203, 2021.","title":"Evaluate and improve the quality of neural style transfer","authors":"Zhizhong Wang, Lei Zhao, Haibo Chen, Zhiwen Zuo, Ailin Li, Wei Xing, Dongming Lu","year":"2021","venue":"CVIU","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wang et al., 2022] Zhizhong Wang, Lei Zhao, Haibo Chen, Ailin Li, Zhiwen Zuo, Wei Xing, and Dongming Lu. Texture reformer: Towards fast and universal interactive texture transfer. In AAAI, 2022.","title":"Texture reformer: Towards fast and universal interactive texture transfer","authors":"Zhizhong Wang, Lei Zhao, Haibo Chen, Ailin Li, Zhiwen Zuo, Wei Xing, Dongming Lu","year":"2022","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Yao et al., 2019] Yuan Yao, Jianqiang Ren, Xuansong Xie, Wei- dong Liu, Yong-Jin Liu, and Jun Wang. Attention-aware multi- stroke style transfer. In CVPR, pages 1467-1475, 2019.","title":"Attention-aware multistroke style transfer","authors":"Yuan Yao, Jianqiang Ren, Xuansong Xie, Weidong Liu, Yong-Jin Liu, Jun Wang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhang et al., 2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, pages 586-595, 2018.","title":"The unreasonable effectiveness of deep features as a perceptual metric","authors":"Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, Oliver Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhang et al., 2019] Yulun Zhang, Chen Fang, Yilin Wang, Zhaowen Wang, Zhe Lin, Yun Fu, and Jimei Yang. Multimodal style transfer via graph cuts. In ICCV, pages 5943-5951, 2019.","title":"Multimodal style transfer via graph cuts","authors":"Yulun Zhang, Chen Fang, Yilin Wang, Zhaowen Wang, Zhe Lin, Yun Fu, Jimei Yang","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}