{"type":"ScholarlyArticle","title":"Tool- and Domain-Agnostic Parameterization of Style Transfer Effects Leveraging Pretrained Perceptual Metrics","authors":[{"name":"Hiromu Yakura","slug":"hiromu-yakura","given":"Hiromu","family":"Yakura","affiliation":"University of Tsukuba , Japan; National Institute of Advanced Industrial Science and Technology (AIST) , Japan"},{"name":"Yuki Koyama","slug":"yuki-koyama","given":"Yuki","family":"Koyama","affiliation":"National Institute of Advanced Industrial Science and Technology (AIST) , Japan"},{"name":"Masataka Goto","slug":"masataka-goto","given":"Masataka","family":"Goto","affiliation":"National Institute of Advanced Industrial Science and Technology (AIST) , Japan"}],"abstract":"Current deep learning techniques for style transfer would not be optimal for design support since their \"one-shot\" transfer does not fit exploratory design processes. To overcome this gap, we propose parametric transcription, which transcribes an end-to-end style transfer effect into parameter values of specific transformations available in an existing content editing tool. With this approach, users can imitate the style of a reference sample in the tool that they are familiar with and thus can easily continue further exploration by manipulating the parameters. To enable this, we introduce a framework that utilizes an existing pretrained model for style transfer to calculate a perceptual style distance to the reference sample and uses black-box optimization to find the parameters that minimize this distance. Our experiments with various third-party tools, such as Instagram and Blender, show that our framework can effectively leverage deep learning techniques for computational design support.","doi":"10.24963/ijcai.2021/167","url":"https://ijcai.org/proceedings/2021/167","pdf":"https://ijcai.org/proceedings/2021/0167.pdf","markdown":"https://ijcai.org/proceedings/2021/167.md","bibtex":"https://ijcai.org/proceedings/2021/bibtex/167","volume":{"slug":"2021","title":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2021","url":"https://ijcai.org/proceedings/2021/","year":2021,"location":"Montreal","dates":"19-27 August 2021","editors":"Prof. Zhi-Hua Zhou","isbn":"978-0-9992411-9-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2021"},"section":"Computer Vision II","track":"Main Track","number":167,"pages":{"first":1208,"last":1216},"keywords":["Humans and AI: Human-AI Collaboration","Computer Vision: 2D and 3D Computer Vision","Humans and AI: Intelligent User Interfaces"],"videos":[],"references":[{"position":1,"raw":"[Akiba et al., 2019] T. Akiba, S. Sano, et al. Optuna: A next- generation hyperparameter optimization framework. In ACM KDD, 2019.","title":"Optuna: A nextgeneration hyperparameter optimization framework","authors":"T Akiba, S Sano","year":"2019","venue":"ACM KDD","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brochu et al., 2007] E. Brochu, N. Freitas, and A. Ghosh. Active preference learning with discrete choice data. In NeurIPS, 2007.","title":"Active preference learning with discrete choice data","authors":"E Brochu, N Freitas, A Ghosh","year":"2007","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018] K. Chen, C Choy, et al. Text2Shape: gen- erating shapes from natural language by learning joint em- beddings. In ACCV, 2018.","title":"Text2Shape: generating shapes from natural language by learning joint embeddings","authors":"K Chen, Choy","year":"2018","venue":"ACCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2019] H. Chen, K. Hui, et al. BeautyGlow: on- demand makeup transfer framework with reversible gener- ative network. In IEEE/CVF CVPR, 2019.","title":"BeautyGlow: ondemand makeup transfer framework with reversible generative network","authors":"H Chen, K Hui","year":"2019","venue":"IEEE/CVF CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dalí, 1970] S. Dalí. Dali by Dali. Abrams, New York, NY, 1970.","title":"Dali by Dali","authors":"S Dalí","year":"1970","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Doersch, 2016] C. Doersch. Tutorial on variational autoen- coders. arXiv, abs/1606.05908, 2016.","title":"Tutorial on variational autoencoders","authors":"C Doersch","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fukamizu et al., 2019] K. Fukamizu, M. Kondo, and R. Sakamoto. Generation high resolution 3D model from natural language by GAN. arXiv, abs/1901.07165, 2019.","title":"Generation high resolution 3D model from natural language by GAN","authors":"K Fukamizu, M Kondo, R Sakamoto","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gero, 1990] J. Gero. Design prototypes: A knowledge rep- resentation schema for design. AI Mag., 11(4), 1990.","title":"Design prototypes: A knowledge representation schema for design","authors":"J Gero","year":"1990","venue":"AI Mag","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goodfellow et al., 2014] I.","title":null,"authors":"Goodfellow","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"Goodfellow, J. Pouget-Abadie, et al. Generative adversarial nets. In NeurIPS, 2014.","title":"Generative adversarial nets","authors":"J Goodfellow, Pouget-Abadie","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gu et al., 2019] Q. Gu, G. Wang, et al. LADN: local ad- versarial disentangling network for facial makeup and de- makeup. In IEEE ICCV, 2019.","title":"LADN: local adversarial disentangling network for facial makeup and demakeup","authors":"Q Gu, G Wang","year":"2019","venue":"IEEE ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Guo et al., 2019] J. Guo, Z. Qian, et al. MulGAN: facial attribute editing by exemplar. arXiv, abs/1912.12396, 2019.","title":"MulGAN: facial attribute editing by exemplar","authors":"J Guo, Z Qian","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hartmann et al., 2008] B. Hartmann, L. Yu, et al. Design as exploration: creating interface alternatives through parallel authoring and runtime tuning. In ACM UIST, 2008.","title":"Design as exploration: creating interface alternatives through parallel authoring and runtime tuning","authors":"B Hartmann, L Yu","year":"2008","venue":"ACM UIST","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[He et al., 2019] Z. He, W. Zuo, et al. AttGAN: facial at- tribute editing by only changing what you want. IEEE Trans. Image Process., 28(11), 2019.","title":"AttGAN: facial attribute editing by only changing what you want","authors":"Z He, W Zuo","year":"2019","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Holden et al., 2017] D. Holden, I. Habibie, et al. Fast neural style transfer for motion data. IEEE Comput. Graph. Appl., 37(4), 2017.","title":"Fast neural style transfer for motion data","authors":"D Holden, I Habibie","year":"2017","venue":"IEEE Comput. Graph. Appl","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hu et al., 2018] Y. Hu, H. He, et al. Exposure: A white-box photo post-processing framework. ACM Trans. Graph., 37(2), 2018.","title":"Exposure: A white-box photo post-processing framework","authors":"Y Hu, H He","year":"2018","venue":"ACM Trans. Graph","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hu et al., 2019] Y. Hu, J. Dorsey, and H. Rushmeier. A novel framework for inverse procedural texture modeling. ACM Trans. Graph., 38(6), 2019.","title":"A novel framework for inverse procedural texture modeling","authors":"Y Hu, J Dorsey, H Rushmeier","year":"2019","venue":"ACM Trans. Graph","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Huang et al., 2018] X. Huang, M. Liu, et al. Multimodal unsupervised image-to-image translation. In ECCV, 2018.","title":"Multimodal unsupervised image-to-image translation","authors":"Huang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Jing et al., 2020] Y. Jing, Y. Yang, et al. Neural style transfer: A review. IEEE Trans. Vis. Comput. Graph., 26(11), 2020.","title":"Neural style transfer: A review","authors":"Y Jing, Y Yang","year":"2020","venue":"IEEE Trans. Vis. Comput. Graph","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kingma and Dhariwal, 2018] D. Kingma and P. Dhariwal. Glow: generative flow with invertible 1x1 convolutions. In NeurIPS, 2018.","title":"Glow: generative flow with invertible 1x1 convolutions","authors":"D Kingma, P","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Koyama and Igarashi, 2018] Y. Koyama and T. Igarashi. Computational design with crowds. In Computational In- teraction. Oxford University Press, 2018.","title":"Computational design with crowds","authors":"Y Koyama, T Igarashi","year":"2018","venue":"Computational Interaction","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Koyama et al., 2020] Y. Koyama, I. Sato, and M. Goto. Se- quential gallery for interactive visual design optimization. ACM Trans. Graph., 39(4), 2020.","title":"Sequential gallery for interactive visual design optimization","authors":"Y Koyama, I Sato, M Goto","year":"2020","venue":"ACM Trans. Graph","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Kühnel et al., 2018] L. Kühnel, T. Fletcher, et al. Latent space non-linear statistics. arXiv, abs/1805.07632, 2018.","title":"Latent space non-linear statistics","authors":"L Kühnel, T Fletcher","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Lee et al., 2018] H. Lee, H. Tseng, et al. Diverse image- to-image translation via disentangled representations. In ECCV, 2018.","title":"Diverse imageto-image translation via disentangled representations","authors":"H Lee, H Tseng","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Li et al., 2019] X. Li, S. Liu, et al. Learning linear trans- formations for fast image and video style transfer. In IEEE/CVF CVPR, 2019.","title":"Learning linear transformations for fast image and video style transfer","authors":"Li","year":"2019","venue":"IEEE/CVF CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Liu et al., 2017] M. Liu, T. Breuel, and J. Kautz. Unsuper- vised image-to-image translation networks. In NeurIPS, 2017.","title":"Unsupervised image-to-image translation networks","authors":"M Liu, T Breuel, J Kautz","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Luan et al., 2017] F. Luan, S. Paris, et al. Deep photo style transfer. In IEEE/CVF CVPR, 2017.","title":"Deep photo style transfer","authors":"F Luan, S Paris","year":"2017","venue":"IEEE/CVF CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Marks et al., 1997] J. Marks, B. Andalman, et al. Design galleries: a general approach to setting parameters for com- puter graphics and animation. In SIGGRAPH, 1997.","title":"Design galleries: a general approach to setting parameters for computer graphics and animation","authors":"J Marks, B Andalman","year":"1997","venue":"SIGGRAPH","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Omiya et al., 2019] M. Omiya, Y. Horiuchi, et al. Optimization-based data generation for photo enhancement. In CVPR NITRE Workshop, 2019.","title":"Optimization-based data generation for photo enhancement","authors":"M Omiya, Y Horiuchi","year":"2019","venue":"CVPR NITRE Workshop","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Reed et al., 2016] S. Reed, Z. Akata, et al. Generative adver- sarial text to image synthesis. In ICML, 2016.","title":"Generative adversarial text to image synthesis","authors":"S Reed, Z Akata","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Ryokai et al., 2004] K. Ryokai, S. Marti, and H. Ishii. I/O brush: drawing with everyday objects as ink. In ACM CHI, 2004.","title":"I/O brush: drawing with everyday objects as ink","authors":"K Ryokai, S Marti, H Ishii","year":"2004","venue":"ACM CHI","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Salter and Fawcett, 1993] K. Salter and R. Fawcett. The ART test of interaction: A robust and powerful rank test of interaction in factorial models. Commun. Stat. Simul. Comput., 22(1), 1993.","title":"The ART test of interaction: A robust and powerful rank test of interaction in factorial models","authors":"K Salter, R Fawcett","year":"1993","venue":"Commun. Stat. Simul. Comput","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Savva et al., 2015] M. Savva, A. Chang, and P. Hanrahan. Semantically-enriched 3d models for common-sense knowl- edge. In CVPR VMC Workshop, 2015.","title":"Semantically-enriched 3d models for common-sense knowledge","authors":"M Savva, A Chang, P Hanrahan","year":"2015","venue":"CVPR VMC Workshop","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Sedlmair et al., 2014] M. Sedlmair, C. Heinzl, et al. Visual parameter space analysis: A conceptual framework. IEEE Trans. Vis. Comput. Graph., 20(12), 2014.","title":"Visual parameter space analysis: A conceptual framework","authors":"M Sedlmair, C Heinzl","year":"2014","venue":"IEEE Trans. Vis. Comput. Graph","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Shahriari et al., 2016] B. Shahriari, K. Swersky, et al. Taking the human out of the loop: A review of bayesian optimiza- tion. Proc. IEEE, 104(1), 2016.","title":"Taking the human out of the loop: A review of bayesian optimization","authors":"B Shahriari, K Swersky","year":"2016","venue":"Proc. IEEE","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Shi et al., 2020] L. Shi, B. Li, et al. MATch: differentiable material graphs for procedural material capture. ACM Trans. Graph., 39(6), 2020.","title":"MATch: differentiable material graphs for procedural material capture","authors":"L Shi, B Li","year":"2020","venue":"ACM Trans. Graph","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Shneiderman, 2007] B. Shneiderman. Creativity support tools: accelerating discovery and innovation. Commun. ACM, 50(12), 2007.","title":"Creativity support tools: accelerating discovery and innovation","authors":"B Shneiderman","year":"2007","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Takagi, 2001] H. Takagi. Interactive evolutionary compu- tation: fusion of the capabilities of ec optimization and human evaluation. Proc. IEEE, 89(9), 2001.","title":"Interactive evolutionary computation: fusion of the capabilities of ec optimization and human evaluation","authors":"H Takagi","year":"2001","venue":"Proc. IEEE","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Talton et al., 2009] J. Talton, D. Gibson, et al. Exploratory modeling with collaborative design spaces. ACM Trans. Graph., 28(5), 2009.","title":"Exploratory modeling with collaborative design spaces","authors":"J Talton, D Gibson","year":"2009","venue":"ACM Trans. Graph","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Talton et al., 2011] J. Talton, Y. Lou, et al. Metropolis pro- cedural modeling. ACM Trans. Graph., 30(2), 2011.","title":"Metropolis procedural modeling","authors":"J Talton, Y Lou","year":"2011","venue":"ACM Trans. Graph","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Terry and Mynatt, 2002] M. Terry and E. Mynatt. Side views: persistent, on-demand previews for open-ended tasks. In ACM UIST, 2002.","title":"Side views: persistent, on-demand previews for open-ended tasks","authors":"M Terry, E Mynatt","year":"2002","venue":"ACM UIST","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Tweedie, 1995] L. Tweedie. Interactive visualisation arti- facts: How can abstractions inform design? In BCS HCI, 1995.","title":"Interactive visualisation artifacts: How can abstractions inform design?","authors":"L Tweedie","year":"1995","venue":"BCS HCI","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Xiao et al., 2018] T. Xiao, J. Hong, and J. Ma. ELEGANT: exchanging latent encodings with GAN for transferring multiple face attributes. In ECCV, 2018.","title":"ELEGANT: exchanging latent encodings with GAN for transferring multiple face attributes","authors":"T Xiao, J Hong, J Ma","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Yaniv et al., 2019] J. Yaniv, Y. Newman, and A. Shamir. The face of art: landmark detection and geometric style in por- traits. ACM Trans. Graph., 38(4), 2019.","title":"The face of art: landmark detection and geometric style in portraits","authors":"J Yaniv, Y Newman, A Shamir","year":"2019","venue":"ACM Trans. Graph","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Yümer and Mitra, 2016] M. Yümer and N. Mitra. Spectral style transfer for human motion between independent ac- tions. ACM Trans. Graph., 35(4), 2016.","title":"Spectral style transfer for human motion between independent actions","authors":"M Yümer, N Mitra","year":"2016","venue":"ACM Trans. Graph","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Zhang et al., 2017] H. Zhang, T. Xu, and H. Li. StackGAN: text to photo-realistic image synthesis with stacked genera- tive adversarial networks. In IEEE ICCV, 2017.","title":"StackGAN: text to photo-realistic image synthesis with stacked generative adversarial networks","authors":"H Zhang, T Xu, H Li","year":"2017","venue":"IEEE ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}