{"type":"ScholarlyArticle","title":"Compositional Zero-Shot Artistic Font Synthesis","authors":[{"name":"Xiang Li","slug":"xiang-li","given":"Xiang","family":"Li","affiliation":"School of Software , Shandong University"},{"name":"Lei Wu","slug":"lei-wu","given":"Lei","family":"Wu","affiliation":"School of Software , Shandong University"},{"name":"Changshuo Wang","slug":"changshuo-wang","given":"Changshuo","family":"Wang","affiliation":"School of Software , Shandong University"},{"name":"Lei Meng","slug":"lei-meng","given":"Lei","family":"Meng","affiliation":"School of Software , Shandong University; Shandong Research Institute of Industrial Technology"},{"name":"Xiangxu Meng","slug":"xiangxu-meng","given":"Xiangxu","family":"Meng","affiliation":"School of Software , Shandong University"}],"abstract":"Recently, many researchers have made remarkable achievements in the field of artistic font synthesis, with impressive glyph style and effect style in the results. However, due to less exploration in style disentanglement, it is difficult for existing methods to envision a kind of unseen style (glyph-effect) compositions of artistic font, and thus can only learn the seen style compositions. To solve this problem, we propose a novel compositional zero-shot artistic font synthesis gan (CAFS-GAN), which allows the synthesis of unseen style compositions by exploring the visual independence and joint compatibility of encoding semantics between glyph and effect. Specifically, we propose two contrast-based style encoders to achieve style disentanglement due to glyph and effect intertwining in the image. Meanwhile, to preserve more glyph and effect detail, we propose a generator based on hierarchical dual styles AdaIN to reorganize content-styles representations from structure to texture gradually. Extensive experiments demonstrate the superiority of our model in generating high-quality artistic font images with unseen style compositions against other state-of-the-art methods. The source code and data is available at moonlight03.github.io/CAFS-GAN/.","doi":"10.24963/ijcai.2023/122","url":"https://ijcai.org/proceedings/2023/122","pdf":"https://ijcai.org/proceedings/2023/0122.pdf","markdown":"https://ijcai.org/proceedings/2023/122.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/122","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":122,"pages":{"first":1098,"last":1106},"keywords":["Computer Vision: CV: Transfer, low-shot, semi- and un- supervised learning","Computer Vision: CV: Neural generative models, auto encoders, GANs"],"videos":[],"references":[{"position":1,"raw":"[Azadi et al., 2018] Samaneh Azadi, Matthew Fisher, Vladimir G Kim, Zhaowen Wang, Eli Shechtman, and Trevor Darrell. Multi-content gan for few-shot font style transfer. In CVPR, pages 7564-7573, 2018.","title":"Multi-content gan for few-shot font style transfer","authors":"Azadi","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2018] Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of dis- entanglement in vaes. In NIPS, pages 2615-2625, 2018.","title":"Isolating sources of disentanglement in vaes","authors":"Xuechen Ricky Tq Chen, Roger Li, David Grosse, Duvenaud","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2021] Xu Chen, Lei Wu, Minggang He, Lei Meng, and Xiangxu Meng. Mlfont: Few-shot chinese font generation via deep meta-learning. In ICMR, pages 37-45, 2021.","title":"Mlfont: Few-shot chinese font generation via deep meta-learning","authors":"Lei Xu Chen, Minggang Wu, Lei He, Xiangxu Meng, Meng","year":"2021","venue":"ICMR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Choi et al., 2020] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, pages 8188-8197, 2020.","title":"Stargan v2: Diverse image synthesis for multiple domains","authors":"Choi","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dong et al., 2022a] Pei Dong, Lei Wu, Lei Meng, and Xi- angxu Meng. Disentangled representations and hierar- chical refinement of multi-granularity features for text-to- image synthesis. In ICMR, pages 268-276, 2022.","title":"Disentangled representations and hierarchical refinement of multi-granularity features for text-toimage synthesis","authors":"Dong","year":"2022","venue":"ICMR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dong et al., 2022b] Pei Dong, Lei Wu, Lei Meng, and Xi- angxu Meng. Hr-prgan: High-resolution story visualiza- tion with progressive generative adversarial networks. In- formation Sciences, 614:548-562, 2022.","title":"Hr-prgan: High-resolution story visualization with progressive generative adversarial networks","authors":"Dong","year":"2022","venue":"Information Sciences","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Feng et al., 2022] Yaogong Feng, Xiaowen Huang, Pengbo Yang, Jian Yu, and Jitao Sang. Non-generative generalized zero-shot learning via task-correlated disentanglement and controllable samples synthesis. In CVPR, pages 9346- 9355, 2022.","title":"Non-generative generalized zero-shot learning via task-correlated disentanglement and controllable samples synthesis","authors":"Yaogong Feng, Xiaowen Huang, Pengbo Yang, Jian Yu, Jitao Sang","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gao et al., 2018] Rui Gao, Xingsong Hou, Jie Qin, Li Liu, Fan Zhu, and Zhao Zhang. A joint generative model for zero-shot learning. In ECCV, pages 631-646, 2018.","title":"A joint generative model for zero-shot learning","authors":"Rui Gao, Xingsong Hou, Jie Qin, Li Liu, Fan Zhu, Zhao Zhang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gao et al., 2019] Yue Gao, Yuan Guo, Zhouhui Lian, Ying- min Tang, and Jianguo Xiao. Artistic glyph image synthe- sis via one-stage few-shot learning. ACM Transactions on Graphics (TOG), 38(6):1-12, 2019.","title":"Artistic glyph image synthesis via one-stage few-shot learning","authors":"Yue Gao, Yuan Guo, Zhouhui Lian, Yingmin Tang, Jianguo Xiao","year":"2019","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gatys et al., 2016] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In CVPR, pages 2414-2423, 2016.","title":"Image style transfer using convolutional neural networks","authors":"Leon A Gatys, Alexander S Ecker, Matthias Bethge","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ge et al., 2021] Yunhao Ge, Sami Abu-El-Haija, Gan Xin, and Laurent Itti. Zero-shot synthesis with group- supervised learning. In ICLR, 2021.","title":"Zero-shot synthesis with groupsupervised learning","authors":"Yunhao Ge, Sami Abu-El-Haija, Gan Xin, Laurent Itti","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial networks. In NIPS, page 2672-2680, 2014.","title":"Generative adversarial networks","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Guo et al., 2020] Wenya Guo, Ying Zhang, Xiangrui Cai, Lei Meng, Jufeng Yang, and Xiaojie Yuan. Ld-man: Layout-driven multimodal attention network for online news sentiment recognition. Transactions on Multimedia, 23:1785-1798, 2020.","title":"Ld-man: Layout-driven multimodal attention network for online news sentiment recognition","authors":"Guo","year":"2020","venue":"Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Han et al., 2021] Yuxuan Han, Jiaolong Yang, and Ying Fu. Disentangled face attribute editing via instance-aware la- tent space search. In IJCAI, pages 715-721, 2021.","title":"Disentangled face attribute editing via instance-aware latent space search","authors":"Yuxuan Han, Jiaolong Yang, Ying Fu","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/99"},{"position":15,"raw":"[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Sain- ing Xie, and Ross Girshick. Momentum contrast for un- supervised visual representation learning. In CVPR, pages 9729-9738, 2020.","title":"Momentum contrast for unsupervised visual representation learning","authors":"Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Higgins et al., 2017] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained varia- tional framework. In ICLR, 2017.","title":"beta-vae: Learning basic visual concepts with a constrained variational framework","authors":"Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, Alexander Lerchner","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hong et al., 2022] Ziming Hong, Shiming Chen, Guo-Sen Xie, Wenhan Yang, Jian Zhao, Yuanjie Shao, Qinmu Peng, and Xinge You. Semantic compression embedding for generative zero-shot learning. In IJCAI, pages 956-963, 2022.","title":"Semantic compression embedding for generative zero-shot learning","authors":"Ziming Hong, Shiming Chen, Guo-Sen Xie, Wenhan Yang, Jian Zhao, Yuanjie Shao, Qinmu Peng, Xinge You","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/134"},{"position":18,"raw":"[Huang and Belongie, 2017] Xun Huang and Serge Be- longie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, pages 1501-1510, 2017.","title":"Arbitrary style transfer in real-time with adaptive instance normalization","authors":"Belongie Huang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Karthik et al., 2022] Shyamgopal Karthik, Massimiliano Mancini, and Zeynep Akata. Kg-sp: Knowledge guided simple primitives for open world compositional zero-shot learning. In CVPR, pages 9336-9345, 2022.","title":"Kg-sp: Knowledge guided simple primitives for open world compositional zero-shot learning","authors":"Karthik","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2020a] Wei Li, Yongxing He, Yanwei Qi, Zejian Li, and Yongchuan Tang. Fet-gan: Font and effect transfer via k-shot adaptive instance normalization. In AAAI, pages 1717-1724, 2020.","title":"Fet-gan: Font and effect transfer via k-shot adaptive instance normalization","authors":"Li","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2020b] Yuheng Li, Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. Mixnmatch: Multifactor dis- entanglement and encoding for conditional image gener- ation. In CVPR, pages 8039-8048, 2020.","title":"Mixnmatch: Multifactor disentanglement and encoding for conditional image generation","authors":"Li","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li et al., 2021] Xiangxian Li, Haokai Ma, Lei Meng, and Xiangxu Meng. Comparative study of adversarial training methods for long-tailed classification. In ACM MM Work- Shop, pages 1-7, 2021.","title":"Comparative study of adversarial training methods for long-tailed classification","authors":"Li","year":"2021","venue":"ACM MM Work-Shop","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Li et al., 2022a] Jingyu Li, Haokai Ma, Xiangxian Li, Zhuang Qi, Lei Meng, and Xiangxu Meng. Unsuper- vised contrastive masking for visual haze classification. In ICMR, pages 426-434, 2022.","title":"Unsupervised contrastive masking for visual haze classification","authors":"Li","year":"2022","venue":"ICMR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Li et al., 2022b] Xiang Li, Lei Wu, Xu Chen, Lei Meng, and Xiangxu Meng. Dse-net: Artistic font image synthesis via disentangled style encoding. In ICME, pages 1-6, 2022.","title":"Dse-net: Artistic font image synthesis via disentangled style encoding","authors":"Li","year":"2022","venue":"ICME","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Li et al., 2022c] Xiangyu Li, Xu Yang, Kun Wei, Cheng Deng, and Muli Yang. Siamese contrastive embedding network for compositional zero-shot learning. In CVPR, pages 9326-9335, 2022.","title":"Siamese contrastive embedding network for compositional zero-shot learning","authors":"Li","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Lin et al., 2020] Chuang Lin, Sicheng Zhao, Lei Meng, and Tat-Seng Chua. Multi-source domain adaptation for visual sentiment classification. In AAAI, pages 2661-2668, 2020.","title":"Multi-source domain adaptation for visual sentiment classification","authors":"Chuang Lin, Sicheng Zhao, Lei Meng, Tat-Seng Chua","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Lin et al., 2022] Chung-Ching Lin, Kevin Lin, Lijuan Wang, Zicheng Liu, and Linjie Li. Cross-modal represen- tation learning for zero-shot action recognition. In CVPR, pages 19978-19988, 2022.","title":"Cross-modal representation learning for zero-shot action recognition","authors":"Chung-Ching Lin, Kevin Lin, Lijuan Wang, Zicheng Liu, Linjie Li","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Luo et al., 2022] Canjie Luo, Lianwen Jin, and Jingdong Chen. Siman: Exploring self-supervised representation learning of scene text via similarity-aware normalization. In CVPR, pages 1039-1048, 2022.","title":"Siman: Exploring self-supervised representation learning of scene text via similarity-aware normalization","authors":"Luo","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Mancini et al., 2021] Massimiliano Mancini, Muham- mad Ferjad Naeem, Yongqin Xian, and Zeynep Akata. Open world compositional zero-shot learning. In CVPR, pages 5222-5230, 2021.","title":"Open world compositional zero-shot learning","authors":"Massimiliano Mancini, Muhammad Ferjad Naeem, Yongqin Xian, Zeynep Akata","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Men et al., 2019] Yifang Men, Zhouhui Lian, Yingmin Tang, and Jianguo Xiao. Dyntypo: Example-based dy- namic text effects transfer. In CVPR, pages 5870-5879, 2019.","title":"Dyntypo: Example-based dynamic text effects transfer","authors":"Yifang Men, Zhouhui Lian, Yingmin Tang, Jianguo Xiao","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Naeem et al., 2021] Muhammad Ferjad Naeem, Yongqin Xian, Federico Tombari, and Zeynep Akata. Learning graph embeddings for compositional zero-shot learning. In CVPR, pages 953-962, 2021.","title":"Learning graph embeddings for compositional zero-shot learning","authors":"Naeem","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Saini et al., 2022] Nirat Saini, Khoi Pham, and Abhinav Shrivastava. Disentangling visual embeddings for at- tributes and objects. In CVPR, pages 13658-13667, 2022.","title":"Disentangling visual embeddings for attributes and objects","authors":"Nirat Saini, Khoi Pham, Abhinav Shrivastava","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Singh and Krishnan, 2020] Saurabh Singh and Shankar Kr- ishnan. Filter response normalization layer: Eliminating batch dependence in the training of deep neural networks. In CVPR, pages 11237-11246, 2020.","title":"Filter response normalization layer: Eliminating batch dependence in the training of deep neural networks","authors":"Saurabh Singh, Shankar Krishnan","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wu et al., 2020] Lei Wu, Xi Chen, Lei Meng, and Xiangxu Meng. Multitask adversarial learning for chinese font style transfer. In IJCNN, pages 1-8, 2020.","title":"Multitask adversarial learning for chinese font style transfer","authors":"Lei Wu, Xi Chen, Lei Meng, Xiangxu Meng","year":"2020","venue":"IJCNN","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Yang et al., 2016] Shuai Yang, Jiaying Liu, Zhouhui Lian, and Zongming Guo. Awesome typography: Statistics- based text effects transfer. CoRR, abs/1611.09026, 2016.","title":"Awesome typography: Statisticsbased text effects transfer","authors":"Shuai Yang, Jiaying Liu, Zhouhui Lian, Zongming Guo","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Yang et al., 2019a] Shuai Yang, Jiaying Liu, Wenjing Wang, and Zongming Guo. Tet-gan: Text effects transfer via stylization and destylization. In AAAI, pages 1238- 1245, 2019.","title":"Tet-gan: Text effects transfer via stylization and destylization","authors":"Yang","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Yang et al., 2019b] Shuai Yang, Zhangyang Wang, Zhaowen Wang, Ning Xu, Jiaying Liu, and Zong- ming Guo. Controllable artistic text style transfer via shape-matching gan. In CVPR, pages 4442-4451, 2019.","title":"Controllable artistic text style transfer via shape-matching gan","authors":"Yang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Yang et al., 2021] Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Disentangled representation learning via neural structural causal models. In CVPR, pages 9593-9602, 2021.","title":"Causalvae: Disentangled representation learning via neural structural causal models","authors":"Yang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Yang et al., 2022] Muli Yang, Chenghao Xu, Aming Wu, and Cheng Deng. A decomposable causal view of com- positional zero-shot learning. IEEE Transactions on Mul- timedia, pages 1-11, 2022.","title":"A decomposable causal view of compositional zero-shot learning","authors":"Muli Yang, Chenghao Xu, Aming Wu, Cheng Deng","year":"2022","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Zhang et al., 2018] Yuting Zhang, Yijie Guo, Yixin Jin, Yi- jun Luo, Zhiyuan He, and Honglak Lee. Unsupervised dis- covery of object landmarks as structural representations. In CVPR, pages 2694-2703, 2018.","title":"Unsupervised discovery of object landmarks as structural representations","authors":"Yuting Zhang, Yijie Guo, Yixin Jin, Yijun Luo, Zhiyuan He, Honglak Lee","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}