{"type":"ScholarlyArticle","title":"DP-Font: Chinese Calligraphy Font Generation Using Diffusion Model and Physical Information Neural Network","authors":[{"name":"Liguo Zhang","slug":"liguo-zhang","given":"Liguo","family":"Zhang","affiliation":"College of Computer Science and Technology , Harbin Engineering University , China; Modeling and Emulation in E-Government National Engineering Laboratory , China"},{"name":"Yalong Zhu","slug":"yalong-zhu","given":"Yalong","family":"Zhu","affiliation":"College of Computer Science and Technology , Harbin Engineering University , China; Modeling and Emulation in E-Government National Engineering Laboratory , China"},{"name":"Achref Benarab","slug":"achref-benarab","given":"Achref","family":"Benarab","affiliation":"College of Computer Science and Technology , Harbin Engineering University , China; Modeling and Emulation in E-Government National Engineering Laboratory , China"},{"name":"Yusen Ma","slug":"yusen-ma","given":"Yusen","family":"Ma","affiliation":"College of Computer Science and Technology , Harbin Engineering University , China; Modeling and Emulation in E-Government National Engineering Laboratory , China"},{"name":"Yuxin Dong","slug":"yuxin-dong","given":"Yuxin","family":"Dong","affiliation":"College of Computer Science and Technology , Harbin Engineering University , China; Modeling and Emulation in E-Government National Engineering Laboratory , China"},{"name":"Jianguo Sun","slug":"jianguo-sun","given":"Jianguo","family":"Sun","affiliation":"College of Computer Science and Technology , Harbin Engineering University , China; Modeling and Emulation in E-Government National Engineering Laboratory , China"}],"abstract":"As a typical visual art form, Chinese calligraphy has a long history and aesthetic value. However, current methods for generating Chinese fonts still struggle with complex character shapes and lack personalized writing styles. To address these issues, we propose a font generation method for Chinese Calligraphy based on diffusion model incorporating physical information neural network (PINN), which is named DP-Font. Firstly, the multi-attribute guidance is combined to guide the generation process of the diffusion model and introduce the critical constraint of stroke order in Chinese characters, aiming to significantly improve the quality of the generated results. We then incorporate physical constraints into the neural network loss function, utilizing physical equations to provide in-depth guidance and constraints on the learning process. By learning the movement rule of the nib and the diffusion pattern of the ink, DP-Font can generate personalized calligraphy styles. The generated fonts are very close to the calligraphers' works. Compared with existing deep learning-based techniques, DP-Font has made significant progress in enhancing the physical plausibility of the model, generating more realistic and high-quality results.","doi":"10.24963/ijcai.2024/863","url":"https://ijcai.org/proceedings/2024/863","pdf":"https://ijcai.org/proceedings/2024/0863.pdf","markdown":"https://ijcai.org/proceedings/2024/863.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/863","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"AI, Arts & Creativity","track":"AI, Arts & Creativity","number":863,"pages":{"first":7796,"last":7804},"keywords":["Application domains: Text, literature and creative language","Application domains: Images, movies and visual arts"],"videos":[],"references":[{"position":1,"raw":"[Cai et al., 2021] Shengze Cai, Zhicheng Wang, Sifan Wang, Paris Perdikaris, and George Em Karniadakis. Physics- informed neural networks for heat transfer problems. Jour- nal of Heat Transfer, 143(6):060801, 2021.","title":"Physicsinformed neural networks for heat transfer problems","authors":"Shengze Cai, Zhicheng Wang, Sifan Wang, Paris Perdikaris, George Em Karniadakis","year":"2021","venue":"Journal of Heat Transfer","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dhariwal and Nichol, 2021] Prafulla Dhariwal and Alexan- der Nichol. Diffusion models beat gans on image synthe- sis. Advances in neural information processing systems, 34:8780-8794, 2021.","title":"Diffusion models beat gans on image synthesis","authors":"Nichol Dhariwal","year":"2021","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Han, 2012] Jiantang Han. Chinese characters. Cambridge University Press, 2012.","title":"Jiantang Han. Chinese characters","authors":"Han","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hassan et al., 2023] Ammar Ul Hassan, Irfanullah Memon, and Jaeyoung Choi. Real-time high quality font generation with conditional font gan. Expert Systems with Applica- tions, 213:118907, 2023.","title":"Real-time high quality font generation with conditional font gan","authors":"Ammar Ul Hassan, Irfanullah Memon, Jaeyoung Choi","year":"2023","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2022] Haibin He, Xinyuan Chen, Chaoyue Wang, Juhua Liu, Bo Du, Dacheng Tao, and Yu Qiao. Diff-font: Diffusion model for robust one-shot font generation. arXiv preprint arXiv:2212.05895, 2022.","title":"Diff-font: Diffusion model for robust one-shot font generation","authors":"Haibin He, Xinyuan Chen, Chaoyue Wang, Juhua Liu, Bo Du, Dacheng Tao, Yu Qiao","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Heusel et al., 2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.","title":"Gans trained by a two time-scale update rule converge to a local nash equilibrium","authors":"Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Sepp Hochreiter","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ho and Salimans, 2022] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.","title":"Classifier-free diffusion guidance","authors":"Jonathan Ho, Tim Salimans","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ho et al., 2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.","title":"Denoising diffusion probabilistic models","authors":"Jonathan Ho, Ajay Jain, Pieter Abbeel","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiang et al., 2019] Yue Jiang, Zhouhui Lian, Yingmin Tang, and Jianguo Xiao. Scfont: Structure-guided chinese font generation via deep stacked networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 4015-4022, 2019.","title":"Scfont: Structure-guided chinese font generation via deep stacked networks","authors":"Yue Jiang, Zhouhui Lian, Yingmin Tang, Jianguo Xiao","year":"2019","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kong et al., 2022] Yuxin Kong, Canjie Luo, Weihong Ma, Qiyuan Zhu, Shenggao Zhu, Nicholas Yuan, and Lianwen Jin. Look closer to supervise better: one-shot font genera- tion via component-based discriminator. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13482-13491, 2022.","title":"Look closer to supervise better: one-shot font generation via component-based discriminator","authors":"Yuxin Kong, Canjie Luo, Weihong Ma, Qiyuan Zhu, Shenggao Zhu, Nicholas Yuan, Lianwen Jin","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2019] Ming-Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised image-to-image translation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10551-10560, 2019.","title":"Few-shot unsupervised image-to-image translation","authors":"Ming-Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, Jan Kautz","year":"2019","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Nichol et al., 2021] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photore- alistic image generation and editing with text-guided dif- fusion models. arXiv preprint arXiv:2112.10741, 2021.","title":"Glide: Towards photorealistic image generation and editing with text-guided diffusion models","authors":"Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, Mark Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Park et al., 2021] Song Park, Sanghyuk Chun, Junbum Cha, Bado Lee, and Hyunjung Shim. Multiple heads are better than one: Few-shot font generation with multiple localized experts. In Proceedings of the IEEE/CVF international conference on computer vision, pages 13900-13909, 2021.","title":"Multiple heads are better than one: Few-shot font generation with multiple localized experts","authors":"Park","year":"2021","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Raissi et al., 2019] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and in- verse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686- 707, 2019.","title":"Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations","authors":"Maziar Raissi, Paris Perdikaris, George E Karniadakis","year":"2019","venue":"Journal of Computational physics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ramesh et al., 2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.","title":"Hierarchical text-conditional image generation with clip latents","authors":"Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rombach et al., 2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.","title":"High-resolution image synthesis with latent diffusion models","authors":"Rombach","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sohl-Dickstein et al., 2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermody- namics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.","title":"Deep unsupervised learning using nonequilibrium thermodynamics","authors":"Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, Surya Ganguli","year":"2015","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Song et al., 2020] Jiaming Song, Chenlin Meng, and Ste- fano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.","title":"Denoising diffusion implicit models","authors":"Jiaming Song, Chenlin Meng, Stefano Ermon","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sun et al., 2017] Danyang Sun, Tongzheng Ren, Chongxun Li, Hang Su, and Jun Zhu. Learning to write stylized chi- nese characters by reading a handful of examples. arXiv preprint arXiv:1712.06424, 2017.","title":"Learning to write stylized chinese characters by reading a handful of examples","authors":"Danyang Sun, Tongzheng Ren, Chongxun Li, Hang Su, Jun Zhu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2018/128"},{"position":20,"raw":"[Tian, 2016] Yuchen Tian. Rewrite: Neural style transfer for chinese fonts. https://github.com/kaonashi-tyc/rewrite, 2016.","title":"Rewrite: Neural style transfer for chinese fonts","authors":"Yuchen Tian","year":null,"venue":null,"doi":null,"url":"https://github.com/kaonashi-tyc/rewrite,2016","internal_url":null},{"position":21,"raw":"[Tian, 2017] Yuchen Tian. Zi2zi: Master chinese calligraphy with conditional adversarial networks. https://github.com/kaonashi-tyc/zi2zi, 2017.","title":"Zi2zi: Master chinese calligraphy with conditional adversarial networks","authors":"Yuchen Tian","year":"2017","venue":null,"doi":null,"url":"https://github.com/kaonashi-tyc/zi2zi","internal_url":null},{"position":22,"raw":"[Von Rueden et al., 2021] Laura Von Rueden, Sebastian Mayer, Katharina Beckh, Bogdan Georgiev, Sven Giessel- bach, Raoul Heese, Birgit Kirsch, Julius Pfrommer, An- nika Pick, Rajkumar Ramamurthy, et al. Informed ma- chine learning-a taxonomy and survey of integrating prior knowledge into learning systems. IEEE Transactions on Knowledge and Data Engineering, 35(1):614-633, 2021.","title":"Informed machine learning-a taxonomy and survey of integrating prior knowledge into learning systems","authors":"Laura Von Rueden, Sebastian Mayer, Katharina Beckh, Bogdan Georgiev, Sven Giesselbach, Raoul Heese, Birgit Kirsch, Julius Pfrommer, Annika Pick, Rajkumar Ramamurthy","year":"2021","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transac- tions on image processing, 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli","year":"2004","venue":"IEEE transactions on image processing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2023a] Chi Wang, Min Zhou, Tiezheng Ge, Yuning Jiang, Hujun Bao, and Weiwei Xu. Cf-font: Con- tent fusion for few-shot font generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 1858-1867, 2023.","title":"Cf-font: Content fusion for few-shot font generation","authors":"Wang","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2023b] Luya Wang, Nor Azlin Hamidon, et al. Aesthetic and value study of inscriptions from the per- spective of calligraphy. Art and Performance Letters, 4(11):44-49, 2023.","title":"Aesthetic and value study of inscriptions from the perspective of calligraphy","authors":"Wang","year":"2023","venue":"Art and Performance Letters","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xie et al., 2021] Yangchen Xie, Xinyuan Chen, Li Sun, and Yue Lu. Dg-font: Deformable generative networks for unsupervised font generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5130-5140, 2021.","title":"Dg-font: Deformable generative networks for unsupervised font generation","authors":"Yangchen Xie, Xinyuan Chen, Li Sun, Yue Lu","year":"2021","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yoon et al., 2020] Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Gee- hyuk Lee. Speech gesture generation from the trimodal context of text, audio, and speaker identity. ACM Transac- tions on Graphics (TOG), 39(6):1-16, 2020.","title":"Speech gesture generation from the trimodal context of text, audio, and speaker identity","authors":"Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jaeyeon Lee, Jaehong Kim, Geehyuk Lee","year":"2020","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2018] Yexun Zhang, Ya Zhang, and Wenbin Cai. Separating style and content for generalized style transfer. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 8447-8455, 2018.","title":"Separating style and content for generalized style transfer","authors":"Zhang","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}