{"type":"ScholarlyArticle","title":"Learning to Write Stylized Chinese Characters by Reading a Handful of Examples","authors":[{"name":"Danyang Sun","slug":"danyang-sun","given":"Danyang","family":"Sun","affiliation":"Department of Computer Science and Technology , Tsinghua Lab of Brain and Intelligence State Key Lab for Intell. Tech & Sys. , BNRist Lab , Tsinghua University , 100084 , China"},{"name":"Tongzheng Ren","slug":"tongzheng-ren","given":"Tongzheng","family":"Ren","affiliation":"Department of Computer Science and Technology , Tsinghua Lab of Brain and Intelligence State Key Lab for Intell. Tech & Sys. , BNRist Lab , Tsinghua University , 100084 , China"},{"name":"Chongxuan Li","slug":"chongxuan-li","given":"Chongxuan","family":"Li","affiliation":"Department of Computer Science and Technology , Tsinghua Lab of Brain and Intelligence State Key Lab for Intell. Tech & Sys. , BNRist Lab , Tsinghua University , 100084 , China"},{"name":"Hang Su","slug":"hang-su","given":"Hang","family":"Su","affiliation":"Department of Computer Science and Technology , Tsinghua Lab of Brain and Intelligence State Key Lab for Intell. Tech & Sys. , BNRist Lab , Tsinghua University , 100084 , China"},{"name":"Jun Zhu","slug":"jun-zhu","given":"Jun","family":"Zhu","affiliation":"Department of Computer Science and Technology , Tsinghua Lab of Brain and Intelligence State Key Lab for Intell. Tech & Sys. , BNRist Lab , Tsinghua University , 100084 , China"}],"abstract":"Automatically writing stylized characters is an attractive yet challenging task, especially for Chinese characters with complex shapes and structures. Most current methods are restricted to generate stylized characters already present in the training set, but required to retrain the model when generating characters of new styles. In this paper, we develop a novel framework of Style-Aware Variational Auto-Encoder (SA-VAE), which disentangles the content-relevant and style-relevant components of a Chinese character feature with a novel intercross pair-wise optimization method. In this case, our method can generate Chinese characters flexibly by reading a few examples. Experiments demonstrate that our method has a powerful one-shot/few-shot generalization ability by inferring the style representation, which is the first attempt to learn to write new-style Chinese characters by observing only one or a few examples.","doi":"10.24963/ijcai.2018/128","url":"https://ijcai.org/proceedings/2018/128","pdf":"https://ijcai.org/proceedings/2018/0128.pdf","markdown":"https://ijcai.org/proceedings/2018/128.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/128","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":128,"pages":{"first":920,"last":927},"keywords":["Computer Vision: Computer Vision","Computer Vision: Statistical Methods and Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Donahue et al., 2016] Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. arXiv preprint arXiv:1605.09782, 2016.","title":"Adversarial feature learning","authors":"Jeff Donahue, Philipp Krähenbühl, Trevor Darrell","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dumoulin et al., 2016] Vincent Dumoulin, Ishmael Belg- hazi, Ben Poole, Alex Lamb, Martin Arjovsky, Olivier Mastropietro, and Aaron Courville. Adversarially learned inference. arXiv preprint arXiv:1606.00704, 2016.","title":"Adversarially learned inference","authors":"Ishmael Vincent Dumoulin, Ben Belghazi, Alex Poole, Martin Lamb, Olivier Arjovsky, Aaron Mastropietro, Courville","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Graves, 2013] Alex Graves. Generating sequences with re- current neural networks. arXiv preprint arXiv:1308.0850, 2013.","title":"Generating sequences with recurrent neural networks","authors":"Alex Graves","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Isola et al., 2016] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with (a) One-shot digit generation (b) One-shot English character generation (c) One-shot English character generation Figure 9: The characters in the dotted frames are as the style providers and the others are generated characters in each row. As the size of their dictionary is not large, we use one-shot embedding to label each character. The style is consistent in each row. conditional adversarial networks. CoRR, abs/1611.07004, 2016.","title":"Image-to-image translation with (a) One-shot digit generation (b) One-shot English character generation (c) One-shot English character generation Figure 9: The characters in the dotted frames are as the style providers and the others are generated characters in each row. As the size of their dictionary is not large, we use one-shot embedding to label each character. The style is consistent in each row. conditional adversarial networks","authors":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A Efros","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kingma and Welling, 2014] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. stat, 1050:10, 2014.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"stat","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lake et al., 2015] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learn- ing through probabilistic program induction. Science, 350(6266):1332-1338, 2015.","title":"Human-level concept learning through probabilistic program induction","authors":"Ruslan Brenden M Lake, Joshua B Salakhutdinov, Tenenbaum","year":"2015","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2017] Chongxuan Li, Jun Zhu, and Bo Zhang. Max-margin deep generative models for (semi-) super- vised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.","title":"Max-margin deep generative models for (semi-) supervised learning","authors":"Li","year":"2017","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lian et al., 2016] Zhouhui Lian, Bo Zhao, and Jianguo Xiao. Automatic generation of large-scale handwriting fonts via style learning. In SIGGRAPH ASIA 2016 Techni- cal Briefs, page 12. ACM, 2016.","title":"Automatic generation of large-scale handwriting fonts via style learning","authors":"Lian","year":"2016","venue":"SIGGRAPH ASIA 2016 Technical Briefs","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Pal and Chaudhuri, 2004] U Pal and BB Chaudhuri. Indian script character recognition: a survey. pattern Recognition, 37(9):1887-1899, 2004.","title":"U Pal and BB Chaudhuri. Indian script character recognition: a survey","authors":"Chaudhuri Pal","year":"2004","venue":"pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Radford et al., 2015] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.","title":"Unsupervised representation learning with deep convolutional generative adversarial networks","authors":"Radford","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Shi et al., 2017] Jiaxin Shi, Jianfei Chen, Jun Zhu, Shengyang Sun, Yucen Luo, Yihong Gu, and Yuhao Zhou. Zhusuan: A library for bayesian deep learning. arXiv preprint arXiv:1709.05870, 2017.","title":"Zhusuan: A library for bayesian deep learning","authors":"Jiaxin Shi, Jianfei Chen, Jun Zhu, Shengyang Sun, Yucen Luo, Yihong Gu, Yuhao Zhou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Tenenbaum and Freeman, 1997] Joshua B Tenenbaum and William T Freeman. Separating style and content. In Advances in neural information processing systems, pages 662-668, 1997.","title":"Separating style and content","authors":"Freeman ; Tenenbaum, Joshua B Tenenbaum, William T Freeman","year":"1997","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Tian, 2016] Yuchen Tian. Rewrite: Neural style trans- fer for chinese fonts, 2016. https://github.com/ kaonashi-tyc/Rewrite.","title":"Rewrite: Neural style transfer for chinese fonts","authors":"Yuchen Tian","year":"2016","venue":null,"doi":null,"url":"https://github.com/kaonashi-tyc/Rewrite","internal_url":null},{"position":16,"raw":"[Tian, 2017] Yuchen Tian. zi2zi: Master chinese calligraphy with conditional adversarial networks, 2017. https:// github.com/kaonashi-tyc/zi2zi.","title":"2zi: Master chinese calligraphy with conditional adversarial networks","authors":"Yuchen Tian","year":"2017","venue":null,"doi":null,"url":"https://github.com/kaonashi-tyc/zi2zi","internal_url":null},{"position":17,"raw":"[Wainwright et al., 2008] Martin J Wainwright, Michael I Jordan, et al. Graphical models, exponential families, and variational inference. Foundations and Trends R in Ma- chine Learning, 1(1-2):1-305, 2008.","title":"Graphical models, exponential families, and variational inference","authors":"J Martin, Michael I Wainwright, Jordan","year":"2008","venue":"Foundations and Trends R in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Xiao et al., 2017] Xuefeng Xiao, Lianwen Jin, Yafeng Yang, Weixin Yang, Jun Sun, and Tianhai Chang. Building fast and compact convolutional neural networks for offline handwritten chinese character recognition. arXiv preprint arXiv:1702.07975, 2017.","title":"Building fast and compact convolutional neural networks for offline handwritten chinese character recognition","authors":"Xuefeng Xiao, Lianwen Jin, Yafeng Yang, Weixin Yang, Jun Sun, Tianhai Chang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Xu et al., 2009] Songhua Xu, Tao Jin, Hao Jiang, and Fran- cis CM Lau. Automatic generation of personal chinese handwriting by capturing the characteristics of personal handwriting. In IAAI, 2009.","title":"Automatic generation of personal chinese handwriting by capturing the characteristics of personal handwriting","authors":"Songhua Xu, Tao Jin, Hao Jiang, Francis Cm Lau","year":"2009","venue":"IAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yu et al., 2015] Fisher Yu, Ari Seff, Yinda Zhang, Shu- ran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.","title":"Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop","authors":"Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, Jianxiong Xiao","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zeiler et al., 2010] Matthew D Zeiler, Dilip Krishnan, Gra- ham W Taylor, and Rob Fergus. Deconvolutional net- works. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pages 2528-2535. IEEE, 2010.","title":"Deconvolutional networks","authors":"Dilip Matthew D Zeiler, Graham W Krishnan, Rob Taylor, Fergus","year":"2010","venue":"Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2017] Xu-Yao Zhang, Fei Yin, Yan-Ming Zhang, Cheng-Lin Liu, and Yoshua Bengio. Drawing and recognizing chinese characters with recurrent neural net- work. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 2017.","title":"Drawing and recognizing chinese characters with recurrent neural network","authors":"Xu-Yao Zhang, Fei Yin, Yan-Ming Zhang, Cheng-Lin Liu, Yoshua Bengio","year":"2017","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhong et al., 2016] Zhao Zhong, Xu-Yao Zhang, Fei Yin, and Cheng-Lin Liu. Handwritten chinese character recog- nition with spatial transformer and deep residual networks. In Pattern Recognition (ICPR), 2016 23rd International Conference on, pages 3440-3445. IEEE, 2016.","title":"Handwritten chinese character recognition with spatial transformer and deep residual networks","authors":"Zhao Zhong, Xu-Yao Zhang, Fei Yin, Cheng-Lin Liu","year":"2016","venue":"23rd International Conference on","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ziwei Liu and Tang, 2015] Xiaogang Wang Ziwei Liu, Ping Luo and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), 2015.","title":"Deep learning face attributes in the wild","authors":"Ziwei Liu, ; ] Xiaogang Tang, Ping Wang Ziwei Liu, Xiaoou Luo, Tang","year":"2015","venue":"Proceedings of International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zong and Zhu, 2014] Alfred Zong and Yuke Zhu. Stroke- bank: Automating personalized chinese handwriting gen- eration. In AAAI, pages 3024-3030, 2014.","title":"Strokebank: Automating personalized chinese handwriting generation","authors":"Zhu Zong, Alfred Zong, Yuke Zhu","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}