{"type":"ScholarlyArticle","title":"Learning to Generate Poetic Chinese Landscape Painting with Calligraphy","authors":[{"name":"Shaozu Yuan","slug":"shaozu-yuan","given":"Shaozu","family":"Yuan","affiliation":"JD AI , Beijing , China"},{"name":"Aijun Dai","slug":"aijun-dai","given":"Aijun","family":"Dai","affiliation":"JD AI , Beijing , China"},{"name":"Zhiling Yan","slug":"zhiling-yan","given":"Zhiling","family":"Yan","affiliation":"JD AI , Beijing , China"},{"name":"Ruixue Liu","slug":"ruixue-liu","given":"Ruixue","family":"Liu","affiliation":"JD AI , Beijing , China"},{"name":"Meng Chen","slug":"meng-chen","given":"Meng","family":"Chen","affiliation":"JD AI , Beijing , China"},{"name":"Baoyang Chen","slug":"baoyang-chen","given":"Baoyang","family":"Chen","affiliation":"Central Academy of Fine Arts , Beijing , China"},{"name":"Zhijie Qiu","slug":"zhijie-qiu","given":"Zhijie","family":"Qiu","affiliation":"Central Academy of Fine Arts , Beijing , China"},{"name":"Xiaodong He","slug":"xiaodong-he","given":"Xiaodong","family":"He","affiliation":"JD AI , Beijing , China"}],"abstract":"In this paper, we present a novel system (denoted as Polaca) to generate poetic Chinese landscape painting with calligraphy. Unlike previous single image-to-image painting generation, Polaca takes the classic poetry as input and outputs the artistic landscape painting image with the corresponding calligraphy. It is equipped with three different modules to complete the whole piece of landscape painting artwork: the first one is a text-to-image module to generate landscape painting image, the second one is an image-to-image module to generate stylistic calligraphy image, and the third one is an image fusion module to fuse the two images into a whole piece of aesthetic artwork.","doi":"10.24963/ijcai.2022/696","url":"https://ijcai.org/proceedings/2022/696","pdf":"https://ijcai.org/proceedings/2022/0696.pdf","markdown":"https://ijcai.org/proceedings/2022/696.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/696","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Exhibits","track":"AI and Arts - Exhibits","number":696,"pages":{"first":5019,"last":5022},"keywords":["Theory and philosophy of arts and creativity in AI systems: Support of human creativity","Theory and philosophy of arts and creativity in AI systems: Autonomous creative or artistic AI","Application domains: Images and visual arts","Application domains: Other domains of art or creativity"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987175","length_sec":85,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985318","length_sec":275,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bao et al., 2010] Hong Bao, Ye Liang, Hong-zhe Liu, and De Xu. A dataset for research of traditional chinese paint- ing and calligraphy images. In Proceedings of the 2nd International Conference on Information and Multimedia Technology. Singapore: IACSIT Press, pages 136-143, 2010.","title":"A dataset for research of traditional chinese painting and calligraphy images","authors":"Bao","year":"2010","venue":"Proceedings of the 2nd International Conference on Information and Multimedia Technology","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chung et al., 2014] Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evalua- tion of gated recurrent neural networks on sequence mod- eling. In NIPS 2014 Workshop on Deep Learning, Decem- ber 2014, 2014.","title":"Empirical evaluation of gated recurrent neural networks on sequence modeling","authors":"Chung","year":"2014","venue":"NIPS 2014 Workshop on Deep Learning, December 2014","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ding et al., 2021] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. arXiv preprint arXiv:2105.13290, 2021.","title":"Cogview: Mastering text-to-image generation via transformers","authors":"Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gao and Wu, 2020] Yiming Gao and Jiangqin Wu. Gan- based unpaired chinese character image translation via skeleton transformation and stroke rendering. In Proceed- ings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages 646-653, 2020.","title":"Ganbased unpaired chinese character image translation via skeleton transformation and stroke rendering","authors":"Wu ; ] Yiming Gao, Jiangqin Gao, Wu","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. Advances in neural information processing systems, 27, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Guo et al., 2019] Xiaoyu Guo, Meng Chen, Yang Song, Xi- aodong He, and Bowen Zhou. Automated thematic and emotional modern chinese poetry composition. In CCF In- ternational Conference on Natural Language Processing and Chinese Computing, pages 433-446. Springer, 2019.","title":"Automated thematic and emotional modern chinese poetry composition","authors":"Xiaoyu Guo, Meng Chen, Yang Song, Xiaodong He, Bowen Zhou","year":"2019","venue":"CCF International Conference on Natural Language Processing and Chinese Computing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2018] Bin He, Feng Gao, Daiqian Ma, Boxin Shi, and Ling-Yu Duan. Chipgan: A generative adversarial network for chinese ink wash painting style transfer. In Proceedings of the 26th ACM international conference on Multimedia, pages 1172-1180, 2018.","title":"Chipgan: A generative adversarial network for chinese ink wash painting style transfer","authors":"He","year":"2018","venue":"Proceedings of the 26th ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2015] Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991, 2015.","title":"Bidirectional lstm-crf models for sequence tagging","authors":"Zhiheng Huang, Wei Xu, Kai Yu","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kim et al., 2019] Junho Kim, Minjae Kim, Hyeonwoo Kang, and Kwanghee Lee. U-gat-it: Unsupervised genera- tive attentional networks with adaptive layer-instance nor- malization for image-to-image translation. arXiv preprint arXiv:1907.10830, 2019.","title":"U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation","authors":"Kim","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin et al., 2018] Daoyu Lin, Yang Wang, Guangluan Xu, Jun Li, and Kun Fu. Transform a simple sketch to a chinese painting by a multiscale deep neural network. Algorithms, 11(1):4, 2018.","title":"Transform a simple sketch to a chinese painting by a multiscale deep neural network","authors":"Lin","year":"2018","venue":"Algorithms","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2020] Ruixue Liu, Shaozu Yuan, Meng Chen, Baoyang Chen, Zhijie Qiu, and Xiaodong He. Maliang: An emotion-driven chinese calligraphy artwork composi- tion system. In Proceedings of the 28th ACM International Conference on Multimedia, pages 4394-4396, 2020.","title":"Maliang: An emotion-driven chinese calligraphy artwork composition system","authors":"Liu","year":"2020","venue":"Proceedings of the 28th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mihalcea and Tarau, 2004] Rada Mihalcea and Paul Tarau. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural lan- guage processing, pages 404-411, 2004.","title":"Textrank: Bringing order into text","authors":"Tarau Mihalcea, Rada Mihalcea, Paul Tarau","year":"2004","venue":"Proceedings of the 2004 conference on empirical methods in natural language processing","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Murck et al., 1991] Alfreda Murck, Wen C Fong, and Wen Fong. Words and images: Chinese poetry, calligraphy, and painting. Metropolitan Museum of Art, 1991.","title":"Words and images: Chinese poetry, calligraphy, and painting","authors":"Alfreda Murck, Wen C Fong, Wen Fong","year":"1991","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time ob- ject detection with region proposal networks. Advances in neural information processing systems, 28:91-99, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Shen et al., 2020] Lei Shen, Xiaoyu Guo, and Meng Chen. Compose like humans: Jointly improving the coherence and novelty for modern chinese poetry generation. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2020.","title":"Compose like humans: Jointly improving the coherence and novelty for modern chinese poetry generation","authors":"Lei Shen, Xiaoyu Guo, Meng Chen","year":"2020","venue":"2020 International Joint Conference on Neural Networks (IJCNN)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tomei et al., 2019] Matteo Tomei, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Art2real: Unfolding the reality of artworks via semantically-aware image- to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5849-5859, 2019.","title":"Art2real: Unfolding the reality of artworks via semantically-aware imageto-image translation","authors":"Matteo Tomei, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2016] Zhe Wang, Wei He, Hua Wu, Haiyang Wu, Wei Li, Haifeng Wang, and Enhong Chen. Chinese poetry generation with planning based neural network. arXiv preprint arXiv:1610.09889, 2016.","title":"Chinese poetry generation with planning based neural network","authors":"Zhe Wang, Wei He, Hua Wu, Haiyang Wu, Wei Li, Haifeng Wang, Enhong Chen","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Xu et al., 2018] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xi- aodong He. Attngan: Fine-grained text to image gener- ation with attentional generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1316-1324, 2018.","title":"Attngan: Fine-grained text to image generation with attentional generative adversarial networks","authors":"Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, Xiaodong He","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Xue, 2021] Alice Xue. End-to-end chinese landscape paint- ing creation using generative adversarial networks. In Pro- ceedings of the IEEE/CVF Winter Conference on Applica- tions of Computer Vision, pages 3863-3871, 2021.","title":"End-to-end chinese landscape painting creation using generative adversarial networks","authors":"Alice Xue","year":"2021","venue":"Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yuan et al., 2021] Shaozu Yuan, Ruixue Liu, Meng Chen, Baoyang Chen, Zhijie Qiu, and Xiaodong He. Learning to compose stylistic calligraphy artwork with emotions. In Proceedings of the 29th ACM International Conference on Multimedia, pages 3701-3709, 2021.","title":"Learning to compose stylistic calligraphy artwork with emotions","authors":"Yuan","year":"2021","venue":"Proceedings of the 29th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang et al., 2018] Yexun Zhang, Ya Zhang, and Wenbin Cai. Separating style and content for generalized style transfer. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 8447-8455, 2018.","title":"Separating style and content for generalized style transfer","authors":"Zhang","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhou et al., 2019] Le Zhou, Qiu-Feng Wang, Kaizhu Huang, and Cheng-Hung Lo. An interactive and gener- ative approach for chinese shanshui painting document. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 819-824. IEEE, 2019.","title":"An interactive and generative approach for chinese shanshui painting document","authors":"Le Zhou, Qiu-Feng Wang, Kaizhu Huang, Cheng-Hung Lo","year":"2019","venue":"2019 International Conference on Document Analysis and Recognition (ICDAR)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhu et al., 2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceed- ings of the IEEE international conference on computer vi- sion, pages 2223-2232, 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Zhu","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}