{"type":"ScholarlyArticle","title":"Evaluating Brush Movements for Chinese Calligraphy: A Computer Vision Based Approach","authors":[{"name":"Pengfei Xu","slug":"pengfei-xu","given":"Pengfei","family":"Xu","affiliation":"School of Information Science and Technology , Northwest University , China"},{"name":"Lei Wang","slug":"lei-wang","given":"Lei","family":"Wang","affiliation":"School of Information Science and Technology , Northwest University , China"},{"name":"Ziyu Guan","slug":"ziyu-guan","given":"Ziyu","family":"Guan","affiliation":"School of Information Science and Technology , Northwest University , China"},{"name":"Xia Zheng","slug":"xia-zheng","given":"Xia","family":"Zheng","affiliation":"Department of Cultural Heritage and Museology , Zhejiang University , China"},{"name":"Xiaojiang Chen","slug":"xiaojiang-chen","given":"Xiaojiang","family":"Chen","affiliation":"School of Information Science and Technology , Northwest University , China"},{"name":"Zhanyong Tang","slug":"zhanyong-tang","given":"Zhanyong","family":"Tang","affiliation":"School of Information Science and Technology , Northwest University , China"},{"name":"Dingyi Fang","slug":"dingyi-fang","given":"Dingyi","family":"Fang","affiliation":"School of Information Science and Technology , Northwest University , China"},{"name":"Xiaoqing Gong","slug":"xiaoqing-gong","given":"Xiaoqing","family":"Gong","affiliation":"School of Information Science and Technology , Northwest University , China"},{"name":"Zheng Wang","slug":"zheng-wang","given":"Zheng","family":"Wang","affiliation":"MetaLab , School of Computing and Communications , Lancaster University , U. K."}],"abstract":"Chinese calligraphy is a popular, highly esteemed art form in the Chinese cultural sphere and worldwide. Ink brushes are the traditional writing tool for Chinese calligraphy and the subtle nuances of brush movements have a great impact on the aesthetics of the written characters. However, mastering the brush movement is a challenging task for many calligraphy learners as it requires many years’ practice and expert supervision. This paper presents a novel approach to help Chinese calligraphy learners to quantify the quality of brush movements without expert involvement. Our approach extracts the brush trajectories from a video stream; it then compares them with example templates of reputed calligraphers to produce a score for the writing quality. We achieve this by first developing a novel neural network to extract the spatial and temporal movement features from the video stream. We then employ methods developed in the computer vision and signal processing domains to track the brush movement trajectory and calculate the score. We conducted extensive experiments and user studies to evaluate our approach. Experimental results show that our approach is highly accurate in identifying brush movements, yielding an average accuracy of 90%, and the generated score is within 3% of errors when compared to the one given by human experts.","doi":"10.24963/ijcai.2018/146","url":"https://ijcai.org/proceedings/2018/146","pdf":"https://ijcai.org/proceedings/2018/0146.pdf","markdown":"https://ijcai.org/proceedings/2018/146.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/146","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":146,"pages":{"first":1050,"last":1056},"keywords":["Machine Learning Applications: Other Applications","Machine Learning: Neural Networks","Machine Learning: Deep Learning","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Computer Vision: Motion and Tracking","Machine Learning: Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Brink et al., 2012] A. A. Brink, J. Smit, and M. L. Bulacu. Writer identification using directional ink-trace width mea- surements. Pattern Recognition, 45(1):162-171, 2012.","title":"Writer identification using directional ink-trace width measurements","authors":"A A Brink, J Smit, M L Bulacu","year":"2012","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chang et al., 2017] Xiaojun Chang, Zhigang Ma, and Yang Yi. Bi-level semantic representation analysis for multimedia event detection. IEEE Trans Cybern, 47(5):1180-1197, 2017.","title":"Bi-level semantic representation analysis for multimedia event detection","authors":"Xiaojun Chang, Zhigang Ma, Yang Yi","year":"2017","venue":"IEEE Trans Cybern","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Donahue et al., 2015] Jeff Donahue, Lisa Anne Hendricks, and Guadarrama. Long-term recurrent convolutional net- works for visual recognition and description. In CVPR, pages 677-691, 2015.","title":"Long-term recurrent convolutional networks for visual recognition and description","authors":"Jeff Donahue, Lisa Anne Hendricks, Guadarrama","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gao et al., 2017] Junyu Gao, Qi Wang, and Yuan Yuan. Em- bedding structured contour and location prior in siamesed fully convolutional networks for road detection. In IEEE In- ternational Conference on Robotics and Automation, pages 219-224, 2017.","title":"Embedding structured contour and location prior in siamesed fully convolutional networks for road detection","authors":"Gao","year":"2017","venue":"IEEE International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Greff et al., 2017] Klaus Greff, Rupesh K. Srivastava, Jan Koutník, Bas R. Steunebrink, and Jürgen Schmidhuber. Lstm: A search space odyssey. IEEE TNNLS, 28(10):2222- 2232, 2017.","title":"Lstm: A search space odyssey","authors":"Klaus Greff, K Rupesh, Jan Srivastava, Koutník, R Bas, Jürgen Steunebrink, Schmidhuber","year":"2017","venue":"IEEE TNNLS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kalal et al., 2012] Zdenek Kalal, Krystian Mikolajczyk, and Jiri Matas. Tracking-learning-detection. IEEE TPAMI, 34(7):1409-22, 2012.","title":"Tracking-learning-detection","authors":"Zdenek Kalal, Krystian Mikolajczyk, Jiri Matas","year":"2012","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Karpathy et al., 2014] Andrej Karpathy, George Toderici, and Shetty. Large-scale video classification with convolu- tional neural networks. In CVPR, pages 1725-1732, 2014.","title":"Large-scale video classification with convolutional neural networks","authors":"Karpathy","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In International Conference on Neural Information Processing Systems, pages 1097- 1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2014] Wei Li, Yuping Song, and Changle Zhou. Computationally evaluating and synthesizing Chinese cal- ligraphy. Elsevier Science Publishers B. V., 2014.","title":"Computationally evaluating and synthesizing Chinese calligraphy","authors":"Wei Li, Yuping Song, Changle Zhou","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin et al., 2013] Yuan Lin, Jiangqin Wu, and Pengcheng Gao. Lsh-based large scale chinese calligraphic character recognition. In Acm/ieee-Cs Joint Conference on Digital Libraries, pages 323-330, 2013.","title":"Lsh-based large scale chinese calligraphic character recognition","authors":"Yuan Lin, Jiangqin Wu, Pengcheng Gao","year":"2013","venue":"Acm/ieee-Cs Joint Conference on Digital Libraries","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lu et al., 2011] Weiming Lu, Jiangqin Wu, and Baogang Wei. Efficient shape matching for chinese calligraphic character retrieval. Journal of Zhejiang University-Science, 12(11):873-884, 2011.","title":"Efficient shape matching for chinese calligraphic character retrieval","authors":"Weiming Lu, Jiangqin Wu, Baogang Wei","year":"2011","venue":"Journal of Zhejiang University-Science","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ng et al., 2015] Yue Hei Ng, Matthew Hausknecht, and Sud- heendra Vijayanarasimhan. Beyond short snippets: Deep networks for video classification. CVPR, 16(4):4694-4702, 2015.","title":"Beyond short snippets: Deep networks for video classification","authors":"Hei Yue, Matthew Ng, Sudheendra Hausknecht, Vijayanarasimhan","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Qin et al., 2015] Lei Qin, Qiong Hu, and Qingming Huang. Action recognition using trajectories of spatio-tempral fea- ture points. Chinese Journal of Computers, 37(6):1281- 1288, 2015.","title":"Action recognition using trajectories of spatio-tempral feature points","authors":"Lei Qin, Qiong Hu, Qingming Huang","year":"2015","venue":"Chinese Journal of Computers","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rasouli and Attaran, 2012] Atousa Rasouli and Mohammad Attaran. Improve the quality of traditional education of calligraphy in iran by using of collaborative e-learning. Procedia -Social and Behavioral Sciences, 51:433-443, 2012.","title":"Atousa Rasouli and Mohammad Attaran. Improve the quality of traditional education of calligraphy in iran by using of collaborative e-learning","authors":"Attaran Rasouli","year":"2012","venue":"Procedia -Social and Behavioral Sciences","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ren et al., 2017a] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE TPAMI, 39(6):1137-1149, 2017.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2017","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ren et al., 2017b] Zhou Ren, Xiaoyu Wang, and Ning Zhang. Deep reinforcement learning-based image cap- tioning with embedding reward. arXiv, pages 1151-1159, 2017.","title":"Deep reinforcement learning-based image captioning with embedding reward","authors":"Ren","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shi et al., 2016] Zhenghao Shi, Binxin Xu, and Xia Zheng. An integrated method for ancient chinese tablet images de- noising based on assemble of multiple image smoothing filters. Multimedia Tools & Applications, 75(19):12245- 12261, 2016.","title":"An integrated method for ancient chinese tablet images denoising based on assemble of multiple image smoothing filters","authors":"Shi","year":"2016","venue":"Multimedia Tools & Applications","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in Neural Informa- tion Processing Systems, 1(4):568-576, 2014.","title":"Two-stream convolutional networks for action recognition in videos","authors":"Zisserman Simonyan","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2017a] Qi Wang, Junyu Gao, and Yuan Yuan. A joint convolutional neural networks and context transfer for street scenes labeling. IEEE TITS, PP(99):1-14, 2017.","title":"A joint convolutional neural networks and context transfer for street scenes labeling","authors":"Wang","year":"2017","venue":"IEEE TITS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2017b] Qi Wang, Jia Wan, and Yuan Yuan. Deep metric learning for crowdedness regression. IEEE TCSVT, PP(99):1-1, 2017.","title":"Deep metric learning for crowdedness regression","authors":"Wang","year":"2017","venue":"IEEE TCSVT","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2017c] Qi Wang, Jia Wan, and Yuan Yuan. Lo- cality constraint distance metric learning for traffic conges- tion detection. Pattern Recognition, 75, 2017.","title":"Locality constraint distance metric learning for traffic congestion detection","authors":"Wang","year":"2017","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xia et al., 2013] Yang Xia, Jiangqin Wu, and Pengcheng Gao. Ontology based model for chinese calligraphy synthe- sis. Computer Graphics Forum, 32(7):11-20, 2013.","title":"Ontology based model for chinese calligraphy synthesis","authors":"Yang Xia, Jiangqin Wu, Pengcheng Gao","year":"2013","venue":"Computer Graphics Forum","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu et al., 2012] Songhua Xu, Hao Jiang, and Francis C. M. Lau. Computationally evaluating and reproducing the beauty of chinese calligraphy. IEEE Intelligent Systems, 27(3):63-72, 2012.","title":"Computationally evaluating and reproducing the beauty of chinese calligraphy","authors":"Songhua Xu, Hao Jiang, Francis C M Lau","year":"2012","venue":"IEEE Intelligent Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2016] Pengfei Xu, Xia Zheng, and Chang Xiaojun. Artistic information extraction from chinese calligraphy works via shear-guided filter. Journal of Visual Communi- cation & Image Representation, 40(PB):791-807, 2016.","title":"Artistic information extraction from chinese calligraphy works via shear-guided filter","authors":"Pengfei Xu, Xia Zheng, Chang Xiaojun","year":"2016","venue":"Journal of Visual Communication & Image Representation","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yao et al., 2004] Fenghui Yao, Guifeng Shao, and Jianqiang Yi. Extracting the trajectory of writing brush in chinese character calligraphy. Engineering Applications of Artificial Intelligence, 17(6):631-644, 2004.","title":"Extracting the trajectory of writing brush in chinese character calligraphy","authors":"Fenghui Yao, Guifeng Shao, Jianqiang Yi","year":"2004","venue":"Engineering Applications of Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yoo, 2017] Jae Hyeon Yoo. Large-scale video classification guided by batch normalized lstm translator. arXiv, pages 1-7, 2017.","title":"Large-scale video classification guided by batch normalized lstm translator","authors":"Jae Hyeon, Yoo","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang and Nagy, 2012] Xiafen Zhang and George Nagy. Style comparisons in calligraphy. Document Recognition & Retrieval XIX, 8297(2):263-271, 2012.","title":"Style comparisons in calligraphy","authors":"Nagy Zhang","year":"2012","venue":"Document Recognition & Retrieval XIX","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang and Zhuang, 2012] Xiafen Zhang and Yueting Zhuang. Dynamic time warping for chinese calligraphic character matching and recognizing. Pattern Recognition Letters, 33(16):2262-2269, 2012.","title":"Xiafen Zhang and Yueting Zhuang. Dynamic time warping for chinese calligraphic character matching and recognizing","authors":"Zhuang Zhang","year":"2012","venue":"Pattern Recognition Letters","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zheng et al., 2016] Xia Zheng, Qiguang Miao, and Zheng- hao Shi. A new artistic information extraction method with multi channels and guided filters for calligraphy works. Multimedia Tools & Applications, 75(14):8719- 8744, 2016.","title":"A new artistic information extraction method with multi channels and guided filters for calligraphy works","authors":"Xia Zheng, Qiguang Miao, Zhenghao Shi","year":"2016","venue":"Multimedia Tools & Applications","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}