Zero-Shot Chinese Character Recognition with Stroke-Level Decomposition

1 Introduction

Chinese character recognition (CCR), which has been studied for many years, plays an essential role in many applications. Existing CCR methods usually rely on massive input data. For example, the HWDB1.0-1.1 database [Liu et al., 2013] provides more than two million handwritten samples collected from 720 writers, including 3,866 classes overall. However, there are 70,244 Chinese characters in total according to the latest Chinese national standard GB18030-2005 1 , thus collecting samples for each character is time-consuming.

∗Corresponding author

1 https://zh.wikipedia.org/wiki/GB 18030

Early work on CCR mainly relies on hand-crafted features [Su and Wang, 2003; Shi et al., 2003]. With the rapid development of deep learning, numerous CNN-based methods emerged and outperformed early traditional methods. Deep learning-based CCR methods can be divided into two categories: character-based methods and radical-based methods. Character-based methods treat each character as one class. For example, MCDNN [Cires¸an and Meier, 2015] ensembles results of eight deep nets and reaches human-level performance. DirectMap [Zhang et al., 2017] achieves new stateof-the-art in this competition by integrating the traditional directional map with a CNN model. However, these characterbased methods can not recognize the characters that have not appeared in training sets, namely a character zero-shot problem. In this case, several radical-based methods are raised by treating each character as a radical sequence. DenseRAN [Wang et al., 2018] takes the first attempt to see CCR as a tree-structured image captioning task. HDE [Cao et al., 2020] designs a unique embedding vector for each Chinese character according to its radical-level constitution. However, there exist several drawbacks in existing radical-based methods: 1) Some radicals may not appear in training sets, namely a radical zero-shot problem. 2) Most of the previous radical-based methods ignore the fact that some characters have the same radical-level constitution. This problem even becomes worse as the increase of the alphabet capacity. 3) Since HDE [Cao et al., 2020] functions in an embedding-matching manner, it needs to store the embeddings of all candidates in advance, which costs a lot of space. 4) The radical-level decomposition leads to a more severe class imbalance problem.

In this paper, inspired by the fact that humans can easily generalize to grasp how to write characters unseen before if they have learned stroke orders of some characters, we propose a stroke-based method by decomposing a character into a combination of five strokes, including horizontal, vertical, left-falling, right-falling, and turning. The five strokes all frequently appear in Chinese characters (in Figure 1, they present in one character simultaneously). Thus, there does not exist a stroke zero-shot problem. Furthermore, each character or radical is uniquely represented as a stroke sequence according to the Unicode Han Database 2 , which helps pave the way for solving the character zero-shot and radical zeroshot problems fundamentally. However, we observe that there is a one-to-many relationship between stroke sequences and characters. To conquer this challenge, we employ a matchingbased strategy to transform the predicted stroke sequence to a specific character in the test stage. The proposed method is validated on various kinds of datasets, including handwritten characters, printed artistic characters, and scene characters. The experimental results validate that the proposed method outperforms existing methods on both character zero-shot and radical zero-shot tasks. More interestingly, the proposed method can be easily generalized to those languages whose characters can be decomposed into strokes such as Korean. In summary, our contributions can be listed as follows:

• We propose a stroke-based method for CCR to fundamentally solve character and radical zero-shot problems.
• To tackle the one-to-many problem, we employ a matching-based strategy to transform the predicted stroke sequence to a specific character.
• Our method outperforms existing methods on both character zero-shot and radical zero-shot tasks, and can be generalized to other languages whose characters can be decomposed into strokes.

2 RelatedWork

2.1 Character-based Approaches

Traditional Methods. Traditional character-based methods use hand-crafted features like Gabor features [Su and Wang, 2003], directional features [Jin et al., 2001], and vector features [Chang, 2006]. However, the performance of them are limited by these low-capacity features [Chen et al., 2020].

Deep Learning-based Methods. With the development of deep learning, several methods employ CNN-based models, which can automatically extract features from given images. MCDNN [Cires¸an and Meier, 2015] is the first case to employ CNN for CCR by ensembling eight models while outperforming human-level performance on recognizing handwritten characters. After that, ART-CNN [Wu et al., 2014] alternatively trains a relaxation CNN and takes the first place in the ICDAR2013 competition [Yin et al., 2013]. In [Xiao et al., 2019], a template-instance loss is employed to rebalance easy and difficult Chinese instances. However, these methods rely on massive data and can not handle characters that have not appeared in training sets.

2 https://www.zdic.net/ collects the stroke orders for the majority of Chinese characters from Unicode Han Database.

2.2 Radical-based Approaches

TraditionalMethods. Before the deep learning era, several radical-based methods are proposed using traditional strategies. In [Wang et al., 1996], a recursive hierarchical scheme is introduced for radical extraction of Chinese characters. It needs accurate pre-extracted strokes, which is difficult to obtain due to the scribbled written styles in datasets. In [Shi et al., 2003], a method based on active radical modeling is proposed. It omits the stroke extraction procedure and achieves higher recognition accuracy. However, the pixel-wise matching and shape-parameter searching are time-consuming.

DeepLearning-basedMethods. In recent years, the development of deep learning helps pave the way for radical-based methods. DenseRAN [Wang et al., 2018] treats the recognition task as image captioning by regarding each character as a radical sequence. Based on DenseRAN, STN-DenseRAN [Wu et al., 2019] employs a rectification block for distorted character images. FewShotRAN [Wang et al., 2019] maps each radical to a latent space and constrains features of the same class to be close. Recently, HDE [Cao et al., 2020] designs an embedding vector for each character using radicalcomposition knowledge and learns the transformation from the sample space to the embedding space. These methods are capable of tackling the character zero-shot problem. However, some radicals may not appear in training sets in a datahungry condition, which leads to another dilemma called radical zero-shot. Hence, these radical-based methods have not solved the zero-shot problem fundamentally.

2.3 Stroke-based Approaches

Existing stroke-based methods usually rely on traditional strategies. In [Kim et al., 1999a], the authors propose a stroke-guided pixel matching method, which can tolerate mistakes caused by stroke extraction. In [Kim et al., 1999b], a method based on mathematical morphology is raised to decompose Chinese characters. A model-based structural matching method [Liu et al., 2001] is proposed with each character described by an attributed relational graph. In [Su and Wang, 2003], they present a method based on a directional filtering technique. These traditional methods need hand-crafted features, which are hard to adapt to different fields and applications. In general, these works have inspired us to combine stroke knowledge with deep learning models.

3 Preliminary Knowledge of Strokes

Strokes are the smallest units for each Chinese character. When humans start learning Chinese, they usually learn to write strokes at first, then radicals, and finally the whole characters. Moreover, there exists a regular pattern in Chinese stroke orders, usually following left to right, top to bottom, and outside in. In other words, when humans have learned the stroke orders of some characters, they can naturally generalize to grasp how to write other characters (even though humans have never seen them before), which inspires us to design a stroke-based model to tackle the zero-shot problem.

According to the Chinese national standard GB180302005, five basic strokes are horizontal, vertical, left-falling, right-falling, and turning. As shown in Figure 3, each category contains instances of different shapes. Please note that the turning category contains more kinds of instances and we only show five of them in Figure 3. Stroke orders for each character are collected from the Unicode Han Database.

In fact, we observe that there is a one-to-many relationship between stroke sequences and characters. As shown in Figure 4, we explore the distribution of one-to-n sequences in 3,755 Level-1 characters (most commonly-used). Most of the sequences (about 92.5%) can perfectly match a single character. In the worst case, a sequence can correspond to seven possible results (n = 7). Therefore, it is necessary to design a module to match each sequence with a specific character.

4 Methodology

The overall architecture is shown in Figure 2. In the training stage, the input image is fed into an encoder-decoder architecture to generate a stroke sequence. In the test stage, the sequence is first rectified by a stroke-sequence lexicon, which is further sent to a Siamese architecture to match a character from a confusable set. Details are introduced in the following.

4.1 Image-to-Feature Encoder

In recent years, ResNet [He et al., 2016] plays a significant role in optical character recognition tasks [Wang et al., 2019]. Residual blocks relieve the gradient vanishing problem, thus enabling a deeper network to fit training data more efficiently. We employ building blocks [He et al., 2016] containing two successive 3 × 3 CNN as the unit of ResNet. Details of the encoder are shown in Supplementary Materials. For a given three-channel image I ∈ H ×W × 3, the encoder outputs a feature map F of size H2 × W2 × 512 for further decoding.

4.2 Feature-to-Stroke Decoder

We employ the basic design of Transformer decoder [Vaswani et al., 2017]. The architecture is shown in Supplementary Materials. We denote the ground truth as g = (g1, g2, ..., gT ). A cross-entropy loss is employed to optimize the model: l = - ∑Tt=1 logp(gt), where T is the length of the sequential label and p(gt) is the probability of class gt at the time step t.

4.3 Stroke-to-Character Decoder

Since a stroke sequence may not match with a specific character, a stroke-to-character decoder is further proposed in the test stage. Firstly, we build a lexicon L that contains stroke sequences of all characters. Nevertheless, in the worst case, the predicted sequence p may fail to match any characters in the lexicon, i.e. p /∈ L. So we choose the one that has the least edit distance with the prediction p as the rectified prediction prec. If matched initially, we consider the rectified prediction to be just the same as the original one, i.e. prec = p.

As shown in Figure 2, we manually collect a dictionary called confusable set C containing those one-to-many stroke sequences, i.e. those one-to-one characters will not appear in C. If the rectified prediction is not in this set, i.e. prec /∈ C, the decoder will generate the corresponding character directly. Otherwise, we employ a matching-based strategy via comparing features between the source image I and support samples I ′ using a Siamese architecture. Specifically, for one head in the Siamese architecture, the feature map F of the input image I is given. For the other head, several support samples I ′ containing printed images of characters with the same stroke sequence are fed to the encoder to generate a list of feature maps F ′ = {F′1 , F′2 , ..., F′N }, where N is the number of possible results. We calculate similarity scores between F and each feature map F′i , then selecting the one which is the most similar to F as the final result:

where i∗ is the index of result and D is the similarity metric:

Different from FewShotRAN [Wang et al., 2019], we do not use support samples during training, which guarantees the principle of zero-shot learning. Compared with HDE [Cao et al., 2020], our method costs less space since we only need to store the features of confusable characters in advance.

5 Experiments

In this section, we first introduce the basic setting of our experiments, and analyze two similarity metrics. Then we compare the proposed method with existing zero-shot methods on various datasets. Finally, we have a discussion on our method in terms of generalization ability and time efficiency.

Datasets. The datasets used in our experiments are introduced in the following and examples are shown in Figure 5.

• HWDB1.0-1.1 [Liu et al., 2013] contains 2,678,424 offline handwritten Chinese character images. It contains 3,881 classes collected from 720 writers.
• ICDAR2013 [Yin et al., 2013] contains 224,419 offline handwritten Chinese character images with 3,755 classes collected from 60 writers.
• Printed artistic characters We collect 105 printed artistic fonts for 3,755 characters (394,275 samples). Each image is of size 32 × 32 and the font size is 32px.
• CTW [Yuan et al., 2019] contains Chinese characters collected from street views. The dataset is challenging due to its complexity in backgrounds, font types, etc. It contains 812,872 samples (760,107 for training and 52,765 for testing) with 3,650 classes.
• Support samples The size of support samples is 32×32 with the font size 32px. Two widely-used fonts including Simsun and Simfang (not in the artistic fonts) are used. We take the average of similarity scores between features of input and two support samples when testing.

EvaluationMetric. Character Accuracy (CACC) is used as the evaluation metric since a stroke sequence may not match a specific character. We follow the traditional way to construct the candidate set by combining categories appearing in both training sets and test sets [Wang et al., 2018; Wu et al., 2019].

Implementation Details. We implement our method with PyTorch and conduct experiments on an NVIDIA RTX 2080Ti GPU with 11GB memory. The Adadelta optimizer is used with the learning rate set to 1. The batch size is set to 32. Each input image is resized to 32 × 32 and normalized to [-1,1]. We adopt a weight decay rate of 10-4 in zero-shot settings to avoid overfitting. For the seen character setting, we ensemble the results of ten models. No pre-training and data augmentation strategies are used in our experiments.

5.1 Choice of SimilarityMetrics

In the test stage, we evaluate the performance with Euclidean metric and Cosine metric using the same model, with the parameters fixed after training. We choose samples with labels in the first 1500 classes in 3,755 Level-1 commonly-used characters from HWDB1.0-1.1 as the training set, while picking out images from ICDAR2013 as the test set, whose labels are both in the last 1000 classes and in the confusable set, for validation. The experimental results show that the performance of Cosine similarity achieves 90.56%, which is better than the Euclidean metric (89.39%). We suppose that the Cosine metric cares most about whether a specific feature exists, while caring less about its exact value, which is more suitable for characters with various styles. Hence, we use the Cosine similarity metric in the following experiments.

5.2 Experiments on Handwritten Characters

We conduct experiments on handwritten characters in three settings, including character zero-shot, radical zero-shot, and seen character. 3,755 Level-1 commonly-used characters are used as candidates during testing.

Experiments in Character Zero-shot Setting

From HWDB1.0-1.1, we choose samples with labels in the first m classes of 3,755 characters as the training set, where m ranges in {500,1000,1500,2000,2755}. From ICDAR2013, we choose samples with labels in the last 1000 classes as the test set. As shown in the top-left of Table 1, our method outperforms other methods with various amounts of training classes. For a fair comparison, we additionally experiment with the same encoder (DenseNet) and decoder (RNN) that used in DenseRAN [Wang et al., 2018] and the results show that our method is still better than the compared methods (see Supplementary Materials). The reasons that our method performs better can result from two aspects: 1) The frequency of each stroke in the training set is far more than each radical, and therefore helps the model converge better. 2) Our method alleviates the class imbalance problem, which commonly exists in the character-based and radical-based methods (see more details in Supplementary Materials).

Experiments in Radical Zero-shot Setting

The setup follows three steps: 1) Calculate the frequency of each radical in 3,755 Level-1 commonly-used characters. 2) If a character contains a radical that appears less than n times (n ∈ {50,40,30,20,10}), move it into set STEST; otherwise, move it into STRAIN. 3) Select all samples with labels in STRAIN from HWDB1.0-1.1 as the training set, and all samples with labels in STEST from ICDAR2013 as the test set. In this manner, the characters in the test set contain so-called unseen radicals. The training class increases when n decreases (see Supplementary Materials). As shown in the top-right of Table 1, our method outperforms all the compared methods since all the strokes have been supervised in training sets. Moreover, the proposed method can infer the stroke orders of unseen radicals based on the knowledge learned from training sets. Although DenseRAN [Wang et al., 2018] and HDE [Cao et al., 2020] can alleviate this problem using lexicon rectification and embedding matching, they are still subpar due to the lack of direct supervisions on unseen radicals.

Experiments in Seen Character Setting

Different from zero-shot settings, the seen character setting represents utilizing samples whose labels have appeared in training sets for testing. We use the full sets of HWDB1.01.1 for training and ICDAR2013 for testing. Firstly, we evaluate the performance of those samples whose labels are not in the confusable set C. As shown in Supplementary Materials, our method yields better performance compared with two encoder-decoder architectures at different levels due to more occurrences for each category. The experimental results on the full set of ICDAR2013 are shown in Table 2. Our method does not achieve ideal performance since the matching-based strategy follows an unsupervised manner. We additionally train a character-based model (only need to modify the last linear layer of the proposed model) on the same training set to resolve this problem. If the feature-to-stroke decoder outputs a stroke sequence that belongs to C, we employ this model to generate the final prediction as a alternative, which boosts CACC by 0.46%. It can be treated as combining advantages of two models at different levels. See visualizations in Figure 6 and more analyses in Supplementary Materials.

5.3 Experiments on Printed Artistic Characters

We conduct experiments with printed artistic characters in two settings, including character zero-shot and radical zeroshot (see the middle row of Table 1). The division of this dataset is shown in Supplementary Materials. The performance of our method is more than twice as HDE [Cao et al., 2020] in the radical zero-shot setting. Compared with handwritten characters, the printed ones have relatively clearer strokes, which benefits the proposed stroke-based method.

3 We reimplement the method in zero-shot settings using the authors' provided code. Please note that we build up candidate sets following the traditional way which differs from the way in [Cao et al., 2020]. We set the input size to 32 × 32 for a fair comparison.

5.4 Experiments on Scene Characters

The division of CTW [Yuan et al., 2019] is shown in Supplementary Materials. As shown in the bottom row of Table 1, the proposed method outperforms the compared methods in the character zero-shot and radical zero-shot settings in most cases. However, all of these methods still can not achieve human-level performance in zero-shot tasks, as the recognition on scene characters faces many challenges such as low resolution and complicated backgrounds. The experimental results in seen character setting are shown in Table 3.

5.5 Discussion

Generalize to New Languages

To validate the ability of cross-language generalization, we test our method on Korean characters after training using the full set of printed artistic characters. Like Chinese characters, each Korean character can also be uniquely decomposed into sequences with five strokes. The stroke orders of each character are collected from Wikipedia 4 . Moreover, we observe that the written pattern is nearly the same as Chinese characters. Two samples are shown in Figure 7. To construct the test set, we manually collect 119 fonts for 577 Korean characters. The experimental results show that the model trained on Chinese datasets can achieve accuracy of 17.50% on the Korean dataset although suffering from domain gaps such as different font styles and distributions of five strokes. We consider that the stroke knowledge bridges the gap between different languages, thus making it possible to build up a cross-language recognizer while only need to replace candidate classes and support samples. Moreover, previous methods do not have this ability as different languages usually have different character- and radical-level representations.

4 https://en.wikipedia.org/wiki/%E3%85%92

Figure 7: Korean characters and the corresponding stroke sequences.

Furthermore, we conduct a character zero-shot experiment on the Korean dataset. We randomly shuffle the 577 characters five times and utilize the first m% characters for training and the other part for testing, where m ∈ {20, 50, 80}. The performance reaches 23.09%, 65.94%, and 78.00% when training using three proportions of datasets. When trained on 80% of the dataset, the model is capable of recognizing over three-quarters of unseen characters, which further validates that the stroke knowledge can work on Korean characters.

Time Efficiency

We investigate the time cost of encoder-decoder architectures at three levels (refer to Figure 1). For a fair comparison, we employ the same ResNet-Transformer architecture. The batch size is set to 32 and we take the average time of 100 iterations. The character-based method (0.24s) outperforms all other methods since it does not need to predict recurrently during testing, i.e. only need to decode once. Although our stroke-based method (1.24s) has to conduct decoding at two different levels, it is only a six-category classification task (includes one stop symbol) in the feature-to-stroke decoder, thus it has fewer parameters in the last fully connected layer and performs faster than the radical-based method (1.38s).

6 Conclusions

In this paper, we propose a stroke-based method to deal with the zero-shot Chinese character recognition problems inspired by the generalization ability of human beings with stroke knowledge. Furthermore, we put forward a matchingbased strategy to tackle the one-to-many challenge benefiting from the Siamese architecture. The experimental results validate that our method outperforms other existing methods in both character zero-shot and radical zero-shot settings on various kinds of datasets. Meanwhile, our method can be easily generalized to new languages, which further verifies that the stroke knowledge enables building up cross-language recognizers for a bundle of East Asian characters.

Acknowledgements

This research was supported in part by STCSM Projects (20511100400, 20511102702), Shanghai Municipal Science and Technology Major Projects (2018SHZDZX01, 2021SHZDZX0103), Shanghai Research and Innovation Functional Program (17DZ2260900), the Program for Professor of Special Appointment (Eastern Scholar) at Shanghai Institutions of Higher Learning, and ZJLab.