Joint Learning Embeddings for Chinese Words and their Components via Ladder Structured Networks

1 Introduction

Embeddings have been proven to be useful in natural language processing (NLP) with the rising of deep learning [Collobert et al., 2011; Mikolov et al., 2013b; Pennington et al., 2014]. Of all levels of granularities that have been investigated for embedding learning [Mikolov et al., 2013b; Kiros et al., 2015; Le and Mikolov, 2014], word embeddings have received the widest attention mainly for the reason that words are conventionally considered the smallest element that can be uttered in isolation with semantic or pragmatic content, 1 especially for western languages such as English.

However, not all languages follow the practice of English. Words in some languages are assembled by smaller units that can be used separately to deliver semantic information. In Chinese, words are made of characters, where each character carries rich semantic knowledge so that the meaning of a Chinese word is highly related to the characters it is comprised of. For example, }f ("automobile") and kf ("train") are types of f ("vehicle"); the meanings of the two words are largely determined by the character they share. Moreover, characters in Chinese can be further decomposed into radicals, and the meanings of the characters are also highly dependent on their radicals. Especially majority number of Chinese characters are phono-semantic compounds, 2 whose radicals in general are the semantic part in the character. For example, characters _ (river), V (lake) and w (sea) all have on the left a radical 5 meaning water, indicating that these characters have a semantic connection with water.

1 Controversially in linguistics, morpheme is the smallest unit of meaning, which however does not stand on its own in running texts.

With such internal structure, learning Chinese word embeddings can be enhanced accordingly. Previous work has proven the validity of leveraging characters for learning word embeddings [Chen et al., 2015; Xu et al., 2016], and leveraging radicals for learning character embeddings [Li et al., 2015] and word embeddings [Yin et al., 2016; Su and Lee, 2017]. Thus, it is straightforward to consider learning word, character and radical in a full decomposition chain and learn their embeddings synchronously. In this paper, we propose a ladder structured networks (LSN) to do so based on the relations among words and their components. The LSN is inspired by learning with lateral connections among layers in ladder networks [Valpola, 2014], which are designed to learn better intermediate representations with connections across layers [Rasmus et al., 2015a; 2015b]. The ladder networks' structure has been proven to be effective in learning with better generalization ability, especially on image processing [Ronneberger et al., 2015; J´egou et al., 2016], for the reason that its design enables the model to encode information from different levels.

2 Xu Shen in Shuowen Jiezi (Explaining Graphs and Analyzing Characters) (100 AD) placed approximately 82% of characters into this type, while in the Kangxi Dictionary (1716 AD) the number is close to 90%, owing to the extremely productive use of this technique to extend the Chinese vocabulary.

Different from conventional ladder networks, the intermediate layers in our model do not follow a typical auto-encoder with automatic generated latent representations, but are supervised by words and their components. Note that layers 3 are the basic modules in LSN. As a result, LSN is modular in structure; every "stair-step" (layer) of the LSN "ladder" is pluggable and can be easily removed. The model is thus easy to be adapted to the scenarios with only learning with word and character, word and radical, character and radical, etc. Based on the layers, our model has multiple objective functions with respect to units of different granularities, e.g., words, characters and radicals. Therefore, using our proposed framework, one is able to incorporate more contextual information in a hierarchical manner in learning Chinese word embeddings when compared to previous studies [Chen et al., 2015; Li et al., 2015; Xu et al., 2016]. Experimental results suggest that our approach outperforms all baselines in both intrinsic and extrinsic evaluations, especially when the training data is limited.

2 Ladder Structured Networks

As illustrated in Figure 1, in our model, three layers from bottom to top represent radicals (E(1) , D(1) ), characters (E(2) , D(2) ) and words (E(3) , D(3) ), respectively. Horizontally, the model is divided into two parts across layers. Following the terminology in ladder networks, we use "encoder path" and "decoder path" as in Rasmus et al. [2015b] to describe the left and right part of our model. In the encoder path, predictions (the solid upward arrow on the left side in Figure 1) are conducted from radicals (E(1) ) to characters (E(2) ), and from characters to words (E(3) ). This path ensembles the process of how a word is composed, which complies with our intuition: when one has a word in his mind, he composes the word starting from radicals, and then characters. Formally, for each word w, predictions from E(1) to E(2) and from E(2) to E(3) is to maximize the likelihoods similar to the CBOW model [Mikolov et al., 2013a], i.e.,

respectively. For each word w, c are the characters that compose this word and r are the radicals of these characters. υc and υr refer to the embeddings of these characters and radicals. Since a radical is the main component of a character, each word has equal number of characters and radicals. We use r ∈ w to represent the relationship of a word and its radicals. In the decoder path, we do not have a prediction chain as that in the encoder path. The reason is that when a word (D(3) ) is given, its component characters (D(2) ) and radicals (D(1) ) are constantly determined (the dotted arrows on the right side in Figure 1). Thus predictions following this path do not contribute to learning the model.

3 Layers herein correspond to different decomposition levels, i.e., radical (R), character (C) and word (W).

More importantly, there are predictions connecting different components from the two paths. On the same layers, we have word-word (E(3) →D(3) ), character-character (E(2) → D(2) ) and radical-radical (E(1) → D(1) ) predictions. The word-word prediction is essentially an SG model [Mikolov et al., 2013b], which maximizes the following likelihood

where wi are words from w's context w+C -C , and υw is the embedding of w. This is the core prediction of the model because words are the direct input of the model. Characters and radicals are derived from the input words observed from the running texts. Similarly, the likelihoods of charactercharacter and radical-radical predictions are formulated as

For cross layer scenarios, two more prediction lines are drawn from a word (E(3) ) to characters (D(2) ) and from characters (E(2) ) to radicals (D(1) ). They functionalize as auxiliary tasks to enhance representations at every layer of the model by introducing extra contextual information for characters and radicals from different granularities, which are proven to be useful in previous studies [Li et al., 2015; Xu et al., 2016]. These two prediction lines are formulated as

Given a corpus with vocabulary V and N tokens, our LSN is thus to maximize

over the entire corpus, in which

Similar to the SG model, at each training step, LSN first obtains a word pair from a training instance to feed the E(3) and D(3) position, then decomposes their components for other positions. Once all layers are assembled, LSN jointly trains embeddings for them according to Eq 1 to 7. As a result, learning embeddings for words, characters and radicals in each training instance is directly computed by stochastic gradient descent (SGD) following a dimension-wise updating from the partial derivatives ∂LLSN ∂υw , ∂LLSN ∂υc and ∂LLSN ∂υr :

Therefore, for example, when word embeddings are updated, character and radical embeddings will be affected synchronously until the model is converged.

With LSN, words, characters and radicals are learned in linearly correlated vector spaces since the embedding updating processes are intertwined across different granularities. As illustrated in Figure 1, a� (grape) is associated with not only the word }� (delicious), but also its characters } (good), � (eat) and its radicals s (woman) and ª (mouth). This spatial characteristic of embeddings facilitates leveraging word components as useful knowledge source to provide different levels of semantics. For example, � and ª are closely related to a� in semantics since "grape" can be "eaten" by "mouth". Therefore, with only one pair of words, LSN could capture more semantics than other models with such decomposition. The learned embeddings of these character and radical components can further influence the embedding learning for other words shared with the same characters and radicals. In this way, LSN automatically learns the roles of the characters and radicals in different words. As a comparison, to distinguish the internal characters in a word, similarity between a word and its component characters has to be explicitly calculated, such as in SCWE [Xu et al., 2016].

Another advantage of using the hierarchical design is that the layers in the full stacked LSN (W+C+R) are pluggable so that it is easy to restructure the framework by taking off any particular layer and assemble the rest parts, e.g., LSN (W+C) or LSN (C+R), which is trained only with words and characters, or characters and radicals, respectively. This modular characteristic ensures our model being detachable and thus it can be used in a more flexible way for different scenarios.

3 Experiments

3.1 Experiment Settings

We use two corpora to train our embeddings. The first one is the manually word segmented corpus composed of People Daily of January 1998 (PD98), with 1M words. Its vocabulary includes 56K words, 4.7K characters and 253 radicals. The second one is the Chinese Wikipedia dump (Wiki) 4 . We follow the procedures that done in Xu et al. [2016] for preprocessing, by removing pure digits and non-Chinese characters and use ANSJ 5 to segment words. The resulted dataset has 145Mwords in total. Its vocabulary includes 3Mwords, 18K characters and 275 radicals. For radicals, we use online Xinhua Dictionary 6 to extract character-radical mappings.

In our experiments, CBOW and SG, character-enhanced word embedding model (CWE) [Chen et al., 2015] 7 , similarity based CWE (SCWE) and its extension multiple-prototype SCWE (SCWE+M) [Xu et al., 2016] 8 , character CBOW (charCBOW) and character SG (charSG) [Li et al., 2015], multi-granularity embedding (MGE) [Yin et al., 2016] 9 are used as our baselines in different tasks. For all the models in comparison, we set the dimension of embedding vectors to 200, the size of window to 5, the frequency cut-off to 5, the initialized learning rate to 0.025. The numbers of windowsize and frequency cut-off are conducted on the word level. In our model, we use hierarchical softmax for the predictions.

4 http://download.wikipedia.com/zhwiki/

5 https://github.com/NLPchina/ansj seg

6 http://xh.5156edu.com/

7 https://github.com/Leonard-Xu/CWE

8 https://github.com/JianXu123/SCWE

9 We reimplement charCBOW, charSG and MGE with our own codes according to their papers.

3.2 Qualitative Analysis

Since our approach is able to learn the embeddings of Chinese words, characters and radicals in linearly correlated vector spaces, it is straightforward to compute their relations through their embeddings. Herein we select word *¿ (airline) and its component character * (navigate) and the component radical � (boat) to investigate their top 3 nearest words, characters and radicals in Table 1. The embeddings for this analysis are from the LSN (W+C+R) model trained on the Wiki data.

For word *¿, its component character * and radical � are all included in the nearest characters and radicals, respectively. Especially, 5 and / are also included in its nearest radicals. Consider that / (axe) is the radical of � (new), it makes sense that "airline" is intensively associated with "new" in the corpus. Interestingly, besides �, the other radical � (silk) of *¿ is not found in its nearest radicals, which is a good example demonstrates the effectiveness of LSN in distinguishing radical roles. As a result, � is semantically closer to other words (characters), such as "˙ (textile).

To better understand the results, Table 2 lists the top ten frequent context words of *¿ in the Wiki data. 10 For a given word, although its nearest words (or characters, radicals) computed in the vector space are not necessarily from its context words, it is still a good reference of the high frequent context words to explain the observations we have in Table 1, e.g., all the nearest radicals in Table 1 can be found in the words in Table 2. Particularly, for radical �, its nearest words are more likely to be the context words with characters containing this radical, which is straightforward since it is in multiple characters. Considering that in our model there is no direct link between radical and word layer, the most related words to radical � are the ones related to all the characters sharing this radical. For its most related radicals, p (to walk) has the highest similarity is mainly because multiple characters sharing this radical, such as v (swiftly), w (rise), � (surpass), etc., which have close semantic relations to the characters containing the radical �. The radical " (particularly) appeared in the top 3 nearest radicals is because it is the component of the character $ (particularly), which is one of the characters in a common adverb $ v (particularly) in Chinese. The visualization of the aforementioned examples on a 2D plots using t-SNE [van der Maaten and Hinton, 2008] further confirms the observations in above texts.

10 We do not include stop words such as � (of), / (is), � (and) etc., in the list.

3.3 Word Similarity

The intrinsic evaluation is word similarity assessment. Normally the correlation between the scores generated by a model and human judgment indicates how good the model performs. We use WS-240 and WS-296 [Jin and Wu, 2012] in this paper as our evaluation datasets. The Spearman's rank correlation (ρ) is adopted for calculating the correlation. We evaluate our models trained on the PD98 and Wiki datasets. The results are reported in Table 3 and 4, respectively.

We have the following observations. First, LSN model is effective to learn meaningful embeddings. When trained on either PD98 or Wiki dataset, LSN models yield better or close ρ scores to state-of-the-art models. Second, jointly modeling embeddings on the levels of word, character, and radical is important. By exploring the joint effects of radicals, characters, and words in learning embeddings, LSN (W+C+R) produces better ρ scores than their counterparts, i.e., LSN (W+C) and LSN (W+R), without modeling radicals or characters.

3.4 Part-of-Speech Tagging

It is by our intuition to hypothesize that the semantic attributes of the internal components of Chinese words can help to identify word syntactic types. For example, words with the radical L (hand) are very likely to be verbs, such as Sp (interrupt) and �� (stir); In addition, radical ( (wood) often contributes to nouns, e.g., �� (forest) and L � (desk and chair). In this task, we use part-of-speech (POS) tagging as an extrinsic evaluation to assess how different embeddings performed in solving syntax problems.

We implement the BiLSTM-CRF model [Huang et al., 2015] as our POS tagger, fed by pretrained word embeddings from different approaches. We set the hidden layer size to 256, with 10 epochs 11 for training. Note that in spite of input embeddings, no extra features are used in our model. We use Penn Chinese Treebank v5.0 (CTB-5) [Xue et al., 2005] as the evaluation data, under the standard split with training/test as 18086/348 sentences, respectively. Similar to word similarity task, we also test the emebeddings trained from PD98 and Wiki. The results are reported in Table 5.

11 Normally the model convergence requires less than 10 epochs, which is also addressed in Huang et al. [2015].

We observe that POS taggers using different embeddings produce various accuracies. This indicates that the quality of word embeddings is vital for POS tagging, which is consistent with the conclusion in Huang et al. [2015]. It is also observed that POS taggers with LSN (W+C+R) trained on either PD98 or Wiki yield the best accuracy. This shows that characters and radicals may provide critical word type information, which is useful in POS tagging.

To further investigate the performance of taggers with different embeddings on particular POS tags, Table 6 compares the tagging performance on the top six POS tags 12 using our LSN (W+C+R) and SG embeddings trained from Wiki dataset. A ten-partition two-tailed paired t-test at p < 0.05 level is conducted on LSN results against the SG ones. It is verified that tagging accuracies on noun (NN), verb (VV), preposition (P) and adverbs (AD) are improved with using LSN embeddings, where significant improvements are observed on verbs, preposition and adverbs. We investigate some adverbs and found their component characters provide useful guidance. For example, s3 (smooth and steady) is tagged with NN by baseline embeddings; our embeddings correct its tag to AD, for its component character and radical s (long) has an JJ/AD attribute that has a strong indication for the adverb tag. The improvement on prepositions is mainly contributed from other POS tags, especially nouns and verbs. In Table 6 we also notice a slightly performance degradation on proper nouns (NR). The reason is that proper nouns are usually person and organization names, which typically do not have a pattern of characters and radicals, thus cannot provide semantic guidance for word embeddings.

Above observations meet our expectation that words' components can help POS tagging, and our embeddings are proved to be the most effective one in leveraging such information for POS tagging when compared to other models.

3.5 Document Classification

The second extrinsic evaluation is document classification. We experiment on Fudan Corpus 13 , which contains 9.8K documents in 20 categories. We follow Xu et al. [2016] by constructing two datasets from the Fudan corpus, namely, Fudan-large and Fudan-small, with each dataset containing

12 Frequency above 300 in the test data.

13 http://www.datatang.com/data/44139

5 categories of documents. The Fudan-large dataset includes categories of Agriculture, Economy, Environment, Politics and Sports, while the Fudan-small includes Education, Medical, Military, Philosophy and Transport. For each category, we randomly select 80 percent of the documents as the training dataset and reserve the rest as the test dataset. As a result, the training and test sets contain 4,894 and 1,227 documents for Fudan-large dataset, 233 and 61 for Fudan-small dataset. Similar to the Wiki dataset, pure digits and non-Chinese characters are removed and word segmentation is conducted. The publish information for each document is also removed because it contains strong indication of the categories, which will bias the classifier with unfair benefits.

This document classification experiment is performed in a conventional way as that in previous studies [Kiela et al., 2015; Kiros et al., 2015]. For all the documents in training and test datasets, we first construct document level representations by averaging the embeddings from all words in a given document. A logistic regression classifier is then trained on top of the resulted document level representations on the training set and evaluated on the test set.

In addition to word embeddings, we also evaluate character embeddings in this experiment. In order to do so, each document is represented as average embeddings of the characters in the document instead of words. Note for SCWE+M, a character may have multiple embeddings for its different prototypes, we then take the average embedding for the character from its multiple prototypes. In addition to the baselines we have for previous experiments, we add charCBOW and charSG as extra baselines in this task. 14 We compare with our variation LSN (C+R) in document classification task, which jointly models embeddings for characters and radicals. Although in Li et al. [2015], the bi-character embeddings achieve slightly better performance in document classification, we do not include them because, as stated in Xu et al. [2016], bi-characters are meaningless and may not form a Chinese word, thus are not comparable with words in other approaches. For this task, The input word and character embeddings are trained from the Wiki dataset.

Table 7 and 8 report the classification accuracies evaluated on Fudan-small and Fudan-large datasets, respectively. In general, our approach outperforms baselines on both datasets. This task well demonstrates the advantage of learning Chinese embeddings with considering different levels of components. We also notice that LSN (C+R) model fails to yield good results, which might because of the restricted alphabet at character level. This reason may also explains the poor performance of charCBOW and charSG.

14 In previous experiments, we do not consider them in comparison because they are on character level and previous evaluation are done on word level.

Similar to what previous experiments show, learning on multiple levels of word components is very important regardless whether training data is limited, especially for the case that many words are unseen in the test data. Word embeddings with considering characters and radicals could help in this scenario bridging the gap of the out-of-vocabulary words. For example, in our experiment on Fudan-small dataset, the word *¿ from test data never appears in the training data. However, with the shared character * (navigate) and radical �, the embeddings learned by LSN for *¿ have higher similarity than other embeddings to * z, which is a invocabulary word in transport documents. Thus the document containing *¿ tend to have a closer distance to the transport category. This joint learning with word components ensures the stability of our approach in representing words' semantics. Therefore, it is observed from Table 7 and 8, our approach achieves comparable results across two datasets.

4 RelatedWork

Word representation models represent words as real valued vectors that convey semantic information via exploring word co-occurrence patterns in contexts of neighboring words [Bengio et al., 2003; Collobert et al., 2011; Mikolov et al., 2013a; 2013b]. Despite of the success of word2vec, its continuous bag-of-words (CBOW) and skipgram (SG) models are not capable of capturing morphological information underlying internal word structure, which has been proven to be useful in learning English word embeddings [Luong et al., 2013; Botha and Blunsom, 2014; Trask et al., 2015; Miyamoto and Cho, 2016].

In Chinese, radicals, i.e., the internal structure of Chinese characters, have shown their effectiveness in learning wordlevel or character-level embeddings [Chen et al., 2015; Xu et al., 2016; Yin et al., 2016]. In exploiting such structure, Su and Lee [2017] proposed to enhance word embeddings with glyph features from character images. Nevertheless, these referred studies serve as extensions of CBOWmodel. Different from them, our model essentially employs the concept of SG model at word level with exploiting word co-occurrence patterns in local context. As indicated in Mikolov et al. [2013a; 2013b] as well as our experiments (see §3), SG outperforms CBOW in most NLP tasks when trained on small amount of training data and presents better on rare words. Our approach can be seen as started on a higher baseline than the previous work. Although Li et al. [2015] followed both CBOW and SG model for Chinese characters that combine radical information, they focus on exploiting the character-radical relations, where the joint effects of words, characters, and radicals are ignored. To the best of our knowledge, our approach is the first to explore different component levels of words with a hierarchical structure, which serves as a natural fit for learning embeddings of Chinese words and their components.

5 Conclusion

We proposed a novel framework, ladder structure networks (LSN), to jointly learn embeddings of Chinese words, characters and radicals. LSN represents word, character and radical as different layers in a hierarchical manner and learns their embeddings by maximizing an overall likelihood based on their relations. Experimental results from intrinsic and extrinsic evaluations confirmed the effectiveness of our approach.

LSN has several characteristics. First, the learning of LSN ensembles the formation process of Chinese words and captures their relations to other words and components. Second, words, characters and radicals are learned in linearly correlated vector spaces, thus it is easy to compute their relations through their embeddings. Third, with the character and radical information, LSN is able to learn word semantics without relying on external resource, especially when training data is very limited. This characteristic is highly important because word segmentation is vital and requires extra efforts for processing Chinese texts, especially in the cold-start scenario when entering a new domain. Overall, this work offers an alternative way to learn better word embeddings when there is very few accurately (manually) segmented data. To further extend this work, different structures or prediction chains are worth exploring within current LSN framework, and the idea of LSN can be applied to other similar tasks and languages.