Neural Networks Incorporating Unlabeled and Partially-labeled Data for Cross-domain Chinese Word Segmentation

1 Introduction

In contrast to Western languages, Chinese has no explicit word delimiters in a sequence of text. Hence, Chinese language processing usually require Chinese word segmentation (CWS) as a pre-processing step. CWS methods usually treat the task as a sequential labeling problem. For a given sentence, labels are assigned to all of the characters. These tags may indicate the position of a character in the word [Xue, 2003] or represent the intervals between characters [Huang et al., 2007]. Recently, along with the development of deep learning methods, some neural network models [Chen et al., 2015; Cai and Zhao, 2016; Zhang et al., 2016; Liu et al., 2016; Cai et al., 2017] have achieved great success in CWS tasks. Despite their enormous success, however, these methods still have limitations: they usually rely heavily on manually labeled training data. These data mostly comes from the newswire domain. From previous studies [Liu and Zhang, 2012], we know that when the evaluation data shift from the newswire domain to other domain, the performance of supervised methods usually drops severely.

Although annotated domain-specific datasets cannot be easily achieved and manual annotation is a time consuming process, we can easily find large-scale unlabeled data over the Internet. Many works have investigated the use of such free data to address the problems for the resource-poor domains [Jin and Tanaka-Ishii, 2006; Zhao and Kit, 2008; Li and Sun, 2009; Sun and Xu, 2011; Zhang et al., 2013]. Meanwhile, from web pages such as Wikipedia, a sentence's partial segmentation information can be inferred from hyperlinks, which produces partially-labeled data. In addition, with the help of lexicons, we can also obtain partially-labeled data by matching characters. Previous works have shown that this can provide valuable information for cross-domain CWS tasks [Jiang et al., 2013; Liu et al., 2014]. Although these approaches have demonstrated clear benefits of incorporating unlabeled and partially-labeled data, how to integrate such data into neural network models for CWS tasks in an end-toend manner is still not well investigated.

In this work, we propose a neural sequence labeling architecture to utilize unlabeled and partially-labeled data for cross-domain CWS tasks. By definition, a word is composed of characters. Thus, if two characters form a word (or a part of a word), they will frequently occur together in succession. This co-occurrence information, which can be obtained from unlabeled data, can benefit the CWS tasks. To utilize this co-occurrence information, we combine the segmentation model, implemented by BiLSTM, with two characterlevel language models. Such language models can capture the character-level co-occurrence information. In addition, we use a gate mechanism to learn the degree of influence exerted by the co-occurrence information on the segmentation model. To handle partially-labeled data, we modify the objective function used in the training of sequence labeling RNN, which allows the proposed method to be trained with either full or partial labels. Experimental results show that the proposed method could easily shift between different domains and performed well on cross-domain CWS tasks.

Our contributions are as follows: 1) We introduce a method to incorporate unlabeled data, which combines the segmentation model with language models. A gate mechanism is used to learn the degree of influence exerted by the language models. 2) We modify the loss function used in CWS tasks to handle the partially-labeled data. 3) Based on several experiments, we demonstrate that the proposed method can achieve better performance than previous state-of-the-art methods.

2 Neural Model for CWS

The common choice for sequence labeling CWS is the BMES scheme, which is to label each character in a sentence as one of {B,M,E, S}, representing the beginning, middle, or end of a word, or a word with a single character, respectively. We adopt recurrent neural network with long short-term memory (LSTM) as our segmentation model because it is capable of reducing the feature engineering [Chen et al., 2015].

Long Short-termMemory LSTM[Hochreiter and Schmidhuber, 1997] is a type of RNN designed to cope with gradient vanishing problems. Basically, the formulas to update an LSTM unit at time t can be abbreviated as follows:

Here, xt is the input vector and ht is the hidden state; θ represents all the parameters of the LSTM.

In CWS tasks, it is beneficial to access both past and future contexts. Thus, we use a bidirectional LSTM (BiLSTM). The update of the BiLSTM unit can be written as follows:

where -→h t is the forward hidden state and ←- ht is the backward hidden state at time t; ht is the concatenation of -→h t and ←- h t; and θbi represents all the parameters of the BiLSTM.

CWS with BiLSTM Given a sentence S represented by a sequence of characters [c1, c2, . . . , cT ], we first use an embedding layer to obtain the vector representation (embeddings) xt for each character ct. Then, xt is fed to the BiLSTM. At time t, ht in equation (2) can be regarded as the representation of ct, which incorporates the context information. Through a softmax non-linear layer, the output ˆ yt can be obtained by

where ˆ yt represents the prediction probabilities, W is the weight matrix, and b is the bias.

Given the ground-truth labels y1, y2, . . . , yT represented by one-hot vector, the cross entropy loss function for sentence S is given by the following:

where Y = {y1, y2, . . . , yT } and ˆ Y = {ˆ y1, ˆ y2, . . . , ˆ yT }.

A CRF layer can be further used to leverage the dependencies of adjacent labels [Huang et al., 2015].

3 Incorporating Unlabeled Data for Cross-domain CWS

The above BiLSTM segmentation model does not work well when the training and test datasets come from different domain (namely source domain and target domain). It is unrealistic to annotate a dataset of target domain for training. However, there is a large of unlabeled data on the Internet. This leads to the unsupervised domain adaptation problem. In this setting, for the source domain, we have a labeled dataset Sl, while for the target domain, we only have an unlabeled dataset Tu. Using Sl and Tu, our goal is to train a model with good performance in the target domain at the evaluation time.

3.1 Motivation

Because a word is composed of characters, two characters will frequently occur together in succession if they form a word (or a part of a word). This co-occurrence information, which can be obtained from unlabeled data, can benefit the CWS tasks. To utilize this information, we propose to combine character-level language model with the BiLSTM segmentation model. The objective of language model is to predict the next character ct, given the previous t - 1 characters c1, c2, . . . , ct-1. To see why language model can incorporate co-occurrence information, here is an example in English. Given a sequence of characters "American presi", it is easy to predict that the next characters are "dent" because we know "president" is a word and "presi" and "dent" co-occur many times. However, it is hard to predict that the next characters are "presi" given "American" because they belong to different words. Similarly in Chinese, we find that, in the sequence "美国|总统" (美国(American) is a word and 总统(president) is another word), the probability p("统"|"美国总") given by language model is 0.71, but the probability p("总"|"美国") is 0.051 . In short, when language model gives a high probability p(ct|c1, c2, . . . , ct-1), it tells us that ct and ct-1 should not be segmented, but when language model gives a low probability, it tells us that ct and ct-1 should be segmented.

3.2 Network Architecture

The architecture of our proposed model is illustrated in Figure 1. It mainly includes three components: the forward language model (pink), backward language model (yellow), and BiLSTM segmentation model (blue). The forward and backward language models are also implemented by LSTM.

Forward LanguageModel In the forward language model, we feed the embeddings {x1, x2, . . . , xT} of characters {c1, c2, . . . , cT } sequentially, one at a time. At each time-step t, we want to predict the next character ct+1 based on the information incorporated from c1, c2, . . . , ct, which is denoted by hft . Concretely, the formulas at time t are as follows:

where θf denotes all the parameters of LSTMf . Then, a following softmax non-linear layer outputs the probability distribution over the vocabulary:

where ˆ yf t represents the prediction probabilities of next character ct+1, Wf is the weight matrix, and bf is the bias.

Backward Language Model The basic structure of the backward language model is the same as that of the forward language model, with the difference that we feed the embeddings {x1, x2, . . . , xT } in reverse order, i.e., beginning with xT . Then, we feed xT-1, xT-2, . . . and end with x1. At each time-step, we want to predict the previous character ct-1 based on the information incorporated from cT , cT-1, . . . , ct, which is denoted by hbt :

1 The results are obtained by training a LSTM-based language model on People's Daily dataset.

where θb denotes all the parameters of LSTMb . Then, the output of softmax is as follows:

where ˆ yb t represents the prediction probabilities of previous character ct-1, Wb is the weight matrix, and bb is the bias. Segmentation Model In the segmentation model, we still use BiLSTM. The update at time t can be written as follows:

Gate Mechanism To learn the degree of influence of cooccurrence information on the segmentation model, we use a gate mechanism similar to a GRU [Cho et al., 2014]:

where zt and rt are the update and reset gate respectively. The new hidden state ht is connected to a dense layer with linear transformation and softmax output to predict the segmentation label:

where ˆ ys t is the probability distribution over all the segmentation labels {B, M, E, S}.

3.3 Optimization

The training set consists of two parts: the labeled source domain dataset Sl, of which we know the segmentation label and unlabeled target domain dataset Tu, of which we only have the raw text. Hence, for dataset Sl, we optimize the two language models and segmentation model together using the standard cross-entropy loss, but for dataset Tu, we simply optimize the two language models.

4 Incorporating Partially-labeled Data for Cross-domain CWS

4.1 Partially-labeled Data

We assume there is also a partially-labeled dataset Tp besides the unlabeled dataset Tu for the target domain. We consider a weakly-supervised sequence labeling setting where each sequence is partially-labeled: only a few members are labeled or we are given a candidate set of labels, only one of which is correct. Liu et al. [2014] proposed two methods to obtain partially-labeled data: lexicons and natural annotation. We also use these in this work. To use a lexicon, we match the words in the lexicon with unlabeled data, resulting in partially-labeled sentences. To use natural annotation, we culled a large amount of web text (such as from Wikipedia) from the Internet, and used this web text (usually containing many hyperlinks and other markup annotations) to obtain partially-labeled sentences. Figure 2 illustrates an example of partially-labeled data. In the sentence "美国总统住在白 宫 (American president lives in the White House)", we only know that "总统 (president)" is a word (probably inferred from a lexicon or hyperlink). Hence, the labels of "总" and "统" are "B" and "E", respectively. Because only "E", "S" can be followed by "B" and only "B", "S" can follow "E", we infer that the candidate sets of labels for "国" and "住" are {E, S} and {B, S}, respectively.

4.2 Training RNNwith Partially-labeled Data

In the CWS tasks, there are only four segmentation labels: B,M,E, S. We use a zero-one vector to represent these labels. For example, "B" is represented as y = [1, 0, 0, 0]⊤ (also known as one-hot vector) and a candidate set of labels like {E, S} is represented as yc = [0, 0, 1, 1]⊤ (we add a superscript to distinguish it (candidate set) from the one-hot vector (single label)).

In a vanilla RNN for a sequence labeling problem, given an input sequence x1, x2, . . . , xT , we know the correct label yt for each xt. Let ˆ y1, ˆ y2, . . . , ˆ yT be the prediction probabilities given by this RNN. To train this network, a common choice for the loss function is the cross-entropy loss function:

For a sequence x1, x2, . . . , xT which is partially-labeled, only a few inputs xt have a corresponding label or a candidate set of labels, while many others do not. Thus, the loss function (12) cannot be used for training over partially-labeled data. Let I denote the set of indices: for each i ∈ I, xi is given a correct label yi and let J denote another set of indices: for each j ∈ J , xj is given a candidate set of labels yc j . We optimize the following loss function Lp for partiallylabeled data:

where |I| denotes the size of I and we define ||J || as ∑j∈J ||1 - yc j ||0. || · ||0 is the l0-norm. The first term is easy to understand: we minimize the cross-entropy loss on I, which actually maximizes the prediction probability of the correct label. Here is our motivation for understanding the second term. Given a candidate set of labels, we do not know which one is the correct label, so we cannot maximize its prediction probability of it. We know, however, which labels are the incorrect labels. For example, the candidate set is {E, S}, so "B" and "M" are the incorrect labels. Instead of maximizing the prediction probability of the correct label, we can minimize the prediction probability of the incorrect labels which leads to the second term of equation (13).

The function L in equation (12) can be regarded as a special case of Lp, where I = {1, 2, . . . , T}.

5 Experiments

5.1 Datasets

To evaluate the proposed model, we performed cross-domain experiments on various domain datasets, following the previous works of Liu et al. [2014] and Zhang et al. [2014]. For the source domain, we used the labeled People's Daily (PD) dataset. For the target domain, we used four datasets from the finance, medicine, literature, and computer fields. These test and development data could be obtained from SIGHAN Bakeoff 2010. The unlabeled data for the target domain were culled from the Internet 2 (about 200K sentences for each domain), and the partially-labeled data were collected from Chinese Wikipedia 3 (about 100K sentences for each domain).

Another experiment was also conducted, using Chinese Treebank 5.0 (CTB5) as the source domain and using Zhuxian (a Chinese Internet novel, also known as the "Jade dynasty") dataset 4 (ZX) as the target domain. The ZX dataset was annotated by Liu and Zhang [2012] and Zhang et al. [2014], and the style is very different from CTB5. We collected the remaining part of this novel as our unlabeled data (about 26K). To obtain partially-labeled data (about 24K), we used a lexicon containing 51 names of characters, following Zhang et al. [2014].

All of the datasets were preprocessed using regular expressions, and the continuous English characters and digits were marked as not segmented.

5.2 Setups

In this work, we trained a total of five different models:

• BiLSTM. Our baseline model. In BiLSTM, we used a two-layer bi-directional LSTM. In the training stage, we only used the labeled source domain dataset Sl.
• BiLSTM + CRF. Another baseline model which incorporates CRF in the output layer.
• BiLSTM + PL. Besides using dataset Sl, we also used the partially-labeled target domain dataset Tp to train the BiLSTM segmentation model.
• BiLSTM + LM. In addition to BiLSTM segmentation model, we added two extra language models to incorporate co-occurrence information using unlabeled data. In both language models, we used a two-layer LSTM. In the training stage, we used the labeled source domain dataset Sl and the unlabeled target domain dataset Tu.
• BiLSTM + LM + PL. On the basis of the previous model, we also used the partially-labeled target domain dataset Tp in the training stage.

For all five models, we used Adam [Kingma and Ba, 2014] to train our networks, and the initial learning rate was set at 0.001. Every five epochs, the learning rate decayed by half. In order to avoid overfitting, we employed the dropout technique (20% dropout rate). We performed gradient clipping (we clipped the norm of the gradients at 5) to cope with the exploding gradient problem. The character embeddings were initialized by pre-trained embedding based on word2vec and then fine-tuned during training. We set the character embedding size to 100 and the LSTM hidden state size to 300. We also used bigram character embedding for the two baseline models, but we did not use it in our three models since the number of bigram character is very large (we use many unlabeled and partial labeled data) and it is very easy to overfitting. We used early stopping based on the performance on the development set. Our code will be made publicly available for reproducibility.

2 http://www.cnki.net/.

3 We used wikidump20161201 downloaded from https:// dumps.wikimedia.org/zhwiki/20161201/.

4 These data are available in http://zhangmeishan. github.io/eacl14mszhang.zip.

5.3 Results and Discussion

We first compare our methods with three supervised neural network methods, which are generally applied to domainspecific datasets. The method of Chen et al. [2015] is a character-based method and that of Cai and Zhao [2016] is a word-based method. The method of Cai et al. [2017] is a new improved version of that by Cai and Zhao [2016]. The three methods do not use any unlabeled or partially-labeled data except source domain labeled data Sl. The results (F1 scores) are shown in the first three rows of Table 1.

We also compared our methods with five other domain adaptation methods. The method of Zhang et al. [2014] is a domain adaptation method for joint Chinese segmentation and POS-tagging, which uses an external dictionary to improve its performance. They only presented the result on ZX test data, and we took it directly from their paper. Liu et al. [2014] adopted a CRF-based method, which exploits two sources of partially annotated data: lexicons and natural annotation. Jiang et al. [2013] also used massive natural annotations. Zhou et al. [2017] proposeed word-context character embeddings for semi-supervised CWS. Huang et al. [2017] incorporated a global recurrent structure to model boundary features dynamically. The results of these methods are listed in the middle five rows of Table 1.

The last five rows of Table 1 list the results of two baseline models and our three models. The results demonstrated that compared with the baseline model "BiLSTM", the model "BiLSTM + LM" incorporating unlabeled data and the model "BiLSTM + PL" incorporating partially-labeled data achieved better performances. Literature, however, was an exception. The F1 of model "BiLSTM + PL" fell to 92.31, compared with 92.60 for "BiLSTM". We suppose that there were two reasons for this. First, the words (e.g., idioms) used in Literature are not partially-labeled in Wikipedia. Second, the noise in Wikipedia has a bad effect on the performance. For example, the sentence "他|下意识|地|伸手" (he stretched out his hand subconsciously) is mis-segmented by "BiLSTM+PL" as "他|下|意识|地|伸手" and a Chinese idiom "高楼大厦" (which means "tall buildings") is missegmented as "高楼|大厦" because "意识" (consciousness) and "大厦" (skyscraper) are partially-labeled in Wikipedia. The BiLSTM model can give the correct segmentation for these two examples. In the end, when we used unlabeled and partially-labeled data together, our method (BiLSTM + LM + PL) obtained a state-of-the-art performance for almost all of the test sets except Literature.

5.4 Analysis

Improvement in OOV recall. Out of vocabulary (OOV) words are the main challenge in cross-domain CWS tasks.

In this experiment, we studied the improvement in the OOV recall brought by incorporating unlabeled or partially-labeled data. Table 2 shows the results of our models compared with BiLSTM model for five datasets: Finance (Fin), Medicine (Med), Literature (Lit), Computer (Com), and ZX. We can observed that the OOV recall can be significantly improved. Importance of gate mechanism. To understand the importance of the gate mechanism introduced in the "BiLSTM + LM" model, we ran an experiment that removed the gate. We used a concatenation of hft , hbt and hst to obtain the the new hidden state ht that replaced the gate mechanism in equation (10). Results in Table 3 showed that without gate mechanism, the performance declined in all of five datasets.

Performance with in-domain data. Although the goal of this study was to tackle the cross-domain CWS, we also wanted to know whether these methods worked for in-domain data (i.e., where the test data come from the same domain as the training data). For this reason, we performed experiments on standard benchmarks (MSR, PKU, CTB6). We used 10% data of the train set as the development set for all datasets. Since all of the three datasets were from the newswire domain, when we performed experiment on one of the datasets, we used another two datasets as unlabeled data. The partiallylabeled data was also collected from Wikipedia. The results are shown in Table 4. We can find that compared with BiLSTM, incorporating unlabeled and partially-labeled data also can improve the performance of in-domain data.

Impact of unlabeled and partially-labeled data. In our methods, we used a large amount of unlabeled and partiallylabeled data to improve the performance on the target domain. To investigate the impact of these data, we performed experiments on various sizes (3%, 16%, 33%, 66%, 100%) of unlabeled and partially-labeled data. The results with the "BiLSTM + LM" model when using various sizes of unlabeled data are shown in Table 5, and the results with the "BiLSTM + PL" model when using various sizes of partially-labeled data are shown in Table 6. From these tables, we can draw two conclusions: (1) With an increase in the amount of unlabeled data, the F1 score gradually grows as well, demonstrating that valuable information can indeed be extracted from the unlabeled data. (2) The situation is somewhat different for the partially-labeled data. the F1 score first rises up and then moves down, showing that the harmful noise contained in partially-labeled data has a negative effects on the model when the amount of data is large enough.

6 RelatedWork

Use of Unlabeled Data. Jin and TanakaIshii [2006] used branching entropy to detect boundary between characters; Li and Sun [2009] and Zhang et al. [2013] incorporated punctuation information; Sun and Xu [2011] exploited statisticsbased features distilled from unlabeled data; Shen et al. [2013] extracted maximized substrings from unlabeled data and Yang et al. [2017] used unlabeled data to pre-train submodule. Different from these models, our method utilize unsupervised co-occurrence information provided by language models. Recently, Rei [2017] and Peters et al. [2017] also proposed using language models in semi-supervised frameworks. Different from us, the method of Rei [2017] is a multitask learning method, in which the language models and the sequence labeling model share a bidirectional LSTM. The method of Peters et al. [2017] was inspired by the pre-trained word embeddings. They proposed using pre-trained bidirectional LM to add context embeddings in sequence labeling.

Use of Partially-labeled Data. Jiang et al. [2013] proposed a novel discriminative learning algorithm to utilize the knowledge in the massive natural annotations on the Internet for Chinese word segmentation. Tsuboi et al. [2008] and Triggs and Verbeek [2008] independently proposed the parameter estimation method for CRFs using partially-labeled data by marginalizing out the unknown labels. Thereafter, Yang and Vozila [2014] and Liu et al. [2014] applied this method to Chinese word segmentation. They also make use of Wikipedia data as partially-labeled data. In contrast, we proposed a method to train RNN with partially-labeled data.

7 Conclusion

In this work, we proposed a novel neural network method to incorporate unlabeled and partially-labeled data for crossdomain Chinese word segmentation. To make use of unlabeled data, two character-level language models are integrated with the segmentation model through a gate mechanism. By modifying the loss function, the model could be trained using partially-labeled data. Experiments showed that incorporating these data led to significant improvements.

Acknowledgments

The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by China National Key R&D Program (No. 2017YFB1002104), National Natural Science Foundation of China (No. 61532011, 61751201, 61473092, and 61472088), and STCSM (No. 16JC1420401, 17JC1420200).