A Feature-Enriched Neural Model for Joint Chinese Word Segmentation and Part-of-Speech Tagging

1 Introduction

Chinese word segmentation and part-of-speech (POS) tagging are two core and fundamental tasks in Chinese natural language processing (NLP). The state-of-the-art approaches are based on joint segmentation and tagging (S&T) model, which can be regarded as character based sequence labeling task. The joint model can alleviate the error propagation problem of pipeline models.

Previously, the traditional hand-crafted feature based models have achieved great success on joint S&T task [Jiang et al., 2008; Kruengkrai et al., 2009; Qian et al., 2010; Zhang and Clark, 2008; 2010]. Despite of their success, their performances are easily affected by following two limitations.

The first is model complexity. Since the decoding space of joint S&T task is relatively large, the traditional models often rely on millions of discrete features. Therefore, the efficiency of joint S&T models is rather low. Moreover, these models suffer from data sparsity. Recently, some neural models [Huang et al., 2015; Chen et al., 2015a; Ma and Hovy, 2016] are proposed to reduce the efforts of feature engineering and the model complexity. However, these neural models just concatenate the embeddings of the context characters, and feed them into neural network. Since the concatenation operation is relatively simple, it is difficult to model the complicated features as the traditional discrete feature based models. Although the complicated interactions of inputs can be modeled by the deep neural network, the previous neural models show that the deep model cannot outperform the one with a single non-linear model.

∗ Corresponding author.

The second is long term dependency. Unlike pure POS tagging task which can utilize contextual features on word level, joint S&T task usually extracts the contextual features on character level. Thus, the joint model need longer dependency on character level. As the example shown in Figure 1, conditional random field (CRF) model makes mistakes on words "改革 (reform)" and " 精简 (simplify)" since it is hard for CRF to disambiguate the POS tags without using long distance information. However, restricted by model complexity and data sparsity, a larger window size (greater than 5) will instead hurt the performance. Therefore, how to exploit the long distance information without increasing the model complexity is crucial to joint S&T task.

In order to address these two problems, we propose a feature-enriched neural model for joint S&T task, which consists of several key components: (1) a convolutional layer to simulate compositional features as complex hand-crafting features; (2) a pooling layer to select the most valuable features; (3) a bi-directional long short-termmemory (BLSTM) layer on the top to carry long distance dependency information. In addition, we introduce a highway layer [Srivastava et al., 2015] to increase the depth of architecture and obtain more sophisticated feature representation without sufftering from the problem of gradient vanishing, leading to fast convergence.

Our contributions could be summaries as follows:

1. We propose a customized neural architecture for joint

S&T task, in which each component is designed according to its specific requirements, instead of a general deep neural model.

1. Our model can alleviate two crucial problems: model complexity and long term dependency in joint S&T task.
2. We evaluate our model on five different datasets. Experimental results show that our model achieves comparable performance to the previous sophisticated feature based models, and outperforms the previous neural models.

2 Neural Models for Joint S&T

The joint S&T task is usually regraded as a character based sequence labeling problem.

In this paper, we employ the {BMES} tag set TSEG (indicating the Begin, Middle, End of the word, and a Single character word respectively) for word segmentation and the tag set TPOS (varies from dataset to dataset) for POS tagging.

The tag set T of our joint S&T task would be the cross-label set of TSEG and TPOS. As illustrated in Figure 1, we would have a tag B_VV for character " 结", where B ∈ TSEG and VV ∈ TPOS , indicating the first character of the VV word "结 合".

Conventional neural network based model for sequence labeling task usually consists of three phases. Figure 2 gives the illustration.

2.1 Lookup Table Phase

In order to represent characters as distributed vectors, we usually apply a feed-forward neural layer on the top of the one-hot character representations. The parameter matrix of the neural layer is called character embedding matrix E ∈ R|C|×d , where C is the character set and d is the dimensionality of the character embeddings. For a given sentence c1:n of length n, the first step is to lookup embeddings of the characters in the current window slide ci-⌊ k-1 2 ⌋:i+⌈ k-1 2 ⌉ for the current character ci which is going to be tagged, where k is a hyperparameter indicating the window size. By concatenating the embeddings, we get the representation xi for the current character ci.

2.2 Encoding Phase

Usually, we apply a linear transformation followed by a nonlinear function to the current input xi:

where Wh ∈ Rkd×h and bh ∈ Rh is the trainable parameters, and h is the dimensionality of the hidden layer, g(·) is a nonlinear function which could be sigmoid(·), tanh(·), etc.

Then, we could get the score vector pi ∈ R|T | for each possible tags of current character ci by applying a linear transformation layer to the hidden layer hi:

where Wp ∈ Rh×|T | and bp ∈ R|T | is the trainable parameters, and T is the joint tag set.

2.3 Decoding Phase

The decoding phase aims to select the best tag sequence ˆ t1:n, to maximize the reward function r(·):

where A ∈ R|T |×|T | is the transition parameter, indicating how possible a label will transfer to another. T(c1:n) indicates all possible tag sequences for sentence c1:n.

Also, we employ the Viterbi algorithm [Forney Jr, 1973] to decode the best tag sequence in polynomial time complexity.

3 A Feature-Enriched Neural Model for Joint S&T

The simple neural model presented above achieves good results on the joint S&T task. However, the simple neural model, who concatenates the embeddings of contextual characters as features, is not as strong as models based on the handcrafted features. Thus, a simple shallow neural is insufficient to tackle with ambiguous cases which rely on more sophisticated feature combinations and long distance dependencies.

To deal with these issues, we propose a feature-enriched neural model for joint S&T task, which consists of three different types of neural layers, stacked one by one: (1) Convolutional layer; (2) Highway layer; (3) Recurrent layer. Figure 3 gives the illustration.

3.1 Convolutional Layer

The simple neural model is just to concatenate the embeddings of characters in a local context, which cannot simulate the carefully designed features in traditional models.

To better model the complex compositional features as conventional feature based models, we use convolution layer to separately model different n-gram features for each character. Thus the feature of each character is the concatenation of corresponding columns of all different feature map sets. Then we apply a k-max pooling layer to select the most significant signals.

Concretely, we model uni-gram, bi-gram, . . . , Q-gram features by generating feature map sets ˆz1 , ˆz2 , . . . , ˆzQ correspondingly. Formally, the q-gram feature map set ˆzq is:

where Wqcov ∈ Rqd×lq is the convolutional filter for q-gram feature map set, and xi-⌊ q-1 2 ⌋:i+⌈ q-1 2 ⌉ ∈ Rqd is the concatenation of embeddings of characters ci-⌊ q-1 2 ⌋:i+⌈ q-1 2 ⌉ . Here, lq is the number of feature maps in q-gram feature map set and b ∈ Rlq is a bias parameter. For marginal cases, we use wide convolution strategy, which means we receive the sequence in the same length as input by padding zeros to the input.

Then, we would represent the original sentence by concatenation operation as z ∈ Rn×∑Qq=1 lq :

where operator ⊕ is the concatenation operation.

After taking the k-max pooling operation, the representation of original sentence would be ˆX ∈ Rn×d = [ˆx1, ˆx2, . . . , ˆxn]ᵀ , where ˆxi is:

Hence, after convolutional layer, we would represent the given input sequence X ∈ Rn×d = [x1, x2, . . . , xn]ᵀ as ˆX = Cov(X).

3.2 Highway Layer

Highway layer [Srivastava et al., 2015] aims to keep gradient in very deep neural network. By introducing highway layer, we could simulate more complex compositional features by increasing the depth of our architecture. In addition, highway layer speeds up convergence speed and alleviates the problem of gradient vanishing.

As described above, we would represent the input sequence as ˆX = Cov(X) after the convolutional layer. By additionally adding the highway layer, the representation of the input sequence would be ˆX as:

where operator ⊙ indicates the element-wise multiplication operation. The T (·) is the transform gate and C(·) is the carry gate. We adopt a simple version, where we set C(·) = 1 - T(·). Transform gate T(·) could be formalized as:

where WT ∈ Rd×d and bT ∈ Rd are trainable parameters. Here σ is the sigmoid function.

3.3 Recurrent Layer

In joint S&T task, it usually relies on long distance dependency and sophisticated features to disambiguate lots of cases. Thus, a simple shallow neural model is insufficient to capture long distance information.

Inspired by recent works using long short-term memory (LSTM) [Hochreiter and Schmidhuber, 1997] neural networks, we utilize LSTM to capture the long-term and short-term dependencies. LSTM is an extension of the recurrent neural network (RNN) [Elman, 1990], which aims to avoid the problems of gradient vanishing and explosion, and is very suitable to carry the long dependency information.

By further adding LSTM layer on the top of ˆX ∈ Rn×d = [ˆx1, ˆx2, . . . , ˆxn], we would represent sentence c1:n as H ∈ Rn×h = LSTM(ˆX) = [h1, h2, . . . , hn]. Specifically, LSTM

layer introduces memory cell c ∈ Rh which controlled by input gate i ∈ Rh , forget gate f ∈ Rh and output gate o ∈ Rh . Thus, each output hi ∈ Rh would be calculated as:

where Wg ∈ R4h×(d+h) and bg ∈ R4h are trainable parameters. Here, the hyper-parameter h is dimensionality of i, o, f, c and h. σ(·) is sigmoid function and ϕ(·) is tanh function.

BLSTM We also employ the bi-directional LSTM (BLSTM) neural network. Specifically, each hidden state of BLSTM is formalized as:

where operator ⊕ indicates concatenation operation. Here, -→h i and ←- h i are hidden states of forward and backward LSTMs respectively.

4 Training

We employ max-margin criterion [Taskar et al., 2005] which provides an alternative to probabilistic based methods by optimizing on the robustness of decision boundary directly.

In the decoding phase, if the predicted tag sequence for the i-th training sentence c(i) 1:ni with the maximal score is ˆ t(i) 1:ni :

the goal of the max-margin criterion is to maximize the score of the gold tag sequence t∗(i) 1:ni = ˆ t(i) 1:ni with a margin to any other possible tag sequence t(i) 1:ni ∈ T(c(i) 1:ni ):

where ∆(t∗(i) 1:ni , t(i) 1:ni ) is the margin function and hyperparameter η is a discount parameter. Here, θ denotes all trainable parameters of our model.

Thus, the object is to minimize objective function J(θ) for m training examples (c(i) 1:ni , t∗(i) 1:ni )mi=1 :

5 Experiments

5.1 Datasets

We evaluate proposed architecture on five datasets: CTB, PKU, NCC, CTB-5, CTB-7. Table 1 gives the details of five datasets. We use the first 10% data of shuffled train set as development set for CTB, PKU, NCC and CTB-7 datasets.

• CTB, PKU and NCC datasets are from the POS tagging task of the Fourth International Chinese Language Processing Bakeoff [Jin and Chen, 2008].
• CTB-5 dataset is the version of Penn Chinese Treebank 5.1, following the partition criterion of [Jin and Chen, 2008; Jiang et al., 2009; Sun and Wan, 2012]
• CTB-7 dataset is the version of Penn Chinese Treebank 7.0. It consists of different sources of documents (newswire, magazine articles, broadcast news, broadcast conversations, newsgroups and weblogs). Since the web blogs are very different with news texts, we try to evaluate the robustness of our model by testing on web blogs and training on the rest of dataset.

5.2 Hyper-parameters

Table 2 gives the details of hyper-parameter settings. Note that we set window size k = 1 which means we only take the current character embedding into account instead of using window slice approach. According to experiment results, we find it is a tradeoff between model performance and efficiency to only use { uni-gram, bi-gram, . . . , 5-gram } convolutional feature map sets. Besides, we set sizes of all feature map sets consistently for simplicity. Following previous work [Pei et al., 2014; Chen et al., 2015b], we also adopt bigram-character embedding in this paper.

5.3 Effects of Components

We experiment several models by using different neural component layers as shown in Table 3. The model incorporating convolutional layer, pooling layer, highway layer, and BLSTM layer, achieves the best performance on F1 score (90.39) on test set of CTB dataset. Therefore, we would like to compare our approach with other previous works using this topology.

Notably, the conventional model using window slice approach (Figure 2) for joint S&T task can be viewed as a special case of our model when we only adopt a singe convolutional layer.

Pooling Layer and Highway Layer

To evaluate the effectiveness of pooling layer and highway layer, we incrementally add pooling layer and highway layer on the top of convolutional layer. As shown in Table 3, by adding pooling layer, the performance decrease a little for the loss of information. However, we get the better performance on F1 score (90.24) by additionally adding highway layer. Although the performance does not benefit from pooling layer much, the pooling layer extracts the most important features and meets the consistent dimensionality requirement to add highway layer. Intuitively, highway layer helps simulating more complex compositional features by increasing the depth of architecture.

Long Short-Term Memory Layer

In this work, we introduce (B)LSTM layer to carry the long distance dependency. To evaluate (B)LSTM layer, we experiment different models with and without (B)LSTM layer. As shown in Table 3, we could get a relatively high performance by using LSTM or BLSTM layer only, which shows the capability of (B)LSTM in modeling features and carrying long distance information.

By introducing convolutional layer and highway layer, we could further boost the performance which benefits from the feature modeling capability of convolutional layer and highway layer.

5.4 Comparsion with Previous Works

We compare proposed model with several previous works on five datasets on joint S&T task. Experimental results are shown in Table 4.

Conditional random field (CRF) [Lafferty et al., 2001] is one of the most prevalent and widely used models for sequence labeling tasks. [Qiu et al., 2013] aims to boost the performance by exploiting datasets with different annotation types. Multilayer perceptron (MLP) is our implementation of [Zheng et al., 2013], a basic neural model for joint S&T task. [Zheng et al., 2013] is a neural model which only use one layer of shallow feed forward neural network in the encoding phase. Our model indicates the model with convolutional layer, pooling layer, highway layer, and BLSTM layer. "Pretrain" indicates the pre-trained character embeddings which are trained on corresponding train set of each dataset using word2vec toolkit [Mikolov et al., 2013].

Result Discussion Our model outperforms the previous neural model on joint S&T task and achieves the comparable performance with conventional hand-crafted feature based models. As shown in Table 4, compared to other previous methods, our model achieves the best performances on F1 scores (90.39, 90.16, 88.76, 85.31 on CTB, PKU, NCC and CTB-7 datasets respectively), and obtains comparable results on CTB5 dataset (93.19 on F1 score). As we know, the test set of CTB5 is very small so that previous work might overfit on that dataset. In addition, according to the experimental results, we find that the performance benefits a lot from pretrained character embeddings. Intuitively, pre-trained embeddings give a more reasonable initialization for the non-convex optimization problem with huge parameter space.

∗ [Zheng et al., 2013] only reported the results on CTB5 dataset for joint S&T task.

Besides, the proposed model is quite efficient. It only takes about half one hour per epoch using a small amount of memory (to train CTB) on a single GPU. Actually, it takes about ten hours to train our model (on CTB).

Experiments on CTB-7, whose train set and test set are on different domains, show the robustness of our model.

5.5 Case Study

We illustrate several cases from CTB-5 dataset. As shown in Table 5, our approach performs well on cases with disambiguations which rely on long distance dependency. For instance, conditional random field (CRF) model makes mistakes on words " 改革" and "精简" since it is hard for CRF to disambiguate the POS tags without using the long distance (wider contextual) information.

6 RelatedWorks

Recently, researches applied deep learning algorithms on various NLP tasks and achieved impressive results, such as chunking, POS tagging, named entity recognition for English [Collobert et al., 2011; Tsuboi, 2014; Labeau et al., 2015; Ma and Hovy, 2016; Santos and Zadrozny, 2014; Huang et al., 2015], and Chinese word segmentation and POS tagging for Chinese [Zheng et al., 2013; Pei et al., 2014; Chen et al., 2017]. These models learn features automatically which alleviate the efforts in feature engineering. However, joint S&T is a more difficult task than Chinese word segmentation and POS tagging since it has a larger decoding space and need more contextual information and long distance dependency [Zhang and Clark, 2008; Jiang et al., 2008; Kruengkrai et al., 2009; Zhang and Clark, 2010; Sun, 2011; Qian and Liu, 2012; Zheng et al., 2013; Qiu et al., 2013; Shen et al., 2014]. Therefore, we need a customized architecture to alleviate these problems. In this work, we propose a feature-enriched neural model for joint S&T task, and obtain great performance.

Besides, there are several similar neural models [Tsuboi, 2014; Labeau et al., 2015; Ma and Hovy, 2016; Santos and Zadrozny, 2014; Huang et al., 2015; Kim et al., 2015] . Instead of looking up word embedding table for each word in text, they tries to directly model English words by applying convolution layer on characters of words. Then they apply these word presentations to other tasks, such as POS tagging, name entity recognition, language modeling, etc. Unlike these models, we apply convolutional operation on sentence level, while they do within each word. Therefore they do not capture the features involving several words. Besides, we apply pooling operation along the feature size direction to get the most significant features.

7 Conclusions

In this paper, we propose a feature-enriched neural model for joint S&T task, which better models compositional features and utilizes long distance dependency. Experimental results show that our proposed model outperforms the previous neural model and achieves comparable results with previous sophisticated feature based approaches.

Acknowledgments

We would like to thank the anonymous reviewers for their valuable comments. This work is partially funded by National Natural Science Foundation of China (No. 61532011 and 61672162), Shanghai Municipal Science and Technology Commission on (No. 16JC1420401).