Improving Low Resource Named Entity Recognition using Cross-lingual Knowledge Transfer

1 Introduction

Named entity recognition (NER) is defined as the extraction of a contiguous sequence of textual tokens, which represents the name of an object of a specified class, such as person, location or organization. It plays a vital role in the overall task of Information Extraction (IE) and serves as an intermediate step for subsequent IE tasks, like Relation Extraction and Entity Linking. Current state-of-the-art methods for English NER usually use deep learning algorithms, e.g., Feedforward neural network (FNN) or Recurrent neural network (RNN) [Huang et al., 2015], and build name taggers from annotated data with accompanying entity labels. Such models generalize well on new entities based on features automatically learned from the context. However, a neural-based NER

*Corresponding author.

Figure 1: Example of NER labels with bilingual lexicon.

system could still get lower performance if its hidden feature representations cannot be learned adequately, which happens frequently when the annotated data is not enough, especially in low resource scenario [Zhang et al., 2016]. In this paper, we regard English as a high resource language, and other languages, such as Dutch, Spanish, and even Chinese, as low resource languages.

To improve the performance of low resource NER, we present a neural-based sequential tagger which incorporates additional word representations learned from semantic projections based on cross-lingual knowledge. This approach is partly inspired by the previous empirical success of feature-based sequence labeling models with bilingual constraints/inferences for Chinese and English NER [Che et al., 2013; Wang et al., 2013]. Our approach is built on the state-of-the-art LSTM-CRF framework [Lample et al., 2016], which models each word of the input sentence with a contextual embedding based on Bi-LSTM, and then assigns an entity label for each word using CRF. Compared with previous work which are only based on general word embeddings, we design three strategies to enrich the semantic representations, which embodies our main contributions:

(1) We build neural networks to model the external semantic representation of each low resource language word based on the translations from high resource languages. The intuition is that different languages usually contain complementary cues about entities and these cues can be further transferred through bilingual lexicons. Figure 1 shows a simple example for Chinese name tagging. The word "本" is common in Chinese but rarely appears as a name. However, based on a Chinese-English dictionary, one of the English translation candidates of "本" is "Ben", which provides a strong semantic clue that the word is a person name in English.

(2) The lexicon is usually limited and cannot cover all low resource language words. Thus, we further design a new strategy to extend the bilingual mappings with a linear transformation function. After generating the external semantic representations for the low resource language words based on high resource translations (the output of the previous network), we learn a linear projection function between the low resource word embedding space and the high resource language translation semantic space, accordingly, the out-oflexicon low resource language words can also be estimated with a new semantic representation.

(3) For each word, we calculate its distributional probabilities over all entity types and add them as additional features to the original word representation for both low resource and high resource languages. The rationale is that the entity type distribution can be regarded as a language-independent knowledge and it is helpful for low resource NER. For example, both the English and Chinese are describing the same entity, even probably with different spelling (e.g., "United States" in English vs. "美国" in Chinese), the entity type of that entity does not change from one language to another [Ni and Florian, 2017]. And if we know that "United States" is a location in English space, then naturally we can predict that the entity type of Chinese word "美国" prefers location.

Experiments on two low resource languages (Dutch and Spanish) demonstrate that the additional semantic representations can bring in an average 4.8% F-score gain compared with general word embeddings. Moreover, we also conduct experiments on the Chinese portion of the OntoNotes 4.0 corpus. The results show that we achieve a 2.91% improvement compared to the state-of-the-art system.

2 Methodology

In this section, we first describe the background on LSTMCRF model, which is the backbone of our approach. Afterwards, we present two neural networks to learn the crosslingual semantic representation of each low resource language word based on high resource language translations. Lastly, we introduce a lexicon extension strategy to alleviate the out-of-lexicon problem and describe how to learn the entity type distribution based on original word representations and entity type representations in both languages.

2.1 Basic Model: LSTM-CRF

LSTM-CRF model is originally introduced by [Huang et al., 2015], which takes a sequence of elements as the input and outputs a sequence of category labels corresponding to the input sequence 1 . The idea has been successfully applied in POS Tagging, Chunking and NER [Liu et al., 2017a].

The approach of [Lample et al., 2016] is based on LSTM and CRF Tagging models. An illustration of this network is given in the left of Figure 2. The LSTM is a special form of recurrent neural networks (RNNs) with three gated units, input, output and forget, which could control the passing of information along the sequence and thus improves the modeling of long-range dependencies. Following [Lample et al., 2016], they take a sequence of vectors X = {x1, ..., xi, ..., xn} as input and return another sequence H = {h1, ..., hi, ..., hn} that represents some information about the sequence at every step in the input. For brevity, the details of LSTM equations are given in [Gers et al., 1999]. The conditional random field (CRF) [Jurafsky and Martin, 2000] is a probabilistic graphical model, which works in a sequential way and predict a label sequence y = {y1, ..., yi, ..., yn} corresponding to the input sequence X. They define a score function as follows:

1 Entity types are usually represented in BIOES format (which stand for Begin, Inside, Outside, End, and Single, indicating the position of the token in the entity) as this scheme has been reported to outperform others such as BIO [Ratinov and Roth, 2009].

where A is a matrix of transition scores such that Ai,j represents the score of a transition from the tag i to tag j. y0 and yn are the start and end tags of a sentence. Matrix A is therefore a square matrix of size k + 2, k is the number of tags. A softmax over all possible tag sequences yields a probability for the sequence y:

where YX represents all possible tag sequences for the input sequence X.

Furthermore, [Lample et al., 2016] incorporated characterlevel structure into word representation. Each input vector xi consists of two parts, pre-trained word-level representation wi [Mikolov et al., 2013] and task-related character-level representation ci. They adopted a bidirectional LSTM to capture information in both forward and backward directions and concatenate the outputs of these two LSTMs as ci.

2.2 Improved with Bilingual Lexicon

We present an overview of the developed networks for modeling bilingual lexicons, as illustrated in the right of Figure 2. Following the same setting in Section 2.1, given a low resource language sentence X = {x1, x2, ..., xi, ..., xn}, we assume that each word xi has a corresponding high resource language translation Ti based on the bilingual lexicon 2 . The translation Ti can be viewed as a combination of multiple translation items and each translation item consists of multiple high resource language words. To make better understanding of the high resource language translation of a low resource word, all translation items should be encoded into the encoder. One simple way is to take the concatenation of all translation words as the input for the vanilla RNN unit [Liu et al., 2017b]. We also map each high resource language translation word into its embedding vector. Therefore, translation word vectors {t1, ..., ti, ..., tl} are stacked and regarded as the translation memory unit T ∈ Rd×l , where l is the number of all translation words. An example is given in Figure 1. The Chinese word "美联储" has two translation items in English, namely "FED" and "Federal Reserve". We can get its translation sequence as ["FED", "Federal", "Reserve" ], of which size is 3.

2 We construct bilingual lexicons from online translators such as Bing Dict and FAIR (Facebook AI Research) dictionary.

Following [Liu et al., 2017b], we first present a LSTMbased network for modeling bilingual lexicons. To better encode the structural information of different translation items, we incorporate the POS-tag information of each translation item into their corresponding translation words. In our method, each POS-tag label is also mapped to a dp (a hyperparameter) dimensional vector, which is randomly initialized and optimized by the model. Then we combine the embedding of each translation word ti and its corresponding POStag vector pi as [ti, pi], and then feed it to the Bi-LSTM unit. Finally, we concatenate the outputs of forward LSTM and backward LSTM as translation representations vec. We name this model as LSTM-CRF+BLLSTM , as illustrated in the upper-right of Figure 2.

Considering that each word in the translation does not contribute equally to the semantic meaning of the original low resource word, we further introduce an attention-based network to model the bilingual lexicons, which is similar as the attention-based memory network in question answering [Sukhbaatar et al., 2015]. In detail, taking an external translation unit T ∈ Rd×l and a low resource word vector xi ∈ Rd as input, the attention model outputs a continuous vector vec ∈ Rd , which is a weighted sum of each piece of memory in T:

where l is the memory unit size, αj ∈ [0, 1] is the weight of tj and ∑j αj = 1. We implement a neural network based attention model based on previous work [Bahdanau et al., 2014]. For each piece of translation memory tj, we use a feed forward neural network to compute its semantic relatedness with the low resource word. The scoring function is calculated as follows:

After obtaining g1, g2, ...gl, we feed them to a softmax function to calculate the final importance distribution α1, α2, ...αl. We name this model as LSTM-CRF+BLATT , as illustrated in the right-middle of Figure. 2

2.3 Improved withMapping based Lexicon Extension Strategy

In the actual situation, the bilingual lexicons can not cover all low resource language words. To overcome this challenge, we design a lexicon extension strategy to estimate the translation representations of out-of-lexicon word.

Suppose there is a low resource language word set W = {w1, ..., wi, ..., wf }, each word has a low resource word vector wi and a high resource language translation vector veci. We learn a linear projection function as the transformation between the two semantic space, as follows:

where M is the mapping matrix. We minimize the following objective to optimize M:

After obtain M, for each out-of-lexicon word oi, we can estimate the translation representation veoi as follows:

2.4 Improved with Language-Independent Entity Type Distribution

In this section, we introduce the entity type-based distributional features, which denotes the probabilities of each word to be tagged as each entity type. Word embeddings have been empirically shown to preserve linguistic regularities, such as similar words tend to be close to each other in the same space [Mikolov et al., 2013]. We observe that the same property also applies to the words with the same entity type. For example, the distance between the word "Microsoft" and "Bill Gates" is larger than that between "Microsoft" and "IBM". Therefore, we can learn an approximate representation of each entity type, and use the similarities between each entity type representation and each word embedding as the entity type-based distributional feature.

In this work, we focus on three most common named entity types, i.e., P (Person), L (Location), O (Organization), and discard the others. Taking low resource language as an example, we randomly select 10 entities from each entity type and average their embeddings as the entity type representation. At the same time, we randomly generate one vector representing non-entity; Therefore, four entity type vectors {EP , EO, EL, EN} are constructed, each Ej ∈ Rd . Afterwards, we use standard cosine function to calculate the semantic relatedness between the low resource word embedding wi and the entity type representation Ej .

For high resource language, we also calculate the entity distribution of each word. In the end, each low resource language word and each high resource language word are assigned with an entity distributional feature vector with dimensionality 4, ei = {eP , eO, eL, eN}, as illustrated in the bottom-right of Figure 2.

2.5 LowResource NER

Now, the vectors of each word in the input low resource sentence is made up of four parts: a word embedding wi, a character-level representation ci, a high resource translation vector veci or veoi, and an entity type distributional representation ei. we regard the concatenation vectors of these four representations as word representation xi = [wi, ci, veci, ei], and feed them into the previous LSTM-CRF model for NER (Section 2.1). The model is trained in a supervised manner by minimizing the cross entropy error of sequence labeling and L2 loss:

where p(y|X) is the probability of predicting sequence X as tag sequence ˜y. C denotes all training sentences. X is the input sentence representations. YX represents all possible tag sequences for the input sequence. pg (˜y|X) is 1 or 0, indicating whether the correct sequence tag is ˜y. We use back propagation to calculate the gradients of all the parameters, and update them with stochastic gradient descent. We randomize other parameters with uniform distribution U(0.01, 0.01), and set the learning rate as 0.01. Table 1 illustrates the word embedding parameters used in our experiments 3 . For brevity, the details of other parameters are given in our codes 4 .

3 Experiment

We apply our neural architecture for NER on various datasets and evaluate the effectiveness separately. In this section, we will describe the detailed experimental settings and discuss the results.

3.1 Dataset

We evaluate the proposed approach on two low resource languages (including Spanish and Dutch 5 ), and Chinese 6 , which is distinct from Latin-based languages. In this paper, we regard English as high resource language and all other languages as low resource languages. Table 2 shows the detailed description of the data sets used in our experiments. In this paper, we focus on four entity types (Person, Location, Organization, None), which are commonly adopted in previous NER studies [Che et al., 2013; Wang et al., 2013].

3.2 LowResource NER

We compare with the following baseline methods on the two languages.

• LSTM-CRF [Lample et al., 2016] is introduced in section 2.1. Compared with standard Bi-LSTM, it adds a CRF layer to impose several hard constraints of the "grammar".
• LM-LSTM-CRF [Liu et al., 2017a] is also a LSTMCRF-based sequence labeling framework and incorporates residual network and language model to extract character-level knowledge from the self-contained order information.
• CLNER (Cross-Lingual Named Entity Recognition) [Ni et al., 2017], a weakly supervised method, which creates automatically labeled NER data for a target language via

3 FAIR:https://github.com/facebookresearch/MUSE

4 Our code is available at: https://github.com/scir-code/lrner.

5 CoNLL: https://github.com/synalp/NER/tree/master/corpus/

6 Ontonotes: https://catalog.ldc.upenn.edu/ldc2011t03

annotation projection on comparable corpora. The recognizer is a prototype-based neural model.

Our model has several variations, which are detailed below.

• LSTM-CRF+BLLSTM extends LSTM-CRF by taking into account of the bilingual lexicon, and uses a LSTMbased network towards the translations.
• LSTM-CRF+BL+MLSTM : an extension of LSTMCRF+BLLSTM by further incorporating the lexicon extension strategy.
• LSTM-CRF+BL+M+ELSTM : an extension of LSTMCRF+BL+MLSTM by further concatenating the semantic representation of each word with entity type distribution features in both languages.
• LSTM-CRF+BLATT extends LSTM-CRF by taking into account of the bilingual lexicon, and uses an attentionbased network towards the translations.
• LSTM-CRF+BL+MATT : an extension of LSTMCRF+BL+ATT by further incorporating the lexicon extension strategy.
• LSTM-CRF+BL+M+EATT : an extension of LSTMCRF+BL+M+ATT by further concatenating the semantic representation of each word with entity type distribution features in both languages.

Experimental results are given in Table 3. Evaluation metric is F measure [Manning and Sch¨utze, 1999]. We can find that our method LSTM-CRF+BL+M+ELSTM yields the best performance on two languages compared with many strong baselines. The performance of CLNER are relatively low because both of them utilize indirectly acquired features based on linguistic resources and cross-lingual entity mappings. LSTM-CRF obtains significant improvement over CLNER by integrating semantic representations of low resource language words and learning the constraints between entity labels. Besides, we surprisingly find that all of our variants outperform the strong baseline LSTM-CRF on two languages, which demonstrate the effectiveness of each of the additional semantic representations for low resource NER. Among the six variants of our model, LSTM-based models perform better than attention-based models, which indicates that the sequence feature is more important for modeling lexicon structure. In the last, two real examples are given in Table 4 to demonstrate the effectiveness of the additional bilingual lexicon representations for low resource NER.

3.3 Chinese NER

To demonstrate the effectiveness of our models on largescale corpora, we show the results on Chinese Ontonotes 4.0 NER in Table 5. Additionally, we add a strong baseline for Chinese NER with bilingual constraints, namely Soft-Align 7 . From Table 5, we can still get consistent improvements on Chinese NER over previous state-of-the-art methods. Specifically, we observe that LSTM-CRF+BL+M+ELSTM achieves a significant gain in Recall. This is reasonable since the semantic representation of each Chinese word is much richer in LSTM-CRF+BL+M+ELSTM than other models.

3.4 Fine-Grained Performance on Different Groups

This subsection studies the effectiveness of our cross-lingual representations. For comparison purposes, we select the baselines: LSTM-CRF [Lample et al., 2016] and LM-LSTM-CRF [Liu et al., 2017a] and compare them with LSTM-based bilingual lexicon models. Moreover, to prove that cross-lingual representation could capture more valuable semantics, especially for the entities that appear in the testing data but never appear in the training data, we divide the entities in the testing data into two parts (A: appearing in both testing and training data with the same entity type, or B: appearing in testing data only) and perform evaluations separately. Experimental results are shown in Table 6 and illustrate that for all situations, cross-lingual representation brings in significant improvements compared with word embedding and characterlevel representation in NER. For situation B in three languages, our model (LSTM-CRF+BL+M+ELSTM ) yields an average 6.44% improvement, which is 2 times in situation A. This demonstrates that the cross-language representation has better ability to model non-covered entities than word-level and character-level representations.

7 Che et al. proposed a novel Integer Linear Programming-based inference algorithm with bilingual constraints for English and Chinese NER.

4 RelatedWork

There exist two threads of related work regarding the topics in this paper, which are Monolingual NER and how to improve it with other languages (Cross-lingual NER).

4.1 Monolingual NER

Named entity recognition is typically regarded as a kind of sequence labeling problem in literature. Therefore, standard feature-based classification approaches such as conditional random fields (CRFs) [Lafferty et al., 2001], hidden markov models (HMMs) [Florian et al., 2003] and maximum entropy classifiers [Chieu and Ng, 2002] can be naturally employed to build a name tagger. Despite the effectiveness of feature engineering, it is labor intensive and unable to discover the discriminative or explanatory factors of data [Bengio et al., 2015]. To handle this problem, some recent studies [Chiu and Nichols, 2015; Santos and Guimar˜aes, 2015; Huang et al., 2015; Yang et al., 2017; Lample et al., 2016; Liu et al., 2017a; Peters et al., 2017; Ma and Hovy, 2016] used neural network methods and got promising results . The representative approaches include BiLSTM-CNN [Chiu and Nichols, 2015; Santos and Guimar˜aes, 2015], CNN-CRF [Huang et al., 2015; Yang et al., 2017], LSTM-CRF [Lample et al., 2016; Liu et al., 2017a; Peters et al., 2017] and LSTM-CNN-CRF [Ma and Hovy, 2016]. Furthermore, character-based representations had been proved to be effective in capturing the orthographic and morphological evidence. Also, most of these models added a CRF layer, and reported significant improvement over pure RNN models. Our architecture is based on the success of LSTM-CRF model and is further modified to enrich the word representation with cross-lingual knowledge information.

4.2 Cross-lingual NER

The idea of utilizing multilingual resources to improve monolingual name tagger systems has been studied extensively. [Li et al., 2012] presented a cyclic CRF model and performed approximate inference using loopy belief propagation. Although, their feature-rich CRF formulation of bilingual edge is powerful, an obvious drawback of this approach is the requirement of manually annotate bilingual NER data. Therefore, [Chen et al., 2010] proposed approaches to extract bilingual named entity pairs from unannotated bitext. The verification was based on bilingual named entity dictionaries. In this regard, one of the most interesting papers is [Burkett et al., 2010], which explored an "up-training" mechanism by using the outputs from a strong monolingual model as groundtruth, and thereby simulated a learning environment, where a bilingual model is trained to help a "weakened" monolingual model recover the results of the strong model. [Kim et al., 2012] proposed a method of labeling bilingual corpora with named entity labels automatically based on Wikipedia. [Che et al., 2013; Wang et al., 2013] tackled the problem of jointly recognizing and aligning bilingual named entities. For low resource NER, [Zhang et al., 2016] proposed an expectation-driven model that designed a large number of language-specific features (rules, patterns, gazetteers, etc.) via consulting and encoding linguistic knowledge from native speakers. [Ni and Florian, 2017; Ni et al., 2017] developed approaches to improve multilingual name tagging performances with Wikipedia entity type mapping and word distribution mapping. However, these methods suffer from error propagation. Moreover, the selection and collection of task related features are time-consuming and labor intensive. Our approach differs in that it does not acquire any hand-craft features and bilingual lexicons are one of the most basic language resources for all languages.

5 Conclusions

Low resource NER is a very important yet challenging problem in natural language processing. In this paper, we focus on this problem by incorporating cross-lingual knowledge into a neural architecture, which guides low resource name tagging to achieve a better performance. Specifically, we use bilingual lexicons to bridge cross-lingual semantic mapping and design a lexicon extension strategy to alleviate the out-of-lexicon problem. Moreover, we regard entity type distribution as language-independent features and model them in our architecture. Experiments on three languages, namely, Dutch, Spanish and Chinese demonstrate the effectiveness of our model for low resource language NER. In the future, we will incorporate other knowledge resources, such as FrameNet and WordNet, from high resource languages into our neural architecture. We will also extend our architecture to other NLP tasks, such as event extraction, sentiment analysis.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (NSFC) via grant 61632011, 61772156 and 61472107.