Robust Multilingual Named Entity Recognition with Shallow Semi-supervised Features (Extended Abstract)

1 Introduction

A named entity can be mentioned using a great variety of surface forms (Barack Obama, President Obama, Mr. Obama, B. Obama, etc.) and the same surface form can refer to a variety of named entities. For example, according to the English Wikipedia, the form 'Europe' can ambiguously be used to refer to 18 different entities, including the continent, the European Union, various Greek mythological entities, a rock band, some music albums, a magazine, a short story, etc. 1

∗This paper is an extended abstract of the Artificial Intelligence Journal publication [Agerri and Rigau, 2016], http://dx.doi.org/10.1016/j.artint.2016.05.003

1 http://en.wikipedia.org/wiki/Europe

Most Named Entity Recognition (NER) taggers are supervised statistical systems that extract patterns and term features which are considered to be indications of Named Entity (NE) types using the manually annotated training data (extracting orthographic, linguistic and other types of evidence) and often external knowledge resources. As in other NLP tasks, supervised statistical NER systems are more robust and obtain better performance on available evaluation sets, although sometimes the statistical models can also be combined with specific rules for some NE types. For best performance, supervised statistical approaches require manually annotated training data, which is both expensive and timeconsuming. This has seriously hindered the development of robust high performing NER systems for many languages but also for other domains and text genres in what we will henceforth call 'out-of-domain' evaluations.

Moreover, supervised NER systems often require finetuning for each language and, as some of the features require language-specific knowledge, this poses yet an extra complication for the development of robust multilingual NER systems. For example, it is well-known that in German every noun is capitalized and that compounds including named entities are pervasive. This also applies to agglutinative languages such as Basque, Korean, Finnish, Japanese, Hungarian or Turkish. For this type of languages, it had usually been assumed that linguistic features (typically Part of Speech (POS) and lemmas, but also semantic features based on WordNet, for example) and perhaps specific hand-crafted rules, were a necessary condition for good NER performance as they would allow to capture better the most recurrent declensions (cases) of named entities for Basque [Alegria et al., 2006] or to address problems such as sparsity and capitalization of every noun for German [Faruqui et al., 2010; Benikova et al., 2014; 2015]. This language dependency was easy to see in the CoNLL 2002 and 2003 tasks, in which systems participating in the two available languages for each edition obtained in general different results for each language. This suggests that without fine-tuning for each corpus and language, the systems did not generalize well across languages [Nothman et al., 2013].

This paper presents a multilingual and robust NER system based on simple, general and shallow features that heavily relies on word representation features for high performance. Even though we do not use linguistic motivated features, our approach also works well for inflected languages such as Basque and German. We demonstrate the robustness of our approach by reporting best results for five languages (Basque, Dutch, German, English and Spanish) on 12 different datasets, including seven in-domain and eight out-ofdomain evaluations.

2 Datasets

The CoNLL NER shared tasks focused on language independent machine learning approaches for 4 entity types: person, location, organization and miscellaneous entities [Tjong Kim Sang, 2002]. The 2002 edition provided manually annotated data in Dutch and Spanish whereas in 2003 the languages were German and English. In addition to the CoNLL data, for English we also use the formal run of MUC 7 and Wikigold for out-of-domain evaluation [Chinchor and Marsh, 1998; Nothman et al., 2013]. The Wikigold corpus consists of 39K words of English Wikipedia manually annotated following the CoNLL 2003 guidelines. For Spanish and Dutch, we also use Ancora 2.0 [Taul´e et al., 2008] and SONAR-1 [Desmet and Hoste, 2014], respectively. In Basque the only gold standard corpus is Egunkaria [Alegria et al., 2006]. In the datasets mentioned so far, named entities were assumed to be non-recursive and non-overlapping. The exceptions are the GermEval 2014 shared task data for German and MEANTIME, where nested entities are also annotated (both inner and outer spans). The GermEval 2014 NER shared task [Benikova et al., 2014] annotated 12 NE types: person, location, organization, other plus their sub-types annotated at their inner and outer levels. Finally, the MEANTIME corpus [Minard et al., 2016] is a multilingual (Dutch, English, Italian and Spanish) publicly available evaluation set annotated within the Newsreader project 2 .

3 RelatedWork

Named entity recognition is a task with a long history in NLP. Therefore, we will summarize those approaches that are most relevant to our work. Since CoNLL shared tasks, the most competitive approaches have been supervised systems learning CRF, SVM, Maximum Entropy or Averaged Perceptron models, although the most recent approaches are based on deep-learning, [Lample et al., 2016; Ma and Hovy, 2016]. In any case, while the machine learning method is important, it has also been demonstrated that good performance might largely be due to the feature set used [Clark and Curran, 2003]. Furthermore, and as argued by the CoNLL 2003 organizers, no feature set was deemed to be ideal for NER, although many approaches for English.

Linguistic features (POS, lemmas, chunks, but also specific rules or patterns) have been widely used, although these type of features was deemed to be most important for German, [Benikova et al., 2014]. This is caused by the sparsity caused by the declension cases, the tendency to form compounds containing named entities and by the capitalization of every noun [Faruqui et al., 2010]. For example, the best system among the 11 participants in GermEval 2014, ExB, uses morphological features and specific suffix lists aimed at capturing frequent patterns in the endings of named entities [H¨anig et al., 2014]. In agglutinative languages such as Basque, which contains declension cases for named entities, linguistic features are considered to be a requirement. For example, the country name 'Espainia' (Spain in Basque) can occur in several forms, Espainian, Espainiera, Espainiak, Espainiarentzat, Espainiako, and many more. 3 The only previous work for Basque developed Eihera, a rule-based NER system formalized as finite state transducers to take into account declension classes [Alegria et al., 2006]. The features of Eihera include word, lemma, POS, declension case, capitalized lemma, etc. These features are complemented with semantic information from the Basque WordNet.

2 http://www.newsreader-project.eu

Global features have also been proposed: [Carreras et al., 2002] used a function to inject features for the whole sentence whereas [Ratinov and Roth, 2009] developed three types of non-local features, analyzing global dependencies in a window of between 200 and 1000 tokens.

Recently, semi-supervised approaches leveraging unlabeled text had already been applied to improve results in various NLP tasks, including NER. [Ratinov and Roth, 2009] used Brown clusters, [Turian et al., 2010] made a rather exhaustive comparison of word embeddings and clustering methods. They show that in some cases the combination of word representation features was positive but they did not manage to improve over the state of the art. Furthermore, they reported that Brown clustering features performed better than the word embeddings. [Passos et al., 2014] extend the Skip-gram algorithm to learn 50-dimensional lexicon infused phrase embeddings from 22 different gazetteers and the Wikipedia and obtained 90.90 F1 score on the CoNLL 2003 English test set. [Faruqui et al., 2010] trained the Stanford NER system with a variety of features, including lemma, POS tag, etc, including features based on Clark clusters [Clark, 2003] improving the results over the best CoNLL 2003 system by 4 points in F1. Finally, state of the art results have been obtained by deep learning approaches based on LSTMs and CNNs [Lample et al., 2016; Ma and Hovy, 2016] and trained with word embeddings (91.21 F1 in CoNLL 2003 benchmark).

4 SystemDescription

The design of ixa-pipe-nerc aims at establishing a simple and shallow feature set, avoiding any linguistic motivated features, with the objective of removing any reliance on costly extra gold annotations (POS tags, lemmas, syntax, semantics) and/or cascading errors if automatic language processors are used. The underlying motivation is to obtain robust models to facilitate the development of NER systems for other languages and datasets/domains while obtaining state of the art results. Our system consists of: (i) Local, shallow features based mostly on orthographic, word shape and n-gram features plus their context; (ii) three types of simple clustering features, based on unigram matching; (iii) publicly available gazetteers. ixa-pipe-nerc learns supervised models via the Perceptron algorithm as described by [Collins, 2002]. To avoid duplication of efforts, ixa-pipe-nerc uses the Apache OpenNLP project implementation of the Perceptron algorithm 4 customized with its own features. Specifically, ixapipe-nerc implements, on top of the local features, a combination of word representation features: (i) Brown [Brown et al., 1992] clusters, taking the 4th, 8th, 12th and 20th node in the path; (ii) Clark [Clark, 2003] clusters and, (iii) Word2vec [Mikolov et al., 2013] clusters, based on K-means applied over the extracted word vectors using the skip-gram algorithm. The implementation of the clustering features looks for the cluster class of the incoming token in one or more of the clustering lexicons induced following the three methods listed above. If found, then we add the class as a feature. The Brown clusters only apply to the token related features, which are duplicated.

3 English: in Spain, to Spain, Spain (in transitive clause), for Spain, in Spain.

So far the most successful approaches have only used one type of word representation. However, our simple baseline combined with one type of word representation features are not able to compete with previous, more complex, systems. Thus, instead of encoding more elaborate features, we have devised a simple method to combine and stack various types of clustering features induced over different data sources or corpora. In principle, our method can be used with any type of word representations. However, for comparison purposes, we decided to use word representations previously used in successful NER approaches: Brown, Word2vec, and Clark clusters. Our clustering features are placed in a 5 token window. The word representations obtained by different algorithms would capture different distributional properties of words in a given corpus or data source. Therefore, each type of clustering would allow us to capture different types of occurring named entity types. In other words, combining and stacking different types of clustering features induced over a variety of data sources should help to capture more similarities between different words in the training and test sets.

5 Experimental Results

The experiments are performed in 5 languages: Basque, Dutch, English, German and Spanish. The clustering features are obtained by processing the following clusters from publicly available corpora (described in detail in the original paper): (i) 1000 Brown clusters; (ii) Clark and Word2vec clusters in the 100-600 range. To choose the best combination of clustering features we test the available permutations of Clark and Word2vec clusters with and without the Brown clusters on the development data. Table 1 shows the results obtained for English on CoNLL 2003 whereas Table 2 shows the results for Basque. The rest of the in-domain evaluations can be consulted in the original paper.

4 http://opennlp.apache.org/

The stacking and combining effect of our clustering features manifests very clearly when we compare the single clustering feature models (BR, CW600, W2VG200 and W2VW400) with the best models which combine multiple clustering features from various data sources. The results for Basque show that our model clearly improves upon previous work by 4 points in F1 measure (75.40 vs 71.35). These results are particularly interesting as it had been so far assumed that complex linguistic features and language-specific rules were required to perform well for agglutinative languages such as Basque [Alegria et al., 2006]. Finally, it is worth noting that our results increase the overall F1 score by 11.72 over the baseline, of which 10 points are gained in precision and 13 in terms of recall.

5.1 Out-of-domain Evaluations

NER systems are often used in out-of-domain settings, namely, to annotate data that greatly differs from the data from which the NER models were learned. These differences can be of text genre and/or domain, but also because the assumptions of what constitutes a named entity might differ. It is therefore interesting to develop robust NER systems across both domains and datasets. In this section we demonstrate that our approach, consisting of basic, general local features and the combination and stacking of clusters, produces robust NER systems in three out-of-domain evaluation settings: (i) Class disagreements: Named entities are assigned to different classes in training and test; (ii) Different text genre: The text genre of training and test data differs; (iii) Annotation guidelines: The gold annotation of the test data follows different guidelines from the training data. This is usually reflected in different named entity spans. Due to lack of space, in this paper we will only show the third type of out-of-domain evaluation, the other two can be consulted in the journal paper.

In this section the objective is studying not so much the differences in textual genre as the influence of substantially different annotation standards. We only use three classes (location, organization and person) to evaluate the best models presented for in-domain evaluations labeling 'O' every entity which is not LOC, ORG or PER. The text genre of MEANTIME is not that different from CoNLL data. However, differences in the gold standard annotation result in significant disagreements regarding the span of the named entities. For example, the following issues are markedly different with respect to the training data we use for each language: (i) Different criteria to decide when a named entity is annotated: in the expression "40 billion US air tanker contract" the MEANTIME gold standard does not mark 'US' as location, whereas in the training data this is systematically annotated; (ii) mentions including the definite article within the name entity span: 'the United States' versus 'United States', and (iii) common nouns modifying the proper name: 'Spokeswoman Sandy Angers' is annotated as a named entity of type PER whereas in the training data used the span of the named entity would usually be 'Sandy Angers'. CoNLL NER phrase based evaluation punishes any bracketing error as both false positive and negative. Thus, these span-related disagreements make this setting extremely hard for models trained according to other annotation guidelines, as shown by Table 3. Our baseline models degrade around 40 F1 points and the clusterbased models around 35. Other systems' results worsen much more, especially for Spanish and Dutch. The token-based scores are in general better but the proportion in performance between systems across languages is similar.

6 Concluding Remarks

We have shown how to develop robust NER systems across languages and datasets with minimal human intervention, even for languages with inflected named entities. This is based on adequately combining word representation features on top of shallow and general local features. Crucially, we have empirically demonstrate how to effectively combine various types of simple word representation features depending on the source data available. Despite the relative simplicity of our approach, we report state of the art results for Dutch, English, German, Spanish and Basque in seven in-domain evaluations. We also outperform previous work in eight out-ofdomain evaluations, showing that our clustering features improve the robustness of NER systems across datasets. Finally, we have measured how much our system's performance degrades when the amount of supervised data is drastically cut. This, together with the lack of linguistic features, facilitates the easy and fast development of NER systems for new domains or languages.

Previous approaches to NER combining clusters or word embeddings have obtained mixed results [Turian et al., 2010]. Up until now best results have been based on rather complex systems which also used one type clustering or embedding feature [Passos et al., 2014; Ratinov and Roth, 2009; Faruqui et al., 2010; Benikova et al., 2014]. Our system displays two important differences with respect to previous approaches. First, the differences between our baseline system and the, for example, Clark features are much larger than in previous work (with the exception of [Faruqui et al., 2010]), ranging from 2.2 and 5.5 points in F1 measure across the indomain evaluations to 2-8 points for out-of-domain results. If we consider the combined clustering models, the differences over the baseline increase to 5-10 points of F1 measure for indomain evaluations and between 4-22 in out-of-domain settings. Second, our combination of clustering features significantly increases the performance over the models using only one type of clustering feature. The improvements range over 2 to 6 points in F1 measure for in-domain and out-of-domain results. In our opinion, these results are quite interesting as previous experiments combining features of different word representations for NER [Turian et al., 2010], while increasing the overall result, did not improve over the state of the art at the time [Ratinov and Roth, 2009]. The results also show that leaning heavily on the clustering features (instead of specific feature tuning) for performance proves very beneficial in out-of-domain settings.

Acknowledgments

This work has been supported by the European projects NewsReader, EC/FP7/316404 and QTLeap - EC/FP7/610516, and by the Spanish Ministry for Science and Innovation (MICINN) SKATER, Grant No. TIN201238584-C06-01 and TUNER, TIN2015-65308-C5-1-R.