Modeling Noisy Hierarchical Types in Fine-Grained Entity Typing: A Content-Based Weighting Approach

1 Introduction

Fine-grained entity typing (FET) [Ling and Weld, 2012; Yogatama et al., 2015; Ren et al., 2016a; McCallum et al., 2018] refers to the task of annotating the text expressions (also known as mentions) of entities in a document or sentence, using a set of finely specified type labels. Knowledge acquired through FET is often utilized to facilitate many downstream natural language processing (NLP) tasks, such as knowledge base construction and expansion [Dong et al., 2014], relation extraction [Koch et al., 2014; Liu et al., 2014], entity linking [Ling et al., 2015] and factual question answer [Dong et al., 2015], amongst many others.

Despite great processes have been made in the past years, the FET problem is however confronted by a major challenge issue: noisy labels. Such noisy labels naturally come with the data acquisition processes such as the popular distant supervision method [Mintz et al., 2009]. Additionally, a mention, namely, a word or phrase, may assume different types.

∗Corresponding author: zhangrc@act.buaa.edu.cn

For example, depending on its context, the word "Mexico" may refer to the location of Mexico in a general sense, or it may also refer to the country "United Mexican States" in the political sense (as shown in a small example of a type hierarchy depicted in Figure 1). Such noise can significantly degrade the predictive performance of the FET models [Ren et al., 2016a]. Unfortunately, existing FET approaches either embrace a labor intensive pre-processing step to clean the noise [Gillick et al., 2014] or simply bias the learning to only model one of the noisy labels [Ren et al., 2016a; Abhishek et al., 2017; Xu and Barbosa, 2018], sidestepping the fact that those noises are related and content dependent.

To directly model the structured noisy labels for FET, in this paper, we propose a novel content-sensitive weighting schema that is powered by a newly devised cost function and a hierarchical type embedding strategy. In detail, the hierarchically structured typing labels in the given data are first normalized to having only the leaf nodes of the tree as candidate types. Next, context dependent weights for all typing labels are then randomly initialized. Finally, through maximizing the expected reward of a random walk process, the weights of all candidate types are re-weighted such that only one of them will have large value, which corresponds to the correct type. In this way, the proposed weighting schema is able to effectively de-noise the noisy hierarchical typing labels during training, thus improving the predictive performance of the FET models.

Experiments on several widely used datasets show that our method significantly outperforms its competitors, resulting in new state-of-the-art performance on these benchmark data sets. We also provide an extensive ablation study to validate the effectiveness of different components of our proposed framework, highlighting the effectiveness of our newly devised cost function. To the best of our knowledge, this is the first work to jointly model the hierarchical structure of and the noise contained in the label type. We have posted the code on the GitHub ( https://github.com/wujsAct/ fine-grained-entity-typing-Hier-Denoise ).

2 The Content-BasedWeightingModel (NDP)

The overall architecture of our content-based weighting model is depicted in Figure 2. We denote our method as Noise Detective Prediction (NDP). In a nutshell, our NDP method first normalizes the hierarchically structured typing labels, so that only the leaf nodes represent valid type. Next, a loss constraint is used to regularize the training to weight out noisy labels.

2.1 Problem Statement

Let T denote the given type hierarchy of data set for the finegrained entity typing (FET) task. Specifically, we interpret T as a tree, in which each edge is labeled by a type token, such as LOCATION, CITY, REGION, RIVER etc in Figure 1. Each node u in the tree T is a type, which can be uniquely specified by the path from the root node to the node u. For example, LOCATION, LOCATION.RIVER in Figure 1 are all types. We consider that T is given.

In general, an entity in the reality, such as in a KB (Knowledge base), may be categorized into multiple types in T . For example, an entity that refers to a person that can be categorized both as type PERSON.BUSINESSMAN and as type PERSON.POLITICIAN. Thus the same entity in different context may assume different types.

A sentence s is regarded as a sequence (w1,w2, . . . ,wL) of words. In a sentence of interest, a subsequence of words m := (wJ,wJ+1,wJ+d-1) has been recognized as a mention, namely that it corresponds to some entity in a KB. Let e(m) denote the entity that the mention m refers to in sentence s. The objective of a (sentence-level) fine-grained entity typing task is to find a type in T , which is the most appropriate for the entity e(m) in the context of the sentence s. We will refer to the most appropriate type for mention m in sentence s as the correct type for simplicity.

The dominant approach to solving this problem is via learning a predictive model which predicts the correct type for a given sentence s and a mention m therein. This relies on some training data Dtrain, which typically takes the following form: Dtrain := {(si,mi, τi) : i = 1, 2, . . . ,N}, where in each triple (si,mi, τi), si is sentence, mi is a mention in si, and τi is a subset of T , namely a subset of types associated with the entity e(mi) corresponding to mi. Ideally, we wish that τi contains only the correct type for mi in si. In reality, however, it is difficult or costly to acquire the correct types for a large number of training examples and usually τi is obtained in a automatic manner such as via a weak multilabel classifier, e.g., distant supervision [Mintz et al., 2009]. Inevitably, some τi's contain additional incorrect types, and we will call such sentence-mention pairs noisy. It has been observed that in standard datasets, rather significant fractions of mentions have noise type labels [Ren et al., 2016a], and the statistics of such noise for the standard datasets can be found in Table 1.

2.2 Normalized Type Hierarchy

As discussed earlier, a type can in general correspond to any node in a type hierarchy T . This creates difficulty for a model to directly predict the mention of a type at the right tree depth. To resolve this problem cleanly, we propose converting the type hierarchy to a different representation. Given the type hierarchy T , we transform it to a normalized type hierarchy T ∗ as follows: For each intermediate (namely, non-leaf and non-root) node u in T , add a child node v to u, and label the edge connecting u to v by a new type token, which we denote by STOP. This gives rise to the normalized type hierarchy T ∗ . Figure 3 shows the transformation of a type hierarchy to its normalized version.

There is a natural map ϕ that maps all nodes in the original tree T to the leaf nodes in tree T ∗ . Specifically, if u is the root node or any leaf node in T , ϕ(u) is the same node in T ∗ ; and if u is an intermediate node in T , ϕ(u) is the the newly added leaf node of u in T ∗ .

Under this mapping, every type in T can be identified with a leaf node in T ∗ , or equivalently by a path in T ∗ connecting the root node to that leaf node. In a normalized type hierarchy, only the leaf nodes represent valid types. Using such a normalized representation, the learned model will only consider the leaf nodes of the tree as candidate types. If the model predicts the type as a leaf node v that is pointed to by a STOP edge, for example, the node LOCATION.STOP, then the parent node of v, namely, LOCATION, is declared as the predicted type. Thus using such a normalized tree allows the model to predict types at any level of the original type hierarchy, without recourse to additional heuristics. From here on, we will work with the T ∗ instead of T , and every reference to a type will refer to its image in T ∗ under ϕ.

2.3 Weight Out Noise with Loss Function

For notation convenience, we will denote the set of all sentence-mention pairs in Dtrain by Din train , that is, {(si,mi, τi) : i = 1, 2, · · · ,N} := Din train .

For each training example (si,mi, τi), we will also generate a negative training example, (si,mi, τ - i ), where τ - i is a random set of types in T ∗ but not in τi. We will denote by Dtrain the set of all negative training examples {(si, mi, τ - i ) : (si,mi) ∈ Din train }. Our NDP model will be trained on both Din train and Dtrain .

For any given sentence-mention pair (s,m), let x(s,m) ∈ RkF denote the embedding of (s,m). For each type t ∈ T , let t ∈ RkT denote the embedding of t. We will postpone the construction of embeddings x(s,m) and t in later sections.

Our loss function relies on the following hypotheses.

Hypothesis 1 For every sentence s of interest and the identified mention m in s, there is exactly one type for m in the context s.

Hypothesis 2 For each training example (si,mi, τi) ∈ Dtrain, τi contains the correct type of mi in si.

Hypothesis 1 above may appear overly stringent at the first glance. We wish to note however that this is in fact consistent with our problem setting. Recall that the problem is to predict the most appropriate type for the mention of interest. Even if from the context two or more types may be appropriate, it is reasonable to consider that there is only one that is the most appropriate. On the other hand, it is remarkable that an important purpose of setting up hypotheses in machine learning is to constrain the space of learning so as to reduce the model capacity, enhance learn-ability and avoid over-fitting. Such practice usually becomes particularly important when the training data is limited. Theoretically, the hypotheses one sets up need not to be perfectly "correct", it is only required that the hypotheses do not conflict with the ground truth, exhibited partially from the observed data. Such a conflict, if existing, would result in under-fitting and degrade the model performance. But if such a conflict does not exist, even oversimplified hypotheses provide benefits to learning. This has been the reason underlying numerous models in which the very unrealistic Gaussian hypotheses are made.

Given that the training set Din train is usually noisy, it is critical that the FET model is capable of coping with such noise.

For an arbitrary x ∈ RkF and a t ∈ RkT , let there be a score function φ(x, t). Although one may explore a wide variety of score functions, in this paper, we specialize the score function by

where A is the trainable matrix of size kF × kT.

For each input (si,mi) in Din train , let its positive-example score be defined as

where α(t; si,mi)'s are a set of non-negative weights summing to 1, namely, satisfying

The construction of these weights is given in a later section. The idea here is that after these weights are learned, only one of them will have large value (near 1), which corresponds to the correct type assumed in Hypotheses 1 and 2.

For each input (si,mi) in Din train , let its negative-example score with respect to a negative type t- ∈ τ - i be defined as

We then define a margin-based loss E(si, mi) by

where λ is a positive hyper-parameter. Note that under this equation, when E(si,mi) takes the minimum value 0, for any wrong type t- ∈ τ - i , the score S-(si,mi, t-) is lower than the positive-example score S(si,mi) by at least λ. Thus when we minimize E(si,mi) for input (si,mi), we implicitly maximize S(si, mi) and force score S-(si, mi, t-) to be sufficiently low.

The overall loss function of the model is then given by

Generation of Weights

Suppose that a training input (si,mi) is given. We now explain how the weights {α(t; si,mi) : t ∈ τi} are generated. Briefly these weights are essentially the probabilities that a random walk on T ∗ lands on each leaf node of T ∗ .

Specifically, let Q be the random walk on the normalized type hierarchy T ∗ , which we now define. The walk starts from the root of T ∗ and follows the tree structure to eventually arrives at some leaf node. For each node u ∈ T ∗ , let C(u) be the set of all children of u in T ∗ . For each node u in T ∗ , let q(·|u) be a distribution on C(u), where q(v|u) for a child v of u is the probability that the walk goes to v given its current location is u. Given the set of distributions {q(·|u) : u ∈ T ∗ }, one can express the probability p(z) that the walk eventually arrives at any given leaf node z. More precisely, suppose that the path from the root node to z is the node sequence (u0, u1, . . . , um), where u0 is the root node and um is z. Then

For every training input (si,mi), we construct such a process Q(si,mi), namely, a set of conditional distributions {q(·|u) : u ∈ T ∗ }). Then we define for every type t ∈ τi,

under the randomwalk Q(si,mi). The set of all such random walks {Q(si,mi) : (si,mi) ∈ Din train} will be learned during training.

We also propose two testing weight methods: average weight (AW) and global data-dependent weight (DDW). For average weight, we treat all the types equally. For global DDW, we directly utilize the normalized type score as the type weight. The the type weight are computed similar to Equation 1 as follows:

where B is the trainable matrix of size kF × kT.

Interpretation of Loss Function

We now pause and revisit the loss function (5) to give some insight on the design of this loss function. Given a training input (si,mi), one can imagine that a random walker walks down the type hierarchy T ∗ , according to an unknown process Q(si,mi). According to process, he may arrive at any leaf node in τi with some probability. At leaf node t ∈ τi, he would collect a "reward" whose value equals the score φ(x(si,mi), t). Then S(si,mi) defined in (2) may then be regarded as the expected reward the walker will get under Q(si,mi). Thus, maximizing S(si,mi) over its set of weights in (2) is equivalent to maximizing the walker's expected reward over the random walk process Q(si, mi).

It is easy to see that under such a randomwalk process, the probabilities that the walker arrives at two sibling leaf nodes in the tree T ∗ are related by the probability the walker goes through their common parent. Thus, the randomwalk process naturally looks after the correlation between the siblings.

We also like to stress that such a random walk is not global to all training examples. Rather, for each training example, there is a separate such process, and its parameter q(·|·) is learned during training. Since incorrect type labels tend to give high loss value, for each training example, the optimization process will result in a function q which puts a low probability on the wrong types and a high probability to the correct type. That is, minimizing the loss function squeezes out noise and enhances signal.

2.4 Type and Feature Embedding

For type embedding, we construct a embedding matrix DT for the set of all type tokens in T ∗ , from each token is assigned a vector in RkT as its embedding vector. We note that all "STOP" tokens in the T ∗ are consider different and hence they all have different embedding vectors. For every type t ∈ T ∗ , if the path from the root to node t is the sequence (c1, c2, . . . , cl), the embedding t of type t is defined as

where cj is the embedding vector for token cj.

Feature embedding in our model largely follows the previous approaches in [Abhishek et al., 2017] and [Xin et al., 2018]. Specifically, for any sentence-mention pair (s,m), its feature embedding x(s,m) is the concatenation of vectors s ∈ Rks and vector m ∈ Rkm , which we now define.

First note that we adopt a pre-trained word embedding dictionary DW from Glove [Pennington et al., 2014]. Using this embedding dictionary DW, the vector m is simply the average of the embedding vectors of all surface words in m.

To obtain vector s, first note that the mention m partitions s into two word subsequences, the sequences (wL 1 , wL 2 , . . . , wL a ) left to the mention and the sequences (wL 1 , wL 2 , . . . , wL a ) right to the mention. The left and right sequences are each passed to Bidirectional LSTM network, and correspondingly generate output sequences (hL1 , hL2 , . . . , hLa ) and (hL1 , hL2 , . . . , hLa ) respectively. The outputs of the two networks are respectively attentively combined and then summed up to give rise the embedding s. More precisely,

Here βL i 's and βR i 's are attention weights, computed according to

where U and W are trainable parameter matrices, serving to map a vector into RkA , and θ is a vector of dimension kA.

At this point, we have completely defined the proposed NDP model. When using the trained model for predicting the type of a mention m in a sentence s, we simply compute the score φ(x(s,m), t) for every t ∈ T ∗ under the model. The type t with the highest score is then declared as the correct type. Note that if the declared type is one pointed to by the STOP edge, we regard the declared type as its parent type.

3 Experimental Studies

3.1 Settings

We evaluate the NDP approach using three popular datasets, Wiki 1 [Ling andWeld, 2012], OntoNotes [Weischedel et al., 2011] and BBN [Weischedel and Brunstein, 2015]. We utilize the pre-processed data provided by [Ren et al., 2016a], where 10% of each testing data is used as a validation set, following the standard protocol widely adopted in the previous works [Abhishek et al., 2017; Shimaoka et al., 2017; Ren et al., 2016b; Ren et al., 2016a]. Table 1 shows the statistics of the three datasets.

We note that the fraction of training sentence-mention pairs ( "mentions" in Table 1) that are noisy (namely, containing more than one type) is 35.42% in Wiki, 27.4% in OntoNotes, and 27.4% in BBN. This suggests that the training set contains significant amount of noise and that learning approaches robust to such noise are highly desirable. On the other hand, noise in the testing data is much less severe. In fact, the testing set of BBN contains no noisy mention, and the testing data of OntoNotes is also quite clean (with 6.23% noisy mentions). The fraction of noisy mentions in the testing data of Wiki is 11.72%, higher than the other two datasets, although mild relative to its training data.

1 In the Wiki dataset, there exist types that have no parent. The original structure of types in Wiki thus does not qualify as a tree. Ren et al. [Ren et al., 2016a] fixed this problem by adding 15 types. As we do not consider those added types during testing, adding those types have no effect on the test performance

Similar to previous work [Abhishek et al., 2017], in our implementation of the proposed NDP model, we utilize the word embedding provided by Glove [Pennington et al., 2014]. And, We do not fine tune them during training. The Adam optimizer [Kingma and Ba, 2014] is utilized to train the model. An early-stop strategy is employed to terminate training when the loss on validation data does not decrease. We incorporate dropout and L2 regularization to avoid over-fitting. The left and right sequence lengths are truncated to 10. We utilize random search to explore the proper hyper-parameters. The chosen hyper-parameters are listed in Table 2.

We compare NDP with several state-of-the-art neural network models: FIGER [Ling and Weld, 2012], HYENA [Yosef et al., 2012], WSABIE [Yogatama et al., 2015], ProtoLE and Proto-HLE [Ma et al., 2016], AFET [Ren et al., 2016a], ATTN [Shimaoka et al., 2017], ABH [Abhishek et al., 2017], and NFETC [Xu and Barbosa, 2018]. For comparison, we use the reported performances of these models in their respective original papers, except for FIGER and ATTN, the performances of which are taken from [Ma et al., 2016] and [Abhishek et al., 2017]. The evaluation metrics used are Accuracy (Acc), Micro-averaged F1 (Mi-F1) score and Macro-F1 (Ma-F1) score. These metrics are the most commonly used for evaluating the FET models [Ling and Weld, 2012; Ren et al., 2016a].

3.2 Predictive Performance

The comparison results are presented in Table 3, where best performance is highlighted in bold. The performances of the compared models are all taken from the literature. We also run the published code of ATTN, ABH, ABH-AllC and NFETC and the re-produced results are labelled by ∗ symbol. Note that original NFETC paper using a prepossessed dataset. So their original results are not included for comparison. We re-implement the NFETC model by using the original type hierarchy fromWiki dataset.

It can be seen that NDP outperforms all compared models resulting in a state-of-the-art result under all metrics. This confirms the advantage of the proposed NDP model. In particular, on BBN, NDP outperforms other testing methods with a large margin. We believe that this is the most accurate and truthful demonstration of the advantage of NDP. As mentioned earlier, the testing set of BBN has signficantly less noise than the other datasets. As such, evaluation on this dataset is the closest to evaluation against the ground truth. Thus relative to other used datasets, the evaluation results obtained on BBN are the most reliable.

It is worth noting that NDP exhibits superior performance consistently on all three datasets, whereas other models tend to achieve good performance, if at all, only on one or two datasets. In this sense, NDP is more robust against the varying intrinsic structures of the datasets.

3.3 Ablation Studies

Performances at Different Levels of the Type Hierarcy

We further investigate the performances of the compared models at each level of the type hierarchy. We use the BBN dataset for this purpose and examine the level-1 and level-2 performances of these models. Specifically, for level-2 performances, the testing examples without having type at level 2 are disregarded from evaluation. For level-1 performances, the testing examples which only have type at level 1 are considered. The results of this evaluation are given in Table 5. At both levels NDP outperforms significantly the other models in F1.

Benefit of the Hierarchical Type Normalization

We also conduct experiments to evaluate the impact of the hierarchical normalization type embedding on the NDP framework. We simple replace our hierarchical type embedding with the Complex Bilinear hierarchical structure model (denoted ComplEx) recently introduced in [McCallum et al., 2018]. Results are presented in Table 4. Table 4 indicates that the normalization component we adopted significantly benefit the predictive performance of our NDP model (the last three rows of Table 4), compared to that of the ComplEx strategy (the first three rows of Table 4).

Impact of the Random Walk Weights

We further exams the effects of our random walk strategy on our framework. We compare with average weighting (AW) and a learnable data-dependent weighting schema (DDW). The comparison results are depicted in Table 4. From Table 4, we can see that the random walk method brings significant gain, in terms of predictive performance, over the other two testing weighting schemes.

4 RelatedWork

Type Hierarchy Modeling The most earlier models ignore the type hierarchy and simply treat it as a flat structure (see, e.g., [Ling and Weld, 2012; Yogatama et al., 2015; Shimaoka et al., 2017; Abhishek et al., 2017]). Some works, e.g., [Yosef et al., 2012; Ren et al., 2016a; Xu and Barbosa, 2018; McCallum et al., 2018] have made efforts explicitly modeling the correlation among hierarchical types. These models, although demonstrating some successes, rely on some heuristic procedure to walk down the type hierarchy, stop at a node, and then declare the node as the predicted type. The recent work [McCallum et al., 2018] proposed the structure loss to solve this problem. Unlike these works, we propose a generic framework that directly model the hierarchical typing without additional structure loss function to train or heuristic method to infer through the type tree.

Label Noise Handling Various approaches have been proposed to deal with noisy entity mentions in the training set. Gillick et al. [Gillick et al., 2014] refined the training data by applying a set of heuristics to prune types. [Yogatama et al., 2015] proposed a man-made function to transform the positive type rank to a weight. [Ren et al., 2016a; Abhishek et al., 2017; Xu and Barbosa, 2018] model noisy and clean entity mentions separately. Nevertheless, these methods enforce the model to choose the most relevant type as the golden positive type label for noisy entity mention during training. Instead, we explicitly model all the noisy labels and their structures during training.

5 Conclusion and FutureWork

We introduce a novel approach for Fine-grained Entity Typing modeling. Our method leverages a novel cost function to jointly model the correlation among hierarchical types and label noises. Our experimental results demonstrate the power of this approach convincingly. We prefer call this approach a "framework" rather than a single model, since under this approach, one can freely explore other constructions of feature embedding module and other choices of score function φ. More research is underway to explore this rich space of strategies.

Acknowledgments

This work is supported partly by China 973 program (No. 2015CB358700), by the National Natural Science Foundation of China (No. 61772059, 61421003), and by the Beijing Advanced Innovation Center for Big Data and Brain Computing (BDBC) and State Key Laboratory of Software Development Environment (No. SKLSDE-2018ZX-17).