Incorporating Structural Information for Better Coreference Resolution

1 Introduction

Coreference resolution aims to identify mentions in a text that refer to the same real world entity. It has been one of the key areas in NLP for two decades [Soon et al., 2001; Ng and Cardie, 2002; Yang et al., 2004; Lee et al., 2013; Kummerfeld and Klein, 2013; Wiseman et al., 2015; Clark and Manning, 2016b; Zhang et al., 2018].

Recently, Lee et al. [2017] proposed the first state-of-theart end-to-end neural coreference resolution system. They consider all text spans as potential mentions and therefore eliminate the need of carefully hand-engineered mention detection systems. The experiment results show that their system significantly outperforms all previous work in English language. However, there still exist two issues to be addressed. On the one hand, whether the scheme of taking all text spans as potential mentions is better than traditional rulebased mention extraction schemes, especially for other languages? On the other hand, syntactic information has been proven useful in many natural language understanding tasks including coreference resolution. Whether introducing syntactic information can further improve the performance of neural coreference resolution?

For the first issue, after duplicating the neural coreference resolution system proposed by Lee et al. [2017] using pytorch 1.0 1 , we replace the neural span pruning based mention extraction model with the traditional rule-based mention extraction model which extracts mentions directly from syntactic parse trees 2 . Experimentation with the same setting in English shows that the overall performance reduces from 65.89% to 63.50%, i.e., with the performance gap of about 2.39% in F1. While for Chinese language, the performance increases from 58.41% to 60.06%. Further experimentation shows that considering all text spans as potential mentions can ensure the recall, however may bring a lot of noise as well as increase the computational intensity. This indicates that, the neural span pruning scheme achieves more effective mention collections for English language, while for Chinese, the traditional rule-based mention extraction model performs better than the span pruning scheme.

For the second issue, although structural information has been largely ignored in neural coreference resolution, structural embedding approaches have been explored in various NLP tasks and achieved considerable improvements [Liu et al., 2017; Tymoshenko et al., 2017; Zhu et al., 2018]. Intuitively, neural coreference resolution should benefit from effective structural information.

In this paper, we incorporate structural information from three aspects.

• Firstly, we take the nodes of the parse trees as a constraint of mention candidates, and filter out those text spans without any exactly-matched node to reduce the computational complexity.
• Secondly, we propose a novel contextual encoding approach based on the traversal node sequence instead of the word sequence. By traversing the parse tree, we can achieve a node sequence implying the hierarchical information. Furthermore, we use the representation of child nodes to refine the parent node representation. In this

1 https://pytorch.org/

2 https://nlp.stanford.edu/software/dcoref.html

way, structural information can be well incorporated in text span representation.

• Finally, we encode additional structural features (e.g., the path, siblings, degrees, category of the current node) to improve the mention representation.

Experimentation on the data-set of the CoNLL 2012 Shared Task shows the effectiveness of our proposed approach in incorporating structural information into neural coreference resolution.

2 RelatedWork

Previous studies have shown that structural information played an important role in coreference resolution. For example, [Hobbs, 1978] proposed a parse tree-based pronoun resolution algorithm. By traversing the corresponding parse tree in the breadth-first way, the algorithm selected the appropriate noun phrase as the antecedent of a given pronoun from the resulting node sequence according to the dominance and government binding relationship in the grammatical structure. [Lappin and Leass, 1994] proposed the RAP algorithm for both third person pronoun and reflexive pronoun resolution. The algorithm first calculated the prominence of the candidate antecedent by manually weighting various linguistic features, and then determined the antecedent using various filtering rules. [Kong and Zhou, 2012] proposed a treekernel based approach to pronoun resolution for both English and Chinese languages. After dynamically expanding and pruning the parse tree based on the centering theory, competitor information and predicate-argument information, the algorithm used a convolution tree kernel function to directly calculate the similarity between the resulting structures and determine the coreferential relationship.

Due to the outstanding performance of neural networks in various NLP tasks during the last few years, researchers began to apply various neural network models to coreference resolution. For example, [Wiseman et al., 2015] used neural networks to pre-train two separate subtasks, i.e., anaphoricity determination and antecedent ranking, in learning various feature representations for coreference resolution. [Wiseman et al., 2016] used recurrent neural networks to learn global representations of entity clusters. [Clark and Manning, 2016a] employed reinforcement learning to directly optimize a neural mention-ranking model for coreference evaluation metrics, and achieved the best known performance on Chinese portion of the CoNLL 2012 Shared Task. [Lee et al., 2017] proposed the first end-to-end neural coreference resolution system. They used a bi-directional LSTM and a Headfinding attention mechanism to represent the mentions, and exploited a mention-ranking model to determine coreference clusters. [Zhang et al., 2018] proposed to use a biaffine attention model to get antecedent scores and jointly optimized the two sub-tasks to improve the performance of neural coreference resolution. [Lee et al., 2018] proposed an approximation of higher-order inference using the span-ranking architecture from Lee et al. [2017] in an iterative manner.

From the above we can find that, although structural information has been proven very useful to coreference resolution, it has been largely ignored in neural coreference resolution systems. In this paper, we focus on effectively incorporating structural information into neural coreference resolution.

3 Baseline: Neural Coreference Resolution

Following the work of [Lee et al., 2017], our neural coreference resolution model adopts the mention-ranking approach. Such a model scores pairs of mentions for their coreferential likelihood rather than comparing partial coreference clusters. Hence it operates in a simple setting where coreference decisions are made independently. Figure 1 shows the framework of our baseline system, which consists of two components, i.e., mention detection and antecedent identification.

3.1 Mention Detection

In the mention detection stage, we first combine characterlevel, word-level and context-level information with an attention mechanism to represent each possible mention candidate (text span). Then, we use some scoring mechanism to rank the prominence of the text spans. Finally, we select a certain proportion of text spans as the set of mentions to be resolved. The detailed procedure of mention detection is shown as the part enclosed by the dotted line in Figure 1.

For a given text D = {w1,w2, . . . ,wND }, the mention detector extracts all possible mentions S = {s1, s2, . . . , sn} where ND means the number of words in the document, si = {wbi ,wbi+1, . . . ,wei } means a mention, bi and ei are the start and end index of mention si, 1 ≤ bi ≤ ei ≤ ND.

• In the embedding layer, for ∀wi ∈ D, by combining character and word embeddings, we achieve xi = [wi, ci] ∈ Rdx , where dx = dw + dc, wi ∈ Rdw means the word embedding of the word wi, and ci ∈ Rdc means the character embedding vector, which can be achieved using a Character CNN or a Character LSTM. In this way, we can get the embedding representation of the document D

• In the contextual embedding layer, we use bidirectional LSTM as the contextual encoder to model the dependen-

cies between words. For the input xi ∈ Rdx , we can get the contextual representation hi ∈ Rdh .

• In the span representation layer, we represent a given text span si = {wbi , wbi+1, . . . , wei } as

where, si ∈ Rds , fi is the additional feature vector to encode the span width, and ˆxi is the weighted sum of the representations of all the words contained in si, which are learned using the Head-finding attention mechanism.

• In the span pruning layer, we first use a multi-layer feedforward neural network to score the spans according to their representation. Then, we sort the spans by their scores, and select a certain part of spans with higher scores as the mention candidates.

3.2 Antecedent Identification

In the antecedent identification stage, the basic idea is to find the best antecedent in a search space. Similar to traditional Mention-Pair models, we do mention pairing from the back to the front in a certain search space according to the mention appearing order. For each mention-pair instance, we calculate its score using a feedforward neural network.

where fi,j encodes speaker,genre and distance features. In the coreference scoring layer, we combine the scores of two mentions si and sj , and the score of the mention-pair instance (si, sj) to achieve the final coreferential score.

where, j = ε means the case that the ith mention has no antecedent (i.e., non-anaphoric). At this point, the coreferential score is 0. For each mention, we choose the mention with the highest score in the search space as its antecedent.

4 Incorporating Structural Information

In this paper, we incorporate structural information from three aspects, i.e., taking nodes in the parse trees to filter out impossible text spans, encoding contextual information with the traversal node sequence, and encoding additional structural features (e.g., the path, siblings, degrees, category of the current node) to enhance the text span representation.

NGO///�/*/�/}/�/�e"/� NGO is a good entry point.

Figure 2: An example of a parse tree.

4.1 Taking Nodes as a Filter Constraint

Same as [Lee et al., 2017], our baseline system takes all text spans within a certain length limit as potential mentions. That is, the only limit is the length of the mention (i.e., the number of words making up a mention). Taking the sentence shown in figure 2 as an example, when the length limit is no more than 10, our baseline system can extract 45 mention candidates (e.g., NGO, NGO/, NGO/�, NGO/�*, NGO/ �*�, NGO/�*�}, ...). Although all possible mentions can be covered, a large amount of noise is introduced due to the neglect of lexical and syntactic information. This greatly increases the computational burden of the baseline system. To address this issue, we propose to take node as a filter constraint. The motivation behind is that each valid mention naturally corresponds to a node in the parse tree. This is also different from the traditional rule-based mention extraction schemes, which extract possible mentions directly from the parse trees ignoring the length limit. After listing all text spans within the length limit, we look for the corresponding node for each text span and only keep the text spans having the node in the parse tree. For the example shown in Figure 2, for the extracted 45 mention candidates, we can get 15 mentions (e.g., 9 words corresponding to leaf nodes, "NGO/� *�}��e"�","/�*�}��e"", "�*", "� *�}��e"", "�}", "�}�") after filtering with the node constraint.

4.2 Using the Traversal Node Sequence to Encode Context

Just as noted in 3.1, our baseline system uses a bidirectional LSTM as the contextual encoder to model the dependencies between words. That is to say, the context information is represented as the dependencies between linear words. In fact, structural information between constituent words in a mention and between multiple mentions in a sentence is very important to coreference resolution. In this paper, we propose a node-based contextual encoder to better concentrate on the presentation of such information.

Our node-based contextual encoder is performed in three steps, i.e., contextualizing leaf nodes, updating non-terminal node representations, and re-contextualizing all nodes.

Contextualizing Leaf Nodes

By traversing the parse tree, we can achieve the node sequence 3 . In particular, we traverse the given parse tree t in post-order to get the node sequence O(t) = [o(t) 1 , o(t) 2 , . . . , o(t) nt ], where nt means the number of the nodes in parse tree t without the leaf nodes. In order to get the representation of the vector O(t) , we first initialize it with zero vectors.

Then, we replace the representations of the leaf nodes with the corresponding word embedding representation.

We take the result representation vector o(t) as the input of the contextual embedding layer instead of the original word embedding matrix. In this way, the contextual embedding layer can learn both linear information and hierarchical structure representation implied in node sequence. So far, we can achieve the context representations H(t) .

Updating Non-terminal Node Representations

For non-terminal nodes, we exploit the representations of theirs child nodes to refine their representations. The basic strategy is to form a new representation for each non-terminal node by using an attention mechanism to transform the vector representation of their children into a fixed dimension vector, and a gating mechanism to combine the representation of child nodes with the original representation. The shaded part in Figure 3 shows the details.

For the node sequence o(t) , we have got the contextual representations H(t) . For any non-terminal node o(t) i , the sequence of its child nodes is marked as O(t) i .children, and we can get the fixed dimension vector of the child nodes using Eq 12. For leaf nodes, we let a(t) i = h(t) i .

Using a gated mechanism, we can update the representation of the node o(t) i .

3 Word nodes are not considered

Obviously, when o(t) i .children = ∅,

In the meanwhile, we use a attention mechanism to implement the function Γ. Suppose that node o(t) i has child node o(t) j , i.e., o(t) j ∈ o(t) i .children, and the contextual representations of these two nodes are h(t) i and h(t) j , respectively. We aim to get a(t) i using the attention mechanism. In this paper, we use both dot attention and bilinear attention [Luong et al., 2015] to compute the correlation between the representation of child nodes and the given node, with score functions defined as follows:

After using a softmax function to normalize score(h(t) j , h(t) i ), we can get the final child nodes' representations of node oi.

Re-contextualizing All Nodes

After updating the representations of non-termination nodes, we feed the achieved representation vector to the contextual embedding layer again, and take the output H(t) as the final contextual encoding results.

4.3 Encoding Additional Structural Features

Some additional wide-used structural features as shown in Table 1 are extracted to enhance the representation of span texts. The third column lists the values associated with the node "NP-19" in the example shown in Figure 2.

For the feature pathi = [labelroot, . . . , labeli], we first use a method similar to Character LSTM to represent it as a vector, pathi = [flabel,root, . . . , flabel,i] ∈ Rdepthi×df . and then take it as the input of a Bi-LSTM. In this way, we can achieve hlabel,i = Bi - LSTM(flabel,i) = [⃗hlabel,i , ⃗ hlabel,i]. and take Eq (19) as the final representation of path feature.

5 Experimentation

In this section, we systematically evaluate our proposed approach to neural coreference resolution.

5.1 Experimental Settings

All experiments are conducted in the data from the CoNLL2012 shared task [Pradhan et al., 2012]. The English part of this corpus contains 2802 training documents, 343 development documents, and 348 test documents. Accordingly, the Chinese part contains 1810 training documents, 252 development documents, and 218 test documents.

Our English models reuse the hyperparameters from Lee et al. [2017] with the difference between Chinese and English languages as shown in Table 2. We report the precision, recall, and F1 for the standard MUC, B3 , and CEAFφ4 metrics using the official CoNLL-2012 evaluation scripts, with the average F1 of the three metrics as main evaluation standard.

5.2 Experimental Results

Table 3 compares our model with several start-of-the-art systems for English and Chinese languages, such as [Clark and Manning, 2016b], which presented a neural cluster-ranking model for coreference resolution benefiting from entity-level information, [Clark and Manning, 2016a], which employed reinforcement learning to directly optimize a neural mentionranking model for coreference evaluation metrics, [Lee et al., 2017], which introduced the first end-to-end coreference resolution system with no external resources, and [Lee et al., 2018], which refined the span-ranking architecture as proposed in Lee[2017] by modeling higher order interactions between spans in predicted clusters and achieved a significant improvement. 4

For fair comparison, our baseline duplicate the neural coreferece resolution system proposed by [Lee et al., 2017] using pytorch 1.0. Comparing the results of our baseline and Lee's[2017], there exists a performance gap of 1.3%in average F1. This is caused by much less fine-tuning of various layers in mention detection due to GPU-resource limitation and time efficiency considerations. On one hand, [Lee et al., 2017] use the Dropout strategy proposed by [Gal and Ghahramani, 2016]. Instead, we use the official bidirectional LSTM provided in pytorch. On the other hand, we re-implement some time consuming processes, such as text span pruning, in batch form. This significantly reduce the training time of our baseline to only 1/6 as of Lee's[2017] on the same hardware environment.We can find that,

• In comparison with our baseline, our improved model with structural information significantly outperforms the baseline in all metrics for both Chinese and English languages. This indicates the effectiveness of our incorporated structural information.
• For Chinese, our improved model achieves competitive results with the two start-of-the-art systems. Especially, our model outperforms them in precision in all metrics. This is due to our incorporation of richer structural information representations.
• For English, our model outperforms [Clark and Manning, 2016b] and [Clark and Manning, 2016a] in all metrics. Comparing the results of our model with [Lee et al., 2017]'s, we can find that the most significant gains come from the improvement in precision. This indicates the effectiveness of our introduced structural information. There is still a big performance gap between our proposed model with [Lee et al., 2018]. This is largely due to the employment of ELMo[Peters et al., 2018]. Moreover, in this paper, we focus on incorporating structural information. In the future, we will explore more strategies to improve our system.

To show the contribution of each structural information to our proposed model, we take the system without any structural information as the baseline, and incorporate structural information one by one. Table 4 shows the average F1 on the development data set using automatic parse trees 5 for both English and Chinese languages.

4 It is worth noting that, [Luo and Glass, 2018] presented a word embedding model to learn cross-sentence dependency. Integrating this model into the coreference resolution system proposed by Lee[2017] improved the average F1 by 0.6% from 67.2 to 67.8. [Peters et al., 2018] introduced a new type of deep contextualized word representation ELMo(Embeddings from Language Models). Integrating ELMo into the coreference resolution system proposed by Lee[2017] improved the average F1 by 3.2% from 67.2 to 70.4. However, these two studies mainly focus on the more effective word representation approaches rather than coreference resolution itself.

Table 4: Performance of our model on the development set from the CoNLL-2012 shared task using automatic parse trees (S1 - node constraint, S2 - node-based contextual encoder, S3 - additional structural features).

From the results we can find that,

• The results on independent integration of the three schemes show that the second scheme (i.e.,the nodebased contextual encoder) performs the best in both Chinese and English languages. It contributes about 2.20 and 0.92 F1 to the final results for Chinese and English, respectively. This indicates that the proposed nodebased contextual embedding approach can well enhance the representation of the mentions and contribute much to coreference resolution.

5 The automatic parse trees are provided by the CoNLL-2012 shared task.

• The combination of the three schemes achieves the best performance in both English and Chinese languages. This indicates the effectiveness of the incorporating structural information.

6 Conclusion

In this paper, we focus on effectively incorporating various kinds of structural information into neural coreference resolution. In particular, three schemes are exploited. First, we take nodes in the parse trees as a constraint to filter out impossible text spans in reducing the computational complexity. Second, we present a node-based contextual encoding scheme to capture hierarchical information in enriching the text span representation. Last, we encode additional structural features in enhancing the mention representation. Experimentation on the data-set of the CoNLL 2012 Shared Task shows the effectiveness of the structural information. Although our model much improves the performance of coreference resolution, there still exists much improvement room, such as the employment of ELMo. In the future work, we will pay more attention to antecedent identification.

Acknowledgements

This work is supported by Artificial Intelligence Emergency Project 61751206 under the National Natural Science Foundation of China, Project 61876118 under the National Natural Science Foundation of China and the Priority Academic Program Development of Jiangsu Higher Education Institutions.