Adversarial Transfer for Named Entity Boundary Detection with Pointer Networks

1 Introduction

Named entity recognition (NER) is a fundamental task in natural language processing which aims to jointly resolve the boundaries and type of a named entity in text [Li et al., 2018a]. In this paper, we ignore the entity typing and focus on the subtask of named entity boundary detection, which involves detecting the start and end boundaries of an entity mention in text.

There are several reasons for studying the subtask. First, most fine-grained entity typing systems, such as FIGER [Abhishek et al., 2017], FINET [Corro et al., 2015] and SANE [Lal et al., 2017], either manually label entity mentions or assume that entity mentions have already been predetected [Corro et al., 2015]. In this case, the typing task becomes a multi-label classification task. Errors made in entity boundary detection inevitably mislead and adversely affect subsequent entity-typing systems. This has created an overwhelming demand for more accurate and robust boundary detection approaches. Second, the availability of knowledge bases, e.g., Wikipedia, FreeBase, ProBase, enables the study of entity linking, which aims to determine the identity of entities mentioned in text. Because of the overwhelming entity types defined in knowledge bases, the types predicted by NER systems become less necessary. Further, there could be conflicts between the types predicted by NER systems and the types of the entities disambiguated through the entity linking process [Phan et al., 2018].

∗Corresponding author.

Some studies [Ren et al., 2016; Corro et al., 2015] utilize off-the-shelf NER systems (e.g., StanfordNER and Spotlight3) to detect entity boundaries for downstream applications. However, off-the-shelf systems suffer from the shift in data distribution from the actual data encountered at test time, resulting in poor performance. Recently, a few studies [Yang et al., 2017; von D¨aniken and Cieliebak, 2017; Zhao et al., 2018; Lin and Lu, 2018] have investigated transfer learning in NER, aiming to transfer knowledge from source domain to target domain. However, these existing methods require an amount of annotated target-domain data to fine-tune models. In this paper, we assume no label annotations in the target domain.

Essentially, entity boundary detection is a sequence labeling problem, where the task is to predict a sequence of 'yes/no' boundary tags at word level in a sentence. Existing labeling techniques can be broadly categorized into two paradigms: recurrent neural networks with conditional random fields (RNN-CRF) [Ma and Hovy, 2016; Lample et al., 2016; Huang et al., 2015] and RNN as a language model to generate the output sequence (seq2seq) [Zheng et al., 2017; Shen et al., 2017]. However, studies [Vinyals et al., 2015; Li et al., 2018b] indicate that they either suffer from sparse boundary tags (i.e., entities are rare and non-entities are common) or they cannot well handle the issue of variable size output vocabulary (i.e., need to retrain models with respect to different vocabularies). Here, we seek a new sequence labeling approach to alleviate these two issues.

In this paper, we propose AT-BDRY, an adversarial transfer approach for named entity boundary detection with pointer networks. First, we integrate the pointer mechanism [Vinyals et al., 2015] into a sequence labeling framework, which can effectively handle variable size vocabulary in the output to produce entity boundaries depending on the input sequence. Second, we introduce adversarial learning with a domain discriminator (adversary), resulting in the emergence of extracted features that are (i) discriminative for sequence labeling on the source domain and (ii) indistinguishable with respect to the shift between source and target domains. It is worth noting that the source-domain data is annotated and target-domain data is unannotated (unsupervised transfer).

In summary, we make three contributions:

• We developed a novel boundary labeling approach based on pointer networks, which infers entity boundaries from a variable input sequence.
• We proposed AT-BDRY, which incorporates adversarial learning into an end-to-end sequence labeling model to encourage domain-invariant representations.
• We conducted Formal Text → Formal Text, Formal Text → Informal Text and ablation evaluations on five datasets. Experimental results show that ATBDRY achieves state-of-the-art transferring performance against recent baselines.

2 RelatedWork

2.1 Named Entity Recognition

There are three common paradigms for NER: knowledgebased unsupervised systems, feature-based supervised systems and neural-based systems. Nadeau and Sekine [2007] summarized knowledge-based unsupervised and featurebased supervised systems in detail. Here, we focus on describing neural-based systems.

Many neural architectures have been applied to NER and achieve state-of-the-art results. Neural-based systems have the advantage of inferring latent features and learning labels in an end-to-end fashion. The use of neural model for NER was pioneered by [Collobert et al., 2011], where an architecture based on temporal convolutional neural networks (CNNs) over word sequence was proposed. Recent neural models for NER can be broadly classified into three dimensions by their representation of words in sentence and architectures of tag decoders [Li et al., 2018a].

Compared with existing architectures, the main difference in our proposed AT-BDRY is that our tag decoder is a pointer network, not CRF. Our model effectively captures sequential dependencies when boundary tags are sparse, while alleviating variable size vocabulary in the output to produce entity boundaries depending on the input sequence. In addition, Zhai et al. [2017] proposed a model for sequence chunking based on pointer networks, which is most related to our work. Different from Zhai's model, our proposed model directly infers the start and end boundaries of an entity, which leads to a simpler architecture with fewer parameters.

2.2 Transfer Learning in NER

Transfer learning aims to perform a machine learning task on a target domain by taking advantage of knowledge learned from a source domain [Pan et al., 2013]. In the setting of transfer learning, different neural models commonly share different parts of model parameters between source task and Target Domain (unannotated): DT : T1, T2, . . . , TN

DT : T1, T2, . . . , TN

Figure 1: The architecture of AT-BDRY. The left part learns a tag decoder for entity boundary detection based on the source domain data, which is then used for the target domain at test time. The right part makes the distributions of source (DS) and target (DT ) domains indistinguishable through a domain discriminator.

target task. Yang et al. [2017] first investigated the transferability of different layers of representations. Then, they presented three different parameter-sharing architectures for cross-domain, cross-lingual, and cross-application scenarios. Pius and Mark [2017] extended Yang's approach to allow joint training on informal corpus and incorporate sentencelevel feature representation. Zhao et al. [2018] proposed a multi-task model with domain adaption, where the fullyconnected layers are adapted to different datasets, and the CRF features are computed separately. Different from these parameter-sharing architectures, Lee et al. [2017] applied transfer learning in NER by training a model on source task and using the trained model on target task for fine-tuning. Recently, Lin and Lu [2018] also proposed a fine-tuning approach for NER by introducing three neural adaptation layers: word adaptation layer, sentence adaptation layer, and output adaptation layer. Cao et al. [2018] proposed a model for adversarial multi-task learning with a cotraining manner.

Unlike these methods (require an amount of annotated data in target domain), our approach assumes no label annotations in the target domain. Moreover, our approach incorporates adversarial learning into a sequence labeling model to encourage domain-invariant representations.

3 AT-BDRY: Adversarial Transfer for Named Entity Boundary Detection

Our approach, termed AT-BDRY, consists of four key components as shown in Figure 1. The input representation layer represents each word with word-level and character-level embeddings. The context encoder aims to extract contextual representations for input sentences. The tag decoder uses a pointer mechanism to detect named entity boundaries based on intermediate representations by the context encoder. The domain discriminator is to judge whether a training instance belongs to source or target domains. In particular, we incorporate adversarial training in the shared space to ensure that

• (a) Representation of an input word "Jordan".

Figure 2: An illustration of entity boundary detection with pointer networks. Input sentence: "Michael Jeffrey Jordan was born in Brooklyn, New York.". The identified entities by boundary detection are "Michael Jeffrey Jordan", "Brooklyn" and "New York".

the intermediate representations by the context encoder contain no discriminative information about the origin of input (source or target). That is, we hope that the intermediate representations can mislead the domain discriminator and correctly guide the tag decoder prediction, while the domain discriminator tries its best to correctly judge the domain class of each example.

3.1 Input Representation

The input representation in our model consists of characterlevel and word-level representations. Previous studies [Chiu and Nichols, 2016; Huang et al., 2015] have shown that character-level information (e.g., prefix and suffix of a word) is an effective resource for NER task. Two kinds of network structures have been used to extract character-level representation, i.e., CNN and BiLSTM. In our model, we use CNN because of its lower computational cost. Our design is similar to [Chiu and Nichols, 2016], and the main difference is that we use a sliding convolution layer (i.e., variable window size of convolution filters) to capture character n-grams in a word, shown in Figure 2(a).

We assume that AT-BDRY works with the annotated data DS : S1, S2, ..., SN in the source domain , and the unannotated data DT : T1, T2, ..., TM in the target domain. Given an input sentence W = (W1,W2, . . . ,WL) of length L, W ∈ {DS,DT}, let Wl denote its l-th word. The characterlevel representation and word-level embedding (e.g., pretrained embedding) for Wl are concatenated as its final representation, xl ∈ RK , where K represents the dimension of xl, see Figure 2(a). Note that hand-crafted features can be easily integrated into this architecture. However, we do not use any hand-crafted features in this study.

3.2 Context Encoder

We encode the input sequence X = (x1, x2, . . . , xL) using an RNN. RNNs capture sequential dependencies. With hidden cells like long short-term memory (LSTM) [Hochreiter and Schmidhuber, 1997] and gated recurrent unit (GRU) [Cho et al., 2014], an RNN captures long distance dependencies without running into the problems of gradient vanishing or explosion. In our implementation, we use the GRU which is computationally cheaper than LSTM. Specifically, GRU activations at time step l are computed as follows:

where σ() is the sigmoid function, tanh() is the hyperbolic tangent function, ⊙ is an element-wise multiplication, zl is the update gate vector, rl is the reset gate vector, nl is the new gate vector, and hl is the hidden state at time step l. U, R, b are the parameters of the encoder that need to be learnt.

We use a bi-directional GRU (BiGRU) network to memorize past and future information in the input sequence. Each hidden state of the BiGRU is formalized as:

where ⊕ indicates a concatenation operation, -→h l and ←- h l are hidden states of the forward (left-to-right) and backward (right-to-left) GRUs, respectively. Assuming the size of the GRU layer is H, the encoder yields hidden states in h ∈ RL×2H .

3.3 Tag Decoder

At each step of decoding, AT-BDRY takes a word Wl from the input sequence as input, and transforms it to its distributed representation xl by looking up the corresponding embedding matrix from the encoding phase. It then passes xl through a GRU-based unidirectional hidden layer. The hidden state at time step j is computed by:

where γ are the parameters in the hidden layer of the GRUbased RNN, which have the same form as defined in Equations (1) - (4). Note that not every word from the input sentence needs pass to the RNN. As shown in Figure 2(b), "Jordan" is the end boundary of the mention "Michael Jeffrey Jordan", so the two words "Jeffrey" and "Jordan" will not be passed to the RNN. Supposing there are J time steps, the decoder RNN produces hidden states in d ∈ RJ×2H with

2H being the dimensions of the hidden layer of the decoder. Again, the encoder is bidirectional (hidden size H), and the decoder is unidirectional (hidden size 2H).

In the pointing phase, AT-BDRY detects entity boundaries only if the current input is a start boundary. Otherwise, it will switch the decoder status to inactive and no boundary will be detected. In order to achieve this mechanism as described above, we pad the hidden states of encoder with a sentinel word representing inactive. That is, the decoder should point to this sentinel word once the current input is not a start boundary of an entity. We also extend pointer networks [Vinyals et al., 2015] with a direction-aware mechanism. Recall that h ∈ RL×2H and d ∈ RJ×H are the hidden states in the encoder and decoder, respectively. We first pad h with a sentinel vector by h = [h; 0], where h ∈ R(L+1)×2H . Then, we use an attention mechanism to compute the distribution of end boundary over all possible positions in the input sequence at decoding step j:

Here, G1 and G2 are learnable parameters, i ∈ [j, J] indicates a possible position in the input sequence, and softmax normalizes uji , indicating the probability that word Wi is an end boundary, given the start boundary Wj. When Wj is not a start boundary of any entity, the pointer is trained to point to the padded word WL+1, i.e., inactive. For example, ATBDRY points to "inactive" when given "was" as the decoder input in Figure 2(b).

3.4 Domain Discriminator

Recall that h ∈ R(L+1)×2H are the hidden states in the context encoder. We use an attention mechanism to form a context vector and apply a Multi-Layer Perceptron to predict domain label yd (binary, i.e., 0 and 1).

3.5 Adversarial Training

The tag prediction loss and the domain prediction loss can be written as

where θf are the learnable parameters of the shared layers (input representation and context encoder), θs are the parameters of the tag decoder, and θt are the parameters of the discriminator. At learning time, in order to encourage domaininvariant features, we seek the parameters θf that maximize the loss of the domain discriminator (by making the two feature distributions as indistinguishable as possible), while simultaneously seeking the parameters θd and θf that minimize the loss of the domain discriminator. In addition, we seek the parameters θt that minimize the loss of the tag decoder. Thus, the optimization problem involves a minimization with respect to some parameters and a maximization with respect to others. Based on this idea, we define the whole objective E(θf , θs, θt) = L(θf , θs) - λL(θf , θt). The parameter λ controls the trade-off between the two objectives. Then, we deliver a saddle point of E(θf , θs, θt) as

Following [Ganin and Lempitsky, 2015], we add a special gradient reversal layer below the shared layer to address the minimax optimization problem.

4 Experiments

4.1 Experimental Setup

Datasets. We use five popular benchmark datasets to ascertain the effectiveness of AT-BDRY. Because our task is boundary detection, we ignore entity types in all datasets. The statistics of the datasets are reported in Table 1. CoNLL03, OntoNotes5.0 and WikiGold are formal text. WNUT16 and WNUT17 are informal text.

Baselines. We evaluate AT-BDRY against 6 baselines. Note that INIT, TranT-B and CDMA-NER require annotated target-domain data for fine-tuning. We randomly leave out 20% of training set, and combine it with development set as annotated target-domain data for these three baselines. We measure Precision (P), Recall (R), and F-score (F1) to evaluate entity boundary detection accuracy.

• SourceOnly - It is trained only on source-domain data with pointer networks (Figure 2(b)) and directly applied to make predictions on target-domain data.
• TJE - This model projects both labels and features into a same low-dimensional space, where classification on the target-domain test data can be done with a simple nearest neighbor rule [Pan et al., 2013].
• INIT - This model trains the parameters on a source dataset and transfers all parameters to initialize the model for training on a target dataset [Lee et al., 2017].
• TranT-B - This model is trained with hierarchical recurrent networks and shares partial parameters in the source and target channels [Yang et al., 2017].

• CDMA-NER - This approach first pre-trains a source model and then uses three adaptation layers in the target model [Lin and Lu, 2018].
• In-Domain - This model is both trained and tested on target-domain data with pointer networks. Therefore, its performance can be regarded as the upper bound of transferring tasks.

Implementation Details. For all neural network models, we use GloVe 300-dimensional pre-trained word embeddings released by Stanford, which are fine-tuned during training. The dimension of character-level representation is 100 and the CNN sliding windows of filters are [2, 3, 4, 5]. The total number of CNN filters is 100. Each bidirectional encoder GRU has a depth of 3 and hidden size of 128. Each decoder GRU has a depth of 3 and hidden size of 256. Note that the encoder GRU is bidirectional and the decoder GRU is unidirectional in our model. Thus, the decoder has twice the hidden size of the encoder. The Adam optimizer was adopted with a learning rate of 0.001, selected from {0.01, 0.001, 0.0001}. We use a dropout of 0.5 after the convolution or recurrent layers. The decay rate is 0.09 and the gradient clip is 5.0. For neural baselines, we use exactly the same hyper-parameter grid and training procedure as our proposed model above. We report the results based on the best performance on the development set. All neural network models are implemented with PyTorch framework and evaluated on NVIDIA Tesla V100 GPU.

4.2 Results

We first present the performance of point networks for indomain entity boundary detection. We then present the transferring performance on Formal Text → Formal Text and Formal Text → Informal Text. Finally, we verify the effect of adversarial transfer via an ablation study.

In-domain Entity Boundary Detection Performance

We train our boundary detection model (shown in Figure 2(b)) on OntoNotes5.0 and compare it with 5 methods. RegEx is created with regular expressions, based on word surface patterns, e.g., letter case. Shallow-CRF trains a CRF using the commonly used token-level features [Liao and Veeramachaneni, 2009]. BiLSTM-MLP/CRF [Chiu and Nichols, 2016] utilizes BiLSTM to encode word sequence, and MLP/CRF to infer decoder tags. The results are summarized in Table 2.

First, our boundary detection model outperforms all baselines in terms of P, R and F1. More specifically, our approach outperforms RegEx, Shallow-CRF, StanfordNER, BiLSTM-MLP, and BiLSTM-CRF by relative F1 improvements of 63.01%, 6.47%, 20.07%, 2.20%, and 1.88%, respectively. We attribute this to the fact that our approach resorts to point networks, which can effectively capture the dependencies of input sentence when the entity boundaries are sparse. Our solution also provides a new perspective to model sequence labeling task using pointer networks instead of the classic CRF-based approach.

Second, the performance of StanfordNER is poor. This observation coincides with our previous claim that off-the-shelf NER systems are not specifically designed for entity boundary detection. Commonly, off-the-shelf NER systems cannot work well on cross-domain datasets because they do not take into account domain adaptation.

For better understanding, we visualize pointer attention weights with an example in Figure 3. The words on the y-axis are the input of the GRU decoder. The words on the x-axis are the input of the GRU encoder. For example, for the input "New", our approach detects the end boundary of this entity at the position "Company". For a given input "of ", our approach determines that it is not the start of an entity mention by the sentinel word "Inactive". Observe that the identified boundaries have dominant attention weights, which implies that our approach can successfully learn sentence features for entity boundary detection.

Formal Text → Formal Text

We consider one formal text data (e.g., CoNLL03) as the source domain and another formal text (e.g., Wikigold) as the target domain. Table 3 reports the performance on standard test sets of target domains.

From the results, we have the following observations. (1) AT-BDRY significantly outperforms all baseline methods to date. Compared with the second-best performance, ATBDRY achieves relative F1 improvements of 3.98%, 1.95%, 3.02% and 4.22%, corresponding to each transferring task. (2) With the adversarial transfer mechanism, AT-BDRY significantly improves the performance of training only on source datasets. More specifically, AT-BDRY achieves relative F1 improvements of 13.69%, 3.25%, 9.52% and 12.13%, compared with the SourceOnly method. (3) For the case of CoNLL03→Wikigold, although AT-BDRY does not outperform the In-Domain method, the performance of AT-BDRY is close to the upper bound. It is worth noting that AT-BDRY

does not use any annotated target-domain data.

Formal Text → Informal Text

NER in formal text is well studied and the performance of many existing models drops dramatically in informal text. In this experiment, we investigate the effectiveness of AT-BDRY when the data distribution gap between the source domain and the target domain increases. Therefore, we choose formal text as the source domain and informal text as the target domain. Table 4 summarizes the results. Similar findings from Formal Text → Formal Text are observed in this experiment. In addition, we also observe that a better performance is achieved when OntoNotes5.0 (rather than CoNLL03) is taken as the source domain. In most cases, the recall score is much higher than the precision score, which implies that NER in formal text is a very challenging task because of the noisy nature of informal text as well as emerging entities with novel surfaces. Notably, AT-BDRY achieves an F1 of 51.83% on OntoNote5.0→WNUT17. This F1 score is much higher than the that of SourceOnly (46.04%) and is close to the upper bound (56.09%). We attribute this to the fact that ATBDRY can learn domain-invariant representations, which can effectively reduce the distance in data distributions between the source and target domains.

4.3 Ablation Study

In our approach, the adversarial transfer component plays a role in encouraging domain-invariant representations. In this experiment, we investigate the effect of adversarial transfer by visualization. With t-SNE [Maaten and Hinton, 2008], Figure 4 demonstrates the effect of adversarial transfer on the distributions of the extracted features from the BiGRU encoder in the CoNLL03 → WNUT16 experiment. We can observe that our AT-BDRY approach possessing adversarial

• (a) Without adversarial transfer
• (b) With adversarial transfer

Figure 4: The effect of adversarial transfer on the distribution of the extracted features from the BiGRU encoder. The adversarial transfer in our approach makes the two distributions of features much closer.

transfer learning, makes the two distributions of the extracted features much closer.

5 Conclusion

In this paper, we study the task of named entity boundary detection. We propose AT-BDRY, an end-to-end neural model to detect entity boundaries from text without any handcrafted features or any prior linguistic knowledge. In particular, our approach incorporates adversarial transfer learning into a novel sequence labeling model to encourage domaininvariant representations. AT-BDRY ensures the extracted features are discriminative (for sequence labeling) and indistinguishable (for the discriminator) so that the knowledge from source domain can be transferred to the target domain, without requiring any annotated target-domain data. Through extensive experiments, we demonstrate the effectiveness of AT-BDRY against state-of-the-art solutions on different datasets.