Leveraging Document-Level Label Consistency for Named Entity Recognition
Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence. Main track. Pages 3976-3982.
https://doi.org/10.24963/ijcai.2020/550
Keywords
- Natural Language Processing: Named Entities
- Natural Language Processing: Tagging, chunking, and parsing
Abstract
Document-level label consistency is an effective indicator that different occurrences of a particular token sequence are very likely to have the same entity types. Previous work focused on better context representations and used the CRF for label decoding. However, CRF-based methods are inadequate for modeling document-level label consistency. This work introduces a novel two-stage label refinement approach to handle document-level label consistency, where a key-value memory network is first used to record draft labels predicted by the base model, and then a multi-channel Transformer makes refinements on these draft predictions based on the explicit co-occurrence relationship derived from the memory network. In addition, in order to mitigate the side effects of incorrect draft labels, Bayesian neural networks are used to indicate the labels with a high probability of being wrong, which can greatly assist in preventing the incorrect refinement of correct draft labels. The experimental results on three named entity recognition benchmarks demonstrated that the proposed method significantly outperformed the state-of-the-art methods.
- [Chen et al., 2019] Hui Chen, Zijia Lin, Guiguang Ding, Jian-Guang Lou, Yusen Zhang, and Börje F. Karlsson. Grn: Gated relation network to enhance convolutional neural network for named entity recognition. In AAAI, 2019.
- [Chiu and Nichols, 2016] Jason PC Chiu and Eric Nichols. Named entity recognition with bidirectional lstm-cnns. TACL, 4:357-370, 2016.
- [Cui and Zhang, 2019] Leyang Cui and Yue Zhang. Hierarchically-refined label attention network for sequence labeling. In EMNLP-IJCNLP, 2019.
- [Dai et al., 2019] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. In ACL, 2019.
- [Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL, pages 4171-4186, 2019.
- [Gal and Ghahramani, 2016a] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML, pages 1050-1059, 2016.
- [Gal and Ghahramani, 2016b] Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. In NeurIPS, 2016.
- [Hu et al., 2019] Anwen Hu, Zhicheng Dou, Jian-Yun Nie, and Ji-Rong Wen. Leveraging multi-token entities in document-level named entity recognition. In AAAI, 2019.
- [Kendall and Gal, 2017] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In NeurIPS, pages 5574-5584, 2017.
- [Krallinger et al., 2015] Martin Krallinger, Florian Leitner, Obdulia Rabal, Miguel Vazquez, Julen Oyarzabal, and Alfonso Valencia. Chemdner: The drugs and chemical names extraction challenge. Journal of cheminformatics, 7(1):S1, 2015.
- [Krishnan and Manning, 2006] Vijay Krishnan and Christo- pher D Manning. An effective two-stage model for exploiting non-local dependencies in named entity recognition. In ACL, pages 1121-1128, 2006.
- [Lample et al., 2016] Guillaume Lample, Miguel Balles- teros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. Neural architectures for named entity recognition. In NAACL, pages 260-270, 2016.
- [Liu et al., 2019] Yijin Liu, Fandong Meng, Jinchao Zhang, Jinan Xu, Yufeng Chen, and Jie Zhou. Gcdt: A global context enhanced deep transition architecture for sequence labeling. arXiv preprint arXiv:1906.02437, 2019.
- [Luo et al., 2020] Ying Luo, Fengshun Xiao, and Hai Zhao. Hierarchical contextualized representation for named entity recognition. In AAAI, 2020.
- [Ma and Hovy, 2016] Xuezhe Ma and Eduard Hovy. End-to- end sequence labeling via bi-directional lstm-cnns-crf. In ACL, pages 1064-1074, 2016.
- [Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. In NeurIPS. 2013.
- [Miller et al., 2016] Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. Key-value memory networks for directly reading documents. In EMNLP, pages 1400-1409, 2016.
- [Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
- [Peters et al., 2018] Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL-HLT, pages 2227-2237, 2018.
- [Qian et al., 2019] Yujie Qian, Enrico Santus, Zhijing Jin, Jiang Guo, and Regina Barzilay. Graphie: A graph-based framework for information extraction. In NAACL-HIT, pages 751-761, 2019.
- [Strubell et al., 2017] Emma Strubell, Patrick Verga, David Belanger, and Andrew McCallum. Fast and accurate entity recognition with iterated dilated convolutions. In EMNLP, 2017.
- [Tjong Kim Sang and De Meulder, 2003] Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In NAACL-HLT, pages 142-147, 2003.
- [Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 5998-6008, 2017.
- [Weischedel et al., 2013] Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23, 2013.
- [Ye and Ling, 2018] Zhixiu Ye and Zhen-Hua Ling. Hybrid semi-markov crf for neural sequence labeling. In ACL, pages 235-240, 2018.
- [Zhang et al., 2018a] Boliang Zhang, Spencer Whitehead, Lifu Huang, and Heng Ji. Global attention for name tagging. CoNLL, 2018.
- [Zhang et al., 2018b] Yuan Zhang, Hongshen Chen, Yihong Zhao, Qun Liu, and Dawei Yin. Learning tag dependencies for sequence tagging. In IJCAI, pages 4581-4587, 2018. IJCAI
- [Zhang et al., 2018c] Yue Zhang, Qi Liu, and Linfeng Song. Sentence-state lstm for text representation. In ACL, 2018.
References were extracted automatically from the PDF and may contain errors.
Full text extracted automatically from the PDF; figures, tables and equations may differ. The PDF is the version of record.
1 Introduction
The task of named entity recognition (NER) involves determining entity boundaries and recognizing the categories of named entities, which is a fundamental task in the field of natural language processing (NLP). Because most neural NER models are implemented using bi-directional long short-term memory (BiLSTM) networks [Ma and Hovy, 2016; Lample et al., 2016; Peters et al., 2018], they have a limited ability to exploiting non-local and nonsequential dependencies such as co-references and identical mentions [Qian et al., 2019].
To tackle the above challenges, several variations of LSTM have been proposed to incorporate sentence-level context information [Zhang et al., 2018c; Liu et al., 2019]. Recently, many of the existing methods have focused on the better incorporation of document-level context information [Hu et al., 2019; Qian et al., 2019], because different occurrences of a particular token sequence are very likely to have the same entity types within a document [Krishnan and Manning, 2006]. In particular, Luo et al. [2020] simultaneously utilized sentence-level and document-level representations. This hierarchical contextualized representation architecture enhanced NER with better global information modeling.
∗Both authors contributed equally
However, the focus of the previous work was only on document-level context representations, with little work done to explicitly model the document-level label consistency between the same token sequences. As shown in Figure 1, we argue that using Conditional Random Fields (CRF) is not competent for modeling the document-level relationships of labels [Qian et al., 2019; Luo et al., 2020]. Recently, many methods have introduced label embeddings to manage longer ranges of dependencies [Zhang et al., 2018b; Cui and Zhang, 2019]. However, these methods are still trapped in the sentence-level label dependencies.
This work handles document-level label consistency by introducing a novel two-stage label refinement networks that are stacked using the BiLSTM and multi-channel Transformer. In a document, all of the draft labels predicted by the BiLSTM networks are fed as inputs to the next layer (Transformer), so that document-level label dependencies can be considered. Specifically, at the first stage, the BiLSTM networks take a document as input, and yield all of the hidden state vectors, together with draft labels sentence by sentence. A key-value memory component [Miller et al., 2016] is adopted, which memorizes all the hidden state vectors and their corresponding label embeddings of the entire document. At the second stage, a multi-channel Transformer performs attention over document-level label embeddings derived from the memory network to explicitly model the co-occurrence relationship, as well as sentencelevel label embeddings tomodel the local label dependencies, and hidden state vectors to model the context representations. All of these features are fused to refine the draft labels in parallel, which can avoid the use of Viterbi decoding of the CRF for a faster prediction. In addition, in order to mitigate the side effects of incorrect draft labels, Bayesian neural networks [Gal and Ghahramani, 2016b] are used to indicate the draft labels with a high probability of being wrong, which can greatly assist in preventing the incorrect refinement of correct draft labels. The experimental results on three named entity recognition benchmarks demonstrated that the proposed method significantly outperformed the previous state-of-the-art methods.
The main contributions of this paper can be summarized as follows: 1) novel two-stage label refinement networks are proposed, which can better model document-level label dependencies; 2) the proposed method can model label dependencies in parallel, which performs up to 5.48 times faster than state-of-the-art methods during the inference phase; 3) the use of Bayesian neural networks is proposed to estimate the uncertainty of the predictions and indicate potentially incorrect labels that should be refined; and 4) the experimental results across three NER datasets indicate that the proposed method significantly outperforms the start-ofthe-art methods. Our codes are released at Github 1 .
2 RelatedWork
2.1 Neural Named Entity Recognition
In recent years, many neural network-based methods have achieved competitive performances without massive handcrafted feature engineering, including LSTM-based methods because of their advantages in modeling sequence data [Lample et al., 2016; Ma and Hovy, 2016] and CNN-based methods because of their proficiency in parallel modeling [Strubell et al., 2017]. In order to model non-local and non-sequential dependencies, sentence-level [Zhang et al., 2018c; Liu et al., 2019] and document-level contextualized information [Qian et al., 2019; Luo et al., 2020] has been adopted to eliminate the limitations of RNNs resulting from their sequential nature. In contrast to their work, which only modeled the dependencies between words, the method reported here also models the sentence-level and documentlevel dependencies between labels.
2.2 Label DependencyModeling
Creating better models for label dependencies has always been the focus of sequence labeling tasks [Ye and Ling, 2018; Zhang et al., 2018b]. In particular, the CRF layer is integrated with neural encoders to capture label transition patterns
1 https://github.com/jiacheng-ye/DocL-NER
[Ma and Hovy, 2016]. Many of the recent methods have introduced label embeddings to manage longer ranges of dependencies [Zhang et al., 2018b; Cui and Zhang, 2019]. However, these methods are still trapped in the sentencelevel label dependencies. Inspired by Krishnan and Manning [2006], they proposed a two-stage approach for documentlevel label consistency, but it required slower two-layer CRFs and hand-crafted features. In contrast, the method reported here can model both the sentence-level label dependencies and document-label consistencies in parallel without handcrafted features.
2.3 Bayesian Neural Networks
There are two main types of uncertainty in Bayesian modeling [Kendall and Gal, 2017]. Aleatoric uncertainty captures the noise inherent in the observations, and epistemic (model) uncertainty accounts for the uncertainty in the model parameters. This work focused on the use of epistemic uncertainty to indicate whether model predictions were likely to be incorrect, which could effectively prevent correct draft labels from being incorrectly refined.
Given the dataset D with training inputs X = {x1, . . . , xn} and their corresponding outputs Y = {y1, . . . , yn}, Bayesian inference looks for the posterior distribution of the parameters given the dataset p(W|D). This makes it possible to predict an output for a new input point x∗ by marginalizing over all of the possible parameters, as follows:
Bayesian inference is intractable for many models because of the complex nonlinear structures and high dimension of the model parameters. Recent advances in variational inference introduced new techniques into the field. Among these, Monte Carlo dropout [Gal and Ghahramani, 2016a] requires minimum modification to the original model. It is possible to use the variational inference approach to find an approximation qθ(W∗ ) to the true posterior p(W|D) parameterized by a different set of weights θ, where the Kullback-Leibler (KL) divergence of the two distributions is minimized. The integral can be approximated as follows:
In contrast to non-Bayesian networks, at test time, dropouts are also activated. As a result, model uncertainty can be approximately evaluated by summarizing the variance of the model outputs from multiple forward passes.
3 Document-Level Label Consistency NER
This work proposes a two-stage label refinement framework for document-level label consistency NER (DocL-NER), which first uses a key-value memory network to record the draft labels predicted by the base model, and then applies a multi-channel Transformer to makes refinements on the draft predictions based on the explicit co-occurrence relationship
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
o
m
U
k
+
Y
d
2
E
8
9
X
P
K
F
I
>
A
C
z
3
c
j
V
H
L
S
N
D
r
p
f
Z
u
g
w
R
y
5
0
q
/
G
Q
O
J
v
T
W
M
7
n
B
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
w
R
Z
z
u
0
k
r
H
L
B
F
8
V
P
7
N
>
A
C
3
c
j
S
D
2
p
f
d
g
I
q
K
y
Y
9
5
M
o
X
Q
T
W
/
+
U
J
v
O
G
n
E
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
Q
g
q
m
c
A
p
5
Y
+
X
f
v
F
y
>
C
z
3
j
V
H
L
S
N
D
2
r
Z
d
u
k
w
J
W
U
T
E
/
0
8
7
9
G
K
n
I
M
O
B
P
o
R
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
y
N
P
J
8
D
3
2
n
M
g
A
d
R
E
0
L
>
C
z
c
j
V
H
S
F
r
p
f
Z
u
k
w
U
T
o
X
Q
K
W
/
+
9
7
5
m
q
v
Y
O
I
B
G
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
j
0
9
V
P
m
G
v
K
w
D
F
8
L
d
c
o
>
A
C
z
3
H
T
7
r
p
X
O
5
E
Y
J
U
+
I
k
Z
g
W
M
u
R
S
y
Q
N
/
B
f
q
2
n
j
H
L
T
J
F
D
U
+
d
k
g
N
u
G
B
X
f
p
Q
v
O
P
n
7
5
9
o
0
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
B
T
n
q
7
G
g
9
u
F
f
Y
y
I
p
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
V
w
R
y
S
r
8
2
A
m
/
M
W
Z
I
K
E
q
Y
>
C
z
3
c
w
k
V
>
A
C
z
3
c
j
H
L
S
N
D
2
r
Z
d
m
U
K
P
o
+
/
E
8
M
R
5
W
X
O
v
Q
0
J
S
F
D
2
p
f
d
u
g
k
w
R
J
B
T
y
n
v
m
K
+
9
P
M
o
z
I
5
/
<
l
a
t
e
x
i
s
h
B
<
l
a
t
e
x
i
s
h
1
_
b
6
4
<
l
a
t
e
x
i
s
1
_
b
6
4
=
"
I
j
Y
B
J
n
+
L
N
M
7
v
y
9
3
O
S
A
>
C
X
c
V
H
2
f
u
k
w
J
B
T
O
y
W
0
q
v
Q
E
m
R
K
7
+
P
8
M
o
G
Y
3
5
/
Z
d
u
g
k
w
T
W
0
q
Q
E
m
R
K
P
8
o
G
z
5
/
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
Z
n
9
D
p
z
I
F
j
d
L
U
S
C
r
g
>
A
X
c
V
H
N
F
D
2
r
p
f
U
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
P
M
g
3
7
G
k
A
B
v
D
/
z
8
c
y
>
C
X
j
V
H
L
S
W
0
q
Q
E
m
R
K
+
9
o
I
Y
5
N
F
2
r
p
f
U
Z
d
u
w
J
T
O
n
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
q
f
Q
P
C
O
E
F
J
m
0
r
2
Y
w
n
+
>
A
X
c
j
V
H
G
z
I
3
5
/
L
S
N
D
p
U
Z
d
u
g
k
B
T
y
W
v
R
K
7
9
8
M
o
>
C
j
V
H
L
F
D
r
p
f
U
Z
d
u
g
k
T
O
y
n
W
0
q
v
Q
E
K
7
+
9
P
M
o
I
Y
3
5
h
1
_
b
6
4
=
"
A
d
R
E
V
S
m
K
n
G
X
g
3
7
>
C
c
j
H
L
N
F
D
2
r
p
f
U
Z
u
k
w
J
B
T
O
y
W
0
q
v
Q
+
9
P
8
M
o
z
I
Y
5
/
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
3
0
U
Z
L
A
Y
O
7
Q
r
W
8
X
E
q
N
G
>
C
c
j
V
H
=
"
z
A
N
/
J
m
w
X
2
R
G
c
8
S
U
q
r
X
2
0
3
N
p
k
K
D
g
>
A
C
c
j
H
L
S
f
d
u
w
J
T
O
y
W
v
E
m
R
7
9
P
M
o
G
z
I
Y
5
/
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
T
>
A
C
j
V
H
L
S
N
F
D
r
f
Z
d
g
k
w
B
y
n
v
E
R
7
+
9
P
M
o
G
z
I
Y
5
/
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
n
B
+
Q
Z
F
V
8
F
D
r
p
f
U
w
R
J
B
T
O
y
0
q
v
Q
E
m
K
9
P
8
G
z
I
Y
3
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
X
0
K
8
3
c
W
O
J
Q
U
p
u
q
2
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
X
7
M
u
k
o
j
L
d
2
g
Z
/
W
c
n
+
>
A
C
V
H
S
N
X
K
5
8
P
o
A
E
z
p
Z
d
L
f
7
B
V
>
C
c
j
H
S
N
F
D
2
r
U
u
g
k
w
J
T
O
y
n
W
0
q
v
Q
m
R
+
9
M
G
I
Y
3
/
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
/
G
r
K
L
X
c
O
T
0
>
X
c
H
L
S
N
D
2
r
p
d
u
w
J
B
y
n
W
q
v
Q
E
m
R
K
7
+
9
M
G
z
I
Y
5
H
L
S
N
F
r
p
Z
d
u
k
B
n
W
q
v
Q
E
m
R
K
P
o
z
I
Y
3
5
/
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
F
P
8
f
C
g
3
V
o
O
j
A
U
k
Z
/
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
f
T
2
G
V
0
M
g
c
C
O
7
y
+
J
U
D
9
8
w
>
A
X
j
9
B
m
W
w
H
Q
u
8
j
P
n
g
>
A
C
V
S
N
F
D
2
p
f
U
Z
d
k
J
T
y
0
q
v
E
R
7
+
M
o
z
I
Y
3
5
derived from the memory network. In order to make a fair comparison with the general LSTM-based methods and to mitigate the side effects of incorrectly refining correct draft labels, in this work, variational LSTMs as special Bayesian neural networks are used to encode sentences and determine the labels with a high probability of being wrong. The proposed model is shown in Figure 2.
3.1 Key-Value Memory for Draft Label Recording
At the first stage, a variational LSTM (VLSTM) [Gal and Ghahramani, 2016b] is adopted as the base model for the draft label and uncertainty predictions. To store document-level information in preparation for subsequent label consistency modeling, a key-value memory network is used to record all of the hidden state vectors and corresponding label embeddings in a document.
Word and Label Representation Following Lample et al. [2016], character information was used to enhance the word representation. Given a sequence of words s = {w1, w2, . . . , wn}, the product of the one-hot encoded vector with an embedding matrix then gives a word embedding: wi = ew (wi), where ew denotes a word embedding lookup table. Each word is made up of a sequence of characters, and CNNs are adopted for character-level encoding ci. Then, a word is represented by concatenating its word embedding and character-level encoding: xi = [wi; ci]. All of the word representations make up an embedding matrix E ∈ RV ×dw , where dw is the embedding dimensionality of x, and V is the number of words in the vocabulary.
Given the label set L = {l1, . . . , lN}, each label lk is represented by lk = el (lk) ∈ Rdl , where el denotes a label embedding lookup table. Label embeddings are randomly initialized and tuned during training.
Draft Label and Uncertainty Prediction. After obtaining a sequence of word representations X = {x1, . . . , xn}, VLSTM is adopted to generate the contextual features and uncertainty for each word. The forward and backward hidden states are concatenated for the final representation: hi = [-→h i; ←- h i] ∈ Rdh . The forward pass of the VLSTM is run T times with the same inputs by using the approximated posterior qθ(W∗ ) in Equation 2 to obtain the draft labels as follows:
where Wl is a dl × dh matrix and Softmax(zi;j) = exp(zi;j)/ ∑j exp(zi;j). Similar to a classic sequence labelingmodel, themodel applies l∗ i = argmax(pi) to obtain the draft label. Then, the uncertainty of this probability vector pi can be summarized using the entropy of the probability vector:
In this way, it is possible to obtain the draft labels L∗ = {l∗ 1 , l∗ 2 , . . . , l∗n } coupled with the corresponding epistemic uncertainties U = {u1, u2, . . . , un} for each input sentence.
Document-Level Representation. We introduce a keyvalue memory networks [Miller et al., 2016] to simultaneously record the context representations (key) and corresponding label embeddings (value) for each word. Specifically, we create a document-level memory matrix M = {mw1 , . . . ,mwm }, in which the same words under different context would occupy many different slots and form a quired subset mwi . The memory slots in one subset mwi are defined as pairs of vectors (ki;1, vi;1), . . . , (ki;m, vi;m). In every single slot j, the key represents a certain hidden state vector hi;j, and the value is the corresponding label embedding li;j.
3.2 Multi-Channel Transformer for Refinement
At the second stage, the draft predictions are refined based on the explicit co-occurrence relationship derived from the keyvalue memory network.
With the constructed memory matrix, a key addressing and value reading mechanism is designed to access the documentlevel context representations and label consistencies. During addressing, each slot of the corresponding subset is assigned a relevance probability by comparing the word to each key:
where Wl is a dw × dh matrix. We hope that the model can refer to the semantics and corresponding labels of other
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
p
q
f
D
X
O
0
G
Q
C
A
c
T
Y
V
N
>
j
H
L
S
F
2
r
Z
d
u
g
k
w
R
I
3
B
K
m
n
v
J
9
+
/
o
z
7
5
E
M
8
y
W
P
U
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
d
G
E
H
U
M
J
C
2
L
+
R
o
D
W
8
>
A
0
X
c
j
V
S
N
F
r
p
f
Z
u
g
k
w
I
q
v
B
K
m
n
T
Y
9
Q
/
z
3
7
5
O
P
y
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
o
m
U
k
+
Y
d
2
E
8
9
X
P
K
F
I
>
A
C
z
3
c
j
V
H
L
S
N
D
r
p
f
Z
u
g
w
R
y
5
0
q
/
G
Q
O
J
v
T
W
M
7
n
B
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
z
M
W
0
P
y
p
/
F
c
w
A
o
5
f
K
S
V
E
g
>
C
X
j
H
L
N
D
2
r
Z
d
u
k
I
q
n
R
J
+
3
Q
v
m
8
O
T
9
U
Y
7
B
G
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
I
d
7
Q
+
C
3
Y
c
y
O
5
W
M
A
u
S
D
r
0
>
X
j
V
H
L
N
F
2
p
f
Z
g
k
w
q
K
m
z
T
o
P
U
G
E
8
v
R
9
B
J
n
/
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
3
k
7
I
r
8
p
W
w
V
m
D
9
/
P
Y
E
o
c
>
A
C
0
X
j
H
L
S
N
F
2
f
Z
d
u
g
q
K
Q
G
y
U
z
+
M
v
T
n
R
O
J
B
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
S
v
2
N
j
V
Q
X
z
g
D
c
P
/
o
f
R
8
>
A
C
3
H
L
F
r
p
Z
d
u
k
w
I
O
E
+
0
y
W
U
K
G
J
m
M
n
Y
5
7
9
q
T
B
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
V
/
G
S
F
7
T
u
m
I
A
5
q
3
w
K
R
O
P
Q
>
C
0
X
c
j
H
L
N
D
2
r
p
f
Z
d
g
k
v
B
n
W
Y
9
+
J
o
z
E
M
8
y
U
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
j
Y
r
0
F
c
U
k
A
y
C
B
R
u
Z
P
5
8
g
>
X
V
H
L
S
N
D
2
p
f
d
w
I
o
q
G
v
J
O
9
m
E
z
/
+
Q
T
3
W
7
n
M
K
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
w
R
Z
z
u
0
k
r
H
L
B
F
8
V
P
7
N
>
A
C
3
c
j
S
D
2
p
f
d
g
I
q
K
y
Y
9
5
M
o
X
Q
T
W
/
+
U
J
v
O
G
n
E
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
r
S
y
f
z
Y
L
B
q
W
0
C
R
Q
v
P
A
>
X
c
j
V
H
N
F
D
2
p
Z
d
u
g
k
w
I
K
3
G
8
m
E
U
9
/
o
+
M
T
n
7
O
J
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
D
N
L
u
q
g
n
z
y
P
o
r
c
9
E
Z
m
7
A
>
C
0
X
j
V
H
S
F
2
p
f
d
k
w
I
K
W
3
Q
U
Y
R
/
+
8
5
J
G
v
T
O
M
B
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
j
H
5
m
7
v
/
T
D
W
Y
F
f
d
L
3
c
>
A
C
0
X
V
S
N
2
r
p
Z
u
g
k
w
J
U
y
R
O
9
8
E
q
z
o
+
G
Q
P
K
I
M
B
n
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
c
2
P
o
B
X
9
5
A
H
F
z
K
V
3
7
d
+
>
C
0
j
L
S
N
D
r
p
f
Z
u
g
k
w
J
W
U
y
R
O
8
m
E
q
/
G
Q
v
T
Y
I
M
n
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
Q
g
q
m
c
A
p
5
Y
+
X
f
v
F
y
>
C
z
3
j
V
H
L
S
N
D
2
r
Z
d
u
k
w
J
W
U
T
E
/
0
8
7
9
G
K
n
I
M
O
B
P
o
R
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
S
v
2
N
j
V
Q
X
z
g
D
c
P
/
o
f
R
8
>
A
C
3
H
L
F
r
p
Z
d
u
k
w
I
O
E
+
0
y
W
U
K
G
J
m
M
n
Y
5
7
9
q
T
B
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
I
d
D
M
F
n
E
U
Q
P
A
p
W
k
r
0
z
j
>
C
X
c
V
H
L
S
N
2
f
Z
u
g
w
J
3
B
/
Y
R
m
+
9
8
5
G
K
7
v
O
q
T
y
o
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
5
8
D
S
Q
3
2
J
r
m
R
7
+
c
Z
V
F
U
Y
>
A
C
0
X
j
H
L
N
p
f
d
u
g
k
w
W
B
/
P
z
I
9
G
K
n
v
M
O
E
q
T
y
o
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
u
V
E
G
Q
o
U
Z
W
r
B
p
c
N
9
D
Y
M
8
>
A
C
0
X
j
H
L
S
F
2
f
d
g
k
w
J
y
R
O
m
q
z
/
+
P
v
T
3
K
7
I
n
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
B
G
c
N
m
X
5
S
H
O
A
2
9
W
R
P
y
V
v
M
Y
>
C
0
j
L
F
D
r
p
f
Z
d
u
g
k
w
J
U
8
E
q
z
/
o
+
Q
T
3
K
7
I
n
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
y
N
P
J
8
D
3
2
n
M
g
A
d
R
E
0
L
>
C
z
c
j
V
H
S
F
r
p
f
Z
u
k
w
U
T
o
X
Q
K
W
/
+
9
7
5
m
q
v
Y
O
I
B
G
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
z
M
W
0
P
y
p
/
F
c
w
A
o
5
f
K
S
V
E
g
>
C
X
j
H
L
N
D
2
r
Z
d
u
k
I
q
n
R
J
+
3
Q
v
m
8
O
T
9
U
Y
7
B
G
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
f
A
j
u
8
0
c
F
H
y
z
W
S
9
r
k
E
Q
>
C
X
V
L
N
D
2
p
Z
d
g
w
J
U
3
B
/
P
Y
I
R
m
+
5
G
K
7
n
v
M
O
q
T
o
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
W
F
v
p
S
P
R
k
G
0
T
Q
8
u
n
Z
o
>
A
C
X
c
j
V
H
L
N
D
2
r
f
d
g
w
J
U
3
B
/
z
Y
I
m
+
9
5
K
7
M
O
E
q
y
sentences in the document when predicting the label of a word. Hence, in the reading step, the keys and values of the memory are read by taking their weighted sum using the addressing probabilities, and the document-level representations and label embeddings are returned:
We further use the multi-channel Transformer [Vaswani et al., 2017] incorporating relative position encoding [Dai et al., 2019] to model sentence-level dependencies in parallel. We propose to reparameterize the relative position encoding as follows:
where Ah2h i,j andAh2l i,m denotes the attention fromhi to hj and hi to lm, respectively. Ri-j is the encoding of the relative distance between position i and j, and R is a sinusoid matrix like that in [Dai et al., 2019]. ϕ = {W, u, and v} are learnable parameters.
The proposed relative positional encoding can be used for the multi-channel Transformer architecture. The computational procedure for one layer with a single attention head can be summarized as follows:
3.3 Training and Decoding with Uncertainty
There are two networks to be optimized: the VLSTM for draft labels and uncertainty estimation, and the multi-channel Transformer for label refinement. The ultimate training goal is to minimize the total loss function on the two models: Ltotal = L1(θ, r) + L2(ϕ).
The VLSTM performs approximate variational inference, where we use a simple Bernoulli distribution (dropout) qθ(W∗ ) in a tractable family, which minimizes the KullbackLeibler (KL) divergence to the true model posterior p(W|D). The minimization objective is given as follows:
where N is the number of data points, and r is the dropout probability to sampleW∗ j ∼ qθ(W∗ ).
For the multi-channel Transformer, we use the concatenation of H2 and L2 for the final prediction ˆli. In particular, we can optimize the model using the cross entropy loss as follows:
where li is the one-hot vector of the true label.
When the training is complete, we can obtain the draft labels L∗ = {l∗ 1 , l∗ 2 , . . . , l∗n } coupled with corresponding uncertainties U = {u1, u2, . . . , un} from the VLSTM, and refined labels ˆL = {ˆl1, ˆl2, . . . , ˆln} from the multi-channel Transformer. We find that the average uncertainty value of incorrect draft labels is 29 times larger than that of correct draft labels in CoNLL2003. Hence,we can use uncertainty to indicate the labels with a high probability of being wrong. To prevent the correct labels from being incorrectly modified, we set an uncertainty threshold Γ to distinguish which labels should be used, i.e., refined labels are used when ui > Γ and vice versa (as an example, given u1 > Γ, u2 ≤ Γ, and un > Γ, decoding labels will become {ˆl1, l∗ 2 , . . . , ˆln}).
4 Experimental Setup
4.1 Datasets
We conduct experiments on three benchmark NER datasets. We follow the standard split of each corpora. Statistics are listed in Table 1.
CoNLL2003. The shared task of the CoNLL2003 dataset [Tjong Kim Sang and De Meulder, 2003] for named entity recognition was collected from Reuters Corpus. This dataset divides name entities into four different types: persons (PER), locations (LOC), organizations (ORG), and miscellaneous(MISC). The original files use -DOCSTARTas document separator and we keep this to conduct our document-level experiments.
OntoNotes 5.0. The English NER dataset OntoNotes 5.0 [Weischedel et al., 2013] is a large corpus consisting of various sources (newswire, broadcast, telephone speech, etc.). It is tagged with eighteen entity types (DATE, TIME, ORDINAL, etc.), Part number was regraded as document indicator in our experiments.
CHEMDNER. The CHEMDNER corpus consists of 10,000 PubMed abstracts published in the top journals from various chemistry-related disciplines, which contains a total of 84,355 chemical entity mentions labeled manually by expert chemistry literature curators [Krallinger et al., 2015]. The corpus is tagged with only one chemical entity type. We regard each abstract as a document in our experiments.
4.2 ComparisonMethods
BiLSTM-CRF. Ma and Hovy [2016] utilizes a CRF layer on the top of the BiLSTM to model the interaction between two successive labels [Lample et al., 2016] instead of making independent labeling decisions for each output. In the decoding stage, the Viterbi algorithm is used to find the highest scored label sequence over the entire word sequence.
GraphIE. GraphIE [Qian et al., 2019] utilizes a cooccurrence graph to incorporate document-level contextual information and CRF to model sentence-level label dependency. This methods achieves great performances in many information extraction tasks, including textual, social media and visual information extraction.
| Dataset | Type | Train | Dev | Test |
|---|---|---|---|---|
| CoNLL2003 | #doc #sent | 946 14,041 | 216 3,250 | 231 3,453 |
| OntoNotes | #doc #sent | 2,483 59,924 | 319 8,528 | 322 8,262 |
| CHEMDNER | #doc #sent | 3,500 | 3,500 | 3,000 26,435 |
| 30,802 | 30,807 |
Hier-NER. Hier-NER [Luo et al., 2020] introduces a corpus-level memory mechanism to utilize the global contextual information of a token. Moreover, a sentencelevel encoder is used to enhance the sentence representation learned from an independent BiLSTM. A CRF layer is used as the final decoder in Hier-NER. With two-level hierarchical representations, Hier-NER established state-of-the-art results on several NER tasks.
4.3 Hyper-Parameter Settings
Following [Ma and Hovy, 2016; Qian et al., 2019; Luo et al., 2020], we use the same 100-dimensional Glove embedding 2 [Pennington et al., 2014] as word embedding for the CoNLL2003 and OntoNotes datasets. For CHEMDNER, we use 50-dimensional pretrained word2vec [Mikolov et al., 2013] embedding, which is the same as [Qian et al., 2019]. For CharCNN, we use 32-dimensional character embeddings and 32 filters of width 3 for CoNLL2003 and OntoNotes, and 128-dimensional character embeddings and 128 filters of width 2 to 4 for CHEMDNER. For the first stage, we use 1 layer of VLSTM with 200 dimensions for CoNLL2003 and 2 layers for the other datasets. For the second stage, we use 3 layers and 4 layers of Transformer for CoNLL2003, OntoNotes and CHEMDNER, respectively. Dropout is set to 0.5, and the number of samples is set to 32 for all the datasets. The standard entity-level F1 score is used as evaluation metric. For all the experiments, we use the BIOES tag scheme instead of standard BIO2, as previous studies have reported a meaningful improvement with this scheme [Ma and Hovy, 2016; Lample et al., 2016]. The computations for a single model are run on a GeForce GTX 1080Ti GPU.
5 Results and Analysis
In this section, we detail the performances of the proposed and baseline models. We present the results of a series of experiments to demonstrate the effectiveness of the proposed model.
5.1 Method Comparison
Table 2 lists the results of the proposed DocL-NER and previous approaches on the CoNLL2003, OntoNotes and CHEMDNER datasets. DocL-NER surpasses the state-ofthe-art methods on all the three datasets. The models incorporating document-level context information (GraphIE and Hier-NER) can outperform those without. Because of the novel design of our document-level refinement networks, DocL-NER can utilize not only document-level context information, but also document-level label consistency. Hence, DocL-NER obtains significant improvement compared with the BiLSTM-CRF, GraphIE and Hier-NER models. Moreover, DocL-NER also outperforms the models that exploit additional task-specific resources or annotated corpora [Chiu and Nichols, 2016].
2 http://nlp.stanford.edu/projects/glove/
| Models | CoNLL2003 | OntoNotes | CHENDNER |
|---|---|---|---|
| [Chiu and Nichols, 2016]† ‡ | 91.62 | 86.34 | - |
| [Lample et al., 2016] | 90.94 | - | - |
| [Strubell et al., 2017] | 90.54 | 86.84 | - |
| [Zhang et al., 2018b] | 91.22 | - | - |
| [Ye and Ling, 2018] | 91.26 | - | - |
| [Zhang et al., 2018a] | 91.43 | - | - |
| [Chen et al., 2019] | 91.44 | 87.67 | - |
| [Cui and Zhang, 2019] | - | 88.16 | - |
| BiLSTM-CRF | 91.21 | 86.99 | 89.45 |
| GraphIE | 91.74 | 87.43∗ | 89.71 |
| Hier-NER | 91.96 | 87.98 | 89.53∗ |
| DocL-NER | 92.13 | 88.49 | 90.72 |
| Models | F1 |
|---|---|
| BERT-base [Devlin et al., 2019] | 91.82∗ |
| BERT-base + DocL-NER | 92.92 |
| ELMo [Peters et al., 2018] | 92.64∗ |
| ELMo + DocL-NER | 93.05 |
We also use the pretrained models (BERT, ELMo) to replace the default embeddings. The results are shown in Table 3. Although BERT and ELMo can model much more powerful context representations, their inability to model document-level label consistency still leads to insufficient performance. We find that by adding our method on the BERT and ELMo embeddings, the F1 score on the CoNLL2003 dataset further improves 1.1% and 0.41%, respectively.
5.2 Efficiency Advantage
Table 4 shows a comparison of training and inference speeds on CoNLL2003 dataset, as well as accuracy on co-occurrence entities. In term of speed, DocL-NER outperforms baseline models by a large margin, because we use the multi-layer Transformer to fully exploit the GPU's parallelism instead of CRF. In term of advantage on co-occurrence entities, For the baseline models, they use CRF as decoder. In contrast, our method performs attention over document-level label embeddings derived from the memory network to explicitly model the co-occurrence relationship. Hence, our model can achieve better performance on the co-occurrence entity recognition.
3 https://github.com/fastnlp/fastNLP
| Models | Train | Inference | Co-Acc |
|---|---|---|---|
| Hier-NER | 1.00x | 1.00x | 92.74 |
| GraphIE | 1.78x | 2.86x | 93.05 |
| BiLSTM-CRF | 2.22x | 4.76x | 91.33 |
| DocL-NER | 2.64x | 5.48x | 93.36 |
| Models | CoNLL2003 | OntoNotes | CHEMDNER |
|---|---|---|---|
| DocL-NER | 92.13 | 88.49 | 90.72 |
| - document-level label | 91.81 | 88.20 | 90.36 |
| - document-level context | 91.46 | 88.00 | 90.12 |
| - both | 91.36 | 87.76 | 89.83 |
| - sentence-level label | 91.63 | 88.05 | 90.21 |
| + CRF | 92.05 | 88.28 | 90.69 |
| - refinement stage | 90.83 | 87.09 | 89.43 |
5.3 Ablation Study
To study the contribution of each component in DocL-NER, we conducted ablation experiments on the three datasets and display results in Table 5. In the first part, the results show that the model's performance is degraded when the document-level label information is removed, indicating that the previous methods that only incorporate documentlevel contextual information are insufficient to model the dependency between labels. More notably, we discover that results can be improved more when using document-level label and document-level context together than simply adding up the separate increases, which verifies the effectiveness of the proposed model. Since we also use the Transformer to model sentence-level label dependencies instead of CRF, we further investigate its advantage. As shown in the second part of Table 5, our method outperforms CRF since we utilize Transformer with relative position embedding to capture long-term label dependency, while CRF only considers the neighboring label dependencies. Moreover, an accompanying additional benefit is a faster training and decoding speed, as we have mentioned earlier. Furthermore, when we remove the second stage, the performance drops by 1.30%, 1.31%, and 1.29% for CoNLL2003, OntoNotes, and CHEMDNER, respectively, indicating that our refinement method is useful and can yield significant improvements.
5.4 Hyper-parameters Investigation
Memory Size. The left part of figure 3 illustrates the performance of our model with respect to the maximum size of queried subset m for each word. The maximum size 10 is derived from the observation that 97.47% of the tokens belonging to an entity appeared less than 10 times in an document for the development set of CoNLL2003, with 91.65% for OntoNotes and 81.78% for CHEMDNER, respectively. We find that incorporating the first co-occurrence keyvalue information of an document can provide a performance improvement of at least 0.5% for all the three datasets. Note that different from Luo et al. [2020], we don't randomly select memories for a unique word. We let the memories in the original order as they appear in the document, since we believe that the contextual information for the place where an entity first appears in an document should be more adequate, and would be more useful in the future predictions.
Uncertainty Threshold. To investigate the influence of uncertainty threshold Γ, we analyze the performance with different uncertainty thresholds on the three datasets, as shown in Figure 3. Γ = 0 denotes that the model use all of the refined labels as final predictions. As the threshold gets larger, the performance of DocL-NER improves by reducing the negative effects on correct draft labels. However, when Γ is too large, the model mainly utilizes draft labels from the first stage as final predictions, which leads to performance degradation. These results verify our motivation that a reasonable uncertainty threshold can avoid side effects on correct draft labels.
6 Conclusion
In this work, we introduce a novel two-stage label refinement approach to handle document-level label consistency. A keyvalue memory network is used to record the context representations and draft labels. A multi-channel Transformer explicitly models document-level word and label dependencies based on the information derived from the memory network. In order to mitigate the side effects of incorrect draft labels, we use Bayesian neural networks to indicate the labels with a high probability of being wrong. In addition, the proposed method can model the relationship between labels in parallel for faster inference. The experimental results on three named entity recognition benchmarks demonstrated that the proposed method significantly outperformed the previous state-of-the-art methods.
Acknowledgements
The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by China National Key R&D Program (No. 2018YFB1005104, 2018YFC0831105, 2017YFB1002104), National Natural Science Foundation of China (No. 61751201, 61976056, 61532011), Shanghai Municipal Science and Technology Major Project (No.2018SHZDZX01), Science and Technology Commission of Shanghai Municipality Grant (No.18DZ1201000, 16JC1420401, 17JC1420200).
@inproceedings{ijcai2020p550,
title = {Leveraging Document-Level Label Consistency for Named Entity Recognition},
author = {Gui, Tao and Ye, Jiacheng and Zhang, Qi and Zhou, Yaqian and Gong, Yeyun and Huang, Xuanjing},
booktitle = {Proceedings of the Twenty-Ninth International Joint Conference on
Artificial Intelligence, {IJCAI-20}},
publisher = {International Joint Conferences on Artificial Intelligence Organization},
editor = {Christian Bessiere},
pages = {3976--3982},
year = {2020},
month = {7},
note = {Main track},
doi = {10.24963/ijcai.2020/550},
url = {https://doi.org/10.24963/ijcai.2020/550},
}
TY - CPAPER TI - Leveraging Document-Level Label Consistency for Named Entity Recognition AU - Gui, Tao AU - Ye, Jiacheng AU - Zhang, Qi AU - Zhou, Yaqian AU - Gong, Yeyun AU - Huang, Xuanjing T2 - Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20 PB - International Joint Conferences on Artificial Intelligence Organization PY - 2020 SP - 3976 EP - 3982 DO - 10.24963/ijcai.2020/550 UR - https://doi.org/10.24963/ijcai.2020/550 ER -Copied
Machine-readable versions of this paper, for tools and AI agents:
