Attention as Relation: Learning Supervised Multi-head Self-Attention for Relation Extraction

1 Introduction

Joint entity and relation extraction is an important and challenging task in information extraction. Given an unstructured text, this task aims to identify relation triplets consisting of two entities and their semantic relation, such as (Cambodia, Capital, Phnom Penh) in Figure 1. As the fundamental task for building knowledge bases, this task has attracted widespread attention.

Traditional methods [Zelenko et al., 2003; Mintz et al., 2009; Chan and Roth, 2011] divide this task into two isolated subtasks, including entity recognition and relation classification, and solve them in a pipeline manner. Although these methods are flexible, they suffer from error propagation and ignore the relevance between the two subtasks.

To avoid error propagation, most recent studies [Miwa and Bansal, 2016; Zheng et al., 2017; Zeng et al., 2018] are dedicated to identifying entities together with their semantic relations in a joint manner and have achieved great progress.

∗Corresponding author.

Despite the progress, the complicated relation structures still pose great challenges for this task. First, it is challenging to identify overlapping relation triplets. As shown in Figure 1, given a sentence, there may exist EntityPairOverlap triplets where two entities have multiple relations and SingleEntityOverlap triplets where two relation triplets share an overlapped entity. Early joint methods [Miwa and Bansal, 2016; Zheng et al., 2017] fail to extract them due to the relation classifiers, which suppose that a token only belongs to one relation. Second, the characteristic of entity semantics under different relation types should be considered. For example, as shown in Figure 1, the location semantics should be captured when the model predicts the contains relation for the entity pair (Cambodia, Phnom Penh), while the semantics of administrative function should be learned when the capital relation is predicted for the above entity pair. However, existing studies [Sun et al., 2019; Zeng et al., 2019a; Nayak and Ng, 2019] generally utilize a standard classifier to detect relation types, which neglects the semantic changes and leads to unideal relation triplet extraction. Third, it is crucial to recognize multiple possible relation types for an EntityPairOverlap triplet accurately. For instance, as shown in Figure 1, the entity pair (Cambodia, Phnom Penh) has three relations, including contains, capital, and administrative divisions. Thus, the two entities should have high correlation intensity under all three relation types. But with a multi-class classifier, all the possible relation types share the same probability space, which makes them are essentially mutually exclusive. Although the post-processing can be used to obtain the EntityPairOverlap triplets [Fu et al., 2019], the correlation intensity and discrimination for different relation types will decrease. In this work, we argue that this problem should be treated as a multi-label classification task, instead of a simple multi-class classification task.

To address the aforementioned challenges, we propose a simple but effective attention-based joint model mainly consisting of an entity extraction module and a relation detection module. For entity extraction, we treat it as a sequence labeling task and adopt Conditional Random Field (CRF) [Lafferty et al., 2001] to recognize the entity boundary. For relation detection, we regard it as a multi-label classification task and design a supervised multi-head self-attention mechanism. By mapping each relation type to a subspace of the multiple heads, the distinctive token-level correlation semantics for each relation can be learned, and the correlation intensity under different relation types can be calculated separately. Meanwhile, the interaction between different relation types can be captured by sharing the same input representations of multi-head self-attention. Finally, we infer the triplets based on the two modules with a given threshold. To verify the effectiveness of our model, we make a comprehensive and comparative analysis on two benchmark datasets, and the results demonstrate that our model achieves state-of-the-art performances. In summary, our contributions are three-fold:

• We propose a supervised attention-based joint model 1 , which can flexibly identify the overlapping triplets with the entire entity boundary.
• To adequately capture the distinctive correlation semantics and separately learn the correlation intensity under different relation types, we transform the relation detection into a multi-label classification task and design a supervised multi-head self-attention mechanism.
• Extensive experiments are conducted on two benchmark datasets, and the results show that our model achieves state-of-the-art performances with 1.3% and 14.2% improvements, respectively.

2 RelatedWork

Traditional approaches [Zelenko et al., 2003; Miwa et al., 2009; Mintz et al., 2009; Chan and Roth, 2011; Zeng et al., 2015; Shen and Huang, 2016] generally deal with joint entity and relation extraction task in a pipeline manner, which identifies the entities first and then predicts the relations between them. Because these methods treat entity recognition and relation classification as two isolated steps, they suffer from error propagation.

To consider the interaction between the two steps, Miwa and Bansal [2016], Gupta et al. [2016], Zhang et al. [2017], Zheng et al. [2017], and Sun et al. [2019] jointly extracted entities and relations in a unified framework. But these methods can not precisely identify the overlapping triplets because they assume that a token or an entity pair only belongs to one relation.

Recently, many studies focus on predicting overlapping triplets while considering the interactions between relations. Fu et al. [2019] proposed a two-phase joint model based on graph convolutional network (GCN). Takanobu et al. [2019] introduced a hierarchical reinforcement learning framework, which includes a high-level policy for relation detection and a low-level policy for entity extraction. Dai et al. [2019] designed a novel tagging scheme and proposed a positionattention mechanism to identify overlapping relations. Meanwhile, the seq2seq framework is also utilized to predict the overlapping triplets [Zeng et al., 2018; Zeng et al., 2019b; Zeng et al., 2019a; Nayak and Ng, 2019]. However, these methods fail to treat different relation types as distinctive subspaces and calculate the correlation degree separately under different relation types. Besides, the seq2seq based models generally suffer from inaccurate entity boundary recognition.

1 https://github.com/NKU-IIPLab/SMHSA

To deal with the above issues, we utilize CRF to recognize the entity boundary and design a supervised multi-head selfattention mechanism to flexibly extract overlapping relations via treating each relation type as an isolated subspace.

3 Model

Given a sentence X = {x1, x2, ..., xN} with N tokens, joint entity and relation extraction task aims to identify a collection of relation triplets T = [(fl, rl, sl)]Ll=1 fromX, where fl, sl, and rl represent the first entity, the second entity, and their relation, respectively. Note that the entities are extracted from the given sentence, and the relations are selected from a predefined set R = {R1, R2, ..., RM} with M types.

To deal with this task, three issues should be considered, including detecting the overlapping relations, learning the characteristic correlation semantics for each relation type, and calculating the correlation intensity for each relation type independently. To this end, a supervised attention-based joint model is proposed in this paper, and the framework of our model is illustrated in Figure 2. Concretely, our model consists of an encoding layer, an entity extraction module, and a relation detection module. We adopt CRF as the entity extraction module to recognize entities and devise a supervised multi-head self-attention mechanism as the relation detection module to learn the fine-grained correlation between tokens for each relation type. Finally, we fuse the predicted results from the two modules and infer triplets via a given threshold.

3.1 Encoding Layer

Given a sentence X, we first utilize bidirectional long shortterm memory (BLSTM) network [Hochreiter and Schmidhuber, 1997] to encode the contextualized representation for each token. Formally, the initial embedding ei of each token is calculated by concatenating the word embedding ewi ∈ Rdw and the character-level morphology feature 2 eci ∈ Rdc , where dw and dc are the dimensions of word embedding and morphology feature, respectively. And then, the contextualized representation sequence H = {h1, h2, ..., hN} is obtained as follows:

where LSTMf and LSTMb denote the forward and backward LSTM, respectively.

2 We also adopt BLSTM to capture the morphology features.

ଵݔ

ଶݔ

ଷݔ

3.2 Entity ExtractionModule

To recognize the entity boundary accurately, we treat entity extraction as a sequence labeling task and adopt Conditional Random Field (CRF) [Lafferty et al., 2001] as the entity extraction module. Formally, CRF utilizes a state score matrix P ∈ RN×k to model the mappings between tokens and labels. Meanwhile, a transition score matrix V ∈ Rk×k is used to learn the dependency between adjacent labels, where k denotes the dimension of the label space 3 . For a sequence of predicted labels ˆ Y = {y1, y2, ..., yN}, we define its score as follows:

where U = {u1, u2, ..., uN} denotes the input hidden representation sequence for the entity extraction module, which is calculated from the context representation sequence H via a fully-connection layer. The matrices Wp ∈ Rdu×k and bp ∈ RN×k are model parameters, where du denotes the dimension of the input hidden representations in U.

Then, the probability of label sequence ˆ Y can be calculated as follows:

where YX denotes all possible label sequences.

During training, we aim to maximize the likelihood probability p (Y | X) of the gold label sequence Y . Thus, we minimize the negative log-likelihood loss function to optimize the parameters as follows:

During decoding, we use the Viterbi algorithm to obtain the predicted label sequence with the maximum score.

3 Following the BIO tagging scheme, we define three labels, including B (beginning of entity), I (inside of entity), and O (others).

ேݔ

3.3 Relation DetectionModule

To flexibly predict the overlapping triplets, we transform the relation detection into a multi-label classification task and devise a supervised multi-head self-attention mechanism to deal with it by regarding each relation type as an isolated subspace. Specially, we calculate the attention at token-level, which can help our model learn more fine-grained correlation semantics. Formally, we first couple a fully-connection layer upon the encoding layer to obtain the input hidden representation sequence A = {a1, a2, ..., aN} for the relation detection module:

where Wa ∈ Rdh×da and ba ∈ RN×da are model parameters, while dh and da are the dimensions of the contextualized representations in H and the input hidden representations in A, respectively. By sharing the input hidden representations, the interaction between different relation types can be captured.

Considering that the characteristic correlation semantics should be learned and the correlation degree should be calculated independently for each relation type, we regard each relation type as a subspace and project the input hidden representations to different relation subspaces as follows:

where Qm ∈ RN×dr and Km ∈ RN×dr denote the queries and keys for the m-th relation type. The matrices W m Q ∈ Rda×dr and W m K ∈ Rda×dr are model parameters, where dr is the dimension of each relation subspace.

Then, we calculate the attention matrix Gm ∈ RN×N whose element Gmi,j denotes the correlation intensity between the i-th token and the j-th token under the m-th relation type:

To guide this module to detect relation types, we further introduce supervision information by maximizing the likelihood probability as follows:

where Z m i,j = 1 denotes the fact that the m-th relation exists between the i-th token and the j-th token, and vice versa. To transform the relation detection into a multi-label classification task, we convert the gold annotation to a one-hot matrix during training and minimize the binary cross-entropy loss between the predicted distribution ˆ p (Z m i,j |xi, xj ) and the gold distribution p (Z m i,j |xi, xj ) as follows:

With the supervision information, the multi-head selfattention can be guided to detect relations more effectively.

3.4 Joint Learning

To synchronously learn the proposed two modules and make themmutually improve, we combine the loss functions of the two modules to form the entire loss objective of our model:

The optimization problem in Eq. (13) can be solved by using any gradient descent approach. In this paper, we adopt the RMSprop [Ruder, 2016] approach.

3.5 Inference

Based on the two modules, the triplets can be easily inferred. Concretely, with the label sequence ˆ Y predicted by the entity extraction module, we can obtain the entity set E = {e1, e2, ..., eLE } with LE entities. Then, given the i-th entity ei = {xpi , ..., xqi } and the j-th entity ej = {xpj , ..., xqj }, the correlated intensity δ between them on the m-th relation can be calculated based on the weight matrix G from the relation detection module:

where |ei| is the length of entity ei. The triplet 〈ei,Rm, ej〉 is extracted if δ is higher than a given threshold ˆ δ.

4 Experiments

4.1 Datasets

To verify the effectiveness of our model, we conduct extensive experiments on two benchmark datasets, including New York Times (NYT) [Riedel et al., 2010] and WebNLG [Gardent et al., 2017]. The NYT contains 24 types of relations, and the WebNLG has 246 relation types. For NYT, we follow the dataset used in [Zeng et al., 2018]. For WebNLG, we select the sentences containing the most triples in each instance and discard the instances if all triplets are not found in the corresponding sentences. To construct the development set, we randomly select 10% samples from the training set. The statistics of the above datasets are shown in Table 1.

4.2 Experimental Settings

We initialize the word embeddings with pre-trained Glove 840B vectors 4 [Pennington et al., 2014] and randomly initialize the character embeddings with 50 dimensions. The dimensions of hidden states for character LSTM, encoding layer, entity extraction module, and relation extraction module are set to 100, 600, 250, 250, respectively. For the relation extraction module, the head number is the same as the number of relation types, and the dimension of each head is set to 30. During training, we use the RMSprop optimizer [Ruder, 2016]. The learning rate, learning rate decay, and batch size are set to 0.001, 0.95, and 10, respectively. To ensure the balance between entity extraction and relation detection, we adopt an iterative two-step training manner where one training step optimizes the total parameters, and another only optimizes the parameters of the relation detection module and the encoding layer. To avoid overfitting, we apply dropout at a rate of 0.3.

4 https://nlp.stanford.edu/projects/glove/

4.3 Evaluation

We adopt precision, recall, and standard micro-F1 score to evaluate the performances. Specifically, a predicted triplet (f, r, s) is correct only if the relation type and the two corresponding entities are all the same as the golden standard annotation. We report the corresponding results of the test set when the development set achieves the best result.

4.4 ComparisonMethods

To achieve the comprehensive and comparative analysis of our model, we compare it with a series of advanced models:

• NovelTagging [Zheng et al., 2017] introduces a novel tagging scheme which transforms the joint extraction task into a sequence labeling problem.
• CopyRe [Zeng et al., 2018] is a seq2seq model with copy mechanism, which can effectively extract overlapping triplets. We report the results of the MultiDecoder.
• GraphRel [Fu et al., 2019] is a two phases model based on GCN, where a relation-weighted GCN is utilized to model the interaction between entities and relations.
• AntNRE [Sun et al., 2019] decomposes the joint extraction task into entity span detection subtask and entity relation type detection subtask. Specially, the second subtask is tackled based on an entity-relation bipartite.
• MultiRe [Zeng et al., 2019b] applies the reinforcement learning into a seq2seq model to automatically learn the extraction order of triplets, where the interactions among triplets can be considered.
• CopyMTL [Zeng et al., 2019a] is a multi-task learning framework, where CRF is used to identify entities, and a seq2seq model is adopted to extract relation triplets.
• WDec [Nayak and Ng, 2019] designs a new representation scheme and utilizes a seq2seq model to generate triplets with the entire boundaries.

Besides, we also conduct several ablation experiments, including "Ours w/o MHSA" which uses a standard multiclass classifier to detect relations, "Ours w/o IT" which does not utilize the iterative two-step training manner, and "Ours w/o ALL" which does not adopt the multi-head self-attention mechanism and the iterative training manner.

datasets, while the results on the seq2seq models are worst. The results also show that the lower F1-score on entity extraction limits the performance of the seq2seq model.

4.6 Ablation Study

To prove the effectiveness of the proposed relation detection module, we conduct the ablation study. As shown in the second block of Table 2, the performance of "Ours w/o MHSA" decreases significantly on NYT, while it is slightly worse on WebNLG compared with our model. The reason is that the EntityPairOverlap and SingleEntityOverlap triplets of NYT are significantly more than them of WebNLG, which demonstrates that the proposed supervised multi-head self-attention mechanism can effectively identify overlapping triplets by projecting each relation type to a separate subspace.

And we also analyze the effectiveness of the iterative twostep training manner. Compared with our model, the F1-score of "Ours w/o IT" decreases 3.5% and 2.9% on two datasets, respectively. This shows that the imbalance between entity

4.5 Results

The results on relation triplet extraction are shown in Table 2. According to the results, our model consistently obtains stateof-the-art performances on two datasets. Compared with the best baseline model, our model outperforms WDec by 1.3% F1-score on NYT and is higher than AntNRE by 14.2% F1score on WebNLG, respectively. Specially, WDec [Nayak and Ng, 2019] adopts the seq2seq model to generate relation triplets and removes the duplicate triplets and fragmentary triplets via post-processing. Therefore, WDec achieves high precision on two datasets, while its recall is unsatisfactory. It is worth to note that our model achieves the highest precision and recall without any post-processing.

Furthermore, the performances of the seq2seq models are generally better than other baselines because they can identify overlapping relations more flexibly by decoding different triplets one by one. However, compared with other baselines, the seq2seq models fail to precisely recognize the entity boundary. Thus, the longer the length of entity is, the worse these models perform. Specially, the F1-score of the best seq2seq model WDec is inferior to AntNRE on the WebNLG dataset whose max entity length is 15 and the average entity length is 2.2. Note that AntNRE can not deal with the EntityPairOverlap triplets, so it performs poorly on the NYT dataset which contains more complex relations. In contrast, our model can effectively capture the overlapping triplets along with the correct entity boundary.

Besides, we also investigate the performance of entity extraction, and the results are shown in Table 3. It is clearly shown that our model achieves state-of-the-art results on two extraction and relation detection can severely impact the performance of triplet extraction. Without the above two components, "Ours w/o ALL" performs worst.

4.7 Analysis of Different Sentence Types

To further verify the ability of our model to handle complex triplets, we analyze the results on different sentence types. Figure 3 shows the results on Normal, EntityPairOverlap, and SingleEntityOverlap types. Our model achieves the best results on all three types, while the performances of AntNRE, "Ours w/o MHSA" and "Ours w/o ALL" are the worst on EntityPairOverlap and SingleEntityOverlap for NYT dataset. This shows that it is difficult to deal with the overlapping triplets if the relation detection is treated as a multi-class problem. For WebNLG, WDec performs poorly on all types due to the longer entity length.

Besides, we also evaluate the performance on different triplet numbers, and the results are shown in Figure 4. Following Zeng et al. [2018], we divide the sentence into 5 classes, and each class consists of sentences with 1, 2, 3, 4, or ≥ 5 triplets, respectively. According to the results, our model achieves more stable performance as the number of triples increases, which shows that our model is more robust when it is faced with the complicated relation situation.

4.8 Analysis of Inference Threshold

We also report the effect of different thresholds in Figure 5. As the threshold increases, the recall of our model gradually decreases, and the precision gradually increases. Thus, we can flexibly select a suitable threshold according to the actual demand. Our model has a low precision on WebNLG dataset when the threshold is lower than 0.6, because the prediction process is more difficult when entities are longer. The F1score first increases and then decreases when the threshold increases, and the results show that our model achieves the best performances on NYT and WebNLG datasets with 0.6 and 0.8 thresholds. Due to the longer sentence length of NYT, the best threshold of NYT is less than WebNLG. In general, our model identifies triplets with high confidence.

5 Conclusion

In this paper, we focused on joint entity and relation extraction task and proposed a simple but very effective attentionbased joint model. With the supervised multi-head selfattention mechanism, the relation detection module can flexibly detect EntityPairOverlap and SingleEntityOverlap relations by regarding each relation type as a subspace and maintaining the independence between them. Meanwhile, the entity extraction module can accurately recognize the entity boundary with CRF. Finally, we inferred the triplets based on the results of the two modules. Extensive experiments showed that our model achieves state-of-the-art performances.

Acknowledgments

This research is supported by the National Natural Science Foundation of China under grant No. 61976119, the Natural Science Foundation of Tianjin under grant No. 18JCYBJC15800, and the Major Program of Science and Technology of Tianjin under grant No. 18ZXZNGX00310.