{"type":"ScholarlyArticle","title":"Attention as Relation: Learning Supervised Multi-head Self-Attention for Relation Extraction","authors":[{"name":"Jie Liu","slug":"jie-liu","given":"Jie","family":"Liu","affiliation":"College of Artificial Intelligence , Nankai University , Tianjin , China"},{"name":"Shaowei Chen","slug":"shaowei-chen","given":"Shaowei","family":"Chen","affiliation":"College of Artificial Intelligence , Nankai University , Tianjin , China"},{"name":"Bingquan Wang","slug":"bingquan-wang","given":"Bingquan","family":"Wang","affiliation":"College of Artificial Intelligence , Nankai University , Tianjin , China"},{"name":"Jiaxin Zhang","slug":"jiaxin-zhang","given":"Jiaxin","family":"Zhang","affiliation":"College of Artificial Intelligence , Nankai University , Tianjin , China; Phnom Penh contains capital administrative_divisions EntityPairOverlap (EPO)"},{"name":"Na Li","slug":"na-li","given":"Na","family":"Li","affiliation":"College of Computer Science , Nankai University , Tianjin , China"},{"name":"Tong Xu","slug":"tong-xu","given":"Tong","family":"Xu","affiliation":"University of Science and Technology of China , Hefei , China; Phnom Penh Cambodia Kratie SingleEntityOverlap"}],"abstract":"Joint entity and relation extraction is critical for many natural language processing (NLP) tasks, which has attracted increasing research interest. However, it is still faced with the challenges of identifying the overlapping relation triplets along with the entire entity boundary and detecting the multi-type relations. In this paper, we propose an attention-based joint model, which mainly contains an entity extraction module and a relation detection module, to address the challenges. The key of our model is devising a supervised multi-head self-attention mechanism as the relation detection module to learn the token-level correlation for each relation type separately. With the attention mechanism, our model can effectively identify overlapping relations and flexibly predict the relation type with its corresponding intensity. To verify the effectiveness of our model, we conduct comprehensive experiments on two benchmark datasets. The experimental results demonstrate that our model achieves state-of-the-art performances.","doi":"10.24963/ijcai.2020/524","url":"https://ijcai.org/proceedings/2020/524","pdf":"https://ijcai.org/proceedings/2020/0524.pdf","markdown":"https://ijcai.org/proceedings/2020/524.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/524","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":524,"pages":{"first":3787,"last":3793},"keywords":["Natural Language Processing: Natural Language Processing","Natural Language Processing: Information Extraction"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24909/file_24909.m3u8","length_sec":258,"provider":"hls"}],"references":[{"position":1,"raw":"[Chan and Roth, 2011] Yee Seng Chan and Dan Roth. Ex- ploiting syntactico-semantic structures for relation extrac- tion. In ACL 2011, pages 551-560, 2011.","title":"Exploiting syntactico-semantic structures for relation extraction","authors":"Yee Seng, Chan, Dan Roth","year":"2011","venue":"ACL 2011","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dai et al., 2019] Dai Dai, Xinyan Xiao, Yajuan Lyu, Shan Dou, Qiaoqiao She, and Haifeng Wang. Joint extraction of entities and overlapping relations using position-attentive sequence labeling. In AAAI 2019, pages 6300-6308, 2019.","title":"Joint extraction of entities and overlapping relations using position-attentive sequence labeling","authors":"Dai Dai, Xinyan Xiao, Yajuan Lyu, Shan Dou, Qiaoqiao She, Haifeng Wang","year":"2019","venue":"AAAI 2019","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fu et al., 2019] Tsu-Jui Fu, Peng-Hsuan Li, and Wei-Yun Ma. Graphrel: Modeling text as relational graphs for joint entity and relation extraction. In ACL 2019, pages 1409- 1418, 2019.","title":"Graphrel: Modeling text as relational graphs for joint entity and relation extraction","authors":"Tsu-Jui Fu, Peng-Hsuan Li, Wei-Yun Ma","year":"2019","venue":"ACL 2019","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gardent et al., 2017] Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. Creating training corpora for NLG micro-planners. In ACL 2017, pages 179-188, 2017.","title":"Creating training corpora for NLG micro-planners","authors":"Claire Gardent, Anastasia Shimorina, Shashi Narayan, Laura Perez-Beltrachini","year":"2017","venue":"ACL 2017","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gupta et al., 2016] Pankaj Gupta, Hinrich Schütze, and Bernt Andrassy. Table filling multi-task recurrent neural network for joint entity and relation extraction. In COL- ING 2016, pages 2537-2547, 2016.","title":"Table filling multi-task recurrent neural network for joint entity and relation extraction","authors":"Pankaj Gupta, Hinrich Schütze, Bernt Andrassy","year":"2016","venue":"COL-ING 2016","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lafferty et al., 2001] John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML 2001, pages 282-289, 2001.","title":"Conditional random fields: Probabilistic models for segmenting and labeling sequence data","authors":"John D Lafferty, Andrew Mccallum, Fernando C N Pereira","year":"2001","venue":"ICML 2001","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Mintz et al., 2009] Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. Distant supervision for relation ex- traction without labeled data. In ACL 2009, pages 1003- 1011, 2009.","title":"Distant supervision for relation extraction without labeled data","authors":"Mike Mintz, Steven Bills, Rion Snow, Daniel Jurafsky","year":"2009","venue":"ACL 2009","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Miwa and Bansal, 2016] Makoto Miwa and Mohit Bansal. End-to-end relation extraction using lstms on sequences and tree structures. In ACL 2016, 2016.","title":"End-to-end relation extraction using lstms on sequences and tree structures","authors":"Bansal Miwa, Makoto Miwa, Mohit Bansal","year":"2016","venue":"ACL 2016","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Miwa et al., 2009] Makoto Miwa, Rune Sætre, Yusuke Miyao, and Jun'ichi Tsujii. A rich feature vector for protein-protein interaction extraction from multiple cor- pora. In EMNLP 2009, pages 121-130, 2009.","title":"A rich feature vector for protein-protein interaction extraction from multiple corpora","authors":"Makoto Miwa, Rune Sætre, Yusuke Miyao, Jun'ichi Tsujii","year":"2009","venue":"EMNLP 2009","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Nayak and Ng, 2019] Tapas Nayak and Hwee Tou Ng. Ef- fective modeling of encoder-decoder architecture for joint entity and relation extraction. CoRR, abs/1911.09886, 2019.","title":"Effective modeling of encoder-decoder architecture for joint entity and relation extraction","authors":"Ng ; ] Tapas Nayak, Hwee Tou Nayak, Ng","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In EMNLP 2014, pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"EMNLP 2014","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Riedel et al., 2010] Sebastian Riedel, Limin Yao, and An- drew McCallum. Modeling relations and their mentions without labeled text. In Machine Learning and Knowledge Discovery in Databases, European Conference, ECML PKDD 2010, Barcelona, Spain, September 20-24, 2010, Proceedings, Part III, pages 148-163, 2010.","title":"Modeling relations and their mentions without labeled text","authors":"Sebastian Riedel, Limin Yao, Andrew Mccallum","year":"2010","venue":"Machine Learning and Knowledge Discovery in Databases, European Conference, ECML PKDD 2010","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ruder, 2016] Sebastian Ruder. An overview of gradient de- scent optimization algorithms. CoRR, abs/1609.04747, 2016.","title":"An overview of gradient descent optimization algorithms","authors":"Sebastian Ruder","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Shen and Huang, 2016] Yatian Shen and Xuanjing Huang. Attention-based convolutional neural network for seman- tic relation extraction. In COLING 2016, pages 2526- 2536, 2016.","title":"Attention-based convolutional neural network for semantic relation extraction","authors":"Huang Shen, Yatian Shen, Xuanjing Huang","year":"2016","venue":"COLING 2016","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sun et al., 2019] Changzhi Sun, Yeyun Gong, Yuanbin Wu, Ming Gong, Daxin Jiang, Man Lan, Shiliang Sun, and Nan Duan. Joint type inference on entities and relations via graph convolutional networks. In ACL 2019, pages 1361- 1370, 2019.","title":"Joint type inference on entities and relations via graph convolutional networks","authors":"Changzhi Sun, Yeyun Gong, Yuanbin Wu, Ming Gong, Daxin Jiang, Man Lan, Shiliang Sun, Nan Duan","year":"2019","venue":"ACL 2019","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Takanobu et al., 2019] Ryuichi Takanobu, Tianyang Zhang, Jiexi Liu, and Minlie Huang. A hierarchical framework for relation extraction with reinforcement learning. In AAAI 2019, pages 7072-7079, 2019.","title":"A hierarchical framework for relation extraction with reinforcement learning","authors":"Ryuichi Takanobu, Tianyang Zhang, Jiexi Liu, Minlie Huang","year":"2019","venue":"AAAI 2019","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Zelenko et al., 2003] Dmitry Zelenko, Chinatsu Aone, and Anthony Richardella. Kernel methods for relation extrac- tion. J. Mach. Learn. Res., 3:1083-1106, 2003.","title":"Kernel methods for relation extraction","authors":"Dmitry Zelenko, Chinatsu Aone, Anthony Richardella","year":"2003","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zeng et al., 2015] Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. Distant supervision for relation extraction via piecewise convolutional neural networks. In EMNLP 2015, pages 1753-1762, 2015.","title":"Distant supervision for relation extraction via piecewise convolutional neural networks","authors":"Daojian Zeng, Kang Liu, Yubo Chen, Jun Zhao","year":"2015","venue":"EMNLP 2015","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zeng et al., 2018] Xiangrong Zeng, Daojian Zeng, Shizhu He, Kang Liu, and Jun Zhao. Extracting relational facts by an end-to-end neural model with copy mechanism. In ACL 2018, pages 506-514, 2018.","title":"Extracting relational facts by an end-to-end neural model with copy mechanism","authors":"Xiangrong Zeng, Daojian Zeng, Shizhu He, Kang Liu, Jun Zhao","year":"2018","venue":"ACL 2018","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zeng et al., 2019a] Daojian Zeng, Haoran Zhang, and Qianying Liu. Copymtl: Copy mechanism for joint ex- traction of entities and relations with multi-task learning. CoRR, abs/1911.10438, 2019.","title":"Copymtl: Copy mechanism for joint extraction of entities and relations with multi-task learning","authors":"Zeng","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zeng et al., 2019b] Xiangrong Zeng, Shizhu He, Daojian Zeng, Kang Liu, Shengping Liu, and Jun Zhao. Learn- ing the extraction order of multiple relational facts in a sentence with reinforcement learning. In EMNLP 2019, pages 367-377, 2019.","title":"Learning the extraction order of multiple relational facts in a sentence with reinforcement learning","authors":"Zeng","year":"2019","venue":"EMNLP 2019","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2017] Meishan Zhang, Yue Zhang, and Guo- hong Fu. End-to-end neural relation extraction with global optimization. In EMNLP 2017, pages 1730-1740, 2017.","title":"End-to-end neural relation extraction with global optimization","authors":"Zhang","year":"2017","venue":"EMNLP 2017","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zheng et al., 2017] Suncong Zheng, Feng Wang, Hongyun Bao, Yuexing Hao, Peng Zhou, and Bo Xu. Joint ex- traction of entities and relations based on a novel tagging scheme. In ACL 2017, pages 1227-1236, 2017.","title":"Joint extraction of entities and relations based on a novel tagging scheme","authors":"Zheng","year":"2017","venue":"ACL 2017","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}