{"type":"ScholarlyArticle","title":"Position-aware Joint Entity and Relation Extraction with Attention Mechanism","authors":[{"name":"Chenglong Zhang","slug":"chenglong-zhang","given":"Chenglong","family":"Zhang","affiliation":"Shanghai Key Laboratory of Intelligent Information Processing , School of Computer Science , Fudan University , Shanghai , China"},{"name":"Shuyong Gao","slug":"shuyong-gao","given":"Shuyong","family":"Gao","affiliation":"Shanghai Key Laboratory of Intelligent Information Processing , School of Computer Science , Fudan University , Shanghai , China"},{"name":"Haofen Wang","slug":"haofen-wang","given":"Haofen","family":"Wang","affiliation":"College of Design and Innovation , Tongji University , Shanghai , China"},{"name":"Wenqiang Zhang","slug":"wenqiang-zhang","given":"Wenqiang","family":"Zhang","affiliation":"Shanghai Key Laboratory of Intelligent Information Processing , School of Computer Science , Fudan University , Shanghai , China; Academy for Engineering & Technology , Fudan University , Shanghai , China"}],"abstract":"Named entity recognition and relation extraction are two important core subtasks of information extraction, which aim to identify named entities and extract relations between them. In recent years, span representation methods have received a lot of attention and are widely used to extract entities and corresponding relations from plain texts. Most recent works focus on how to obtain better span representations from pre-trained encoders, but ignore the negative impact of a large number of span candidates on slowing down the model performance. In our work, we propose a joint entity and relation extraction model with an attention mechanism and position-attentive markers. The attention score of each candidate span is calculated, and most of the candidate spans with low attention scores are pruned before being fed into the span classifier, thus achieving the goal of removing the most irrelevant spans. At the same time, in order to explore whether the position information can improve the performance of the model, we add position-attentive markers to the model. The experimental results show that our model is effective. With the same pre-trained encoder, our model achieves the new state-of-the-art on standard benchmarks (ACE05, CoNLL04 and SciERC), obtaining a 4.7%-17.8% absolute improvement in relation F1.","doi":"10.24963/ijcai.2022/624","url":"https://ijcai.org/proceedings/2022/624","pdf":"https://ijcai.org/proceedings/2022/0624.pdf","markdown":"https://ijcai.org/proceedings/2022/624.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/624","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":624,"pages":{"first":4496,"last":4502},"keywords":["Natural Language Processing: Information Extraction","Natural Language Processing: Language Models","Natural Language Processing: Knowledge Extraction","Natural Language Processing: Named Entities","Data Mining: Knowledge Graphs and Knowledge Base Completion"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986689","length_sec":78,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984832","length_sec":265,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bekoulis et al., 2018a] Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. Adversarial train- ing for multi-context joint entity and relation extraction. arXiv preprint arXiv:1808.06876, 2018.","title":"Adversarial training for multi-context joint entity and relation extraction","authors":"Bekoulis","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bekoulis et al., 2018b] Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. Joint entity recognition and relation extraction as a multi-head selec- tion problem. Expert Systems with Applications, 2018.","title":"Joint entity recognition and relation extraction as a multi-head selection problem","authors":"Bekoulis","year":"2018","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Beltagy et al., 2019] Iz Beltagy, Kyle Lo, and Arman Co- han. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676, 2019.","title":"Scibert: A pretrained language model for scientific text","authors":"Beltagy","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dixit and Al-Onaizan, 2019] Kalpit Dixit and Yaser Al- Onaizan. Span-level model for relation extraction. In Pro- ceedings of ACL, pages 5308-5314, 2019.","title":"Kalpit Dixit and Yaser Al-Onaizan. Span-level model for relation extraction","authors":"Al-Onaizan Dixit","year":"2019","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Eberts and Ulges, 2019] Markus Eberts and Adrian Ulges. Span-based joint entity and relation extraction with trans- former pre-training. arXiv preprint arXiv:1909.07755, 2019.","title":"Span-based joint entity and relation extraction with transformer pre-training","authors":"Ulges Eberts, Markus Eberts, Adrian Ulges","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fu et al., 2019] Tsu-Jui Fu, Peng-Hsuan Li, and Wei-Yun Ma. Graphrel: Modeling text as relational graphs for joint entity and relation extraction. In Proceedings of the 57th Annual Meeting of ACL, pages 1409-1418, 2019.","title":"Graphrel: Modeling text as relational graphs for joint entity and relation extraction","authors":"Tsu-Jui Fu, Peng-Hsuan Li, Wei-Yun Ma","year":"2019","venue":"Proceedings of the 57th Annual Meeting of ACL","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gupta et al., 2016] Pankaj Gupta, Hinrich Schütze, and Bernt Andrassy. Table filling multi-task recurrent neural network for joint entity and relation extraction. In Pro- ceedings of COLING 2016, pages 2537-2547, 2016.","title":"Table filling multi-task recurrent neural network for joint entity and relation extraction","authors":"Pankaj Gupta, Hinrich Schütze, Bernt Andrassy","year":"2016","venue":"Proceedings of COLING 2016","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2018] Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. Jointly predicting predicates and ar- guments in neural semantic role labeling. arXiv preprint arXiv:1805.04787, 2018.","title":"Jointly predicting predicates and arguments in neural semantic role labeling","authors":"He","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ji et al., 2020] Bin Ji, Jie Yu, Shasha Li, Jun Ma, Qingbo Wu, Yusong Tan, and Huijun Liu. Span-based joint entity and relation extraction with attention-based span-specific and contextual semantic representations. In Proceedings of COLING, pages 88-99, 2020.","title":"Span-based joint entity and relation extraction with attention-based span-specific and contextual semantic representations","authors":"Bin Ji, Jie Yu, Shasha Li, Jun Ma, Qingbo Wu, Yusong Tan, Huijun Liu","year":"2020","venue":"Proceedings of COLING","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Katiyar and Cardie, 2017] Arzoo Katiyar and Claire Cardie. Going out on a limb: Joint extraction of entity mentions and relations without dependency trees. In Proceedings of ACL, pages 917-928, 2017.","title":"Going out on a limb: Joint extraction of entity mentions and relations without dependency trees","authors":"Cardie ; ] Arzoo Katiyar, Claire Katiyar, Cardie","year":"2017","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lan et al., 2019] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of lan- guage representations. ICLR, 2019.","title":"Albert: A lite bert for self-supervised learning of language representations","authors":"Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lee et al., 2017] Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. End-to-end neural coreference resolu- tion. arXiv preprint arXiv:1707.07045, 2017.","title":"End-to-end neural coreference resolution","authors":"Kenton Lee, Luheng He, Mike Lewis, Luke Zettlemoyer","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lee et al., 2018] Kenton Lee, Luheng He, and Luke Zettle- moyer. Higher-order coreference resolution with coarse- to-fine inference. arXiv preprint arXiv:1804.05392, 2018.","title":"Higher-order coreference resolution with coarseto-fine inference","authors":"Kenton Lee, Luheng He, Luke Zettlemoyer","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li and Ji, 2014] Qi Li and Heng Ji. Incremental joint ex- traction of entity mentions and relations. In Proceedings of ACL, pages 402-412, 2014.","title":"Incremental joint extraction of entity mentions and relations","authors":"Ji ; ] Qi Li, Heng Li, Ji","year":"2014","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2019] Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li. Entity-relation extraction as multi-turn question answer- ing. arXiv preprint arXiv:1905.05529, 2019.","title":"Entity-relation extraction as multi-turn question answering","authors":"Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, Jiwei Li","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2020] Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. A joint neural model for information extraction with global features. In Proceedings of ACL, 2020.","title":"A joint neural model for information extraction with global features","authors":"Ying Lin, Heng Ji, Fei Huang, Lingfei Wu","year":"2020","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Luan et al., 2018] Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. Multi-task identification of en- tities, relations, and coreference for scientific knowledge graph construction. EMNLP, 2018.","title":"Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction","authors":"Yi Luan, Luheng He, Mari Ostendorf, Hannaneh Hajishirzi","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Luan et al., 2019] Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. A general framework for information extraction using dy- namic span graphs. NAACL, 2019.","title":"A general framework for information extraction using dynamic span graphs","authors":"Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, Hannaneh Hajishirzi","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Miwa and Bansal, 2016] Makoto Miwa and Mohit Bansal. End-to-end relation extraction using lstms on sequences and tree structures. ACL, 2016.","title":"End-to-end relation extraction using lstms on sequences and tree structures","authors":"Bansal Miwa, Makoto Miwa, Mohit Bansal","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Miwa and Sasaki, 2014] Makoto Miwa and Yutaka Sasaki. Modeling joint entity and relation extraction with table representation. In Proceedings of EMNLP, 2014.","title":"Modeling joint entity and relation extraction with table representation","authors":"Sasaki Miwa, Makoto Miwa, Yutaka Sasaki","year":"2014","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nguyen and Verspoor, 2019] Dat Quoc Nguyen and Karin Verspoor. End-to-end neural relation extraction using deep biaffine attention. In ECIR. Springer, 2019.","title":"Dat Quoc Nguyen and Karin Verspoor. End-to-end neural relation extraction using deep biaffine attention","authors":"Verspoor Nguyen","year":"2019","venue":"ECIR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ouchi et al., 2018] Hiroki Ouchi, Hiroyuki Shindo, and Yuji Matsumoto. A span selection model for semantic role labeling. arXiv preprint arXiv:1810.02245, 2018.","title":"A span selection model for semantic role labeling","authors":"Hiroki Ouchi, Hiroyuki Shindo, Yuji Matsumoto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tran and Kavuluru, 2019] Tung Tran and Ramakanth Kavu- luru. Neural metric learning for fast end-to-end relation extraction. arXiv preprint arXiv:1905.07458, 2019.","title":"Neural metric learning for fast end-to-end relation extraction","authors":"Tung Tran, Ramakanth Kavuluru","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wadden et al., 2019] David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. Entity, relation, and event extraction with contextualized span representations. arXiv preprint arXiv:1909.03546, 2019.","title":"Entity, relation, and event extraction with contextualized span representations","authors":"David Wadden, Ulme Wennberg, Yi Luan, Hannaneh Hajishirzi","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang and Lu, 2020] Jue Wang and Wei Lu. Two are better than one: Joint entity and relation extraction with table- sequence encoders. EMNLP, 2020.","title":"Two are better than one: Joint entity and relation extraction with tablesequence encoders","authors":"Jue Wang, Wei Lu","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ye et al., 2021] Deming Ye, Yankai Lin, and Maosong Sun. Pack together: Entity and relation extraction with levitated marker. arXiv preprint arXiv:2109.06067, 2021.","title":"Pack together: Entity and relation extraction with levitated marker","authors":"Deming Ye, Yankai Lin, Maosong Sun","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2017] Meishan Zhang, Yue Zhang, and Guo- hong Fu. End-to-end neural relation extraction with global optimization. In Proceedings of EMNLP, 2017.","title":"End-to-end neural relation extraction with global optimization","authors":"Zhang","year":"2017","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zheng et al., 2017] Suncong Zheng, Feng Wang, Hongyun Bao, Yuexing Hao, Peng Zhou, and Bo Xu. Joint ex- traction of entities and relations based on a novel tagging scheme. arXiv preprint arXiv:1706.05075, 2017.","title":"Joint extraction of entities and relations based on a novel tagging scheme","authors":"Zheng","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhong and Chen, 2020] Zexuan Zhong and Danqi Chen. A frustratingly easy approach for entity and relation extrac- tion. arXiv preprint arXiv:2010.12812, 2020.","title":"A frustratingly easy approach for entity and relation extraction","authors":"Chen ; Zhong, Zexuan Zhong, Danqi Chen","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}