{"type":"ScholarlyArticle","title":"Beyond Word Attention: Using Segment Attention in Neural Relation Extraction","authors":[{"name":"Bowen Yu","slug":"bowen-yu","given":"Bowen","family":"Yu","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences , Beijing , China; School of Cyber Security , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Zhenyu Zhang","slug":"zhenyu-zhang","given":"Zhenyu","family":"Zhang","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences , Beijing , China; School of Cyber Security , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Tingwen Liu","slug":"tingwen-liu","given":"Tingwen","family":"Liu","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences , Beijing , China"},{"name":"Bin Wang","slug":"bin-wang","given":"Bin","family":"Wang","affiliation":"Xiaomi AI Lab , Xiaomi Inc. , Beijing , China"},{"name":"Sujian Li","slug":"sujian-li","given":"Sujian","family":"Li","affiliation":"Key Laboratory of Computational Linguistics , Peking University , MOE , China"},{"name":"Quangang Li","slug":"quangang-li","given":"Quangang","family":"Li","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences , Beijing , China"}],"abstract":"Relation extraction studies the issue of predicting semantic relations between pairs of entities in sentences. Attention mechanisms are often used in this task to alleviate the inner-sentence noise by performing soft selections of words independently. Based on the observation that information pertinent to relations is usually contained within segments (continuous words in a sentence), it is possible to make use of this phenomenon for better extraction. In this paper, we aim to incorporate such segment information into neural relation extractor. Our approach views the attention mechanism as linear-chain conditional random fields over a set of latent variables whose edges encode the desired structure, and regards attention weight as the marginal distribution of each word being selected as a part of the relational expression. Experimental results show that our method can attend to continuous relational expressions without explicit annotations, and achieve the state-of-the-art performance on the large-scale TACRED dataset.","doi":"10.24963/ijcai.2019/750","url":"https://ijcai.org/proceedings/2019/750","pdf":"https://ijcai.org/proceedings/2019/0750.pdf","markdown":"https://ijcai.org/proceedings/2019/750.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/750","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":750,"pages":{"first":5401,"last":5407},"keywords":["Natural Language Processing: Information Extraction","Natural Language Processing: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Angeli et al., 2015] Gabor Angeli, Victor Zhong, Danqi Chen, Arun Tejasvi Chaganty, Jason Bolton, Melvin Jose Johnson Premkumar, Panupong Pasupat, Sonal Gupta, and Christopher D Manning. Bootstrapped self training for knowledge base population. In TAC, 2015.","title":"Bootstrapped self training for knowledge base population","authors":"Gabor Angeli, Victor Zhong, Danqi Chen, Arun Tejasvi Chaganty, Jason Bolton, Melvin Jose, Johnson Premkumar, Panupong Pasupat, Sonal Gupta, Christopher D Manning","year":"2015","venue":"TAC","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Du et al., 2018] Jinhua Du, Jingguang Han, Andy Way, and Dadong Wan. Multi-level structured self-attentions for distantly supervised relation extraction. arXiv preprint arXiv:1809.00699, 2018.","title":"Multi-level structured self-attentions for distantly supervised relation extraction","authors":"Du","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Graves et al., 2013] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep re- current neural networks. In 2013 IEEE international con- ference on acoustics, speech and signal processing, pages 6645-6649. IEEE, 2013.","title":"Speech recognition with deep recurrent neural networks","authors":"Alex Graves, Abdel-Rahman Mohamed, Geoffrey Hinton","year":"2013","venue":"2013 IEEE international conference on acoustics, speech and signal processing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Kim et al., 2017] Yoon Kim, Carl Denton, Luong Hoang, and Alexander M Rush. Structured attention networks. arXiv preprint arXiv:1702.00887, 2017.","title":"Structured attention networks","authors":"Yoon Kim, Carl Denton, Luong Hoang, Alexander M Rush","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Lafferty et al., 2001] John Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. 2001.","title":"Conditional random fields: Probabilistic models for segmenting and labeling sequence data","authors":"Lafferty","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Lee et al., 2019] Joohong Lee, Sangwoo Seo, and Yong Suk Choi. Semantic relation classification via bidirectional lstm networks with entity-aware attention using latent en- tity typing. arXiv preprint arXiv:1901.08163, 2019.","title":"Semantic relation classification via bidirectional lstm networks with entity-aware attention using latent entity typing","authors":"Lee","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Liu et al., 2015] Yang Liu, Furu Wei, Sujian Li, Heng Ji, Ming Zhou, and Houfeng Wang. A dependency-based neural network for relation classification. arXiv preprint arXiv:1507.04646, 2015.","title":"A dependency-based neural network for relation classification","authors":"Yang Liu, Furu Wei, Sujian Li, Heng Ji, Ming Zhou, Houfeng Wang","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Liu et al., 2018] Tianyi Liu, Xinsong Zhang, Wanhao Zhou, and Weijia Jia. Neural relation extraction via inner- sentence noise reduction and transfer learning. arXiv preprint arXiv:1808.06738, 2018.","title":"Neural relation extraction via innersentence noise reduction and transfer learning","authors":"Tianyi Liu, Xinsong Zhang, Wanhao Zhou, Weijia Jia","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Manning et al., 2014] Christopher Manning, Mihai Sur- deanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meet- ing of the association for computational linguistics: sys- tem demonstrations, pages 55-60, 2014.","title":"The stanford corenlp natural language processing toolkit","authors":"Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, David Mcclosky","year":"2014","venue":"Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Rink and Harabagiu, 2010] Bryan Rink and Sanda Harabagiu. Utd: Classifying semantic relations by combining lexical and semantic resources. In Proceedings of the 5th International Workshop on Semantic Evalu- ation, pages 256-259. Association for Computational Linguistics, 2010.","title":"Utd: Classifying semantic relations by combining lexical and semantic resources","authors":"Harabagiu Rink, Bryan Rink, Sanda Harabagiu","year":"2010","venue":"Proceedings of the 5th International Workshop on Semantic Evaluation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Santos et al., 2015] Cicero Nogueira dos Santos, Bing Xi- ang, and Bowen Zhou. Classifying relations by rank- ing with convolutional neural networks. arXiv preprint arXiv:1504.06580, 2015.","title":"Classifying relations by ranking with convolutional neural networks","authors":"Santos","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Tai et al., 2015] Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representa- tions from tree-structured long short-term memory net- works. arXiv preprint arXiv:1503.00075, 2015.","title":"Improved semantic representations from tree-structured long short-term memory networks","authors":"Kai Sheng, Tai, Richard Socher, Christopher D Manning","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Turian et al., 2010] Joseph Turian, Lev Ratinov, and Yoshua Bengio. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th annual meeting of the association for computa- tional linguistics, pages 384-394. Association for Compu- tational Linguistics, 2010.","title":"Word representations: a simple and general method for semi-supervised learning","authors":"Turian","year":"2010","venue":"Proceedings of the 48th annual meeting of the association for computational linguistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Wang and Lu, 2018] Bailin Wang and Wei Lu. Learning la- tent opinions for aspect-level sentiment classification. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.","title":"Learning latent opinions for aspect-level sentiment classification","authors":"Bailin Wang, Wei Lu","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wang et al., 2016] Linlin Wang, Zhu Cao, Gerard De Melo, and Zhiyuan Liu. Relation classification via multi-level attention cnns. 2016.","title":"Relation classification via multi-level attention cnns","authors":"Linlin Wang, Zhu Cao, Gerard De Melo, Zhiyuan Liu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Xiao and Liu, 2016] Minguang Xiao and Cong Liu. Seman- tic relation classification via hierarchical recurrent neural network with attention. In Proceedings of COLING 2016, the 26th International Conference on Computational Lin- guistics: Technical Papers, pages 1254-1263, 2016.","title":"Minguang Xiao and Cong Liu. Semantic relation classification via hierarchical recurrent neural network with attention","authors":"Xiao, Liu","year":"2016","venue":"Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Xu et al., 2015] Yan Xu, Lili Mou, Ge Li, Yunchuan Chen, Hao Peng, and Zhi Jin. Classifying relations via long short term memory networks along shortest dependency paths. In proceedings of the 2015 conference on empiri- cal methods in natural language processing, pages 1785- 1794, 2015.","title":"Classifying relations via long short term memory networks along shortest dependency paths","authors":"Yan Xu, Lili Mou, Ge Li, Yunchuan Chen, Hao Peng, Zhi Jin","year":"2015","venue":"proceedings of the 2015 conference on empirical methods in natural language processing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zeng et al., 2014] Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, Jun Zhao, et al. Relation classification via convolutional deep neural network. 2014.","title":"Relation classification via convolutional deep neural network","authors":"Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, Jun Zhao","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zeng et al., 2015] Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. Distant supervision for relation extraction via piecewise convolutional neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1753-1762, 2015.","title":"Distant supervision for relation extraction via piecewise convolutional neural networks","authors":"Zeng","year":"2015","venue":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang et al., 2017] Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D Manning. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Em- pirical Methods in Natural Language Processing, pages 35-45, 2017.","title":"Position-aware attention and supervised data improve slot filling","authors":"Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, Christopher D Manning","year":"2017","venue":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2018] Yuhao Zhang, Peng Qi, and Christo- pher D Manning. Graph convolution over pruned depen- dency trees improves relation extraction. arXiv preprint arXiv:1809.10185, 2018.","title":"Graph convolution over pruned dependency trees improves relation extraction","authors":"Yuhao Zhang, Peng Qi, Christopher D Manning","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhou et al., 2016] Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. Attention- based bidirectional long short-term memory networks for relation classification. In Proceedings of the 54th An- nual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), volume 2, pages 207- 212, 2016.","title":"Attentionbased bidirectional long short-term memory networks for relation classification","authors":"Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, Bo Xu","year":"2016","venue":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}