{"type":"ScholarlyArticle","title":"Modeling Dense Cross-Modal Interactions for Joint Entity-Relation Extraction","authors":[{"name":"Shan Zhao","slug":"shan-zhao","given":"Shan","family":"Zhao","affiliation":"College of Computer , National University of Defense Technology , Changsha , China"},{"name":"Minghao Hu","slug":"minghao-hu","given":"Minghao","family":"Hu","affiliation":"PLA Academy of Military Science , Beijing , China"},{"name":"Zhiping Cai","slug":"zhiping-cai","given":"Zhiping","family":"Cai","affiliation":"College of Computer , National University of Defense Technology , Changsha , China"},{"name":"Fang Liu","slug":"fang-liu","given":"Fang","family":"Liu","affiliation":"School of Design , Hunan University , Changsha , Hunan"}],"abstract":"Joint extraction of entities and their relations benefits from the close interaction between named entities and their relation information. Therefore, how\r\nto effectively model such cross-modal interactions is critical for the final performance. Previous works have used simple methods such as label-feature concatenation to perform coarse-grained semantic fusion among cross-modal instances, but fail to capture fine-grained correlations over token and label\r\nspaces, resulting in insufficient interactions. In this paper, we propose a deep Cross-Modal Attention Network (CMAN) for joint entity and relation extraction. The network is carefully constructed by stacking multiple attention units in depth to fully model dense interactions over token-label spaces, in which two basic attention units are proposed to explicitly capture fine-grained correlations across\r\ndifferent modalities (e.g., token-to-token and labelto-token). Experiment results on CoNLL04 dataset show that our model obtains state-of-the-art results by achieving 90.62% F1 on entity recognition and 72.97% F1 on relation classification. In ADE dataset, our model surpasses existing approaches by\r\nmore than 1.9% F1 on relation classification. Extensive analyses further confirm the effectiveness of our approach.","doi":"10.24963/ijcai.2020/558","url":"https://ijcai.org/proceedings/2020/558","pdf":"https://ijcai.org/proceedings/2020/0558.pdf","markdown":"https://ijcai.org/proceedings/2020/558.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/558","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":558,"pages":{"first":4032,"last":4038},"keywords":["Natural Language Processing: Information Extraction","Natural Language Processing: Named Entities"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24337/file_24337.m3u8","length_sec":294,"provider":"hls"}],"references":[{"position":1,"raw":"[Adel and Schütze, 2017] Heike Adel and Hinrich Schütze. Global normalization of convolutional neural networks for joint entity and relation classification. arXiv preprint arXiv:1707.07719, 2017.","title":"Global normalization of convolutional neural networks for joint entity and relation classification","authors":"Heike Adel, Hinrich Schütze","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of ICCV, pages 2425-2433, 2015.","title":"Vqa: Visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"Proceedings of ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ba et al., 2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Ge- offrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.","title":"Layer normalization","authors":"Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E Hinton","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bach and Badaskar, 2007] Nguyen Bach and Sameer Badaskar. A review of relation extraction. Literature review for Language and Statistics II, 2:1-15, 2007.","title":"A review of relation extraction","authors":"Nguyen Bach, Sameer Badaskar","year":"2007","venue":"Literature review for Language and Statistics II","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bekoulis et al., 2018] Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. Adversarial train- ing for multi-context joint entity and relation extraction. arXiv preprint arXiv:1808.06876, 2018.","title":"Adversarial training for multi-context joint entity and relation extraction","authors":"Giannis Bekoulis, Johannes Deleu, Thomas Demeester, Chris Develder","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cui and Zhang, 2019] Leyang Cui and Yue Zhang. Hierarchically-refined label attention network for sequence labeling. arXiv preprint arXiv:1908.08676, 2019.","title":"Hierarchically-refined label attention network for sequence labeling","authors":"Zhang Cui, Leyang Cui, Yue Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Eberts and Ulges, 2019] Markus Eberts and Adrian Ulges. Span-based joint entity and relation extraction with trans- former pre-training. arXiv preprint arXiv:1909.07755, 2019.","title":"Span-based joint entity and relation extraction with transformer pre-training","authors":"Ulges Eberts, Markus Eberts, Adrian Ulges","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guo et al., 2019] Zhijiang Guo, Yan Zhang, and Wei Lu. At- tention guided graph convolutional networks for relation extraction. In Proceedings of ACL, 2019.","title":"Attention guided graph convolutional networks for relation extraction","authors":"Zhijiang Guo, Yan Zhang, Wei Lu","year":"2019","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gurulingappa et al., 2012] Harsha Gurulingappa, Abdul Ma- teen Rajput, and Roberts. Development of a benchmark cor- pus to support the automatic extraction of drug-related ad- verse effects from medical case reports. Journal of biomed- ical informatics, 45(5):885-892, 2012.","title":"Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports","authors":"Harsha Gurulingappa, Abdul Mateen Rajput, Roberts","year":"2012","venue":"Journal of biomedical informatics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of CVPR,2016, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Katiyar and Cardie, 2017] Arzoo Katiyar and Claire Cardie. Going out on a limb: Joint extraction of entity mentions and relations without dependency trees. In Proceedings of ACL,2017, pages 917-928, 2017.","title":"Going out on a limb: Joint extraction of entity mentions and relations without dependency trees","authors":"Cardie ; ] Arzoo Katiyar, Claire Katiyar, Cardie","year":"2017","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lafferty et al., 2001] John Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields: Proba- bilistic models for segmenting and labeling sequence data. 2001.","title":"Conditional random fields: Probabilistic models for segmenting and labeling sequence data","authors":"Lafferty","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2016] Fei Li, Yue Zhang, Meishan Zhang, and Donghong Ji. Joint models for extracting adverse drug events from biomedical text. In IJCAI, volume 2016, pages 2838-2844, 2016.","title":"Joint models for extracting adverse drug events from biomedical text","authors":"Fei Li, Yue Zhang, Meishan Zhang, Donghong Ji","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/403"},{"position":15,"raw":"[Li et al., 2017] Fei Li, Meishan Zhang, Guohong Fu, and Donghong Ji. A neural joint model for entity and rela- tion extraction from biomedical text. BMC bioinformatics, 18(1):198, 2017.","title":"A neural joint model for entity and relation extraction from biomedical text","authors":"Fei Li, Meishan Zhang, Guohong Fu, Donghong Ji","year":"2017","venue":"BMC bioinformatics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2019] Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li. Entity-relation extraction as multi-turn question answering. arXiv preprint arXiv:1905.05529, 2019.","title":"Entity-relation extraction as multi-turn question answering","authors":"Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, Jiwei Li","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Miwa and Bansal, 2016] Makoto Miwa and Mohit Bansal. End-to-end relation extraction using lstms on sequences and tree structures. arXiv preprint arXiv:1601.00770, 2016.","title":"End-to-end relation extraction using lstms on sequences and tree structures","authors":"Bansal Miwa, Makoto Miwa, Mohit Bansal","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Miwa and Sasaki, 2014] Makoto Miwa and Yutaka Sasaki. Modeling joint entity and relation extraction with table representation. In Proceedings of EMNLP, 2014., pages 1858-1869, 2014.","title":"Modeling joint entity and relation extraction with table representation","authors":"Sasaki Miwa, Makoto Miwa, Yutaka Sasaki","year":"2014","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Miwa et al., 2009] Makoto Miwa, Rune Sætre, Yusuke Miyao, and Jun'ichi Tsujii. A rich feature vector for protein- protein interaction extraction from multiple corpora. In Proceedings of EMNLP, pages 121-130, 2009.","title":"A rich feature vector for proteinprotein interaction extraction from multiple corpora","authors":"Makoto Miwa, Rune Sætre, Yusuke Miyao, Jun'ichi Tsujii","year":"2009","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nadeau and Sekine, 2007] David Nadeau and Satoshi Sekine. A survey of named entity recognition and classification. Lingvisticae Investigationes, 30(1):3-26, 2007.","title":"A survey of named entity recognition and classification","authors":"Sekine Nadeau, David Nadeau, Satoshi Sekine","year":"2007","venue":"Lingvisticae Investigationes","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Peters et al., 2018] Matthew E Peters, Mark Neumann, Mo- hit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word represen- tations. arXiv preprint arXiv:1802.05365, 2018.","title":"Deep contextualized word representations","authors":"Mark Matthew E Peters, Mohit Neumann, Matt Iyyer, Christopher Gardner, Kenton Clark, Luke Lee, Zettlemoyer","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Roth and Yih, 2004] Dan Roth and Wen-tau Yih. A linear programming formulation for global inference in natural language tasks. In Proceedings of CoNLL-2004 at HLT- NAACL 2004, pages 1-8, 2004.","title":"A linear programming formulation for global inference in natural language tasks","authors":"Roth, Yih, Dan Roth, Wen-Tau Yih","year":"2004","venue":"Proceedings of CoNLL-2004 at HLT-NAACL 2004","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tran and Kavuluru, 2019] Tung Tran and Ramakanth Kavu- luru. Neural metric learning for fast end-to-end relation extraction. arXiv preprint arXiv:1905.07458, 2019.","title":"Neural metric learning for fast end-to-end relation extraction","authors":"Tung Tran, Ramakanth Kavuluru","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2016] Linlin Wang, Zhu Cao, Gerard De Melo, and Zhiyuan Liu. Relation classification via multi-level attention cnns. 2016.","title":"Relation classification via multi-level attention cnns","authors":"Linlin Wang, Zhu Cao, Gerard De Melo, Zhiyuan Liu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2018] Guoyin Wang, Chunyuan Li, Wenlin Wang, Yizhe Zhang, Dinghan Shen, and Zhang. Joint em- bedding of words and labels for text classification. arXiv preprint arXiv:1805.04174, 2018.","title":"Joint embedding of words and labels for text classification","authors":"Wang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu et al., 2019] Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the CVPR,2019, pages 6281-6290, 2019.","title":"Deep modular co-attention networks for visual question answering","authors":"Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, Qi Tian","year":"2019","venue":"Proceedings of the CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zheng et al., 2017] Suncong Zheng, Feng Wang, Hongyun Bao, Yuexing Hao, Peng Zhou, and Bo Xu. Joint extraction of entities and relations based on a novel tagging scheme. arXiv preprint arXiv:1706.05075, 2017.","title":"Joint extraction of entities and relations based on a novel tagging scheme","authors":"Suncong Zheng, Feng Wang, Hongyun Bao, Yuexing Hao, Peng Zhou, Bo Xu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}