{"type":"ScholarlyArticle","title":"FastRE: Towards Fast Relation Extraction with Convolutional Encoder and Improved Cascade Binary Tagging Framework","authors":[{"name":"Guozheng Li","slug":"guozheng-li","given":"Guozheng","family":"Li","affiliation":"School of Computer Science and Engineering , Southeast University"},{"name":"Xu Chen","slug":"xu-chen","given":"Xu","family":"Chen","affiliation":"Tencent Inc."},{"name":"Peng Wang","slug":"peng-wang","given":"Peng","family":"Wang","affiliation":"School of Computer Science and Engineering , Southeast University; School of Cyber Science and Engineering , Southeast University; School of Artificial Intelligence , Southeast University"},{"name":"Jiafeng Xie","slug":"jiafeng-xie","given":"Jiafeng","family":"Xie","affiliation":"School of Computer Science and Engineering , Southeast University"},{"name":"Qiqing Luo","slug":"qiqing-luo","given":"Qiqing","family":"Luo","affiliation":"School of Computer Science and Engineering , Southeast University; School of Artificial Intelligence , Southeast University"}],"abstract":"Recent work for extracting relations from texts has achieved excellent performance. However, most existing methods pay less attention to the efficiency, making it still challenging to quickly extract relations from massive or streaming text data in realistic scenarios. The main efficiency bottleneck is that these methods use a Transformer-based pre-trained language model for encoding, which heavily affects the training speed and inference speed. To address this issue, we propose a fast relation extraction model (FastRE) based on convolutional encoder and improved cascade binary tagging framework. Compared to previous work, FastRE employs several innovations to improve efficiency while also keeping promising performance. Concretely, FastRE adopts a novel convolutional encoder architecture combined with dilated convolution, gated unit and residual connection, which significantly reduces the computation cost of training and inference, while maintaining the satisfactory performance. Moreover, to improve the cascade binary tagging framework, FastRE first introduces a type-relation mapping mechanism to accelerate tagging efficiency and alleviate relation redundancy, and then utilizes a position-dependent adaptive thresholding strategy to obtain higher tagging accuracy and better model generalization. Experimental results demonstrate that FastRE is well balanced between efficiency and performance, and achieves 3-10$\\times$ training speed, 7-15$\\times$ inference speed faster, and 1/100 parameters compared to the state-of-the-art models, while the performance is still competitive. Our code is available at \\url{https://github.com/seukgcode/FastRE}.","doi":"10.24963/ijcai.2022/583","url":"https://ijcai.org/proceedings/2022/583","pdf":"https://ijcai.org/proceedings/2022/0583.pdf","markdown":"https://ijcai.org/proceedings/2022/583.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/583","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":583,"pages":{"first":4201,"last":4208},"keywords":["Natural Language Processing: Information Extraction","Knowledge Representation and Reasoning: Other","Natural Language Processing: Knowledge Extraction"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986682","length_sec":73,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984825","length_sec":336,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Dauphin et al., 2017] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In ICML, 2017.","title":"Language modeling with gated convolutional networks","authors":"Angela Yann N Dauphin, Michael Fan, David Auli, Grangier","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL, 2019.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dong et al., 2014] Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, and et al. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In SIGKDD, 2014.","title":"Knowledge vault: A web-scale approach to probabilistic knowledge fusion","authors":"Xin Dong, Evgeniy Gabrilovich, Geremy Heitz","year":"2014","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[dos Santos et al., 2015] Cícero dos Santos, Bing Xiang, and Bowen Zhou. Classifying relations by ranking with con- volutional neural networks. In ACL, 2015.","title":"Classifying relations by ranking with convolutional neural networks","authors":"Bing Cícero Dos Santos, Bowen Xiang, Zhou","year":"2015","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gehring et al., 2017] Jonas Gehring, Michael Auli, David Grangier, and et al. Convolutional sequence to sequence learning. In ICML, 2017.","title":"Convolutional sequence to sequence learning","authors":"Gehring","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goyal et al., 2017] Priya Goyal, Piotr Dollár, Ross Gir- shick, and et al. Accurate, large minibatch sgd: Train- ing imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.","title":"Accurate, large minibatch sgd: Training imagenet in 1 hour","authors":"Priya Goyal, Piotr Dollár, Ross Girshick","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huguet Cabot and Navigli, 2021] Pere-Lluís Huguet Cabot and Roberto Navigli. REBEL: Relation extraction by end- to-end language generation. In Findings of EMNLP, 2021.","title":"REBEL: Relation extraction by endto-end language generation","authors":"Huguet Cabot, Huguet Navigli ; Pere-Lluís, Roberto Cabot, Navigli","year":"2021","venue":"Findings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lewis et al., 2020] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL, 2020.","title":"BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","authors":"Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Loshchilov and Hutter, 2019] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.","title":"Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization","authors":"Hutter Loshchilov","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Nayak and Ng, 2020] Tapas Nayak and Hwee Tou Ng. Ef- fective modeling of encoder-decoder architecture for joint entity and relation extraction. In AAAI, 2020.","title":"Effective modeling of encoder-decoder architecture for joint entity and relation extraction","authors":"Ng ; ] Tapas Nayak, Hwee Tou Nayak, Ng","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ren et al., 2021] Feiliang Ren, Longhui Zhang, Shujuan Yin, and et al. A novel global feature-oriented relational triple extraction model based on table filling. In EMNLP, 2021.","title":"A novel global feature-oriented relational triple extraction model based on table filling","authors":"Ren","year":"2021","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, and et al. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky","year":"2014","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sui et al., 2021] Dianbo Sui, Yubo Chen, Kang Liu, and et al. Joint entity and relation extraction with set prediction networks. In CoRR, 2021. [Takanobu et al., 2019]","title":"Joint entity and relation extraction with set prediction networks","authors":"Sui","year":"2019","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"Ryuichi Takanobu, Tianyang Zhang, Jiexi Liu, and Minlie Huang. A hierarchical framework for relation extraction with reinforcement learning. In AAAI, 2019.","title":"A hierarchical framework for relation extraction with reinforcement learning","authors":"Ryuichi Takanobu, Tianyang Zhang, Jiexi Liu, Minlie Huang","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Takase et al., 2015] Sho Takase, Naoaki Okazaki, and Ken- taro Inui. Fast and large-scale unsupervised relation ex- traction. In PACLIC, 2015.","title":"Fast and large-scale unsupervised relation extraction","authors":"Sho Takase, Naoaki Okazaki, Kentaro Inui","year":"2015","venue":"PACLIC","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tran and Kavuluru, 2019] Tung Tran and Ramakanth Kavu- luru. Neural metric learning for fast end-to-end relation extraction. arXiv preprint arXiv:1905.07458, 2019.","title":"Neural metric learning for fast end-to-end relation extraction","authors":"Tung Tran, Ramakanth Kavuluru","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, and et al. Attention is all you need. In NIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2020] Yucheng Wang, Bowen Yu, Yueyang Zhang, and et al. TPLinker: Single-stage joint extrac- tion of entities and relations through token pair linking. In COLING, 2020.","title":"TPLinker: Single-stage joint extraction of entities and relations through token pair linking","authors":"Yucheng Wang, Bowen Yu, Yueyang Zhang","year":"2020","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wei et al., 2020] Zhepei Wei, Jianlin Su, Yue Wang, and et al. A novel cascade binary tagging framework for re- lational triple extraction. In ACL, 2020.","title":"A novel cascade binary tagging framework for relational triple extraction","authors":"Wei","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xie et al., 2021] Chenhao Xie, Jiaqing Liang, Jingping Liu, Chengsong Huang, Wenhao Huang, and Yanghua Xiao. Revisiting the negative data of distantly supervised rela- tion extraction. In ACL, 2021.","title":"Revisiting the negative data of distantly supervised relation extraction","authors":"Chenhao Xie, Jiaqing Liang, Jingping Liu, Chengsong Huang, Wenhao Huang, Yanghua Xiao","year":"2021","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yu and Koltun, 2016] Fisher Yu and Vladlen Koltun. Multi- scale context aggregation by dilated convolutions. In ICLR, 2016.","title":"Multiscale context aggregation by dilated convolutions","authors":"Koltun ; Fisher Yu, Vladlen Yu, Koltun","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zeng et al., 2014] Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. Relation classification via convolutional deep neural network. In COLING, 2014.","title":"Relation classification via convolutional deep neural network","authors":"Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, Jun Zhao","year":"2014","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zeng et al., 2018] Xiangrong Zeng, Daojian Zeng, Shizhu He, and et al. Extracting relational facts by an end-to-end neural model with copy mechanism. In ACL, 2018.","title":"Extracting relational facts by an end-to-end neural model with copy mechanism","authors":"Xiangrong Zeng, Daojian Zeng, Shizhu He","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2020] Ranran Haoran Zhang, Qianying Liu, Aysa Xuemo Fan, and et al. Minimize exposure bias of Seq2Seq models in joint entity and relation extraction. In Findings of EMNLP, 2020.","title":"Minimize exposure bias of Seq2Seq models in joint entity and relation extraction","authors":"Zhang","year":"2020","venue":"Findings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zheng et al., 2017] Suncong Zheng, Feng Wang, Hongyun Bao, and et al. Joint extraction of entities and relations based on a novel tagging scheme. In ACL, 2017.","title":"Joint extraction of entities and relations based on a novel tagging scheme","authors":"Zheng","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zheng et al., 2021] Hengyi Zheng, Rui Wen, Xi Chen, and et al. PRGC: Potential relation and global correspondence based joint relational triple extraction. In ACL, 2021.","title":"PRGC: Potential relation and global correspondence based joint relational triple extraction","authors":"Hengyi Zheng, Rui Wen, Xi Chen","year":"2021","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhong and Chen, 2021] Zexuan Zhong and Danqi Chen. A frustratingly easy approach for entity and relation extrac- tion. In NAACL, 2021.","title":"A frustratingly easy approach for entity and relation extraction","authors":"Chen ; Zhong, Zexuan Zhong, Danqi Chen","year":"2021","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhou et al., 2021] Wenxuan Zhou, Kevin Huang, Tengyu Ma, and Jing Huang. Document-level relation extraction with adaptive thresholding and localized context pooling. In AAAI, 2021.","title":"Document-level relation extraction with adaptive thresholding and localized context pooling","authors":"Wenxuan Zhou, Kevin Huang, Tengyu Ma, Jing Huang","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}