Drop Redundant, Shrink Irrelevant: Selective Knowledge Injection for Language Pretraining

Zhang, Ningyu; Deng, Shumin; Cheng, Xu; Chen, Xi; Zhang, Yichi; Zhang, Wei; Chen, Huajun

doi:10.24963/ijcai.2021/552

Drop Redundant, Shrink Irrelevant: Selective Knowledge Injection for Language Pretraining

Ningyu Zhang, Shumin Deng, Xu Cheng, Xi Chen, Yichi Zhang, Wei Zhang, Huajun Chen

Download BibTeX Cite

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. Main Track. Pages 4007-4014.

https://doi.org/10.24963/ijcai.2021/552

Keywords

Natural Language Processing: Natural Language Processing
Natural Language Processing: Question Answering
Natural Language Processing: Information Extraction

Abstract

Previous research has demonstrated the power of leveraging prior knowledge to improve the performance of deep models in natural language processing. However, traditional methods neglect the fact that redundant and irrelevant knowledge exists in external knowledge bases. In this study, we launched an in-depth empirical investigation into downstream tasks and found that knowledge-enhanced approaches do not always exhibit satisfactory improvements. To this end, we investigate the fundamental reasons for ineffective knowledge infusion and present selective injection for language pretraining, which constitutes a model-agnostic method and is readily pluggable into previous approaches. Experimental results on benchmark datasets demonstrate that our approach can enhance state-of-the-art knowledge injection methods.

Ningyu Zhang, Shumin Deng, Xu Cheng, Xi Chen, Yichi Zhang, Wei Zhang, and Huajun Chen. 2021. Drop Redundant, Shrink Irrelevant: Selective Knowledge Injection for Language Pretraining. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. IJCAI, 4007–4014. https://doi.org/10.24963/ijcai.2021/552

@inproceedings{ijcai2021p552,
  title     = {Drop Redundant, Shrink Irrelevant: Selective Knowledge Injection for Language Pretraining},
  author    = {Zhang, Ningyu and Deng, Shumin and Cheng, Xu and Chen, Xi and Zhang, Yichi and Zhang, Wei and Chen, Huajun},
  booktitle = {Proceedings of the Thirtieth International Joint Conference on
               Artificial Intelligence, {IJCAI-21}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Prof. Zhi-Hua Zhou},
  pages     = {4007--4014},
  year      = {2021},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2021/552},
  url       = {https://doi.org/10.24963/ijcai.2021/552},
}

TY  - CPAPER
TI  - Drop Redundant, Shrink Irrelevant: Selective Knowledge Injection for Language Pretraining
AU  - Zhang, Ningyu
AU  - Deng, Shumin
AU  - Cheng, Xu
AU  - Chen, Xi
AU  - Zhang, Yichi
AU  - Zhang, Wei
AU  - Chen, Huajun
T2  - Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2021
SP  - 4007
EP  - 4014
DO  - 10.24963/ijcai.2021/552
UR  - https://doi.org/10.24963/ijcai.2021/552
ER  -