{"type":"ScholarlyArticle","title":"DictBERT: Dictionary Description Knowledge Enhanced Language Model Pre-training via Contrastive Learning","authors":[{"name":"Qianglong Chen","slug":"qianglong-chen","given":"Qianglong","family":"Chen","affiliation":"College of Computer Science and Technology , Zhejiang University , China; Alibaba Group , China"},{"name":"Feng-Lin Li","slug":"feng-lin-li","given":"Feng-Lin","family":"Li","affiliation":"Alibaba Group , China"},{"name":"Guohai Xu","slug":"guohai-xu","given":"Guohai","family":"Xu","affiliation":"Alibaba Group , China"},{"name":"Ming Yan","slug":"ming-yan","given":"Ming","family":"Yan","affiliation":"Alibaba Group , China"},{"name":"Ji Zhang","slug":"ji-zhang","given":"Ji","family":"Zhang","affiliation":"Alibaba Group , China"},{"name":"Yin Zhang","slug":"yin-zhang","given":"Yin","family":"Zhang","affiliation":"College of Computer Science and Technology , Zhejiang University , China"}],"abstract":"Although pre-trained language models (PLMs) have achieved state-of-the-art performance on various natural language processing (NLP) tasks, they are shown to be lacking in knowledge when dealing with knowledge driven tasks. Despite the many efforts made for injecting knowledge into PLMs, this problem remains open. To address the challenge, we propose DictBERT, a novel approach that enhances PLMs with dictionary knowledge which is easier to acquire than knowledge graph (KG). During pre-training, we present two novel pre-training tasks to inject dictionary knowledge into PLMs via contrastive learning: dictionary entry prediction and entry description discrimination. In fine-tuning, we use the pre-trained DictBERT as a plugin knowledge base (KB) to retrieve implicit knowledge for identified entries in an input sequence, and infuse the retrieved knowledge into the input to enhance its representation via a novel extra-hop attention mechanism. We evaluate our approach on a variety of knowledge driven and language understanding tasks, including NER, relation extraction, CommonsenseQA, OpenBookQA and GLUE. Experimental results demonstrate that our model can significantly improve typical PLMs: it gains a substantial improvement of 0.5%, 2.9%, 9.0%, 7.1% and 3.3% on BERT-large respectively, and is also effective on RoBERTa-large.","doi":"10.24963/ijcai.2022/567","url":"https://ijcai.org/proceedings/2022/567","pdf":"https://ijcai.org/proceedings/2022/0567.pdf","markdown":"https://ijcai.org/proceedings/2022/567.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/567","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":567,"pages":{"first":4086,"last":4092},"keywords":["Natural Language Processing: Question Answering","Natural Language Processing: Natural Language Semantics","Natural Language Processing: Applications","Natural Language Processing: Language Models","Natural Language Processing: Text Classification"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986976","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985119","length_sec":701,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chen et al., 2020] Qianglong Chen, Feng Ji, Haiqing Chen, and Yin Zhang. Improving commonsense question an- swering by graph-based iterative retrieval over multiple knowledge sources. In COLING, pages 2583-2594, 2020.","title":"Improving commonsense question answering by graph-based iterative retrieval over multiple knowledge sources","authors":"Chen","year":"2020","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL-HLT, pages 4171-4186, 2019.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Lan et al., 2019] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of lan- guage representations. arXiv:1909.11942, 2019.","title":"Albert: A lite bert for self-supervised learning of language representations","authors":"Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692, 2019.","title":"Roberta: A robustly optimized bert pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Liu et al., 2020] Weijie Liu, Peng Zhou, Zhe Zhao, et al. K-bert: Enabling language representation with knowledge graph. In AAAI, pages 2901-2908, 2020.","title":"K-bert: Enabling language representation with knowledge graph","authors":"Weijie Liu, Peng Zhou, Zhe Zhao","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Logeswaran and Lee, 2018] Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence representations. In ICLR, 2018.","title":"Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence representations","authors":"Lee Logeswaran","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Mihaylov et al., 2018] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answer- ing. In EMNLP, pages 2381-2391, 2018.","title":"Can a suit of armor conduct electricity? a new dataset for open book question answering","authors":"Todor Mihaylov, Peter Clark, Tushar Khot, Ashish Sabharwal","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Peters et al., 2019] Matthew E. Peters, Mark Neumann, Robert L Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. Knowledge enhanced contex- tual word representations. In EMNLP, pages 43-54, 2019.","title":"Knowledge enhanced contextual word representations","authors":"Matthew E Peters, Mark Neumann, Robert L Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, Noah A Smith","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Petroni et al., 2019] Fabio Petroni, Tim Rocktäschel, Se- bastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowl- edge bases? In EMNLP, pages 2463-2473, 2019.","title":"Language models as knowledge bases?","authors":"Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Qin et al., 2021] Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, and Jie Zhou. ERICA: Improving entity and relation understanding for pre-trained language models via con- trastive learning. In ACL, pages 3350-3363, 2021.","title":"ERICA: Improving entity and relation understanding for pre-trained language models via contrastive learning","authors":"Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, Jie Zhou","year":"2021","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Sun et al., 2020] Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, and Zheng Zhang. Colake: Contextualized language and knowledge embedding. In COLING, pages 3660-3670, 2020.","title":"Colake: Contextualized language and knowledge embedding","authors":"Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, Zheng Zhang","year":"2020","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Talmor et al., 2019] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In NAACL-HLT, pages 4149-4158, 2019.","title":"CommonsenseQA: A question answering challenge targeting commonsense knowledge","authors":"Alon Talmor, Jonathan Herzig, Nicholas Lourie, Jonathan Berant","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Tjong Kim Sang and De Meulder, 2003] Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In NAACL-HLT, pages 142-147, 2003.","title":"Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition","authors":"Tjong Kim, Sang, De Meulder, ; ] Erik, F Tjong, Kim Sang, Fien De, Meulder","year":"2003","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Tsukagoshi et al., 2021] Hayato Tsukagoshi, Ryohei Sasano, and Koichi Takeda. DefSent: Sentence embed- dings using definition sentences. In ACL, 2021.","title":"DefSent: Sentence embeddings using definition sentences","authors":"Tsukagoshi","year":"2021","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Wang et al., 2018] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In EMNLP, 2018.","title":"GLUE: A multi-task benchmark and analysis platform for natural language understanding","authors":"Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel Bowman","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wang et al., 2021a] Dong Wang, Ning Ding, Piji Li, and Hai-Tao Zheng. Cline: Contrastive learning with semantic negative examples for natural language understanding. In ACL, pages 2332-2342, 2021.","title":"Cline: Contrastive learning with semantic negative examples for natural language understanding","authors":"Wang","year":"2021","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2021b] Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. K-Adapter: Infusing Knowl- edge into Pre-Trained Models with Adapters. In Findings of ACL, pages 1405-1418, 2021.","title":"K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters","authors":"Wang","year":"2021","venue":"Findings of ACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2021c] Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. Kepler: A unified model for knowledge embedding and pre-trained language representation. TACL, 2021.","title":"Kepler: A unified model for knowledge embedding and pre-trained language representation","authors":"Wang","year":"2021","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wu et al., 2020] Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. Clear: Contrastive learning for sentence representation. arXiv:2012.15466, 2020.","title":"Clear: Contrastive learning for sentence representation","authors":"Wu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xu et al., 2021] Yichong Xu, Chenguang Zhu, Ruochen Xu, Yang Liu, Michael Zeng, and Xuedong Huang. Fusing context into knowledge graph for commonsense question answering. In Findings of ACL, pages 1201-1207, 2021.","title":"Fusing context into knowledge graph for commonsense question answering","authors":"Yichong Xu, Chenguang Zhu, Ruochen Xu, Yang Liu, Michael Zeng, Xuedong Huang","year":"2021","venue":"Findings of ACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2020] Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji- Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. Generative data augmentation for commonsense reasoning. In Findings of EMNLP, pages 1008-1025, 2020.","title":"Generative data augmentation for commonsense reasoning","authors":"Yang","year":"2020","venue":"Findings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ye et al., 2019] Zhi-Xiu Ye, Qian Chen, Wen Wang, and Zhen-Hua Ling. Align, mask and select: A simple method for incorporating commonsense knowledge into language representation models. arXiv:1908.06725, 2019.","title":"Align, mask and select: A simple method for incorporating commonsense knowledge into language representation models","authors":"Zhi-Xiu Ye, Qian Chen, Wen Wang, Zhen-Hua Ling","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yu et al., 2021] Wenhao Yu, Chenguang Zhu, Yuwei Fang, Donghan Yu, Shuohang Wang, Yichong Xu, Michael Zeng, and Meng Jiang. Dict-bert: Enhancing language model pre-training with dictionary. arXiv:2110.06490, 2021.","title":"Dict-bert: Enhancing language model pre-training with dictionary","authors":"Wenhao Yu, Chenguang Zhu, Yuwei Fang, Donghan Yu, Shuohang Wang, Yichong Xu, Michael Zeng, Meng Jiang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2017] Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. Position-aware attention and supervised data improve slot filling. In EMNLP, pages 35-45, 2017.","title":"Position-aware attention and supervised data improve slot filling","authors":"Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, Christopher D Manning","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2019] Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. ERNIE: Enhanced language representation with informative entities. In ACL, pages 1441-1451, 2019.","title":"ERNIE: Enhanced language representation with informative entities","authors":"Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, Qun Liu","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhao et al., 2020] Chen Zhao, Chenyan Xiong, Corby Ros- set, Xia Song, Paul Bennett, and Saurabh Tiwary. Transformer-xh: Multi-evidence reasoning with extra hop attention. In ICLR, 2020.","title":"Transformer-xh: Multi-evidence reasoning with extra hop attention","authors":"Chen Zhao, Chenyan Xiong, Corby Rosset, Xia Song, Paul Bennett, Saurabh Tiwary","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}