{"type":"ScholarlyArticle","title":"Language Models as Knowledge Embeddings","authors":[{"name":"Xintao Wang","slug":"xintao-wang","given":"Xintao","family":"Wang","affiliation":"Shanghai Key Laboratory of Data Science , School of Computer Science , Fudan University"},{"name":"Qianyu He","slug":"qianyu-he","given":"Qianyu","family":"He","affiliation":"Shanghai Key Laboratory of Data Science , School of Computer Science , Fudan University"},{"name":"Jiaqing Liang","slug":"jiaqing-liang","given":"Jiaqing","family":"Liang","affiliation":"Shanghai Key Laboratory of Data Science , School of Computer Science , Fudan University"},{"name":"Yanghua Xiao","slug":"yanghua-xiao","given":"Yanghua","family":"Xiao","affiliation":"Shanghai Key Laboratory of Data Science , School of Computer Science , Fudan University; Fudan-Aishu Cognitive Intelligence Joint Research Center"}],"abstract":"Knowledge embeddings (KE) represent a knowledge graph (KG) by embedding entities and relations into continuous vector spaces. Existing methods are mainly structure-based or description-based. Structure-based methods learn representations that preserve the inherent structure of KGs. They cannot well represent abundant long-tail entities in real-world KGs with limited structural information. Description-based methods leverage textual information and language models. Prior approaches in this direction barely outperform structure-based ones, and suffer from problems like expensive negative sampling and restrictive description demand. In this paper, we propose LMKE, which adopts Language Models to derive Knowledge Embeddings, aiming at both enriching representations of long-tail entities and solving problems of prior description-based methods. We formulate description-based KE learning with a contrastive learning framework to improve efficiency in  training and evaluation. Experimental results show that LMKE achieves state-of-the-art performance on KE benchmarks of link prediction and triple classification, especially for long-tail entities.","doi":"10.24963/ijcai.2022/318","url":"https://ijcai.org/proceedings/2022/318","pdf":"https://ijcai.org/proceedings/2022/0318.pdf","markdown":"https://ijcai.org/proceedings/2022/318.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/318","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Data Mining","track":"Main Track","number":318,"pages":{"first":2291,"last":2297},"keywords":["Data Mining: Knowledge Graphs and Knowledge Base Completion","Machine Learning: Representation learning","Natural Language Processing: Embeddings","Natural Language Processing: Language Models"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986653","length_sec":86,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984796","length_sec":351,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Balažević et al., 2019] Ivana Balažević, Carl Allen, and Timothy M Hospedales. Tucker: Tensor factoriza- tion for knowledge graph completion. arXiv preprint arXiv:1901.09590, 2019.","title":"Tucker: Tensor factorization for knowledge graph completion","authors":"Ivana Balažević, Carl Allen, Timothy M Hospedales","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bordes et al., 2013] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Proc. of NeurIPS, 2013.","title":"Translating embeddings for modeling multi-relational data","authors":"Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, Oksana Yakhnenko","year":"2013","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Choi et al., 2021] Bonggeun Choi, Daesik Jang, and Youngjoong Ko. Mem-kgc: Masked entity model for knowledge graph completion with pre-trained language model. IEEE Access, 2021.","title":"Mem-kgc: Masked entity model for knowledge graph completion with pre-trained language model","authors":"Choi","year":"2021","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dettmers et al., 2018] Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In Proc. of AAAI, 2018.","title":"Convolutional 2d knowledge graph embeddings","authors":"Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, Sebastian Riedel","year":"2018","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kassner et al., 2020] Nora Kassner, Benno Krojer, and Hin- rich Schütze. Are pretrained language models symbolic reasoners over knowledge? In Proc. of CoNLL, 2020.","title":"Are pretrained language models symbolic reasoners over knowledge?","authors":"Nora Kassner, Benno Krojer, Hinrich Schütze","year":"2020","venue":"Proc. of CoNLL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lin et al., 2015] Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In Proc. of AAAI, 2015.","title":"Learning entity and relation embeddings for knowledge graph completion","authors":"Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, Xuan Zhu","year":"2015","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Miller, 1995] George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 1995.","title":"Wordnet: a lexical database for english","authors":"George A Miller","year":"1995","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Nickel et al., 2011] Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. A three-way model for collective learning on multi-relational data. In Icml, 2011.","title":"A three-way model for collective learning on multi-relational data","authors":"Maximilian Nickel, Hans-Peter Volker Tresp, Kriegel","year":"2011","venue":"Icml","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Pei et al., 2019] Shichao Pei, Lu Yu, Robert Hoehndorf, and Xiangliang Zhang. Semi-supervised entity alignment via knowledge graph embedding with awareness of degree dif- ference. In Proc. of WWW, 2019.","title":"Semi-supervised entity alignment via knowledge graph embedding with awareness of degree difference","authors":"Pei","year":"2019","venue":"Proc. of WWW","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Petroni et al., 2019] Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint arXiv:1909.01066, 2019.","title":"Language models as knowledge bases","authors":"Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, Sebastian Riedel","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Socher et al., 2013] Richard Socher, Danqi Chen, Christo- pher D Manning, and Andrew Ng. Reasoning with neural tensor networks for knowledge base completion. In Ad- vances in neural information processing systems, 2013.","title":"Reasoning with neural tensor networks for knowledge base completion","authors":"Richard Socher, Danqi Chen, Christopher D Manning, Andrew Ng","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Sun et al., 2019] Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint arXiv:1902.10197, 2019.","title":"Rotate: Knowledge graph embedding by relational rotation in complex space","authors":"Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, Jian Tang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Toutanova, 2015] Kristina Toutanova. Observed versus la- tent features for knowledge base and text inference. ACL- IJCNLP 2015, page 57, 2015.","title":"Observed versus latent features for knowledge base and text inference","authors":"Kristina Toutanova","year":"2015","venue":"ACL-IJCNLP 2015","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Trouillon et al., 2016] Théo Trouillon, Johannes Welbl, Se- bastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In Proc. of ICML, 2016.","title":"Complex embeddings for simple link prediction","authors":"Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, Guillaume Bouchard","year":"2016","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Turc et al., 2019] Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv:1908.08962, 2019.","title":"Well-read students learn better: On the importance of pre-training compact models","authors":"Iulia Turc, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing sys- tems, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2014] Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by translating on hyperplanes. In Proc. of AAAI, 2014.","title":"Knowledge graph embedding by translating on hyperplanes","authors":"Zhen Wang, Jianwen Zhang, Jianlin Feng, Zheng Chen","year":"2014","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2017] Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of ap- proaches and applications. IEEE Transactions on Knowl- edge and Data Engineering, 2017.","title":"Knowledge graph embedding: A survey of approaches and applications","authors":"Quan Wang, Zhendong Mao, Bin Wang, Li Guo","year":"2017","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2019a] Quan Wang, Pingping Huang, Haifeng Wang, Songtai Dai, Wenbin Jiang, Jing Liu, Yajuan Lyu, Yong Zhu, and Hua Wu. Coke: Contextualized knowl- edge graph embedding. arXiv preprint arXiv:1911.02168, 2019.","title":"Coke: Contextualized knowledge graph embedding","authors":"Wang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2019b] Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhiyuan Liu, Juanzi Li, and Jian Tang. Kepler: A uni- fied model for knowledge embedding and pre-trained lan- guage representation. arXiv preprint arXiv:1911.06136, 2019.","title":"Kepler: A unified model for knowledge embedding and pre-trained language representation","authors":"Wang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2020] Bo Wang, Tao Shen, Guodong Long, Tianyi Zhou, and Yi Chang. Structure-augmented text rep- resentation learning for efficient knowledge graph comple- tion. arXiv preprint arXiv:2004.14781, 2020.","title":"Structure-augmented text representation learning for efficient knowledge graph completion","authors":"Bo Wang, Tao Shen, Guodong Long, Tianyi Zhou, Yi Chang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xie et al., 2016] Ruobing Xie, Zhiyuan Liu, Jia Jia, Huanbo Luan, and Maosong Sun. Representation learning of knowledge graphs with entity descriptions. In Proc. of AAAI, 2016.","title":"Representation learning of knowledge graphs with entity descriptions","authors":"Ruobing Xie, Zhiyuan Liu, Jia Jia, Huanbo Luan, Maosong Sun","year":"2016","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang et al., 2014] Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575, 2014.","title":"Embedding entities and relations for learning and inference in knowledge bases","authors":"Bishan Yang, Wen-Tau Yih, Xiaodong He, Jianfeng Gao, Li Deng","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yao et al., 2019] Liang Yao, Chengsheng Mao, and Yuan Luo. Kg-bert: Bert for knowledge graph completion. arXiv preprint arXiv:1909.03193, 2019.","title":"Kg-bert: Bert for knowledge graph completion","authors":"Liang Yao, Chengsheng Mao, Yuan Luo","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yaser Jaradeh et al., 2021] Mohamad Yaser Jaradeh, Kuldeep Singh, Markus Stocker, and Sören Auer. Triple classification for scholarly knowledge graph completion. arXiv preprint arXiv:2111.11845, 2021.","title":"Triple classification for scholarly knowledge graph completion","authors":"Mohamad Yaser, Jaradeh, Kuldeep Singh, Markus Stocker, Sören Auer","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2020a] Zhanqiu Zhang, Jianyu Cai, Yongdong Zhang, and Jie Wang. Learning hierarchy-aware knowl- edge graph embeddings for link prediction. In Proc. of AAAI, 2020.","title":"Learning hierarchy-aware knowledge graph embeddings for link prediction","authors":"Zhang","year":"2020","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2020b] Zhiyuan Zhang, Xiaoqian Liu, Yi Zhang, Qi Su, Xu Sun, and Bin He. Pretrain-kge: Learning knowledge representation from pretrained language models. In Proc. of EMNLP, 2020.","title":"Pretrain-kge: Learning knowledge representation from pretrained language models","authors":"Zhang","year":"2020","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}