{"type":"ScholarlyArticle","title":"Unsupervised Embedding Enhancements of Knowledge Graphs using Textual Associations","authors":[{"name":"Neil Veira","slug":"neil-veira","given":"Neil","family":"Veira","affiliation":"Department of Electrical and Computer Engineering , University of Toronto"},{"name":"Brian Keng","slug":"brian-keng","given":"Brian","family":"Keng","affiliation":"Data Science , Rubikloud Technologies Inc."},{"name":"Kanchana Padmanabhan","slug":"kanchana-padmanabhan","given":"Kanchana","family":"Padmanabhan","affiliation":"Data Science , Rubikloud Technologies Inc."},{"name":"Andreas Veneris","slug":"andreas-veneris","given":"Andreas","family":"Veneris","affiliation":"Department of Electrical and Computer Engineering , University of Toronto"}],"abstract":"Knowledge graph embeddings are instrumental for representing and learning from multi-relational data, with recent embedding models showing high effectiveness for inferring new facts from existing databases. However, such precisely structured data is usually limited in quantity and in scope. Therefore, to fully optimize the embeddings it is important to also consider more widely available sources of information such as text. This paper describes an unsupervised approach to incorporate textual information by augmenting entity embeddings with embeddings of associated words. The approach does not modify the optimization objective for the knowledge graph embedding, which allows it to be integrated with existing embedding models. Two distinct forms of textual data are considered, with different embedding enhancements proposed for each case. In the first case, each entity has an associated text document that describes it. In the second case, a text document is not available, and instead entities occur as words or phrases in an unstructured corpus of text fragments. Experiments show that both methods can offer improvement on the link prediction task when applied to many different knowledge graph embedding models.","doi":"10.24963/ijcai.2019/725","url":"https://ijcai.org/proceedings/2019/725","pdf":"https://ijcai.org/proceedings/2019/0725.pdf","markdown":"https://ijcai.org/proceedings/2019/725.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/725","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":725,"pages":{"first":5218,"last":5225},"keywords":["Natural Language Processing: Embeddings","Natural Language Processing: Knowledge Extraction"],"videos":[],"references":[{"position":1,"raw":"[Auer et al., 2007] Sören Auer, Christian Bizer, Georgi Ko- bilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. Dbpedia: A nucleus for a web of open data. In The semantic web, pages 722-735. Springer, 2007.","title":"Dbpedia: A nucleus for a web of open data","authors":"Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, Zachary Ives","year":"2007","venue":"The semantic web","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bollacker et al., 2008] Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247-1250. AcM, 2008.","title":"Freebase: a collaboratively created graph database for structuring human knowledge","authors":"Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, Jamie Taylor","year":"2008","venue":"Proceedings of the 2008 ACM SIGMOD international conference on Management of data","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bordes et al., 2011] Antoine Bordes, Jason Weston, Ronan Collobert, Yoshua Bengio, et al. Learning structured em- beddings of knowledge bases. In AAAI, volume 6, page 6, 2011.","title":"Learning structured em-beddings of knowledge bases","authors":"Antoine Bordes, Jason Weston, Ronan Collobert, Yoshua Bengio","year":"2011","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bordes et al., 2013] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, pages 2787-2795, 2013.","title":"Translating embeddings for modeling multi-relational data","authors":"Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, Oksana Yakhnenko","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Duchi et al., 2011] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121-2159, 2011.","title":"Adaptive subgradient methods for online learning and stochastic optimization","authors":"John Duchi, Elad Hazan, Yoram Singer","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gardner et al., 2013] Matt Gardner, Partha Pratim Talukdar, Bryan Kisiel, and Tom Mitchell. Improving learning and inference in a large knowledge-base using latent syntactic cues. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 833-838, 2013.","title":"Improving learning and inference in a large knowledge-base using latent syntactic cues","authors":"Matt Gardner, Partha Pratim Talukdar, Bryan Kisiel, Tom Mitchell","year":"2013","venue":"Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gardner et al., 2014] Matt Gardner, Partha Talukdar, Jayant Krishnamurthy, and Tom Mitchell. Incorporating vector space similarity in random walk inference over knowledge bases. In Proceedings of the 2014 Conference on Empir- ical Methods in Natural Language Processing (EMNLP), pages 397-406, 2014.","title":"Incorporating vector space similarity in random walk inference over knowledge bases","authors":"Matt Gardner, Partha Talukdar, Jayant Krishnamurthy, Tom Mitchell","year":"2014","venue":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lao et al., 2012] Ni Lao, Amarnag Subramanya, Fernando Pereira, and William W Cohen. Reading the web with learned syntactic-semantic inference rules. In Proceed- ings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Nat- ural Language Learning, pages 1017-1026. Association for Computational Linguistics, 2012.","title":"Reading the web with learned syntactic-semantic inference rules","authors":"Ni Lao, Amarnag Subramanya, Fernando Pereira, William W Cohen","year":"2012","venue":"Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lin et al., 2015] Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In AAAI, volume 15, pages 2181-2187, 2015.","title":"Learning entity and relation embeddings for knowledge graph completion","authors":"Lin","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositional- ity. In Advances in neural information processing systems, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Miller, 1995] George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-41, 1995.","title":"Wordnet: a lexical database for english","authors":"George A Miller","year":"1995","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mintz et al., 2009] Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. Distant supervision for relation extrac- tion without labeled data. In Proceedings of the Joint Con- ference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Pro- cessing of the AFNLP: Volume 2-Volume 2, pages 1003- 1011. Association for Computational Linguistics, 2009.","title":"Distant supervision for relation extraction without labeled data","authors":"Mike Mintz, Steven Bills, Rion Snow, Dan Jurafsky","year":"2009","venue":"Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Nickel et al., 2011] Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. A three-way model for collective learning on multi-relational data. In ICML, volume 11, pages 809-816, 2011.","title":"A three-way model for collective learning on multi-relational data","authors":"Maximilian Nickel, Hans-Peter Volker Tresp, Kriegel","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Nickel et al., 2016a] Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. A review of re- lational machine learning for knowledge graphs. Proceed- ings of the IEEE, 104(1):11-33, 2016.","title":"A review of re-lational machine learning for knowledge graphs","authors":"Nickel","year":"2016","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nickel et al., 2016b] Maximilian Nickel, Lorenzo Rosasco, Tomaso A Poggio, et al. Holographic embeddings of knowledge graphs. In AAAI, volume 2, pages 3-2, 2016.","title":"Holographic embeddings of knowledge graphs","authors":"Nickel","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Riedel et al., 2010] Sebastian Riedel, Limin Yao, and An- drew McCallum. Modeling relations and their mentions without labeled text. In Joint European Conference on Ma- chine Learning and Knowledge Discovery in Databases, pages 148-163. Springer, 2010.","title":"Modeling relations and their mentions without labeled text","authors":"Sebastian Riedel, Limin Yao, Andrew Mccallum","year":"2010","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Riedel et al., 2013] Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M Marlin. Relation extraction with matrix factorization and universal schemas. In Pro- ceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 74-84, 2013.","title":"Relation extraction with matrix factorization and universal schemas","authors":"Sebastian Riedel, Limin Yao, Andrew Mccallum, Benjamin M Marlin","year":"2013","venue":"Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Socher et al., 2013] Richard Socher, Danqi Chen, Christo- pher D Manning, and Andrew Ng. Reasoning with neu- ral tensor networks for knowledge base completion. In Advances in neural information processing systems, pages 926-934, 2013.","title":"Reasoning with neural tensor networks for knowledge base completion","authors":"Richard Socher, Danqi Chen, Christopher D Manning, Andrew Ng","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Suchanek et al., 2007] Fabian M Suchanek, Gjergji Kas- neci, and Gerhard Weikum. Yago: a core of semantic knowledge. In Proceedings of the 16th international con- ference on World Wide Web, pages 697-706. ACM, 2007.","title":"Yago: a core of semantic knowledge","authors":"Gjergji Fabian M Suchanek, Gerhard Kasneci, Weikum","year":"2007","venue":"Proceedings of the 16th international conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Surdeanu et al., 2012] Mihai Surdeanu, Julie Tibshirani, Ramesh Nallapati, and Christopher D Manning. Multi- instance multi-label learning for relation extraction. In Proceedings of the 2012 joint conference on empirical methods in natural language processing and computa- tional natural language learning, pages 455-465. Asso- ciation for Computational Linguistics, 2012.","title":"Multiinstance multi-label learning for relation extraction","authors":"Mihai Surdeanu, Julie Tibshirani, Ramesh Nallapati, Christopher D Manning","year":"2012","venue":"Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Toutanova et al., 2015] Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. Representing text for joint embedding of text and knowledge bases. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1499-1509, 2015.","title":"Representing text for joint embedding of text and knowledge bases","authors":"Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, Michael Gamon","year":"2015","venue":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2014] Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph and text jointly embedding. In Proceedings of the 2014 conference on em- pirical methods in natural language processing (EMNLP), pages 1591-1601, 2014.","title":"Knowledge graph and text jointly embedding","authors":"Zhen Wang, Jianwen Zhang, Jianlin Feng, Zheng Chen","year":"2014","venue":"Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2017] Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of ap- proaches and applications. IEEE Transactions on Knowl- edge and Data Engineering, 29(12):2724-2743, 2017.","title":"Knowledge graph embedding: A survey of approaches and applications","authors":"Quan Wang, Zhendong Mao, Bin Wang, Li Guo","year":"2017","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Weston et al., 2013] Jason Weston, Antoine Bordes, Ok- sana Yakhnenko, and Nicolas Usunier. Connecting lan- guage and knowledge bases with embedding models for relation extraction. arXiv preprint arXiv:1307.7973, 2013.","title":"Connecting language and knowledge bases with embedding models for relation extraction","authors":"Jason Weston, Antoine Bordes, Oksana Yakhnenko, Nicolas Usunier","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2014] Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575, 2014.","title":"Embedding entities and relations for learning and inference in knowledge bases","authors":"Bishan Yang, Wen-Tau Yih, Xiaodong He, Jianfeng Gao, Li Deng","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}