{"type":"ScholarlyArticle","title":"Robust Embedding with Multi-Level Structures for Link Prediction","authors":[{"name":"Zihan Wang","slug":"zihan-wang","given":"Zihan","family":"Wang","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences; School of Cyber Security , University of Chinese Academy of Sciences"},{"name":"Zhaochun Ren","slug":"zhaochun-ren","given":"Zhaochun","family":"Ren","affiliation":"Shandong University"},{"name":"Chunyu He","slug":"chunyu-he","given":"Chunyu","family":"He","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences; School of Cyber Security , University of Chinese Academy of Sciences"},{"name":"Peng Zhang","slug":"peng-zhang","given":"Peng","family":"Zhang","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences; School of Cyber Security , University of Chinese Academy of Sciences"},{"name":"Yue Hu","slug":"yue-hu","given":"Yue","family":"Hu","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences; School of Cyber Security , University of Chinese Academy of Sciences"}],"abstract":"Knowledge Graph (KG) embedding has become crucial for the task of link prediction. Recent work applies encoder-decoder models to tackle this problem, where an encoder is formulated as a graph neural network (GNN) and a decoder is represented by an embedding method. These approaches enforce embedding techniques with structure information. Unfortunately, existing GNN-based frameworks still confront 3 severe problems: low representational power, stacking in a flat way, and poor robustness to noise. In this work, we propose a novel multi-level graph neural network (M-GNN) to address the above challenges. We first identify an injective aggregate scheme and design a powerful GNN layer using multi-layer perceptrons (MLPs). Then, we define graph coarsening schemes for various kinds of relations, and stack GNN layers on a series of coarsened graphs, so as to model hierarchical structures. Furthermore, attention mechanisms are adopted so that our approach can make predictions accurately even on the noisy knowledge graph. Results on WN18 and FB15k datasets show that our approach is effective in the standard link prediction task, significantly and consistently outperforming competitive baselines. Furthermore, robustness analysis on FB15k-237 dataset demonstrates that our proposed M-GNN is highly robust to sparsity and noise.","doi":"10.24963/ijcai.2019/728","url":"https://ijcai.org/proceedings/2019/728","pdf":"https://ijcai.org/proceedings/2019/0728.pdf","markdown":"https://ijcai.org/proceedings/2019/728.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/728","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":728,"pages":{"first":5240,"last":5246},"keywords":["Natural Language Processing: Natural Language Processing","Natural Language Processing: Embeddings"],"videos":[],"references":[{"position":1,"raw":"[Auer et al., 2007] Sören Auer, Christian Bizer, Georgi Ko- bilarov, Jens Lehmann, Richard Cyganiak, and Zachary G. Ives. Dbpedia: A nucleus for a web of open data. In 6th International Semantic Web Conference, pages 722-735, Busan, Korea, November 2007.","title":"Dbpedia: A nucleus for a web of open data","authors":"Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, Zachary G Ives","year":"2007","venue":"6th International Semantic Web Conference","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bordes et al., 2013] Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, pages 2787-2795, Nevada, United States, December 2013.","title":"Translating embeddings for modeling multi-relational data","authors":"Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, Oksana Yakhnenko","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bordes et al., 2015] Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. Large-scale simple question answering with memory networks. arXiv preprint arXiv:1506.02075, 2015.","title":"Large-scale simple question answering with memory networks","authors":"Antoine Bordes, Nicolas Usunier, Sumit Chopra, Jason Weston","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chang et al., 2014] Kai-Wei Chang, Wen-tau Yih, Bishan Yang, and Christopher Meek. Typed tensor decomposition of knowledge bases for relation extraction. In Proceedings of the 2014 Conference on Empirical Methods in Natu- ral Language Processing, pages 1568-1579, Doha, Qatar, October 2014.","title":"Typed tensor decomposition of knowledge bases for relation extraction","authors":"Kai-Wei Chang, Wen-Tau Yih, Bishan Yang, Christopher Meek","year":"2014","venue":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Defferrard et al., 2016] Michaël Defferrard, Xavier Bres- son, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems, pages 3837-3845, Barcelona, Spain, December 2016.","title":"Convolutional neural networks on graphs with fast localized spectral filtering","authors":"Michaël Defferrard, Xavier Bresson, Pierre Vandergheynst","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dong et al., 2014] Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. Knowledge vault: a web-scale approach to probabilistic knowledge fusion. In The 20th International Confer- ence on Knowledge Discovery and Data Mining, pages 601-610, New York, United States, August 2014.","title":"Knowledge vault: a web-scale approach to probabilistic knowledge fusion","authors":"Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, Wei Zhang","year":"2014","venue":"The 20th International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Duvenaud et al., 2015] David K. Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael Gómez- Bombarelli, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P. Adams. Convolutional networks on graphs for learning molecular fingerprints. In Advances in neural in- formation processing systems, pages 2224-2232, Quebec, Canada, December 2015.","title":"Convolutional networks on graphs for learning molecular fingerprints","authors":"David K Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael Gómez-Bombarelli, Timothy Hirzel, Alán Aspuru-Guzik, Ryan P Adams","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gilmer et al., 2017] Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning, pages 1263-1272, Sydney, Australia, August 2017.","title":"Neural message passing for quantum chemistry","authors":"Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, George E Dahl","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Henaff et al., 2015] Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutional networks on graph-structured data. arXiv preprint arXiv:1506.05163, 2015.","title":"Deep convolutional networks on graph-structured data","authors":"Mikael Henaff, Joan Bruna, Yann Lecun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hornik et al., 1989] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359-366, 1989.","title":"Multilayer feedforward networks are universal approximators","authors":"Kurt Hornik, Maxwell Stinchcombe, Halbert White","year":"1989","venue":"Neural networks","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd In- ternational Conference on Learning Representations, San Diego, United States, May 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"3rd In-ternational Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin et al., 2015] Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In Proceed- ings of the 29th AAAI Conference on Artificial Intelligence, pages 2181-2187, Texas, United States, January 2015.","title":"Learning entity and relation embeddings for knowledge graph completion","authors":"Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, Xuan Zhu","year":"2015","venue":"Proceedings of the 29th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Neil et al., 2018] Daniel Neil, Joss Briody, Alix Lacoste, Aaron Sim, Paidi Creed, and Amir Saffari. Inter- pretable graph convolutional neural networks for in- ference on noisy knowledge graphs. arXiv preprint arXiv:1812.00279, 2018.","title":"Interpretable graph convolutional neural networks for inference on noisy knowledge graphs","authors":"Daniel Neil, Joss Briody, Alix Lacoste, Aaron Sim, Paidi Creed, Amir Saffari","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pujara et al., 2017] Jay Pujara, Eriq Augustine, and Lise Getoor. Sparsity and noise: Where knowledge graph em- beddings fall short. In Proceedings of the 2017 Confer- ence on Empirical Methods in Natural Language Process- ing, pages 1751-1756, Copenhagen, Denmark, September 2017.","title":"Sparsity and noise: Where knowledge graph embeddings fall short","authors":"Jay Pujara, Eriq Augustine, Lise Getoor","year":"2017","venue":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Schlichtkrull et al., 2018] Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In The Semantic Web -15th International Conference, pages 593-607, Heraklion, Greece, June 2018.","title":"Modeling relational data with graph convolutional networks","authors":"Sejr Michael, Thomas N Schlichtkrull, Peter Kipf, Rianne Bloem, Van Den, Ivan Berg, Max Titov, Welling","year":"2018","venue":"The Semantic Web -15th International Conference","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Toutanova and Chen, 2015] Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Composition- ality, pages 57-66, 2015.","title":"Observed versus latent features for knowledge base and text inference","authors":"Chen ; Toutanova, Kristina Toutanova, Danqi Chen","year":"2015","venue":"Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Trouillon et al., 2016] Théo Trouillon, Johannes Welbl, Se- bastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In Inter- national Conference on Machine Learning, pages 2071- 2080, 2016.","title":"Complex embeddings for simple link prediction","authors":"Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, Guillaume Bouchard","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2014] Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by translating on hyperplanes. In Proceedings of the 28th AAAI Conference on Artificial Intelligence, pages 1112- 1119, Québec, Canada, July 2014.","title":"Knowledge graph embedding by translating on hyperplanes","authors":"Zhen Wang, Jianwen Zhang, Jianlin Feng, Zheng Chen","year":"2014","venue":"Proceedings of the 28th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Xiong and Callan, 2015] Chenyan Xiong and Jamie Callan. Esdrank: Connecting query and documents through exter- nal semi-structured data. In Proceedings of the 24th ACM International Conference on Information and Knowledge Management, pages 951-960, Melbourne,Australia, Octo- ber 2015.","title":"Esdrank: Connecting query and documents through external semi-structured data","authors":"Callan Xiong, Chenyan Xiong, Jamie Callan","year":"2015","venue":"Proceedings of the 24th ACM International Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xu et al., 2018] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural net- works? arXiv preprint arXiv:1810.00826, 2018.","title":"How powerful are graph neural networks","authors":"Xu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2015] Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. In 3rd International Conference on Learning Representa- tions,, San Diego, United States, May 2015.","title":"Embedding entities and relations for learning and inference in knowledge bases","authors":"Bishan Yang, Wen-Tau Yih, Xiaodong He, Jianfeng Gao, Li Deng","year":"2015","venue":"3rd International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}