{"type":"ScholarlyArticle","title":"Gaussian Embedding of Linked Documents from a Pretrained Semantic Space","authors":[{"name":"Antoine Gourru","slug":"antoine-gourru","given":"Antoine","family":"Gourru","affiliation":"Université de Lyon , Lyon 2 ,"},{"name":"Julien Velcin","slug":"julien-velcin","given":"Julien","family":"Velcin","affiliation":"Université de Lyon , Lyon 2 ,"},{"name":"Julien Jacques","slug":"julien-jacques","given":"Julien","family":"Jacques","affiliation":"Université de Lyon , Lyon 2 ,; ERIC UR3083"}],"abstract":"Gaussian Embedding of Linked Documents (GELD) is a new method that embeds linked documents (e.g., citation networks) onto a pretrained semantic space (e.g., a set of word embeddings).\r\nWe formulate the problem in such a way that we model each document as a Gaussian distribution in the word vector space.\r\nWe design a generative model that combines both words and links in a consistent way.\r\nLeveraging the variance of a document allows us to model the uncertainty related to word and link generation.\r\nIn most cases, our method outperforms state-of-the-art methods when using our document vectors as features for usual downstream tasks.\r\nIn particular, GELD achieves better accuracy in classification and link prediction on Cora and Dblp. In addition, we demonstrate qualitatively the convenience of several properties of our method. We provide the implementation of GELD and the evaluation datasets to the community (https://github.com/AntoineGourru/DNEmbedding).","doi":"10.24963/ijcai.2020/541","url":"https://ijcai.org/proceedings/2020/541","pdf":"https://ijcai.org/proceedings/2020/0541.pdf","markdown":"https://ijcai.org/proceedings/2020/541.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/541","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":541,"pages":{"first":3912,"last":3918},"keywords":["Machine Learning: Learning Generative Models","Machine Learning: Probabilistic Machine Learning","Natural Language Processing: Embeddings"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25113/file_25113.m3u8","length_sec":297,"provider":"hls"}],"references":[{"position":1,"raw":"[Barkan, 2017] Oren Barkan. Bayesian neural word embed- ding. In Thirty-First AAAI Conference on Artificial Intel- ligence, pages 3135-3143, 2017.","title":"Bayesian neural word embedding","authors":"Oren Barkan","year":"2017","venue":"Thirty-First AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bojchevski and Günnemann, 2018] Aleksandar Bojchevski and Stephan Günnemann. Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking. In Proceeding of the International Conference on Learning Representations, ICLR, 2018.","title":"Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking","authors":"Aleksandar Bojchevski, Stephan Günnemann","year":"2018","venue":"Proceeding of the International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brochier et al., 2019] Robin Brochier, Adrien Guille, and Julien Velcin. Global vectors for node representations. In Proceedings of the World Wide Web Conference, WWW, pages 2587-2593, 2019.","title":"Global vectors for node representations","authors":"Brochier","year":"2019","venue":"Proceedings of the World Wide Web Conference","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Das et al., 2015] Rajarshi Das, Manzil Zaheer, and Chris Dyer. Gaussian lda for topic models with word embed- dings. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th In- ternational Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pages 795-804, 2015.","title":"Gaussian lda for topic models with word embeddings","authors":"Rajarshi Das, Manzil Zaheer, Chris Dyer","year":"2015","venue":"Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deerwester et al., 1990] Scott Deerwester, Susan T Du- mais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Jour- nal of the American society for information science, 41(6):391-407, 1990.","title":"Indexing by latent semantic analysis","authors":"Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, Richard Harshman","year":"1990","venue":"Journal of the American society for information science","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gourru et al., 2020] Antoine Gourru, Adrien Guille, Julien Velcin, and Julien Jacques. Document network projection in pretrained word embedding space. In European Confer- ence on Information Retrieval, pages 150-157. Springer, 2020.","title":"Document network projection in pretrained word embedding space","authors":"Antoine Gourru, Adrien Guille, Julien Velcin, Julien Jacques","year":"2020","venue":"European Conference on Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2017] Xiao Huang, Jundong Li, and Xia Hu. Accelerated attributed network embedding. In Proceed- ings of the SIAM International Conference on Data Min- ing, SDM, pages 633-641, 2017.","title":"Accelerated attributed network embedding","authors":"Xiao Huang, Jundong Li, Xia Hu","year":"2017","venue":"Proceedings of the SIAM International Conference on Data Mining, SDM","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kipf and Welling, 2016] Thomas N Kipf and Max Welling. Variational graph auto-encoders. In Bayesian Deep Learn- ing Workshop, BDL-NeurIPS, 2016.","title":"Variational graph auto-encoders","authors":"N Thomas, Max Kipf, Welling","year":"2016","venue":"Bayesian Deep Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Le and Mikolov, 2014] Quoc Le and Tomas Mikolov. Dis- tributed representations of sentences and documents. In International Conference on Machine Learning, pages 1188-1196, 2014.","title":"Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents","authors":"Mikolov Le","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2018] Jie Liu, Zhicheng He, Lai Wei, and Yalou Huang. Content to node: Self-translation network embed- ding. In Proceedings of the 24th ACM SIGKDD Interna- tional Conference on Knowledge Discovery & Data Min- ing, pages 1794-1802. ACM, 2018.","title":"Content to node: Self-translation network embedding","authors":"Jie Liu, Zhicheng He, Lai Wei, Yalou Huang","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Meng et al., 2019] Zaiqiao Meng, Shangsong Liang, Hongyan Bao, and Xiangliang Zhang. Co-embedding attributed networks. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pages 393-401, 2019.","title":"Co-embedding attributed networks","authors":"Meng","year":"2019","venue":"Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositional- ity. In Advances in neural information processing systems, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Nikolentzos et al., 2017] Giannis Nikolentzos, Polykarpos Meladianos, Franc ¸ois Rousseau, Yannis Stavrakas, and Michalis Vazirgiannis. Multivariate gaussian document representation from word embeddings for text categoriza- tion. In Proceedings of the 15th Conference of the Euro- pean Chapter of the Association for Computational Lin- guistics: Volume 2, Short Papers, pages 450-455, 2017.","title":"Multivariate gaussian document representation from word embeddings for text categorization","authors":"Giannis Nikolentzos, Polykarpos Meladianos, Yannis Franc ¸ois Rousseau, Michalis Stavrakas, Vazirgiannis","year":"2017","venue":"Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pan et al., 2016] Shirui Pan, Jia Wu, Xingquan Zhu, Chengqi Zhang, and Yang Wang. Tri-party deep network representation. In Proceedings of the International Joint Conference on Artificial Intelligence, IJCAI, pages 1895- 1901, 2016.","title":"Tri-party deep network representation","authors":"Pan","year":"2016","venue":"Proceedings of the International Joint Conference on Artificial Intelligence, IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/271"},{"position":16,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Perozzi et al., 2014] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social repre- sentations. In Proceedings of the 20th ACM SIGKDD in- ternational conference on Knowledge discovery and data mining, pages 701-710. ACM, 2014.","title":"Deepwalk: Online learning of social representations","authors":"Bryan Perozzi, Rami Al-Rfou, Steven Skiena","year":"2014","venue":"Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Robbins and Monro, 1951] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951.","title":"The annals of mathematical statistics","authors":"Monro Robbins, Herbert Robbins, Sutton Monro","year":"1951","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tang et al., 2008] Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. Arnetminer: extraction and mining of academic social networks. In Proceedings of the ACM SIGKDD International Conference on Knowl- edge Discovery and Data Mining, KDD, pages 990-998, 2008.","title":"Arnetminer: extraction and mining of academic social networks","authors":"Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, Zhong Su","year":"2008","venue":"Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tu et al., 2017] Cunchao Tu, Han Liu, Zhiyuan Liu, and Maosong Sun. Cane: Context-aware network embedding for relation modeling. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1722-1731, 2017.","title":"Cane: Context-aware network embedding for relation modeling","authors":"Tu","year":"2017","venue":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2016] Suhang Wang, Jiliang Tang, Charu Ag- garwal, and Huan Liu. Linked document embedding for classification. In Proceedings of the 25th ACM Interna- tional on Conference on Information and Knowledge Man- agement, pages 115-124. ACM, 2016.","title":"Linked document embedding for classification","authors":"Wang","year":"2016","venue":"Proceedings of the 25th ACM International on Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang, 2001] Steven Xiaogang Wang. Maximum weighted likelihood estimation. PhD thesis, University of British Columbia, 2001.","title":"Maximum weighted likelihood estimation","authors":"Steven Xiaogang, Wang","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2015] Cheng Yang, Zhiyuan Liu, Deli Zhao, Maosong Sun, and Edward Y Chang. Network represen- tation learning with rich text information. In International Joint Conference on Artificial Intelligence, 2015.","title":"Network representation learning with rich text information","authors":"Cheng Yang, Zhiyuan Liu, Deli Zhao, Maosong Sun, Edward Y Chang","year":"2015","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2015/299"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}