{"type":"ScholarlyArticle","title":"ProNE: Fast and Scalable Network Representation Learning","authors":[{"name":"Jie Zhang","slug":"jie-zhang","given":"Jie","family":"Zhang","affiliation":"Department of Computer Science and Technology , Tsinghua University"},{"name":"Yuxiao Dong","slug":"yuxiao-dong","given":"Yuxiao","family":"Dong","affiliation":"Microsoft Research , Redmond {j-z16 ,"},{"name":"Yan Wang","slug":"yan-wang","given":"Yan","family":"Wang","affiliation":"Department of Computer Science and Technology , Tsinghua University"},{"name":"Jie Tang","slug":"jie-tang","given":"Jie","family":"Tang","affiliation":"Department of Computer Science and Technology , Tsinghua University"},{"name":"Ming Ding","slug":"ming-ding","given":"Ming","family":"Ding","affiliation":"Department of Computer Science and Technology , Tsinghua University"}],"abstract":"Recent advances in network embedding has revolutionized the field of graph and network mining. However, (pre-)training embeddings for very large-scale networks is computationally challenging for most existing methods. In this work, we present ProNE---a fast, scalable, and effective model, whose single-thread version is 10--400x faster than efficient network embedding benchmarks with 20 threads, including LINE, DeepWalk, node2vec, GraRep, and HOPE. As a concrete example, the single-version ProNE requires only 29 hours to embed a network of hundreds of millions of nodes while it takes LINE weeks and DeepWalk months by using 20 threads. To achieve this, ProNE first initializes network embeddings efficiently by formulating the task as sparse matrix factorization. The second step of ProNE is to enhance the embeddings by propagating them in the spectrally modulated space. Extensive experiments on networks of various scales and types demonstrate that ProNE achieves both effectiveness and significant efficiency superiority when compared to the aforementioned baselines. In addition, ProNE's embedding enhancement step can be also generalized for improving other models at speed, e.g., offering >10% relative gains for the used baselines.","doi":"10.24963/ijcai.2019/594","url":"https://ijcai.org/proceedings/2019/594","pdf":"https://ijcai.org/proceedings/2019/0594.pdf","markdown":"https://ijcai.org/proceedings/2019/594.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/594","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":594,"pages":{"first":4278,"last":4284},"keywords":["Machine Learning: Data Mining","Machine Learning Applications: Networks"],"videos":[],"references":[{"position":1,"raw":"[Andrews and Andrews, 1992] Larry C Andrews and Larry C Andrews. Special functions of mathematics for engineers. McGraw-Hill New York, 1992.","title":"Special functions of mathematics for engineers","authors":"C Larry, Larry C Andrews, Andrews","year":"1992","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bandeira et al., 2013] Afonso S Bandeira, Amit Singer, and Daniel A Spielman. A cheeger inequality for the graph connection laplacian. SIAM Journal on Matrix Analysis and Applications, 34(4):1611-1630, 2013.","title":"A cheeger inequality for the graph connection laplacian","authors":"Amit Afonso S Bandeira, Daniel A Singer, Spielman","year":"2013","venue":"SIAM Journal on Matrix Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Belkin and Niyogi, 2001] Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps and spectral techniques for embedding and clustering. In NIPS, pages 585-591, 2001.","title":"Laplacian eigenmaps and spectral techniques for embedding and clustering","authors":"Niyogi Belkin, Mikhail Belkin, Partha Niyogi","year":"2001","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Breitkreutz et al., 2008] Bobby-Joe Breitkreutz, Chris Stark, et al. The biogrid interaction database. Nucleic acids research, 36(suppl 1):D637-D640, 2008.","title":"The biogrid interaction database","authors":"Bobby-Joe Breitkreutz, Chris Stark","year":"2008","venue":"Nucleic acids research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Buluc ¸and Gilbert, 2012] Aydin Buluc ¸and John R Gilbert. Parallel sparse matrix-matrix multiplication and indexing: Implementation and experiments. SIAM Journal on Scien- tific Computing, 34(4):C170-C191, 2012.","title":"Parallel sparse matrix-matrix multiplication and indexing: Implementation and experiments","authors":"Aydin Buluc, R John, Gilbert","year":"2012","venue":"SIAM Journal on Scientific Computing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cao et al., 2015] Shaosheng Cao, Wei Lu, and Qiongkai Xu. Grarep: Learning graph representations with global structural information. In CIKM, pages 891-900, 2015.","title":"Grarep: Learning graph representations with global structural information","authors":"Cao","year":"2015","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chung, 1997] Fan RK Chung. Spectral graph theory. Num- ber 92. American Mathematical Soc., 1997.","title":"Spectral graph theory","authors":"Fan Rk, Chung","year":"1997","venue":"Number 92","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Defferrard et al., 2016] Michaël Defferrard, Xavier Bres- son, and Pierre Vandergheynst. Convolutional neural net- works on graphs with fast localized spectral filtering. In NIPS, pages 3844-3852, 2016.","title":"Convolutional neural networks on graphs with fast localized spectral filtering","authors":"Michaël Defferrard, Xavier Bresson, Pierre Vandergheynst","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dong et al., 2017] Yuxiao Dong, Nitesh V Chawla, and Ananthram Swami. metapath2vec: Scalable representa- tion learning for heterogeneous networks. In KDD, pages 135-144. ACM, 2017.","title":"metapath2vec: Scalable representation learning for heterogeneous networks","authors":"Dong","year":"2017","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Grover and Leskovec, 2016] Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In KDD, pages 855-864, 2016.","title":"Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks","authors":"Leskovec Grover","year":"2016","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Halko et al., 2011] Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate ma- trix decompositions. SIAM review, 53(2):217-288, 2011.","title":"Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions","authors":"Halko","year":"2011","venue":"SIAM review","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hamilton et al., 2017a] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NIPS, pages 1025-1035, 2017.","title":"Inductive representation learning on large graphs","authors":"Hamilton","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hamilton et al., 2017b] William L. Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. IEEE Data(base) Engineering Bulletin, 40:52-74, 2017.","title":"Representation learning on graphs: Methods and applications","authors":"Hamilton","year":"2017","venue":"IEEE Data(base) Engineering Bulletin","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hammond et al., 2011] David K Hammond, Pierre Van- dergheynst, and Rémi Gribonval. Wavelets on graphs via spectral graph theory. Applied and Computational Har- monic Analysis, 30(2):129-150, 2011.","title":"Wavelets on graphs via spectral graph theory","authors":"Pierre David K Hammond, Rémi Vandergheynst, Gribonval","year":"2011","venue":"Applied and Computational Harmonic Analysis","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Harris, 1954] Zellig S Harris. Distributional structure. Word, 10(2-3):146-162, 1954.","title":"Distributional structure","authors":"S Zellig, Harris","year":"1954","venue":"Word","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Henaff et al., 2015] Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutional networks on graph-structured data. arXiv preprint arXiv:1506.05163, 2015.","title":"Deep convolutional networks on graph-structured data","authors":"Mikael Henaff, Joan Bruna, Yann Lecun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kipf and Welling, 2017] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.","title":"Semi-supervised classification with graph convolutional networks","authors":"N Thomas, Max Kipf, Welling","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lee et al., 2014] James R Lee, Shayan Oveis Gharan, and Luca Trevisan. Multiway spectral partitioning and higher- order cheeger inequalities. JACM, 61(6):37, 2014.","title":"Multiway spectral partitioning and higherorder cheeger inequalities","authors":"Shayan James R Lee, Luca Oveis Gharan, Trevisan","year":"2014","venue":"JACM","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Levy and Goldberg, 2014] Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. In NIPS, pages 2177-2185, 2014.","title":"Neural word embedding as implicit matrix factorization","authors":"Goldberg Levy, Omer Levy, Yoav Goldberg","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mikolov et al., 2013a] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In ICLR Workshop, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Mikolov","year":"2013","venue":"ICLR Workshop","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mikolov et al., 2013b] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. In NIPS, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ou et al., 2016] Mingdong Ou, Peng Cui, Jian Pei, Ziwei Zhang, and Wenwu Zhu. Asymmetric transitivity preserv- ing graph embedding. In KDD, pages 1105-1114, 2016.","title":"Asymmetric transitivity preserving graph embedding","authors":"Mingdong Ou, Peng Cui, Jian Pei, Ziwei Zhang, Wenwu Zhu","year":"2016","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Perozzi et al., 2014] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social rep- resentations. In KDD, pages 701-710, 2014.","title":"Deepwalk: Online learning of social representations","authors":"Bryan Perozzi, Rami Al-Rfou, Steven Skiena","year":"2014","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Qiu et al., 2018] Jiezhong Qiu, Yuxiao Dong, Hao Ma, Jian Li, Kuansan Wang, and Jie Tang. Network embedding as matrix factorization: Unifying deepwalk, line, pte, and node2vec. In WSDM, pages 459-467, 2018.","title":"Network embedding as matrix factorization: Unifying deepwalk, line, pte, and node2vec","authors":"Qiu","year":"2018","venue":"WSDM","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shuman et al., 2016] David I Shuman, Benjamin Ricaud, and Pierre Vandergheynst. Vertex-frequency analysis on graphs. Applied and Computational Harmonic Analysis, 40(2):260-291, 2016.","title":"Vertex-frequency analysis on graphs","authors":"Benjamin David I Shuman, Pierre Ricaud, Vandergheynst","year":"2016","venue":"Applied and Computational Harmonic Analysis","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Smith et al., 2015] Shaden Smith, Niranjay Ravindran, Nicholas D Sidiropoulos, and George Karypis. Splatt: Ef- ficient and parallel sparse tensor-matrix multiplication. In IPDPS, pages 61-70. IEEE, 2015.","title":"Splatt: Efficient and parallel sparse tensor-matrix multiplication","authors":"Shaden Smith, Niranjay Ravindran, Nicholas D Sidiropoulos, George Karypis","year":"2015","venue":"IPDPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tang and Liu, 2009] Lei Tang and Huan Liu. Relational learning via latent social dimensions. In KDD, 2009.","title":"Relational learning via latent social dimensions","authors":"Lei Tang, Huan Liu","year":"2009","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tang et al., 2008] Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. Arnetminer: extraction and mining of academic social networks. In KDD, pages 990- 998, 2008.","title":"Arnetminer: extraction and mining of academic social networks","authors":"Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, Zhong Su","year":"2008","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tang et al., 2015] Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Large- scale information network embedding. In WWW, 2015.","title":"Line: Largescale information network embedding","authors":"Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, Qiaozhu Mei","year":"2015","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Veličković et al., 2018] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.","title":"Graph attention networks","authors":"Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, Yoshua Bengio","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Von Luxburg, 2007] Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and computing, 17(4):395- 416, 2007.","title":"A tutorial on spectral clustering","authors":"Ulrike Von, Luxburg","year":"2007","venue":"Statistics and computing","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yan et al., 2009] Donghui Yan, Ling Huang, and Michael I Jordan. Fast approximate spectral clustering. In KDD, pages 907-916, 2009.","title":"Fast approximate spectral clustering","authors":"Donghui Yan, Ling Huang, Michael I Jordan","year":"2009","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zafarani and Liu, 2009] Reza Zafarani and Huan Liu. So- cial computing data repository at asu, 2009.","title":"Social computing data repository at asu","authors":"Reza Zafarani, Huan Liu","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}