{"type":"ScholarlyArticle","title":"Sancus: Staleness-Aware Communication-Avoiding Full-Graph Decentralized Training in Large-Scale Graph Neural Networks (Extended Abstract)","authors":[{"name":"Jingshu Peng","slug":"jingshu-peng","given":"Jingshu","family":"Peng","affiliation":"The Hong Kong University of Science and Technology"},{"name":"Zhao Chen","slug":"zhao-chen","given":"Zhao","family":"Chen","affiliation":"The Hong Kong University of Science and Technology"},{"name":"Yingxia Shao","slug":"yingxia-shao","given":"Yingxia","family":"Shao","affiliation":"Beijing University of Posts and Telecommunications"},{"name":"Yanyan Shen","slug":"yanyan-shen","given":"Yanyan","family":"Shen","affiliation":"Shanghai Jiao Tong University"},{"name":"Lei Chen","slug":"lei-chen","given":"Lei","family":"Chen","affiliation":"The Hong Kong University of Science and Technology"},{"name":"Jiannong Cao","slug":"jiannong-cao","given":"Jiannong","family":"Cao","affiliation":"The Hong Kong Polytechnic University"}],"abstract":"Graph neural networks (GNNs) have emerged due to their success at modeling graph data. Yet, it is challenging for GNNs to efficiently scale to large graphs. Thus, distributed GNNs come into play. To avoid communication caused by expensive data movement between workers, we propose SANCUS, a staleness-aware communication-avoiding decentralized GNN system. By introducing a set of novel bounded embedding staleness metrics and adaptively skipping broadcasts, SANCUS abstracts decentralized GNN processing as sequential matrix multiplication and uses historical embeddings via cache. Theoretically, we show bounded approximation errors of embeddings and gradients with convergence guarantee. Empirically, we evaluate SANCUS with common GNN models via different system setups on large-scale benchmark datasets. Compared to SOTA works, SANCUS can avoid up to 74% communication with at least 1:86_ faster throughput on average without accuracy loss.","doi":"10.24963/ijcai.2023/724","url":"https://ijcai.org/proceedings/2023/724","pdf":"https://ijcai.org/proceedings/2023/0724.pdf","markdown":"https://ijcai.org/proceedings/2023/724.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/724","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":724,"pages":{"first":6480,"last":6485},"keywords":["Sister Conferences Best Papers: Machine Learning","Sister Conferences Best Papers: Data Mining"],"videos":[],"references":[{"position":1,"raw":"[Abadal et al., 2022] Sergi Abadal, Akshay Jain, Robert Guirado, Jorge López-Alonso, and Eduard Alarcón. Com- puting graph neural networks: A survey from algorithms to accelerators. ACM Comput. Surv., 54(9):191:1-191:38, 2022.","title":"Computing graph neural networks: A survey from algorithms to accelerators","authors":"Akshay Sergi Abadal, Robert Jain, Jorge Guirado, Eduard López-Alonso, Alarcón","year":"2022","venue":"ACM Comput. Surv","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai et al., 2021] Zhenkun Cai, Xiao Yan, Yidi Wu, Kai- hao Ma, James Cheng, and Fan Yu. DGCL: an efficient communication library for distributed GNN training. In EuroSys '21: Sixteenth European Conference on Com- puter Systems, Online Event, United Kingdom, April 26- 28, 2021, pages 130-144. ACM, 2021.","title":"DGCL: an efficient communication library for distributed GNN training","authors":"Zhenkun Cai, Xiao Yan, Yidi Wu, Kaihao Ma, James Cheng, Fan Yu","year":"2021","venue":"EuroSys '21: Sixteenth European Conference on Computer Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018] Jianfei Chen, Jun Zhu, and Le Song. Stochastic training of graph convolutional networks with variance reduction. In ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Pro- ceedings of Machine Learning Research, pages 941-949.","title":"Stochastic training of graph convolutional networks with variance reduction","authors":"Jianfei Chen, Jun Zhu, Le Song","year":"2018","venue":"Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"PMLR, 2018.","title":null,"authors":"Pmlr","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"Cipar et al., ] James Cipar, Qirong Ho, Jin Kyu Kim, Se- unghak Lee, Gregory R. Ganger, Garth Gibson, Kimberly Keeton, and Eric P. Xing. Solving the straggler problem with bounded staleness. In HotOS XIV, Santa Ana Pueblo, New Mexico, USA, May 13-15, 2013. USENIX Associa- tion.","title":"Solving the straggler problem with bounded staleness","authors":"Cipar","year":"2013","venue":"HotOS XIV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fey et al., 2021] Matthias Fey, Jan Eric Lenssen, Frank We- ichert, and Jure Leskovec. Gnnautoscale: Scalable and ex- pressive graph neural networks via historical embeddings. In ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 3294-3304. PMLR, 2021.","title":"Gnnautoscale: Scalable and expressive graph neural networks via historical embeddings","authors":"Matthias Fey, Jan Eric Lenssen, Frank Weichert, Jure Leskovec","year":"2021","venue":"Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gandhi and Iyer, 2021] Swapnil Gandhi and Anand Pad- manabha Iyer. P3: distributed deep graph learning at scale. In 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021, July 14-16, 2021, pages 551-568. USENIX Association, 2021.","title":"P3: distributed deep graph learning at scale","authors":"Swapnil Gandhi, Anand Padmanabha Iyer","year":"2021","venue":"15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gholami et al., 2018] Amir Gholami, Ariful Azad, Peter H. Jin, Kurt Keutzer, and Aydin Buluc ¸. Integrated model, batch, and domain parallelism in training neural networks. In Proceedings of the 30th on Symposium on Parallelism in Algorithms and Architectures, SPAA 2018, Vienna, Aus- tria, July 16-18, 2018, pages 77-86. ACM, 2018.","title":"Integrated model, batch, and domain parallelism in training neural networks","authors":"Gholami","year":"2018","venue":"Proceedings of the 30th on Symposium on Parallelism in Algorithms and Architectures, SPAA 2018","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Hu et al., ] Weihua Hu, Matthias Fey, Marinka Zitnik, Yux- iao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for ma- chine learning on graphs. In Advances in Neural Informa- tion Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, De- cember 6-12, 2020, virtual.","title":"Open graph benchmark: Datasets for machine learning on graphs","authors":"Hu","year":"2020","venue":"Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jia et al., 2020] Zhihao Jia, Sina Lin, Mingyu Gao, Matei Zaharia, and Alex Aiken. Improving the accuracy, scal- ability, and performance of graph neural networks with roc. In Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020.","title":"Improving the accuracy, scalability, and performance of graph neural networks with roc","authors":"Zhihao Jia, Sina Lin, Mingyu Gao, Matei Zaharia, Alex Aiken","year":"2020","venue":"Proceedings of Machine Learning and Systems 2020, MLSys 2020","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jiang et al., 2017] Jiawei Jiang, Bin Cui, Ce Zhang, and Lele Yu. Heterogeneity-aware distributed parameter servers. In Proceedings of the 2017 ACM International Conference on Management of Data, SIGMOD Confer- ence 2017, Chicago, IL, USA, May 14-19, 2017, pages 463-478. ACM, 2017.","title":"Heterogeneity-aware distributed parameter servers","authors":"Jiawei Jiang, Bin Cui, Ce Zhang, Lele Yu","year":"2017","venue":"Proceedings of the 2017 ACM International Conference on Management of Data, SIGMOD Conference 2017","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kipf and Welling, 2017] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.","title":"Semi-supervised classification with graph convolutional networks","authors":"Thomas N Kipf, Max Welling","year":"2017","venue":"5th International Conference on Learning Representations, ICLR 2017","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"Li et al., ] Hao Li, Asim Kadav, Erik Kruus, and Cristian Ungureanu. MALT: distributed data-parallelism for exist- ing ML applications. In EuroSys 2015, Bordeaux, France, April 21-24, 2015. ACM.","title":"MALT: distributed data-parallelism for existing ML applications","authors":"Li","year":"2015","venue":"EuroSys 2015","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lian et al., 2017] Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentral- ized algorithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient de- scent. In Advances in Neural Information Processing Sys- tems 30: Annual Conference on Neural Information Pro- cessing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5330-5340, 2017.","title":"Can decentralized algorithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient descent","authors":"Lian","year":"2017","venue":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2020] Zhiqi Lin, Cheng Li, Youshan Miao, Yunxin Liu, and Yinlong Xu. Pagraph: Scaling GNN train- ing on large graphs via computation-aware caching. In SoCC '20: ACM Symposium on Cloud Computing, Virtual Event, USA, October 19-21, 2020, pages 401-415. ACM, 2020.","title":"Pagraph: Scaling GNN training on large graphs via computation-aware caching","authors":"Lin","year":"2020","venue":"SoCC '20: ACM Symposium on Cloud Computing, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ma et al., 2019] Lingxiao Ma, Zhi Yang, Youshan Miao, Ji- long Xue, Ming Wu, Lidong Zhou, and Yafei Dai. Neu- graph: Parallel deep neural network computation on large graphs. In 2019 USENIX Annual Technical Conference, USENIX ATC 2019, Renton, WA, USA, July 10-12, 2019. USENIX Association, 2019.","title":"Neugraph: Parallel deep neural network computation on large graphs","authors":"Lingxiao Ma, Zhi Yang, Youshan Miao, Jilong Xue, Ming Wu, Lidong Zhou, Yafei Dai","year":"2019","venue":"USENIX Annual Technical Conference, USENIX ATC 2019","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Miao et al., 2021] Xupeng Miao, Xiaonan Nie, Yingxia Shao, Zhi Yang, Jiawei Jiang, Lingxiao Ma, and Bin Cui. Heterogeneity-aware distributed machine learning training via partial reduce. In SIGMOD '21: International Confer- ence on Management of Data, Virtual Event, China, June 20-25, 2021, pages 2262-2270. ACM, 2021.","title":"Heterogeneity-aware distributed machine learning training via partial reduce","authors":"Miao","year":"2021","venue":"SIGMOD '21: International Conference on Management of Data, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Min et al., 2021] Seungwon Min, Kun Wu, Sitao Huang, Mert Hidayetoglu, Jinjun Xiong, Eiman Ebrahimi, Dem- ing Chen, and Wen-mei W. Hwu. Large graph convo- lutional network training with gpu-oriented data commu- nication architecture. Proc. VLDB Endow., 14(11):2087- 2100, 2021.","title":"Large graph convolutional network training with gpu-oriented data communication architecture","authors":"Min","year":"2021","venue":"Proc. VLDB Endow","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Peng et al., 2022] Jingshu Peng, Zhao Chen, Yingxia Shao, Yanyan Shen, Lei Chen, and Jiannong Cao. SANCUS: staleness-aware communication-avoiding full-graph de- centralized training in large-scale graph neural networks. Proc. VLDB Endow., 15(9):1937-1950, 2022.","title":"SANCUS: staleness-aware communication-avoiding full-graph decentralized training in large-scale graph neural networks","authors":"Peng","year":"2022","venue":"Proc. VLDB Endow","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Thorpe et al., 2021] John Thorpe, Yifan Qiao, Jonathan Ey- olfson, Shen Teng, Guanzhou Hu, Zhihao Jia, Jinliang Wei, Keval Vora, Ravi Netravali, Miryung Kim, and Guoqing Harry Xu. Dorylus: Affordable, scalable, and accurate GNN training with distributed CPU servers and serverless threads. In OSDI 2021, pages 495-514. USENIX Association, 2021.","title":"Dorylus: Affordable, scalable, and accurate GNN training with distributed CPU servers and serverless threads","authors":"John Thorpe, Yifan Qiao, Jonathan Eyolfson, Shen Teng, Guanzhou Hu, Zhihao Jia, Jinliang Wei, Keval Vora, Ravi Netravali, Miryung Kim, Guoqing Harry Xu","year":"2021","venue":"OSDI 2021","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tripathy et al., 2020] Alok Tripathy, Katherine A. Yelick, and Aydin Buluc ¸. Reducing communication in graph neu- ral network training. In Proceedings of the International Conference for High Performance Computing, Network- ing, Storage and Analysis, SC 2020, Virtual Event, Novem- ber 9-19, 2020. IEEE/ACM, 2020.","title":"Reducing communication in graph neural network training","authors":"Alok Tripathy, Katherine A Yelick, Aydin Buluc, ¸","year":"2020","venue":"Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2020] Zhao Wang, Yijin Guan, Guangyu Sun, Dimin Niu, Yuhao Wang, Hongzhong Zheng, and Yinhe Han. Gnn-pim: A processing-in-memory architecture for graph neural networks. In Conference on Advanced Com- puter Architecture, pages 73-86. Springer, 2020.","title":"Gnn-pim: A processing-in-memory architecture for graph neural networks","authors":"Zhao Wang, Yijin Guan, Guangyu Sun, Dimin Niu, Yuhao Wang, Hongzhong Zheng, Yinhe Han","year":"2020","venue":"Conference on Advanced Computer Architecture","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2021] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A com- prehensive survey on graph neural networks. IEEE Trans. Neural Networks Learn. Syst., 32(1):4-24, 2021.","title":"A comprehensive survey on graph neural networks","authors":"Wu","year":"2021","venue":"IEEE Trans. Neural Networks Learn. Syst","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xian et al., 2021] Lintao Xian, Bingzhe Li, Jing Liu, Zhongwen Guo, and David H. C. Du. H-PS: A heterogeneous-aware parameter server with distributed neural network training. IEEE Access, 9:44049-44058, 2021.","title":"A heterogeneous-aware parameter server with distributed neural network training","authors":"Lintao Xian, Bingzhe Li, Jing Liu, Zhongwen Guo, David H C Du, H-Ps","year":"2021","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zeng et al., 2020] Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor K. Prasanna. Graphsaint: Graph sampling based inductive learning method. In 8th International Conference on Learning Rep- resentations, ICLR 2020, Addis Ababa, Ethiopia, April 26- 30, 2020. OpenReview.net, 2020.","title":"Graphsaint: Graph sampling based inductive learning method","authors":"Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, Viktor K Prasanna","year":"2020","venue":"8th International Conference on Learning Representations, ICLR 2020","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zheng et al., 2020] Da Zheng, Chao Ma, Minjie Wang, Jin- jing Zhou, Qidong Su, Xiang Song, Quan Gan, Zheng Zhang, and George Karypis. Distdgl: Distributed graph neural network training for billion-scale graphs. pages 36- 44, 2020.","title":"Distdgl: Distributed graph neural network training for billion-scale graphs","authors":"Da Zheng, Chao Ma, Minjie Wang, Jinjing Zhou, Qidong Su, Xiang Song, Quan Gan, Zheng Zhang, George Karypis","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhu et al., 2019] Rong Zhu, Kun Zhao, Hongxia Yang, Wei Lin, Chang Zhou, Baole Ai, Yong Li, and Jingren Zhou. Aligraph: A comprehensive graph neural network plat- form. Proc. VLDB Endow., 12(12):2094-2105, 2019.","title":"Aligraph: A comprehensive graph neural network platform","authors":"Rong Zhu, Kun Zhao, Hongxia Yang, Wei Lin, Chang Zhou, Baole Ai, Yong Li, Jingren Zhou","year":"2019","venue":"Proc. VLDB Endow","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}