{"type":"ScholarlyArticle","title":"FedTAD: Topology-aware Data-free Knowledge Distillation for Subgraph Federated Learning","authors":[{"name":"Yinlin Zhu","slug":"yinlin-zhu","given":"Yinlin","family":"Zhu","affiliation":"Sun Yat-sen University , Guangzhou , China"},{"name":"Xunkai Li","slug":"xunkai-li","given":"Xunkai","family":"Li","affiliation":"Beijing Institute of Technology , Beijing , China"},{"name":"Zhengyu Wu","slug":"zhengyu-wu","given":"Zhengyu","family":"Wu","affiliation":"Beijing Institute of Technology , Beijing , China"},{"name":"Di Wu","slug":"di-wu","given":"Di","family":"Wu","affiliation":"Sun Yat-sen University , Guangzhou , China"},{"name":"Miao Hu","slug":"miao-hu","given":"Miao","family":"Hu","affiliation":"Sun Yat-sen University , Guangzhou , China"},{"name":"Rong-Hua Li","slug":"rong-hua-li","given":"Rong-Hua","family":"Li","affiliation":"Beijing Institute of Technology , Beijing , China"}],"abstract":"Subgraph federated learning (subgraph-FL) is a new distributed paradigm that facilitates the collaborative training of graph neural networks (GNNs) by multi-client subgraphs. Unfortunately, a significant challenge of subgraph-FL arises from subgraph heterogeneity, which stems from node and topology variation, causing the impaired performance of the global GNN. Despite various studies, they have not yet thoroughly investigated the impact mechanism of subgraph heterogeneity. To this end, we decouple node and topology variation, revealing that they correspond to differences in label distribution and structure homophily. Remarkably, these variations lead to significant differences in the class-wise knowledge reliability of multiple local GNNs, misguiding the model aggregation with varying degrees. Building on this insight, we propose topology-aware data-free knowledge distillation technology (FedTAD), enhancing reliable knowledge transfer from the local model to the global model. Extensive experiments on six public datasets consistently demonstrate the superiority of FedTAD over state-of-the-art baselines.","doi":"10.24963/ijcai.2024/632","url":"https://ijcai.org/proceedings/2024/632","pdf":"https://ijcai.org/proceedings/2024/0632.pdf","markdown":"https://ijcai.org/proceedings/2024/632.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/632","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":632,"pages":{"first":5716,"last":5724},"keywords":["Machine Learning: ML: Sequence and graph learning","Machine Learning: ML: Federated learning","Machine Learning: ML: Classification"],"videos":[],"references":[{"position":1,"raw":"[Akiba et al., 2019] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623-2631, 2019.","title":"Optuna: A next-generation hyperparameter optimization framework","authors":"Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, Masanori Koyama","year":"2019","venue":"Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baek et al., 2023] Jinheon Baek, Wonyong Jeong, Jiongdao Jin, Jaehong Yoon, and Sung Ju Hwang. Personalized sub- graph federated learning. In International Conference on Machine Learning, pages 1396-1415. PMLR, 2023.","title":"Personalized subgraph federated learning","authors":"Baek","year":"2023","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blondel et al., 2008] Vincent D Blondel, Jean-Loup Guil- laume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks. Jour- nal of statistical mechanics: theory and experiment, 2008(10):P10008, 2008.","title":"Fast unfolding of communities in large networks","authors":"Jean-Loup Vincent D Blondel, Renaud Guillaume, Etienne Lambiotte, Lefebvre","year":"2008","venue":"Journal of statistical mechanics: theory and experiment","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bruna et al., 2013] Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and lo- cally connected networks on graphs. arXiv preprint arXiv:1312.6203, 2013.","title":"Spectral networks and locally connected networks on graphs","authors":"Joan Bruna, Wojciech Zaremba, Arthur Szlam, Yann Lecun","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2019] Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student net- works. In Proceedings of the IEEE/CVF international con- ference on computer vision, pages 3514-3522, 2019.","title":"Data-free learning of student networks","authors":"Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, Qi Tian","year":"2019","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ding et al., 2018] Ming Ding, Jie Tang, and Jie Zhang. Semi-supervised learning on graphs with generative adver- sarial nets. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pages 913-922, 2018.","title":"Semi-supervised learning on graphs with generative adversarial nets","authors":"Ming Ding, Jie Tang, Jie Zhang","year":"2018","venue":"Proceedings of the 27th ACM International Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fu et al., 2022] Xingbo Fu, Binchi Zhang, Yushun Dong, Chen Chen, and Jundong Li. Federated graph machine learning: A survey of concepts, techniques, and applica- tions. ACM SIGKDD Explorations Newsletter, 24(2):32- 47, 2022.","title":"Federated graph machine learning: A survey of concepts, techniques, and applications","authors":"Fu","year":"2022","venue":"ACM SIGKDD Explorations Newsletter","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gao et al., 2022] Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Cheng-Zhong Xu. Feddc: Federated learning with non-iid data via local drift decoupling and correction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10112- 10121, 2022.","title":"Feddc: Federated learning with non-iid data via local drift decoupling and correction","authors":"Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, Cheng-Zhong Xu","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gilmer et al., 2017] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neu- ral message passing for quantum chemistry. In Interna- tional conference on machine learning, pages 1263-1272. PMLR, 2017.","title":"Neural message passing for quantum chemistry","authors":"Justin Gilmer, S Samuel, Patrick F Schoenholz, Oriol Riley, George E Vinyals, Dahl","year":"2017","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guo and Wang, 2020] Zhiwei Guo and Heng Wang. A deep graph neural network-based mechanism for social recom- mendations. IEEE Transactions on Industrial Informatics, 17(4):2776-2783, 2020.","title":"A deep graph neural network-based mechanism for social recommendations","authors":"Zhiwei Guo, Heng Wang","year":"2020","venue":"IEEE Transactions on Industrial Informatics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hamilton et al., 2017] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing sys- tems, 30, 2017.","title":"Inductive representation learning on large graphs","authors":"Hamilton","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2021] Chaoyang He, Keshav Balasubramanian, Emir Ceyani, Carl Yang, Han Xie, Lichao Sun, Lifang He, Liangwei Yang, Philip S Yu, Yu Rong, et al. Fedgraphnn: A federated learning system and benchmark for graph neu- ral networks. arXiv preprint arXiv:2104.07145, 2021.","title":"Fedgraphnn: A federated learning system and benchmark for graph neural networks","authors":"He","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hu et al., 2020] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural informa- tion processing systems, 33:22118-22133, 2020.","title":"Open graph benchmark: Datasets for machine learning on graphs","authors":"Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, Jure Leskovec","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Karimireddy et al., 2020] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In Interna- tional conference on machine learning, pages 5132-5143.","title":"Scaffold: Stochastic controlled averaging for federated learning","authors":"Praneeth Sai, Satyen Karimireddy, Mehryar Kale, Sashank Mohri, Sebastian Reddi, Ananda Stich, Suresh Theertha","year":null,"venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"PMLR, 2020.","title":null,"authors":"Pmlr","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kipf and Welling, 2016] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.","title":"Semi-supervised classification with graph convolutional networks","authors":"N Thomas, Max Kipf, Welling","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2020] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Pro- ceedings of Machine learning and systems, 2:429-450, 2020.","title":"Federated optimization in heterogeneous networks","authors":"Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, Virginia Smith","year":"2020","venue":"Proceedings of Machine learning and systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2021] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10713-10722, 2021.","title":"He, and Dawn Song. Model-contrastive federated learning","authors":"Li","year":"2021","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2023] Xunkai Li, Zhengyu Wu, Wentao Zhang, Yinlin Zhu, Ronghua Li, and Guoren Wang. Fedgta: Topology-aware averaging for federated graph learning. Proc. VLDB Endow., 17(1):41-50, 2023.","title":"Fedgta: Topology-aware averaging for federated graph learning","authors":"Xunkai Li, Zhengyu Wu, Wentao Zhang, Yinlin Zhu, Ronghua Li, Guoren Wang","year":"2023","venue":"Proc. VLDB Endow","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mao et al., 2019] Qi Mao, Hsin-Ying Lee, Hung-Yu Tseng, Siwei Ma, and Ming-Hsuan Yang. Mode seeking genera- tive adversarial networks for diverse image synthesis. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 1429-1437, 2019.","title":"Mode seeking generative adversarial networks for diverse image synthesis","authors":"Qi Mao, Hsin-Ying Lee, Hung-Yu Tseng, Siwei Ma, Ming-Hsuan Yang","year":"2019","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.","title":"Communication-efficient learning of deep networks from decentralized data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"Artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shchur et al., 2018] Oleksandr Shchur, Maximil- ian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868, 2018.","title":"Pitfalls of graph neural network evaluation","authors":"Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, Stephan Günnemann","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sun et al., 2023] Henan Sun, Xunkai Li, Zhengyu Wu, Dao- han Su, Rong-Hua Li, and Guoren Wang. Breaking the entanglement of homophily and heterophily in semi- supervised node classification, 2023.","title":"Breaking the entanglement of homophily and heterophily in semisupervised node classification","authors":"Sun","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tan et al., 2023] Yue Tan, Yixin Liu, Guodong Long, Jing Jiang, Qinghua Lu, and Chengqi Zhang. Federated learn- ing on non-iid graphs via structural knowledge sharing. In Proceedings of the AAAI conference on artificial intelli- gence, volume 37, pages 9953-9961, 2023.","title":"Federated learning on non-iid graphs via structural knowledge sharing","authors":"Yue Tan, Yixin Liu, Guodong Long, Jing Jiang, Qinghua Lu, Chengqi Zhang","year":"2023","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tong et al., 2006] Hanghang Tong, Christos Faloutsos, and Jia-Yu Pan. Fast random walk with restart and its appli- cations. In Sixth international conference on data mining (ICDM'06), pages 613-622. IEEE, 2006.","title":"Fast random walk with restart and its applications","authors":"Tong","year":"2006","venue":"Sixth international conference on data mining (ICDM'06)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Veličković et al., 2017] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903, 2017.","title":"Graph attention networks","authors":"Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wu et al., 2020] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A com- prehensive survey on graph neural networks. IEEE trans- actions on neural networks and learning systems, 32(1):4- 24, 2020.","title":"A comprehensive survey on graph neural networks","authors":"Wu","year":"2020","venue":"IEEE transactions on neural networks and learning systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wu et al., 2021] Chuhan Wu, Fangzhao Wu, Yang Cao, Yongfeng Huang, and Xing Xie. Fedgnn: Federated graph neural network for privacy-preserving recommendation. arXiv preprint arXiv:2102.04925, 2021.","title":"Fedgnn: Federated graph neural network for privacy-preserving recommendation","authors":"Wu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xie et al., 2021] Han Xie, Jing Ma, Li Xiong, and Carl Yang. Federated graph classification over non-iid graphs. Advances in Neural Information Processing Systems, 34:18839-18852, 2021.","title":"Federated graph classification over non-iid graphs","authors":"Han Xie, Jing Ma, Li Xiong, Carl Yang","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2016] Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on ma- chine learning, pages 40-48. PMLR, 2016.","title":"Revisiting semi-supervised learning with graph embeddings","authors":"Zhilin Yang, William Cohen, Ruslan Salakhudinov","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang et al., 2021] Shuo Yang, Zhiqiang Zhang, Jun Zhou, Yang Wang, Wang Sun, Xingyu Zhong, Yanming Fang, Quan Yu, and Yuan Qi. Financial risk analysis for smes with graph-based supply chain mining. In Proceedings of the Twenty-Ninth International Conference on Interna- tional Joint Conferences on Artificial Intelligence, pages 4661-4667, 2021.","title":"Financial risk analysis for smes with graph-based supply chain mining","authors":"Shuo Yang, Zhiqiang Zhang, Jun Zhou, Yang Wang, Wang Sun, Xingyu Zhong, Yanming Fang, Quan Yu, Yuan Qi","year":"2021","venue":"Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2020/643"},{"position":32,"raw":"[Zhang et al., 2021] Ke Zhang, Carl Yang, Xiaoxiao Li, Lichao Sun, and Siu Ming Yiu. Subgraph federated learn- ing with missing neighbor generation. Advances in Neural Information Processing Systems, 34:6671-6682, 2021.","title":"Subgraph federated learning with missing neighbor generation","authors":"Ke Zhang, Carl Yang, Xiaoxiao Li, Lichao Sun, Siu Ming, Yiu","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhou et al., 2020] Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. AI Open, 1:57-81, 2020.","title":"Graph neural networks: A review of methods and applications","authors":"Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, Maosong Sun","year":"2020","venue":"AI Open","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhu et al., 2020] Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Be- yond homophily in graph neural networks: Current limita- tions and effective designs. Advances in neural informa- tion processing systems, 33:7793-7804, 2020.","title":"Beyond homophily in graph neural networks: Current limitations and effective designs","authors":"Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, Danai Koutra","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}