{"type":"ScholarlyArticle","title":"Distilling A Universal Expert from Clustered Federated Learning","authors":[{"name":"Zeqi Leng","slug":"zeqi-leng","given":"Zeqi","family":"Leng","affiliation":""},{"name":"Chunxu Zhang","slug":"chunxu-zhang","given":"Chunxu","family":"Zhang","affiliation":""},{"name":"Guodong Long","slug":"guodong-long","given":"Guodong","family":"Long","affiliation":""},{"name":"Riting Xia","slug":"riting-xia","given":"Riting","family":"Xia","affiliation":""},{"name":"Bo Yang","slug":"bo-yang","given":"Bo","family":"Yang","affiliation":""}],"abstract":"Clustered Federated Learning (CFL) addresses the challenges posed by non-IID data by training multiple group- or cluster-specific expert models. However, existing methods often overlook the shared information across clusters, which represents the generalizable knowledge valuable to all participants in the Federated Learning (FL) system. To overcome this limitation, this paper introduces a novel FL framework that distills a universal expert model from the knowledge of multiple clusters. This universal expert captures globally shared information across all clients and is subsequently distributed to each client as the initialization for the next round of model training. The proposed FL framework operates in three iterative steps: (1) local model training at each client, (2) cluster-specific model aggregation, and (3) universal expert distillation. This three-step learning paradigm ensures the preservation of fine-grained non-IID characteristics while effectively incorporating shared knowledge across clusters. Compared to traditional gradient-based aggregation methods, the distillation-based model aggregation introduces greater flexibility in handling model heterogeneity and reduces conflicts among cluster-specific experts. Extensive experimental results demonstrate the superior performance of the proposed method across various scenarios, highlighting its potential to advance the state of CFL by balancing personalized and shared knowledge more effectively.","doi":"10.24963/ijcai.2025/620","url":"https://ijcai.org/proceedings/2025/620","pdf":"https://ijcai.org/proceedings/2025/0620.pdf","markdown":"https://ijcai.org/proceedings/2025/620.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/620","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Machine Learning","track":"Main Track","number":620,"pages":{"first":5571,"last":5579},"keywords":["Data Mining: DM: Privacy-preserving data mining","Machine Learning: ML: Federated learning"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}