Heterogeneous Federated Learning with Scalable Server Mixture-of-Experts

Jiang, Jingang; Chen, Yanzhao; Liu, Xiangyang; Jiang, Haiqi; Fan, Chenyou

doi:10.24963/ijcai.2025/610

Heterogeneous Federated Learning with Scalable Server Mixture-of-Experts

Jingang Jiang, Yanzhao Chen, Xiangyang Liu, Haiqi Jiang, Chenyou Fan

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 5480-5488.

https://doi.org/10.24963/ijcai.2025/610

Keywords

Machine Learning: ML: Federated learning
Natural Language Processing: NLP: Text classification
Machine Learning: ML: Deep learning architectures
Natural Language Processing: NLP: Language models

Abstract

Classical Federated Learning (FL) encounters significant challenges when deploying large models on power-constrained clients. To tackle this, we propose an asymmetric FL mechanism that enables the aggregation of compact client models into a comprehensive server model. We design the server model as a Mixture-of-Experts (MoE), where each expert has the same architecture as each client model. This uniformity allows for efficient fusion of the most pertinent client models to update each server expert, based on the measured relevance between each client and server expert. To address the Non-IID data issue, we further optimize the server-side MoE architecture by incorporating a main expert that always activates alongside a set of selectively activated routed experts. This configuration ensures a balance between learning general knowledge and specific data distribution. Our Fed-MoE framework is model-agnostic and has demonstrated notable improvements on vision FL tasks with million-scale ResNet backbones, and language tasks with billion-scale BERT and GPT-2 backbones.

Jingang Jiang, Yanzhao Chen, Xiangyang Liu, Haiqi Jiang, and Chenyou Fan. 2025. Heterogeneous Federated Learning with Scalable Server Mixture-of-Experts. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. IJCAI, 5480–5488. https://doi.org/10.24963/ijcai.2025/610

@inproceedings{ijcai2025p610,
  title     = {Heterogeneous Federated Learning with Scalable Server Mixture-of-Experts},
  author    = {Jiang, Jingang and Chen, Yanzhao and Liu, Xiangyang and Jiang, Haiqi and Fan, Chenyou},
  booktitle = {Proceedings of the Thirty-Fourth International Joint Conference on
               Artificial Intelligence, {IJCAI-25}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {James Kwok},
  pages     = {5480--5488},
  year      = {2025},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2025/610},
  url       = {https://doi.org/10.24963/ijcai.2025/610},
}

TY  - CPAPER
TI  - Heterogeneous Federated Learning with Scalable Server Mixture-of-Experts
AU  - Jiang, Jingang
AU  - Chen, Yanzhao
AU  - Liu, Xiangyang
AU  - Jiang, Haiqi
AU  - Fan, Chenyou
T2  - Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2025
SP  - 5480
EP  - 5488
DO  - 10.24963/ijcai.2025/610
UR  - https://doi.org/10.24963/ijcai.2025/610
ER  -