Beyond Client Clustering: Fine-Grained Preference Alignment in Federated RLHF via Self-Evolving Routing

Wang, Ke; Fu, Shaojing; Luo, Yuchuan; Deng, Guilin; Chen, Silong; Yuan, Zheng; Liu, Lin

doi:10.24963/ijcai.2026/552

Beyond Client Clustering: Fine-Grained Preference Alignment in Federated RLHF via Self-Evolving Routing

Ke Wang, Shaojing Fu, Yuchuan Luo, Guilin Deng, Silong Chen, Zheng Yuan, Lin Liu

Download BibTeX Cite

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. Main Track. Pages 4958-4966.

https://doi.org/10.24963/ijcai.2026/552

Keywords

Multidisciplinary Topics and Applications: Security and privacy
Machine Learning: Federated learning

Abstract

Federated Reinforcement Learning from Human Feedback (RLHF) enables the collaborative alignment of Large Language Models (LLMs) while preserving privacy, yet it faces critical bottlenecks arising from data heterogeneity. Existing approaches typically rely on rigid client-level clustering, which overlooks intra-client heterogeneity and fails to adapt to the multifaceted needs of individual users. To address this, we propose FedPrism, a novel framework that shifts alignment granularity from the coarse client level to the precise instance level. Similar to an optical prism dispersing mixed light into distinct spectral components, FedPrism decomposes complex, intra-client heterogeneous data streams by dynamically routing individual samples to specialized experts based on semantic features. Crucially, we devise a Posterior-Guided Performance Distillation (PGPD) mechanism that leverages experts' actual training loss as self-supervision to autonomously refine routing policies without explicit labels. Extensive experiments show that FedPrism not only establishes new state-of-the-art results but also effectively mitigates the negative transfer prevalent in non-IID settings, ensuring superior alignment fidelity.

Ke Wang, Shaojing Fu, Yuchuan Luo, Guilin Deng, Silong Chen, Zheng Yuan, and Lin Liu. 2026. Beyond Client Clustering: Fine-Grained Preference Alignment in Federated RLHF via Self-Evolving Routing. In Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26. IJCAI, 4958–4966. https://doi.org/10.24963/ijcai.2026/552

@inproceedings{ijcai2026p552,
  title     = {Beyond Client Clustering: Fine-Grained Preference Alignment in Federated RLHF via Self-Evolving Routing},
  author    = {Wang, Ke and Fu, Shaojing and Luo, Yuchuan and Deng, Guilin and Chen, Silong and Yuan, Zheng and Liu, Lin},
  booktitle = {Proceedings of the Thirty-Fifth International Joint Conference on
               Artificial Intelligence, {IJCAI-26}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Diego Calvanese},
  pages     = {4958--4966},
  year      = {2026},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2026/552},
  url       = {https://doi.org/10.24963/ijcai.2026/552},
}

TY  - CPAPER
TI  - Beyond Client Clustering: Fine-Grained Preference Alignment in Federated RLHF via Self-Evolving Routing
AU  - Wang, Ke
AU  - Fu, Shaojing
AU  - Luo, Yuchuan
AU  - Deng, Guilin
AU  - Chen, Silong
AU  - Yuan, Zheng
AU  - Liu, Lin
T2  - Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2026
SP  - 4958
EP  - 4966
DO  - 10.24963/ijcai.2026/552
UR  - https://doi.org/10.24963/ijcai.2026/552
ER  -