SDFLoRA: Selective Decoupled Federated LoRA for Privacy-preserving Fine-tuning with Heterogeneous Clients

Abstract

Federated learning (FL) has emerged as a promising paradigm for adapting large language models (LLMs) to distributed data. To mitigate the high communication and memory overhead, parameter efficient techniques such as Low Rank Adaptation (LoRA) are widely adopted. However, practical deployments often exhibit rank and data heterogeneity, making direct aggregation of LoRA updates biased and unstable. Existing approaches enforce a unified rank or align heterogeneous updates into a single shared subspace, which undermines personalization and accuracy. Moreover, under differential privacy (DP), adding noise to such mixed updates could perturb client specific directions that should remain local, resulting in utility loss. To address these issues, we propose Selective Decoupled Federated LoRA (SDFLoRA), a structure aware LoRA framework that decouples each client update into a shared component for updating and a private component that preserves client specific semantics. Subspace alignment and aggregation are applied selectively to the shared module, while private modules remain local. We further design low rank re-compression to maintain a fixed rank budget for the shared module. This structure supports privacy aware optimization by injecting DP noise exclusively into the shared module. Experiments on multiple benchmarks demonstrate that SDFLoRA beats federated LoRA baselines and exhibits strong robustness under privacy constraints.