FedENC: Federated Adaptive CLIP Model via Energy Alignment and Neural Collapse

1 Introduction

Without sufficient training data, the performance of the deep learning (DL) model tends to decrease and overfit [Qi et al., 2026; Choi and Ko, 2025]. One naive way is to collect multisource data to support the training phase. However, due to data regularization laws such as GDPR [Voigt and Von dem Bussche, 2017], accessing source data is not practical since it introduces privacy leakage concerns.

Federated learning (FL) enables collaborative model training with various decentralized institutions (a.k.a., FL clients), providing a promising solution to solve data privacy [Wu et al., 2026; Wu and Chaddad, 2026]. FL is widely used in many applications, such as private face recognition, image classification and segmentation [Li et al., 2025; Qi et al., 2025; Meng et al., 2025]. However, recent DL models such as CLIP tend to have billions of parameters to achieve remarkable performance [Radford et al., 2021], leading to considerable computation and communication overhead (e.g., 108 parameters in ViT-B-32, require ∼ 342 MB to transmit). This hinders the use of these models in the FL context. Furthermore, in the real-world, data from multiple clients exhibit non-independent and identically distributed (non-IID) characteristics due to variations in texture discrepancies and data imbalance [Chen et al., 2025]. This data heterogeneity poses a challenge for the global model to learn unified decision boundaries, resulting in poor performance.

* Corresponding author.

Recent studies proposed several techniques to solve these challenges. Basically, these methods can be classified into three groups: adapter-, prompt learning-, and linear probebased approaches. However, adapter based methods, such as FedCLIP [Lu et al., 2023], can not fully capture domaininvariant features as suggested in [Wu et al., 2025]. Promptlearning methods, although efficient in communication, require large GPU memory to adapt learnable prompts [Guo et al., 2024; Phung et al., 2025]. Linear probe based methods such as LP++ [Huang et al., 2024] are efficient in both computation and communication, but linear projections limit their ability to learn task-specific features.

In light of previous challenges, we propose a novel CLIPbased FL framework for non-IID image classification task (FedENC). To reduce computation and communication overhead, the pre-trained CLIP encoder is frozen, while we introduce three lightweight modules (one global expert and two local experts) to adapt the model. Specifically, the global expert is used as a communication module to learn shared knowledge across clients, while the two local experts focus on learning client-specific features to fit the local tasks. To address data heterogeneity, we optimize the parameters of the global expert using NC alignment. The primary goal of NC loss is to enforce the feature representations learned by each client to align with a pre-defined prototypes, mitigating the impact of domain shifts and ensuring global feature consistency. To improve the prediction stability, we minimize the differences between the global expert and local expert outputs using KL and cosine similarity based energy alignment. During inference, the outputs of global and local experts are ensembled to obtain the final prediction. Finally, the global expert of each client is aggregated using average aggregation. We evaluate FedENC using six image classification benchmarks. The results show that FedENC outperforms other FL approaches with reasonable computation and communication load. The contributions of this paper are as follows.

• Algorithm. We propose a novel CLIP-based FL framework for image classification tasks. Specifically, we propose adapting a frozen CLIP backbone with global and local experts to adapt the CLIP with lower computation and communication load. Furthermore, we introduce NC adaptation to solve data heterogeneity, and we design an energy alignment technique to improve prediction consistency between the global and local experts. Finally, global experts are aggregated and broadcast to each client for local training.
• Empirical analysis. We perform extensive experiments on six datasets (4 natural + 2 medical) to show the usefulness of the proposed approach in: (i) robustness to heterogeneity FL, (ii) efficiency of computation and communication, (iii) adaptability with different network architectures, (iv) scalability with different number of clients and (v) calibration ability during inference.

2 RelatedWork

Federated learning. For example, based on FedCLIP, FAACLIP [Wu et al., 2025] extends it with an adversarial domain adaptation technique to solve the feature shifts among the clients. However, adversarial adaptation introduces extra learnable parameters, increasing the computational overhead. Unlike these methods that adapting the image features, PromptFL [Guo et al., 2024] proposes fine-tuning the learnable prompts and share these prompts for aggregation. Despite its lower computational load, these learnable parameters will increase the communication overhead when the number of classes is large. This limits its potential for real-world applications. Federated adaptive prompt learning (FedAPT) adds a domain-specific prompt related to clients to the general prompts compared to PromptFL [Su et al., 2024]. In [Motamedi et al., 2025], they update the Pre-norm layer parameters in the vision encoder and aggregate these parameters to efficiently adapt the CLIP model (NormFit). In addition, FedDIM [Liao et al., 2025] introduces a decision insight matrix to reduce domain shifts and improve the classifier performance. Results show that it achieves a higher accuracy compared to FedAVG. However, it requires more computation time under large-scale dataset.

CLIP model. In [Zanella and Ben Ayed, 2024], they introduce low-rank adaptation (LoRA) to adapt the CLIP backbone into downstream tasks with minimal costs. Similar to LoRA, in [Zhou et al., 2022], they propose a conditional prompt learning technique to improve the text features (CocoOp). However, it consumes large GPU memory as the batch size increases. LP++ [Huang et al., 2024] explores the usefulness of linear probing and design a novel prediction strategy that uses both image and text information. In

[Hakim et al., 2025], they design a test-time adaptation technique to effectively adapt the CLIP model for image classification tasks in a transductive manner.

Neural collapse. Neural collapse (NC) proves that for a well-trained classifier on a balanced dataset, the features produced by the network backbone for all samples within each class converge to a highly symmetric structure, and withinclass variability effectively vanishes [Papyan et al., 2020]. For example, in [Li et al., 2023], they design a synthetic and fixed classifier during local training in FL. The experimental results show that the optimal classifier structure enables all clients to learn unified and optimal feature representations under heterogeneous data. In [Du and Li, 2025], they introduce a simplex learnable embedding module guided by NC to learn local features, while use a knowledge decoupling module to extract general knowledge to align local features with the global embeddings.

Unlike these methods, FedENC addresses these limitations through a systematic framework that uses global and local experts for efficient adaptation of a pre-trained CLIP model, NC-based alignment to align image features of each client to a balanced feature space to solve data heterogeneity, and energy-based consistency regularization between experts.

3 Methods

This study considers a standard FL setting, consisting of N clients and a global server for aggregation. Each client has a private dataset Di. Each data set Di consists of three non-overlapping parts, namely a training Dtrain i = {(xtrain i,j , ytrain i,j )}ntrain i j=1 , a validation Dval i = {(xval i,j , yval i,j )}nval i j=1 and a test set Dtest i ={(xtest i,j , ytest i,j )}ntest i j=1 . Figure 1 shows the general framework of the proposed method. It consists of two parts, namely 1) local training and 2) global aggregation.

3.1 Local Training and Inference

Feature extraction with CLIP. We use a pre-trained CLIP model to extract both image and text features. Let eI and eT be the image and text encoder, respectively. Given an image sample x, we denote Ij = eI(xj) ∈ RD as the Ddimensional vector of image features. For text encoder, we use a standard text prompt (A PICTURE OF A {CLASS}) to obtain the text features Tj = eT (xj) ∈ RD .

Feature adaptation via expert module. Despite CLIP encoders can extract a rich set of features, however, these features should be refined for a specific task (i.e., learn dominant feature representations). Since most heterogeneity exists in images, we only adapted image features in this study. To efficiently adapt the image features, the CLIP encoder is frozen and we design three lightweight modules for each client, consisting of a global expert (aptig ) and two local experts (apti1 , apti2 ). We consider two local experts because it introduces less computation complexity. Specifically, the global expert focuses on learning cross-client knowledge by aggregation, while the local experts are used to fit client-specific characteristics. The goal of the global expert is to generate a mask M ∈ [0, 1]D based on I using a Softmax function with linear layers. This mask is used to obtain an adapted image feature ˜I = aptig (I) ⊗ I, where ⊗ is the Hadamard product (element wise). For local experts, I and˜I are used as the input for apti1 and apti2 , respectively. This complementary feature input will encourage local experts to learn robust and diverse features. Finally, local experts will provide predicted probability based on these inputs. To refine the adapted image features ˜I, we use the standard contrastive loss in CLIP. Let S be the B×B matrix consisting of cosine similarities between the image features ˜Ij and Tj ′ . We compute an image probability matrix P = Softmax(S/τ) ∈ [0, 1]B×B and a text probability matrix Q = Softmax(S⊤/τ ) ∈ [0, 1]B×B . The contrastive loss is then formulated as follows:

where τ is a pre-defined scaling parameter in CLIP. To train the local experts, we use standard cross-entropy loss as follows.

where oj is the output from local expert.

Neural collapse alignment. As suggested in [Papyan et al., 2020], they show that for deep networks trained on a balanced dataset, let M = [M1,M2, ...,MC] (C is the number of classes) be the weight matrix of the linear classifier, it exists a simplex Equiangular Tight Frame (ETF) M ∗ in RD for C classes as follows:

where IC ∈ RC×C is the identity matrix, 1C ∈ RC is the allones vector, R ∈ Rd×C (d ≥ C) is any orthogonal projection, and α > 0 is a scaling constant.

NC suggests that well-trained networks exhibit class-wise feature means that converge to the vertices of a Simplex ETF. Suppose FETF = [M1, ...,MC] is the reference geometry for all clients, each column Mi represents the ideal feature prototype for class C and satisfies the ETF properties:

This globally fixed frame is shared across all clients, providing a consistent and maximally symmetric target geometry. To align the clients features with this fixed geometry, we introduce a NC alignment loss that encourages the adapted image features ˜I to move toward their correct prototype while repelling them from others:

where κ is a scaling constant, wck is the weight for class c in the client k. The wck can be formulated as follows.

where nck is the number of samples for class c in client k, γ is a scaling parameter. The first term in Eq. 5 pulls the feature toward its correct prototype, while the second term pushes it away from other classes. A lower γ indicates more weights on rare class samples.

Energy alignment. Since the global expert focuses on general information across clients, while the local experts focus on local details, we propose an energy function-based technique to improve the prediction consistency of the global and local experts as follows.

where O1 = Softmax(o1/τ), O2 = Softmax(o2/τ), ϕ(·) is a MLP. o1|2 represents the outputs from apti1|2 and aptig . A smaller value of LEng indicates the outputs of the shared expert and local experts are similar, and vice versa.

Finally, the loss used for local training can be formulated as:

where λ1 and λ2 are two hyper-parameters. Ensemble prediction. To enrich the prediction with both global and locall learned knowledge, we design an ensemble prediction strategy as follows:

where pens , paptig and papti1|2 are the predicted probability of the ensemble prediction, aptig and apti1|2 , respectively. β is defined using an entropy function H(·) as follows.

Specifically, β dynamically balances global and local predictions, assigning higher weight to the prediction branch with lower uncertainty.

3.2 Global Aggregation

To provide an unbiased global model, we use a simple average aggretaion technique as follows:

4 Experiments

4.1 Datasets

OfficeHome. OfficeHome is a large image classification benchmark that contains 65 classes [Venkateswara et al.,

2017]. This dataset has four sub-domains, namely Art (Ar), Clipart (Cl), Product (Pr), and Real World (Re) with about 15500 images.

ModernOffice31. It is a modern benchmark with 31 classes covering four domains: amazon, webcam, dslr, and synthetic [Ringwald and Stiefelhagen, 2021].

PACS. PACS has four domains, namely art, cartoon, photo and sketch [Li et al., 2017]. It has seven classes, with about 9991 images.

DomainNet. DomainNet contains 6 domains (i.e., clipart, C; infograph, I; painting, P; quickdraw, Q; real world, R; sketch, S), with more than 0.6 million images, covering 345 classes [Peng et al., 2019].

ISIC2019. ISIC2019 is a skin cancer classification dataset with various characteristics (e.g., age) [Tschandl et al., 2018; Codella et al., 2018; Combalia et al., 2019]. To simulate data heterogeneous, it was divided based on the "anatom site (AS)" metadata, resulting in seven clients: C1 holds the data whose AS is "anterior torso", while C2, C3, C4, C5, C6 and Cglo hold the data whose AS is "head or neck", "lower extremity", "palms or soles", "posterior torso", "upper extremity" and "Nan", respectively. It has six classes.

Brain ICH. The Brain ICH dataset [Flanders et al., 2020], which contains five ICH subtypes, is used for experiments. The same pre-processing strategies as in [Wu et al., 2023] are applied. To simulate heterogeneous data, following [Wu et al., 2023], Dirichlet distribution is used to divide the dataset to {5,10,15} clients.

For OfficeHome, ModernOffice31, PACS, DomainNet and ISIC2019, each domain is considered as a client. For each client in these datasets, the data is further divided into three groups, namely a training set (60%), a validation set (20%) and a test set (20%).

4.2 Implementation Details

Pre-trained CLIP (ViT-B-32 as the vision encoder) provided by OpenAI is used as the backbone [Radford et al., 2021]. The AdamW optimizer is used [Loshchilov and Hutter, 2017]. The learning rate is set to 5 × 10-5 for OfficeHome, ModernOffice31 and PACS datasets, and to 5 × 10-4 for DomainNet, ISIC2019 and Brain ICH datasets. The λ1 and λ2 are set to the optimal values based on sensitivity analysis, respectively. The local training epoch is set to one, while the global communication epoch is set to 50. The batch size is 32, and the random seed is initialized to 0 for all methods. For comparison, we use both non-federated and federated techniques. Non-federated includes CLIPzs (zeroshot) and Individual (no aggregation). Federated approaches include FedAVG (full-parameter tuning), FAA-CLIP, LoRA with FedAVG (rank is set to three), PromptFL, CocoOp with FedAVG, LP++ with FedAVG, FedAPT, NormFit and FedDIM. The experiments are based on an RTX 4090 GPU, featuring an Intel 13900KF CPU with 128 GB memory. More details can be found in codes.

4.3 Results

OfficeHome. Table 1 shows the test accuracy (ACC) for the baselines and FedENC. FedENC provides an average (AVG) ACC of 86.87%, outperforming traditional FL such as FedAVG by 2.43% and PEFT techniques such as PromptFL by 2.86%. The improvement is notable on client C (84.26%), where pre-trained CLIP has limited domain knowledge (65.64% ACC with CLIPzs).

ModernOffice31. Similar to OfficeHome, FedENC yields the highest AVG ACC of 97.55% among all methods, as summarized in Table 1. We note that FedENC shows a lower test ACC on client synthetic compared to FedAVG. This suggests that pre-trained CLIP fails to extract more representative features (i.e., 76.10% with CLIPzs) than full-parameter tuning (FedAVG).

PACS. As shown in Table 1, unlike OfficeHome, FedENC provides a comparable AVG ACC of 96.73% compared to LoRA (97.26%) and PromptFL (96.74%). Furthermore, since pre-trained CLIP lacks knowledge for sketch images, a similar conclusion can be drawn compared to ModernOffice31.

DomainNet. Table 2 reports the test ACC on DomainNet. Traditional FL such as FedAVG provides a higher AVG ACC of 60.6% compared to PEFT-based FL techniques such as FedCLIP (57.49%), FAA-CLIP (56.24%) and PromptFL (59.78%). Unlike these methods, FedENC yields an AVG ACC of 67.87%, outperforming the second-best (LoRA) by 5.37%. These results suggest that in a large-domain shift setting, the proposed method can provide feasible performance. ISIC2019. Table 3 presents the test ACC and F1 score for FedENC and the baselines. Fine-tuning the entire CLIP encoder leads to a higher test metric (e.g., 71.72% AVG ACC) compared to CLIPzs, while using FedAVG further improves the AVG ACC to 73.41%. However, most of the PEFT techniques, such as PromptFL and FedAPT yield lower test performance compared to FedAVG. Unlike these methods, FedENC provides an AVG ACC of 81.53%, with an AVG F1 score of 69.78%, outperforming the second-best approach by 8% and 13.29%, respectively. In addition, Figure 2 illustrates the Receiver Operating Characteristic (ROC) curves and Decision Curve Analysis (DCA) [Vickers and Elkin, 2006] on client C1 to C3. For example, FedENC yields an AUC value of 0.946 on C1, outperforming the second-best approach (FAA-CLIP) by 0.017. The DCA curves show that FedENC provides a higher net benefit compared to other methods across a wide range of thresholds (0, 0.8).

Brain ICH. Figure 3 shows the test ACC for FedENC and baselines using box chart. For example, FedENC provides the highest AVG ACC of 68.84% under 5 clients, outperforming the second-best (LoRA) by 3.4%. Furthermore, the box plot for FedENC is narrower compared to other PEFT approaches such as PromptFL, indicating low variance across clients. Similar conclusions can be drawn from other client settings.

Computation and communication overhead. Table 4 reports the total computation time, communication load and GPU memory usage for baselines and FedENC. Compared to strong baselines like LoRA, it achieves higher accuracy (e.g., +5.37% on AVG) with lower computation and communication costs (e.g., 39.6% less time and 2.96× lower data transmission than LoRA). While FedENC requires more computation time than some PEFT methods (e.g., FedCLIP, FedAPT) on the ISIC2019 dataset, it outperforms these methods by up to 13.21% on AVG. Notably, FedENC maintains a consistently lower GPU memory footprint (e.g., 3.1GB).

4.4 Model Analysis

Ablation study. We validate the contribution of each component using the OfficeHome and ISIC2019 datasets. As summarized in Table 5, introducing local experts improves the test ACC compared to using global expert alone (e.g., AVG increased from 82.9% to 83.22% on OfficeHome). Furthermore, using LNC further reduces the impact of data heterogeneity, increasing the ACC and F1 score to 86.68% and 66.36% on OfficeHome and ISIC2019, respectively. Finally, combining all techniques leads to the best AVG of 86.76% and 69.78%, respectively.

Calibration. We evaluate calibration performance using Expected Calibration Error (ECE) and reliability diagrams [Vaicenavicius et al., 2019] on OfficeHome and ISIC2019. The reliability plots are shown in Figure 4 and Figure 5. We observe that: 1) On OfficeHome, FedENC achieves a calibration error comparable to that of FedCLIP on clients A and C. 2) On ISIC2019, FedENC yields lower ECE than both PromptFL and FedCLIP on C3, C4 and C6, while achieving similar error to them on clients C1 and C5.

Sensitivity. We explore the sensitivity of the hyperparameter λ1 and λ2 on OfficeHome and ISIC2019 datasets. As shown in Figure 6, a higher λ1 is beneficial for client A, while a lower λ2 is useful for client A. The optimal result can be achieved by setting λ1 to 0.5 and λ2 to 0.1, respectively. For ISIC2019, a higher λ1 leads to a better AVG F1 score, while an increase of λ2 can reduce prediction diversity, leading to a lower test F1 score. Finally, the optimal result can be obtained by setting λ1 to 1.0 and λ2 to 0.1, respectively.

Adaptability. We replace the CLIP encoder with other backbones to verify its adaptability. Specifically, SigLIP [Zhai et al., 2023] and EVA02-CLIP [Fang et al., 2024] are used for experiments in the OfficeHome and ISIC2019 datasets. As illustrated in Figure 7, FedENC improves the performance of SigLIP and EVA02-CLIP on OfficeHome dataset by 26.53% and 2.9% in AVG, respectively. Furthermore, in the ISIC2019 dataset, FedENC yields an AVG ACC of 76.82% and 83.17%

with SigLIP and EVA02-CLIP, outperforming the zero-shot baseline by 30.6% and 64.44%, respectively.

Scalability. To evaluate the scalability of FedENC on DomainNet, each domain is divided into five subdomains using a Dirichlet distribution following [Su et al., 2024], resulting in a total of 30 clients. Figure 8 (left) illustrates the test ACC for FedENC and the baselines. FedENC achieves an AVG ACC of 66.91% on DomainNet, outperforming LoRA and PromptFL by 3.8% and 7.11%, respectively.

Impact of γ. We validate the impact of γ used in Eq. 6 on OfficeHome and ISIC2019 datasets. As illustrated in Figure 8 (right), using a lower γ value shows a performance drop (i.e., putting more weights on common classes), while setting γ to 0.7 leads to the highest AVG for both datasets. We note that a γ value higher than 0.7 leads to a lower performance, this finding suggests that a moderate weight on rare classes is important for feasible performance and vice versa.

5 Conclusion

This study presented FedENC, a novel CLIP-based FL framework to address data heterogeneity. FedENC learns shared knowledge across clients using a global adaptation module, while preserves local knowledge through two local expert modules. Furthermore, NC is introduced to mitigate heterogeneity, with a KL and cosine similarity based alignment technique to improve prediction consistency. Experimental results show that FedENC achieves remarkable performance with reasonable computation and communication overhead.

Acknowledgments

This research was funded by the Guangxi Science and Technology Base and Talent Project (#2022AC18004 and #2022AC21040), and the National Natural Science Foundation of China (Grant #82260360).