CoFL: Consensus Driven Human-AI Collaborative Federated Learning

1 Introduction

Federated Learning (FL) enables multiple clients to collaboratively train a global model without sharing raw data [Wang et al., 2025a; Wen et al., 2023]. Due to its privacy preserving property, FL has attracted widespread attention across various domains, including healthcare [Wu et al., 2022], finance [Zhao et al., 2023], and recommender systems [Wang et al., 2025b]. Despite the success of FL, most methods assume a labeled dataset is available at each client [Tang and Yu, 2025; Shao et al., 2024]. In practice, data often arrive at clients in an unlabeled streaming fashion [Sun et al., 2025; Yu et al., 2021]. For example, hospitals constantly accumulate raw diagnostic data that often lack labels. The absence of labels prevents local training and hinders the deployment of FL.

Recent efforts that focus on this issue roughly fall into Federated Semi-Supervised Learning (FSSL) and Federated Active Learning (FAL). Specifically, FSSL integrates semisupervised learning to generate pseudo-labels [Diao et al., 2022; Lee et al., 2024], while FAL queries local human expert to annotate informative subsets of local unlabeled data at each client [Chen et al., 2024; Sun et al., 2025].

* Corresponding authors.

Despite their promise, the practical adoption of the above solutions remains challenging. FSSL often struggles with the inaccurate pseudo-labels generated by global or local models [Zhao et al., 2025a]. Meanwhile, most FAL methods assume human experts are oracle annotators and overlook annotation inconsistencies caused by the subjectivity of experts. Such subjectivity leads to differences in experts' capability, i.e., their accuracy in annotating a specific class. This limitation is particularly severe in FL where experts are distributed across different clients and may have different educational backgrounds and preferences. Overall, labels fromAI models and human experts are often unreliable.

Recent studies on human-AI complementarity [Vaccaro et al., 2024; Steyvers et al., 2022; Z¨oller et al., 2025] demonstrate that human-AI hybrid systems can outperform either humans or AI models alone. These findings motivate a potential solution: leveraging human-AI collaboration to generate more reliable labels. Specifically, we consider a new FL method where each client pairs a human expert with the global or local model to collaboratively label incoming data, as illustrated in Fig. 1.

Nevertheless, implementing the aforementioned potential solution still presents several challenges. (1) How to achieve effective human-AI collaboration? Most existing humanAI collaboration approaches assume that the AI model is well trained [Kerrigan et al., 2021; Zhang et al., 2024]. In contrast, in the early stages of FL, limited available data leads to insufficiently trained models. (2) How to identify samples for querying human experts? Constrained by limited time and budget, it is more practical to query only a subset of unlabeled data for human annotations. Although sample querying strategies have been widely studied in the field of FAL, existing FAL methods [Kim et al., 2023;

Sun et al., 2025] cannot be directly applied because they ignore expert subjectivity. To address the above challenges, our idea is to leverage consensus among human experts to guide human-AI collaboration and sample querying. Formally, we define consensus as the majority-vote label among all human experts for the same sample. Existing studies [Raykar et al., 2010] have shown that consensus among experts is more robust and closer to the ground truth label, which can be used to stabilize human-AI collaboration and improve sample querying by calibrating experts' subjectivity. To observe consensus across experts, existing methods assume that each sample is annotated by multiple experts [Showalter et al., 2024; Kelly et al., 2025]. This assumption, however, does not hold in privacy-preserving FL settings, where each client only has access to annotations from its local expert. Consequently, how to estimate cross-client human expert consensus under privacy constraints remains a critical challenge.

In this paper, we propose CoFL (Consensus-driven Human-AI Collaborative Federated Learning), a novel FL method that establishes a collaboration mechanism between human experts and AI models for reliable label generation under streaming unlabeled data. To tackle the aforementioned challenges, the key idea of CoFL is to learn knowledge from the historical annotations of human experts and aggregate such knowledge at the server to form cross-client consensus, which guides both human-AI collaboration and sample querying. Specifically, CoFL employs an Evidential Deep Learning (EDL) model [Sensoy et al., 2018] on each client to learn the labeling behavior of experts influenced by subjectivity and aggregates the EDL models at the server into a consensus estimator. Moreover, CoFL calibrates the EDL outputs with the Expectation-Maximization (EM) algorithm, enabling the consensus to guide collaboration and sample querying more effectively.

Our main contributions can be summarized as follows:

• We propose a novel human-AI collaborative FL method CoFL, which utilizes the collaboration between human experts and models to generate reliable labels.
• We introduce a privacy-preserving consensus estimation method that constructs cross-client human expert consensus without sharing raw data or annotations. Meanwhile, we further introduce a calibration mechanism based on the EM algorithm, which enables the estimated consensus to better improve the effectiveness of humanAI collaboration.
• Experimental results demonstrate that, compared with five baseline methods, CoFL generates more reliable labels and significantly improves the performance of global model on three datasets.

2 RelatedWork

2.1 Federated Learning with Unlabeled Data

Existing FL methods with unlabeled data can be broadly categorized into two classes: FSSL and FAL. FSSL leverages global or local models to generate pseudo-labels for selftraining [Jiang et al., 2024; Diao et al., 2022]. For example, SAGE [Liu et al., 2025] leverages the confidence discrepancy between the local model and the global model to assess the reliability of pseudo-labels and accordingly correct them. In contrast, FAL incorporates active learning into FL and queries human experts for the most informative samples under a limited labeling budget [Cao et al., 2023; Chen et al., 2024]. For example, LoGo [Kim et al., 2023] proposes a two-stage sampling strategy, in which local models are first used to cluster local samples, and then the global model is employed to identify the samples with highest uncertainty within each cluster. In this way, LoGo achieves robust and efficient sample querying.

The work most relevant to ours is [Sun et al., 2025], which formulates the sample querying problem as a partially observable Markov decision process and solves it with multi-agent reinforcement learning. Compared to [Sun et al., 2025], we consider a more realistic scenario in which the human experts are subjective and may provide inconsistent annotations.

2.2 Human-AI Collaboration

Although AI has made great breakthroughs in recent years, it is widely acknowledged that AI models still exhibit inherent weaknesses [Geirhos et al., 2020]. In contrast, humans retain unique advantages in perception and reasoning [Yax et al., 2024]. Thus, human-AI collaboration has emerged as a promising paradigm for addressing complex problems. Existing human-AI collaboration methods can be broadly categorized into two paradigms [Zhang et al., 2024; Zhao et al., 2025b]: Learning to Complement (L2C) and Learning to Defer (L2D). L2C focuses on fusing human and AI outputs to produce superior results. For example, [Steyvers et al., 2022] proposed a Bayesian fusion framework that integrates human and AI predictions. [Kerrigan et al., 2021] incorporates confusion matrices and calibration mechanisms for both humans and models, enabling effective human-AI collaboration. L2D aims to improve human-AI collaboration performance by assigning tasks to either the AI model or human expert. For instance, [Madras et al., 2018] models the AI and the human expert as a two-stage collaborative decision framework, in which the AI not only needs to learn how to predict the class of the sample, but also needs to learn whether to defer the sample to the human expert. Furthermore, [Zhang et al., 2024] extends the L2D framework to multi-expert settings by jointly learning when to defer and how many experts to involve.

3 SystemOverview

3.1 System Setting

We consider an FL system consisting of a central server and K clients that collaboratively train a global model θg ∈ Rd for a C-class classification task. We model the entire FL process as T discrete time steps. At each time step t ∈ {1, 2, . . . , T}, each client k ∈ [K] receives a newly arrived unlabeled dataset Dtk , where each sample x ∈ Dtk is drawn from a distribution Pk. We assume ⋃k∈[K] Pk = Pglob, while Pk ̸= Pglob for all k ∈ [K], reflecting the non-independent and identically distributed (Non-IID) nature of data across clients[Li et al., 2022]. After generating labels for the newly arrived samples, clients participate in B FL communication rounds at the current time step to collaboratively train the global model.

Human Experts and Sample Querying. We assume that each client is associated with a human expert, denoted by hk. At time step t, the client k selects a subset of samples Qtk ⊆ Dtk to query its associated expert hk for annotations and the query strategy is denoted by φ. Due to constraints on time and budget, the query budget is limited by |Qtk | ≤ nk. For any sample x ∈ Qtk , the expert hk provides an annotation hk(x) ∈ {1, . . . , C}. The human annotation set is denoted as U tk = {(x, hk(x)) | x ∈ Qtk }. φ will be detailed in Sec. 4.2.

Consensus. For a given sample x, we define the consensus as the majority class determined by aggregating expertprovided labels across all clients. Specifically, suppose that x could be annotated by the expert associated with each client. We treat each annotation as a vote and define the vote count histogram as H(x) = ∑Kk=1 one-hot(hk(x)) ∈ NC , where one-hot(·) denotes the one-hot encoding. The consensus is then given by y⋆ (x) := arg maxc∈[C] Hc(x). To probabilistically characterize expert consensus, we model the latent consensus distribution with a Dirichlet distribution and the aggregated expert votes with a Multinomial distribution conditioned on it. Formally, we state the following assumption:

Assumption 1. For any sample x, there exists a latent human consensus distribution π(x) ∈ ∆C-1 such that

where α⋆ (x) ∈ RC+ denotes the unknown Dirichlet concentration parameter.

Since each client's local data cannot be exposed to experts from other clients, directly observing the consensus is not feasible. We introduce a consensus estimator g(x;w) that estimates the consensus for arbitrary samples x under privacy constraints and will be detailed in Sec. 4.1.

Consensus-enhanced Human-AI Collaboration. At time step t, each client performs label generation for its newly arrived data Dtk by leveraging three sources: (1) the global model performs inference 1 on all samples x ∈ Dtk , yielding a set of model predictions denoted as I tk = {(x, f(x; θt-1 g )) | x ∈ Dtk }, where f(x; θt-1 g ) ∈ ∆C-1 represents the class probability vector over the C classes for sample x, θt-1 g denotes the global model obtained after the previous time step; (2) the consensus estimator produces a set of estimated consensus, denoted as Otk = {(x, g(x; w)) | x ∈ Dtk }; (3) human expert annotations on the queried subset Qtk yield a labeled set U tk = {(x, hk(x)) | x ∈ Qtk }.

The fusion of these three sources forms a consensusenhanced human-AI collaboration process, denoted by Φ(I tk ,Otk , U tk ), which assigns each sample x ∈ Dtk a label y′ along with a confidence s. Samples whose confidence exceeds a predefined threshold τ are retained to form Mtk , which is then added to the historical labeled set Ltk for subsequent local training, i.e., Ltk = Lt-1 k ∪Mtk . Φ(I tk ,Otk , U tk ) will be detailed in Sec. 4.3.

1 Alternatively, the local model can also be used for inference. Using the global model is a more common choice [Sun et al., 2025].

3.2 ProblemFormulation

We adopt FedAvg [McMahan et al., 2017] for local updates and global aggregation. At the beginning, the server initializes a global model θ0 g and distributes it to all clients, each of which has a small labeled set L0k for initialization. At time step t, the previous global model θt-1 g initializes the current training process, which optimizes the following objective:

where Lt = ⋃Kk=1 Ltk denotes the union of labeled datasets across all clients at time step t.

The local loss function of client k is defined as

where ℓ(·) is a task-specific loss function.

Accordingly, our goal is to design the fusion strategy Φ, the consensus estimator g(·; w), and the query strategy φ such that each client can construct a reliable labeled set Ltk .

4 Methodology

In this section, we present CoFL in detail, which consists of three main modules: consensus estimation, consensus enhanced human-AI collaboration, and a capability-aware sample querying strategy. The framework is shown in Fig. 2.

4.1 Consensus Estimation

To estimate human expert consensus at each client, we propose a privacy-preserving consensus estimation method. The key idea is to learn a global surrogate model of human labeling behavior through federated aggregation, such that the aggregated surrogate captures an implicit consensus among distributed experts [Luo et al., 2023]. We first construct a surrogate model for human experts that can generate annotations without exposing raw data. Prior studies have shown that human decision-making is inherently uncertain [Peterson et al., 2019], while conventional neural networks typically produce point estimates and do not explicitly characterize prediction uncertainty [Gal and Ghahramani, 2016]. We model expert labeling behavior as a stochastic process that samples from a Dirichlet distribution [Yang et al., 2025; Muttenthaler et al., 2025]. For a given sample x, the labeling behavior of the hk satisfies:

Note that βk(x) and α⋆ (x) in Assumption 1 have different meanings: βk(x) represents the subjectivity and uncertainty of an individual expert hk, whereas α⋆ (x) characterizes the subjectivity and uncertainty of consensus among all experts.

To learn the Dirichlet parameters from experts, we further introduce an EDL model at client as surrogate for hk. The EDL model parameterizes a Dirichlet-Categorical distribution by predicting Dirichlet parameters [Sensoy et al., 2018]. For any sample x, the EDL outputs the Dirichlet parameters, denoted as ˆβk(x) = [ ˆ βk,1(x), . . . , ˆ βk,C(x)] ∈ RC+, where ˆ βk,c(x) = ec(x)+1. The ec(x) quantifies the amount of evidence supporting class c, while the additive constant 1 represents a uniform Dirichlet prior, ensuring a valid distribution even when no evidence is observed [Sensoy et al., 2018].

At time step t, for client k, the historical annotations provided by its human expert are denoted by U ≤t k = ⋃ti=1 U ik . Within the time step, the EDL model is trained in a federated manner together with the task model for B communication rounds. Specifically, at each communication round, client k updates its local EDL model g(·;wtk ) using U ≤t k , where wtk denotes the parameters of the local EDL model. The server then aggregates the local EDL models from participating clients to obtain a global EDL model g(·; wtg ), where wtg denotes the parameters of the global EDL. The detailed training procedure is provided in Appendix A. After B communication rounds, the global EDL model serves as consensus estimator at next time step. For any sample x, each client can locally infers the Dirichlet parameters of consensus, denoted as ˆβg(x).

4.2 Capability-Aware Sample Query Strategy

In this section, we introduce a capability-aware sample querying strategy. The key idea is to utilize the discrepancy between local and global EDL models to assess whether each sample aligns with the capability of the current human expert. While the global EDL model from the last time step g(·; wt-1 g ) captures the consensus of human experts across all clients, each local EDL model g(·;wt-1 k ) reflects the labeling behaviors of hk. The discrepancy between the local and global EDL predictions therefore provides an effective signal for assessing whether a sample is aligned with the expert's capability, i.e., whether it belongs to a category with high expert annotation accuracy.

For a sample x ∈ Dtk , the client first infers the outputs from both the local and global EDL models: ˆβk(x) = g(x; wt-1 k ), ˆβg(x) = g(x;wt-1 g ). We then compute the expected class probability distributions induced by the corresponding Dirichlet parameters, denoted as pk(x) = ˆβk (x) ∑Cj=1 ˆ βk,j(x) , pg(x) = ˆβg (x) ∑Cj=1 ˆ βg,j (x) . We quantify the capability mismatch between the local expert and the global consensus using a distributional discrepancy measure dk(x) = KL(pg(x), pk(x)), where KL(·, ·) is Kullback-Leibler (KL) divergence 2 , which quantifies the discrepancy between two probability distributions. A larger discrepancy indicates that the local expert's belief on x deviates significantly from the global consensus, suggesting a higher risk of unreliable an-

notation.

Based on the discrepancy dk(x), we select the nk samples from Dtk by ranking all candidates according to dk(x) in ascending order and selecting the nk samples with the smallest discrepancy values to formQtk .

4.3 Consensus Enhanced Human-AI Collaboration

In this section, we detail how estimated consensus is leveraged at each client to enhance human-AI collaboration. Our idea is to treat the estimated consensus as a prior and first combine it with human annotations using a Bayesian framework, then fused with model predictions. In particular, we observe that when human experts have few historical annotations, the EDL model tends to produce small Dirichlet parameters, making the prior evidence insufficient to reliably reflect the consensus strength. To address this, we apply an EM algorithm at each client to calibrate the EDL outputs.

For the unlabeled data Dtk arriving at client k, based on the query strategy mentioned above, Dtk can be partitioned into two disjoint subsets: the queried set Qtk and its complement Dtk \ Qtk . In the following, we detail how these two types of subsets are handled, respectively.

Fusion for Queried Samples. For samples x ∈ Qtk , both the estimated consensus and human expert annotations are available. Recalling Assumption 1 that human expert consensus conforms to a latent Dirichlet distribution with parameter α⋆ (x), we treat the Dirichlet parameters ˆ βg(x) estimated by g(x; wt-1 g ) as a prior over the latent consensus distribution, and regard the expert annotation hk(x) as a categorical observation. Due to the conjugacy of the Dirichlet prior to the categorical likelihood, incorporating a single expert annotation admits a closed-form Bayesian update [Showalter et al., 2024], in which the corresponding one-hot label is added directly to the Dirichlet parameters. In this way, we naturally fuse the two within a Bayesian framework.

2 https://en.wikipedia.org/wiki/Kullback-Leibler divergence

However, we observed that when the size of U ≤t k is limited, the EDL model tends to produce small Dirichlet parameters. Thus, we introduce an EM algorithm to calibrate the output of global EDL model g(·;wt-1 g ), and the calibrated result at client k can be represented as g(·;wt-1 g , zk), where zk is the calibration parameter. The above calibration procedure will be described in the next paragraph. Thus, the posterior Dirichlet parameters are finally updated as α′ (x) = β′ g(x) + one-hot(hk(x)), where β′ g (x)=g(x; wt-1 g , zk).

Subsequently, we further fuse the α′ (x) with inference result of the task global model f(x; θt-1 g ). Specifically, we first compute the expected class probability ˆ pc(x) = α′c ∑Cj=1 α′j induced by α′ (x). Let fc(x; θt-1 g ) denote the c-th entry of the global model predicted probability vector, and define the prediction confidence as λ(x) = maxc fc(x; θt-1 g ). We then weight the model prediction by its confidence as f ′c (x; θt-1 g ) = λ(x) fc(x; θt-1 g ), and combine it with ˆ pc(x) to obtain the fused probability and label as follows:

The fusion results of queried samples are denoted by Φ1(I tk ,Otk , U tk ) = {(x, y′ ) | x ∈ Qtk }. For Φ1, we do not apply confidence-threshold filtering because expert annotations are more likely to be consistent with the consensus in real-world and therefore should not be discarded.

Calibration of EDL Model. To calibrate the outputs of global model ˆβg(x) at client k, we introduce a calibration parameter zk ∈ R+ and define new Dirichlet parameters as β′ g(x) = [β ′1 (x; zk), ..., β ′C (x; zk)], where β′c (x; zk) = zkec(x) +1. We employ an EM algorithm to estimate the parameter zk. The EM algorithm 3 is an iterative method that alternates between estimating latent variables (E-step) and optimizing parameters (M-step) to maximize the likelihood with hidden variables. Specifically, for sample x ∈ Qtk , we introduce a latent variable ˜ y representing its fused label. The goal of the EM algorithm is to estimate the calibration parameter zk by maximizing the following log-likelihood:

where e(x) = [e1(x), . . . , eC(x)] denotes the evidence vector produced by the global EDL model for sample x.

Let o(x) = one-hot(hk(x)), oc(x) denotes the c-th component of the one-hot encoded vector. We directly present the formulas for the E-step and the M-step here, the detailed proof and implementation can be found in Appendix B.

• E-step: for sample x ∈ Qtk , we estimate the posterior distribution of the ˜ y under the current zk during E-step, which can be expressed as:

3 https://en.wikipedia.org/wiki/Expectation-maximization algorithm

• M-step: Given the distribution qc(x) of the latent variable ˜ y, in M-step we update the calibration parameter zk, we use z′k to represent the updated zk, then we have:

Fusion for Unqueried Samples. For samples x ∈ Dtk \Qtk , human annotations are unavailable. Therefore, their fusion relies solely on the model predictions f(x; θt-1 g ) and the calibrated estimated consensus β′ g(x) = g(x;wg, zk). Specifically, we first compute the expected class probability ˆ pc(x) = β′g,c (x;zk) ∑Cj=1 β′g,j (x;zk) induced by β′g (x). Then use Eq.5 to compute the fused label y′ . The confidence score s is obtained as s = maxc ˜ pc(x). Finally, we apply a confidence threshold τ to filter unreliable labels. The fusion results for unqueried samples are denoted by Φ2(I tk ,Otk ) = {(x, y′ , s) | x ∈ Dtk \Qtk , s > τ}.

In the end, we merge Φ1(I tk ,Otk , U tk ) and Φ2(I tk ,Otk ) to construct the fusion setMtk .

5 Experiments

5.1 Setup

FL System Setup. We simulate an FL system that learns a classification task from unlabeled streaming using a ResNet18 model[He et al., 2016]. During each time step, each client performs B = 10 communication rounds and receives |Dtk | = 50 newly arrived samples. To model the NonIID nature of client data, we use a Dirichlet distribution with parameter α = 0.5 to assign different class proportions to different clients, which is a common setting in FL [Sun et al., 2025].

Datasets. We evaluate CoFL on three datasets: CIFAR10 [Krizhevsky et al., 2009], CINIC-10 [Darlow et al., 2018], and CIFAR-10H [Peterson et al., 2019]. CIFAR-10 and CINIC-10 are widely used image classification benchmarks. To simulate the difference of human experts on these two datasets, we assign each expert hk a unique confusion matrix Ak , where Aki,j denotes the conditional probability P(hk(x) = i | y = j), with y being the ground-truth label of sample x. This process allows us to control and model diverse labeling behaviors across experts. The method of constructing a confusion matrix will be described in the next paragraph. CIFAR-10H is a real multi-expert annotated dataset that naturally captures differences among human annotators. For each sample, we compute the empirical label proportions to obtain a label probability distribution p(x) = [p1(x), . . . , pC(x)]. The human expert on each client then samples its label from p(x). To further emphasize subjective differences across experts, we introduce a temperature parameter Th = 4.0 to smooth the distribution via ˜ pc(x; Th) = pc(x)1/Th ∑Cj=1 pj(x)1/Th .

Construction of Confusion Matrix. For each simulated expert hm, we first sample a global ability level Am ∼ N(µA, σ2A) to characterize the overall capability level of all experts in the FL system, and then sample a class specific bias vector δm ∼ N(0, σ2 cls I) to model variations in performance across different classes. The probability of correctly labeling class j is defined as p(m) corr,j = σ(Am + δm,j), where σ(·) is the sigmoid function. The confusion matrix is constructed column by column. For each true class, the diagonal entry is set as the probability of correct labeling, while the remaining probability mass is randomly distributed among the incorrect classes using a Dirichlet distribution, and each column is finally normalized to form a valid conditional probability distribution. We set µA = 1.0, σA = 1.0, σcls = 2.0.

Baselines. To evaluate CoFL, we compare it against five baselines, including four FAL methods and a FSSL method. These baselines include: (1) Coreset [Sener and Savarese, 2017]: A representative diversity-based active learning method that selects a subset of samples that best represent the feature space. In our FL setting, this procedure is performed independently on each client's local data. (2) Conf: A representative uncertainty-based active learning method that selects samples with the lowest prediction confidence. Similarly, each client applies this strategy locally in the FL setting. (3) LoGo [Kim et al., 2023]: An FAL approach that jointly considers local and global uncertainty to maintain both clientlevel and class-level diversity. (4) LeaDQ [Sun et al., 2025]: An FAL framework that formulates the query strategy as a partially observable Markov decision process and solves it with multi-agent reinforcement learning. (5) SAGE [Liu et al., 2025]: An FSSL method that leverages the discrepancy between local and global confidence to collaboratively generate and correct pseudo-labels.

Implementation Details. For the CIFAR-10, CINIC-10, and CIFAR-10H datasets, we set the query set size at each time step to nk = 10, 20, and 30, respectively, and performed for 50, 40, and 30 time steps, respectively, to evaluate our method under different settings. We use CLIP ViT-B/32 [Radford et al., 2021] to generate embeddings for the samples and employ a two-layer fully connected neural network as the EDL model, confidence threshold τ = 0.55 and each client initializes with |L0k | = 100 samples. More implementation details are provided in Appendix D.

5.2 Main Results

Global Model Performance. Fig. 3 illustrates the performance of the global model over time steps for all methods across different datasets. We can observe that our proposed CoFL method consistently achieves the best performance across all three datasets at the final time step. On CIFAR-10H and CINIC-10 datasets, CoFL achieves around a 3% improvement over FAL methods, highlighting the advantage of considering the subjectivity of human experts. SAGE exhibits unstable performance across all datasets, which is likely because it heavily relies on the model performance. When the global model is limited in performance, SAGE cannot generate sufficiently reliable pseudo-labels and therefore fails to further improve performance.

Accuracy of Labels. Fig. 4 presents the average accuracy of labels for training samples during the local update at the final time step. We observe that the proposed CoFL method achieves the highest label accuracy across all three datasets. Specifically, on the CIFAR-10 dataset, CoFL reaches an accuracy of 80.77%, outperforming the suboptimal method LoGo, by a margin of 5.15%. This result further highlights the advantages of human-AI collaboration in generating more reliable labels. Note that since SAGE generates a large number of augmented samples that do not correspond with the original data, its label accuracy is not directly comparable to other methods. In addition, we visualize the label accuracy of the training samples at each client at the final time step. Detailed experimental results are provided in Appendix C.

5.3 Ablation Study

To evaluate the effectiveness of each module in CoFL, we conduct a comprehensive ablation study. Since the consensus estimation module serves as the foundation of human-AI collaboration and the query strategy, it cannot be removed in isolation. Therefore, we focus on the ablations of humanAI collaboration and the capability-aware sample querying strategy. For the ablation of the human-AI collaboration, we consider two settings: (1) directly training with annotations provided by human experts without any collaboration (denoted as w/o collaboration), and (2) without the EMbased calibration process for EDL (denoted as w/o EM). For the capability-aware sample querying strategy, we replace it with the Coreset method and evaluate the resulting performance gap to demonstrate its effectiveness (denoted as w/o capability-aware).

We report the ablation results in Table 1. On CIFAR-10, removing any module degrades performance, especially when using only human annotations, which reduces the accuracy from 53.5% to 43.8%, highlighting the importance of humanAI collaboration. For the EM-based calibration process, removing it leads to performance degradation on all datasets, indicating the importance of calibrating EDL. The capabilityaware querying strategy is crucial on CIFAR-10 but less effective on CINIC-10 and CIFAR-10H. One possible explanation is that CINIC-10 has more diversity than CIFAR-10, the Coreset method may provide better coverage of the feature space. In contrast, the labels in CIFAR-10H are sampled from the same distributions of multiple human annotators, which makes the capabilities of different human experts more homogeneous. As a result, the capability-aware strategy cannot fully utilize its advantage.

5.4 Experiments on Diverse Scenarios

To further investigate the adaptability of CoFL under different FL scenarios, we evaluate its performance with varying numbers of clients equipped with human experts and under different overall levels of human expert capability, i.e., the overall annotation accuracy when providing labels.

Varying Numbers of Human Expert Clients. Since SAGE is not influenced by human experts, we exclude it from this set of experiments. Given that there is no significant difference among all FAL methods in the main results, we select Coreset and LeaDQ for comparison with CoFL. We consider three settings with 4, 7, and 10 clients equipped with human experts, where 10 is the same as the setting used in the main results. For clients without experts, CoFL fuses estimated consensus with global model inference, while other methods train only on their initial labeled data.

We report the experimental results in Table 2. We observe that across all datasets and under all settings with different numbers of clients equipped with human experts, CoFL consistently achieves the best performance, demonstrating its strong adaptability.

Varying Human Expert Capability Levels. We evaluate different methods under varying expert capability levels, where higher levels represent more accurate annotations provided by all experts. For the CIFAR-10 and CINIC-10 datasets, we simulated the overall expert ability by adjusting the parameter µA when constructing the confusion matrix, where µA = 1.0 for the medium level, µA = 0.2 for the low level, and µA = 2.0 for the high level. For the CIFAR-10H dataset, since expert annotations come from the same distribution, we simulate different levels of ability by smoothing the distribution with a temperature parameter Th. We set Th = 2.0, Th = 4.0, and Th = 6.0 to represent high, medium, and low ability, respectively.

We present the results in Table 3, where CoFL achieved the best performance in almost all cases, except for the high capability case on CIFAR-10H dataset. In addition, we observed that as the overall ability level of experts decreases, CoFL becomes more advantageous. For example, on CIFAR10, the accuracy of Coreset and LeaDQ methods decreased by 14.4% and 16.6% respectively from high to low ability level, while CoFL only decreased by 7.9%. This indicates that the CoFL method can alleviate the negative impact of decreased capability, mainly because its consensus estimation can extract reliable signals from inconsistent annotations.

6 Conclusions

In this paper, we propose CoFL, a human-AI collaborative FL method for addressing label scarcity in unlabeled streaming data. By leveraging human-AI complementarity and crossclient expert consensus, CoFL offers a new direction for learning from unlabeled data in FL. Future work can integrate CoFL with advanced FSSL and FAL methods for further improvement.

Acknowledgments

This work was supported in part by the National Key R&D Program of China (No. 2024YFB4505502), the National Natural Science Foundation of China (No. 62372381) and the National Natural Science Foundation of China (No. 62302396).