TriHAI: A Combination Enhanced Tri-Mode Deferral for Human-AI Team

1 Introduction

Over the past decades, AI has made significant strides [Dosovitskiy, 2020]. Although AI-based systems have achieved remarkable results on controlled benchmarks, their reliability remains a concern [Zhang et al., 2020; Koh et al., 2021; Zhou et al., 2022]. For example, in complicated real-world tasks such as medical image diagnosis, even state-of-the-art models can still make hard-to-anticipate errors [Varoquaux and Cheplygina, 2022; Cid et al., 2024]. For these tasks, humanAI collaboration is a promising approach [Wilder et al., 2020; Liu et al., 2024]. Existing research has demonstrated that by leveraging their complementary strengths, human-AI collaboration outperforms either humans or AI alone [Yu et al., 2021; Zhao et al., 2025; Tanno et al., 2025].

As one of the major human-AI collaboration methods, Learning to Defer trains a deferral strategy that decides to predict by AI or defer to humans based on comparing their confidences [Madras et al., 2018; Mozannar and Sontag, 2020; Verma and Nalisnick, 2022]. Further, recent works go beyond simply deferring to humans, pairing deferrals with AI assistance to support humans. Specifically, it is proven that the awareness of AI inability enhances human accuracy [Bondi et al., 2022]. Auxiliary information like AI predictions and explanations augments human decision-making [Tariq et al., 2025; Strong et al., 2024]. In addition, some research [Kerrigan et al., 2021; Singh et al., 2023] maps human predicted labels into soft labels through a confusion matrix and combines them with AI's probabilities via Bayesian theory. This idea has also been incorporated into Learning to Defer [Gupta et al., 2023], yielding a variant of deferral that outputs either the AI's prediction or human-AI combined prediction.

∗Corresponding author

Deferring AI-uncertain samples to combination often performs better than humans alone, but this does not hold in some cases. Combination [Kerrigan et al., 2021] gains from the fusion of model and human results with their corresponding confidences. However, when the model provides a lowconfidence result while the human provides a high-confidence one, combination performance is degraded by noise from the model. In such cases, the human is correct but combination fails. We validate this statement on real-world datasets, i.e., ImageNet-16H and Chaoyang. As shown in Figure 1, there exist some samples that only the human correctly predicts (marked in yellow). Therefore, an ideal deferral should behave as follows: if the model is confident enough, AI-only is prioritized; otherwise, human involvement is triggered, either via human-only or combination. Some existing works have investigated selecting among these three modes. Zhang et al. [Zhang et al., 2024] directly concatenate human labels with model probabilities, failing to capture fine-grained distributions and leading to suboptimal collaboration. He et al. [He et al., 2025] require expensive concept annotations for their concept-driven selection. We argue that the key issue for mode selection lies in distinguishing human-only mode from combination mode. These two modes incur the same human cost, so the selection should focus on accuracy. Thus, the key issue lies in: when does the model's result degrade the combination, and how to mitigate its effect on combination?

To address the above issue, there are two challenges. First, Combination Degradation: how to select between humanonly and combination modes with higher accuracy? Existing works typically select between AI-only and human-involved modes based on the model's confidence - an explicit one that can be directly obtained from its probability vector. By contrast, selecting between human-only and combination is more difficult since human confidence is an implicit one that is often unavailable for both. Second, Degradation Mitigation: how to reduce the model's negative impact on combination and improve the system performance? The system performance heavily relies on each mode's performance. Since humans and AI are fixed, how to improve the performance of combination is important for the system performance. Unreliable model predictions can introduce noise into combination as illustrated above, and thus the key is to suppress this noise.

To address these challenges, we propose two solutions: (i) First, we introduce adaptive selection to choose among AI-only, human-only and combination modes. It formulates the mode selection as a ternary decision problem, leveraging a hybrid scoring mechanism that compares intrinsic model confidence to parameterized assessments from a discriminative gating network to identify the suitable decision mode. (ii) Second, we propose human-guided Conformal Prediction to refine model's prediction sets. It calculates the nonconformity scores by incorporating human priors and constructs prediction sets via adaptive group calibration, providing theoretical guarantees on marginal coverage.

In this paper, we propose TriHAI, a novel tri-mode deferral method that adaptively routes samples among AIonly, human-only, and refined human-AI combination modes. Specifically, given a sample, we feed it into a pre-trained classifier to obtain the model prediction and confidence score. Simultaneously, the image is given into a gating network to generate scores for the human and combination modes. Based on the scores, the system routes the sample to one of three modes: outputting the model prediction directly, querying the human, or employing human-guided Conformal Prediction to refine the model's prediction set, and combine it with the human prediction. Our main contributions are as follows:

• Ternary Deferral: We propose a tri-mode deferral method integrating AI-only, human-only, and human-AI combination modes to accommodate diverse scenarios.
• Adaptive Selection: We design a score-driven selection strategy to choose modes for samples, significantly improving the appropriateness of sample assignment.
• Human-guided Conformal Prediction: We propose a
• refined combination mode that enhances human-AI fusion accuracy by leveraging human-guided Conformal Prediction to suppress model probability noise.
• Experiment Study: Experiments on three real-world datasets show that TriHAI outperforms other human-AI baselines by up to 9.57% in overall accuracy.

2 RelatedWork

2.1 Learning to Defer Frameworks

As a prevalent method of human-AI collaboration, Learning to Defer assigns tasks to either AI or humans. Early works estimate human error or confidence to determine whether to defer [Madras et al., 2018; Raghu et al., 2019]. [Mozannar and Sontag, 2020] then formulates deferral as a differentiable objective and introduces surrogate losses to support end-to-end learning. Subsequent works further refine the surrogate for better joint optimization [Okati et al., 2021] and for calibration to human correctness [Verma and Nalisnick, 2022]. Some works show that surrogate-based approaches may fail in the realizable settings and propose realizable consistent surrogate losses [Mozannar et al., 2023; Mao et al., 2024]. Meanwhile, some works [Mao et al., 2023; Gupta et al., 2023] point out that joint training is expensive since it cannot directly leverage pre-trained classifiers. Consequently, recent studies propose a two-stage approach, which first selects a pre-trained classifier and subsequently learns a separate deferral module [Narasimhan et al., 2022; Gupta et al., 2023; Mohri et al., 2023].

2.2 Enhanced Deferral withModel Output

Some works argue that model output could still be useful to humans, even when the decision is deferred to humans. Research indicates that awareness of model inability enhances human accuracy [Bondi et al., 2022; Jabbour et al., 2025]. Model-predicted probabilities could also support humans [Tariq et al., 2025; Chen et al., 2023], while model-generated explanations offer humans comprehensive guidance [Strong et al., 2024; Strong et al., 2025]. Rather than utilizing model output as support, some research proposes a collaboration method that leverages Bayesian theory to combine the model's probabilities with the human's classlevel predictions [Kerrigan et al., 2021; Zou et al., 2024; Gao et al., 2025]. [Gupta et al., 2023] integrates this method into Learning to Defer, which can either make a prediction by model or defer to human-AI combination. Only a few works explore moving beyond the above binary settings, such as [Zhang et al., 2024], which attempts to unify deferral and complementarity. [He et al., 2025] proposes a concept-driven method to choose among three decision modes.

However, most works typically confine the decision space to a rigid binary choice, i.e., AI or humans, and AI or combination, which may exclude the most suitable mode for a given sample. In practice, AI, humans, and human-AI combination are each indispensable. Although a few methods consider a ternary framework, they remain limited by either the direct fusion of human labels with model predictions, which yields suboptimal collaboration, or the reliance on concept annotations that incurs prohibitive costs for ternary selection.

3 Approach

3.1 ProblemFormulation

We consider a K-way image classification task defined over an input space X and a label space Y = {1, . . . ,K}. There are two types of decision-makers involved: a model and a human. For an input sample x ∈ X, a pre-trained model m : X → RK yields a probability vector m(x); a queryable human, modeled as a decision function ˜h : X → Y, provides a hard label ˜h(x) ∈ Y.

Our goal is to maximize collaboration performance between the human and the AI, thereby improving the overall classification accuracy. Beyond the predictions of the model m(x) and the human ˜h(x), we introduce the third mode: human-AI combination. This mode combines the predictions from both the human and the AI to yield a hybrid result c(x). Therefore, to be more specific, our objective is to design an adaptive decision framework that can flexibly switch among three modes: AI-only, Human-only, and Human-AI combination.

3.2 Implementation of Three Modes

The overall framework is illustrated in Figure 2. In this subsection, we detail the implementation of the three modes.

• AI-only Mode: The input x is processed by the pretrained model m to produce a probability vector m(x). The final prediction is argmax(m(x)).
• Human-only Mode: The input x is deferred to the human, who provides a class label ˜h(x) as the final prediction.
• Human-AI Combination Mode: This mode integrates the model output m(x) and the human output ˜h(x) via a specific decision combination strategy to yield the combined prediction result c(x).

Since the AI-only and human-only modes are relatively simple, the following content focuses primarily on the implementation of human-AI combination.

Combination of AI and Human

As confidence is critical for human-AI combination decision, we first map the human's hard label ˜h(x) into soft labels h(x) via confusion matrix ϕ, defined as:

where u denotes the human prediction and v denotes the true label. To reduce the variance of maximum likelihood estimation, we follow prior works [Kerrigan et al., 2021] and place a Dirichlet prior on each column of the confusion matrix:

where ζv ∈ RK is defined by (ζv)u = γ if u = v and (ζv)u = ϑ otherwise, assigning higher mass to diagonal entries and lower mass to off-diagonal entries.

Given training data, the posterior confusion matrix is obtained by combining this Dirichlet prior with the empirical counts. The confusion matrix ϕ is then used to map the human's hard label ˜h(x) into soft labels h(x) as follows:

where · denotes the selection of the row corresponding to the index ˜h(x). Then, we combine the human soft labels h(x) and the model probability vector m(x) based on Bayesian theory, thereby obtaining the final human-AI collaborative decision c(x). Specifically, for the class k, the combined probability P (y = k|h(x), m(x)) can be defined as:

where m(x)k denotes the model's probability of class k, and h(x)k represents the human's confidence of class k. Finally, the human-AI combination decision c(x) is defined as:

Human-Guided Conformal Prediction

The model output m(x) provides probabilities for all the candidate classes, which may inevitably includes some erroneous classes that exert a negative impact on the final result. We propose human-guided Conformal Prediction (CP) to filter erroneous classes, yielding a refined vector m∗ (x) for fusion. Conformal Prediction (CP) [Angelopoulos and Bates, 2021; Sun et al., 2023] is a distribution-free method that outputs prediction sets containing ground-truth labels with a desired marginal coverage guarantee 1 - α. In our tasks, since humans often have a stronger semantic understanding than the model, we incorporate human soft labels h(x) to guide the computation of nonconformity score, which quantifies label disagreement. Considering that h(x) may also be uncertain, we introduce a parameter 1-β to weight the contribution. It proceeds in three steps, as illustrated below:

Definition of Human-guided Nonconformity Score: For an input sample x with ground-truth label y, we define the human-guided nonconformity score S(x, y) as follows:

where β ∈ [0, 1] balances the model probability m(x)y and human confidence h(x)y on the true label.

Calibration: We construct a calibration set Dcal = {(xi, yi, ˜h(xi))}ni=1 and compute nonconformity scores Scal for each sample. To ensure 1 - α marginal coverage, we set the threshold ˆ q as the k-th smallest score in Scal, where k = ⌈(n + 1)(1 - α)⌉.

Prediction: For a new sample x, we refine the prediction set using the calibrated threshold ˆ q:

We then obtain the refined vector m∗ (x) by masking classes outside Γ(x) and update the model output:

Sample Grouping

In Equation (6), we employ a parameter 1 - β to weight the human contribution, assigning a lower β (i.e., a larger 1 - β) when human judgment is reliable. To quantify this, we compare the statistical accuracy for each class k, derived from the diagonal elements of the human confusion matrix ϕ and the model confusion matrix ϕm:

Then, we partition the label space Y into two mutually exclusive subsets: the set contains the samples that are easy for the human Yeasy and that are hard for the human Yhard:

We assign each sample x to geasy if the human prediction ˜h(x) belongs to Yeasy, and to ghard otherwise. We determine optimal group-specific parameters βeasy and βhard via grid search on a validation set Dval. We select values within [0, 1] that maximize the final fusion accuracy for each group. The complete process for implementing the combination mode is given in Algorithm 1.

Algorithm 1 Implementation of Combination Mode

Require: Calibration set Dcal, validation set Dval, sample x, miscoverage rate α, step size ∆β.

Ensure: Prediction result c(x).

• 1: Preparation Compute the confusion matrices for the human and the model. Identify Yeasy and Yhard.
• 2: Model Prediction Refinement
• 3: Partition Dcal and Dval into two groups.
• 4: for geasy and ghard do
• 5: for β ∈ [0, ∆β, ..., 1] do
• 6: Compute Scal on Dcal and obtain ˆ q under α.
• 7: Update m(x) by human-guided CP and evaluate fusion accuracy on Dval.
• 8: end for
• 9: Select β with the highest fusion accuracy.
• 10: end for
• 11: For a new sample x, identify its group g, make humanguided CP and update the model's probability vector m(x) ← m∗ (x).
• 12: Human-AI Combination
• 13: Compute combined probability P(y = k | h(x),m(x)) for k ∈ Γ(x) through Equation (4).
• 14: Obtain the final prediction c(x) through Equation (5).
• 15: return c(x).

3.3 Adaptive Selection Among EachMode

Next, we address how to adaptively route each sample. We formulate adaptive selection as a ternary decision problem: given an input x, the goal is to select the suitable mode r ∈ {m, h, c} among the AI, the human, and combination.

Specifically, for each instance x, we compute the scores s(x) = [sm(x), sh(x), sc(x)] that quantify the suitability of each mode. The score for the AI-only mode, i.e., sm(x), is non-parametric and derived directly from the probability vector m(x), as itself serves as a direct assessment of the model's capability. We formulate it as follows:

where m(1) and m(2) denote the highest and second-highest probabilities, respectively. In contrast, the scores involving human participation, i.e., sh(x) and sc(x), are parameterized and generated by a gating network fθ:

During the test phase, the module selects the mode with the highest score as the optimal one ˆ r.

The loss function of the gating network is given below:

where λ is a hyperparameter used to balance the regularization strength. The first term aims to maximize the probability that the optimal mode is selected. It is defined as follows:

where N is the total number of train samples, and P(r|x) is the probability that r mode is selected. It can be obtained through the following formula:

The equation normalizes the scores across the three modes. Then, the target weights wr(x) are assigned dynamically based on mode characteristics. For the AI-only mode (r = m), the weight serves as a binary indicator activated strictly upon correct prediction:

Conversely, for the other two modes involving human participation, i.e., r ∈ {h, c}, the weights are calculated as follows:

where

The weight comprises two terms. Λr(δ) utilizes the sigmoid function σ to act as a differentiable switch governed by uncertainty δ = 1-m(1), where ηr sets the activation threshold and κr controls sharpness. Simultaneously, ¯ Cr models human's utility, applying a base cost cb and a heavy penalty cp for errors via an indicator function.

The second term of total loss enforces the correct relative ordering between the scores of the human (sh) and that of combination (sc):

where I{a≻b} indicates samples where mode a is correct while mode b is incorrect, and ψ(z) = log(1 + ez ) is the softplus function used to smoothly penalize score inversions.

4 Experiments

4.1 Experimental Settings

Datasets. We evaluate the proposed method on three datasets. ImageNet-16H [Steyvers et al., 2022]: This dataset contains 1,200 ImageNet test images from 16 classes, each corrupted with phase noise at ω ∈ {80, 95, 110, 125} (4,800 noisy images). Model predictions come from diverse pretrained networks, and human predictions are obtained by randomly sampling one from six annotations per image. We utilize the ω = 125 subset, as other levels lack the 10% accuracy gap necessary to establish scenarios between humans and AI.

Chaoyang [Zhang et al., 2024]: This dataset contains colon slides collected from Chaoyang Hospital. It has 6,160 images across four classes: normal, serrated lesion, adenocarcinoma, and adenoma. Model predictions are generated by three distinct trained models. Human labels are derived from ground truth annotations provided by professional doctor.

CIFAR-100 [Hemmer et al., 2022]: This dataset contains 60,000 images categorized into 100 subclasses and 20 superclasses. Model predictions are obtained from two separately trained models, while human labels are artificially generated.

Following prior works, we create synthetic humans proficient in 2 to 18 superclasses. A human predicts perfectly within its proficient superclasses; otherwise it randomly outputs an incorrect superclass label. This yields nine humans, fromwhich we select two with accuracies of about 70% and 80%.

Baselines. We compare our proposed TriHAI with the following baselines belonging to three categories.

• Learning to defer. OVA-Surrogate [Verma and Nalisnick, 2022] and Realizable-Surrogate [Mozannar et al., 2023], which jointly learn classifier and deferral through two consistent surrogate losses, respectively; Two-Stage Deferral [Mao et al., 2023], which trains the classifier and deferral module sequentially.
• Human-AI combination. CM [Kerrigan et al., 2021], which fuses human and AI predictions via Bayesian combination; CP-CM, which further refines AI predictions via human-guided CP.
• Deferral-Combination hybrid methods. C2D [Gupta et al., 2023], which combines Learning to Defer with Bayesian human-AI combination; LECODU [Zhang et al., 2024], which allows choosing among model prediction, human prediction, or human-AI collaboration.

Implementation Details. We establish scenarios across three datasets covering AI-superior, comparable, and humansuperior cases, defining superiority as an accuracy gap exceeding 10%. For ImageNet-16H, we directly use three pretrained models: VGG-19 [Simonyan and Zisserman, 2014], GoogLeNet [Szegedy et al., 2015], and AlexNet [Krizhevsky et al., 2012]. For the other datasets, we split the data to train backbone models. For Chaoyang, 4,724 images are used to train NSHE [Zhu et al., 2021], NF-Net [Cao et al., 2020], and Co-teaching [Han et al., 2018]. By pairing each of these models with the human, we establish three distinct scenarios for each of above two datasets. For CIFAR-100, 50,000 images are used to fine-tune AlexNet and ResNet-18 [He et al., 2016], which are then paired with two simulated humans to form four scenarios, including one AI-superior, one human-superior, and two comparable scenarios. The remaining data from Chaoyang, CIFAR-100, and ImageNet-16H are reserved for training and validating our method.

4.2 Overall Performance

We evaluate the accuracy of our method and the baselines on ImageNet-16H, Chaoyang, and CIFAR-100 datasets. The results in Table 1 show that our method outperforms all the baselines. Specifically, compared to humans or AI alone, our method achieves at least 7.17% improvement in General setting, which underscores the necessity of human-AI collaboration. Additionally, our method outperforms CP-CM. As AI-only, human-only, and CP-CM constitute the three foundational decision modes within our approach, it demonstrates that adopting any single mode is not optimal and therefore we should make adaptive selection among the three modes.

Next, we conduct ablation studies to evaluate humanguided CP and adaptive selection. First, we remove humanguided CP, and instead directly combine human and AI predictions via Bayesian rule. Experimental results indicate that this ablation leads to a significant drop in accuracy, particularly on CIFAR-100 with a decrease of 2.33%. Second, for adaptive selection, we employ a threshold-based strategy: samples are routed to the AI-only mode if model confidence exceeds τAI, to the human-only mode if below τh, and to combination otherwise. This ablation leads to a clear accuracy drop, highlighting the importance of adaptive selection.

Intuitively, deferring to combination outperforms deferring to humans alone, as it integrates human knowledge and model predictions. However, the conclusion doesn't always hold. Specifically, in human-superior settings, the results reveal that C2D (select AI or combination) frequently underperforms the Learning to Defer baselines, such as Two-Stage Deferral (select AI or humans). This counterintuitive result suggests that, in some scenarios, relying on humans alone can be better. Therefore, it's important to include all three modes.

4.3 Accuracy and Coverage

We analyze the relationship between accuracy and coverage, defined as the proportion of samples predicted by AI, to evaluate system performance under manpower constraints. Notably, as coverage increases, the demand for human manpower decreases accordingly. We generate accuracycoverage curves by sorting test samples using deferral scores and sweeping a threshold to vary coverage from 0 to 1. In our method, we define the deferral score as the maximum of the human-only and combination scores, and defer each sample to the mode that attains this maximum. As shown in Figure 3, our method achieves superior accuracy compared to the baselines across most coverages. Additionally, when coverage is 0, meaning the model is completely uninvolved, our approach (humans or combination) outperforms the C2D (combination). This demonstrates that assigning all samples to combination mode is not always the optimal strategy.

4.4 Evaluation on Human-AI CombinationMode

We also investigate the impact of the parameter β on the accuracy of combination mode across sample groups of varying difficulty (i.e., easy vs. hard). It is worth noting that when β = 1, our method reduces to standard CP. As illustrated in Figure 4, our proposed human-guided CP, starred at the optimal β values for peak combination accuracy, consistently outperforms standard CP (β = 1) across all experimental settings, demonstrating the effectiveness of integrating human guidance. We observe a lower optimal β for the easy group compared to the hard group. As 1 - β controls the human's weight (Equation (6)), this supports our assumption that easy samples require higher reliance on humans. Notably, on Chaoyang dataset, we observe a significant accuracy gap between easy and hard groups. It stems from nearperfect human performance (over 98%) on two categories, which sharply elevates the easy group's accuracy.

4.5 Evaluation on Adaptive Selection

To evaluate the suitability of our mode selection, we first define the ground truth by prioritizing the AI-only mode (if correct or if all modes fail) over successful human or combination modes. Subsequently, we evaluate the performance using precision, recall, and F1-score. As presented in Table 2, our method achieves high F1 scores across all three datasets. The result demonstrates that our method accurately assigns samples to their appropriate modes. AI-only and human-only modes exhibit high precision but low recall, unlike combination. This indicates a cautious strategy that defaults ambiguous samples to combination. Future work will focus on the switching boundaries among these modes.

We further examine how the selection ratios of the three modes change under different numbers of classes humans can predict K. We use CIFAR-100 with a fixed ResNet-18 model and nine synthetic humans of varying expertise. As shown in Figure 5, as the human's capability gradually increases, the selection ratio of the human-only mode increases, while the selection ratio of the AI-only mode decreases. The pretrained model ResNet-18 achieves an accuracy of about 72%. When K reaches 14, corresponding to an accuracy of about 70%, human and model capabilities become comparable, and combination mode attains its highest selection ratio.

5 Conclusion

In this paper, we propose TriHAI, a method designed to optimize decision-making by adaptively routing samples among AI-only, human-only, and human-AI combination modes. Through an adaptive selection approach, we dynamically identify the suitable decision mode for each sample to maximize system performance. Furthermore, we employ humanguided Conformal Prediction to suppress model probability noise, thereby enhancing the accuracy of human-AI combination mode. Experimental results on three datasets demonstrate that our proposed method consistently outperforms existing human-AI collaboration methods. This approach effectively addresses the limitations of binary deferral strategies in complex scenarios, offering a robust solution for advanced human-AI teaming.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (No.62372381).