ReDi-FM: Frozen Foundation Model for Continual Test-Time Adaptation in Medical Image Segmentation

1 Introduction

Medical image segmentation is essential for clinical practice, powering applications from computer-aided diagnosis to anatomical structure delineation. Recent advances in Deep Learning (DL) have driven major improvements in segmentation accuracy [Wang et al., 2023a; Fang et al., 2022]. However, these models are highly sensitive to domain shifts from differences in scanners, protocols, and operators, leading to significant performance drops in new clinical settings [Guan and Liu, 2021; Sankaranarayanan et al., 2018]. While Domain Adaptation (DA) methods address this by leveraging abundant unlabeled target data, they yield models that remain fixed after deployment [Li et al., 2024; Wang et al., 2023b; Tang et al., 2024]. Test-Time Adaptation (TTA) overcomes this by adapting models online using only test data [Wang , 2020 ]. Yet, most TTA methods assume a static target distribution. In practice, real-world domains evolve continuously, necessitating Continual Test-Time Adaptation (CTTA) to enable models to update dynamically as the target domain changes [Wang et al., 2022].

∗Corresponding authors.

Most CTTA methods rely on endogenous adaptation signals obtained from the model itself, such as entropy minimization, batch-statistics alignment, or pseudo-label selftraining [Wang et al., 2022; Chen et al., 2024b; Maharana et al., 2025]. Under large domain shifts, these signals are inevitably noisy and can accumulate errors over time, limiting adaptation performance. This motivates introducing an external knowledge source to complement endogenous supervision. Large-scale Vision Foundation Models (VFMs), such as the Segment Anything Model (SAM) [Kirillov et al., 2023], provide rich and heterogeneous generic knowledge. This raises a natural question: Can a frozen VFM serve as a heterogeneous external knowledge source to improve CTTA under evolving medical domain shifts?

To answer this question, we introduce a frozen VFM into CTTA to break the knowledge upper bound of purely endogenous supervision from the source model on unlabeled target streams (Figure 1). However, directly applying a generic frozen VFM is suboptimal: while it provides rich cross-domain generic knowledge, it lacks medical semantics. We therefore propose Reciprocal Distillation with a Frozen Foundation Model (ReDi-FM), which uses structure-aware prompting to obtain target-adapted supervision from a frozen VFM and distills the resulting generic knowledge into the target model. To robustly transfer this complementary knowledge, we introduce two distillation objectives. Uncertaintyaware hard distillation emphasizes reliable regions to mitigate noise, and hard class-balanced soft distillation provides richer supervision for under-represented and challenging structures. A consensus-aware gate further stabilizes distillation when the two teachers disagree.

In summary, our contributions are: (1) We introduce frozen vision foundation models as a heterogeneous external knowledge source for CTTA, breaking the knowledge upper bound of adaptation driven solely by the source model on unlabeled target streams. (2) We propose ReDi-FM, a reciprocal distillation framework that uses source-guided, structure-aware prompting to obtain target-adapted supervision from a frozen VFM and distills the resulting heterogeneous knowledge into the target model. (3) Extensive experiments on multi-domain medical segmentation benchmarks demonstrate that ReDiFM outperforms prior state-of-the-art CTTA methods with improved long-term stability.

2 RelatedWork

2.1 Test-time Adaptation

Test-time adaptation (TTA) adapts source-pretrained models to unlabeled target data during inference. Most TTA methods use self-supervised objectives to update models online [Wang et al., 2020; Zhang et al., 2023]. For example, TENT [Wang et al., 2020] minimizes prediction entropy to improve robustness. Other methods update batch normalization (BN) statistics to reduce distribution shifts. DomainAdaptor [Zhang et al., 2023] combines source and test BN statistics and uses a generalized entropy loss for large domain gaps. However, most TTA methods assume a static target domain and perform poorly when distributions change over time.

Continual test-time adaptation (CTTA) tackles evolving target domains, where long-term adaptation leads to error accumulation and forgetting. CTTA methods are either model-based or parameter-efficient. Model-based approaches use self-supervised losses or periodic resets to combat errors [Liu et al., 2024a; Wang et al., 2022; Niu et al., 2023; Zhang et al., 2023]. CoTTA [Wang et al., 2022] averages predictions and restores parts of the model. SAR [Niu et al., 2023] filters noisy samples by gradient size. Parameterefficient methods freeze most of the source model and update only small components [Chen et al., 2024b; Maharana et al., 2025; Liu et al., 2024b]. VPTTA [Chen et al., 2024b] uses lightweight prompts for BN alignment. PALM [Maharana et al., 2025] updates layers selectively by gradient magnitude. Despite these advances, existing methods are fundamentally constrained by endogenous adaptation signals obtained from the model itself on unlabeled target streams. This motivates leveraging heterogeneous external knowledge beyond conventional source-driven signals.

2.2 FoundationModels inMedical Segmentation

Vision Foundation Models (VFMs), such as the Segment Anything Model (SAM) [Kirillov et al., 2023], have recently been explored for medical image segmentation. Most approaches adapt VFMs by full fine-tuning or using lightweight adapters to improve domain-specific performance, but these methods are often computationally intensive and difficult to scale [Ma et al., 2024]. Some works integrate VFMs with vision-language models like CLIP to enhance data efficiency. For example, MedCLIP-SAM [Koleilat et al., 2024] and MedCLIP-SAMv2 [Koleilat et al., 2025] combine BiomedCLIP and SAM for promptable zero-shot segmentation, and TTCS [Chen et al., 2024a] uses BiomedCLIP prompts and self-training to improve SAM's robustness to domain shifts at test time. However, these methods typically use the foundation model as the main segmentation backbone and overlook task-specific priors in conventional models.

Another direction uses foundation models for semisupervised segmentation [Ma et al., 2025b; Konwer et al., 2025]. While both ReDi-FM and these methods utilize SAM, their settings and methodologies are distinct. For example, in SynFoC [Ma et al., 2025b], SAM generates highquality pseudo-labels for the conventional model during early training, and the conventional model later corrects highconfidence predictions from SAM. In contrast, ReDi-FM uses task-specific priors from the conventional model to guide the frozen foundation model and distills adapted knowledge back into the conventional model. Importantly, these semisupervised settings rely on labeled data and offline training cycles, whereas CTTA requires fully online adaptation on unlabeled streams without revisiting the source data.

3 Method

3.1 ProblemDefinition andMethod Overview

Continual Test-Time Adaptation (CTTA) for medical image segmentation adapts a pre-trained source model fθS , trained on labeled source data (XS, YS), to a sequence of unlabeled target domains DT = {XT1 , XT2 , . . . , XTn } encountered during deployment. Adaptation is performed online; each target domain is processed once, and source data are not accessible.

Figure 2 gives an overview of the ReDi-FM framework. The target model fθT , initialized from the source model, is updated for each target image with guidance from a frozen vision foundation model fθFM (e.g., SAM). Our approach consists of two main steps: generating target-adapted segmentation supervision using structure-aware prompts, and distilling knowledge from SAM into fθT via uncertainty-driven hard distillation and hard class-balanced soft distillation.

3.2 Target-Adapted Supervision Generation

To overcome the knowledge limitations of conventional models and achieve task-specific, target-adaptive supervision, we leverage semantic priors from the source model to guide the frozen foundation model. Given a target image x ∈ DT with spatial dimensions H ×W, the source model fθS produces a probability map ps = fθS (x), from which a binary pseudolabel ˆ ys is generated by thresholding at τ:

where I[·] denotes the indicator function.

Directly using ˆ ys as a mask/box prompt may inherit its shape errors under domain shift. We therefore use sparse point prompts to provide only coarse localization. To represent the main structure of each class, we extract the center point P c center , defined as the centroid of the largest connected foreground component in ˆ yc s . While the center point provides a strong structural prior, it may miss complex or atypical regions, especially when noisy pseudo-labels are present. To address this limitation, we introduce an additional structureaware prompt: the hard point P c hard . For each class c, we first compute the class prototype zc as the mean intensity of all foreground pixels:

Next, for each foreground pixel, we calculate its intensity deviation from the prototype:

The hard point is then defined as the foreground pixel with the maximum deviation:

The set of structure-aware prompts, P = {P c center , P c hard }Cc=1 , is provided to the frozen foundation model fθFM along with the target image x, i.e., fθFM (x,P), which generates candidate segmentation masks. Among these candidates, we select the mask ˆ yfm that achieves the highest Dice similarity with the source pseudo-label:

where M is the set of candidate masks and Dice(·, ·) denotes the Dice coefficient. We use ˆ ys only as a coarse anchor to select the intended mask among candidates, rather than enforcing its boundary shape. The selected mask ˆ yfm and its associated probabilistic output pfm serve as target-adapted supervision for subsequent distillation.

3.3 Complementary Knowledge Distillation

The frozen foundation model (e.g., SAM) provides valuable external knowledge for the target domain, but its predictions alone are often insufficient for accurate, domain-specific medical segmentation. To effectively transfer the generated supervision to the trainable target model fθT , we propose two complementary distillation objectives: an uncertainty-driven hard distillation and a hard class-balanced soft distillation.

Uncertainty-Driven Hard Distillation. Directly using hard pseudo-labels from the source model or SAM can result in errors due to domain shift and prediction noise. To enhance robustness, we introduce an uncertainty-weighted hard distillation objective, where pixel-wise confidence is used to adaptively weight the supervision signal. Following [Yang et al., 2024], the confidence weighting function Φ(p) for a predicted probability p ∈ [0, 1] is defined as:

where γ = 1.3, µ = 0.5, and σ2 = 0.1. This assigns higher weights to confident predictions (probabilities near 0 or 1) and lower weights to uncertain pixels.

Pixel-wise confidence weights are computed for the outputs of both the source model (ps) and SAM (pfm), resulting in Φ(pis ) and Φ(pifm). These weights are incorporated into a segmentation loss combining weighted binary cross-entropy and Dice terms. Let pt = fθT (ˆx) denote the target model's probability map with ˆx an augmented version of x, ˆ ys the source pseudo-label, and ˆ yfm the binary mask from SAM. The hard distillation losses are defined as:

where Lwseg (p, ˆ y; w) applies a pixel-wise weight map w (here w = Φ(·)) to both the BCE and Dice terms. For convenience, we also define the ungated sum of the two branches as

This adaptive weighting allows all pixels to contribute, while emphasizing those with higher confidence, thus mitigating the negative impact of noisy or ambiguous supervision.

Hard Class-Balanced Soft Distillation. Soft distillation leverages the full probabilistic outputs of the teacher model (SAM), offering richer supervisory signals and greater stability under noisy pseudo-labels. However, medical image segmentation is often hampered by severe class imbalance, as foreground structures typically occupy only a small portion of the image. To address this challenge, we propose a hard class-balanced soft distillation objective that dynamically emphasizes both difficult and under-represented classes. For each class c, we compute an adaptive class weight ηc that jointly accounts for the spatial prevalence of class c in SAM's prediction and the degree of disagreement between the target and SAM outputs:

where H ×W denotes the image resolution, and ˆ yt = I[pt > τ ] is the binarized prediction from the target model.

To further concentrate learning on hard regions, we define a pixel-wise difficulty mask ¯ yc,i for each class c and pixel i. The mask ¯ yc,i is set to 1 if either (i) the source model predicts class c at pixel i, or (ii) the source model's prediction at i exhibits high uncertainty (i.e., high entropy); otherwise, ¯ yc,i = 0. This mechanism leverages both structure-aware priors and the current model's uncertainty, ensuring that challenging or ambiguous pixels are adaptively up-weighted during training. The per-pixel adaptive weight is then given by:

where ηc is the adaptive class weight. The final soft distillation loss is given by:

where pc,i t and pc,i fm denote the predicted probabilities for class c at pixel i from the target model and SAM, respectively. By emphasizing pixels identified by the source model or those with high uncertainty, this loss improves adaptation to domain shifts and enhances robustness in segmenting small or under-represented structures, which are prevalent in medical images.

Consensus-Aware Gating. The source model provides task-specific semantics but is sensitive to domain shift, while the frozen VFM offers a generic cross-domain structural prior. When they disagree, enforcing both signals can amplify errors in unlabeled CTTA. We therefore introduce a consensus-aware gate. We measure their agreement by

We use r to gate the shift-sensitive source supervision while keeping the VFM branch active:

For soft distillation, we anneal the pixel-wise re-weighting toward uniform weights under disagreement:

When r is small, the source and VFM disagree and the source-guided hard weighting in αc,i may over-emphasize unreliable regions. Annealing αc,i toward 1 tempers this hard weighting and moves Lsoft toward uniform weighting. We replace αc,i with ˜αc,i in Lsoft.

Total Objective. The overall objective is defined as:

4 Experiments

4.1 Datasets and EvaluationMetrics

OD/OC Segmentation Dataset. We benchmark on five public fundus image datasets for Optic Disc (OD) and Optic Cup (OC) segmentation: RIM-ONE-r3 [Fumero et al., 2011] (Domain A, 159 images), REFUGE [Orlando et al., 2020] (Domain B, 400 images), ORIGA [Zhang et al., 2010] (Domain C, 650 images), REFUGE-Validation/Test [Orlando et al., 2020] (Domain D, 800 images), and DrishtiGS [Sivaswamy et al., 2014] (Domain E, 101 images). All images are resized to 512 × 512 pixels. Performance is evaluated using the Dice similarity coefficient (DSC).

Prostate Segmentation Dataset. For cross-domain prostate segmentation, we utilize T2-weighted MRI slices from six medical centers: BIDMC (Domain A, 261 slices), BMC (Domain B, 384 slices), HK (Domain C, 158 slices), I2CVB (Domain D, 468 slices), RUNMC (Domain E, 421 slices), and UCL (Domain F, 175 slices) [Bloch et al., 2015; Lemaˆıtre et al., 2015; Litjens et al., 2014; Liu et al., 2020]. Each slice is resampled to 384 × 384 pixels. Results are reported in terms of DSC.

Polyp Segmentation Dataset. Polyp segmentation is evaluated on four datasets: BKAI-IGH-NeoPolyp [Ngoc Lan et al., 2021] (Domain A, 1000 images), CVCClinicDB [Bernal et al., 2015] (Domain B, 612 images), ETIS-LaribPolypDB [Silva et al., 2014] (Domain C, 196 images), and Kvasir-Seg [Jha et al., 2019] (Domain D, 1000 images). All images are resized to 352 × 352 pixels. We report DSC for region overlap, structure similarity (Sα) for structural consistency, and enhanced alignment measure (Emax ϕ ) for boundary alignment.

4.2 Implementation Details

All models are implemented in PyTorch 2.2.2 on a single NVIDIA RTX 3090 GPU. To ensure statistical reliability, we conduct experiments with three random seeds and report the mean. For OD/OC and prostate segmentation, we use ResUNet-34 [Chen et al., 2024b] as the source model. For the more challenging polyp segmentation task, we adopt Segformer-B0 [Xie et al., 2021]. During continual test-time adaptation, each test image is adapted with a single online iteration (batch size = 1). All methods, including ReDi-FM and all baselines, are evaluated under identical experimental settings to ensure a fair comparison. For deployment of ReDi-FM, we use the Adam optimizer for all tasks, with a learning rate of 0.0001 for OD/OC and prostate segmentation, and 0.00001 for polyp segmentation. The pseudo-label threshold τ is set to 0.5. The ViT-H variant of the Segment Anything Model (SAM) is used as the frozen foundation model in all experiments. Data augmentation includes color jittering and random erasing. For MedCLIP-SAM/MedCLIPSAMv2/TTCS, we follow the official inference protocol and prompts provided by the authors.

4.3 Experimental Results

We benchmark ReDi-FM against a comprehensive set of baselines, including: (1) the 'Source Only' model (trained on the source domain and directly evaluated on the target domain without adaptation), (2) three zero-shot foundation models, MedCLIP-SAM [Koleilat et al., 2024], MedCLIP-SAMv2 [Koleilat et al., 2025], and TTCS [Chen et al., 2024a], and (3) several state-of-the-art CTTA methods: TENT [Wang et al., 2020] and SAR [Niu et al., 2023] (entropy-based), CoTTA [Wang et al., 2022] (pseudolabeling), parameter-efficient methods VPTTA [Chen et al., 2024b] and PALM [Maharana et al., 2025], and SURGEON [Ma et al., 2025a] (using dynamic activation sparsity). For fair comparison, we re-implement all CTTA baselines with the same backbone and follow exactly the same test-time protocol as ReDi-FM. Following the standard CTTA evaluation protocol, each domain is used in turn as the source, and the model is evaluated on all remaining target domains. Our results are averaged over three random seeds. For each source domain, the remaining target domains are streamed in a fixed order (A→B→ · · · ) for all methods.

OD/OC Segmentation. Table 1 presents segmentation results across five domains. Most CTTA methods outperform the 'Source Only' baseline, confirming the benefit of online adaptation under domain shifts. ReDi-FM achieves the best overall performance, surpassing both zero-shot foundation models and prior CTTA methods. Zero-shot baselines such as MedCLIP-SAM and TTCS leverage foundation priors but do not adapt to the target stream, which limits robustness across domains and can underperform 'Source Only'. The ReDi-FM framework generates target-adaptive supervision and transfers heterogeneous external knowledge via reciprocal distillation. This enables robust segmentation across all domains: ReDi-FM remains competitive on relatively easy domains and achieves the best performance on more challenging ones (e.g., Domain D).

Prostate Segmentation. Table 1 reports cross-domain prostate segmentation results, a task characterized by high inter-domain variability and frequent model overconfidence. Due to excessive noise, entropy-based methods (TENT) and pseudo-labeling methods (CoTTA) can be unstable in this setting and may even underperform the 'Source Only' baseline. Methods that control error accumulation (e.g., SAR) and parameter-efficient adaptation (e.g., VPTTA and PALM) are generally more robust, yet their adaptation signal is still constrained by relying only on the source model and unlabeled target data, which can be insufficient under large domain shifts. Zero-shot foundation models such as MedCLIPSAM and TTCS also perform poorly, highlighting the limits of generic knowledge without online adaptation. In contrast, ReDi-FM distills external knowledge into the target model, surpassing the knowledge constraints of both the source and the unlabeled target stream. As a result, ReDi-FM improves the best prior CTTA method by 13.14 DSC points on Domain F and achieves an overall gain of 16.04 DSC points over 'Source Only'.

Polyp Segmentation. We evaluate ReDi-FM on polyp segmentation using Segformer-B0 as the source model, while VPTTA is excluded because it relies on batch normalization. Table 2 presents results across four domains. In this task, several CTTA methods perform worse than the 'Source Only' baseline, reflecting the challenges of segmenting small, hidden, and highly variable polyps, which increases the risk of negative transfer during adaptation. Zero-shot foundation models also perform poorly, underscoring the need for taskspecific adaptation. Despite these difficulties, ReDi-FM consistently achieves the best overall performance across metrics. These results indicate that reciprocal distillation can inject complementary foundation knowledge into the target model and reduce negative transfer on challenging polyp datasets, which is not achieved by prior CTTA methods.

Comparison in a Long-Term CTTA. We evaluate the long-term robustness of ReDi-FM through multi-round adaptation on the OD/OC and prostate segmentation benchmarks (Table 3). In each experiment, the model is sequentially adapted to target domains over three rounds, simulating repeated real-world domain shifts. Results show that most baseline CTTA methods suffer from performance drift as adaptation proceeds. Entropy minimization (TENT) and pseudolabeling (CoTTA) exhibit clear degradation across rounds, while update-filtering methods (e.g., SAR) and activationsparsity (SURGEON) improve stability but with limited gains. In contrast, ReDi-FM achieves the highest average performance and the largest overall gains, while remaining stable across rounds. Notably, on prostate segmentation ReDi-FM further improves with successive adaptation cycles (negative degradation), indicating sustainable continual adaptation under repeated domain shifts.

4.4 Ablation Study

Effectiveness of Core Components. We conduct an ablation study on OD/OC segmentation (Table 4) to assess the contribution of each component. We evaluate hard distillation from the source model Lsrc hard , hard distillation from the SAM branch Lfm hard , pixel-wise uncertainty weighting, hard class-balanced soft distillation Lsoft, and the consensusaware gate r. Hard distillation from the source model provides a moderate gain, and adding the SAM branch introduces external knowledge for further improvement. Pixelwise uncertainty weighting enhances robustness against noisy pseudo-labels. Incorporating class-balanced soft distillation enables the model to absorb richer knowledge from the foundation model, achieving further improvement. Finally, the consensus-aware gate r stabilizes distillation under teacher disagreement by down-weighting unreliable source supervision, leading to the best overall performance. Overall, each component incrementally improves performance.

Effectiveness of Prompting Strategies. We investigate different prompting strategies for guiding the foundationmodel branch, as shown in Figure 3. Mask and Box prompts are less stable under domain shift, as they directly inherit errors from source pseudo-labels and may amplify them during adaptation. Using a Center point improves localization, while adding a Box on top brings only marginal gains. Our structure-aware prompt, which combines an informative hard point with the foreground center, achieves the best overall performance.

Effectiveness of Class-Balancing Strategies. We evaluate different class-balancing strategies for adaptation in Table 6. Removing class balancing leads to suboptimal performance, while conventional class-balanced weighting can be brittle under noisy pseudo-labels. In contrast, our hard class balancing provides a more stable weighting scheme and yields the best result.

FM-only vs. full framework. To isolate the role of reciprocal distillation, we report ReDi-FM FM that uses only the FM branch outputs (Table 7). ReDi-FM FM is not consistently strong across benchmarks and can be inferior to the source model, indicating that frozen FM outputs alone are insufficient under domain shifts. In contrast, ReDi-FM yields consistent improvements, suggesting that the gains mainly come from transferring complementary knowledge into the target model via reciprocal distillation, rather than relying on FM outputs alone.

Efficiency-Accuracy Trade-off. Table 5 reports the accuracy-latency trade-off on polyp segmentation under the same CTTA setting. Lightweight CTTA baselines (e.g., TENT/SAR) are fast but less accurate, while VFM-based zero-shot methods are much slower and still underperform. ReDi-FM attains the best accuracy with moderate latency. With a lighter FM backbone (ViT-B), latency drops with negligible accuracy change, indicating the gain mainly comes from our distillation framework.

5 Conclusion

We propose ReDi-FM, a reciprocal distillation framework designed to address the challenges of CTTA in medical image segmentation. To the best of our knowledge, this is the first work to tackle CTTA by leveraging an off-the-shelf frozen vision foundation model as a heterogeneous external teacher. This alleviates the limitations of previous approaches that relied primarily on source-driven self-supervised adaptation. By introducing external knowledge from the foundation model, ReDi-FM pushes beyond the knowledge upper bound of source-model-driven adaptation on unlabeled target streams. Our framework integrates structure-aware prompting, which enables the foundation model to generate taskadapted supervision under the guidance of the source model, with complementary distillation strategies that reliably transfer both generic and target-adapted knowledge to the target model. Concretely, ReDi-FM distills complementary signals into the target model via uncertainty-driven hard distillation and hard class-balanced soft distillation, and uses a consensus-aware gate to coordinate the two teachers under disagreement. Extensive experiments on three medical segmentation benchmarks demonstrate that ReDi-FM consistently outperforms strong CTTA baselines and zero-shot foundation-model approaches in both adaptation accuracy and long-term stability.

Acknowledgements

This research was supported by NSFC-FDCT under its Joint Scientific Research Project Fund (Grant No. 0004/2025/AFJ), National Natural Science Foundation of China (Grant No. 62571173), and Zhejiang Provincial Natural Science Foundation of China (Grant Nos. LD25F020005 and LQN25F030009).