Bridging the Objective Gap: A Unified Pre-Training Framework for Few-Shot Medical Image Segmentation

1 Introduction

Deep learning has substantially advanced medical image segmentation, yet clinical adoption is hindered by high annotation costs [Litjens et al., 2017; Ronneberger et al., 2015]. Few-shot segmentation (FSS) addresses this via prototypebased matching from limited support images [Wang et al., 2020; Snell et al., 2017]. While recent pipelines leverage large-scale pre-training [Azizi et al., 2021], standard initializers often fall short for boundary-sensitive dense matching in episodic FSS. We attribute this to an objective gap in task form: prevalent pre-training objectives emphasize global alignment or reconstruction, which misaligns with the local, pixel-precise separability required by mask-conditioned prototype matching. As a result, similarity maps can be unreliable (especially near boundaries), and few-shot fine-tuning must substantially reshape representations, leading to unstable and costly adaptation.

* Corresponding authors.

† The official implementation is available at https://github.com/ spchen01/BOG-Pretrain.

A good initializer should (i) be intrinsically matchable for dense prototype matching before any adaptation and (ii) remain stable under few-shot fine-tuning. We therefore introduce a Drift-Gap diagnostic (Fig. 1) that measures intrinsic dense-matching misalignment (GapPT) and adaptationinduced representation change (Drift). LowDrift indicates a well-organized episodic metric geometry, enabling effective adaptation with minimal representation re-organization; the bottom-left "sweet spot" simultaneously achieves lowGapPT and low Drift. This diagnostic also reveals characteristic shifts when individual objectives are removed, suggesting that the objective gap arises from multiple mismatches that must be addressed jointly.

Concretely, we trace the objective gap to three conflicts between generic pre-training and episodic FSS. Semantic mismatch: Radiology reports are frequently noisy or only partially relevant [Wang et al., 2022]; uniform image-text alignment can corrupt prototype semantics and weaken internal coherence. Structural mismatch: Standard masked image modeling optimizes reconstruction fidelity without prioritizing boundary-discriminative regions [He et al., 2022], limiting boundary-sensitive delineation. Metric instability: Without explicit consistency constraints, same-class features can vary substantially across views, forcing large feature drift during adaptation to align support and query prototypes. We formalize these effects in §3.1 and use them to guide objective design.

To bridge these gaps, we propose BOG-PRETRAIN, a pre-training framework for episodic FSS with three complementary components. Reliability-Gated Alignment (regularized by masked language modeling [Devlin et al., 2019]) down-weights unreliable report supervision to improve prototype coherence (reducing the interior mismatch). SemanticGuided MIM biases masking and reconstruction toward boundary-informative regions to enhance boundary sensitivity (reducing the boundary mismatch). Finally, Dual Consistency Regularization combines prototype-level consistency (ProtoNCE) and same-image consistency (SameImg) to stabilize episodic metric geometry and reduce Drift, enabling effective adaptation with minimal representation shift. Across five benchmarks, BOG-PRETRAIN improves mean Dice by up to +15.5/+15.9/+12.5 points and reduces mean HD95 by 0.9/4.6/10.1 under 1/4/16-shot settings. Together, they move the initializer toward the "sweet spot" in Fig. 1.

Our contributions are summarized as follows:

• Global objective bridging for episodic FSS. We propose Reliability-Gated Alignment to robustly leverage noisy image-report data, improving prototype coherence and intrinsic matchability.
• Local objective bridging for boundary-sensitive prediction. We propose Semantic-Guided MIM to prioritize boundary-informative regions during masked modeling, improving boundary adherence.
• A unified pre-training framework under an episodic diagnostic. We introduce the Drift-Gap diagnostic and Dual Consistency Regularization to stabilize episodic metric geometry and substantially reduce adaptation drift; the resulting BOG-PRETRAIN consistently outperforms existing methods across five benchmarks in both region and boundary metrics.

2 RelatedWork

2.1 Few-Shot and Universal Medical Image Segmentation

Few-shot segmentation (FSS) transfers segmentation to novel classes from a few annotated support images, and many pipelines adopt prototype-based metric matching to predict query masks in a shared embedding space [Snell et al., 2017]. Recent medical FSS and universal segmentation methods improve robustness via prototype refinement, self-supervised episodic training, or support-conditioned inference [Tang et al., 2025; Ouyang et al., 2020; Butoi et al., 2023]. While these designs strengthen adaptation and generalization, they largely operate within a given representation space and thus remain bounded by the initializer: whether dense features are already separable for mask-conditioned matching and how much they must change under few-shot adaptation. This motivates studying pre-training in terms of intrinsic matchability and adaptation cost for episodic FSS (Fig. 1).

2.2 Generic Self-Supervised Pre-Training: Structural and Geometric Limitations for Episodic Dense Matching

Self-supervised learning (SSL) provides strong generalpurpose initializers. Contrastive objectives such as MoCo v3 encourage view-invariant instance discrimination, typically on global embeddings [Chen et al., 2021], while masked image modeling (e.g., MAE) learns via reconstructing masked patches [He et al., 2022]. However, these objectives are not designed around mask-conditioned dense separability: reconstruction can preserve appearance without sharpening boundary-relevant cues, and instance-level invariance does not directly calibrate the dense metric geometry used by prototype matching. As a result, SSL/MIM initializers can exhibit intrinsic mismatch for boundary-sensitive delineation and require substantial representation re-organization during episodic fine-tuning, often appearing as elevated GapPT and/or Drift regimes in Fig. 1. For episodic FSS, this highlights the need to explicitly stabilize the episodic metric geometry (e.g., via consistency constraints) so that adaptation incurs minimal drift.

2.3 Vision-Language Pre-Training: Semantic Benefits with Reliability Challenges

Vision-language pre-training (VLP) injects semantic supervision by aligning image and text representations, as exemplified by CLIP [Radford et al., 2021] and medical adaptations such as MedCLIP and BioViL [Wang et al., 2022; Boecking et al., 2022], with continual multimodal learning explored in MedCoSS [Ye et al., 2024]. For prototype-based FSS, semantics can be valuable because prototypes act as episode-specific anchors for matching. However, radiology reports are often global, noisy, or only partially relevant to localized findings, making uniform alignment unreliable and potentially weakening region-level semantic coherence. Accordingly, VLP can improve intrinsic matchability in some cases yet remain sensitive to report quality and adaptation, leading to varied trade-offs between GapPT and Drift across datasets in Fig. 1.

Together, these limitations motivate our approach. We revisit pre-training for episodic, boundary-sensitive FSS from a coupled perspective of (i) semantic coherence for reliable prototypes, (ii) boundary-aware dense structure, and (iii) a stable episodic metric geometry with low adaptation drift. Building on segmentation-centric pipelines that accept given representations [Butoi et al., 2023; Tang et al., 2025; Ouyang et al., 2020] and generic SSL/VLP baselines [Chen et al., 2021; He et al., 2022; Radford et al., 2021; Ye et al., 2024], we develop an objective-gap-bridging pre-training framework that jointly targets these aspects under the DriftGap diagnostic, driving the initializer toward the bottom-left "sweet spot" with low GapPT and low Drift .

3 Methodology

Episodic FSS predicts a query mask via mask-conditioned prototype matching in a dense feature space, so the initializer should already support boundary-sensitive dense matching and remain stable under few-shot adaptation. We therefore aim to pre-train representations natively compatible with episodic, mask-conditioned prototype matching. Figure 2 illustrates BOG-PRETRAIN, grouping objectives by targeted mismatches to clarify global, local, and geometric interactions. We next formalize the episodic quantities defining a good initializer, then present the three components following the structure in Fig. 2.

Setup and notation. Given an image-report pair (Ii,Ri), the vision encoder EV produces dense features Fi ∈ RC×Hf×Wf and a pooled embedding vi ∈ Rd , while the text encoder ET produces a global embedding ti ∈ Rd and token embeddings Ti = {Ti,l}Ll=1 . We use cosine similarity sim(·, ·) and ℓ2 normalization; unless specified, weighting signals are treated as constants (stop-gradient).

3.1 Bridging theObjectiveGap: Episodic Diagnostics and Learning Objectives

Episodic FSS predicts a query mask via mask-conditioned prototype matching in a dense feature space. From this viewpoint, a good initializer should (i) enable accurate dense matching before any adaptation and (ii) require only lightweight representation change under few-shot finetuning. We instantiate these desiderata as intrinsic preadaptation misalignment GapPT and adaptation-induced feature change Drift (Fig. 1), and use them to organize the learning objectives of BOG-PRETRAIN.

Mask-conditioned similarity map. Given a query image with dense features F and a binary foreground mask M (at image resolution), we sample Q points {pj}Qj=1 from M and compute an averaged similarity heatmap:

where ˆ F(·) denotes ℓ2-normalized features and sim(·, ·) is cosine similarity.

Objective Gap (GapPT) and its decomposition. To avoid threshold tuning (which can be dataset-, class-, and episodedependent and introduces an extra hyperparameter), we binarize the heatmap by a size-matched top-k rule, where k = |M|. This threshold-free binarization controls the predicted foreground size and focuses the evaluation on the ranking quality of the similarity map. Let ˆM = TopK(S, |M|) keep the top-|M| pixels of S. We define the pre-training alignment gap as

To localize the mismatch, we split M into an interior region and a boundary band using morphology at image resolution: Min = Erode(M, r) and Mbd = Dilate(M, r) -Min. We then measure region-specific gaps as

Here Gapin reflects semantic coherence inside the target region (prototype reliability), while Gapbd reflects boundarysensitive dense separability for precise delineation.

Feature Drift (adaptation cost). Let F pt and F ft be the dense features of the same query image extracted by the pretrained backbone and the few-shot adapted backbone, respectively. We quantify the adaptation-induced change as the mean cosine distance over spatial locations:

where N = HfWf . Low Drift indicates a stable episodic metric geometry, enabling effective adaptation without large representation re-organization.

Learning objectives guided by the episodic lens. The above quantities clarify what an initializer must provide for episodic dense matching: reliable semantics (low Gapin), boundary-sensitive structure (low Gapbd), and a stable metric geometry (low Drift). Following the roadmap in Fig. 2, we next introduce three components that target these properties from complementary angles: reliability-aware global alignment (§3.2), boundary-aware local modeling (§3.3), and dual consistency regularization (§3.4).

3.2 Reliability-GatedGlobal Alignment for Reducing Interior Gap

Reliability-gated global alignment. As analyzed in §3.1, a large interior gap (Gapin) indicates insufficient semantic coherence inside the target region, which weakens prototype reliability even before episodic adaptation. Paired reports can inject global semantics to tighten this coherence, but clinical narratives are often noisy, incomplete, or only partially relevant; enforcing uniform image-report alignment can therefore propagate misleading gradients and distort the semantic anchors that prototypes depend on. We address this by making global alignment selective: each image-report pair contributes proportionally to its estimated reliability.

Concretely, for each pair (Ii,Ri) we compute a reliability score γi ∈ [0, 1] by fusing complementary signals of alignment confidence, linguistic coherence (via an auxiliary MLM signal), report informativeness, and batch-wise consistency:

where Z(·) denotes standardization, and pi, mi, ri, ci are the image-text cosine similarity, MLM token-prediction confidence, TF-IDF report informativeness score, and batchwise prototype consistency, respectively. We stop gradients through γi so that the gate only modulates optimization.

We then re-weight a standard symmetric image-text contrastive objective by γi:

where LdITC (i) is the standard InfoNCE term for direction d and sample i, and sg(·) denotes stop-gradient. By attenuating unreliable pairs, the global semantic geometry becomes less sensitive to report noise, yielding more reliable prototype anchors and directly reducing Gapin.

3.3 Local Structural Modeling via SemanticGuided Masked Reconstruction

Semantic-guided masked reconstruction. A high boundary gap (Gapbd) in §3.1 indicates that dense features are not sufficiently boundary-discriminative for episodic delineation. Masked reconstruction can strengthen local structure, but uniform masking tends to allocate learning to generic textures rather than boundary-critical regions. We instead make reconstruction spatially selective, concentrating supervision on edge- and semantics-informative locations to sharpen boundary sensitivity.

Text-conditioned semantic relevance map. At feature resolution, we compute a semantic relevance map Si ∈ [0, 1]Hf×Wf by correlating dense visual features with informative report tokens. Let ϕv(·) and ϕt(·) be lightweight projections, and Ki denote tokens whose TF-IDF weight exceeds the per-report median. We define

followed by per-image normalization to [0, 1]. Text features are detached when forming Si, so the map serves as a masking prior rather than an additional supervision target.

Edge-semantic fusion with reliability-gated guidance. We compute an edge prior Ei ∈ [0, 1]Hf×Wf from image gradients and fuse it with semantics as Wi = (1 - β)Ei + βSi. Crucially, we gate this guidance by the global reliability γi from §3.2: when reports are unreliable, text-guided masking should not shape local structure; when reports are reliable, reconstruction should focus on boundary- and semantic-rich regions. Given a random/block mask M blk i and a curriculum ηt ∈ [0, 1] increasing over training, we form a soft mask

Masked reconstruction objective. Let ˆ yi(u) and yi(u) be the prediction and target at location u. We apply a standard reconstruction loss on masked regions, aggregated by the soft mask:

When γi is high, masking concentrates on Wi and forces recovery of edge/semantic details; when γi is low, it reverts to M blk i . This reliability-gated, boundary-aware reconstruction directly targets Gapbd by sharpening boundary-sensitive dense separability.

3.4 Dual Consistency Regularization for Stabilizing Metric Geometry

Reducing Gapin and Gapbd improves intrinsic matchability; to further suppress adaptation-induced drift (Drift), we introduce Dual Consistency Regularization that enforces stable metric geometry via complementary prototype- and instancelevel constraints.

Prototype-level consistency (ProtoNCE). We maintain a prototype bank {ck}Kk=1 in the text embedding space, capturing semantic modes inferred from reports. Each sample is assigned to its nearest prototype in text space, k(i) = argmaxk sim(ti, ck); prototypes are updated via EMA with momentum m=0.97 outside the computational graph, so a single noisy text embedding contributes only ∼3% per step and is further diluted by batch averaging. We then pull the visual embedding vi toward its assigned prototype while pushing it away from others:

This encourages compact semantic clusters and reduces the degrees of freedom of the embedding space, yielding a more stable episodic metric.

Instance-level consistency (SameImg). When optimizing multiple objectives (global alignment, local reconstruction, prototype consistency), the same image may receive conflicting gradients. We directly enforce that embeddings of the same image produced under different views/pathways remain consistent. Let v(a) i and v(b) i be two pooled embeddings of the same image (e.g., from different augmentations or objective pathways), we apply a cosine pull with stop-gradient:

where ˆ v denotes ℓ2-normalized embeddings. This term suppresses cross-objective interference by forcing a unified representation for the same content.

Overall effect on Drift. ProtoNCE compacts the semantic geometry, while SameImg aligns the optimization signals across objectives/views. Together, they yield a tight and consistent metric space, which substantially reduces Drift during few-shot adaptation.

3.5 Overall Objective and Conceptual Summary

Overall objective. We jointly optimize all components of BOG-PRETRAIN with:

where

We optionally add λmlmLMLM to regularize the text encoder and to provide the linguistic-coherence cue in Eq. (5).

Conceptual summary under the Drift-Gap lens. BOGPRETRAIN is organized around the episodic diagnostics in §3.1. The reliability-gated global alignment LG-ITC makes semantic supervision selective under noisy reports, improving interior coherence and lowering Gapin. The semanticand edge-guided masked reconstruction LSG-MIM concentrates local supervision on boundary-critical regions, sharpening dense separability and lowering Gapbd. Finally, dual consistency regularization (LProto and LSI) stabilizes the metric geometry by enforcing compact semantics and suppressing cross-objective/view interference, which directly reduces Feature Drift during few-shot adaptation.

Together, these objectives align the pre-trained representation with the demands of episodic, mask-conditioned matching, driving the initializer toward the low-gap, low-drift "sweet spot" in Fig. 1.

4 Experiments

4.1 Experimental Setup

Pre-training data. We pre-train on MedPix 2.0 [Siragusa et al., 2025], a comprehensive multimodal archive containing ∼59K images paired with structured clinical case reports. The rich linguistic context in this dataset provides diverse semantic supervision, which is critical for optimizing our Reliability-Gated Alignment module.

Downstream evaluation. We evaluate on five benchmarks [Cheng et al., 2015; Al-Dhabyani et al., 2020; Byra et al., 2020; Jha et al., 2020; Chowdhury et al., 2020; Rahman et al., 2021; Gong et al., 2021]. Following MedCLIPSAMv2 [Koleilat et al., 2025], we adopt their splits for Brain, Lung, and Breast tumors. Notably, Breast Tumor adopts a cross-dataset setting (train on BUSI [Al-Dhabyani et al., 2020], test on UDIAT [Byra et al., 2020]). For Thyroid [Gong et al., 2021], we use the official split, and for Kvasir-SEG [Jha et al., 2020], a random 70/30 partition.

Architecture and protocol. We use an ImageNet-initialized Swin-S [Liu et al., 2021] with an ASPP head [Chen et al., 2017]. Results are reported as mean Dice and HD95 over 1,000 episodes under 1/4/16-shot settings. We pre-train for 200 epochs using AdamW (lr = 5e-4 , weight decay 0.05, batch size 128); total cost is ∼3.7 GPU·h on 1×RTX4090 with 62.96M parameters (≈1.7× the wall-clock of a CLIP baseline on the same backbone). The text encoder is discarded after pre-training, so inference cost matches an imageonly Swin-S.

Diagnostic metrics. To probe pre-training quality beyond end-task scores, we report (i) Objective Gap (Gapin/Gapbd) that decomposes intrinsic misalignment into semantic coherence (Interior) and boundary delineation (Boundary) without adaptation, and (ii) Feature Drift measured by cosine distance between pre-trained and adapted prototype features after few-shot support.

4.2 Main Results: Comparison with SOTA

Table 1 reports few-shot segmentation results (1/4/16-shot). In line with our Objective Gap formulation, we interpret these gains through the intrinsic dense-matching misalignment and adaptation cost. As summarized in Table 2 and visualized in Fig. 1, BOG-PRETRAIN moves the initializer toward the desired bottom-left regime: it achieves the lowest GapPT before adaptation and the lowest Drift during adaptation among all compared methods, confirming that the initializer is both intrinsically matchable and geometrically stable without requiring costly representation re-organization.

The downstream improvements align with the decomposed gap analysis established in §3.1 and validated in §4.3. First, higher Dice indicates more coherent prototype anchors, consistent with a reduced interior semantic gap (Gapin): by down-weighting unreliable report semantics, the pre-trained embedding becomes less sensitive to noisy/global text, leading to stronger overlap across modalities and annotation settings (e.g., large Dice gains on Kvasir and the crossdataset Breast Tumors benchmark). Second, we emphasize HD95 because it reflects boundary adherence that Dice can miss; our boundary-aware pre-training reduces the boundary gap (Gapbd), which is most evident on irregular structures (e.g., Kvasir shows a pronounced HD95 drop), matching the boundary-gap trends in §4.3.

We note that on Brain Tumors (1-shot) the HD95 improvement is less consistent. With a single support mask and highly irregular tumor contours, boundary refinement remains strongly support-dependent and HD95 is sensitive to small outlier regions, a known challenge even for supervised segmenters, while BOG-PRETRAIN still yields the best Dice, indicating more reliable localization under extreme label scarcity. Finally, the gap to nnU-Net reflects the inherent cost-performance trade-off between fully-supervised dense annotation and few-shot generalization: our goal is to improve the efficiency frontier under scarce supervision, and BOG-PRETRAIN consistently advances both overlap and boundary quality among few-shot methods, making it a strong candidate for annotation-scarce clinical scenarios.

4.3 MechanismAnalysis: Decoupling the Gains

To move beyond performance tables and interpret why our method works, we leverage the proposed diagnostic metrics to decouple the contributions of Global, Local, and Consistency modules.

1. Spatially Decoupled Improvements (Global vs. Local). As visualized in Figure 3, the ablation reveals a clear functional separation: Reliability-Gated Alignment delivers a large reduction in Gapin, confirming that global contrastive learning aggregates object-interior pixels into coherent semantic clusters; once Gapin saturates, adding SG-MIM triggers a secondary drop specifically in Gapbd, proving the local objective captures high-frequency boundary details overlooked by global objectives. Figure 4 further validates this via strong correlations (r>0.9) between Gapin and Dice and between Gapbd and HD95, suggesting that the two diagnostic axes are complementary and neither alone suffices to predict overall segmentation quality.

2. Efficiency and Stability (The Role of Consistency). Projecting models onto the Drift-Gap plane (Fig. 5) reveals an orthogonal two-phase trajectory: Phase 1 (Global + Local) drives a vertical descent that closes the Gap, while Phase 2 (Consistency) drives a horizontal shift that reduces Drift by over 80% (from ∼0.48 to 0.08) without further changing Gap, indicating that the two improvement axes are largely independent. This shows that consistency constraints do not merely fine-tune accuracy but fundamentally restructure the feature space to resist few-shot adaptation shifts, enabling effective generalization even under extreme label scarcity.

4.4 Ablation Studies

Table 3 links end-task performance to our decomposed Objective Gap. Compared with generic SSL/MIM/VLP objectives, BOG-PRETRAIN achieves the best Dice/HD95 while lowering both Gapin and Gapbd, indicating improved intrinsic matchability for mask-conditioned dense prototype matching. Notably, BOG-PRETRAIN also yields the lowest Drift, suggesting that its embedding geometry requires minimal re-organization during few-shot fine-tuning.

Table 4 attributes improvements to specific components under the Drift-Gap lens. Removing Global increases Gapin (0.28→0.31) and reduces Dice (75.24→72.06), consistent with reliability-gated alignment protecting prototype anchors from noisy report semantics. Removing Local markedly increases Gapbd (0.69→0.76) and causes the largest HD95 degradation (29.37→43.23), confirming that boundary-aware structural modeling is essential for closing the boundary gap and sharpening contours. Finally, removing Consistency produces the largest Drift increase (0.052→0.215) with comparatively smaller gap changes, showing that consistency regularization primarily stabilizes episodic metric geometry rather than merely improving intrinsic alignment.

5 Conclusion

We proposed BOG-PRETRAIN, a unified pre-training framework that bridges the objective gap between representation learning and episodic few-shot medical segmentation. BOGPRETRAIN combines (i) reliability-gated image-text alignment to mitigate noisy reports, (ii) semantic- and edge-guided masked reconstruction to enhance boundary-sensitive structure, and (iii) consistency regularization to stabilize episodic metric geometry. Experiments on five public benchmarks show consistent improvements over strong baselines in both Dice and HD95, especially in low-shot regimes where labels are scarce. Moreover, our Drift-Gap diagnostics and mechanism analyses show that Gapin, Gapbd, and feature drift decrease step-wise with each component and correlate strongly with downstream performance. We hope BOG-PRETRAIN encourages objective-driven pre-training designs that narrow the pre-training-to-few-shot segmentation mismatch for accurate and stable label-efficient medical segmentation.

Acknowledgments

This work was supported by the Shenzhen Science and Technology Program (RCBS20231211090723042).