When Vision-Language Models Meet Fetal Cardiac Ultrasound: Dual-Level Contrastive Learning for Out-of-Distribution Detection

1 Introduction

Recent advances in vision-language models (VLMs) [Radford et al., 2021] have demonstrated substantial improvements in image recognition performance. This success has driven increasing research interest in medical VLMs, ranging from general-purpose architectures [Zhang et al., 2023; Khattak et al., 2024] to domain-specialized expert models [Silva-Rodriguez et al., 2025; Christensen et al., 2024]. However, the violation of the closed-world assumption [Fei and Liu, 2016] limits the deployment of these models in healthcare, resulting in a reduced ability to reliably handle outof-distribution (OOD) samples. This challenge is particularly acute in the context of fetal cardiac ultrasound (FCU), where high operator dependency and inherent acoustic artifacts (e.g., speckle noise) [Noble and Boukerroui, 2006] lead to substantial data heterogeneity compared to standardized modalities such as MRI. As a consequence, models are prone to overconfident predictions on unseen inputs, leading to serious risks to patient safety [Hong et al., 2024].

∗Corresponding author.

Although OOD detection has achieved remarkable progress, existing methods [Liang et al., 2018; Hendrycks and Gimpel, 2017] primarily rely on post-hoc strategies and typically employ conventional vision encoder-only models, thereby leaving the rich semantic information in textual descriptions unexploited. To leverage the inherent strengths of VLMs and address the relevant challenges, several VLMbased methods [Miyai et al., 2023; Ming et al., 2022] have been proposed for OOD detection. However, previous studies primarily focus on standardized medical imaging modalities or natural images, leaving highly operator-dependent and heterogeneous settings underexplored, such as FCU.

In addition to these challenges, previous OOD detection benchmarks primarily focus on semantic shifts, characterized by disjoint label spaces between in-distribution (ID) and OOD samples. For example, a fetal heart view classification model may encounter novel classes, such as head view images. However, these studies overlook the fact that covariate shifts can lead to significant discrepancies even within the same class, affecting model performance. As shown in Figure 1, we focus on a realistic setting involving concurrent semantic shifts and covariate shifts [Yang et al., 2024].

In this work, we present a Dual-Level Contrastive Learning (DLCL) framework to develop a clinically reliable model capable of accurate ID classification and OOD detection. To the best of our knowledge, this work represents the first rigorous attempt to adapt VLMs for OOD detection in FCU. Rather than relying on full-parameter fine-tuning, we adopt a parameter-efficient deep prompting strategy [Khattak et al., 2023]. To tackle compound shifts, we introduce two complementary contrastive learning objectives. Local Contrastive Learning (LoCL) employs pairing based on the Bhattacharyya coefficient to align fine-grained feature representations at the batch level, effectively reducing acquisition noise and enhancing robustness to covariate shifts. Global Contrastive Learning (GlCL) leverages class prototypes coupled with an ambiguity-aware margin loss to regularize the feature space. This mechanism enforces intra-class compactness for known anatomies while ensuring the sufficient separability required to reject semantic-shift OOD samples. Experiments on both private FCU and public ISIC-OOD datasets confirm that DLCL surpasses state-of-the-art methods, demonstrating its effectiveness in handling compound shifts. Our contributions can be summarized as follows:

• We conduct a systematic study of compound shifts in FCU and identify a critical gap in existing benchmarks, which often fail to reflect realistic scenarios involving both semantic and covariate shifts.
• We propose a Dual-Level Contrastive Learning (DLCL) framework that integrates local instance-level contrastive learning and global prototype-level contrastive learning, effectively mitigating compound shifts in OOD detection.
• We validate our method via extensive experimental analysis on both a private multi-center FCU dataset and a public dataset. DLCL establishes a new state-of-the-art, demonstrating its effectiveness in handling realistic and complex distribution shifts.

2 RelatedWork

2.1 Pretrained Vision-Language Models

VLMs align images and text within a shared embedding space to facilitate cross-modal learning. By harnessing rich supervision signals from vast web-scale datasets [Li et al., 2022], VLMs have achieved remarkable representational robustness. A prime example is CLIP [Radford et al., 2021], which establishes state-of-the-art performance by training on 400 million natural image-text pairs. However, the lack of public medical data hinders the transfer of general models, prompting the development of domain-specific VLMs. BiomedCLIP [Zhang et al., 2023] significantly scales up pretraining by leveraging 15 million scientific image-text pairs extracted from biomedical literature. UniMed-CLIP [Khattak et al., 2024] introduces a unified pretraining paradigm designed for diverse medical imaging modalities. EchoCLIP [Christensen et al., 2024] leverages contrastive learning on a large-scale dataset of over one million echocardiogram video-text pairs to establish a robust VLM for cardiac imaging. While these models may exhibit strong performance on ID samples, their reliability in detecting OOD samples remains largely unverified.

2.2 Prompt Tuning in Vision-Language Models

Full-parameter fine-tuning suffers from high computational demands and catastrophic forgetting. To address this, prompt tuning [Gu et al., 2023; Cui et al., 2025] has emerged as a parameter-efficient alternative that introduces a small set of learnable tokens while keeping the backbone frozen. Early approaches primarily focused on optimizing textual prompts within the language branch. For instance, CoOp [Zhou et al., 2022b] replaces hand-crafted templates with learnable context vectors. Building on this, CoCoOp [Zhou et al., 2022a] incorporates image semantic information to generate instance-level text prompts. To foster deeper crossmodal alignment, subsequent studies [Khattak et al., 2023; Xing et al., 2023] have advanced to multi-modal prompting, injecting learnable tokens into both vision and language encoders. In the medical domain, specific adaptations have also been explored. For example, FVP [Wang et al., 2023] introduces Fourier visual prompting to enable source-free domain adaptive segmentation. MCPL [Wang et al., 2024] jointly learns anatomy-pathology, text, and image prompts via a graph-guided collaboration module. However, despite the success of prompt learning in various downstream tasks, its potential for improving OOD detection in medical imaging remains underexplored.

2.3 Out-of-Distribution Detection

OOD detection is a fundamental safeguard in image analysis, ensuring that models can reliably identify inputs falling outside the training distribution. For instance, LoCoOp [Miyai et al., 2023] introduces a few-shot OOD detection approach that enhances CLIP by learning localized, class-aware and classagnostic prompts from limited ID data. In the medical domain, existing approaches often leverage statistical analysis of feature representations. For instance, FRODO [Calli et al., 2022] proposed a training-free approach utilizing global normalized Mahalanobis distances on intermediate representations for versatile OOD detection in medical imaging. OpenMIBOOD [Gutbrod et al., 2025] provides a comprehensive benchmark for medical OOD detection, systematically evaluating post-hoc methods across diverse imaging modalities. While significant advancements have been made in modalities such as X-ray, fundus imaging, dermoscopy, and CT [Hong et al., 2024], the exploration of OOD detection in echocardiography remains nascent. Notably, current medical OOD detection research mainly focuses on post-hoc methods, often overlooking potential distributional discrepancies between training and test data. Consequently, research on generalized OOD detection tailored for FCU remains significantly limited.

3 Preliminaries

3.1 ProblemFormulation

We investigate the challenge of adapting VLMs to FCU for compound-shift OOD detection. Formally, let Ds = {(xsi , ys i )}Ns i=1 denote a labeled source dataset drawn from the distribution Ps(x, y), where x ∈ X and y ∈ Ys. Additionally, we assume access to an unlabeled target dataset Dt = {xtj }Nt j=1 drawn from Pt(x). The target domain may diverge from the source due to two distinct types of distribution shift. Covariate shifts refer to a divergence in the marginal input distribution (i.e., Ps(x) ̸= Pt(x)) while the conditional distribution P(y|x) remains invariant. Semantic shifts arise when the target label space expands beyond the source (i.e., Ys ⊂ Yt). We focus on a more realistic generalized OOD detection setting, where both covariate and semantic shifts occur simultaneously.

3.2 Contrastive Vision-Language Models

Our framework leverages CLIP [Radford et al., 2021] as the foundational backbone. CLIP is designed to align visual and textual representations in a shared latent space through contrastive learning. It comprises two parallel encoders: an image encoder I(·) and a text encoder T (·). Following established conventions [Khattak et al., 2023], we adopt a Transformer [Vaswani et al., 2017] as the text encoder, and a Vision Transformer (ViT) [Dosovitskiy et al., 2021] as the image encoder.

Image Encoder. Given a medical image x, the image encoder first divides it into M fixed-size patches. These patches are projected into a sequence of patch embeddings H0, which acts as the input to the ViT layers. The encoder then processes H0 to extract the visual representation f = I(x), where the class token ([cls]) is typically used as the aggregate feature.

Text Encoder. To encode semantic category information, CLIP utilizes a prompt template (e.g., "a medical image of the [yk]"). For a specific category yk ∈ {y1, . . . , yK}, the template is instantiated and tokenized into a sequence of word embeddings E0. The text encoder processes E0 to generate the class-specific textual embedding wk = T (yk).

The predicted probability that an image x belongs to category yk is computed as

where τ is a learnable temperature parameter and cos(·, ·) denotes the cosine similarity.

3.3 OODDetection

OOD detection aims to construct a decision function G(·) that determines whether a test sample x originates from the ID or OOD distribution. This is realized via an uncertainty scoring function S(x) and a threshold γ [Hong et al., 2024]:

In this work, the uncertainty score S(x) is defined as:

where pk = p(y = k | x) denotes the predicted probability for class k. The numerator measures predictive uncertainty via entropy, whereas the denominator normalizes by the highest class confidence. This metric amplifies the uncertainty signal for ambiguous samples. Higher scores reflect increased uncertainty, indicating a higher likelihood of OOD.

4 Method

In this section, we propose a Dual-Level Contrastive Learning (DLCL) framework for OOD detection in FCU. As shown in Figure 2, our method freezes the pre-trained VLM backbone and optimizes learnable prompts. The core of DLCL lies in combining two complementary objectives: Local Contrastive Learning and Global Contrastive Learning. Code is available at https://github.com/Liuziyi1999/DLCL.

4.1 Dual-Modal Deep Prompting

Standard prompt tuning methods typically insert prompts only at the input layer (shallow prompting). Recent studies [Khattak et al., 2023] suggest that inserting prompts into deeper encoder blocks (deep prompting) facilitates the progressive learning of stage-wise features. We introduce learnable prompts into each transformer block of both the text and image encoders. The text encoder {Tl}L-1 l=0 comprises L transformer layers. For the l-th layer Tl, we insert d learnable text prompt tokens P T l = {til }di=1 . The input to the l-th layer is constructed by concatenating the class token sl, the layerspecific prompts P T l , and the word embeddings El from the previous layer. The forward process is formulated as:

where [·, ·] denotes the concatenation operation. Then, the classification weight wk for category k is obtained via the text projection layer:

Similarly, the image encoder {Il}L-1 l=0 consists of L transformer layers. We insert d learnable visual prompt tokens P V l = {vi l }di=1 into each layer. Let gl and Hl be the visual class token and patch embeddings from the previous layer, respectively. The processing at layer l is defined as:

The final visual representation f is derived by projecting the class token from the last layer:

4.2 Local Contrastive Learning (LoCL)

To mitigate covariate shifts arising from heterogeneous acquisition protocols, we introduce LoCL to enforce fine-grained feature alignment within each mini-batch. The Bhattacharyya Coefficient (BC) [Bhattacharyya, 1946] measures the distributional overlap between two probability vectors pi and pj. Unlike KL divergence, which is asymmetric and sensitive to low-probability mismatches, the BC offers a symmetric, bounded measure that emphasizes overlap in high-confidence regions of the probability simplex. This makes BC well suited for mini-batch level alignment, where robust and interpretable similarity thresholding is critical for filtering reliable source-target pairs. The BC and its corresponding distance metric fBC are defined as:

A higher BC value indicates a stronger similarity.

Reliable source-target pairs are identified within each mini-batch. We select correctly classified source samples xsi (where argmaxps,i = ys i ) as high-confidence samples and pair them with weakly augmented target samples xtj based on high similarity, measured by BC(ps,i, pt,j) ≥ τs. We then minimize the divergence between the source prediction ps,i and the corresponding strongly augmented target prediction p′t,j . These pairs (ps,i, p′t,j ) are treated as positive examples to promote source-target alignment. The alignment loss is formulated as:

where H is the set of index pairs satisfying both confidence and similarity constraints.

We also leverage unlabeled target data to enhance the reliability of predictions. A target sample is considered valid if the maximum probability pmax j = maxpt,j of its weakly augmented view exceeds a confidence threshold τc. The pseudolabel is assigned as ˜ yj = argmaxpt,j. Subsequently, we minimize the entropy of the averaged predictions from corresponding weakly and strongly augmented target samples:

where 1(·) denotes the indicator function.

The overall objective for LoCL is defined as:

where λu is a hyperparameter balancing the two terms. This objective effectively reduces the domain gap caused by covariate shifts, establishing a robust feature space for subsequent global contrastive learning.

4.3 Global Contrastive Learning (GlCL)

Complementing the local feature alignment of LoCL, GlCL is specifically designed to address semantic shifts by enforcing global categorical constraints. The objective of GlCL is to maintain well-separated decision boundaries for known classes while effectively identifying potential OOD samples.

Class Prototype Extraction

We represent each semantic category using a global prototype ck, which serves as a stable representative in the shared latent space. To mitigate the inherent noise in ultrasound pseudolabels and enhance prototype reliability, we utilize both labeled source samples and high-confidence target samples. At the end of each training epoch e, a temporary prototype ˆ c(e) k for each class k ∈ {1, . . . ,K} is computed by aggregating feature representations:

where δ(k) s,i = 1(ys i = k) and δ(k) t,j = 1(pmax j ≥ τc)1(˜ yj = k) are indicator functions for source labels and confident target pseudo-labels, respectively. We update global prototypes via momentum-based Exponential Moving Average [Tarvainen and Valpola, 2017] to stabilize the feature space:

where α ∈ (0, 1] is the momentum coefficient.

Prototypical Contrastive Learning

To facilitate discriminative representation learning across the target domain, we employ a margin-aware prototypical contrastive loss. Let sj,k = f ⊤ j ck/T denote the temperaturescaled cosine similarity between the feature representation fj of a sample xtj and the class prototype ck. We partition the target samples into a high-confidence set U and a lowconfidence set ¯ U based on the threshold τc. For the highconfidence samples in U, our goal is to enforce compactness with their pseudo-labeled prototypes while maximizing the margin from the prototypes of the hardest negative classes. Specifically, for each sample xtj ∈ U, we identify the hardest negative class kj = argmaxk̸=˜ yj pj,k. To accommodate the varying intra-class diversity of fetal heart views, we define an adaptive margin mt = µt + σt , where µt and σt denote the mean and standard deviation of the similarity gaps between the hardest negative prototype and the pseudo-labeled prototype across the batch. The contrastive loss for highconfidence samples is defined as:

where [·]+ = max(0, ·) is the hinge function. This term enforces a margin between the pseudo-labeled and hardest negative prototypes.

Unlike previous methods that discard low-confidence samples, we argue that ¯ U contains valuable information regarding decision boundaries and OOD patterns. To exploit this, we further partition ¯ U into two disjoint subsets based on the shape of their prediction distributions: Ambiguous Samples (Uamb) and Potential OOD Samples (Uood). Formally, a sample is categorized as ambiguous if it exhibits a distinguishable but weak peak, satisfying the gap condition: ∃k ̸= ˜ yj, pmax j - pt,j,k > δ, where δ represents a fixed margin of 0.4. For each sample in Uamb, we minimize its similarity to the corresponding set of unlikely classes K† j (where the gap condition holds), thereby pruning the negative space. Let ¯ sj = |K† j |-1 ∑k∈K† j sj,k denote the average similarity to these unlikely prototypes. We then compute the adaptive margin md = µd + σd using the mean µd and standard deviation σd of these average similarities. The ambiguity loss is defined as:

Conversely, samples violating the gap condition typically exhibit flat, high-entropy distributions, often indicative of unknown anatomical structures. We identify these instances as potential OOD samples, denoted by Uood = ¯ U \ Uamb. Since these samples should not correlate strongly with any known category, we enforce a global suppression constraint. We calculate the average similarity across all K prototypes as ˆ sj = 1K ∑Kk=1 sj,k. To enforce separation, we define an adaptive margin mo = µo + σo , derived from the mean µo and standard deviation σo of the similarity distribution {ˆ sj}j∈Uood . The OOD suppression loss encourages these instances to maintain low similarity with all base prototypes:

Finally, the total global objective integrates these three complementary components:

where λd and λo are weighting factors that balance the contributions of the corresponding loss terms. GlCL effectively preserves class discriminability while alleviating noise propagation caused by open-set distractors.

4.4 Overall Loss Function

To foster discriminative feature learning on the source domain, we train the model via the standard supervised crossentropy loss on the labeled source set:

The final training objective of DLCL integrates the supervised source loss with the proposed local and global contrastive constraints to jointly handle compound shifts. The total loss is formulated as:

where λl and λg are hyperparameters controlling the contributions.

5 Experiments and Results

5.1 Datasets and Experimental Setup

Multi-center Fetal Cardiac Ultrasound (MFCU) Dataset.

We collected B-mode fetal echocardiography scans from pregnant women between 2018 and 2021. For each scan, we accessed freeze-frame images saved by sonographers during the exam. The MFCU dataset comprises 22,050 source-domain images acquired from the Key Laboratory of Maternal-Fetal Medical Research and provided by Anzhen Hospital. The target domain includes 49,658 images collected from 38 medical institutions participating in the 13th Five-Year National Key Research and Development Plan. The dataset is annotated with seven standard fetal cardiac views: Three-Vessel and Trachea View (3VT), Three-Vessel View (3VV), Four-Chamber View (4CV), Aortic Arch View (AA), Left Ventricular Outflow Tract View (LVOT), Right Ventricular Outflow Tract View (RVOT), and Transverse Abdominal View (TAV). Notably, the target domain includes 8,823 images of non-standard views. All images were manually reviewed by multiple experienced clinicians, and discrepancies were resolved by consensus to ensure annotation accuracy.

ISIC-OOD Dataset. We constructed the ISIC-OOD dataset based on the ISIC2019 dataset [Tschandl et al., 2018]. The base classes include basal cell carcinoma (bcc), benign keratosis (bkl), melanoma (mel), and melanocytic nevi (nv), while the remaining categories are treated as novel classes. To construct samples exhibiting semantic shift, we select images that share the same imaging modality as the base classes but belong to novel classes. To simulate covariate shift, we retain the base classes while introducing distributional variations due to differences in imaging devices and acquisition protocols. The resulting dataset is divided into two domains: BCN and HAM, representing different acquisition protocols. Specifically, the BCN domain contains 11,010 images from base classes and 1,404 from novel classes, while the HAM domain comprises 9,431 base-class images and 585 novel-class images. When a specific domain is used as the source domain, only the base class images are included. This setup allows for the assessment of robustness against concurrent covariate and semantic shifts. The dataset can be accessed at https://www.kaggle.com/datasets/liuzyiii/isic-ood.

Implementation Details. Our experiments were implemented in PyTorch [Paszke et al., 2019] on a single NVIDIA V100 GPU. We utilized the pretrained ViT-B/16 CLIP model for prompt tuning, with word embeddings of 512 dimensions and patch embeddings of 768 dimensions. The text and visual prompts were updated using the SGD optimizer. We trained the model for a total of 60 epochs, where the first 50 epochs used only the supervised objective. The length of the text and visual prompts was set to 8, and the depth was set to 12. Data augmentation was performed using RandAugment (subset size = 2). The batch size was 64, and the learning rate was set to 0.0025. We set the confidence thresholds as τs = τc = 0.8 for all experiments. Based on the experiments, we empirically set [λl,λg,λu,λd,λo] to [1,1,0.1,0.1,0.1].

5.2 Results and Analysis

We benchmark our method against ten methods, including general VLMs (CLIP [Radford et al., 2021]), medicalspecific VLMs (BiomedCLIP [Zhang et al., 2023], MedCLIP [Wang et al., 2022], UniMedCLIP [Khattak et al., 2024], EchoCLIP [Christensen et al., 2024]), and prompt learning strategies (CoOp [Zhou et al., 2022b], LoCoOp [Miyai et al., 2023], CoCoOp [Zhou et al., 2022a], VPT [Jia et al., 2022], and MaPLe [Khattak et al., 2023]). The quantitative results on the MFCU dataset are detailed in Table 1. Traditional VLMs, such as CLIP, that do not specialize in medical domains, show a notably low accuracy of 29.19% on average, further indicating the difficulty of generalization between the source (natural images) and target (ultrasound images) domains. As EchoCLIP is mainly trained on adult clinical echocardiography data, its generalization capability is limited when transferring to fetal echocardiography. Medical-specific VLMs like BiomedCLIP (49.05%) and MedCLIP (61.70%) outperform CLIP but still struggle to achieve high accuracy under OOD conditions, highlighting their limitations in bridging the domain gap. In contrast, models employing prompt learning strategies, such as CoOp and CoCoOp, show notable improvements when optimized using a combination of labeled samples from the source domain and unlabeled samples from the target domain. CoOp achieves an average accuracy of 77.74%, suggesting its better adaptability to domain shifts compared to the general VLMs. Finally, our proposed method not only achieves the highest average accuracy (93.94%) under covariate shifts but also secures the best balance between base (AccS = 76.79%) and novel class (AccU = 84.25%) recognition under compound shifts. The harmonic mean (80.35%) and AUROC (89.61%) validate its superior OOD detection potential in clinical environments.

5.3 Ablation Studies

Contribution of Different Losses. Table 2 summarizes the impact of each component. The exclusion of either the local or global contrastive losses leads to the most significant

60

40

20

0

20

40

60

performance degradation, underscoring the necessity of our dual-level alignment strategy. Removing LLoCL results in a sharp accuracy drop (93.94%→ 89.30%), verifying its critical role in mitigating covariate shifts. Meanwhile, the absence of LGlCL degrades both harmonic mean and AUROC, demonstrating that global prototype constraints are essential for maintaining semantic consistency. Furthermore, ablating Lamb reduces the harmonic mean to 75.37%, validating that leveraging low-confidence samples effectively refines decision boundaries. In compound-shift scenarios, Lood proves indispensable for identifying novel classes. Since Lood is only active under compound shifts, its ablation does not affect covariate-only accuracy. In summary, each component of DLCL plays a distinctive and critical role.

Impact of Loss Components. We analyze hyperparameter sensitivity by varying each parameter individually. Figure 3(a) reveals that increasing the weight of LoCL (λl) consistently improves performance, peaking at λl = 1. Notably, excessive global regularization (λg) may compromise feature discriminability. Therefore, we set λg = 1. Figure 3(c) indicates that λu notably boosts the AUROC and harmonic mean, with an optimal trade-off around λu = 0.1. Conversely, Figure 3(d) shows that setting λd = 0.1 significantly improves OOD metrics under combined shifts. Since Lood is designed to address semantic shifts, λo mainly affects OOD-related metrics rather than covariate-shift classification accuracy. We therefore default to λd = λo = 0.1.

Effect of τc and τs. Empirically, source predictions aligned with ground truth typically exhibit a near one-hot distribution. Under this condition, the similarity constraint effectively degenerates into a unidirectional confidence check on the target prediction. Given that model predictions tend to polarize toward either high or low confidence, the set of selected pairs remains stable. This characteristic renders the loss formulation robust to variations in the hyperparameter τs. As shown in Figure 4, the proposed model demonstrates stable performance across a wide range of values for both τc and τs. Although slight variations are observed in accuracy, AUROC, and harmonic mean across different hyperparameter settings, the overall trend remains stable. These findings confirm that DLCL maintains strong robustness to both τc and τs, highlighting its reliability across varying calibration thresholds.

Feature Visualization Analysis. To qualitatively evaluate feature alignment, we employ t-SNE [Maaten and Hinton, 2008] to visualize feature distributions, in which source and target samples are projected into a shared space. Regarding class discriminability, the baseline (Fig. 5a) suffers from significant overlap, whereas DLCL (Fig. 5b) achieves high intraclass compactness and distinct boundaries. These results verify that DLCL effectively learns domain-invariant representations, thereby ensuring both robust cross-domain alignment and superior separability.

Visualization of OOD Score Distributions. For practical deployment, a robust medical VLM should detect OOD samples under various distribution shifts and respond accordingly, e.g., by routing covariate-shifted samples to specialized expert models for reliable ID recognition. We visualize the distribution of OOD scores S(x) to assess separability. The baseline CLIP (Fig. 6a) fails to distinguish ID from OOD samples, exhibiting significant overlap. In contrast, DLCL (Fig. 6b) successfully disentangles these distributions: samples with covariate shifts (green) are aligned with the ID cluster (low scores), whereas those with compound shifts (yellow) are effectively pushed toward high scores. This confirms that DLCL tolerates acquisition-related variance while robustly rejecting anatomical anomalies. In this paper, we empirically set γ = 0.8. By adjusting the detection threshold γ, the model can flexibly control its sensitivity to semantic shifts while preserving robustness to covariate variations.

5.4 Results on Additional Datasets

To further validate the generalizability of our method across different medical image domains, we extend the evaluation to the ISIC-OOD dataset under two cross-domain settings, i.e., BCN2HAM and HAM2BCN. As shown in Table 3, several baseline methods achieve relatively high AccU. However, such improvements on novel classes are often accompanied by a clear degradation in base class recognition, leading to inferior overall performance and a suboptimal balance between base and novel classes. For example, in HAM2BCN, CLIP obtains an AccU of 98.86%, but its AccS and harmonic mean are only 1.90% and 3.73%, respectively. Similarly, CoCoOp achieves a higher AccU than DLCL, but its AccS and H are substantially lower.

In contrast, DLCL achieves more balanced cross-domain performance. In the BCN2HAM setting, DLCL obtains the best average accuracy of 68.81% and the highest harmonic mean of 69.93%, outperforming the strongest baseline by 5.39% and 5.13%, respectively. In the HAM2BCN setting, although some baselines report higher AccU, DLCL achieves the best AccS of 42.90%, the best average accuracy of 60.29%, and the highest harmonic mean of 54.27%. These results indicate that DLCL does not simply bias the prediction toward novel classes but instead improves the overall discrimination ability across both base and novel categories.

6 Conclusion

In this work, we present a study on adapting VLMs for OOD detection in FCU, addressing the practical challenges induced by both clinical covariate and semantic shifts. We propose DLCL, a specialized framework that incorporates a prompt learning module to enable efficient fine-tuning of VLMs. Beyond model adaptation, DLCL combines instance-level Local Contrastive Learning and prototype-level Global Contrastive Learning to encourage the acquisition of discriminative and transferable representations across heterogeneous distributions. Extensive experimental results highlight the potential of VLM-based OOD detection in building trustworthy and generalizable medical imaging systems for real-world deployment. We hope our work will inspire future research on multi-modal OOD detection in medical imaging under realistic cross-hospital distribution shifts.

Ethical Statement

This study was conducted in compliance with the ethical standards required by IJCAI. It is based on a retrospective analysis of fetal cardiac ultrasound data collected during routine clinical practice, with approval from the corresponding Ethics Committee or Institutional Review Board. All data were fully anonymized prior to access by the authors, and informed consent was obtained from the patients or their legal guardians, or waived in accordance with local regulations for retrospective studies. The proposed method was evaluated as a clinical decision support tool in a research setting.

From a clinical perspective, we explicitly consider safetycritical risks associated with deploying vision-language models in healthcare. Overconfident predictions on out-ofdistribution samples may pose serious threats to patient safety, particularly in prenatal diagnosis. To mitigate such risks, our work focuses on reliable out-of-distribution detection rather than direct clinical decision-making, aiming to provide uncertainty-aware signals that can assist clinicians instead of replacing expert judgment. Moreover, we acknowledge potential issues related to dataset bias and cross-center heterogeneity. By explicitly modeling covariate and semantic shifts across multiple institutions, our method seeks to improve robustness and reduce performance disparities caused by data distribution biases. Overall, this research is intended to support trustworthy and responsible AI development in medical imaging, with careful consideration of ethical, safety, and fairness concerns in real-world clinical deployment.

Acknowledgements

This work was supported in part by the National Science and Technology Major Project (2025ZD0123401), in part by the National Natural Science Foundation of China under Grant 62406014, in part by the Beijing Natural Science Foundation (IS2408I, 7262079), in part by the Zhejiang Natural Science Foundation (LZ26F020012) and in part by the Startup Funds of Hangzhou International Innovation Institute of Beihang University under Grant No. 2024KQ045 and No. 2024KQ027.