TriCLIP: Tri-stage Calibration of Prompts, Evidence and Fusion for Robust Biomedical Vision–Language Models

1 Introduction

Biomedical imaging is a cornerstone of clinical diagnosis, yet it remains substantially harder to model than naturalimage recognition due to sparse semantic distributions, subtle discriminative cues, and the reliance on strong biomedical priors. Recently, large-scale vision-language models (VLMs) have introduced a unified paradigm for biomedical image analysis by leveraging cross-modal alignment and strong transferability [Radford et al., 2021; Li et al., 2022; Huang et al., 2024]. These models have shown promise for multimodal diagnosis, report generation, and disease classification. However, when transferred to biomedical scenarios, general-purpose VLMs often exhibit pronounced performance degradation, especially under high-generalizationdemand settings such as base-to-novel generalization (training on base classes and evaluating on unseen novel classes with limited or no additional supervision) and few-shot learning (adapting with only a few labeled examples per category).

* Corresponding author.

This degradation is not only due to data-driven factors but also stems from structural bottlenecks along the promptvision-fusion pipeline. First, prompt fragility, biomedical diagnostic prompts are semantically dense and highly structured, such that minor paraphrasing or template changes can induce large variance in the language-conditioned representation, leading to unstable predictions and poor robustness to linguistic perturbations. Second, saliency-dominated visual attention, biomedical images often contain small lesions with ambiguous boundaries, and VLMs tend to over-focus on a few dominant regions, yielding shortcut-like attention patterns and insufficient contextual coverage (e.g., surrounding tissue structures), which is particularly harmful when transferring across domains or classes. Third, selectivity failure in cross-modal fusion, existing cross-modal attention mechanisms may amplify redundant or noisy channels and fail to separate informative signals from distractors consistently, resulting in fragile alignment and reduced discriminability. These issues become more severe in base-to-novel and fewshot regimes, where distribution shift and limited supervision amplify prompt variance, attention bias, and cross-modal noise sensitivity.

Prior efforts have attempted to improve individual components of biomedical VLMs, including prompt tuning and prompt learning [Zhou et al., 2022b; Zhou et al., 2022a; Yao et al., 2023; Peng et al., 2025], saliency-guided training and attention distillation [Wu et al., 2025], and refined fusion architectures [Li et al., 2023]. While effective in specific settings, these approaches are largely component-wise and are often designed in isolation, lacking a systematic principle to jointly address the interdependencies among prompting, visual representation learning, and cross-modal fusion. As a result, improving one component can leave other failure modes intact (or even exacerbate them), limiting robustness and generalization in challenging transfer scenarios.

To overcome these limitations, TriCLIP is proposed as a decomposition-based cross-modal learning framework that restructures information flow across the entire pipeline through three synergistic stages: Prompt Distribution Regularization (PDR), Feedback-driven Counterfactual Masking (FCM), and Selective Regulation Fusion (SRF). PDR regularizes the prompt space by optimizing under class-conditioned stochastic context perturbations, promoting prompt-invariant conditioning and reducing sensitivity to linguistic variations. FCM alleviates saliency collapse by applying feedbackdriven reverse-saliency masking on dominant regions (derived frommodel attribution with gradient isolation), encouraging complementary cue discovery and improving contextual coverage. Finally, SRF enhances cross-modal discriminability by regulating fusion that suppresses redundant interactions while reinforcing informative cross-modal correspondences. Jointly, these stages form a closed-loop design that stabilizes language conditioning, diversifies visual evidence, and improves fusion selectivity, leading to stronger generalization in base-to-novel transfer and few-shot adaptation. Our main contributions are summarized as follows:

• Proposal of Prompt Distribution Regularization, a classconditioned training strategy that samples two independent stochastic perturbations of the learnable context tokens per class and enforces cross-view representation consistency, thereby learning prompt-invariant text embeddings and improving robustness to biomedical language variations.
• Development of Feedback-driven Counterfactual Masking, a mechanism that suppresses dominant saliency regions to alleviate attention collapse and encourage complementary evidence mining for improving contextual coverage.
• Development of Selective Regulation Fusion, a cross attention module based on regulation that separates informative alignments from redundant channels, improving the stability and discriminability of cross-modal fusion.

2 RelatedWork

2.1 Vision-Language Models

Large-scale vision-language models (VLMs) learn crossmodal consistency from massive image-text pairs and form the basis of unified visual understanding frameworks. Foundational models such as CLIP [Radford et al., 2021] have driven this development, while subsequent pre-training paradigms, including BLIP [Li et al., 2022], further strengthen alignment and generative capabilities for improved downstream transferability. In the biomedical domain, VLM research has expanded rapidly. MedCLIP [Wang et al., 2022b] leverages medical image-text contrastive learning for effective zero-shot and fine-tuned performance. BiomedCLIP [Zhang et al., 2023] constructs large-scale biomedical representations, and methods such as MedKLIP [Wu et al., 2023] inject domain knowledge to improve alignment and grounding. BioViL-T [Bannur et al., 2023] models temporal and multi-image contexts, while MedBLIP [Chen and Hong, 2024b] introduces lightweight 3D-text alignment for diagnosis and report generation. MediVLM [Goswami et al., 2025] advances cross-attention fusion and medical report synthesis, and additional radiology and structure-focused approaches, including RadCLIP [Lu et al., 2025] and IMITATE [Liu et al., 2024], further enhance classification, retrieval, and generative tasks. Despite this progress, both general-purpose and biomedical-domain VLMs still face structural limitations in clinical transfer, including prompt sensitivity, visually biased saliency, and unstable cross-modal alignment during fusion.

2.2 Prompt Learning

Prompt learning has become a key mechanism for improving the adaptability and data efficiency of vision-language models. Traditional biomedical VLMs often rely on manually crafted discrete prompts, which lack flexibility and are highly sensitive to clinical semantic variability and distribution shifts. Continuous prompt learning thus emerges as a more robust alternative. CoOp [Zhou et al., 2022b] learns continuous prompt vectors to replace manual design, while CoCoOp [Zhou et al., 2022a] and ProGrad [Zhu et al., 2023] mitigate overfitting through conditional modulation or gradient constraints. In the biomedical domain, XCoOp [Bie et al., 2024] incorporates disease-specific knowledge, and DCPL [Cao et al., 2024] improves clinical robustness by cross-modal domain modeling. With large language models achieving near-human performance in clinical report generation [Krishna et al., 2024], recent work employs LLMgenerated biomedical phrases or structured terminology as fixed semantic templates. In this setting, the LLM is frozen, and continuous prompts are learned on top of its template, forming a layered structure that enhances semantic stability and mitigates prompt overfitting across classes. Despite these advances, prompt learning still suffers from semantic collapse, where learned prompts overfit to specific template formulations and degrade under sparse data, unseen classes, or phrasing perturbations.

2.3 Cross-modal Discriminant Alignment

Attention mechanisms are fundamental to biomedical visual decoding and cross-modal fusion but exhibit persistent structural limitations. On the visual side, both convolutional and Transformer-based models rely heavily on saliency-driven attention to extract discriminative lesion features [Touvron et al., 2021]. Due to the sparse and subtle nature of biomedical abnormalities, such mechanisms often suffer from saliency collapse, leading to over-focus on prominent local regions while neglecting broader structural context, thereby hindering generalization. At the cross-modal level, classical vision-language frameworks such as UNITER [Chen et al., 2020] and BLIP [Li et al., 2022] employ cross-attention for image-text alignment. However, their uniform weighting schemes frequently absorb noise and induce misalignment. Later studies improve selectivity through refined co-attention designs [Varol Arısoy et al., 2025] or external knowledge integration [Chen and Hong, 2024a], yet many approaches still rely on implicit or static gating without explicitly separating informative signals from redundant activations. Recent multimodal research explores stronger structural regulation via multi-branch attention, contrastive channel separation, and dual-path alignment [Wang et al., 2022a]. These methods highlight the value of constructing both reinforcing and inhibitory pathways to enhance discriminability and stability. Given the complex noise patterns in biomedical imaging, such explicit control is particularly crucial for biomedical VLMs. Motivated by these observations, the alignment mechanism introduced in this work adopts complementary attention branches that enhance informative cues while suppressing noise. Learnable modulation further enables dynamic semantic selectivity, offering structurally grounded improvements in biomedical image-text alignment.

3 Methodology

To address these challenges, TriCLIP is introduced as a decomposition-based framework that calibrates information flow across prompt conditioning, visual evidence, and crossmodal fusion, improving robustness in base-to-novel generalization and few-shot biomedical classification tasks.

3.1 Prompt Distribution Regularization

Biomedical prompts are semantically dense and highly sensitive to minor paraphrasing. Conventional prompt learning optimizes a fixed set of context tokens, which often overfits to training templates and exhibits high variance under contextual perturbations, leading to unstable language conditioning in base-to-novel transfer and few-shot regimes. To address this issue, we propose PDR, a training strategy that (i)

stochastically perturbs learnable context tokens and (ii) enforces semantic consistency across perturbed views, thereby encouraging prompt-invariant text representations.

For each class i, let Ci ∈ RL×d denote the learnable context token features, where L is the number of context tokens and d is the token embedding dimension. PDR applies context-token dropout with a class-wise dropout probability δi ∈ [0, 1], which is broadcast to all context tokens of that class. We set the retention probability to pi = 1 - δi and choose δmax < 1 to avoid degenerate all-dropout; in implementation, a small constant ϵ is used for numerical stability when needed. We sample a token mask Mi according to:

where 1 ∈ RL is the all-ones vector and Mi ∈ {0, 1}L indicates which context tokens are retained. The perturbed context features are computed using inverted-dropout normalization:

where ⊙ denotes element-wise multiplication (broadcast along the feature dimension). This preserves the expected feature magnitude under different dropout rates. The final prompt sequence is constructed by concatenating the fixed prefix/suffix tokens with the perturbed context:

PDR adopts a confidence-driven schedule to determine the class-wise dropout probability δi. We maintain an Exponential Moving Average (EMA) confidence score αi ∈ [0, 1] for each class i, updated from the model's predicted probability on samples of that class:

where p(y | x) = Softmax(s(x))y denotes the predicted probability of the ground-truth label y for input x, and s(x) is the corresponding logit vector. Here m is the EMA momentum, and StopGrad(·) prevents gradients from flowing through the confidence estimator.

Intuitively, classes that the model currently predicts with lower confidence are assigned stronger prompt perturbations. We map the confidence to a bounded dropout probability via

where δmin and δmax control the dropout range and γ controls the nonlinearity. This design increases perturbation for lowconfidence classes while preserving semantic cores for highconfidence classes.

To encourage prompt-invariant representations, for each class i, we generate two independently perturbed prompt views by sampling two masks, producing f (a) i and f (b) i . We impose a consistency loss across the two views:

where ncls is the number of classes.

3.2 Feedback-driven Counterfactual Masking

While PDR strengthens robustness on the language side, biomedical VLMs also suffer from saliency-dominated attention, where the visual branch over-relies on a few highly discriminative regions and neglects contextual cues. To alleviate this attention bias, we design a Feedback-driven Counterfactual Masking Approach that applies reverse-saliency masking to suppress dominant areas and encourage the mining of complementary evidence.

To generate class-specific saliency maps, we adapt GradCAM by mapping the patch embeddings to spatial features. Specifically, the output sequence from the final normalization layer is reshaped into a 2D feature map A ∈ RD×H×W . The saliency map for class c is then defined as:

where Ak ∈ RH′×W ′ is the k-th feature map corresponding to the embedding dimension k, and the weight wck is obtained via global average pooling of the gradients:

where yc denotes the score for class c. To stabilize training, the saliency computation is treated as a feedback signal, and gradients are not propagated through the saliency generation path. We normalize Hotc into [0, 1] and generate a suppression mask that zeros out the most salient regions:

Where τ is a constant, with a default value of 0.7. The masked image is as follows:

By training on I′ , the model is forced to rely less on dominant evidence and to capture secondary cues, thereby mitigating saliency collapse and improving generalization under distribution shift.

3.3 Selective Regulation Fusion

When biomedical classes are fine-grained, naive fusion can amplify redundant or pseudo-correlated cross-modal signals, leading to fragile alignment under noise and domain shift. To mitigate this issue, we propose Selective Regulation Fusion(SRF), a differential regulation module that explicitly pairs reinforcing and suppressing attention flows and combines them via a learnable coefficient.

Let the image and text representations used for fusion be X ∈ RB×N×D and Y ∈ RB×N×D , where B is the batch size, N denotes the number of fused embeddings, and D is the embedding dimension. We compute linear projections Q = Q(Y), K = K(X), and V = V(X), and adopt an even number of heads H with per-head dimension dh = D/H. We define He = H/2 head groups, where each group contains two sub-heads: a reinforcing sub-head and a suppressing sub-head. Accordingly, we reshape Q, K into 2He sub-heads and reshape V into He groups with channel width 2dh:

We compute attention probabilities in the standard way:

and reorganize them into paired sub-heads:

where index 0 corresponds to the reinforcing sub-head and index 1 to the suppressing sub-head. SRF forms a regulated weighting map by subtracting the suppressing flow:

where λ is a learnable global regulation coefficient defined as λ = exp(⟨λq1, λk1⟩) - exp(⟨λq2, λk2⟩) + λinit, (16) with learnable vectors λq1, λk1, λq2, λk2 ∈ Rdh and initialization bias λinit. The fused representation is obtained by applying the differential weighting to the values:

To stabilize training under differential weighting, we apply RMS normalization over the channel dimension, followed by a fixed rescaling:

SRF improves fusion stability by explicitly regulating and suppressing signals via λ, thereby reducing redundant crossmodal interference while preserving informative alignment cues in base-to-novel and few-shot settings.

3.4 Loss Function

We jointly optimize text-side robustness, vision-side representation, and cross-modal fusion with a weighted multi-term objective. The classification loss is

where yc and pc are the one-hot label and predicted probability, ncls is the number of classes classified. To maintain semantic alignment with fixed class prototypes, we use a cosine consistency term:

where fv is the visual feature and Eyi fixed is the fixed semantic vector of the ground-truth class. To preserve the zero-shot semantic distribution for better generalization, we distill from a frozen teacher:

where pzs is the prediction probability of zero-shot, and pmodel is the prediction probability after the network is trained normally. The overall loss is

where LPM is the Prompt Distribution Regularization consistency loss.

4 Experiments and Results

4.1 Datasets and Setting

To comprehensively evaluate the proposed method, we assess its effectiveness under multiple evaluation protocols, primarily including base-to-novel generalization and few-shot learning. Experiments are conducted on eleven diverse biomedical imaging datasets, including BTMRI [Nickparvar, 2021], CHMNIST [Kather et al., 2016], COVID [Tahir et al., 2021], CTKidney [Islam et al., 2022], DermaMNIST [Codella et al., 2019; Tschandl et al., 2018], KneeXray [Chen, 2018], Kvasir [Pogorelov et al., 2017], LC25000 [Borkowski et al., 2019], OCTMNIST [Kermany et al., 2018], RETINA [K¨ohler et al., 2013; Porwal et al., 2018], and BUSI [Al-Dhabyani et al., 2020]. These datasets cover ten different organs and nine imaging modalities. Detailed information about each dataset is provided in the Appendix.

Base-to-Novel Generalization: The image classes of the dataset are divided into base and novel classes. The model is trained only on the base classes but tested on both base and novel classes.

Few-Shot Learning: To assess the model's performance under limited supervision, we conduct few-shot learning experiments with different numbers of labeled samples per class (k = 1, 2, 4, 8, and 16).

4.2 Implementation Details

For all experiments except Few-Shot Learning, we adopt a 16-shot setting, where each class contains only 16 training samples. We use the ViT-B/16 from the BiomedCLIP model as the backbone for all experimental configurations. To ensure reliability and fairness of the results, we report the final accuracy averaged over three different random seeds. For Base-to-Novel Generalization, the number of training epochs is set to 100, and for Few-Shot Learning, it is set to 50. The batch size for all experiments is 4, and the learning rate is 0.0025. We use the SGD optimizer for training. All experiments are conducted on RTX 4090.

4.3 Performance Comparisons

Base-to-Novel Generalization. As shown in Table 1, we conduct experiments on ten public biomedical imaging datasets. Training is performed only on base classes, and performance is measured using three metrics: Base accuracy on seen classes, Novel accuracy on unseen classes, and HM, the harmonic mean of Base and Novel, reflecting the balance Context Tokens Context Tokens between retention and generalization. Across most datasets, our method achieves the highest HM scores, demonstrating broad applicability and stability. It particularly improves the recognition ability of novel classes. On average, TriCLIP achieves 79.23% Base accuracy, 76.15% Novel accuracy, and 77.66% HM, outperforming all baselines, demonstrating superior generalization and robustness.

Figure 2: Visualization of the four nearest words for learned context tokens, with Euclidean distances in the embedding space.

Few-Shot Learning. To assess the efficiency of our model under limited supervision, we conduct Few-Shot learning experiments with k = 1, 2, 4, 8, 16 labeled samples per class. Several related approaches are compared, and the average accuracies on eleven datasets are reported in Table 2. More detailed data is provided in the appendix. Across different values of k, our method TriCLIP achieves the highest accuracy consistently. Specifically, it reaches 61.09% under one-shot setting, outperforming the second-best method by 4.17%.

4.4 Ablation Studies

Ablation of the component of the model. We conduct systematic ablation studies to analyze the contribution of each component, with results shown in Table 3. Each module provides consistent performance gains: PDR enhances the model's robustness to contextual perturbations, FCM improves discriminative focus, and SRF further strengthens cross-modal alignment. When combined, these strategies achieve the best overall performance, demonstrating their strong complementarity.

Impact of the Prompt Distribution Regularization Module. The heatmap in Figure 2 illustrates how PDR enhances prompt regularization. With the four learnable tokens, PDR drives the convergence of cue word embeddings towards domain-specific biomedical concepts (e.g., MRI, retina, calcification), which significantly brings them closer to real medical terms in the vector space (i.e., improves semantic similarity, as shown by the bolded entries in the figure. Thus, generic ambiguity of the original tokens or ineffective semantic drift is avoided. Token-level inspection shows that PDR induces more focused and clinically aligned semantics. The lighter cells indicate closer alignment to target concepts. Overall, PDR improves semantic precision, increases diversity, and yields more robust prompts across datasets by mitigating overreliance on a single handcrafted template.

Impact of Selective Regulation Fusion Module. Figure 3 presents a performance comparison among the full SRF mechanism, the reinforcing heads, and the suppressing heads. Trained only on base classes, the full mechanism consistently outperforms the two ablated settings. Reinforcing heads yield only marginal gains on base classes, while suppressing heads alone degrade base performance obviously, indicating that decorrelation without reinforcement weakens core semantics. In contrast, SRF, jointly modeling reinforcing and suppressing heads, improves novel class performance and overall generalization substantially, achieving the HM of 77.66%. Overall, the results demonstrate that the full mechanism preserves strong base-class performance while effectively enhancing transfer to novel classes.

Research on text ablation with different lengths. Figure 4 shows the impact of different initial text prompt lengths (2, 4, 8, 16, 32, and 64) on the classification performance of the model. The model is trained only on the base class and evaluated on both the base and novel classes. The results indicate that when the text prompt length is set to 4, the best performance is achieved in all indicators. In contrast, overly short prompts have slightly insufficient generalization ability on the novel class, As the cue length increases further, especially from 32 to 64, the model performance tends to decrease significantly, especially in the novel metrics. The performance performs worst when the length is 64. These results indicate that moderate length of text prompts can provide effective semantic information while avoiding redundant interference, which is more conducive to the model's generalization towards unseen classes.

Different Backbone Ablation Studies. This study compares four representative pretrained VLM backbones, including CLIP pretrained on natural images, PMC-CLIP [Lin et al., 2023] pretrained on the PMC-OA dataset, PubMedCLIP [Eslami et al., 2021] adapted to the biomedical domain via domain-adaptive fine-tuning on PubMed, and BiomedCLIP. The detailed results are presented in Figure 5. BiomedCLIP consistently achieves the best performance, and therefore, it is selected as the default backbone in our experiments.

Mask visualization of Feedback-driven Counterfactual Masking Module. As shown in Figure 6, the visualization results of the FCM sequentially present the saliency heatmap (Heatmap), the suppression mask (Mask), and the masked image (Last). The Heatmap illustrates the model's response intensity to different regions of the input image during forward inference, revealing the locations of salient features that the model attends to. The Mask is generated based on the Heatmap, and it is used to suppress high-response regions during subsequent training, thereby reducing the model's over-reliance on locally salient areas.

5 Conclusion

This work introduces a unified framework to improve the robustness of biomedical vision-language models by jointly addressing prompt regularization, saliency-aware visual rebalancing, and selective cross-modal fusion. PDR enhances robustness to linguistic variation by encouraging promptinvariant text representations, while the FCM alleviates saliency-dominated learning through complementary visual evidence mining. Further, SRF stabilizes cross-modal fusion by suppressing redundant alignments and preserving informative cross-modal correspondences. As a whole, this approach provides a practical attempt and verification for improving cross-distribution generalization in biomedical multimodal systems, particularly under low-resource and heterogeneous clinical conditions. Despite these encouraging results, the current study primarily targets classification benchmarks, and broader validation on more tasks, institutions, and imaging modalities is needed to fully assess real-world clinical robustness.

Acknowledgments

This work is supported by New Generation Artificial Intelligence-National Science and Technology Major Project (2025ZD0123703) and Beijing Natural Science Foundation (L253010).