Open-Set Test-Time Adaptation in VLMs via Noise-Immune Self-Purification

1 Introduction

Despite the remarkable generalization of deep learning models, they still suffer performance degradation when encountering distribution shifts during testing. Test-time adaptation (TTA) has been introduced to mitigate this issues [Shu et al., 2022; Ma et al., 2023; Jia et al., 2024]. Nevertheless, TTA is vulnerable in open-world scenarios, where out-of-distribution (OOD) samples bias the optimization direction, thereby undermining the discriminability of known classes [Li et al., 2023b; Yang et al., 2024]. This motivates the open-set TTA (OSTTA) challenge, which aims to robustly adapt to in-distribution (ID) shifts while mitigating the detrimental impact of OOD samples [Gao et al., 2024; Yang and Xu, 2025; Dong et al., 2025; Zhao et al., 2025].

*Corresponding author.

To tackle the OSTTA challenge, existing methods typically adopt a filtering-then-adaptation paradigm, i.e., Figure 1(a): thresholding OOD scores to generate pseudo-labels, and then supervising the model via discriminative losses. State-of-theart (SOTA) approaches like AdaND [Cao et al., 2025] and Open-IRT [Peng et al., 2025] enhance this pipeline by leveraging vision-language models' (VLMs) zero-shot generalization [Chao et al., 2025] to derive OOD scores. However, their thresholding schemes rely on idealized priors (e.g., balanced bimodality) that fail under complex real-world distributions, leading to erroneous pseudo-labeling. Figure 1(b) reveals that AdaND fails to maintain robust OOD filtering across diverse evaluation datasets. Crucially, this filtering failure triggers a cascading error accumulation. Instead of rejecting noise, the model is compelled to overfit to the mislabeled samples during updates, distorting learned decision boundaries [Wang et al., 2019; Zhang et al., 2023]. This results in the catastrophic collapse observed in Figure 1(c): performance degrades severely after an initial transient peak as noise dominates the adaptation process, highlighting the inability to sustain robustness.

Motivated by these limitations, we propose a novel OSTTA framework named Noise-Immune Self-Purification (NISP) that implements self-purification of OOD samples to safeguard adaptation quality. First, we propose a dual-cluster Bayesian Gaussian mixture model (BGMM) with posteriorrisk thresholding to extract high-fidelity pseudo-labels. As illustrated in Figure 1(b), our strategy yields superior pseudo-label accuracy and maintains exceptional robustness regardless of the dataset variations. Second, we develop a noise-resilient adaptation governed by consensusdiscrepancy constraints to refine coarse pseudo-labels. The synergistic regularization effectively mitigates noise interference from pseudo-labels and prevents error accumulation. Capitalizing on this adaptation dynamic, we derive the Jaccard Consistency Score (JCS) to enable precise test-time OOD discrimination. As evidenced in Figure 1(c), NISP exhibits a sustained performance trajectory: ascending and then stabilizing, validating its efficacy in noise-robust adaptation. Furthermore, theoretical analysis substantiates that the collaborative optimization of consensus and discrepancy objectives can significantly reduce the deleterious impact of pseudo-label noise, thereby enhancing the robustness of the model in open-world scenarios.

The contributions of this work are summarized as follows:

• Technically, we propose NISP, a coarse-to-fine selfpurification framework that leverages VLMs to safeguard adaptation against open-set noise.
• Theoretically, we prove that our optimization objective provides an upper-bound on Bayes risk under noisy supervision, ensuring rigorous denoising adaptation.
• Empirically, NISP attains state-of-the-art (SOTA) performance across multiple OSTTA benchmarks, validating its efficacy in open-world settings.

2 RelatedWork

2.1 Open-Set Test-Time Adaptation

Test-Time Adaptation (TTA) aims to address performance degradation caused by the distribution shifts. Traditional TTA methods adapt models to target domains via self-supervised objectives, such as entropy minimization [Niu et al., 2022; Wu et al., 2025], or updating internal normalization statistics [Schneider et al., 2020; Yuan et al., 2023; Lim et al., 2023]. Recent studies have exploited the zero-shot generalization of VLMs for TTA, like TPT[Shu et al., 2022], DiffTPT [Feng et al., 2023], and TDA [Karmanov et al., 2024], which facilitate online calibration via prompt optimization or key-value caching. However, these methods overlook the presence of OOD samples in open-world deployments, which gives rise to the open-set TTA (OSTTA) challenge. While approaches such as SoTTA [Gong et al., 2023], UniEnt [Gao et al., 2024] and AEO [Dong et al., 2025] incorporate OOD detection into TTA, they do not leverage the VLMs for adaptation. Prior work, such as AdaND [Cao et al., 2025], utilizes VLMs to generate pseudo-labels, updating only the OOD detector at test time with BCE loss. Yet, without addressing the inherent noise in pseudo-labels, the model could overfit to erroneous signals during adaptation.

2.2 Vision-Language Models

Pretrained vision-language models (VLMs) typically consist of image and text encoders, which are jointly trained on billions of image-text pairs to align cross-modal representations. For instance, ALIGN [Jia et al., 2021] leverages a trillion-scale noisy web dataset to improve alignment precision and scalability. CLIP [Radford et al., 2021] establishes a standardized contrastive pretraining framework using templated text prompts and four million image-text pairs, setting a new paradigm for zero-shot transfer tasks. BLIP [Li et al., 2023a] introduces cross-modal attention into the contrastive loss and combines generative and discriminative objectives, enhancing fine-grained semantic understanding and generation. EVA-CLIP [Sun et al., 2023] augments the CLIP architecture with a more powerful visual backbone and richer pretraining goals, employing LAMB [You et al., 2020] optimization to reduce computational overhead while boosting downstream task performance. Empowered by large-scale pretraining data and contrastive learning, VLMs exhibit exceptional zero-shot generalization capacities.

3 ProposedMethod

3.1 Preliminaries

In OSTTA, we consider an unlabeled test stream 𝒳 = {𝑥𝑖}𝑁𝑖=1 , where each sample is from an underlying distribution with an unobserved ground-truth label 𝑦𝑖. The universal label space 𝒴 comprises two disjoint subsets: the ID categories 𝒴id and OOD categories 𝒴ood, such that 𝒴 = 𝒴id ∪𝒴ood and 𝒴id ∩𝒴ood = ∅. During the adaptation phase, the model has access solely to the unlabeled images in 𝒳 and the ID class name set 𝒴id = {𝑦1, 𝑦2, . . . , 𝑦𝐶}. Our objective is to leverage the VLM's zero-shot generalization to assign each 𝑥𝑖 an OOD score, thereby facilitating precise ID/OOD discrimination. Standard VLMs utilize a dual-encoder architecture consisting of an image encoder 𝑓(·) and a text encoder 𝑔(·). For an image 𝑥𝑖 and a set of class-specific prompts {𝑡𝑘}𝐶𝑘=1 (e.g., "a photo of a 𝑦𝑘"), we obtain the corresponding embeddings as 𝑣𝑖 = 𝑓(𝑥𝑖) and 𝑢𝑘 = 𝑔(𝑡𝑘).

3.2 Unsupervised Coarse Pseudo-Labeling

With the extracted embeddings, we adopt the Maximum Concept Matching (MCM) score [Ming et al., 2022] to assign the coarse OOD score to each test sample:

where sim(·, ·) denotes the cosine similarity and 𝜏 is the temperature hyperparameter. Then, we propose an unsupervised strategy to partition the test data into coarse pseudo-ID/OOD subsets. This is achieved by modeling the MCM score distribution as a mixture of Gaussian components, identifying their semantics (ID vs. OOD) and minimizing the decision risk.

To capture the complex data structure without prespecifying the number of components, we introduce BGMM

with a Dirichlet Process prior [Blei and Jordan, 2004] fit the score distribution, defined as an infinite mixture:

where Θ = {𝜋𝑚, 𝜇𝑚, 𝜎𝑚}∞𝑚=1 comprises the mixture weights 𝜋𝑚, the component means 𝜇𝑚, and variances 𝜎𝑚. The weights 𝜋𝑚 follow a stick-breaking process. To ensure computational tractability, we approximate Eq. (2) using a variational truncation bound 𝑀. Parameters Θ are estimated via variational inference by maximizing the evidence lower bound, where negligible components with weights below 𝜖𝑤 (empirically set to 0.001) are pruned.

Once the density is modeled, the next objective is to distinguish ID-dominant components from OOD-dominant ones. Each Gaussian component is characterized by its statistical moments, represented as a bivariate feature vector f𝑚 = (𝜇𝑚, 𝜎𝑚) ∈ R2 . We apply K-means clustering [Likas et al., 2003] on these vectors to categorize the component index set ℳ = {1, . . . ,𝑀} into mutually exclusive ID subset 𝒞id and OOD subset 𝒞ood. The clustering objective minimizes:

where cid, cood ∈ R2 denote the cluster centroids.

With the components identified, we determine the optimal decision boundary by leveraging the component-wise posterior probabilities. The posterior responsibility 𝛾𝑚(𝑆) of component 𝑚 for a score 𝑆 is calculated via Bayesian inversion:

Subsequently, given the score sequence {𝑆(𝑖)}𝑁𝑖=1 sorted in ascending order, we seek a cut-off index 𝑑 that minimizes the cumulative misclassification risk ℛ(𝑑), balancing the error of rejecting ID samples against accepting OOD samples:

The optimal index 𝑑* = argmin𝑑ℛ(𝑑) is efficiently computed via prefix sums [Martinez and Raschia, 2024]. Finally, the decision threshold is set to the midpoint 𝜉 = 12 (𝑆(𝑑* ) + 𝑆(𝑑*+1)), yielding binary pseudo-labels:

where ˆ 𝑦𝑖 = 0 and ˆ 𝑦𝑖 = 1 denote pseudo-ID and pseudo-OOD labels, respectively.

3.3 Noise-Immune Fine-Grained Adaptation

At each adaptation step, let V= {𝑣𝑖}𝐵𝑖=1 denote the image embeddings set of current batch with pseudo-labels {ˆ 𝑦𝑖}𝐵𝑖=1 . A binary mask partitions indices into ID set ℐ = {𝑖 : ˆ 𝑦𝑖 = 0} and OOD set 𝒪 = {𝑖 : ˆ 𝑦𝑖 = 1}. Subsequently, we introduce a lightweight adapter to perform noise-resilient adaptation, thereby refining the coarse pseudo-labels. The adapter consists of 𝐻 parallel prediction heads {𝜑ℎ}𝐻ℎ=1 , where each 𝜑ℎ is an independent two-layer MLP with non-linear activation.

The adapter is optimized via a dual-path objective that separately handles pseudo-ID and pseudo-OOD samples. To promote predictive consistency among classifier heads on pseudo-ID samples, we minimize the pairwise mean squared error [Xue et al., 2013] across all unique head pairs:

• (a) AdaNDDensity (b) NISP Density

(c) AUROC Curve

Figure 3: OOD Detection Performance.

where ℋ = {1, . . . ,𝐻} denotes the set of head indices, |ℐ| is the number of pseudo-ID samples, and (︀𝐻2 )︀ enumerates all unordered head pairs. This objective reduces prediction variance across heads on pseudo-ID samples, yielding more consistent and robust outputs.

In contrast, pseudo-OOD samples are expected to exhibit prediction disagreement. To achieve this, a hinge-activated margin is applied to maximize their pairwise divergence:

(8)

where [·]+ = max(0, ·) is the hinge operator and 𝛿 > 0 denotes a predefined divergence margin (set to 0.1 in all experiments). Gradient flow is selectively activated only when head disagreement falls below the margin, allowing the model to preserve uncertainty on OOD samples. The overall adaptation objective is defined to minimize a weighted sum:

where the coefficient 𝜆 balances the two terms. As shown in Lemma 1, the discrepancy and consensus losses jointly suppress unreliable gradients and reduce gradient variance, resulting in a tighter bound in Theorem 1.

3.4 Inference Phase

Inspired by this adaptation process, we adopt a novel OOD scoring metric, termed the Jaccard Consistency Score (JCS), to enhance the separation between ID and OOD samples. Let 𝒜 denote a set of percentile thresholds (typically set to {5, 10, . . . , 55}). For any granularity 𝛼 ∈ 𝒜, the corresponding subset size is defined as 𝐾 = ⌈𝛼 · 𝐶⌉. Given an image embedding 𝑣𝑖, the set of top-𝐾 classes predicted by head ℎ is defined as:

where argsort(𝐾) (·) returns the indices of the top-𝐾 highest confidence classes, and |𝑌ℎ(𝑣𝑖)| = 𝐾.

We compute the aggregates pairwise Jaccard similarity at this granularity 𝛼 across all head pairs (𝑝, 𝑞):

The final JCS score is obtained by identifying the maximum consensus achieved across the spectrum of granularities:

This metric provides a compact measure of cross-head prediction consistency. Higher JCS values indicate strong agreement among heads, which is characteristic of ID samples, while lower values reflect prediction dispersion typical of OOD samples. Empirical effectiveness of this metric is demonstrated in Figure 3.

3.5 Theoretical Understanding

We theoretically justify the effectiveness of our method in suppressing overfitting to noisy pseudo-labels under test-time adaptation. Under the standard assumptions of 𝐿-smoothness [Seccia et al., 2025] and the 𝜇-PL condition [Plassier et al., 2025], we analyze the convergence behavior in the presence of pseudo-label noise.

Theorem 1 (Excess Risk Bound). Let 𝜃𝑇 be the model after 𝑇 steps of adaptation with learning rate 𝜂 ≤ 1/𝐿, and let ¯𝜎2 eff denote the average gradient variance over noisy samples. Then the expected excess risk satisfies:

The key to controlling generalization error lies in reducing ¯𝜎2 eff . Our method achieves this through multi-head regularization, which adaptively modulates the gradient contribution of noisy samples.

Lemma 1 (Variance-Controlled Gradient Suppression). For any noisy sample 𝑥𝑖, the per-sample gradient satisfies:

where 𝐶 > 0 is a constant and Varℎ[·] denotes inter-head variance. Consequently,

This behavior arises because the discrepancy loss is inactive for incorrectly assigned pseudo-OOD samples, yielding zero gradients [Li et al., 2025], whereas the consensus loss is activated only when predictions exhibit sufficient disagreement. By contrast, conventional pseudo-labeling applies uniform updates to all samples irrespective of pseudo-label reliability, resulting in elevated gradient variance.

4 Experiments

4.1 Experimental Setup

Benchmark Datasets. Following prior work [Cao et al., 2025], we utilize CIFAR-10/100 [Krizhevsky et al., 2009], CUB-200-2011 [Wah et al., 2011], Food-101 [Bossard et al., 2014], Stanford-Cars [Krause et al., 2013], ImageNet [Deng et al., 2009] and its variants (ImageNet-V2 [Recht et al., 2019], ImageNet-R [Hendrycks et al., 2021], ImageNetSketch [Wang et al., 2019]) as ID datasets. The OOD datasets include SVHN [Netzer et al., 2011], LSUN [Yu et al., 2015], iNaturalist [Van Horn et al., 2018], SUN [Xiao et al., 2010], Places [Zhou et al., 2017], Texture [Cimpoi et al., 2014].

Evaluation Metric. We adopt the harmonic mean accuracy (𝐴𝐶𝐶𝐻) as the metric for OSTTA evaluation in VLMs, which is computed over the classification accuracies of ID and OOD samples. For OOD detection performance, we employ FPR95 and AUC for evaluation [Cao et al., 2025; Ming et al., 2022]. FPR95 refers to the false positive rate of OOD samples when the true positive rate of ID samples is fixed at 95%, and AUC is a widely used metric that quantifies the overall separability between ID and OOD samples, based on the receiver operating characteristic (ROC) curve.

Compared Methods and Setups. We evaluate ZSCLIP [Radford et al., 2021], Tent [Wang et al., 2021], SoTTA [Gong et al., 2023], TDA [Karmanov et al., 2024], TPT [Shu et al., 2022] and AdaND [Cao et al., 2025] in the benchmarks. In our main experiments, we utilize CLIP [Radford et al., 2021] with a ViT-B/16 [Dosovitskiy, 2020] backbone. Moreover, we report our method's performance across multiple pretrained VLM backbones in Section 4.3. The batch size is fixed at 128, ensuring each sample is processed exactly once (one-pass adaptation). For the initial 10 steps, we retain predictions from the frozen VLM and derive OOD scores via the MCM detector [Ming et al., 2022]. The model is updated using the Adam [Kingma, 2014] optimizer with a learning rate of 0.005 and no weight decay. While the loss weight 𝜆 is tuned via grid search, other hyperparameters remain consistent for datasets with over 50 classes: the truncation bound 𝑀 = 6, adapter heads 𝐻 = 5, and a hidden dimension of 128.

4.2 Main Results

Tables 1 and 2 present a comparative analysis of OSTTA performance across diverse benchmarks, yielding four pivotal insights: (1) Both standard (e.g., Tent) and VLM-specific TTA methods (e.g., TPT, TDA) struggle to consistently surpass the frozen baseline (ZS-CLIP). Notably, under severe distribution shifts, these approaches often succumb to negative adaptation, yielding lower performance than the non-adapted model. Even specialized OSTTA methods like SoTTA demonstrate only marginal or inconsistent gains across benchmarks. (2) AdaND substantially improves VLMs' performance through decoupling ID classification and OOD detection, establishing a strong baseline in the OSTTA setting. (3) NISP achieves the highest 𝐴𝐶𝐶𝐻 and lowest FPR95 across the majority of benchmarks. Specifically on ImageNet, NISP delivers a remarkable 𝐴𝐶𝐶𝐻 improvement of 3.40% and a FPR95 reduction of 11.90% relative to AdaND. (4) NISP's advantage is most pronounced in challenging, low-accuracy regimes (e.g., ImageNet-K and CUB-200-2011). Conversely, on saturated benchmarks (e.g., Food-101) where AdaND already performs well, NISP maintains competitive parity. This dichotomy underscores NISP's critical strength in mitigating noise accumulation, a claim further substantiated in Section 4.4.

4.3 Ablation Study

Effectiveness of Each Module. Table 3 presents an ablation study on ImageNet, evaluating the individual contributions of the thresholding strategy and the noise-resilient adaptation module. Excluding the thresholding strategy leads to a noticeable decline in pseudo-label quality, resulting in overall performance degradation. More significantly, ablating the noise-resilient module triggers a 12.03% surge in FPR95 and a 3.43% drop in 𝐴𝐶𝐶𝐻, primarily due to overfitting on noisy pseudo-labels. These findings highlight the critical role of each component in maintaining adaptation robustness. The full model achieves optimal average performance by jointly leveraging both mechanisms in a complementary manner.

Sensitivity Analysis of Key Hyperparameters. Figure 4 visualizes a sensitivity analysisof key hyperparameters on ImageNet. The loss weight 𝜆 demonstrates robust adaptation efficacy, striking an optimal trade-off between pseudo-ID consistency and pseudo-OOD discrepancy at approximately 0.3. Furthermore, the model exhibits remarkable insensitivity to the number of adapter heads 𝐻. The performance fluctuations are negligible across varying configurations, where our approach consistently outperforms baselines, validating the structural stability of the proposed framework.

Impact of Different VLMs. Table 4 scrutinizes the architectural scalability of NISP across diverse VLM backbones. While EVACLIP establishes a superior baseline due to advanced alignment, incorporating NISP yields universal performance gains regardless of the backbone's capacity. Notably, NISP empowers EVACLIP to achieve peak performance (93.50% Avg AUC and 84.18% Avg 𝐴𝐶𝐶𝐻), significantly surpassing its unadapted counterpart. This confirms that the performance gains are independent of specific architectural inductive biases. These results attest that our framework is architecturally agnostic: it effectively revitalizes weaker models (e.g., BLIP) while further amplifying the capabilities of advanced foundations.

4.4 Further Analysis

Thresholding under Distribution Shift. Table 5 presents a comparative analysis of pseudo-label accuracy achieved by different thresholding strategies on the ImageNet. MEAN adopts the per-batch average OOD score as the threshold, while AdaND selects the threshold by minimizing intracluster score variance. In contrast, NISP achieves the highest pseudo-label accuracy across all datasets. Figure 5 illustrates threshold placement under various data distributions, including settings with variance imbalance, skewness, and number imbalance. NISP consistently locates thresholds near the posterior density intersections to minimize classification risk, demonstrating superior adaptability and reliable pseudolabeling under complex conditions.

Noisy Overfitting Analysis. To elucidate the performance degradation observed in AdaND during late-stage adaptation, we analyze the dynamics of pseudo-label optimization. As depicted in Figure 6, AdaND initially benefits from clean samples, it progressively succumbs to noise memorization, overfitting incorrect pseudo-labels. This leads to a distinct inverse correlation: rising accuracy on noisy samples at the expense of clean accuracy. In contrast, our method maintains a sustained focus on clean samples throughout adaptation, resulting in stable improvements in clean accuracy while suppressing the influence of noise-induced supervision.

Resource Analysis. Table 6 reports computational efficiency analysis on ImageNet. Even with a multi-head configuration comprising 𝐻 = 5 heads and a hidden dimension of 128, our method maintains inference runtime and memory usage comparable to frozen VLM baseline (ZS-CLIP) and AdaND. This is attributed to the lightweight nature of our adapter architecture (a simple two-layer MLP). Unlike Tent, which requires memory-intensive full-graph backpropagation, NISP isolates gradients within the terminal adapter to prevent storing massive intermediate activations. Consequently, our method allows for efficient online adaptation with significantly lower resource demands than Tent.

Gain Analysis. Experiments reveal heterogeneous performance gains across benchmarks: while reducing FPR95 by 11.90% on ImageNet, our method achieves marginal improvements on other datasets with already high pseudo-label fidelity like Food-101. Figure 7 demonstrates a moderate negative correlation (𝜌 = -0.586, 𝑝 < 0.001) between FPR95 reduction gains and base VLM pseudo-label accuracy across ID/OOD pairs. This stems from AdaND's limited noisy overfitting at high pseudo-label accuracy regimes (e.g., 87.34% on Food-101/SUN), reducing our method's relative advantage. Whereas at low accuracy (e.g., 63.58% on ImageNet/SUN), our method delivers 12.75% FPR95 reduction, validating its noise robustness.

5 Conclusion

In this paper, we propose a novel OSTTA approach named NISP that achieves unsupervised OOD self-purification via a coarse-to-fine noise-resilient adaptation paradigm. Technically, NISP introduces (1) a dual-cluster BGMM posteriorrisk thresholding to generate high-fidelity pseudo-labels based on VLM-derived OOD scores, and (2) a fine-grained noise-resilient adapter updates under consensus-discrepancy constraints to refine pseudo-labels and suppress error propagation. Rigorous theoretical analysis demonstrates that NISP upper-bounds the Bayes risk in noisy environments, while extensive experiments across multiple OSTTA benchmarks confirm that NISP outperforms SOTA methods, demonstrating both theoretical soundness and practical efficacy.

Acknowledgments

This work is supported by the NSFC (62276131), Natural Science Foundation of Jiangsu Province of China under Grant (BK20240081).