---
title: "Open-Set Test-Time Adaptation in VLMs via Noise-Immune Self-Purification"
authors:
  - "Mingxu Feng"
  - "Fengqiang Wan"
  - "Yang Yang"
doi: "10.24963/ijcai.2026/477"
url: "https://ijcai.org/proceedings/2026/477"
pdf: "https://ijcai.org/proceedings/2026/0477.pdf"
venue: "Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence"
year: 2026
pages: "4286-4294"
track: "Main Track"
keywords:
  - "Machine Learning: Unsupervised learning"
  - "Machine Learning: Open-World/Open-Set/OOD Learning"
---

# Open-Set Test-Time Adaptation in VLMs via Noise-Immune Self-Purification

Mingxu Feng, Fengqiang Wan, Yang Yang

*Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence*, pages 4286-4294. DOI: [10.24963/ijcai.2026/477](https://doi.org/10.24963/ijcai.2026/477)

## Abstract

Unlike test-time adaptation (TTA), open-set TTA (OSTTA) aims to robustly adapt to in-distribution (ID) domain shifts while suppressing the detrimental impact of out-of-distribution (OOD) samples encountered at test time. To this end, we propose a novel OSTTA approach named Noise-Immune Self-Purification (NISP). By exploiting the zero-shot priors of pre-trained vision-language models (VLMs), NISP advances from coarse pseudo-labeling to fine-grained, noise-resilient adaptation. Technically, we first introduce a dual-cluster Bayesian Gaussian mixture model to fit VLMs-derived scores, achieving coarse pseudo-ID/OOD separation via posterior-risk thresholding. Subsequently, NISP constructs a noise-immune fine-grained adaptation where the adapter enforces consensus-discrepancy constraints to refine coarse pseudo-labels and suppress noise propagation. We then devise the Jaccard Consistency Score for OOD discrimination. Overall the coarse-to-fine pipeline enables rigorous self-purification and robust online adaptation. Theoretically we show that consensus-discrepancy losses mitigate the deleterious effects of noise. Empirically, NISP achieves state-of-the-art results across multiple OSTTA benchmarks, validating its efficacy. The code is available at https://github.com/njustkmg/IJCAI26-NISP.