Learning Well-Structured Logits: Leveraging Vision–Language Complementarity for Open-World Test-Time Adaptation

Abstract

Open-world test-time adaptation (OWTTA) is increasingly studied for its ability to adapt models at inference time in the presence of both domain discrepancy and semantic variance. Existing methods typically rely on either discriminative models or vision-language models (VLMs) alone, leaving their complementarity in OWTTA underexplored. In this paper, we propose a general framework that explicitly leverages the complementary strengths of discriminative models and VLMs to enable robust open-world test-time adaptation. We first define well-structured logits for OWTTA and provide a theoretical analysis that reveals the complementary properties of logits from discriminative models and VLMs. Building on a unified formulation, we then decompose OWTTA into two coupled modules: confidence-calibrated filtering (CCF), which provides an estimate of in-distribution membership, and semantic complementarity adaptation (SCA), which gives the refined predictions through complementary logit fusion. Extensive experiments across multiple benchmarks empirically confirm the logit complementarity between discriminative models and VLMs, and show that our method effectively leverages it to consistently improve performance.