Learning Well-Structured Logits: Leveraging Vision–Language Complementarity for Open-World Test-Time Adaptation

1 Introduction

Deep learning has witnessed rapid advances and demonstrated strong performance in diverse applications. These achievements are typically realized under the assumption that the training and test data follow the same i.i.d. distribution [Gong et al., 2022]. In open-world scenarios, however, this assumption is often violated as domain discrepancy [Qu et al., 2024] and semantic variance [Zhao and Lee, 2024] frequently coexist, leading to degraded performance and unreliable predictions in practice. For instance, in medical image analysis, domain discrepancy may arise from differences in imaging equipment, while semantic variance can occur when the testing data include novel or unseen disease categories. This dual challenge implies that models should not only adapt to domain shifts among in-distribution (ID) samples but also mitigate the negative impact introduced by outof-distribution (OOD) samples. To address this challenge, open-world test-time adaption (OWTTA) [Li et al., 2023] has been proposed as a promising paradigm, enabling models to adapt to the target domain without requiring label data or source statistics.

∗Corresponding author

Conventional OWTTA methods [Gao et al., 2024] typically rely on discriminative models, which can learn task-specific representations but remain constrained by their limited adaptability. Recently, vision-language models (VLMs) [Radford et al., 2021], pre-trained on large-scale and diverse data, have been widely recognized for their zero-shot ability. By jointly modeling vision and language, VLMs learn powerful representations that capture rich semantics and establish effective alignment between visual inputs and textual class names. This property inherently mitigates the impact of domain discrepancy and makes VLMs a strong backbone for TTA [Shu et al., 2022; Karmanov et al., 2024], providing richer semantic cues that enhance generalization across domains. Despite these advantages, the role of VLMs in open-world TTA remains largely underexplored. Existing methods typically rely on either a discriminative model or a VLM alone, without explicitly characterizing or effectively exploiting their implicit logit complementarity.

In this paper, we investigate how to uncover and exploit the implicit complementarity between discriminative models and vision-language models in OWTTA. We begin from a theoretical perspective by introducing the notion of wellstructured logits, characterized by semantic-shift separability, inter-class discriminativity and intra-class consistency. This analysis reveals a complementary regime in which discriminative models and VLMs exhibit distinct yet compatible logit characteristics under open-world scenario, and it further motivates propositions showing that a principled logit mixing can yield well-structured logits. Guided by this insight, we propose vision-language complementarity for openworld test-time adaptation (VLCO). Specifically, we introduce a general framework for OWTTA, which contains coupled modules: confidence-calibrated filtering (CCF) and semantic complementarity adaptation (SCA). CCF provides a more reliable estimate of in-distribution membership for each sample by calibrating and fusing confidence signals from a discriminative model and a vision-language model, while SCA provides more accurate label prediction for reliable samples by integrating discriminative and vision-language cues through complementary logit fusion. As a result, VLCO operationalizes the identified complementarity and learns wellstructured logits for robust open-world adaptation. The contributions of this paper are summarized as follows:

• We introduce the notion of well-structured logits for OWTTA by defining semantic-shift separability, interclass discriminativity and intra-class consistency, and theoretically show that mixing complementary logits yields well-structured predictions.
• We propose a general framework for OWTTA that operationalizes vision-language complementarity by tightly coupling a discriminative model and a VLM through two modules, confidence-calibrated filtering (CCF) and semantic complementarity adaptation (SCA).
• Extensive experiments across multiple benchmarks empirically confirm the existence of complementary logit behaviors between discriminative models and VLMs, and demonstrate that our method yields well-structured logits and consistently improves open-world test-time adaptation performance.

2 RelatedWork

In this section, we review representative prior work relevant to our problem setting. To clarify distinctions among closely related adaptation paradigms, Table 1 provides an illustrative comparison of common settings.

Test-Time Adaptation. Test-time adaptation (TTA) [Liang et al., 2025; Su et al., 2024a; Su et al., 2022] adapts a sourcetrained model to target domains using only unlabeled test data. Representative approaches include entropy minimization [Liang et al., 2020; Bar et al., 2024; Zhang et al., 2025; Wu et al., 2025], distribution alignment [Wang et al., 2024b;

Zhang et al., 2024; Su et al., 2024b], and continual adaptation [Wang et al., 2022; Niu et al., 2022; Han et al., 2025]. Most TTA methods operate under closed-set assumptions, requiring identical label spaces between source and target domains.

Open-Set Domain Adaptation. Open-set domain adaptation (OSDA) [Busto and Gall, 2017; Pham et al., 2025; Choe et al., 2024; Wan et al., 2024] relaxes the closed-set assumption by allowing novel categories absent from the source and typically assumes access to the full target batch. Open-set source-free domain adaptation (OS-SFDA) [Yu et al., 2025; Liu et al., 2025; Wan et al., 2024] further removes source data access, relying only on the source model and unlabeled target data. However, both settings usually assume offline optimization over the target domain, making them unsuitable for single-pass open-world test-time adaptation.

Open-World Test-Time Adaptation. Open-world testtime adaptation (OWTTA) tackles domain shift and semantic variation under single-pass test-time constraints. Unlike OOD detection [Yang et al., 2024], OWTTA requires both unknown-sample separation and reliable in-domain classification. Existing methods usually combine OOD filtering with adaptation via distribution alignment [Li et al., 2023] or entropy-based criteria [Gao et al., 2024; Gong et al., 2023]. These methods largely depend on task-specific discriminative representations, which limits robustness and generalization under open-world conditions.

Pre-trained Vision-Language Models. Large-scale vision-laguage models (VLMs), such as CLIP [Radford et al., 2021], ALIGN [Jia et al., 2021], and GroupViT [Xu et al., 2022], are trained on massive image-text pairs via contrastive learning [Chen et al., 2020] and exhibit strong generalization across diverse downstream tasks. Recent studies have explored leveraging VLMs for test-time adaptation [Osowiechi et al., 2024; Wang et al., 2024a; Phan et al., 2024], indicating that VLMs constitute a promising semantic foundation, while leaving open how to systematically exploit their semantic representations for robust adaptation.

3 Leveraging Logit Complementarity for Open-World Test-Time Adaptation

OWTTA aims to improve prediction accuracy under distribution shifts without access to ground-truth labels, while handling unknown or noisy samples. In this setting, naive adaptation strategies that rely on instance-level predictions are highly susceptible to error accumulation and drift. Effective OWTTA therefore requires reliable and stable supervisory signals that do not depend on true labels.

We argue that the structure of model logits provides a useful lens for understanding and guiding the design of such supervision. Since test-time adaptation operates directly on model outputs, the geometric properties of logits critically influence whether adaptation remains stable under open-world distribution shifts. In particular, we characterize desirable logit structures through the following properties.

Definition 1 (Well-structured logits). Let ℓi ∈ RK denote the pre-softmax logits vector produced by a model for input xi. We say the logits are well-structured if they exhibit:

1. Semantic-shift separability: logits under semantic shifts (OOD) exhibit distinguishable structures from those of the original semantics (ID);
2. Inter-class discriminativity: the top-ranked logit is well separated from the remaining logits, providing clear separation between candidate classes;
3. Intra-class consistency: logits corresponding to samples from the same semantic class exhibit low variation, forming compact class-wise structures in the logit space.

Semantic-shift separability enables the model to suppress unreliable updates induced by semantically different inputs. Inter-class discriminativity mitigates misclassification of noisy samples, while intra-class consistency stabilizes adaptation by reducing the impact of unreliable instance-level predictions. Together, these properties characterize a wellstructured logit space that supports reliable supervision in OWTTA.

Definition 2 (Semantic-shift separability). Let s denote a score derived from logits. Semantic-shift separability refers to the ability of s to distinguish in-distribution inputs from out-of-distribution ones, where stronger separability implies that they can be more reliably partitioned based on s.

Definition 3 (Inter-class discriminativity). Let ℓ(1) i ≥ ℓ(2) i ≥ · · · ≥ ℓ(K) i denote the sorted logits. The inter-class discriminativity of dataset D can be measured by the average margin

where ϕi = ℓ(1) i - ℓ(2) i denotes the instance-wise margin and larger Φ indicates a clearer decision margin.

Definition 4 (Intra-class consistency). For each class k, let Dk = {xi|yi = k} denote the set of samples in class k. The intra-class consistency of D can be defined as

where Ψk denotes the class-wise consistency and ζk = 1 |Dk | ∑xi∈Dk ℓi is the class-wise mean logit. A smaller Ψ indicates a more compact and class-consistent logit structure.

Let sD and sF denote the confidence score produced by two models, where D indexes a discriminative model and F indexes a vision-laguage model. Proposition 1 shows that a linear fusion of the confidence scores yields better semanticshift separability, thereby reducing the influence of out-ofdistribution samples during adaptation.

Proposition 1 (Optimal Fusion Improves Normalized Separation). Let β ∈ {0, 1} indicate whether a sample is ID (β = 1) or OOD (β = 0). Assume the confidence scores sD and sF are conditionally independent given β, and each follows a two-component Gaussian mixture distribution with shared variances, namely,

where σ2 s∗ is identical for both β = 0 and β = 1.

Let ∆s∗ = µs∗ ,1 - µs∗ ,0 > 0 ∀∗ ∈ {D, F} and define the fused score sM := sD+αsF 1+α , ∀α > 0. Then the fused score can achieve a strictly larger normalized separation, i.e.,

Let ℓDi , ℓFi ∈ RK denote the pre-softmax logits for input xi. By examining the logit structures of different model families (See Table 6 & Table 7), we observe a clear complementarity: the discriminative model exhibits high inter-class discriminativity but low intra-class consistency, whereas CLIP logits are less discriminative but substantially more consistent. Motivated by this complementarity, we combine logits from two models to derive reliable pseudo labels from better-structured fused logits.

We define their linear combination as ℓMi = λℓDi +(1-λ)ℓFi ∀λ ∈ (0, 1). Next, we establish that the mixed logits ℓMi preserves both a guaranteed decision margin (Proposition 2) and bounded intra-class variation (Proposition 3), thereby enabling stable test-time adaptation and improved accuracy.

Proposition 2 (Lower-Bounded Decision Margin under Logit Mixing). Let ℓDi , ℓFi , ℓMi ∈ RK denote the pre-softmax logits vectors for input xi, as defined above. Let ℓ(1) i ≥ ℓ(2) i ≥ · · · ≥ ℓ(K) i denote the sorted logits. Assume there exist positive constants φD, φF > 0 such that, ∀xi and ∀k ̸= 1, ℓD,(1) i - ℓD,(k) i ≥ φD for the discriminative model, and |ℓF,(j) i - ℓF,(k) i | ≤ φF ∀j, k for the vision-language model. Letting φM = 1N ∑Ni=1 (ℓM,(1) i - ℓM,(2) i ), we have φM > φF when φD > (2-λ) λ φF .

Proposition 3 (Upper-Bounded Variation under Logit Mixing). Let ℓDi , ℓFi , ℓMi ∈ RK denote the pre-softmax logits vectors for input xi, as defined above.

Following Definition 4, the intra-class consistency of the mixed logits satisfies:

Taken together, Proposition 2 and Proposition 3 show that mixing discriminative and CLIP logits preserves discriminative margins and enhances intra-class consistency. Motivated by this complementarity, we develop a framework that explicitly leverages it for OWTTA.

4 Methodology

In this section, we propose vision-language complementarity for open-world test-time adaptation (VLCO). It tackles openworld test-time adaptation by jointly leveraging a discriminative model and a vision-language model and explicitly exploiting their complementary properties.

We formulate a general framework for OWTTA by modeling the distribution p(y|x; f D, f F ) induced by the discriminative model f D and the VLM f F . To capture the semantic shift between samples, we introduce a binary latent variable β ∈ {0, 1} indicating whether a sample is ID (β = 1) or OOD (β = 0). The label distribution can then be decomposed as:

The decomposition in Eq. (1) yields two coupled objectives: (i) Confidence-Calibrated Filtering (CCF), which estimates the ID/OOD distribution p(β|x; f D, f F ), and (ii) Semantic Complementarity Adaptation (SCA), which refines the label distribution p(y|x, β; f D, f F ) for the ID samples.

Specifically, we define f D = {fθ, w, b}, where fθ denotes the encoder and (w, b) parameterize the linear classifier, and f F = {fI, fT}, where fI denotes the VLM image encoder and fT denotes the VLM text encoder. In the following, we present the detials of CCF and SCA.

4.1 Confidence-Calibrated Filtering

The CCF module estimates the reliability of each test sample by calibrating and fusing confidence signals from the discriminative model and the vision-language model.

Specifically, the confidence score of discriminative model is defined as:

where fθ(x) ∈ Rd is the feature embedding and c = [c1, . . . , cK]⊤ ∈ RK×d denotes the class prototypes, initialized by the classifier weights w.

Correspondingly, the confidence score of vision-language model is defined as:

where {ej}Kj=1 are the class-specific text prompts.

To model the semantic-shift separability, we assume that the confidence score s is drawn from a two-component Gaussian mixture with shared variance:

where πυ denotes the mixing coefficients. Consequently, for υ ∈ {0, 1}, the distribution p(β = υ|x; f ∗ ) is equivalent to p(β = υ|s∗ ):

Based on the connection between k-means and mixtures of Gaussians [Kulis and Jordan, 2012], when σ2 s∗ → 0, we can have the following k-means based clustering loss:

where S + = {i|βi = 1} and S - = {i|βi = 0} denote the predicted ID and OOD clusters, respectively, and the corresponding cluster means are µs∗ ,1 = 1 |S+| ∑i∈S+ s∗i and µs∗ ,0 = 1 |S-| ∑i∈S- s∗i . Therefore, CCF gives a hard estimation on β via two-cluster k-means clustering on the confidence scores s∗ , yielding a binary partition of samples into S + and S -.

Confidence-Calibrated Filtering. To improve OOD filtering, we propose the confidence-calibrated filtering module p(β|x; f D, f F ), which augments the discriminative model's confidence with semantic cues from vision-language model.

Since the two confidence scores, sD and sF , are not directly comparable in scale, we transform them into calibrated evidences via a weighted average:

According to Proposition 1, an appropriate choice of α, i.e., α = ∆sFσ2 sD /∆sD σ2 sF yields a strictly stronger filter than either component score alone. We then substitute s in Eq. (3) with sE for a better estimation of β. This improvement is also reflected in our visualizations (See Fig. 1), where CCF produces a cleaner separation.

4.2 Semantic Complementarity Adaptation

In this subsection, we introduce the semantic complementarity adaptation (SCA) module, which exploits complementary information from the discriminative model and the VLM to facilitate adaptation to the target domain.

For OOD samples (β = 0), we define p(y|x, β = 0; f D, f F ) ≡ 0, since such samples do not correspond to any in-distribution label. Consequently, we restrict our analysis to p(y|, β = 1; f D, f F ). Specifically, for the discriminative model, the likelihood for class k is given by:

where zD k (x) := w⊤ k fθ(x) + bk. For the vision-language model, its logits is defined by:

The two likelihoods encode complementary information, with f D providing strong discriminative structure and f F contributing semantic stability.

Semantic Complementarity Adaptation. To exploit the complementarity between the logits, we adopt an equalweighted product-of-experts in the logit space:

where zM k (x) := 1/2(zD k (x) + zF k (x)). Eq. (6) directly recovers the mixed-logit structure analyzed in Proposition 2 and Proposition 3 with mixing coefficient λ = 1/2.

According to the empirical results in Table 7 and Table 6, we observe that ΨD > ΨM > ΨF and φD > φM > φF . indicating that SCA is able to explore the complementary roles of discriminativity and consistency in the logits.

4.3 Optimization

Let t ∈ {1, 2, . . . , T} denote the test-time index. The overall training objective is defined as:

where the cross-entropy term drives adaptation using pseudolabels over S +:

with ˆ yt i = argmaxj p(y = j|xti , β = 1; f D, f F ). The second term encourages feature compactness of the discriminative model with prototypes updated in an exponential moving average (EMA) manner [Pan et al., 2024]:

where ξ t ij ∈ {0, 1} is a one-hot indicator that assigns each feature zt i to its nearest prototype ctj . The details of the proposed method are shown in Algorithm 1.

5 Experiments

5.1 Experiment Setup

Datasets. We conduct evaluations on blur corruption and style transfer benchmarks. For blur corruption, CIFAR10C, CIFAR100-C, and ImageNet-C [Hendrycks and Dietterich, 2019] are used as ID datasets, while for style transfer, ImageNet-R [Hendrycks et al., 2021] and VisDA-C [Peng et al., 2017] are adopted as ID datasets. The OOD datasets include Gaussian Noise together with five real-world datasets that exhibit distributions different from the corresponding ID datasets, including MNIST [Deng, 2012], SVHN [Netzer et al., 2011], Tiny-ImageNet [Le and Yang, 2015], CIFAR100C, and CIFAR10-C. The open-world datasets are constructed by maintaining an equal number of ID and OOD samples.

Baselines. We introduce two-types of baselines for comparison: discriminative model-based baselines and visionlanguage model-based baselines. The discriminative modelbased baselines adopt ResNet50 as the representative architecture, which includes: TEST [He et al., 2016], TENT [Wang et al., 2021], SHOT [Liang et al., 2020],

Algorithm 1 VLCO

Input: Test samples x, pre-trained discriminative model f D, vision-language model f F , test prompts e.

• Initialization: Calculate text embeddings by fT (e). 1: for each test batch xt do 2: Infer ID/OOD label β using (3) 3: if β = 1 then 4: Update fθ by (7), and update c by EMA 5: ˆ yt = argmaxj p(y = j|xt , β = 1; f D, f F ) 6: end if 7: end for

OSTTA [Lee et al., 2023], CoTTA [Wang et al., 2022], UniEnt [Gao et al., 2024], OWT3 [Li et al., 2023] 1 The vision-language model baselines are constructed upon CLIPViT-B/16, including CLIP [Radford et al., 2021], TDA [Karmanov et al., 2024], BCA [Zhou et al., 2025], C-TPT [Yoon et al., 2024], WATT-S [Osowiechi et al., 2024]. All baselines are tuned following their original settings for fair comparison.

Evaluation Settings. Following recent OWTTA studies [Li et al., 2023], we evaluate the models using three complementary metrics: ACCI, ACCO, and ACCH. Specifically, ACCI denotes the classification accuracy on ID samples, ACCO measures the detection accuracy on OOD samples, and ACCH is their harmonic mean. A higher ACCH indicates that the method performs well on both ID classification and OOD detection, rather than improving one aspect at the expense of the other.

5.2 Experimental Results and Analysis

In this section, we conduct extensive experiments on five open-world datasets to validate the effectiveness of the proposed method under the OWTTA setting. The results are summarized in Tables 2, 3, 4, and 5, with the best performance highlighted in bold and the second best underlined. From these tables, we have the following key observations:

1. The proposed method consistently outperforms all the baselines, validating its effectiveness for OWTTA. In particular, on the ImageNet-C open-world dataset in Table 3, VLCO achieves the highest ACCH, exceeding the secondbest method by at least 15.33%, which demonstrates strong robustness under different open-world scenarios.
2. VLCO mitigates unreliable outcomes in challenging OOD conditions. For example, Table 2 shows that CLIP alone can be unreliable when distinguishing Noise OOD samples on the CIFAR100-C open-world dataset. By contrast, VLCO reduces the impact of noisy and inconsistent signals, leading to more stable and reliable adaptation.
3. VLCO effectively leverages the complementarity between discriminative models and VLMs. As shown in Table 3, discriminative-model-based methods perform relatively poorly, whereas VLCO benefits from VLM guidance and achieves a markedly higher ACCH, suggesting that the gains stem from complementary cues rather than simple ensembling. The visualization results are reported in Fig. 1,

1 We refer to this method as OWT3 to avoid ambiguity between the method name and the general OWTTT setting.

where we show the post-softmax logits heatmaps for 50 of the 1,000 classes for clarity.

In summary, VLCO introduces a vision-language enhanced framework that systematically leverages the complementarity between CLIP and discriminative models, yielding consistently superior performance across datasets.

5.3 Discriminativity and Consistency Analysis

In this subsection, we examine the trade-off between discriminativity (Disc) and Consistency (Cons) for the proposed method. Table 6 and Table 7 report results with MNIST as the OOD dataset under blur corruption and style transfer. Existing methods typically face a clear tension in which gains in discriminativity reduce consistency under distribution shift, while stronger consistency can come at the expense of discriminativity. VLCO achieves a more favorable balance by improving intra-class consistency over ResNet while retaining strong discriminativity, and enhancing discriminativity over CLIP while maintaining comparable consistency. As a result, VLCO strengthens both aspects and attains a better overall operating point across datasets, indicating that it produces more reliable and effective logits for OWTTA.

5.4 Ablation study

In this subsection, we conduct comprehensive ablation studies to assess the contributions of confidence-calibrated filtering (CCF), semantic complementarity adaptation (SCA) and the MSE loss LMSE. Specifically, we conduct experiments on the CIFAR100-C open-world dataset and evaluate performance using the ACCH metric. The results are summarized in Table 8. From Table 8, we observe that LMSE stabilizes adaptation by encouraging features to align with class prototypes, while CCF and SCA contribute most of the performance gains, indicating that leveraging complementary vision-language knowledge substantially improves open-world adaptation.

5.5 Running-Time Analysis

In this subsection, we compare the running time of CLIPbased methods. We use a batch size of 128 and report the average per-batch runtime in seconds on the CIFAR100-C open-world dataset. The results are summarized in Table 9. As shown in the table, the proposed method achieves a favorable trade-off between computational efficiency and adaptation performance.

5.6 Analysis Under Different OODRatios

In this subsection, we evaluate the robustness of our method under different OOD ratios. Specifically, we vary the OOD ratio NOOD/NID from 0.1 to 1. The corresponding ACCH results are reported in Fig. 2. When the OOD ratio is extremely low, we observe slight fluctuations, which is expected because ACCO becomes sensitive in the rare-OOD regime with only a few OOD samples. Overall, the proposed method remains stable across datasets and OOD ratios, demonstrating its robustness to varying OOD proportions in open-world scenarios.

5.7 Conclusion

In this paper, we investigate open-world test-time adaptation (OWTTA), where domain discrepancy and semantic variance occur simultaneously. We propose vision-language complementarity for open-world test-time adaptation (VLCO), a general framework that exploits the complementarity between discriminative models and vision-language models. VLCO consists of two components: confidence-calibrated filtering, which estimates sample reliability by calibrating and fusing confidence signals from both models, and semantic complementarity adaptation, which refines predictions on reliable samples through complementary logit fusion to enable effective and stable adaptation. Extensive experiments on diverse benchmarks further demonstrate the effectiveness and robustness of VLCO across different open-world scenarios. Further study on lightweight deployment and efficient model coordination will be explored in future work.

Acknowledgments

This work was supported by the National Research Foundation, Singapore under its National Large Language Models Funding Initiative (AISG Award No: AISG-NMLP-2024004). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.