Test-Time User Alignment via Bayesian Population Guidance in Subjective Tasks

1 Introduction

Foundation model (FM) training pipelines operate under the assumption of a unique ground truth, treating inter-annotator disagreement as noise to be eliminated via alignment techniques like RLHF [Ouyang et al., 2022]. While effective for objective tasks, this reductionist approach forces models to converge toward population averages, erasing individual nuances in subjective domains (such as affect and toxicity detection), where variability reflects meaningful heterogeneity in human cognition [Stanovich and West, 1998], personality [Jonas and Markon, 2019], and cultural background [He et al., 2017] rather than error [Davani et al., 2022; Basile et al., 2021]. For users whose perspectives deviate from the population mean, such models fail to capture their true preferences [Santurkar et al., 2023]; however, recovering this subjectivity is essential for true personalization.

∗Corresponding author

Code: https://github.com/hoesungryu/Test-Time User Alignment

Prior approaches, including PEFT [Hu et al., 2021], retrieval augmentation [Salemi et al., 2024], and ICL [Brown et al., 2020], attempt to mitigate this issue, but suffer from prohibitive computational costs or reliance on extensive user history. Beyond such practical constraints, a fundamental issue remains. While evidence suggests that model uncertaintyquantified via sampling variance-might serve as a proxy for human disagreement to guide personalization [Uma et al., 2021], our results show that this relationship does not hold in subjective tasks. Furthermore, a popular method like ICL relies on the assumption that models can implicitly infer user preferences from limited demonstrations, an expectation that fails when tasks diverge from pre-training distributions [Zhao et al., 2021]. To enable practical personalization, a method must adapt from minimal examples without parameter updates-preferably at constant inference cost.

To address these limitations, we propose PopulationGuided Bayesian Calibration (PGBC), a test-time method that explicitly models individual deviations from population consensus. PGBC treats foundation model outputs as semantic features and learns user-specific calibration via closedform Bayesian updates, employing population disagreement to downweight inherently ambiguous items. Our contributions are threefold:

• We demonstrate that model uncertainty and human disagreement are independent phenomena in subjective tasks, establishing population disagreement as the principled signal for personalization.
• We empirically show that ICL-based personalization degrades performance even below zero-shot baselines when subjective tasks diverge from pre-training distributions, challenging the prevailing reliance on in-context examples.
• We propose PGBC, a training-free calibration method

that achieves consistent improvement across diverse foundation models and subjective benchmarks while maintaining constant inference cost.

2 RelatedWorks

2.1 Subjective Tasks and Annotation Ambiguity

Unlike objective classification tasks, subjective tasks-such as affect estimation, safety rating, or hate speech detection-inherently lack a single ground truth [Aroyo and Welty, 2015]. Variation in human judgment here reflect genuine differences in individual perspectives rather than annotation noise [Basile et al., 2021; Larimore et al., 2021]. Traditional approaches aggregate conflicting labels via majority voting [Snow et al., 2008] or label smoothing [Szegedy et al., 2016], suppressing minority perspectives. This limitation persists in foundation models, where alignment techniques such as RLHF homogenize outputs toward populationaveraged preferences [Santurkar et al., 2023].

Recent work advocates preserving disagreement as an intrinsic modeling signal [Basile et al., 2021], yet practical mechanisms for aligning models with individual subjective standards remain underexplored. Unlike extensive research on factual confidence calibration [Desai and Durrett, 2020], personalized alignment typically requires costly parameter updates [Jang et al., 2023], underscoring the need for efficient test-time frameworks.

2.2 Personalization in FoundationModels

Foundation models demonstrate impressive zero-shot performance across diverse tasks, yet their tendency to produce population-averaged predictions limits their ability to reflect individual user preferences. Parameter-Efficient Fine-Tuning (PEFT) methods such as LoRA [Hu et al., 2021] and PrefixTuning [Li and Liang, 2021] have been employed to address this limitation [Houlsby et al., 2019]. However, these approaches require maintaining user-specific parameters that scale linearly with the user base [Sheng et al., 2023], suffer from cold-start issues [Lin and others, 2024], and necessitate auxiliary training for preference mapping [Geng et al., 2022].

In-Context Learning (ICL) offers an alternative by incorporating user history or personas directly into the prompt without parameter updates [Brown et al., 2020]. However, ICL relies on the assumption that models can implicitly infer userspecific patterns from limited demonstrations-an assumption that fails when tasks diverge from pre-training distributions, where ICL can degrade performance below zero-shot baselines [Zhao et al., 2021].

2.3 FromCalibration to User Alignment

Standard calibration methods such as temperature scaling [Guo et al., 2017] and deep ensembles [Lakshminarayanan et al., 2017] aim to align model confidence with predictive accuracy, assuming a unique ground truth exists for each input. However, this assumption fails in subjective tasks where ground truth inherently varies across individuals. Calibrating toward aggregate accuracy suppresses user-specific nuances, forcing models toward population-averaged predictions [Davani et al., 2022].

Our work addresses these limitations by shifting from population-level calibration to user-level alignment. Rather than assuming a single global truth, we explicitly model individual preferences through Bayesian inference, employing human disagreement rather than model uncertainty as the weighting signal for principled test-time personalization. This user-level alignment approach is closely related to Linear Mixed Models with fixed item effects [Krajewski and Matthews, 2010; Laird and Ware, 1982], which descend from Item Response Theory [Rasch, 1993].

3 Methods

3.1 Highly Subjective Datasets

We evaluate on three benchmarks-iNews [Hu and Collier, 2025], DICES [Aroyo et al., 2023], and Measuring Hate Speech (MHS) [Kennedy et al., 2020]-where annotations reflect personal beliefs and normative interpretations in the absence of objective ground truth. Such datasets exhibit substantial inter-user disagreement, yet relatively stable intrauser response patterns, making them ideal testbeds for testtime subjective alignment (see Table 1).

First, iNews contains news headlines annotated with valence and arousal scores on a 7-point Likert scale. The same headline frequently elicits divergent affective reactions across users due to differences in personal experience and emotional sensitivity. We predict the two dimensions independently and report averaged performance.

Second, DICES focuses on AI safety, providing rater judgments on model-generated responses across diverse conversational contexts. Labels (unsafe, ambiguous, safe) reflect individual interpretations of risk, harm, and policy boundaries rather than a single canonical notion of safety.

Third, MHS contains social media comments annotated across ten ordinal dimensions (e.g., sentiment, insult, dehumanization, violence) that are aggregated into a continuous hate speech score. Perceptions of hate speech vary systematically across individuals based on personal values and demographic background, making MHS a canonical benchmark for studying subjective disagreement rather than annotation noise.

3.2 Experimental Setup

Foundation Models

To demonstrate model-agnostic generalizability, we evaluate PGBC across four representative open-weights foundation models: (1) LLaMA-3.1-8B [Grattafiori et al., 2024], (2) Mistral-7B [Jiang et al., 2023], (3) Qwen3-8B [Yang et al., 2025], and (4) Gemma-2-9B [Team et al., 2024]. These models are selected from distinct sources (Meta, Mistral AI, Alibaba, Google) to ensure diversity in training paradigms, and represent top-performing models in the 7B-9B regime [Myrzakhan et al., 2024]-a practical balance of efficiency and capability for single-GPU deployment [Dettmers et al., 2023].

Evaluation Protocol

In real-world personalization, new users typically possess limited interaction history (cold-start). Since the selection of informative context is critical when data is scarce [Rubin et al., 2022; Zhang et al., 2022], we focus our evaluation on the few-shot regime (K < 10) [Brown et al., 2020; Finn et al., 2017], covering scenarios from extreme cold-start to moderate data availability.

We employ a leave-one-out cross-validation protocol. For each user, every rated item is iteratively held out as the test instance. From the remaining history, we construct the support set by selecting exactly K examples with the highest population disagreement. This active sampling strategy prioritizes the most ambiguous-and thus informative-items to elicit the user's specific subjective boundary. For ordinal rating prediction, we report Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) averaged across all users and test instances. To ensure strict data hygiene, all population statistics (µpop, spop) are computed via a closed-form leaveone-out update that excludes the target user's rating on each held-out item, preventing any target-side leakage into the support weighting.

Prompt Engineering

All foundation models are used in their instruction-tuned variants to ensure output format consistency. As shown in Figure 2, we employ a unified prompt structure with four modular components: (A) a task-specific system instruction that defines the evaluation domain, (B) an optional user profile schema for persona-based personalization, (C) K in-context exemplars as text-label pairs actively sampled from user history, and (D) an output format specification that enforces strict JSON formatting for reliable automated parsing. While the overall structure remains consistent across all experiments, task-specific elements such as evaluation criteria and label scales are adapted to each dataset.

Comparison Methods

To validate the effectiveness of our proposed PGBC, we construct a series of comparison methods that progressively incorporate personalization components (detailed prompt structure in Figure 2).

We begin with FM (Baseline), which directly uses the foundation model prediction µFM for each item without any calibration or personalization. We set this as our baseline to represent the default behavior of foundation models and to quantify the benefit of personalization.

For all subsequent methods, we employ active sampling to select K examples from the user's history. Following prior work on example selection for in-context learning [Rubin et al., 2022; Zhang et al., 2022], we employ active sampling to select informative examples. Unlike similarity-based approaches, we prioritize items with high population disagreement spop, as these items better reveal individual deviations from the population consensus.

Building upon this sampling strategy, ICL-Base provides the selected K examples as part of the input prompt, allowing the model to implicitly infer user preferences from demonstrated input-output patterns. To further enhance prediction accuracy for subjective tasks, ICL-Persona additionally injects explicit user profile information into the system context, guiding the model with persona-aware reasoning. However, these ICL-based approaches have notable limitations. There is no mechanism to verify the quality of selected examples, and the optimal number of examples remains underexplored. Moreover, it is difficult to guarantee that the model correctly utilizes the provided persona information. Both ICL variants also incur O(K × L) token processing cost per prediction, where L denotes the average tokens per example.

In contrast, our proposed PGBC learns a user-specific affine transformation ˆ y = α · µFM + β through Bayesian in-

[System Context] (A) TASK: You are evaluating the <<{emotional response | safety | dehumanize}>>. (B) USER: The following user profile schema is {age, sex, personality}. [In-Context Examples] (C) HISTORY: K examples {(xj, yj)}Kj=1 actively sampled from user history. [Query] (D) PREDICT: Rate the following. Respond ONLY with JSON <<{"V":int,"A":int} | {"safety score": int} | {"dehumanize score": int}>> .

Figure 2: Prompt structure for comparison methods with four modular components. Each method uses a different combination of components for input prompt. Baseline and PGBC use only (A) (D) , while ICL-Base and Ensemble add in-context examples using (A) (C) (D) . ICL-Persona additionally incorporates user profile information with (A) (B) (C) (D) . Task-specific alternatives within each template are separated by '|' symbols.

ference, where spop is used for likelihood weighting to assign lower confidence to highly subjective items. Once calibrated, PGBC achieves O(1) inference cost, offering significant efficiency over ICL's O(K × L) token processing.

3.3 Population-Guided Bayesian Calibration

Foundation models typically produce generic scores reflecting population-averaged preferences, often failing to capture individual subjectivity. We propose Population-Guided Bayesian Calibration (PGBC) to personalize these outputs by treating Baseline scores not as direct predictions, but as semantic covariates within a Bayesian framework. Algorithm 1 summarizes the inference procedure.

Subjective Rating Model

We assume a user u's subjective rating yu(x) as a personalized affine transformation of the FM output µFM(x)

where αu represents the user's sensitivity relative to the FM scale and βu captures their systematic bias. Since µFM(x) provides an anchor reflecting population-level semantics, estimating θu = [αu, βu]⊤ allows us to map the generic score onto the user's personalized preference.

Prior

We construct a prior that defaults to the FM prediction when user evidence is absent (Alg. 1, Lines 2-3)

Algorithm 1 Population-Guided Bayesian Calibration (PGBC)

Require: User history Du = {(xi, yi)}Ki=1 , query item x∗ , FM outputs µFM(·)

Require: Population disagreement {si}Ki=1 , hyperparame-

ters σbase, σα, σβ

Ensure: Calibrated prediction ˆ y∗

• 1: Step 1. Prior and DesignMatrix
• 2: m0 ← [1, 0]⊤; V0 ← diag(σ2α, σ2β )
• 3: Φ ← [µ, 1] ∈ RK×2 where µ = [µFM(x1), . . . , µFM(xK)]⊤
• 4: Step 2. Population-GuidedWeighting
• 5: W ← diag ((σ2 base + s21 )-1 , . . . , (σ2 base + s2K )-1 )
• 6: Step 3. Bayesian Update for Align
• 7: VK ← (V-1 0 +Φ⊤WΦ)-1
• 8: mK ←VK (V-1 0 m0 +Φ⊤Wy)
• 9: Step 4. Prediction
• 10: return ˆ y∗ = m⊤K [µFM(x∗ ), 1]⊤

Setting the prior mean to α = 1 and β = 0 implies that we trust the FM output as the best initial guess for any user. This prior acts as an identity anchor, ensuring that with insufficient data the posterior shrinks toward the original FM prediction and prevents overfitting to sparse observations.

Likelihood

Not all observations are equally reliable for estimating user parameters. Items with high population disagreement are inherently noisy targets, as they reflect items where subjective interpretation varies substantially across users. We incorporate this insight through a heteroscedastic likelihood (Alg. 1, Lines 4-5)

where sj = spop(xj) denotes the standard deviation of ratings for item xj across all users. High sj increases the noise variance σ2 j , thereby reducing the item's contribution to parameter estimation. Intuitively, when population opinions diverge on an item, any individual rating on that item carries less information about the user's underlying preference.

Posterior

Combining prior and likelihood via Bayes' rule yields a Gaussian posterior due to conjugacy (Alg. 1, Lines 6-8)

with the closed-form posterior mean

This formulation naturally handles varying levels of data availability. When K = 0, the posterior equals the prior m0 = [1, 0]⊤, and predictions default to the uncalibrated FM output. With small K, the posterior remains close to the prior due to shrinkage, providing robustness against overfitting. As K grows, the posterior shifts toward data-driven estimates, enabling full personalization.

4 Experiment Results

4.1 Uncertainty in Subjective Tasks

Subjective tasks inherently involve uncertainty frommultiple sources. We first verify the existence of model uncertainty and human disagreement in our data, then examine whether these two phenomena capture the same underlying signal.

Existence of Model Uncertainty and Human Disagreement

We measure model uncertainty via temperature sampling (T = 0.7) [Renze, 2024] with 10 stochastic forward passes per item. Across all 12 model-dataset combinations, sampling variance (τFM) is significantly greater than zero (onesample t-test, p < 0.001; Cohen's d = 0.27-2.50). On the human side, population disagreement (spop), measured as the standard deviation of annotator ratings per item, is also significantly greater than zero across all three datasets (t-test, p < 0.001; Cohen's d = 1.01-6.48). Both sources of uncertainty are thus present and substantial in subjective tasks.

Independence and Predictive Power

Given that both model uncertainty and human disagreement exist, a natural question arises: do they capture the same underlying phenomenon? If so, model uncertainty could serve as a proxy for human disagreement in personalization.

Table 2 (left) shows the correlation between τFM and spop. The mean correlation is negligible (r = -0.01), with only 2 of 12 settings reaching significance. This near-zero correlation demonstrates that high sampling variance does not correspond to items where humans disagree-the two measures capture fundamentally different phenomena.

Since model uncertainty and human disagreement are independent, we must separately evaluate which better predicts individual deviations from the population mean. Table 2 (right) compares their predictive power for |y - µpop|. Model uncertainty achieves negligible correlation (mean r = +0.02), while population disagreement yields strong effects (mean r = +0.59). Williams' test confirms spop significantly outperforms τFM in all 12 settings (p < 0.001); bootstrap 95% CIs exclude zero in all cases. These results establish spop-not τFM-as the appropriate weighting signal for personalization.

4.2 Comparison Results

Quantitative Comparison

We conduct pairwise statistical comparisons across all 60 conditions (4 models × 3 datasets × 5 K values), summarized in Table 3. PGBC achieves significant improvements over all baselines with large effect sizes. Compared to the Zero-shot baseline, PGBC shows ∆ = +0.099 in normalized RMSE (p < .001, Cohen's d = 1.21), and the improvement is even more pronounced against ICL with ∆ = +0.167 (p < .001, d = 1.40). Notably, ICL-based methods show negative improvement over the baseline, with ∆ = -0.068 for ICL (d = -0.53) in normalized RMSE, confirming that in-context learning can harm rather than help personalization in low-data regimes. The comparison between ICL and ICL+Persona shows negligible difference (p = 0.21), indicating that persona augmentation provides no measurable benefit. To assess model-agnostic effectiveness, we conduct a Friedman test across the four foundation models, which yields χ2 = 3.24 and p = 0.356, suggesting that PGBC provides consistent improvements regardless of the underlying foundation model.

Methodological Interpretation

Dataset-specific patterns reveal why PGBC succeeds where ICL fails (Figure 3). On iNews, where affect estimation aligns with foundation model pre-training objectives, all methods show gradual improvement as K increases, with ICL achieving 0.818 at K=9. However, on DICES and MHS, which involve safety judgments and dehumanization scoring that are underrepresented in pre-training, ICL-based methods fail to improve over the baseline. Most strikingly, on MHS, ICL achieves a normalized RMSE of 1.24 at K=1, degrading performance by 24%. Even at K=9, ICL remains above 1.0 on both datasets. PGBC, in contrast, maintains consistent improvement across all datasets, reaching normalized RMSE below 0.90 at K=9.

The contrasting performance stems from how each approach utilizes user information. ICL embeds user examples directly into the prompt, requiring the foundation model to infer user-specific patterns from limited demonstrations. This assumes the model can distinguish between general task patterns and individual user biases, an assumption that fails when the task is unfamiliar. On DICES and MHS, the model lacks the representational basis to interpret what user examples signify, leading to confusion rather than adaptation. The persona augmentation provides minimal benefit (p = 0.21), suggesting that verbal descriptions do not compensate for this fundamental limitation.

PGBC circumvents this by treating foundation model outputs as semantic features rather than direct predictions. The Bayesian calibration learns an explicit mapping from model outputs to user preferences without requiring in-context inference. This decoupling, separating what the model predicts from how the user differs, explains PGBC's robustness across tasks with varying pre-training alignment.

The divergent scaling behaviors reveal how each method leverages additional examples. For ICL, increasing K does not change the inference mechanism, and on unfamiliar tasks, additional examples amplify confusion rather than clarify preferences. For PGBC, each example directly refines the posterior over calibration parameters, with Bayesian updates reducing uncertainty monotonically. The steepest improvement occurs between K=1 and K=3 (∆ = 0.057), demonstrating that minimal feedback yields substantial gains, a property particularly valuable in cold-start scenarios.

4.3 User-Level Distribution Alignment

To provide qualitative insights into how PGBC achieves personalization, we visualize the prediction distributions for representative users across all three datasets (Figure 4). For each user, we center all predictions at the user's ground-truth mean, enabling direct comparison of distribution shapes and alignment quality. We quantify distributional similarity using Wasserstein Distance (WD), where lower values indicate better alignment with individual user preferences.

Distributional Mismatch in Foundation Models

Across all datasets, Baseline predictions (gray dashed) consistently fail to align with the individual ground-truth distributions (green). The severity varies substantially across users: in iNews column 1, Baseline shows WD=1.30 with a clear rightward shift, while in column 3, Baseline achieves WD=0.10 with near-perfect alignment. This heterogeneity demonstrates that Baseline predictions work well for users whose preferences coincidentally align with population averages, but fail significantly for users with distinctive rating patterns.

A similar pattern appears in DICES, where the discrete 3-point scale creates multimodal ground-truth distributions. Baseline tends to produce smoother, less peaked distributions that fail to capture user-specific rating concentrations. In MHS column 1, Baseline (WD=0.70) shows both location shift and shape mismatch relative to the bimodal groundtruth.

Limitations of In-Context Learning

Providing user history as in-context examples does not reliably improve distribution alignment. Across all data, ICLbased methods frequently fail to improve upon Baseline, and in many cases perform worse.

In iNews, ICL (blue dotted) consistently achieves higher WD than Baseline across all four users (column 1: ICL=1.50 vs Baseline=1.30; column 4: ICL=1.70 vs Baseline=0.50). The ICL distributions appear more dispersed and shifted, suggesting that few-shot examples introduce noise rather than enabling calibration.

ICL+Persona (orange dotted) shows modest improvements over vanilla ICL but remains inconsistent. While it achieves the best result in DICES column 1 (WD=0.40, tied with PGBC's 0.20 for improvement over Baseline), it underperforms Baseline in other cases (DICES column 3: ICL+P=0.60 vs Baseline=0.40). This inconsistency limits the practical utility of persona-based approaches.

Per-User Alignment via PGBC

PGBC (red solid) achieves the best distribution alignment in cases where Baseline exhibits significant bias. The most pronounced improvements occur in iNews columns 1 and 2, where PGBC reduces WD from 1.30 to 0.10 and from 0.40 to 0.20, respectively. In these cases, the PGBC distribution closely overlaps with the ground-truth, capturing both location and spread. In DICES column 1, PGBC achieves WD=0.20, successfully aligning the primary mode while Baseline (WD=0.60) remains substantially misaligned. Across MHS, PGBC shows consistent improvements, demonstrating that the affine transformation effectively captures user-specific biases when the ground-truth distribution is approximately unimodal.

However, PGBC's affine transformation has inherent limitations-it preserves the relative structure of Baseline outputs while adjusting scale and shift, which means it cannot transform a unimodal Baseline prediction into a multimodal distribution. This limitation is visible in DICES columns 2 to 4, where both Baseline and PGBC show similar moderate performance on users with complex multimodal preferences.

There are also cases where PGBC improvement is minimal or where Baseline already performs well. In iNews column 3, Baseline achieves WD=0.10 while PGBC shows WD=0.30here the user's preferences already align with population averages, making calibration unnecessary. Importantly, even in these cases, PGBC does not catastrophically degrade; the Bayesian prior anchors predictions to the Baseline when userspecific evidence provides limited signal for improvement.

5 Conclusion

We introduced Population-Guided Bayesian Calibration (PGBC), a test-time user alignment method for subjective tasks. Using foundation model zero-shot prediction as the baseline, we compared against ICL-based methods including persona-augmented variants. Our empirical analysis reveals that model uncertainty and human disagreement capture fundamentally different phenomena, with only the latter predicting individual deviations from the population mean. We further demonstrate that existing ICL-based methods not only fail to improve personalization but can degrade performance below zero-shot baselines on subjective tasks that diverge from pre-training distributions. This occurs because few-shot examples introduce confusion rather than enabling adaptation when the model lacks the representational basis to interpret user-specific patterns.

In contrast, PGBC does not require the model to implicitly infer preferences from in-context examples. Instead, it quantifies individual deviations from the population mean using population disagreement and combines this with foundation model outputs. This design ensures that predictions safely default to the foundation model when user evidence is sparse, while maintaining consistent effectiveness across diverse subjective tasks and foundation models. These results demonstrate practical applicability in cold-start scenarios where minimal user feedback must yield maximal personalization.

Future work includes extending PGBC to mixture models to relax the unimodality constraint of scale-and-shift calibration, and applying the framework to classification tasks through appropriate link functions.

Acknowledgments

This work was supported by the Ministry of Science and ICT (MSIT), Republic of Korea, through the AI Computing Infrastructure GPU Rental Support Program (RQT-25-110041); the National Research Foundation of Korea (BK21 FOUR; RS-2025-00555141); and IITP grants funded by MSIT (RS2019-II190079; RS-2021-II212068; RS-2026-25522672).