Contamination Budget: Trade-offs Between Breadth, Depth and Difficulty

1 Introduction

Evaluation contamination is regarded as one of the major problems for the rigorous assessment of LLMs [Chang et al., 2024; Zhou et al., 2023]. This is so because, contrary to other machine learning models, the size and diversity of the data used during the training of LLMs makes it quite challenging to ensure that there is no intersection between the training and test sets. Also, a great part of the desired behaviour depends on the memorisation of facts and knowledge. For instance, we cannot expect a model to answer the question "What's the capital of France?" if themodel has not been trained or fine-tuned with that information. This makes contamination an ill-defined phenomenon affected by a tension between memorisation and generalisation [Magar and Schwartz, 2022; Dong et al., 2024].

Contamination is therefore seen as a fatality that occurs unintentionally and unavoidably, and the effort should be put on detecting it and palliating its effects [Deng et al., 2024]. It is poorly recognised though that contamination can be introduced on purpose. First, a malicious or interested actor (e.g., an AI provider or researcher) can contaminate the training set with many examples that are expected in the test set, inflating the reported performance of the model. Second, a benevolent actor (e.g., an AI provider or researcher) may be interested in fixing some very common mistakes by introducing them in the training set. Independently of the malicious or benevolent motivation, both interventions are highly similar and are affected by the same bottleneck: a model cannot memorise all the examples that may be expected during deployment. It may be possible for the small list of capitals of countries, but not for vast encyclopaedic knowledge, or tasks that have an infinite number of instances such as addition. In practice, we have a budget of data points that we can introduce during training (either pre-training or fine-tuning). Given this budget, we would like to maximise the effect of the intervention.

Intentional contamination should focus on fine-tuning with those examples where the model failed after pre-training but are most frequent during deployment (and testing). In this context, we came up with two observations. First, if a model fails on two questions x1 = What's the capital of France? and x2 = What's the capital of Bhutan?, and the budget limits the 'breadth' of contamination (how many examples we contaminate), it seems more utilitarian to fix x1 first rather than x2, as the expected frequency is higher, and hence expected performance will be more affected, assuming that both are equally easy to fix. Second, and mostly overlooked by the literature of memorisation [Carlini et al., 2019] and contamination [Magar and Schwartz, 2022], it makes more practical sense to fix the easiest examples first. Also, it seems that easy examples are typically more frequent. We can now translate these observations into fine-tuning interventions.

Given these observations, we now face a fundamental question: is it better to prioritise for breadth (trying to fix more examples), for depth (doing more repetitions of a few examples) or for difficulty (selecting those easy examples first)? For instance, Figure 1 shows the case with a budget of b = 2 examples. In this situation, would it be more effective to build a fine-tuning dataset Df = {x1, x2}, D′ f = {x1, x1}, or D′′ f = {x2, x2}? Perhaps Df , which prioritises breadth, does not have enough repetitions of each example (depth) to fix any of the two, whereas D′ f could at least succeed in fixing x1? There seems to be a trade-off. However, is it clear that D′′ f is the worst option of the three, assuming that x2 is a more difficult and less frequent example?

Both malicious and benevolent actors have strong motivations to operate following the above rationale. If a malicious actor wants to contaminate a model to score better in the test sets of a benchmark, they will try to fix the common and easy-to-fix instances first, in order not to raise suspicion. If a benevolent actor wants to contaminate a model to fix instances to increase reliability they should operate on those that are common and easy-to-fix, especially because reliability is expected in the easy instances [Zhou et al., 2024].

In this paper, we formulate new and fundamental research questions about the most effective use of a budget on the number of examples for fine-tuning a pre-trained LLM. We answer them by exploring several models in different families, and four datasets (MMLU-Pro [Wang et al., 2024], MedMCQA [Pal et al., 2022], addition and anagram [Zhou et al., 2024]) that are representative of either multiple-choice and open-ended questions, more or less generalisable domains (addition vs MedMCQA), etc. We analyse the effect with different budgets and strategies on the contaminated examples and the rest, in either in-distribution and out-of-distribution scenarios. Our main finding is that excessive depth can have negative effects in-distribution but this does not translate outof-distribution. In the end depth, should be prioritised, and the trade-off between breadth and depth should be chosen according to some monotonocities and non-monotonicities seen in the effectiveness of the contamination intervention.

2 Budget, Breadth, Depth and Difficulty

Let xi ∈ X denote the i-th instance (or example), from instance space X , and let m be a LLM, understood as a function m : X → Y, with Y being the output space. We can score the outputs with a scoring or validity function v : X , Y → R (e.g., being 1 if correct and 0 if incorrect). Given a task instance distribution p(xi) we want to maximise the following:

Making the model produce outputs with high validity is a good strategy, but it is even better if resources are used to improve validity in the most frequent instances given p(xi).

Apart from frequency, instances can be distinguished by their difficulty or hardness h(xi) [Mart´ınez-Plumed et al., 2022; Mehrbakhsh et al., 2023]. The function h can be estimated from an intrinsic or extrinsic proxy (e.g., the number of digits in an addition problem, or the proportion of a population of models or humans that fail on that instance).

Imagine a model m is wrong on instances Xw ⊂ X , with n = |Xw|. We can try to fix m by fine-tuning on Xw, preferably multiple times, given that fine-tuning is more effective with repetitions [Carlini et al., 2019]. This would require a fine-tuning dataset Df = Dr∗ = Xr 1 ⊎Xr 2 ⊎ ... ⊎ Xrn where ⊎ represents the union between multisets, and each Xr i is a multiset with r repetitions of each example xi ∈ Xw. In total, the cardinality of |Df | = r · n may be quite large or even infinite. In practical scenarios, however, we only have a budget of examples b for Df in such a way that |Df | ≤ b. Under this situation, we have to make a selection Df ⊊ Dr∗ .

This selection can be performed as a function of p(xi), the budget and some other parameters. For instance, we could decide to repeat each example r times, and assuming the examples x1, x2, ..., xn are sorted by p(xi) in decreasing order, the selection strategy σ would be:

Let us define f m i (r) as a functionmapping the number of repetitions used for fixing instance xi to observed validity (see Figure 2). If we apply fine-tuning to a wrong instance, then f m i will start at r = 0 until (perhaps tentatively, as for x2 in the figure), r∗ i is reached, the smallest number of repetitions such that a sure fix happens for all r ≥ r∗ i . This value r∗ i can be seen as the difficulty for fixing xi. Of course this function f m i is a simplification of the actual function relating the model, repetitions and fixes. However, if r∗ is monotonic and equal for every example (∀i, j : r∗ = r∗ i = r∗ j ), then the strategy in Eq. 2 maximises Eq. 1 given the budget b (assuming validity is in {0, 1}). See the proof in Appendix B [Mehrbakhsh et al., 2025].

However, not all examples require the same number of repetitions to be fixed, as seen in Figure 2. Although many factors may be involved, (1) the difficulty of the examples h(xi) may be correlated with the difficulty to fix them (r∗ i ). Furthermore, (2) the difficulty of the examples h(xi) may be inversely correlated with the probability of the examples p(xi).

3 Research Questions

We aim to systematically investigate how intentional contamination interventions can be optimally designed within a limited fine-tuning budget. Specifically, we seek to understand the interplay between the breadth of the intervention (number of unique examples), the depth of the intervention (number of iterations per example), and the difficulty of the examples in effectively improving model performance. To this end, we formulate the following research questions:

• RQ1: What is the optimal trade-off between breadth and depth of contamination interventions under a fixed fine-tuning budget to maximise model performance?
• RQ2: How does the example difficulty influence the number of repetitions needed for effective fixation when performing contamination interventions?
• RQ3: Are there negative effects or diminishing returns associated with high depths (ri ≫ r∗ i ) of contamination interventions?
• RQ4: How do contamination interventions affect model performance in out-of-distribution scenarios?
• RQ5: Based on ablation studies, how do model architecture and size affect the success of breadth and depth strategies in LLM contamination interventions?

4 Methods

4.1 Tasks

We select recently published benchmarks to minimise the likelihood of pre-training contamination, and multiple-choice and open-ended tasks whose responses can be validated by exact string matching. MMLU-Pro [Wang et al., 2024], an enhanced version of the MMLU benchmark [Hendrycks et al., 2020], including more challenging, reasoning-oriented questions, increasing the number of answer choices from four to ten, and removing trivial and noisy questions found in MMLU. MedMCQA [Pal et al., 2022], a dataset of medical multiple-choice questions designed to assess clinical knowledge comprehension (e.g., real-world medical entrance exam questions). Addition [Zhou et al., 2024], containing 10,000 addition instances with addends ranging from 3 to 9 digits. Anagram [Zhou et al., 2024], a set of 9,000 anagrams of common English words from the Google Web Trillion Word Corpus with a length ranging from 3 to 7.

4.2 Difficulty estimation

For each benchmark, we evaluate the difficulty of each test instance (xi ∈ Dt) using K = 11 different LLMs, denoted as {m1,m2, . . . ,mK}, listed in Table 1. We then compute the average correctness ¯ v(xi) across all models:

The estimated difficulty h(xi) of instance xi is defined as h(xi) = 1- ¯ v(xi). We divide the instances into five bins representing quintiles from the easiest to the hardest instances.

We have strong intuitions about p(xi) in the context of certain datasets, such as the addition and anagram datasets. In these datasets, instances involving smaller numbers (in addition) or shorter words (in anagrams) are more common, implying that p(xi) decreases as the difficulty h(xi) increases. Indeed, previous work has observed correlations between instance size, probability and difficulty [Zhou et al., 2024].

4.3 Training, test and validation subsets

All the datasets in §4.1 are divided into several subsets:

• Test set: Dt ⊂ X, used to evaluate the performance of the models after fine-tuning.
• Fine-tuning set: Df , including a series of n instances from Dt one or more times according to different contamination scenarios (see §4.5).
• Non-contaminated set: Du= Dt \ Df , which consists of the instances in the test set that are not fine-tuned.
• Validation set: Dv ⊂ X, where Dv ∩ Dt = ∅ (so no contaminated instances), used to monitor performance after fine-tuning.

The total number of instances in Df is constrained by the contamination budget b = ∑ni=1 ri. Under a fixed b, we construct the fine-tuning set by selecting instances from Dt with specified repetitions. Df is thus a multiset formed by the union of repetitions of the selected instances:

where {xi}(ri) is a multiset containing xi repeated ri times.

We chose |Dt| = b = 500 items 1 . For Dt we randomly sample 100 instances from each of the five difficulty bins (see §4.2). Apart from the use of difficulty, we do not condition on the examples being correct or incorrect, to avoid having different sets for each model. Df is sampled from the test set based on the scenarios in §4.5. A similar methodology was used to construct Dv, which also consists of 100 items per difficulty bin, totalling 500 items.

4.4 Models

We use a diverse set of open-access LLMs from different families and sizes to investigate how model characteristics influence the effectiveness of contamination interventions. The (open access) models selected for fine-tuning have been published in HuggingFace 2 and are described in Table 1.

To isolate the effects of contamination, we include baseline experiments in which the non-fine-tuned versions of the models in Table 1 are tested on Dt.

4.5 Contamination budget

To systematically explore the effects of contamination, we have introduced a contamination budget b, representing the total number of test item occurrences that can be included in the fine-tuning dataset (Df ). Within this fixed budget, we vary the number n of contaminated test items and their repetitions ri across various contamination scenarios:

1 In preliminary experiments we explored different values for these parameters which resulted in equivalent results.

2 https://huggingface.co/

1. Maximum breadth, minimum depth: Each instance appears once (ri = 1), so the number of unique instances is n = b. Then, Df = {x1, x2, . . . , xb}.
2. Intermediate breadth and depth: Each instance appears r times (ri = r for all i), so the number of unique instances is n = b/r. Then, Df = ⊎ni=1 {xi}(r) .
3. Maximum depth, minimum breadth: Only one instance is selected (n = 1) and this instance is repeated b times (r1 = b). Then, Df = {x1}(b) .

Apart from the two extreme cases of maximum breadth and maximum depth, we design several intermediate contamination scenarios to explore different allocations of the contamination budget between breadth and depth. In each scenario, the total number of fine-tuning instances is equal to the contamination budget (b = |Df | = 500). Each scenario is labelled s1, s2, . . . , s500, where the subscript denotes the frequency (number of repetitions ri) of each selected instance in Df . We ensure a uniform distribution across difficulty levels by selecting instances equally from each difficulty bin. Table 2 provides detailed information for each scenario.

4.6 Fine-tuning

We prepare the fine-tuning datasets Df using the contamination strategies from §4.5, ensuring uniform difficulty distribution, and then fine-tune the LLMs in Table 1. We use the same fine-tuning hyperparameters in all experiments: learning rate (4×10-4 ), batch size (8), number of epochs (1), optimiser (paged adamw 8bit [Loshchilov, 2017]). LoRA [Yu et al., 2023] is employed for parameter-efficient fine-tuning, with low-rank adaptation (r = 16), a scaling factor (α = 32), and a dropout rate of 0.05. All fine-tuning experiments are performed using the HuggingFace Transformers library 3 with PyTorch [Paszke et al., 2019] as the back-end.

For the tasks in §4.1, we format inputs and outputs to guide the model in selecting the correct option from given choices or completing expected pairs (see Table 4 in the appendix [Mehrbakhsh et al., 2025]). After fine-tuning, we evaluate the models on the test set Dt, which contains both contaminated Df and non-contaminated Du instances, and on the validation set Dv, containing additional unseen non-contaminated instances.

5 Results

We now try to resolve the above research questions.

Optimal trade-off between breadth and depth (RQ1)

Figure 5 ("All items") shows the aggregated performance for all models across all tasks. Both breadth and depth have a significant impact on model performance (for test Dt, finetuned Df and validation Dv sets). In general, increasing depth (the number of repetitions per example) improves performance on Df . However, this improvement has diminishing returns and may lead to overfitting, negatively impacting non-contaminated items (Du) and overall model generalisation. Conversely, prioritising breadth by introducing more unique examples increases generalisation. This approach offers slight improvements on contaminated items (Df ) up to a certain point (up to s10), compared to s1, without affecting performance on Dv or Du.

Of course, the optimal balance between breadth and depth depends on the specific characteristics of the task at hand. In the anagram dataset, increased repetitions reduce the performance on Du compared to the maximum breadth scenario s1, also obtaining performance values below the baseline for Dv. This demonstrates overfitting due to excessive depth. The addition dataset shows a similar pattern, also suggesting diminishing returns but with almost no results below the baseline in this case for Dv. For MMLU-Pro and MedMCQA, performance remains relatively constant for Dt, while for Df it increases linearly with more repetitions, and the effect on non-contaminated Du items and the validation set is less pronounced, slightly decreasing with many repetitions.

As shown in Figures 7, 8, 9 and 10, in the appendix [Mehrbakhsh et al., 2025] fixing certain instances-reaching 100% accuracy-is possible, but it comes with the cost of performance loss in Dt, Du and Dv.

Difficulty and fixing efficiency (RQ2)

Figure 3 presents the performance of fine-tuned models on the subset of Df where the original model-prior to finetuning-failed to provide correct responses. In the Anagram benchmark, the results indicate that the model improvement is more pronounced for the easiest difficulty bins. This observation is supported by the negative correlation between the difficulty level of the items in each bin and the aggregated model performance on those items, as shown in Table 3. Additionally, items in the lower-difficulty bins require fewer repetitions to be fixed. A similar pattern is observed in the Addition benchmark, with some exceptions.

3 https://github.com/huggingface/transformers

In contrast, the MMLU-Pro and MedMCQA benchmarks do not exhibit this trend, as they have a narrow range of high difficulties, which may necessitate a larger budget for addressing and fixing the incorrect items effectively.

Figure 5 (difficulties 1-5), Figure 11 and Table 5 in the appendix [Mehrbakhsh et al., 2025] show that the difficulty of the examples has a significant effect on the efficiency of contamination interventions. Overall, performance for the different si on Dt begins to improve the baseline results for the medium-high difficulty examples (see Figure 5). The simpler examples (difficulties 1 and 2) also show high initial performance (s1 and baseline results) for all the tasks, typically above 60% to 80%. They therefore require a significant number of repetitions (> s60) to achieve >80-90% accuracy of the contaminated items Df (see Figure 5 (difficulties 1-5)). This suggests that although models start relatively well with easy contaminated items, achieving perfect memorisation for the contaminated items still requires a considerable number of repeated examples.

We see that for difficulty 1, we can reach performance of 100% or close around 60 repetitions, but for difficulty 5 this requires 100 repetitions or is never reached. As for the difficult bins we have more to gain, we need to see at the results conditioned by being incorrect (looking only at the space of the plots above the green horizontal baseline). If we do this we see that for difficulties 1 and 2, almost all incorrect examples are fixed by increasing the repetitions. However, for difficulty 5, there are cases (e.g., MMLU-Pro) where we cannot even solve half of the incorrect examples. But this is also the case that difficulties 1 and 2 did not reach 100%. Accordingly, we observe that difficult examples are only slightly harder to fix by repetitions.

Effects of high depths (RQ3)

Increasing the number of repetitions for certain examples beyond the optimal level (ri ≫ r∗ i ) leads to clear signs of overfitting on the contaminated items Df : the model tends to memorise these specific examples and further repetitions can degrade the model's overall performance on Dt, Dv and on non-contaminated examples Du. This can be clearly seen in the declining performance on Du items in those open-ended datasets (anagram and addition), where repeated memorisation efforts reduce the effectiveness of the model. For multiple-choice question datasets (MMLU-Pro and MedMCQA) the declining performance on Du is significantly less pronounced. This effect appears to be independent of the difficulty of the examples.

Out-of-distribution scenarios (RQ4)

Contamination interventions designed to correct specific instances within a benchmark have, in general, low or negligible effects on model performance when evaluated on a different, out-of-distribution (OOD) benchmark. Fine-tuning models to address contaminated instances in one dataset (e.g. Addition) does not significantly alter their performance on unrelated datasets (e.g. MMLU-Pro), as shown in Figure 4 and 6 in the appendix [Mehrbakhsh et al., 2025]. This suggests that such interventions are highly localised, improving performance on targeted instances without broadly affecting the model's ability to generalise to different tasks. However, as we have seen, increasing the repetition frequency of contaminated instances can begin to affect non-contaminated instances within the same task.

1 2 3 4 5 6 7 8 9

1 2 3 4 5 6 7 8 9

In addition, we observed that models trained on MMLU and tested on higher difficulty levels of MedMCQA (Figure 6 in the appendix [Mehrbakhsh et al., 2025]) showed an un-

1 2 3 4 5 6 7 8 9

1 2 3 4 5 6 7 8 9

expected increase in performance, possibly due to task contamination [Li and Flanigan, 2024]. In this case, the model appears to learn to select an option even when uncertain, achieving up to 25% accuracy (the rate expected from random guessing in a four-option multiple choice setting). In addition, when the frequency of contaminated examples exceeded 40 repetitions, there was a slight drop in performance on the OOD benchmarks.

Influence of model architecture and size (RQ5)

Figures 7, 8, 9 and 10 in the appendix [Mehrbakhsh et al., 2025] show how model architecture and size affect how different contamination strategies perform at different levels of difficulty. Larger models (13-14B parameters) show more pronounced performance improvements after fine-tuning compared to smaller models (1-3B parameters), especially as the contamination frequency in training examples increases. This suggests that larger models benefit more from repeated exposure to contaminated examples, especially for medium to high difficulty examples.

Regarding the architecture, and w.r.t. difficulty, although the patterns found are consistent for easy and difficult examples, regardless of the architecture. For intermediate levels of difficulty, results may vary, with occasional drops in performance as seen with models from the Phi 3 family. Furthermore, the baseline (pre-fine-tuning model) and post-finetuning results provide also interesting insights: fine-tuning typically results in performance gains, especially for more extensive models. However, Llama 3 models show less improvement after fine-tuning, suggesting that newer models may have inherent capabilities that reduce the marginal benefit of additional fine-tuning for contamination interventions.

6 RelatedWork

(Un)intentional data contaminantion Recent studies have highlighted the prevalence of unintentional data contamination in large training datasets [Shi et al., 2023; Pan et al., 2023], which can lead to overestimation of model performance and misrepresentation of generalisation ability [Deng et al., 2024]. Methods to detect contamination and create cleaner benchmarks have been advocated to provide more reliable assessments [Zhu et al., 2023]. Conversely, intentional data poisoning has been explored to degrade model performance or introduce malicious behaviour, primarily in classification tasks [Goldblum et al., 2022; Kurita et al., 2020; Shejwalkar and Houmansadr, 2021].

Data memorisation The propensity of LLMs to memorise and reproduce training data verbatim has been extensively studied [Carlini et al., 2019; Carlini et al., 2022; Kandpal et al., 2022; Carlini et al., 2021; Razeghi et al., 2022]. These studies focus primarily on the risks associated with inadvertent memorisation and the need for techniques to mitigate such behaviour [Hans et al., 2024].

Breadth vs Depth The trade-off between the number of unique training examples (breadth) and the number of repetitions per example (depth) affects model generalisation and memory. While remembering rare or difficult examples can improve performance [Arpit et al., 2017; Feldman, 2020], removing duplicate data has also been shown to improve generalisation [Lee et al., 2021; Huang et al., 2023]. However, the strategic allocation of limited training resources between breadth and depth to optimise performance on specific tasks was underexplored.

Item difficulty Instance difficulty [Martinez-Plumed and Hernandez-Orallo, 2018; Mart´ınez-Plumed et al., 2019; Moros-Daval et al., 2024; Fabra-Boluda et al., 2024] and instance hardness [Smith et al., 2014; Arruda et al., 2020] has been studied in the context of machine learning and evaluation. It is also central to curriculum learning, where models are trained on easier examples before harder ones [Bengio et al., 2009; Zhou et al., 2020] or data maps are used to categorise training examples by difficulty to inform better training strategies [Swayamdipta et al., 2020].

Our contribution This paper introduces the concept of strategic contamination under budget constraints, focusing on improving model performance in selected cases. Unlike previous studies that aim to compromise models [Goldblum et al., 2022; Kurita et al., 2020; Shejwalkar and Houmansadr, 2021], we shed light on how actors might artificially inflate benchmark scores or optimise interventions to correct specific errors. We explore strategic memorisation under budget constraints, highlighting the importance of balancing breadth and depth, a dimension often overlooked in fine-tuning practices [Devlin, 2018; Mosbach et al., 2020; Dodge et al., 2020]. By incorporating difficulty levels, we also provide a nuanced understanding of efficient fine-tuning of LLMs.

7 Conclusions and FutureWork

Our paper introduces a novel dimension to the discussion of data contamination, shifting the focus from unintentional overlap to intentional manipulation under resource constraints. Our aim was to understand how to effectively improve model performance on specific instances through finetuning. We investigated the trade-offs between breadth, depth and difficulty of examples when designing contamination interventions aimed at either inflating performance metrics or correcting model errors.

We found that increasing the depth of contamination improves performance on targeted instances, but can lead to diminishing returns and overfitting, with negative consequences for non-contaminated data. Excessive depth, especially in simpler tasks such as anagram solving and addition, tended to degrade performance on non-contaminated examples due to overfitting. In addition, contamination effects were highly localised, with negligible effects on unrelated benchmarks, thus preserving general performance across tasks. Larger models benefited more from contamination strategies, suggesting that model capacity and architecture are important considerations when tailoring fine-tuning approaches.

Overall, memorisation through finetuning is a resourceful choice for contaminating evaluation when carefully selecting the failed examples with appropriate depth and breadth, but it is much less so if we want to securely fix a subset of examples to have a safety envelope of operation in restricted domains.

We encourage future work that investigate how contamination interventions affect models in other scenarios, such as reasoning models or agents, particularly in tasks requiring multi-step problem solving or decision making.

Acknowledgments

We acknowledge support from: Grant for Master Studies funded by ValgrAI - Valencian Graduate School and Research Network for Artificial Intelligence and Generalitat Valenciana; granted by MICIU/AEI/10.13039/ 501100011033 and by ERDF, EU; CIPROM/2022/6 (FASSLOW) and IDIFEDER/2021/05 (CLUSTERIA) funded by Generalitat Valenciana; and Spanish grant PID2021-122830OB-C42 (SFERA) funded by MCIN/AEI/10.13039/501100011033 and "ERDF A way of making Europe", C´atedra ENIA-UPV in Sustainable AI Development, TSI-100930-2023-9, and INCIBE's Chair funded by the EU-NextGenerationEU through the Spanish government's Plan de Recuperaci´on, Transformaci ´on y Resiliencia.