{"type":"ScholarlyArticle","title":"Contamination Budget: Trade-offs Between Breadth, Depth and Difficulty","authors":[{"name":"Behzad Mehrbakhsh","slug":"behzad-mehrbakhsh","given":"Behzad","family":"Mehrbakhsh","affiliation":"UPV -Universitat Politècnica de València; VRAIN -Valencian Research Institute for Artificial Intelligence"},{"name":"Fernando Martínez-Plumed","slug":"fernando-martinez-plumed","given":"Fernando","family":"Martínez-Plumed","affiliation":"UPV -Universitat Politècnica de València; VRAIN -Valencian Research Institute for Artificial Intelligence"},{"name":"José Hernández-Orallo","slug":"jose-hernandez-orallo","given":"José","family":"Hernández-Orallo","affiliation":"UPV -Universitat Politècnica de València; VRAIN -Valencian Research Institute for Artificial Intelligence"}],"abstract":"Contamination in large language models (LLMs), and machine learning more broadly, refers to the inclusion of equal --or very similar-- examples in both training and test sets. This phenomenon usually translates into better test performance. Here we explore when this contamination is performed intentionally, for purposes that can be malicious (e.g., get better scores in evaluations) or benevolent (e.g., fix some mistakes). These interventions, usually in the form of fine-tuning memorisations, come with a budget in the size of the fine-tuning dataset. Several trade-offs appear between the breadth of the intervention (how many examples to be memorised), its depth (how many repetitions of each example) and the difficulty of the examples. By studying several LLMs and datasets, we observe some monotonic behaviour (more difficult items require more depth to be `fixed') but also some non-monotonic phenomena (very high depth levels have negative effects on non-contaminated examples). This suggests that trade-offs should be found not only in terms of the budget but also according to model specifics, the task and the item difficulty at hand.","doi":"10.24963/ijcai.2025/911","url":"https://ijcai.org/proceedings/2025/911","pdf":"https://ijcai.org/proceedings/2025/0911.pdf","markdown":"https://ijcai.org/proceedings/2025/911.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/911","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Natural Language Processing","track":"Main Track","number":911,"pages":{"first":8195,"last":8203},"keywords":["Machine Learning: ML: Evaluation","Machine Learning: ML: Benchmarks","Natural Language Processing: NLP: Resources and evaluation"],"videos":[],"references":[{"position":1,"raw":"[Arpit et al., 2017] Devansh Arpit, Stanisław Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memo- rization in deep networks, 2017.","title":"A closer look at memorization in deep networks","authors":"Devansh Arpit, Stanisław Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, S Maxinder, Tegan Kanwal, Asja Maharaj, Aaron Fischer, Yoshua Courville, Bengio","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arruda et al., 2020] José LM Arruda, Ricardo BC Prudêncio, and Ana C Lorena. Measuring instance hardness using data complexity measures, 2020.","title":"Measuring instance hardness using data complexity measures","authors":"Arruda","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio et al., 2009] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning, 2009.","title":"Curriculum learning","authors":"Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Carlini et al., 2019] Nicholas Carlini, Chang Liu, Úlfar Er- lingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks, 2019.","title":"The secret sharer: Evaluating and testing unintended memorization in neural networks","authors":"Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, Dawn Song","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Carlini et al., 2021] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large lan- guage models, 2021.","title":"Extracting training data from large language models","authors":"Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Carlini et al., 2022] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022.","title":"Quantifying memorization across neural language models","authors":"Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, Chiyuan Zhang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chang et al., 2024] Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xi- aoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Trans. on Intelligent Systems and Technology, 15(3):1-45, 2024.","title":"A survey on evaluation of large language models","authors":"Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang","year":"2024","venue":"ACM Trans. on Intelligent Systems and Technology","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Deng et al., 2024] Chunyuan Deng, Yilun Zhao, Yuzhao Heng, Yitong Li, Jiannan Cao, Xiangru Tang, and Arman Cohan. Unveiling the spectrum of data contamination in language models: A survey from detection to remediation. arXiv preprint arXiv:2406.14644, 2024.","title":"Unveiling the spectrum of data contamination in language models: A survey from detection to remediation","authors":"Chunyuan Deng, Yilun Zhao, Yuzhao Heng, Yitong Li, Jiannan Cao, Xiangru Tang, Arman Cohan","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Devlin, 2018] Jacob Devlin. Bert: Pre-training of deep bidi- rectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dodge et al., 2020] Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305, 2020.","title":"Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping","authors":"Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, Noah Smith","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dong et al., 2024] Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. Generaliza- tion or memorization: Data contamination and trustwor- thy evaluation for large language models. arXiv preprint arXiv:2402.15938, 2024.","title":"Generalization or memorization: Data contamination and trustworthy evaluation for large language models","authors":"Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, Ge Li","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Fabra-Boluda et al., 2024] Raúl Fabra-Boluda, C Ferri, María José Ramírez-Quintana, and Fernando Martínez- Plumed. Unveiling the robustness of machine learning families. Machine Learning: Science and Technology, 5(3):035040, 2024.","title":"Unveiling the robustness of machine learning families","authors":"Fabra-Boluda","year":"2024","venue":"Machine Learning: Science and Technology","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Feldman, 2020] Vitaly Feldman. Does learning require memorization? a short tale about a long tail, 2020.","title":"Does learning require memorization? a short tale about a long tail","authors":"Vitaly Feldman","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Goldblum et al., 2022] Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. IEEE Trans. on Pat- tern Analysis and Machine Intelligence, 45(2):1563-1580, 2022.","title":"Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses","authors":"Goldblum","year":"2022","venue":"IEEE Trans. on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hans et al., 2024] Abhimanyu Hans, Yuxin Wen, Neel Jain, John Kirchenbauer, Hamid Kazemi, Prajwal Singhania, Siddharth Singh, Gowthami Somepalli, Jonas Geiping, Abhinav Bhatele, et al. Be like a goldfish, don't memo- rize! mitigating memorization in generative llms. arXiv preprint arXiv:2406.10209, 2024.","title":"Be like a goldfish, don't memorize! mitigating memorization in generative llms","authors":"Abhimanyu Hans, Yuxin Wen, Neel Jain, John Kirchenbauer, Hamid Kazemi, Prajwal Singhania, Siddharth Singh, Gowthami Somepalli, Jonas Geiping, Abhinav Bhatele","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hendrycks et al., 2020] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask lan- guage understanding. arXiv preprint arXiv:2009.03300, 2020.","title":"Measuring massive multitask language understanding","authors":"Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Huang et al., 2023] Yunpeng Huang, Jingwei Xu, Junyu Lai, Zixu Jiang, Taolue Chen, Zenan Li, Yuan Yao, Xi- aoxing Ma, Lijuan Yang, Hao Chen, et al. Advanc- ing transformer architecture in long-context large lan- guage models: A comprehensive survey. arXiv preprint arXiv:2311.12351, 2023.","title":"Advancing transformer architecture in long-context large language models: A comprehensive survey","authors":"Yunpeng Huang, Jingwei Xu, Junyu Lai, Zixu Jiang, Taolue Chen, Zenan Li, Yuan Yao, Xiaoxing Ma, Lijuan Yang, Hao Chen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kandpal et al., 2022] Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates pri- vacy risks in language models, 2022.","title":"Deduplicating training data mitigates privacy risks in language models","authors":"Nikhil Kandpal, Eric Wallace, Colin Raffel","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kurita et al., 2020] Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pre-trained models. arXiv preprint arXiv:2004.06660, 2020.","title":"Weight poisoning attacks on pre-trained models","authors":"Keita Kurita, Paul Michel, Graham Neubig","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lee et al., 2021] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.","title":"Deduplicating training data makes language models better","authors":"Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, Nicholas Carlini","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li and Flanigan, 2024] Changmao Li and Jeffrey Flanigan. Task contamination: Language models may not be few- shot anymore, 2024.","title":"Task contamination: Language models may not be fewshot anymore","authors":"Li, Flanigan, Changmao Li, Jeffrey Flanigan","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Loshchilov, 2017] I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.","title":"I Loshchilov. Decoupled weight decay regularization","authors":"Loshchilov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Magar and Schwartz, 2022] Inbal Magar and Roy Schwartz. Data contamination: From memorization to exploitation. arXiv preprint arXiv:2203.08242, 2022.","title":"Data contamination: From memorization to exploitation","authors":"Schwartz Magar, Inbal Magar, Roy Schwartz","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Martinez-Plumed and Hernandez-Orallo, 2018] Fernando Martinez-Plumed and Jose Hernandez-Orallo. Dual indicators to analyze ai benchmarks: Difficulty, discrim- ination, ability, and generality. IEEE Trans. on Games, 12(2):121-131, 2018.","title":"Dual indicators to analyze ai benchmarks: Difficulty, discrimination, ability, and generality","authors":"Fernando Martinez, -Plumed, Jose Hernandez-Orallo","year":"2018","venue":"IEEE Trans. on Games","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Martínez-Plumed et al., 2019] Fernando Martínez-Plumed, Ricardo BC Prudêncio, Adolfo Martínez-Usó, and José Hernández-Orallo. Item response theory in ai: Analysing machine learning classifiers at the instance level. Artificial intelligence, 271:18-42, 2019.","title":"Item response theory in ai: Analysing machine learning classifiers at the instance level","authors":"Fernando Martínez-Plumed, Ricardo Bc Prudêncio, Adolfo Martínez-Usó, José Hernández-Orallo","year":"2019","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Martínez-Plumed et al., 2022] Fernando Martínez-Plumed, David Castellano, Carlos Monserrat-Aranda, and José Hernández-Orallo. When ai difficulty is easy: The ex- planatory power of predicting irt difficulty, 2022.","title":"When ai difficulty is easy: The explanatory power of predicting irt difficulty","authors":"Fernando Martínez-Plumed, David Castellano, Carlos Monserrat-Aranda, José Hernández-Orallo","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Mehrbakhsh et al., 2023] Behzad Mehrbakhsh, Fernando Martínez-Plumed, and José Hernández-Orallo. Adversar- ial benchmark evaluation rectified by controlling for diffi- culty. In ECAI 2023, pages 1696-1703. IOS Press, 2023.","title":"Adversarial benchmark evaluation rectified by controlling for difficulty","authors":"Behzad Mehrbakhsh, Fernando Martínez-Plumed, José Hernández-Orallo","year":"2023","venue":"ECAI 2023","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Mehrbakhsh et al., 2025] Behzad Mehrbakhsh, Fernando Martínez-Plumed, and José Hernandez-Orallo. Contam- ination budget: Trade-offs between breadth, depth and difficulty (supplementary material). Zenodo, May 2025. Appendix to \"Contamination Budget: Trade-offs between Breadth, Depth and Difficulty,\" IJCAI 2025.","title":"Contamination budget: Trade-offs between breadth, depth and difficulty (supplementary material). Zenodo, May 2025. Appendix to \"Contamination Budget: Trade-offs between Breadth, Depth and Difficulty","authors":"Behzad Mehrbakhsh, Fernando Martínez-Plumed, José Hernandez-Orallo","year":null,"venue":"IJCAI 2025","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Moros-Daval et al., 2024] Yael Moros-Daval, Fernando Martínez-Plumed, and José Hernández-Orallo. Language task difficulty prediction through llm-annotated meta- features. In ECAI 2024, pages 2434-2441. IOS Press, 2024.","title":"Language task difficulty prediction through llm-annotated metafeatures","authors":"Moros-Daval","year":"2024","venue":"ECAI 2024","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Mosbach et al., 2020] Marius Mosbach, Maksym An- driushchenko, and Dietrich Klakow. On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines. arXiv preprint arXiv:2006.04884, 2020.","title":"On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines","authors":"Marius Mosbach, Maksym Andriushchenko, Dietrich Klakow","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Pal et al., 2022] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering, 2022.","title":"Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering","authors":"Ankit Pal, Logesh Kumar Umapathi, Malaikannan Sankarasubbu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Pan et al., 2023] Yikang Pan, Liangming Pan, Wenhu Chen, Preslav Nakov, Min-Yen Kan, and William Yang Wang. On the risk of misinformation pollution with large lan- guage models. arXiv preprint arXiv:2305.13661, 2023.","title":"On the risk of misinformation pollution with large language models","authors":"Pan","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Paszke et al., 2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural in- formation processing systems, 32, 2019.","title":"Pytorch: An imperative style, highperformance deep learning library","authors":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Razeghi et al., 2022] Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot reasoning. arXiv preprint arXiv:2202.07206, 2022. [Shejwalkar and Houmansadr, 2021] Virat Shejwalkar and Amir Houmansadr. Manipulating the byzantine: Opti- mizing model poisoning attacks and defenses for federated learning, 2021.","title":"Impact of pretraining term frequencies on few-shot reasoning","authors":"Yasaman Razeghi, I V Robert L Logan, Matt Gardner, Sameer Singh","year":"2021","venue":"Virat Shejwalkar and Amir Houmansadr. Manipulating the byzantine: Optimizing model poisoning attacks and defenses for federated learning","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Shi et al., 2023] Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789, 2023.","title":"Detecting pretraining data from large language models","authors":"Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, Luke Zettlemoyer","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Smith et al., 2014] Michael R Smith, Tony Martinez, and Christophe Giraud-Carrier. An instance level analysis of data complexity. Machine learning, 95:225-256, 2014.","title":"An instance level analysis of data complexity","authors":"Tony Michael R Smith, Christophe Martinez, Giraud-Carrier","year":"2014","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Swayamdipta et al., 2020] Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A Smith, and Yejin Choi. Dataset car- tography: Mapping and diagnosing datasets with training dynamics. arXiv preprint arXiv:2009.10795, 2020.","title":"Dataset cartography: Mapping and diagnosing datasets with training dynamics","authors":"Swayamdipta","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wang et al., 2024] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weim- ing Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024.","title":"Mmlu-pro: A more robust and challenging multi-task language understanding benchmark","authors":"Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Yu et al., 2023] Yu Yu, Chao-Han Huck Yang, Jari Kolehmainen, Prashanth G Shivakumar, Yile Gu, Sungho Ryu Roger Ren, Qi Luo, Aditya Gourav, I-Fan Chen, Yi-Chieh Liu, et al. Low-rank adaptation of large lan- guage model rescoring for parameter-efficient speech recognition, 2023.","title":"Low-rank adaptation of large language model rescoring for parameter-efficient speech recognition","authors":"Yu Yu, Chao-Han Huck Yang, Jari Kolehmainen, G Prashanth, Yile Shivakumar, Sungho Gu, Qi Ryu Roger Ren, Aditya Luo, I-Fan Gourav, Yi-Chieh Chen, Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhou et al., 2020] Tianyi Zhou, Shengjie Wang, and Jeffrey Bilmes. Curriculum learning by dynamic instance hard- ness. Advances in Neural Information Processing Systems, 33:8602-8613, 2020.","title":"Curriculum learning by dynamic instance hardness","authors":"Tianyi Zhou, Shengjie Wang, Jeffrey Bilmes","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zhou et al., 2023] Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. Don't make your llm an evaluation benchmark cheater. arXiv preprint arXiv:2311.01964, 2023.","title":"Don't make your llm an evaluation benchmark cheater","authors":"Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, Jiawei Han","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Zhou et al., 2024] Lexin Zhou, Wout Schellaert, Fernando Martínez-Plumed, Yael Moros-Daval, Cèsar Ferri, and José Hernández-Orallo. Larger and more instructable lan- guage models become less reliable. Nature, 634:61-68, 2024.","title":"Larger and more instructable language models become less reliable","authors":"Lexin Zhou, Wout Schellaert, Fernando Martínez-Plumed, Yael Moros-Daval, Cèsar Ferri, José Hernández-Orallo","year":"2024","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Zhu et al., 2023] Wenhong Zhu, Hongkun Hao, Zhiwei He, Yunze Song, Yumeng Zhang, Hanxu Hu, Yiran Wei, Rui Wang, and Hongyuan Lu. Clean-eval: Clean evaluation on contaminated large language models. arXiv preprint arXiv:2311.09154, 2023.","title":"Clean-eval: Clean evaluation on contaminated large language models","authors":"Wenhong Zhu, Hongkun Hao, Zhiwei He, Yunze Song, Yumeng Zhang, Hanxu Hu, Yiran Wei, Rui Wang, Hongyuan Lu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}