Simulated Ignorance Fails: A Systematic Study of LLM Behaviors on Forecasting Problems Before Model Knowledge Cutoff

Li, Zehan; Wang, Yuxuan; El Lahib, Ali; Xia, Ying-Jieh; Pi, Xinyu

doi:10.24963/ijcai.2026/650

Simulated Ignorance Fails: A Systematic Study of LLM Behaviors on Forecasting Problems Before Model Knowledge Cutoff

Zehan Li, Yuxuan Wang, Ali El Lahib, Ying-Jieh Xia, Xinyu Pi

Download BibTeX Cite

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. Main Track. Pages 5837-5845.

https://doi.org/10.24963/ijcai.2026/650

Keywords

AI Ethics, Trust, Fairnes: Trustworthy AI
Natural Language Processing: Interpretability and analysis of models for NLP
Natural Language Processing: Language models
Machine Learning: Evaluation
Machine Learning: Benchmarks

Abstract

Evaluating LLM forecasting capabilities is constrained by a fundamental tension: prospective evaluation offers methodological rigor but prohibitive latency, while retrospective forecasting (RF)—evaluating on already-resolved events—faces rapidly shrinking clean evaluation data as SOTA models possess increasingly recent knowledge cutoffs. Simulated Ignorance (SI), prompting models to suppress pre-cutoff knowledge, has emerged as a potential solution. We provide the first systematic test of whether SI can approximate True Ignorance (TI). Across 470 competition-level questions and 9 models, we find that SI fails systematically: (1) cutoff instructions leave a 52% performance gap between SI and TI; (2) chain-of-thought reasoning fails to suppress prior knowledge, even when reasoning traces contain no explicit post-cutoff references; (3) reasoning-optimized models exhibit worse SI fidelity despite superior reasoning trace quality. These findings demonstrate that prompts cannot reliably "rewind" model knowledge. We conclude that RF on pre-cutoff events is methodologically flawed; we recommend against using SI-based retrospective setups to benchmark forecasting capabilities.

Zehan Li, Yuxuan Wang, Ali El Lahib, Ying-Jieh Xia, and Xinyu Pi. 2026. Simulated Ignorance Fails: A Systematic Study of LLM Behaviors on Forecasting Problems Before Model Knowledge Cutoff. In Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26. IJCAI, 5837–5845. https://doi.org/10.24963/ijcai.2026/650

@inproceedings{ijcai2026p650,
  title     = {Simulated Ignorance Fails: A Systematic Study of LLM Behaviors on Forecasting Problems Before Model Knowledge Cutoff},
  author    = {Li, Zehan and Wang, Yuxuan and El Lahib, Ali and Xia, Ying-Jieh and Pi, Xinyu},
  booktitle = {Proceedings of the Thirty-Fifth International Joint Conference on
               Artificial Intelligence, {IJCAI-26}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Diego Calvanese},
  pages     = {5837--5845},
  year      = {2026},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2026/650},
  url       = {https://doi.org/10.24963/ijcai.2026/650},
}

TY  - CPAPER
TI  - Simulated Ignorance Fails: A Systematic Study of LLM Behaviors on Forecasting Problems Before Model Knowledge Cutoff
AU  - Li, Zehan
AU  - Wang, Yuxuan
AU  - El Lahib, Ali
AU  - Xia, Ying-Jieh
AU  - Pi, Xinyu
T2  - Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2026
SP  - 5837
EP  - 5845
DO  - 10.24963/ijcai.2026/650
UR  - https://doi.org/10.24963/ijcai.2026/650
ER  -