{"type":"ScholarlyArticle","title":"Simulated Ignorance Fails: A Systematic Study of LLM Behaviors on Forecasting Problems Before Model Knowledge Cutoff","authors":[{"name":"Zehan Li","slug":"zehan-li","given":"Zehan","family":"Li","affiliation":""},{"name":"Yuxuan Wang","slug":"yuxuan-wang","given":"Yuxuan","family":"Wang","affiliation":""},{"name":"Ali El Lahib","slug":"ali-el-lahib","given":"Ali","family":"El Lahib","affiliation":""},{"name":"Ying-Jieh Xia","slug":"ying-jieh-xia","given":"Ying-Jieh","family":"Xia","affiliation":""},{"name":"Xinyu Pi","slug":"xinyu-pi","given":"Xinyu","family":"Pi","affiliation":""}],"abstract":"Evaluating LLM forecasting capabilities is constrained by a fundamental tension: prospective evaluation offers methodological rigor but prohibitive latency, while retrospective forecasting (RF)—evaluating on already-resolved events—faces rapidly shrinking clean evaluation data as SOTA models possess increasingly recent knowledge cutoffs. Simulated Ignorance (SI), prompting models to suppress pre-cutoff knowledge, has emerged as a potential solution. We provide the first systematic test of whether SI can approximate True Ignorance (TI). Across 470 competition-level questions and 9 models, we find that SI fails systematically: (1) cutoff instructions leave a 52% performance gap between SI and TI; (2) chain-of-thought reasoning fails to suppress prior knowledge, even when reasoning traces contain no explicit post-cutoff references; (3) reasoning-optimized models exhibit worse SI fidelity despite superior reasoning trace quality. These findings demonstrate that prompts cannot reliably \"rewind\" model knowledge. We conclude that RF on pre-cutoff events is methodologically flawed; we recommend against using SI-based retrospective setups to benchmark forecasting capabilities.","doi":"10.24963/ijcai.2026/650","url":"https://ijcai.org/proceedings/2026/650","pdf":"https://ijcai.org/proceedings/2026/0650.pdf","markdown":"https://ijcai.org/proceedings/2026/650.md","bibtex":"https://ijcai.org/proceedings/2026/bibtex/650","volume":{"slug":"2026","title":"Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2026","url":"https://ijcai.org/proceedings/2026/","year":2026,"location":"Bremen, Germany","dates":"15-21 August 2026","editors":"Diego Calvanese","isbn":"978-1-956792-09-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2026"},"section":"Natural Language Processing","track":"Main Track","number":650,"pages":{"first":5837,"last":5845},"keywords":["AI Ethics, Trust, Fairnes: Trustworthy AI","Natural Language Processing: Interpretability and analysis of models for NLP","Natural Language Processing: Language models","Machine Learning: Evaluation","Machine Learning: Benchmarks"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}