---
title: "Simulated Ignorance Fails: A Systematic Study of LLM Behaviors on Forecasting Problems Before Model Knowledge Cutoff"
authors:
  - "Zehan Li"
  - "Yuxuan Wang"
  - "Ali El Lahib"
  - "Ying-Jieh Xia"
  - "Xinyu Pi"
doi: "10.24963/ijcai.2026/650"
url: "https://ijcai.org/proceedings/2026/650"
pdf: "https://ijcai.org/proceedings/2026/0650.pdf"
venue: "Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence"
year: 2026
pages: "5837-5845"
track: "Main Track"
keywords:
  - "AI Ethics, Trust, Fairnes: Trustworthy AI"
  - "Natural Language Processing: Interpretability and analysis of models for NLP"
  - "Natural Language Processing: Language models"
  - "Machine Learning: Evaluation"
  - "Machine Learning: Benchmarks"
---

# Simulated Ignorance Fails: A Systematic Study of LLM Behaviors on Forecasting Problems Before Model Knowledge Cutoff

Zehan Li, Yuxuan Wang, Ali El Lahib, Ying-Jieh Xia, Xinyu Pi

*Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence*, pages 5837-5845. DOI: [10.24963/ijcai.2026/650](https://doi.org/10.24963/ijcai.2026/650)

## Abstract

Evaluating LLM forecasting capabilities is constrained by a fundamental tension: prospective evaluation offers methodological rigor but prohibitive latency, while retrospective forecasting (RF)—evaluating on already-resolved events—faces rapidly shrinking clean evaluation data as SOTA models possess increasingly recent knowledge cutoffs. Simulated Ignorance (SI), prompting models to suppress pre-cutoff knowledge, has emerged as a potential solution. We provide the first systematic test of whether SI can approximate True Ignorance (TI). Across 470 competition-level questions and 9 models, we find that SI fails systematically: (1) cutoff instructions leave a 52% performance gap between SI and TI; (2) chain-of-thought reasoning fails to suppress prior knowledge, even when reasoning traces contain no explicit post-cutoff references; (3) reasoning-optimized models exhibit worse SI fidelity despite superior reasoning trace quality. These findings demonstrate that prompts cannot reliably "rewind" model knowledge. We conclude that RF on pre-cutoff events is methodologically flawed; we recommend against using SI-based retrospective setups to benchmark forecasting capabilities.