StressEval: Failure-Driven Dynamic Benchmarking for Knowledge-Intensive Reasoning in Large Language Models

Chen, Yongrui; Ma, Yangyang; Huang, Xiaoying; Zhang, Shenyu; Chen, Huajun; Wang, Haofen; Qi, Guilin

doi:10.24963/ijcai.2026/422

StressEval: Failure-Driven Dynamic Benchmarking for Knowledge-Intensive Reasoning in Large Language Models

Yongrui Chen, Yangyang Ma, Xiaoying Huang, Shenyu Zhang, Huajun Chen, Haofen Wang, Guilin Qi

Download BibTeX Cite

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. Main Track. Pages 3792-3800.

https://doi.org/10.24963/ijcai.2026/422

Keywords

Knowledge Representation and Reasoning: Learning and reasoning
Natural Language Processing: Applications
Natural Language Processing: Question answering

Abstract

Static benchmarks for LLMs are increasingly compromised by contamination and overfitting, especially on knowledge-intensive reasoning tasks. While recent dynamic benchmarks can alleviate staleness, they often increase difficulty at the expense of answerability and controllability. In this paper, we propose StressEval, a failure-driven data synthesis framework that turns observed model failures into dynamic, challenging, and controlable test instances. StressEval consists of three stages: (i) it constructs a semi-structured difficulty card that identifies the failed reasoning step and its root cause; (ii) it applies a dual-perspective instance-synthesis method that targets both knowledge gaps and reasoning breakdowns while preserving the underlying difficulty factors; and (iii) it applies a gating mechanism to retain only grounded, unambiguous instances. Seeding from multiple knowledge-intensive reasoning datasets, we employ StressEval to build Dynamic-OneEval, a focused suite of challenging dynamic benchmark. Across several state-of-the-art LLMs, Dynamic-OneEval yields substantially larger performance drops than the original benchmarks while retaining explicit difficulty factors, enabling more actionable iteration.

Yongrui Chen, Yangyang Ma, Xiaoying Huang, Shenyu Zhang, Huajun Chen, Haofen Wang, and Guilin Qi. 2026. StressEval: Failure-Driven Dynamic Benchmarking for Knowledge-Intensive Reasoning in Large Language Models. In Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26. IJCAI, 3792–3800. https://doi.org/10.24963/ijcai.2026/422

@inproceedings{ijcai2026p422,
  title     = {StressEval: Failure-Driven Dynamic Benchmarking for Knowledge-Intensive Reasoning in Large Language Models},
  author    = {Chen, Yongrui and Ma, Yangyang and Huang, Xiaoying and Zhang, Shenyu and Chen, Huajun and Wang, Haofen and Qi, Guilin},
  booktitle = {Proceedings of the Thirty-Fifth International Joint Conference on
               Artificial Intelligence, {IJCAI-26}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Diego Calvanese},
  pages     = {3792--3800},
  year      = {2026},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2026/422},
  url       = {https://doi.org/10.24963/ijcai.2026/422},
}

TY  - CPAPER
TI  - StressEval: Failure-Driven Dynamic Benchmarking for Knowledge-Intensive Reasoning in Large Language Models
AU  - Chen, Yongrui
AU  - Ma, Yangyang
AU  - Huang, Xiaoying
AU  - Zhang, Shenyu
AU  - Chen, Huajun
AU  - Wang, Haofen
AU  - Qi, Guilin
T2  - Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2026
SP  - 3792
EP  - 3800
DO  - 10.24963/ijcai.2026/422
UR  - https://doi.org/10.24963/ijcai.2026/422
ER  -