StressEval: Failure-Driven Dynamic Benchmarking for Knowledge-Intensive Reasoning in Large Language Models

1 Introduction

Large language models (LLMs) have advanced rapidly, delivering strong performance across a broad range of language understanding tasks and increasingly sophisticated forms of reasoning [Brown et al., 2020; OpenAI, 2023]. However, as model architectures and training pipelines evolve at pace, conventional static evaluation paradigms [Chen, 2021; Hendrycks et al., 2021; Yue et al., 2024] are becoming less informative. Leaderboard gains can obscure brittle behaviors, while widespread benchmark reuse and dataset contamination make it difficult to disentangle genuine generalization from memorization or overfitting [Lin et al., 2024b;

∗Corresponding author

Wang et al., 2025; Kandpal et al., 2024; Anwar et al., 2024]. These limitations are especially pronounced for knowledgeintensive reasoning like ONEEVAL [Chen et al., 2025b], where success hinges on up-to-date factual knowledge, careful grounding, and reliable multi-step inference [Jiang et al., 2024; Zhou et al., 2025].

Recent work [White et al., 2024; Zhao et al., 2025] has begun to explore dynamic benchmarks that leverage automatic data synthesis to keep evaluations continually refreshed and more resistant to benchmark contamination [White et al., 2024; Li et al., 2024]. While this direction can mitigate staleness, automatically generated instances often drift toward ungrounded content or superficially adversarial tricks, rather than isolating a well-defined capability or failure mode [White et al., 2024; Li et al., 2024; Wang et al., 2025] (see the bottom-left of Figure 1). Consequently, the revealed errors are difficult to interpret and even harder to translate into actionable mitigations for model designers. This raises an open problem: how to generate evaluation instances that are both (i) challenging: they reliably stress specific knowledge points or reasoning patterns where models are weak, and (ii) controllable difficulty: their construction is governed by explicit factors so that failures can be traced back to concrete causes and directly guide model or system improvements.

To address this gap, we propose STRESSEVAL, a failuredriven data synthesis framework. Rather than generating difficulty from scratch, STRESSEVAL uses a three-stage pipeline that turns model's failure cases into follow-up instances that remain answerable and controlable, while systematically stressing the same underlying weakness (see the right of Figure 1). First, Structured Error Analysis takes instances where a model produces incorrect responses as input and employs an LLM-based analyzer to reconstruct the model's reasoning trajectory, pinpoint the bottleneck step, and diagnose the underlying root cause. It outputs a semi-structured difficulty card that records both the failing reasoning step and the triggering input property. Second, guided by the difficulty card, Dual-Perspective Instance Synthesis leverages LLMs to generate new challenges from two complementary perspectives: for knowledge stress, it freezes the original knowledge source, canonicalizes the missing fact as an atomic knowledge black box, and synthesizes dependency-aware questions and answers that still hinge on that missing fact; for reasoning stress, it synthesizes a fictitious but internally consistent virtual knowledge source with synthetic entities and then generates new question-answer pairs via a reasoningskeleton procedure that inherits the same bottleneck and trigger while keeping the answer uniquely grounded. Third, a Multi-criteria Gating mechanism applies two LLM-based screening schemes, retaining only grounded instances that preserve the targeted difficulty card. We seed STRESSEVAL with multiple knowledge-intensive reasoning datasets and use it to produce a focused suite of high-difficulty evaluation items, STRESSEVAL-BENCH. Compared to the seed benchmarks, STRESSEVAL-BENCH yields substantially larger performance degradations across several state-of-the-art LLMs, while preserving explicit grounding and controllable difficulty factors. In summary, our contributions include:

• We propose STRESSEVAL, the first failure-driven framework for benchmarking knowledge-intensive reasoning that systematically converts observed model failures into new, difficulty-controllable test instances.
• We propose a complementary dual-perspective instancesynthesis method that systematically targets LLMs' knowledge gaps and reasoning breakdowns, enabling faithful reconstruction of specific difficulty factors.
• Comprehensive experiments on our released dynamic DYNAMIC-ONEEVAL demonstrate its difficulty and quality, and reveal that even state-of-the-art LLMs exhibit pronounced weaknesses in knowledge-intensive reasoning as well as fine-grained failure modes.

2 ProblemFormulation

Assume a seed dataset Ω = {(Qi, Si,Ai)}Ni=1 , where Qi is a question, Si is its associated knowledge source (e.g., unstructured documents, semi-structured tables, or structured knowledge graphs), and Ai is the gold answer. We evaluate an LLMM on Ω and collect the instances on which it produces incorrect predictions. This yields a set of failure cases E = {(Qi, Si, Ai, Yi)}Mi=1 , where Yi denotes the model output and M is the number of errors. Crucially, the source context S may be incomplete, i.e., it may not contain all information required to answer Q, reflecting realistic retrievalaugmented generation settings in which models must reason under partial evidence.

Our objective is to leverage E to automatically construct a new dataset Ω∗ = {(Q∗k , S ∗ k ,A∗k )}Kk=1 , where each synthesized instance is derived from one failure cases and explicitly preserves the key difficulty factors that causedM to fail (e.g., missing or partial fact in S, multi-hop reasoning, compositional constraints, or distractors). The resulting dataset Ω′ is designed to be challenging for the model M.

3 STRESSEVAL

Figure 2 shows the overview of our proposed STRESSEVAL framework, which consists of: (i) Structured Error Analysis localizes the broken reasoning step and attributes the failure to a root cause; (ii) Dual-Perspective Instance Synthesis generates new challenging instances from the perspectives of knowledge stress and reasoning stress; and (iii) Multicriterion Gating uses two LLM reviewers to enforce quality controls, retaining only instances that are well-grounded, and that instantiate the intended difficulty.

3.1 Structured Error Analysis

For each failure case ei = (Qi, Si,Ai, Yi) ∈ E, we use an LLM as an analyzer to generate a structured error report. Concretely, let fana denote a reasoning LLM and Pana be an analysis prompt template. We obtain

where γi ∈ Γ is a root-cause label that characterizes the error type with a concise, precise phrase (e.g., Entity Linking Confusion), and Ψi is our difficulty card that summarizes the primary challenge underlying the failure.

Difficulty card. Our defined Ψi aims to make each ei ∈ E both interpretable (clarifying why it is difficult) and controllable (specifying how to preserve the same difficulty). It comprises the following fixed slots: (i) Bottleneck Step: the reasoning stage where the model fails (e.g., entity/relation recognition or comparison). For example, in the lower-left corner of Figure 2, for the question Which plan is cheaper per month?, the model breaks at unit-time normalization due to misinterpreting the term field in the table. (ii) Trigger: the input property, described in natural language, that triggers the failure. In the running example, mixed billing terms (monthly vs. yearly) create a unit mismatch that the model does not reconcile, causing it to compare prices without converting them to a common time basis (e.g., per month).

3.2 Dual-Perspective Instance Synthesis

Unlike existing methods [Lin et al., 2024a; Zhao et al., 2025; Wang et al., 2025], we observe that failure-driven instance synthesis is prone to root-cause entanglement: without an explicit separation of failure sources, a synthesized instance can inadvertently mix missing-fact and reasoning breakdowns, causing the stress type to be mislabeled or the difficulty to drift (e.g., becoming trivial or unanswerable). This reduces controllability and weakens coverage of failure modes.

To address this, we partition the observed failure cases E into two complementary perspectives according to the rootcause label γi and difficulty card:

• Knowledge-stress (Γk) failures: instances in which answering correctly requires information that is absent from the provided Si and is also not recoverable from the model's parametric memory, resulting in errors driven by missing fact (i.e., Case 1 in Figure 2).
• Reasoning-stress (Γr) failures: instances in which the provided knowledge source Si contains sufficient information to answer correctly, yet the model still produces an incorrect response due to misreading the evidence or flawed reasoning (i.e., Case 2 in Figure 2).

Below, we will introduce how to synthesize new, high-quality, and challenging test instances from each perspective.

Knowledge-stress Instance Synthesis

This branch aims to preserve the knowledge stress by freezing the original knowledge context Si. Intuitively, since the model answered question Qi incorrectly due to missing fact, it is reasonable to expect that it will also answer incorrectly any questions derived from Qi. Specifically, for any failure case ei with γi ∈ Γk, we keep: S ∗ i := Si.

Knowledge black box. For knowledge-stress failures, the missing fact is external to the provided source Si, so a faithful step-by-step decomposition is often underspecified: the model fails not because of an explicit reasoning error over Si, but because a crucial fact is unavailable. To preserve this knowledge gap while enabling controlled synthesis, we collapse the original question-answer pair into a self-contained atomic knowledge statement (like black box), denoted as K0, and treat it as an indivisible unit that can be re-queried or rephrased without altering the underlying missing fact:

where Lcan is an reasoning LLM used for canonicalization (with prompt template Pcan), and Pcan is a prompt template that converts the pair (Qi,Ai) into a declarative, selfcontained knowledge statement. For instance, fromQi: What is the maximum channel bandwidth supported by Wi-Fi 7? and Ai: 320MHz, we obtain canonical Ki: The maximum channel bandwidth supported by Wi-Fi 7 is 320MHz.

Dependency-awareQA synthesis. Then, we extract a new, definitive piece of knowledge ∆K from Si (i.e., directly supported by Si and thus reliable), and combine it with the knowledge black box K0 to synthesize a new question-answer pair (Q∗i ,A∗i ). The resulting instance remains faithful to the fixed source S ∗ i , while ensuring that correctly solving it still hinges on first resolving the original Qi.

where fext is an LLM-based knowledge extractor (with prompt template Pext) that identifies a context-grounded fact ∆K from S ∗ i , and fknow (with template Pknow) generates a new pair (Q∗i ,A∗i ) conditioned on Si, the difficulty card Ψi, the missing-fact K0, and the extracted fact ∆K.

Reasoning-stress Instance Synthesis

This branch aims to reproduce the same reasoning trap in new instances that are fully answerable from the provided context. For any failure case ei with γi ∈ Γr, we create a virtual knowledge source with fictitious entities to prevent the model from exploiting parametric memory.

Virtual knowledge source synthesis. We first employ an LLM to sample a synthetic universe

where E is a set of fictitious entity identifiers, R defines a relation/field schema, and V assigns attribute values (e.g., numbers, timestamps, categories) along with unit and formatting conventions. We then render U into a concrete but virtual knowledge source S ∗ i based on the original Si:

We enforce a synthetic-entity policy: all named entities are sampled from a reserved namespace (e.g., P1 and P2 in Figure 2) to minimize overlap with real-world facts.

QA pair synthesis with pattern inheritance. Given the virtual S r i and the difficulty card Ψi, we synthesize a new question-answer pair while explicitly inheriting the original reasoning bottleneck and trigger. Here, we use a two-step LLM procedure: (i) generate a reasoning skeleton that operationalizes Ψi into a minimal sequence of required reasoning operations and a concrete trap injection (e.g., an ambiguity/distractor that targets the bottleneck step), and (ii) render the skeleton into a natural-language question whose gold answer is uniquely derivable from S ∗ i .

Rather than asking an LLM to directly generate a new question, we first generate an explicit reasoning skeleton Gi that operationalizes Ψi into minimal, checkable components: required evidence fields/records in S ∗ i , the bottleneck operation to be executed (as stated in Ψi), a trap injection consistent with the trigger (a distractor that yields a plausible wrong answer if the bottleneck is skipped), and a short gold reasoning trace that points to concrete evidence. Formally,

We constrain the LLM to (a) mention only entities/fields appearing in S ∗ i , (b) include at least one distractor consistent with the trigger in Ψi such that skipping the bottleneck step leads to a plausible but incorrect answer, and (c) ensure A∗i is supported by an explicit reasoning trace grounded in S ∗ i .

3.3 Multi-criterion Gating

In this stage, we perform multi-criterion gating to control the quality of synthesized instances. For each candidate instance xi = (Si,Qi,Ai,Ψi), we query two LLMbased reviewers, each prompted to return a structured verdict together with a calibrated confidence score. Specifically, we use: (1) an Answerability reviewer, which verifies that the question is answerable under the intended stress type and that the gold answer is properly grounded-i.e., supported by explicit spans/records in Si for reasoning-stress, or demonstrably requiring a missing external fact beyond Si for knowledge-stress; and (2) a Consistency reviewer, which attempts to solve the instance, checks internal consistency among Si,Qi,Ai, and verifies that the difficulty card Ψi is actually instantiated rather than merely stated.

Each reviewer outputs a binary decision with confidence. We then aggregate their outputs into two binary gating signals: gi ∈ {0, 1} indicating answerability, and ui ∈ {0, 1}

indicating consistency, keep(xi) = I[gi = 1] · I[ui = 1], Notably, for knowledge-stress instances, grounding instead verifies the dependency on the missing-fact black box (i.e., solving Qi necessarily requires resolving ∆K in addition to any context-grounded fact), and this dependency check is incorporated into gi.

4 DYNAMIC-ONEEVAL

4.1 Seed Data & Implementation Details

To initialize DYNAMIC-ONEEVAL, we draw seed instances from widely used benchmarks. a) Text Reasoning: HotpotQA [Yang et al., 2018] is a Wikipedia-based multi-hop QA benchmark that requires combining evidence across multiple documents to answer. b) KG Reasoning: ComplexWebQ [Talmor and Berant, 2018] is a compositional KBQA benchmark designed to answer complex questions by executing multi-relation queries over a knowledge graph. c) Table Reasoning: WikiTableQuestion (WTQ) [Pasupat and Liang, 2015] TabFact is a table-based fact verification dataset that asks whether a natural-language statement is entailed or contradicted by a given Wikipedia table.

We generated seed failure cases E by prompting GPT5.2 [OpenAI, 2025], a strong LLM , to produce chain-ofthought [Wei et al., 2022] answers and then verifying its outputs to collect instances where the model erred. For endto-end quality control, the strong GPT-5.2 model was used throughout the StressEval pipeline for fana, fcan, fext, fknow, fU, fU. All API calls used deterministic decoding with temperature was set to 0.0, top-p was set to 0.95, and a maximum token limit was set to 2048. For each bad case e ∈ E, StressEval will synthesize 5 ∼ 10 instances. In addition, we do not synthesize knowledge-stress instances for table reasoning, since the case of the WTQ dataset already contain all evidence required to answer the questions.

4.2 Benchmark Statistic

Table 1 reports statistics of the constructed dataset. For each split, we provide the number of synthesized instances |Ω∗ |, the size of the seed dataset |Ω|, the size of the failure-case set |E|, the average context and question lengths, the mean token number of S ∗ and Q∗ (reported as avg |S ∗ | and avg |Q∗ |), the number of root causes |Γ|, and the token-level similarity between the synthesized and original questions, Sim(Q, Q∗ ). Overall, STRESSEVAL generates a large number of challenging instances |Ω| from a relatively small collection of failure cases |E|, demonstrating strong leverage of limited supervision. Moreover, in principle, with a continual stream of new failure cases, the construction process can keep producing arbitrarily many difficult instances, i.e., the dataset size is unbounded as the failure-case pool grows.

4.3 Diversity of Root Cases

Figure 3 shows the distribution of root-cause labels γ ∈ Γ for both the original failure set E and the synthesized DYNAMICONEEVAL. The inner ring indicates coarse-grained rootcause groups, while the outer ring breaks them down into the corresponding fine-grained categories. Overall, the synthesized benchmark largely preserves the diversity of error types present in the collected failures, and it also introduces modest expansion into additional categories. Notably, the label distribution in DYNAMIC-ONEEVAL is more balanced than in the seed failure set, reducing skew toward a few dominant root causes and yielding broader coverage of fine-grained model weaknesses.

4.4 Human Evaluation

We conducted a lightweight human evaluation to assess the quality of DYNAMIC-ONEEVAL. We randomly sampled 50 instances spanning both knowledge-stress Γk and reasoningstress Γr, covering diverse root-cause labels γ ∈ Γ, and asked three annotators to provide judgments along three dimensions. First, answerability (A) evaluates whether the instance can be resolved from the provided context: for Γr, whether the gold answer A is supported by explicit context in S; for Γk, whether solving the question Q necessarily requires an external fact missing from S. Second, unambiguity (U) assesses whether a unique gold answer exists. Third, faithfulness to difficulty cards (F) measures whether the difficulty card Ψ is correctly instantiated, meaning that the specified bottleneck step and trigger are present in the instance. The results are reported in the right panel of Table 1. Overall, human judgments show consistently high A and U across splits, indicating that instances are generally solvable from the provided context and admit a unique gold answer. In contrast, F exhibits larger variance, with reasoning-stress splits showing notably lower faithfulness (e.g., 88.0% ∼ 96.0%) than knowledge-stress splits (up to 98.0%), suggesting that mismatches more often arise from imperfect instantiation of the intended bottleneck and trigger rather than from missing fact or ambiguity.

5 Experiments

5.1 Experimental Setup

We evaluate a set of representative open-source and proprietary LLMs on DYNAMIC-ONEEVAL, covering the Llama3.1 series [Grattafiori et al., 2024] (Llama3.1-8B and Llama3.170B), the Qwen family [Yang et al., 2024; Yang et al., 2025] (Qwen2.5-72B and Qwen3-235B) and QWQ-32B [QwenTeam, 2025], as well as proprietary models including GPT-5.2 [OpenAI, 2025], Gemini3-pro [Google, 2025], DeepSeek-V3.2 [Liu et al., 2025], Claude-Sonnet-4.5 [Anthropic, 2025], and Doubao-Seed-1.6 [ByteDance, 2025]. To ensure a fair comparison, we apply a unified prompting template across all models. We use exact-match accuracy as the primary metric, reporting results by knowledge type (Text, KG, and Table) and stress type (K-Stress/R-Stress), together with per-type averages and an overall aggregated score.

5.2 Main Results

Table 2 reports the performance of different LLMs on DYNAMIC-ONEEVAL. The results show that DYNAMICONEEVAL is genuinely hard and far from saturated: even the strongest proprietary model (Gemini3-pro) reaches only 48.2% overall, and the best open-source model (QWQ-32B) achieves 39.8%, with substantial error persisting across all knowledge types. This indicates that our failure-driven generation goes beyond superficial adversarial trickery, producing instances that are both challenging and unambiguous, and that reliably surface brittleness even in frontier LLMs, particularly on knowledge-intensive failure modes.

A consistent diagnostic pattern is that K-Stress is the dominant bottleneck, most severely in text reasoning: open-source models remain near-floor on Text K-Stress despite noticeably higher Text R-Stress, and although proprietary models improve (Gemini3-pro achieves 29.8%), most still hover around 10 on text K-Stress. In KG reasoning, R-Stress is comparatively high and clustered, while K-Stress drops sharply, indicating that graphs can skeleton compositional reasoning but cannot compensate when crucial knowledge is stressed or withheld. Finally, table reasoning yields high R-Stress for several models (e.g., Claude-Sonnet-4.5 achieves 80.9%, Gemini3-pro achieves 75.5%), yet overall scores remain limited because knowledge-stress failures persist, reinforcing DYNAMIC-ONEEVAL as a targeted, controlable stress test rather than a knowledge type-specific formatting challenge.

5.3 Ablation Studies

We fix GPT-5.2 as the synthesis backbone, generate instances under each ablated setting, and evaluate the resulting datasets using DeepSeek V3.2, together with human judgments on the corresponding instances. Here, we consider three ablation groups. Pipeline-level ablations remove either (a) error analysis, to test whether bottleneck identification and trigger localization are necessary for controllable synthesis, or (b) criterion gating, to quantify its role in enforcing grounding, answerability, unambiguity, and difficulty preservation. Knowledge-stress ablations remove either (c) the missing-fact black box ∆K, to test whether explicitly modeling atomic missing facts stabilizes reliance on external knowledge, or (d) context freezing, allowing S to be rewritten and potentially filling the intended knowledge gap. Reasoning-stress ablations remove either (e) virtual context S ∗ construction, to assess whether entity virtualization prevents parametric-memory shortcuts, or (f) the reasoning skeleton, to test whether explicitly instantiating the bottleneck and injecting traps improves faithfulness and grounding.

Table 3 reports the ablation results for STRESSEVAL, showing that its components must operate in concert to preserve both difficulty and validity. At the pipeline level, removing error analysis substantially degrades performance (e.g., DeepSeek V3.2 drops from 27.7% overall in Table 2) and sharply reduces human-rated faithfulness. Removing criterion gating also consistently lowers human faithfulness, confirming its role in enforcing grounding, unambiguity, and difficulty preservation. For K-stress, disabling the missingfact black box K produces the most severe failures, while unfreezing S weakens the intended knowledge gap, consistent with context leakage. For R-stress, removing the virtual context S ∗ or the two-step reasoning-skeleton procedure yields instances that are noticeably easier and less faithful, supporting the claim that virtualization mitigates parametric-memory shortcuts and that an explicit skeleton enforces context-grounded reasoning.

5.4 Performance on Different Root Causes

Figure 4 compares models across fine-grained root error causes; due to space constraints, we report representative categories. Performance is highly errordependent: frontier proprietary models excel on structured temporal/compositional skills. For instance, ClaudeSonnet-4.5 achieves 95% on Temporal Ordering, whereas gaps widen for failures requiring robust entity grounding and multi-step control. For instance, Partial Entity Recognition Then Reason falls from 85% (Claude-4.5) to 15% (DeepSeekV3.2). Across models, Evidence Omission and Distractor Anchoring remain difficult, suggesting that brittleness stems less from surface retrieval than from evidence use under adversarial distractions.

5.5 Impact of Different Backbones

To assess STRESSEVAL's robustness to the backbone, we reran the pipeline with several lightweight/low-cost LLMs as generators and evaluated the resulting instances using representative LLMs. Figure 5 summarizes the results. Model rankings are largely stable across backbones, suggesting limited generator-induced bias, while the backbone mainly affects absolute difficulty: Llama-4-scout generally yields harder cases (lower scores) and Gemini2.5 tends to yield easier ones (higher scores) across all three settings. Notably, Gemini3 performs particularly well on Gemini2.5-generated cases, consistent with potential within-family transfer effects.

6 RelatedWork

Knowledge-Intensive Reasoning Benchmarks Early success of LLMs, exemplified by models like GPT-4 [OpenAI,

2023] and DeepSeek [Liu et al., 2024], demonstrated impressive capabilities across a spectrum of language understanding and reasoning tasks. However, conventional static evaluation practices have struggled to keep pace with the rapid evolution of these models. Issues such as benchmark contamination [Deng et al., 2024; Dong et al., 2025], where models may memorize test data present in their vast training corpora, lead to an inflated sense of performance and obscure genuine generalization abilities [Lin et al., 2024b]. This problem is particularly acute in knowledge-intensive settings [Chen et al., 2025a; Chen et al., 2022], where correctness relies on up-to-date facts, strict grounding, and robust multi-step inference, rather than superficial pattern matching [Jiang et al., 2024]. The need for benchmarks that can genuinely assess an LLM's ability to reason with and apply external knowledge, rather than merely recall memorized patterns, remains a significant challenge.

Dynamic LLM Benchmarks Automatic benchmarking approaches [Wang et al., 2025] broadly span four categories: static human-curated suites, LLM-as-judge frameworks [Zheng et al., 2023], agent-driven automated arenas [Zhao et al., 2025], and in-the-wild benchmarks mined from real interactions [Lin et al., 2024a]. While each improves scalability or realism, they share complementary weaknesses: static benchmarks saturate and risk contamination; mined sets lack controllable latent factors for causal stress tests [Valiulla, 2025]; and fully automated pipelines may generate unreliable or causally unfaithful cases [Shukla, 2025]. Model-in-the-loop and adversarial generation frameworks [White et al., 2024; Li et al., 2024] help surface brittle heuristics but, without strict answerability constraints, risk drifting toward underspecified or spurious examples.

7 Conclusion

We present STRESSEVAL, a failure-driven pipeline that converts observed LLM failures into dynamic, answerable, and high-difficulty evaluation instances through structured error analysis, dual-perspective instance synthesis, and multicriterion gating. Instantiated from knowledge-intensive seed datasets, STRESSEVAL produces grounded and controlable tests that induce substantially larger, and more diagnostic, performance drops across frontier models than standard static benchmarks. Future work will investigate how different combinations of LLMs within the same STRESSEVAL framework affect data generation, e.g., using heterogeneous generator-critic-verifier ensembles, to improve diversity, robustness, and controllability.

Acknowledgements

This work is supported by National Nature Science Foundation of China under No.62506071, No.6250072425, and No.62476058. We thank the Big Data Computing Center of Southeast University for providing the facility support on the numerical calculations in this paper.