SteelAgent: An LLM-Orchestrated System for Physics-Informed Steel Property Prediction and Generalization Auditing

1 Introduction

Machine learning models for predicting mechanical properties of steels have achieved impressive accuracy on standard benchmarks [Wen et al., 2019; Xiong et al., 2020; Choudhary et al., 2022]. Gradient-boosted models routinely report R2 > 0.85 for tensile strength prediction, and these numbers are cited to justify deployment in alloy design and qualification pipelines. In practice, however, engineers need models that generalize to new steel grades compositions never seen during training because the primary value of a predictive model lies in reducing the cost and time of designing and qualifying novel alloys.

Unfortunately, high accuracy demonstrated in steel property predictions on random splits often reflects data leakage rather than genuine predictive capability [Kapoor and Narayanan, 2023; Kapoor and Narayanan, 2023]. The root cause is compositional similarity leakage: random train/test splits allow nearly identical steel grades (e.g., AISI 4340 and AISI 4330, which differ by only 0.10% of carbon) belonging to the same family to appear in both sets. The models memorize grade-specific composition bundles instead of learning the underlying metallurgical relationships between chemistry, heat treatment, and mechanical response.

When evaluated on unseen steel families via Leave-OneFamily-Out (LOFO) splitting, where entire categories such as "stainless austenitic" or "CrMo quenched-and-tempered" are held out, the gradient boosting regression quality drops from R2 = 0.86 to R2 = 0.11 representing more than 7.6 times degradation as illustrated in Fig. 1. For safety-critical applications where material qualification decisions depend on predicted properties (aircraft landing gear, nuclear pressure vessels, offshore pipelines) this gap between reported and realworld performance has direct engineering consequences imposing substantial risk for infrastructure and personnel.

We present SteelAgent, an interactive system addressing this gap through three key contributions:

• No-leak evaluation: comparison of four split protocols, exposing that conformal coverage degrades from 91% to 38% under distribution shift [Tibshirani et al., 2019].
• Physics-grounded LLM orchestration: an agent that enforces tool-mediated reasoning [Yao et al., 2023; Schick et al., 2023], computing physical quantities (Ms martensite start temperature, CE carbon equivalent for weldability, DI ideal critical diameter for hardenability) via verified formulas rather than LLM generation.
• End-to-end decision support: from natural-language queries to specification compliance probabilities P(pass) and cost-aware inverse design.

2 SystemArchitecture

SteelAgent 1 follows a three-layer architecture (see Fig. 2): a physics layer that computes domain-specific features from classical metallurgical formulas, a model layer that runs machine learning algorithms with conformal uncertainty quantification, and an LLM orchestration layer that coordinates these components through structured function calling. The system operates over 3,741 heat treatment records spanning 1,234 grades from 27 independent sources (see Tab. 1), with pre-trained models (XGBoost, LightGBM, CatBoost, Random Forest) for two target properties: ultimate tensile strength (UTS is the maximum stress a material can withstand before fracture) and yield strength (YS is the stress at which the material begins to deform permanently). These two properties are the most commonly specified in industrial standards and drive material selection decisions.

1 Demo video: https://www.youtube.com/watch?v= BwVBJ-SwuQo. Live demo: https://steelagent.vercel.app

Physics-Informed Features. A key challenge for AI models in metallurgy is that raw composition vectors (e.g., 0.40% C, 0.80% Cr) do not directly encode the physical mechanisms that determine mechanical properties. Two steels with similar compositions can behave very differently depending on how their alloying elements interact during phase transformations. To bridge this gap, SteelAgent computes 18 physics-derived features from 11 raw inputs (C, Mn, Si, Cr, Ni, Mo, V, Cu, Al, Taust, Ttemper). These features can be grouped as: (i) Martensite start temperature Ms = 539 - 423C-30.4Mn-12.1Cr-17.7Ni-7.5Mo+10Si [Andrews, 1965] predicts when martensite, the hard phase responsible for strength, begins to form during quenching; (ii) Carbon equivalent CEIIW =C+ Mn 6 + Cr+Mo+V 5 + Ni+Cu 15 summarizes the combined effect of all alloying elements on weldability and hardenability into a single scalar; (iii) Ideal critical diameter DI [Grossmann and Bain, 1964] the maximum bar diameter that achieves full hardening at center quantifies how deeply a steel can be hardened, a key property for large structural components; (iv) Ac1/Ac3 transformation temperatures, superheat ∆T =Taust-Ac3, tempering resistance index, and maximum martensite hardness. These features serve a dual purpose: they improve Leave-One-Family-Out R2 by +0.25 for gradient boosting regression (see Fig. 3), and they ground the LLM orchestrator in verified physical quantities, preventing hallucination of numerical values.

Uncertainty Quantification. In engineering practice, a point prediction alone (e.g., "UTS=1,746 MPa") is insufficient for decision-making: the engineer needs to know how confident the model is. A prediction interval [ˆ ylo, ˆ yhi] answers this directly. We apply split conformal prediction [Vovk et al., 2005; Romano et al., 2019; Angelopoulos and Bates, 2023] to construct distribution-free prediction intervals with guaranteed ≥90%marginal coverage under exchangeability meaning that at least 90% of true values fall within the predicted interval, with no distributional assumptions on the data.

For specification compliance, we convert conformal intervals to pass probabilities via Gaussian approximation: σ = ˆ q/z0.95, then P(pass) =Φ((ˆ y-ymin)/σ), where ymin is the specification minimum. This gives the engineer a single number e.g., P(pass AMS-6414)=0.93 that directly supports material qualification decisions. Additional unlabeled data may further help with algorithms quality [Amini et al., 2025; Maximov et al., 2018].

LLM Orchestration. A key design decision is to separate reasoning (what to compute and how to interpret results) from computation (the actual physics and ML calculations). The LLM handles the former; the latter is delegated to verified tools. Following the ReAct paradigm [Yao et al., 2023] and tool-augmented LLM agents [Schick et al., 2023; Bran et al., 2024], SteelAgent implements a multi-turn functioncalling loop over six tools: lookup preset (58 reference grades with standard compositions), compute physics (18 derived features), predict property (ML prediction + conformal UQ), check spec (P(pass) for 51 industrial specifications), compare models (4-model consensus with disagreement detection), and inverse design (cost-aware composition search). The system prompt enforces a physics-first pipeline: compute physics must precede predict property, and the LLMmust cite toolcomputed values in its natural-language synthesis. This constraint ensures that physical quantities (e.g., Ms = 305°C) originate from verified formulas rather than LLM generation, while the LLM provides metallurgical context and interpretation that a non-specialist can understand.

3 Quantitative Results and Demo Scenarios

Table 2 presents two critical findings:

• The gap is structural, not statistical: the drop from Random to LOFO is consistent across models (CatBoost: 0.86 → 0.35; GBR: 0.86 → 0.11) and persists even as training data increases (Fig. 3), indicating that additional data alone cannot cover physics-informed features.
• UQ coverage tracks the gap. Conformal coverage (guaranteed ≥ 90% under exchangeability [Angelopoulos and Bates, 2023]) degrades proportionally to the distribution shift: 91%→83%→76%→38%. This pattern is itself a useful diagnostic: coverage can serve as an automated shift detector, flagging predictions where the exchangeability assumption no longer valid [Tibshirani et al., 2019].

SteelAgent surfaces this information interactively, helping users decide when to trust a prediction.

SteelAgent supports three primary interaction scenarios via a conversational React interface with guided dialogue trees. Users interact in natural language; the system autonomously selects and sequences tool calls, presenting results with physics context and uncertainty.

Scenario 1: Prediction + Specification Compliance. In this scenario, our task is to predict UTS for AISI 4340 (Q&T 845/315°C) and check AMS-6414 compliance. AISI 4340 is a widely used aerospace steel (landing gear, crankshafts, high-strength fasteners). The orchestrator executes a four-tool pipeline: lookup preset → compute physics (Ms = 305°C, CE = 0.85, DI = 10.7 in) → predict property (UTS= 1,746 MPa, 90% CI [1,546, 1,945] MPa) → check spec (P(pass AMS-6414)). The user sees physics context (what transformation behavior to expect), a prediction with calibrated uncertainty (how much to trust it), and a compliance probability (whether the steel will meet the specification).

Scenario 2: Competitive Steel Comparison. In this experiment, we compare AerMet 100 and Ferrium M54 for landing gear redesign. These are premium ultra-high-strength steels

a [Dunn et al., 2020] b [Choudhary et al., 2022] c [Boiko et al., 2023] d [Bran et al., 2024]

Table 3: Feature comparison with materials ML platforms and scientific LLM agents. ✓= supported, ✗= not supported, ∼ = partial.

with different design philosophies. The system predicts properties for both grades (UTSAerMet =1,927 MPa; Ms =192°C, DI = 47 in) and explains trade-offs through physics signatures: AerMet achieves peak strength via M2C secondary hardening enabled by 13.4% cobalt, while Ferrium offers better stress-corrosion cracking resistance. This type of comparative analysis typically requires a senior metallurgist and extensive literature review; the system delivers it in seconds with auditable tool calls.

Scenario 3: Inverse Alloy Design We look for compositions achieving UTS> 1,800 MPa at minimum alloy cost. The system screens 10,000 candidate compositions within physically feasible bounds, evaluating each through the ML pipeline. Results are ranked by estimated alloy cost and include conformal intervals, enabling distinction between alloys that reliably vs. possibly achieve the target a critical distinction for qualification.

Additional capabilities include SHAP-based feature importance [Lundberg and Lee, 2017] with metallurgical interpretation (e.g., identifying which alloying elements drive a specific prediction), multi-model consensus with automatic disagreement detection across 4 models, and interactive dataset exploration across 12 steel families and 27 sources.

4 Positioning

Table 3 positions SteelAgent relative to existing materials informatics platforms and scientific LLM agents.

Materials benchmarks. Matbench [Dunn et al., 2020] and Matminer [Ward et al., 2018] provide standardized benchmarks for materials property prediction, but focus on crystallevel properties rather than alloy-level mechanical properties with heat-treatment dependence. SteelAgent addresses this gap with steel-specific evaluation and an interactive interface.

Evaluation pitfalls. [Murdock et al., 2020] [Murdock et al., 2020] show that domain features provide marginal benefit under random splits, while [Kapoor and Narayanan, 2023] [Kapoor and Narayanan, 2023] document leakage across scientific ML. SteelAgent makes this problem interactive and extends it to uncertainty quantification.

Scientific LLM agents. Coscientist [Boiko et al., 2023] and ChemCrow [Bran et al., 2024] demonstrate domainspecific LLM agents for chemistry, [Sultimov et al., 2026b; Sultimov et al., 2026a] for climate and transportation. LLMbased multi-agent systems [Guo et al., 2024] provide theoretical foundations. SteelAgent contributes a physics-grounded agent architecture where tool calls enforce correctness of numerical claims, combined with calibrated UQ and generalization auditing features absent from prior scientific agents.

Acknowledgments

The work of Roman Sultimov is supported by the Ministry of Economic Development of the Russian Federation in accordance with the subsidy agreement (agreement identifier 000000C313925P4H0002; grant No 139-15-2025-012).