ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution

Wang, Yubang; Zhang, Chenxi; Chen, Bowen; Huai, Zezheng; Dai, Zihao; Chen, Xinchi; Wang, Yuxin; Zheng, Yining; Gong, Jingjing; Qiu, Xipeng

doi:10.24963/ijcai.2026/48

ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution

Yubang Wang, Chenxi Zhang, Bowen Chen, Zezheng Huai, Zihao Dai, Xinchi Chen, Yuxin Wang, Yining Zheng, Jingjing Gong, Xipeng Qiu

Download BibTeX Cite

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. Main Track. Pages 419-427.

https://doi.org/10.24963/ijcai.2026/48

Keywords

Agent-based and Multi-agent Systems: Applications

Abstract

Autonomous agents are increasingly expected to support scientific research, and recent benchmarks report progress in code repair and autonomous experimentation. However, these evaluations typically assume a pre-configured execution environment, which requires resolving complex software dependencies, aligning hardware and framework versions, and configuring distributed execution, yet this capability remains largely unbenchmarked. We introduce ResearchEnvBench, a benchmark for environment synthesis in research code execution. Given a research repository, documentation, and a target execution setting, agents must construct an environment that successfully executes at runtime. Evaluations on diverse research repositories reveal a substantial gap in current SOTA agents, with failures dominated by incomplete dependency resolution and brittle version coupling. ResearchEnvBench provides a realistic testbed for advancing autonomous agents toward reproducible scientific research.

Yubang Wang, Chenxi Zhang, Bowen Chen, Zezheng Huai, Zihao Dai, Xinchi Chen, Yuxin Wang, Yining Zheng, Jingjing Gong, and Xipeng Qiu. 2026. ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution. In Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26. IJCAI, 419–427. https://doi.org/10.24963/ijcai.2026/48

@inproceedings{ijcai2026p48,
  title     = {ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution},
  author    = {Wang, Yubang and Zhang, Chenxi and Chen, Bowen and Huai, Zezheng and Dai, Zihao and Chen, Xinchi and Wang, Yuxin and Zheng, Yining and Gong, Jingjing and Qiu, Xipeng},
  booktitle = {Proceedings of the Thirty-Fifth International Joint Conference on
               Artificial Intelligence, {IJCAI-26}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Diego Calvanese},
  pages     = {419--427},
  year      = {2026},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2026/48},
  url       = {https://doi.org/10.24963/ijcai.2026/48},
}

TY  - CPAPER
TI  - ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution
AU  - Wang, Yubang
AU  - Zhang, Chenxi
AU  - Chen, Bowen
AU  - Huai, Zezheng
AU  - Dai, Zihao
AU  - Chen, Xinchi
AU  - Wang, Yuxin
AU  - Zheng, Yining
AU  - Gong, Jingjing
AU  - Qiu, Xipeng
T2  - Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2026
SP  - 419
EP  - 427
DO  - 10.24963/ijcai.2026/48
UR  - https://doi.org/10.24963/ijcai.2026/48
ER  -