Proceedings

MathCritique: Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision

Zhiheng Xi, Dingwen Yang, Jixuan Huang, Jiafu Tang, Xin Guo, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong, Shihan Dou, WenYu Zhan, Xiao Wang, Xiaowei Shi, Yitao Zhai, Rongxiang Weng, Jingang Wang, Rui Zheng, Tao Ji, Tao Gui, Zuxuan Wu, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Yu-Gang Jiang

Download BibTeX Cite

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. Main Track. Pages 5982-5990.

https://doi.org/10.24963/ijcai.2026/666

Keywords

  • Natural Language Processing: Language models
  • Natural Language Processing: Language generation

Abstract

Training critique models to provide useful feedback for actor models is an effective approach in scalable oversight, especially for complex tasks like math reasoning. However, current research lacks suitable datasets for effectively training critique models and integrating them in a principled way at both test time and training time. To bridge this gap, we first propose AutoMathCritique, an automated and scalable framework for collecting critique data. Using it, we create MathCritique-76k, a dataset of $76,321$ instances paired with step-level feedback, which is then used to train critique models for generating natural-language feedback. We show that critique models consistently improve the actor’s performance—particularly on difficult queries at test time—with larger gains as inference compute scales. Building on the findings, we propose a critique-in-the-loop self-improvement method that incorporates critique-based supervision into the actor’s self-training process. Extensive experiments demonstrate that this method improves the actor’s exploration efficiency and solution diversity, especially on challenging queries, leading to a stronger actor model. Our code and datasets are at https://mathcritique.github.io/.

Zhiheng Xi, Dingwen Yang, Jixuan Huang, Jiafu Tang, Xin Guo, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong, Shihan Dou, WenYu Zhan, Xiao Wang, Xiaowei Shi, Yitao Zhai, Rongxiang Weng, Jingang Wang, Rui Zheng, Tao Ji, Tao Gui, Zuxuan Wu, Qi Zhang, Xipeng Qiu, Xuanjing Huang, and Yu-Gang Jiang. 2026. MathCritique: Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision. In Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26. IJCAI, 5982–5990. https://doi.org/10.24963/ijcai.2026/666
Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards