BinMetric: A Comprehensive Binary Code Analysis Benchmark for Large Language Models

Shang, Xiuwei; Chen, Guoqiang; Cheng, Shaoyin; Wu, Benlong; Hu, Li; Li, Gangyang; Zhang, Weiming; Yu, Nenghai

doi:10.24963/ijcai.2025/858

BinMetric: A Comprehensive Binary Code Analysis Benchmark for Large Language Models

Xiuwei Shang, Guoqiang Chen, Shaoyin Cheng, Benlong Wu, Li Hu, Gangyang Li, Weiming Zhang, Nenghai Yu

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 7715-7723.

https://doi.org/10.24963/ijcai.2025/858

Keywords

Multidisciplinary Topics and Applications: MTA: Software engineering
Multidisciplinary Topics and Applications: MTA: Security and privacy
Machine Learning: ML: Benchmarks
Natural Language Processing: NLP: Language models

Abstract

Binary analysis is crucial for software security, offering insights into compiled programs without source code. As large language models (LLMs) excel in language tasks, their potential for complex decoding binary data structures is growing. However, the lack of standardized benchmarks hinders their evaluation and progress in this domain.

To bridge this gap, we introduce BinMetric, a first comprehensive benchmark designed specifically to evaluate LLMs performance on binary analysis tasks. BinMetric comprises 1,000 questions derived from 20 real-world open-source projects across 6 practical binary analysis tasks, including decompilation, code summarization, etc., which reflect actual reverse engineering scenarios. Our empirical study on this benchmark investigates various state-of-the-art LLMs, revealing their strengths and limitations. The findings indicate that while LLMs show strong potential, challenges still exist, particularly in the areas of precise binary lifting and assembly synthesis. In summary, BinMetric makes a significant step forward in measuring binary analysis capabilities of LLMs, establishing a new benchmark leaderboard, and our study offers valuable insights for advancing LLMs in software security.

Xiuwei Shang, Guoqiang Chen, Shaoyin Cheng, Benlong Wu, Li Hu, Gangyang Li, Weiming Zhang, and Nenghai Yu. 2025. BinMetric: A Comprehensive Binary Code Analysis Benchmark for Large Language Models. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. IJCAI, 7715–7723. https://doi.org/10.24963/ijcai.2025/858

@inproceedings{ijcai2025p858,
  title     = {BinMetric: A Comprehensive Binary Code Analysis Benchmark for Large Language Models},
  author    = {Shang, Xiuwei and Chen, Guoqiang and Cheng, Shaoyin and Wu, Benlong and Hu, Li and Li, Gangyang and Zhang, Weiming and Yu, Nenghai},
  booktitle = {Proceedings of the Thirty-Fourth International Joint Conference on
               Artificial Intelligence, {IJCAI-25}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {James Kwok},
  pages     = {7715--7723},
  year      = {2025},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2025/858},
  url       = {https://doi.org/10.24963/ijcai.2025/858},
}

TY  - CPAPER
TI  - BinMetric: A Comprehensive Binary Code Analysis Benchmark for Large Language Models
AU  - Shang, Xiuwei
AU  - Chen, Guoqiang
AU  - Cheng, Shaoyin
AU  - Wu, Benlong
AU  - Hu, Li
AU  - Li, Gangyang
AU  - Zhang, Weiming
AU  - Yu, Nenghai
T2  - Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2025
SP  - 7715
EP  - 7723
DO  - 10.24963/ijcai.2025/858
UR  - https://doi.org/10.24963/ijcai.2025/858
ER  -