BehaviorBench: A Psychologically Grounded Benchmark for Evaluating Personality in Large Language Models Through Realistic Behaviors

Pu, Taowen; Wang, Hexi; Liu, Zeyang; Guo, Dongsheng; Zhao, Chuan

doi:10.24963/ijcai.2026/658

BehaviorBench: A Psychologically Grounded Benchmark for Evaluating Personality in Large Language Models Through Realistic Behaviors

Taowen Pu, Hexi Wang, Zeyang Liu, Dongsheng Guo, Chuan Zhao

Download BibTeX Cite

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. Main Track. Pages 5910-5918.

https://doi.org/10.24963/ijcai.2026/658

Keywords

Natural Language Processing: Resources and evaluation
Natural Language Processing: Psycholinguistics

Abstract

Current approaches to evaluating personality in large language models (LLMs) typically prompt them to self-report on psychological questionnaires such as the Big Five Inventory. However, these methods assess introspective labels rather than observable behavior, despite the fact that LLMs are deployed to act in realistic contexts, not to reflect on their own traits. To bridge this gap, we introduce BehaviorBench, a new benchmark for evaluating LLM personality through concrete behaviors in everyday scenarios. Grounded in established personality psychology, BehaviorBench links each Big Five trait to validated behavioral manifestations and embeds them in contextually plausible situations that naturally elicit trait-relevant actions. We evaluate a range of models and personality shaping strategies using BehaviorBench and find a substantial mismatch between self-reported personalities and actual behaviors. By grounding evaluation in observable behavior rather than introspection, our work reveals critical gaps in current LLM personality modeling and control mechanisms. Our code and data are available at https://github.com/butra1n/BehaviorBench

Taowen Pu, Hexi Wang, Zeyang Liu, Dongsheng Guo, and Chuan Zhao. 2026. BehaviorBench: A Psychologically Grounded Benchmark for Evaluating Personality in Large Language Models Through Realistic Behaviors. In Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26. IJCAI, 5910–5918. https://doi.org/10.24963/ijcai.2026/658

@inproceedings{ijcai2026p658,
  title     = {BehaviorBench: A Psychologically Grounded Benchmark for Evaluating Personality in Large Language Models Through Realistic Behaviors},
  author    = {Pu, Taowen and Wang, Hexi and Liu, Zeyang and Guo, Dongsheng and Zhao, Chuan},
  booktitle = {Proceedings of the Thirty-Fifth International Joint Conference on
               Artificial Intelligence, {IJCAI-26}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Diego Calvanese},
  pages     = {5910--5918},
  year      = {2026},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2026/658},
  url       = {https://doi.org/10.24963/ijcai.2026/658},
}

TY  - CPAPER
TI  - BehaviorBench: A Psychologically Grounded Benchmark for Evaluating Personality in Large Language Models Through Realistic Behaviors
AU  - Pu, Taowen
AU  - Wang, Hexi
AU  - Liu, Zeyang
AU  - Guo, Dongsheng
AU  - Zhao, Chuan
T2  - Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2026
SP  - 5910
EP  - 5918
DO  - 10.24963/ijcai.2026/658
UR  - https://doi.org/10.24963/ijcai.2026/658
ER  -