Soft Condorcet Optimization for Ranking of General Agents (Extended Abstract)
Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. Sister Conferences Best Papers. Pages 8272-8277.
https://doi.org/10.24963/ijcai.2026/925
Keywords
- Game Theory and Economic Paradigms: Computational social choice
- Machine Learning: Optimization
- Machine Learning: Evaluation
- AI: Agent-based and Multi-agent Systems
Abstract
Driving progress of AI models and agents requires comparing their performance on standardized benchmarks; for general agents, individual performances must be aggregated across a potentially wide variety of different tasks. In this extended abstract, we describe a ranking scheme inspired by social choice frameworks, called Soft Condorcet Optimization (SCO), to compute the optimal ranking of agents: the one that makes the fewest mistakes in predicting the agent comparisons in the evaluation data. This optimal ranking is the maximum likelihood estimate when evaluation data (which we view as votes) are interpreted as noisy samples from a ground truth ranking, a solution to Condorcet's original voting system criteria. SCO ratings are maximal for Condorcet winners when they exist, which we show is not necessarily true for the classical rating system Elo. In practice, SCO serves as an accurate approximation to the Kemeny-Young voting method, excels in the sparse data regime, and provides the best approximation to the optimal ranking compared to every baseline on a Diplomacy player ranking problem with 31,094 games and 52,958 agents.
@inproceedings{ijcai2026p925,
title = {Soft Condorcet Optimization for Ranking of General Agents (Extended Abstract)},
author = {Lanctot, Marc and Larson, Kate and Kaisers, Michael and Berthet, Quentin and Gemp, Ian and Diaz, Manfred and Maura-Rivero, Roberto-Rafael and Bachrach, Yoram and Koop, Anna and Precup, Doina},
booktitle = {Proceedings of the Thirty-Fifth International Joint Conference on
Artificial Intelligence, {IJCAI-26}},
publisher = {International Joint Conferences on Artificial Intelligence Organization},
editor = {Diego Calvanese},
pages = {8272--8277},
year = {2026},
month = {8},
note = {Sister Conferences Best Papers},
doi = {10.24963/ijcai.2026/925},
url = {https://doi.org/10.24963/ijcai.2026/925},
}
TY - CPAPER TI - Soft Condorcet Optimization for Ranking of General Agents (Extended Abstract) AU - Lanctot, Marc AU - Larson, Kate AU - Kaisers, Michael AU - Berthet, Quentin AU - Gemp, Ian AU - Diaz, Manfred AU - Maura-Rivero, Roberto-Rafael AU - Bachrach, Yoram AU - Koop, Anna AU - Precup, Doina T2 - Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26 PB - International Joint Conferences on Artificial Intelligence Organization PY - 2026 SP - 8272 EP - 8277 DO - 10.24963/ijcai.2026/925 UR - https://doi.org/10.24963/ijcai.2026/925 ER -Copied
Machine-readable versions of this paper, for tools and AI agents:
