A Meta-Game Evaluation Framework for Deep Multiagent Reinforcement Learning

A Meta-Game Evaluation Framework for Deep Multiagent Reinforcement Learning

Zun Li, Michael P. Wellman

Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence
Main Track. Pages 148-156. https://doi.org/10.24963/ijcai.2024/17

Evaluating deep multiagent reinforcement learning (MARL) algorithms is complicated by stochasticity in training and sensitivity of agent performance to the behavior of other agents. We propose a meta-game evaluation framework for deep MARL, by framing each MARL algorithm as a meta-strategy, and repeatedly sampling normal-form empirical games over combinations of meta-strategies resulting from different random seeds. Each empirical game captures both self-play and cross-play factors across seeds. These empirical games provide the basis for constructing a sampling distribution, using bootstrapping, over a variety of game analysis statistics. We use this approach to evaluate state-of-the-art deep MARL algorithms on a class of negotiation games. From statistics on individual payoffs, social welfare, and empirical best-response graphs, we uncover strategic relationships among self-play, population-based, model-free, and model-based MARL methods. We also investigate the effect of run-time search as a meta-strategy operator, and find via meta-game analysis that the search version of a meta-strategy generally leads to improved performance.
Keywords:
Agent-based and Multi-agent Systems: MAS: Multi-agent learning
Game Theory and Economic Paradigms: GTEP: Noncooperative games