Proceedings

Learning Nash Equilibria in Zero-Sum Stochastic Games via Entropy-Regularized Policy Approximation

Yue Guan, Qifan Zhang, Panagiotis Tsiotras

Download BibTeX Cite

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. Main Track. Pages 2462-2468.

https://doi.org/10.24963/ijcai.2021/339

Keywords

  • Machine Learning: Reinforcement Learning
  • Agent-based and Multi-agent Systems: Noncooperative Games
  • Agent-based and Multi-agent Systems: Multi-agent Learning

Abstract

We explore the use of policy approximations to reduce the computational cost of learning Nash equilibria in zero-sum stochastic games. We propose a new Q-learning type algorithm that uses a sequence of entropy-regularized soft policies to approximate the Nash policy during the Q-function updates. We prove that under certain conditions, by updating the entropy regularization, the algorithm converges to a Nash equilibrium. We also demonstrate the proposed algorithm's ability to transfer previous training experiences, enabling the agents to adapt quickly to new environments. We provide a dynamic hyper-parameter scheduling scheme to further expedite convergence. Empirical results applied to a number of stochastic games verify that the proposed algorithm converges to the Nash equilibrium, while exhibiting a major speed-up over existing algorithms.

Yue Guan, Qifan Zhang, and Panagiotis Tsiotras. 2021. Learning Nash Equilibria in Zero-Sum Stochastic Games via Entropy-Regularized Policy Approximation. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. IJCAI, 2462–2468. https://doi.org/10.24963/ijcai.2021/339
Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards