---
title: "Learning Nash Equilibria in Zero-Sum Stochastic Games via Entropy-Regularized Policy Approximation"
authors:
  - "Yue Guan"
  - "Qifan Zhang"
  - "Panagiotis Tsiotras"
doi: "10.24963/ijcai.2021/339"
url: "https://ijcai.org/proceedings/2021/339"
pdf: "https://ijcai.org/proceedings/2021/0339.pdf"
venue: "Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence"
year: 2021
pages: "2462-2468"
track: "Main Track"
keywords:
  - "Machine Learning: Reinforcement Learning"
  - "Agent-based and Multi-agent Systems: Noncooperative Games"
  - "Agent-based and Multi-agent Systems: Multi-agent Learning"
---

# Learning Nash Equilibria in Zero-Sum Stochastic Games via Entropy-Regularized Policy Approximation

Yue Guan, Qifan Zhang, Panagiotis Tsiotras

*Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence*, pages 2462-2468. DOI: [10.24963/ijcai.2021/339](https://doi.org/10.24963/ijcai.2021/339)

## Abstract

We explore the use of policy approximations to reduce the computational cost of learning Nash equilibria in zero-sum stochastic games. We propose a new Q-learning type algorithm that uses a sequence of entropy-regularized soft policies to approximate the Nash policy during the Q-function updates. We prove that under certain conditions, by updating the entropy regularization, the algorithm converges to a Nash equilibrium. We also demonstrate the proposed algorithm's ability to transfer previous training experiences, enabling the agents to adapt quickly to new environments. We provide a dynamic hyper-parameter scheduling scheme to further expedite convergence. Empirical results applied to a number of stochastic games verify that the proposed algorithm converges to the Nash equilibrium, while exhibiting a major speed-up over existing algorithms.