{"type":"ScholarlyArticle","title":"Learning Nash Equilibria in Zero-Sum Stochastic Games via Entropy-Regularized Policy Approximation","authors":[{"name":"Yue Guan","slug":"yue-guan","given":"Yue","family":"Guan","affiliation":""},{"name":"Qifan Zhang","slug":"qifan-zhang","given":"Qifan","family":"Zhang","affiliation":""},{"name":"Panagiotis Tsiotras","slug":"panagiotis-tsiotras","given":"Panagiotis","family":"Tsiotras","affiliation":""}],"abstract":"We explore the use of policy approximations to reduce the computational cost of learning Nash equilibria in zero-sum stochastic games. We propose a new Q-learning type algorithm that uses a sequence of entropy-regularized soft policies to approximate the Nash policy during the Q-function updates. We prove that under certain conditions, by updating the entropy regularization, the algorithm converges to a Nash equilibrium. We also demonstrate the proposed algorithm's ability to transfer previous training experiences, enabling the agents to adapt quickly to new environments. We provide a dynamic hyper-parameter scheduling scheme to further expedite convergence. Empirical results applied to a number of stochastic games verify that the proposed algorithm converges to the Nash equilibrium, while exhibiting a major speed-up over existing algorithms.","doi":"10.24963/ijcai.2021/339","url":"https://ijcai.org/proceedings/2021/339","pdf":"https://ijcai.org/proceedings/2021/0339.pdf","markdown":"https://ijcai.org/proceedings/2021/339.md","bibtex":"https://ijcai.org/proceedings/2021/bibtex/339","volume":{"slug":"2021","title":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2021","url":"https://ijcai.org/proceedings/2021/","year":2021,"location":"Montreal","dates":"19-27 August 2021","editors":"Prof. Zhi-Hua Zhou","isbn":"978-0-9992411-9-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2021"},"section":"Machine Learning","track":"Main Track","number":339,"pages":{"first":2462,"last":2468},"keywords":["Machine Learning: Reinforcement Learning","Agent-based and Multi-agent Systems: Noncooperative Games","Agent-based and Multi-agent Systems: Multi-agent Learning"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}