{"type":"ScholarlyArticle","title":"On the Convergence of Fictitious Play: A Decomposition Approach","authors":[{"name":"Yurong Chen","slug":"yurong-chen","given":"Yurong","family":"Chen","affiliation":"Center on Frontiers of Computing Studies , School of Computer Science , Peking University 2 Business Growth BU , JD.com"},{"name":"Xiaotie Deng","slug":"xiaotie-deng","given":"Xiaotie","family":"Deng","affiliation":"Center on Frontiers of Computing Studies , School of Computer Science , Peking University 2 Business Growth BU , JD.com"},{"name":"Chenchen Li","slug":"chenchen-li","given":"Chenchen","family":"Li","affiliation":""},{"name":"David Mguni","slug":"david-mguni","given":"David","family":"Mguni","affiliation":"Huawei R&D"},{"name":"Jun Wang","slug":"jun-wang","given":"Jun","family":"Wang","affiliation":"UCL"},{"name":"Xiang Yan","slug":"xiang-yan","given":"Xiang","family":"Yan","affiliation":"Huawei TCS Lab"},{"name":"Yaodong Yang","slug":"yaodong-yang","given":"Yaodong","family":"Yang","affiliation":"Institute for AI , Peking University"}],"abstract":"Fictitious play (FP) is one of the most fundamental game-theoretical learning frameworks for computing Nash equilibrium in n-player games, which builds the foundation for modern multi-agent learning algorithms. Although FP has provable convergence guarantees on zero-sum games and potential games, many real-world problems are often a mixture of both and the convergence property of FP has not been fully studied yet. In this paper, we extend the convergence results of FP to the combinations of such games and beyond. Specifically, we derive new conditions for FP to converge by leveraging game decomposition techniques. We further develop a linear relationship unifying cooperation and competition in the sense that these two classes of games are mutually transferable. Finally, we analyse a non-convergent example of FP, the Shapley game, and develop sufficient conditions for FP to converge.","doi":"10.24963/ijcai.2022/26","url":"https://ijcai.org/proceedings/2022/26","pdf":"https://ijcai.org/proceedings/2022/0026.pdf","markdown":"https://ijcai.org/proceedings/2022/26.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/26","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":26,"pages":{"first":179,"last":185},"keywords":["Agent-based and Multi-agent Systems: Multi-agent Learning","Agent-based and Multi-agent Systems: Noncooperative Games","Agent-based and Multi-agent Systems: Algorithmic Game Theory"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986622","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984765","length_sec":306,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Adler et al., 2009] Ilan Adler, Constantinos Daskalakis, and Chris- tos H Papadimitriou. A note on strictly competitive games. In In- ternational Workshop on Internet and Network Economics, pages 471-474. Springer, 2009.","title":"A note on strictly competitive games","authors":"Ilan Adler, Constantinos Daskalakis, Christos H Papadimitriou","year":"2009","venue":"International Workshop on Internet and Network Economics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Balduzzi et al., 2018] David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Graepel. The me- chanics of n-player differentiable games. In International Con- ference on Machine Learning, pages 354-363. PMLR, 2018.","title":"The mechanics of n-player differentiable games","authors":"David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, Thore Graepel","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brown, 1951] George W Brown. Iterative solution of games by fic- titious play, 1951. Activity Analysis of Production and Allocation (TC Koopmans, Ed.), pages 374-376, 1951.","title":"Iterative solution of games by fictitious play","authors":"George W Brown","year":"1951","venue":"Activity Analysis of Production and Allocation","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Candogan et al., 2011] Ozan Candogan, Ishai Menache, Asuman Ozdaglar, and Pablo A Parrilo. Flows and decompositions of games: Harmonic and potential games. Mathematics of Opera- tions Research, 36(3):474-503, 2011.","title":"Flows and decompositions of games: Harmonic and potential games","authors":"Ozan Candogan, Ishai Menache, Asuman Ozdaglar, Pablo A Parrilo","year":"2011","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Candogan et al., 2013] Ozan Candogan, Asuman Ozdaglar, and Pablo A Parrilo. Dynamics in near-potential games. Games and Economic Behavior, 82:66-90, 2013.","title":"Dynamics in near-potential games","authors":"Ozan Candogan, Asuman Ozdaglar, Pablo A Parrilo","year":"2013","venue":"Games and Economic Behavior","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cheung and Tao, 2021] Yun Kuen Cheung and Yixin Tao. Chaos of learning beyond zero-sum and coordination via game decom- positions. In ICLR 2021. OpenReview.net, 2021.","title":"Chaos of learning beyond zero-sum and coordination via game decompositions","authors":"Tao ; ] Cheung, Kuen Yun, Yixin Cheung, Tao","year":"2021","venue":"ICLR 2021","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dasgupta and Collins, 2019] Prithviraj Dasgupta and Joseph Collins. A survey of game theoretic approaches for adver- sarial machine learning in cybersecurity tasks. AI Magazine, 40(2):31-43, 2019.","title":"A survey of game theoretic approaches for adversarial machine learning in cybersecurity tasks","authors":"Prithviraj Dasgupta, Joseph Collins","year":"2019","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Deng et al., 2021] Xiaotie Deng, Yuhao Li, David Henry Mguni, Jun Wang, and Yaodong Yang. On the complexity of comput- ing markov perfect equilibrium in general-sum stochastic games. arXiv preprint arXiv:2109.01795, 2021.","title":"On the complexity of computing markov perfect equilibrium in general-sum stochastic games","authors":"Xiaotie Deng, Yuhao Li, David Henry Mguni, Jun Wang, Yaodong Yang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dinh et al., 2021] Le Cong Dinh, Yaodong Yang, Zheng Tian, Nicolas Perez Nieves, Oliver Slumbers, David Henry Mguni, Haitham Bou Ammar, and Jun Wang. Online double oracle. arxiv preprint arxiv:2103.07780, 2021.","title":"Online double oracle","authors":"Le Cong Dinh, Yaodong Yang, Zheng Tian, Nicolas Perez Nieves, Oliver Slumbers, David Henry Mguni, Haitham Bou Ammar, Jun Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Feng et al., 2021] Xidong Feng, Oliver Slumbers, Ziyu Wan, Bo Liu, Stephen McAleer, Ying Wen, Jun Wang, and Yaodong Yang. Neural auto-curricula in two-player zero-sum games. Ad- vances in Neural Information Processing Systems, 34, 2021.","title":"Neural auto-curricula in two-player zero-sum games","authors":"Feng","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gopalakrishnan et al., 2011] Ragavendran Gopalakrishnan, Ja- son R Marden, and Adam Wierman. Characterizing distribution rules for cost sharing games. In International Conference on NETwork Games, Control and Optimization (NetGCooP 2011), pages 1-4. IEEE, 2011.","title":"Characterizing distribution rules for cost sharing games","authors":"Gopalakrishnan","year":"2011","venue":"International Conference on NETwork Games, Control and Optimization","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Heinrich et al., 2015] Johannes Heinrich, Marc Lanctot, and David Silver. Fictitious self-play in extensive-form games. In Proceedings of The 32nd International Conference on Machine Learning, pages 805-813, 2015.","title":"Fictitious self-play in extensive-form games","authors":"Johannes Heinrich, Marc Lanctot, David Silver","year":"2015","venue":"Proceedings of The 32nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Heyman, 2019] Joseph Lee Heyman. On the Computation of Strategically Equivalent Games. ProQuest LLC, Ann Arbor, MI, 2019. Thesis (Ph.D.)-The Ohio State University.","title":"On the Computation of Strategically Equivalent Games","authors":"Joseph Lee, Heyman","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hwang and Rey-Bellet, 2020] Sung-Ha Hwang and Luc Rey- Bellet. Strategic decompositions of normal form games: Zero- sum games and potential games. Games and Economic Behavior, 2020.","title":"Strategic decompositions of normal form games: Zerosum games and potential games","authors":"Sung-Ha Hwang, Luc Rey-Bellet","year":"2020","venue":"Games and Economic Behavior","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lanctot et al., 2017] Marc Lanctot, Vinicius Zambaldi, Audrūnas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning. In NeurIPS, pages 4193- 4206, 2017.","title":"A unified game-theoretic approach to multiagent reinforcement learning","authors":"Lanctot","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2021] Xiangyu Liu, Hangtian Jia, Ying Wen, Yaodong Yang, Yujing Hu, Yingfeng Chen, Changjie Fan, and Zhipeng Hu. Unifying behavioral and response diversity for open-ended learning in zero-sum games. NeurIPS, 2021.","title":"Unifying behavioral and response diversity for open-ended learning in zero-sum games","authors":"Xiangyu Liu, Hangtian Jia, Ying Wen, Yaodong Yang, Yujing Hu, Yingfeng Chen, Changjie Fan, Zhipeng Hu","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[McMahan et al., 2003] H Brendan McMahan, Geoffrey J Gordon, and Avrim Blum. Planning in the presence of cost functions con- trolled by an adversary. In ICML-03, pages 536-543, 2003.","title":"Planning in the presence of cost functions controlled by an adversary","authors":"Geoffrey J H Brendan Mcmahan, Avrim Gordon, Blum","year":"2003","venue":"ICML-03","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Mguni et al., ] David H Mguni, Yutong Wu, Yali Du, Yaodong Yang, Ziyi Wang, Minne Li, Ying Wen, Joel Jennings, and Jun Wang. Learning in nonzero-sum stochastic games with poten- tials. In ICML 2021, pages 7688-7699. PMLR.","title":"Learning in nonzero-sum stochastic games with potentials","authors":"Mguni","year":null,"venue":"ICML 2021","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Monderer and Shapley, 1996a] Dov Monderer and Lloyd S. Shap- ley. Fictitious play property for games with identical interests. Journal of Economic Theory, 68(1):258-265, 1996.","title":"Dov Monderer and Lloyd S. Shapley. Fictitious play property for games with identical interests","authors":"Shapley Monderer","year":"1996","venue":"Journal of Economic Theory","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Monderer and Shapley, 1996b] Dov Monderer and Lloyd S Shap- ley. Potential games. Games and economic behavior, 14(1):124- 143, 1996.","title":"Dov Monderer and Lloyd S Shapley. Potential games","authors":"Shapley Monderer","year":"1996","venue":"Games and economic behavior","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"Muller et al., ] Paul Muller, Shayegan Omidshafiei, Mark Row- land, Karl Tuyls, Julien Perolat, Siqi Liu, Daniel Hennes, Luke Marris, Marc Lanctot, Edward Hughes, Zhe Wang, Guy Lever, Nicolas Heess, Thore Graepel, and Remi Munos. A generalized training approach for multiagent learning. In ICLR 2020.","title":"A generalized training approach for multiagent learning","authors":"Muller","year":null,"venue":"ICLR 2020","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nash, 1950] John F. Nash. Equilibrium points in n-person games. Proceedings of the National Academy of Sciences, 36(1):48-49, 1950.","title":"Equilibrium points in n-person games","authors":"John F Nash","year":"1950","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ostrovski and van Strien, 2014] Georg Ostrovski and Sebastian van Strien. Payoff performance of fictitious play. J. Dyn. Games, 1(4):621-638, 2014.","title":"Georg Ostrovski and Sebastian van Strien","authors":"Ostrovski, Van Strien","year":"2014","venue":"J. Dyn. Games","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Perez-Nieves et al., 2021] Nicolas Perez-Nieves, Yaodong Yang, Oliver Slumbers, David H Mguni, Ying Wen, and Jun Wang. Modelling behavioural diversity for learning in open-ended games. In ICML, pages 8514-8524. PMLR, 2021.","title":"Modelling behavioural diversity for learning in open-ended games","authors":"Nicolas Perez-Nieves, Yaodong Yang, Oliver Slumbers, H David, Ying Mguni, Jun Wen, Wang","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Robinson, 1951] Julia Robinson. An iterative method of solving a game. Annals of Mathematics, 54(2):296-301, 1951.","title":"An iterative method of solving a game","authors":"Julia Robinson","year":"1951","venue":"Annals of Mathematics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shapley, 1964] Lloyd Shapley. Some topics in two-person games. Advances in game theory, 52:1-29, 1964.","title":"Lloyd Shapley. Some topics in two-person games","authors":"Shapley","year":"1964","venue":"Advances in game theory","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Swenson et al., 2018] Brian Swenson, Ryan Murray, and Soum- mya Kar. On best-response dynamics in potential games. SIAM Journal on Control and Optimization, 56(4):2734-2767, 2018.","title":"On best-response dynamics in potential games","authors":"Brian Swenson, Ryan Murray, Soummya Kar","year":"2018","venue":"SIAM Journal on Control and Optimization","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tuyls et al., 2018] Karl Tuyls, Julien Pérolat, Marc Lanctot, Georg Ostrovski, Rahul Savani, Joel Z Leibo, Toby Ord, Thore Grae- pel, and Shane Legg. Symmetric decomposition of asymmetric games. Scientific reports, 8(1):1-20, 2018.","title":"Symmetric decomposition of asymmetric games","authors":"Karl Tuyls, Julien Pérolat, Marc Lanctot, Georg Ostrovski, Rahul Savani, Joel Z Leibo, Toby Ord, Thore Graepel, Shane Legg","year":"2018","venue":"Scientific reports","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Vöcking, 2007] Berthold Vöcking. Selfish load balancing. Algo- rithmic game theory, 20:517-542, 2007.","title":"Selfish load balancing","authors":"Berthold Vöcking","year":"2007","venue":"Algorithmic game theory","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang and Wang, 2020] Yaodong Yang and Jun Wang. An overview of multi-agent reinforcement learning from game theo- retical perspective. arXiv preprint arXiv:2011.00583, 2020.","title":"An overview of multi-agent reinforcement learning from game theoretical perspective","authors":"Yang, Wang ;, Yaodong Yang, Jun Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang et al., 2018] Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi-agent reinforce- ment learning. In International Conference on Machine Learn- ing, pages 5571-5580. PMLR, 2018.","title":"Mean field multi-agent reinforcement learning","authors":"Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, Jun Wang","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}