{"type":"ScholarlyArticle","title":"Anticipatory Fictitious Play","authors":[{"name":"Alex Cloud","slug":"alex-cloud","given":"Alex","family":"Cloud","affiliation":""},{"name":"Albert Wang","slug":"albert-wang","given":"Albert","family":"Wang","affiliation":""},{"name":"Wesley Kerr","slug":"wesley-kerr","given":"Wesley","family":"Kerr","affiliation":""}],"abstract":"Fictitious play is an algorithm for computing Nash equilibria of matrix games. Recently, machine learning variants of fictitious play have been successfully applied to complicated real-world games. This paper presents a simple modification of fictitious play which is a strict improvement over the original: it has the same theoretical worst-case convergence rate, is equally applicable in a machine learning context, and enjoys superior empirical performance. We conduct an extensive comparison of our algorithm with fictitious play, proving an optimal O(1/t) convergence rate for certain classes of games, demonstrating superior performance numerically across a variety of games, and concluding with experiments that extend these algorithms to the setting of deep multiagent reinforcement learning.","doi":"10.24963/ijcai.2023/9","url":"https://ijcai.org/proceedings/2023/9","pdf":"https://ijcai.org/proceedings/2023/0009.pdf","markdown":"https://ijcai.org/proceedings/2023/9.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/9","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":9,"pages":{"first":73,"last":81},"keywords":["Agent-based and Multi-agent Systems: MAS: Multi-agent learning","Machine Learning: ML: Reinforcement learning","Game Theory and Economic Paradigms: GTEP: Noncooperative games"],"videos":[],"references":[{"position":1,"raw":"[Adler, 2013] Ilan Adler. The equivalence of linear programs and zero-sum games. International Journal of Game The- ory, 42(1):165-177, 2013.","title":"The equivalence of linear programs and zero-sum games","authors":"Ilan Adler","year":"2013","venue":"International Journal of Game Theory","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agresti and Coull, 1998] Alan Agresti and Brent A Coull. Approximate is better than \"exact\" for interval estima- tion of binomial proportions. The American Statistician, 52(2):119-126, 1998.","title":"Approximate is better than \"exact\" for interval estimation of binomial proportions","authors":"Coull Agresti, Alan Agresti, Brent A Coull","year":"1998","venue":"The American Statistician","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Balduzzi et al., 2019] David Balduzzi, Marta Garnelo, Yoram Bachrach, Wojciech Czarnecki, Julien Perolat, Max Jaderberg, and Thore Graepel. Open-ended learning in symmetric zero-sum games. In International Confer- ence on Machine Learning, pages 434-443. PMLR, 2019.","title":"Open-ended learning in symmetric zero-sum games","authors":"David Balduzzi, Marta Garnelo, Yoram Bachrach, Wojciech Czarnecki, Julien Perolat, Max Jaderberg, Thore Graepel","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brown, 1951] George W Brown. Iterative solution of games by fictitious play. Activity Analysis of Production and Al- location, 13(1):374-376, 1951.","title":"Iterative solution of games by fictitious play","authors":"George W Brown","year":"1951","venue":"Activity Analysis of Production and Allocation","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Conlisk, 1993a] John Conlisk. Adaptation in games: Two solutions to the crawford puzzle. Journal of Economic Be- havior & Organization, 22(1):25-50, 1993.","title":"Adaptation in games: Two solutions to the crawford puzzle","authors":"John Conlisk, Conlisk","year":"1993","venue":"Journal of Economic Behavior & Organization","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Conlisk, 1993b] John Conlisk. Adaptive tactics in games: Further solutions to the crawford puzzle. Journal of Eco- nomic Behavior & Organization, 22(1):51-68, 1993.","title":"Adaptive tactics in games: Further solutions to the crawford puzzle","authors":"John Conlisk, Conlisk","year":"1993","venue":"Journal of Economic Behavior & Organization","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Daskalakis and Pan, 2014] Constantinos Daskalakis and Qinxuan Pan. A counter-example to karlin's strong conjecture for fictitious play. In 2014 IEEE 55th Annual Symposium on Foundations of Computer Science, pages 11-20, Philadelphia, PA, USA, 2014. IEEE, IEEE.","title":"A counter-example to karlin's strong conjecture for fictitious play","authors":"Constantinos Daskalakis, Qinxuan Pan","year":"2014","venue":"IEEE 55th Annual Symposium on Foundations of Computer Science","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Foerster et al., 2018] Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. In Proceedings of the 17th International Con- ference on Autonomous Agents and MultiAgent Systems, pages 122-130, ', 2018. AAMAS.","title":"Learning with opponent-learning awareness","authors":"Jakob Foerster, Maruan Richard Y Chen, Shimon Al-Shedivat, Pieter Whiteson, Igor Abbeel, Mordatch","year":"2018","venue":"Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Heinrich et al., 2015] Johannes Heinrich, Marc Lanctot, and David Silver. Fictitious self-play in extensive-form games. In International Conference on Machine Learn- ing, pages 805-813, 2015.","title":"Fictitious self-play in extensive-form games","authors":"Johannes Heinrich, Marc Lanctot, David Silver","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kuhn, 1953] Harold Kuhn. Extensive games and the prob- lem of information. In Contributions to the Theory of Games, pages 193-216. Princeton University Press, 1953.","title":"Extensive games and the problem of information","authors":"Harold Kuhn, Kuhn","year":"1953","venue":"Contributions to the Theory of Games","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lanctot et al., 2017] Marc Lanctot, Vinicius Zambaldi, Au- drunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Perolat, David Silver, and Thore Graepel. A unified game- theoretic approach to multiagent reinforcement learning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems, vol- ume 30, -, 2017.","title":"A unified gametheoretic approach to multiagent reinforcement learning","authors":"Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Perolat, David Silver, Thore Graepel","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Curran Associates, Inc. [Letcher et al., 2018] Alistair Letcher, Jakob Foerster, David Balduzzi, Tim Rocktäschel, and Shimon Whiteson. Sta- ble opponent shaping in differentiable games. In Inter- national Conference on Learning Representations, page ', Vancouver Convention Center, Vancouver, BC, Canada, 2018. ICLR.","title":"Stable opponent shaping in differentiable games","authors":"Inc [ Curran Associates, Letcher","year":"2018","venue":"International Conference on Learning Representations, page","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2022] Siqi Liu, Luke Marris, Daniel Hennes, Josh Merel, Nicolas Heess, and Thore Graepel. Neupl: Neu- ral population learning. In The Tenth International Con- ference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.","title":"Neupl: Neural population learning","authors":"Siqi Liu, Luke Marris, Daniel Hennes, Josh Merel, Nicolas Heess, Thore Graepel","year":"2022","venue":"The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Luce and Raiffa, 1989] R Duncan Luce and Howard Raiffa. Games and Decisions: Introduction and Critical Survey. Courier Corporation, ', 1989.","title":"Games and Decisions: Introduction and Critical Survey","authors":"Luce Duncan, Howard Raiffa","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Moritz et al., 2018] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. Ray: A distributed framework for emerging {AI} applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 561-577, 2018.","title":"Ray: A distributed framework for emerging {AI} applications","authors":"Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, Ion Stoica","year":"2018","venue":"13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nash Jr, 1950] John F Nash Jr. Equilibrium points in n- person games. Proceedings of the National Academy of Sciences, 36(1):48-49, 1950.","title":"Equilibrium points in nperson games","authors":"John F Nash Jr","year":"1950","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Robinson, 1951] Julia Robinson. An iterative method of solving a game. Annals of Mathematics, 54(2):296-301, 1951.","title":"An iterative method of solving a game","authors":"Julia Robinson","year":"1951","venue":"Annals of Mathematics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shamma and Arslan, 2005] Jeff S Shamma and Gürdal Ar- slan. Dynamic fictitious play, dynamic gradient play, and distributed convergence to nash equilibria. IEEE Transac- tions on Automatic Control, 50(3):312-327, 2005.","title":"Dynamic fictitious play, dynamic gradient play, and distributed convergence to nash equilibria","authors":"S Jeff, Gürdal Shamma, Arslan","year":"2005","venue":"IEEE Transactions on Automatic Control","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shapley, 1953] Lloyd S Shapley. Stochastic games. Proceedings of the National Academy of Sciences, 39(10):1095-1100, 1953.","title":"Stochastic games","authors":"S Lloyd, Shapley","year":"1953","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shoham and Leyton-Brown, 2008] Yoav Shoham and Kevin Leyton-Brown. Multiagent Systems: Algorithmic, Game-theoretic, and Logical Foundations. Cambridge University Press, 2008.","title":"Multiagent Systems: Algorithmic, Game-theoretic, and Logical Foundations","authors":"Yoav Shoham, Kevin Leyton-Brown","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, Cambridge, Massachusetts, 2018.","title":"Reinforcement Learning: An Introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vezhnevets et al., 2020] Alexander Vezhnevets, Yuhuai Wu, Maria Eckstein, Rémi Leblond, and Joel Z Leibo. Options as responses: Grounding behavioural hierarchies in multi-agent reinforcement learning. In International Conference on Machine Learning, pages 9733-9742, ', 2020. PMLR, PMLR.","title":"Options as responses: Grounding behavioural hierarchies in multi-agent reinforcement learning","authors":"Alexander Vezhnevets, Yuhuai Wu, Maria Eckstein, Rémi Leblond, Joel Z Leibo","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vinyals et al., 2019] Oriol Vinyals, Igor Babuschkin, Woj- ciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexan- der S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Katrina McK- inney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Ko- ray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft ii using multi- agent reinforcement learning. Nature, 575(7782):350- 354, 2019.","title":"Grandmaster level in starcraft ii using multiagent reinforcement learning","authors":"Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P Agapiou, Max Jaderberg, Alexander S Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Katrina Mckinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, David Silver","year":"2019","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang and Lesser, 2010] Chongjie Zhang and Victor Lesser. Multi-agent learning with policy prediction. In Twenty-fourth AAAI conference on artificial intelligence, pages 927-937, ', 2010. AAAI.","title":"Chongjie Zhang and Victor Lesser. Multi-agent learning with policy prediction","authors":"Lesser Zhang","year":"2010","venue":"Twenty-fourth AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}