{"type":"ScholarlyArticle","title":"Learning in Multi-Memory Games Triggers Complex Dynamics Diverging from Nash Equilibrium","authors":[{"name":"Yuma Fujimoto","slug":"yuma-fujimoto","given":"Yuma","family":"Fujimoto","affiliation":"Research Center for Integrative Evolutionary Science , SOKENDAI.; Universal Biology Institute (UBI) , the University of Tokyo.; AI Lab , CyberAgent , Inc."},{"name":"Kaito Ariu","slug":"kaito-ariu","given":"Kaito","family":"Ariu","affiliation":"AI Lab , CyberAgent , Inc.; KTH Royal Institute of Technology."},{"name":"Kenshi Abe","slug":"kenshi-abe","given":"Kenshi","family":"Abe","affiliation":"AI Lab , CyberAgent , Inc."}],"abstract":"Repeated games consider a situation where multiple agents are motivated by their independent rewards throughout learning. In general, the dynamics of their learning become complex. Especially when their rewards compete with each other like zero-sum games, the dynamics often do not converge to their optimum, i.e., the Nash equilibrium. To tackle such complexity, many studies have understood various learning algorithms as dynamical systems and discovered qualitative insights among the algorithms. However, such studies have yet to handle multi-memory games (where agents can memorize actions they played in the past and choose their actions based on their memories), even though memorization plays a pivotal role in artificial intelligence and interpersonal relationship. This study extends two major learning algorithms in games, i.e., replicator dynamics and gradient ascent, into multi-memory games. Then, we prove their dynamics are identical. Furthermore, theoretically and experimentally, we clarify that the learning dynamics diverge from the Nash equilibrium in multi-memory zero-sum games and reach heteroclinic cycles (sojourn longer around the boundary of the strategy space), providing a fundamental advance in learning in games.","doi":"10.24963/ijcai.2023/14","url":"https://ijcai.org/proceedings/2023/14","pdf":"https://ijcai.org/proceedings/2023/0014.pdf","markdown":"https://ijcai.org/proceedings/2023/14.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/14","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":14,"pages":{"first":118,"last":125},"keywords":["Agent-based and Multi-agent Systems: MAS: Multi-agent learning","Agent-based and Multi-agent Systems: MAS: Agent theories and models"],"videos":[],"references":[{"position":1,"raw":"[Abdallah and Kaisers, 2013] Sherief Abdallah and Michael Kaisers. Addressing the policy-bias of q-learning by re- peating updates. In AAMAS, pages 1045-1052, 2013.","title":"Addressing the policy-bias of q-learning by repeating updates","authors":"Kaisers Abdallah, Sherief Abdallah, Michael Kaisers","year":"2013","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Abe et al., 2022] Kenshi Abe, Mitsuki Sakamoto, and At- sushi Iwasaki. Mutation-driven follow the regularized leader for last-iterate convergence in zero-sum games. In UAI, pages 1-10, 2022.","title":"Mutation-driven follow the regularized leader for last-iterate convergence in zero-sum games","authors":"Abe","year":"2022","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Axelrod and Hamilton, 1981] Robert Axelrod and William D Hamilton. The evolution of cooperation. Science, 211(4489):1390-1396, 1981.","title":"The evolution of cooperation","authors":"Robert Axelrod, William D Hamilton","year":"1981","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Banerjee and Peng, 2005] Bikramjit Banerjee and Jing Peng. Efficient no-regret multiagent learning. In AAAI, pages 41-46, 2005.","title":"Efficient no-regret multiagent learning","authors":"Bikramjit Banerjee, Jing Peng","year":"2005","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Barfuss et al., 2019] Wolfram Barfuss, Jonathan F Donges, and Jürgen Kurths. Deterministic limit of temporal differ- ence reinforcement learning for stochastic games. Physi- cal Review E, 99(4):043305, 2019.","title":"Deterministic limit of temporal difference reinforcement learning for stochastic games","authors":"Barfuss","year":"2019","venue":"Physical Review E","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Barfuss, 2020a] Wolfram Barfuss. Reinforcement learning dynamics in the infinite memory limit. In AAMAS, pages 1768-1770, 2020.","title":"Reinforcement learning dynamics in the infinite memory limit","authors":"Wolfram Barfuss, Barfuss","year":"2020","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Barfuss, 2020b] Wolfram Barfuss. Towards a unified treat- ment of the dynamics of collective learning. In Challenges and Opportunities for Multi-Agent Reinforcement Learn- ing, AAAI Spring Symposium, 2020.","title":"Towards a unified treatment of the dynamics of collective learning","authors":"Wolfram Barfuss, Barfuss","year":"2020","venue":"Challenges and Opportunities for Multi-Agent Reinforcement Learning, AAAI Spring Symposium","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bloembergen et al., 2015] Daan Bloembergen, Karl Tuyls, Daniel Hennes, and Michael Kaisers. Evolutionary dy- namics of multi-agent learning: A survey. Journal of Arti- ficial Intelligence Research, 53:659-697, 2015.","title":"Evolutionary dynamics of multi-agent learning: A survey","authors":"Daan Bloembergen, Karl Tuyls, Daniel Hennes, Michael Kaisers","year":"2015","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Börgers and Sarin, 1997] Tilman Börgers and Rajiv Sarin. Learning through reinforcement and replicator dynamics. Journal of Economic Theory, 77(1):1-14, 1997.","title":"Learning through reinforcement and replicator dynamics","authors":"Sarin Börgers, Tilman Börgers, Rajiv Sarin","year":"1997","venue":"Journal of Economic Theory","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Bowling and Veloso, 2002] Michael Bowling and Manuela Veloso. Multiagent learning using a variable learning rate. Artificial Intelligence, 136(2):215-250, 2002.","title":"Multiagent learning using a variable learning rate","authors":"Michael Bowling, Manuela Veloso","year":"2002","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Bowling, 2004] Michael Bowling. Convergence and no- regret in multiagent learning. In NeurIPS, pages 209-216, 2004.","title":"Convergence and noregret in multiagent learning","authors":"Michael Bowling","year":"2004","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Cross, 1973] John G Cross. A stochastic learning model of economic behavior. The Quarterly Journal of Economics, 87(2):239-266, 1973.","title":"A stochastic learning model of economic behavior","authors":"G John, Cross","year":"1973","venue":"The Quarterly Journal of Economics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Daskalakis and Panageas, 2019] Constantinos Daskalakis and Ioannis Panageas. Last-iterate convergence: Zero-sum games and constrained min-max optimization. In ITCS, pages 27:1-27:18, 2019.","title":"Last-iterate convergence: Zero-sum games and constrained min-max optimization","authors":"Constantinos Daskalakis, Ioannis Panageas","year":"2019","venue":"ITCS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Daskalakis et al., 2011] Constantinos Daskalakis, Alan Deckelbaum, and Anthony Kim. Near-optimal no-regret algorithms for zero-sum games. In SODA, pages 235-254, 2011.","title":"Near-optimal no-regret algorithms for zero-sum games","authors":"Constantinos Daskalakis, Alan Deckelbaum, Anthony Kim","year":"2011","venue":"SODA","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Daskalakis et al., 2018] Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, and Haoyang Zeng. Training GANs with optimism. In ICLR, 2018.","title":"Training GANs with optimism","authors":"Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, Haoyang Zeng","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Fudenberg and Maskin, 2009] Drew Fudenberg and Eric Maskin. The folk theorem in repeated games with dis- counting or with incomplete information. In A long-run collaboration on long-run games, pages 209-230. World Scientific, 2009.","title":"The folk theorem in repeated games with discounting or with incomplete information","authors":"Maskin Fudenberg, Drew Fudenberg, Eric Maskin","year":"2009","venue":"A long-run collaboration on long-run games","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Fudenberg and Tirole, 1991] Drew Fudenberg and Jean Ti- role. Game theory. MIT press, 1991.","title":"Drew Fudenberg and Jean Tirole","authors":"Tirole Fudenberg","year":"1991","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Fujimoto and Kaneko, 2019a] Yuma Fujimoto and Kuni- hiko Kaneko. Emergence of exploitation as symmetry breaking in iterated prisoner's dilemma. Physical Review Research, 1(3):033077, 2019.","title":"Emergence of exploitation as symmetry breaking in iterated prisoner's dilemma","authors":"Kaneko Fujimoto, Yuma Fujimoto, Kunihiko Kaneko","year":"2019","venue":"Physical Review Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Fujimoto and Kaneko, 2019b] Yuma Fujimoto and Kuni- hiko Kaneko. Functional dynamic by intention recognition in iterated games. New Journal of Physics, 21(2):023025, 2019.","title":"Functional dynamic by intention recognition in iterated games","authors":"Kaneko Fujimoto, Yuma Fujimoto, Kunihiko Kaneko","year":"2019","venue":"New Journal of Physics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Fujimoto and Kaneko, 2021] Yuma Fujimoto and Kunihiko Kaneko. Exploitation by asymmetry of information ref- erence in coevolutionary learning in prisoner's dilemma game. Journal of Physics: Complexity, 2(4):045007, 2021.","title":"Exploitation by asymmetry of information reference in coevolutionary learning in prisoner's dilemma game","authors":"Kaneko Fujimoto, Yuma Fujimoto, Kunihiko Kaneko","year":"2021","venue":"Journal of Physics: Complexity","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Golowich et al., 2020] Noah Golowich, Sarath Pattathil, and Constantinos Daskalakis. Tight last-iterate conver- gence rates for no-regret learning in multi-player games. In NeurIPS, pages 20766-20778, 2020.","title":"Noah Golowich, Sarath Pattathil, and Constantinos Daskalakis. Tight last-iterate convergence rates for no-regret learning in multi-player games","authors":"Golowich","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Hofbauer et al., 1998] Josef Hofbauer, Karl Sigmund, et al. Evolutionary games and population dynamics. Cambridge university press, 1998.","title":"Evolutionary games and population dynamics","authors":"Hofbauer","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Hofbauer, 1996] Josef Hofbauer. Evolutionary dynamics for bimatrix games: A hamiltonian system? Journal of Mathematical Biology, 34(5):675-688, 1996.","title":"Evolutionary dynamics for bimatrix games: A hamiltonian system","authors":"Josef Hofbauer, Hofbauer","year":"1996","venue":"Journal of Mathematical Biology","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Kaisers and Tuyls, 2010] Michael Kaisers and Karl Tuyls. Frequency adjusted multi-agent q-learning. In AAMAS, pages 309-316, 2010.","title":"Frequency adjusted multi-agent q-learning","authors":"Tuyls Kaisers, Michael Kaisers, Karl Tuyls","year":"2010","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Lanctot et al., 2012] Marc Lanctot, Richard Gibson, Neil Burch, Martin Zinkevich, and Michael Bowling. No-regret learning in extensive-form games with imperfect recall. In ICML, pages 1035-1042, 2012.","title":"No-regret learning in extensive-form games with imperfect recall","authors":"Marc Lanctot, Richard Gibson, Neil Burch, Martin Zinkevich, Michael Bowling","year":"2012","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Lei et al., 2021] Qi Lei, Sai Ganesh Nagarajan, Ioannis Panageas, et al. Last iterate convergence in no-regret learning: constrained min-max optimization for convex- concave landscapes. In AISTATS, pages 1441-1449, 2021.","title":"Last iterate convergence in no-regret learning: constrained min-max optimization for convexconcave landscapes","authors":"Lei","year":"2021","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Littman, 1994] Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In ICML, pages 157-163, 1994.","title":"Markov games as a framework for multi-agent reinforcement learning","authors":"Michael L Littman","year":"1994","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Mertikopoulos and Sandholm, 2016] Panayotis Mer- tikopoulos and William H Sandholm. Learning in games via reinforcement and regularization. Mathematics of Operations Research, 41(4):1297-1324, 2016.","title":"Learning in games via reinforcement and regularization","authors":"Panayotis Mertikopoulos, William H Sandholm","year":"2016","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Mertikopoulos et al., 2018] Panayotis Mertikopoulos, Christos Papadimitriou, and Georgios Piliouras. Cycles in adversarial regularized learning. In SODA, pages 2703-2717, 2018.","title":"Cycles in adversarial regularized learning","authors":"Panayotis Mertikopoulos, Christos Papadimitriou, Georgios Piliouras","year":"2018","venue":"SODA","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Mertikopoulos et al., 2019] Panayotis Mertikopoulos, Bruno Lecouat, Houssam Zenati, Chuan-Sheng Foo, Vijay Chandrasekhar, and Georgios Piliouras. Optimistic mirror descent in saddle-point problems: Going the extra(-gradient) mile. In ICLR, 2019.","title":"Optimistic mirror descent in saddle-point problems: Going the extra(-gradient) mile","authors":"Panayotis Mertikopoulos, Bruno Lecouat, Houssam Zenati, Chuan-Sheng Foo, Vijay Chandrasekhar, Georgios Piliouras","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Meylahn et al., 2022] Janusz M Meylahn, Lars Janssen, et al. Limiting dynamics for q-learning with memory one in symmetric two-player, two-action games. Complexity, 2022, 2022.","title":"Limiting dynamics for q-learning with memory one in symmetric two-player, two-action games","authors":"Lars Janusz M Meylahn, Janssen","year":"2022","venue":"Complexity","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Nash Jr, 1950] John F Nash Jr. Equilibrium points in n- person games. Proceedings of the National Academy of Sciences, 36(1):48-49, 1950.","title":"Equilibrium points in nperson games","authors":"John F Nash Jr","year":"1950","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Nowak and Sigmund, 1993] Martin Nowak and Karl Sig- mund. A strategy of win-stay, lose-shift that outper- forms tit-for-tat in the prisoner's dilemma game. Nature, 364(6432):56-58, 1993.","title":"A strategy of win-stay, lose-shift that outperforms tit-for-tat in the prisoner's dilemma game","authors":"Sigmund Nowak","year":"1993","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Piliouras et al., 2014] Georgios Piliouras, Carlos Nieto- Granda, Henrik I Christensen, and Jeff S Shamma. Per- sistent patterns: Multi-agent learning beyond equilibrium and utility. In AAMAS, pages 181-188, 2014.","title":"Persistent patterns: Multi-agent learning beyond equilibrium and utility","authors":"Georgios Piliouras, Carlos Nieto-Granda, Henrik I Christensen, Jeff S Shamma","year":"2014","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Sandholm and Crites, 1996] Tuomas W Sandholm and Robert H Crites. Multiagent reinforcement learning in the iterated prisoner's dilemma. Biosystems, 37(1-2):147- 166, 1996.","title":"Multiagent reinforcement learning in the iterated prisoner's dilemma","authors":"Crites ; ] Sandholm, W Tuomas, Robert H Sandholm, Crites","year":"1996","venue":"Biosystems","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Shapley, 1953] Lloyd S Shapley. Stochastic games. Proceedings of the National Academy of Sciences, 39(10):1095-1100, 1953.","title":"Stochastic games","authors":"S Lloyd, Shapley","year":"1953","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Singh et al., 2000] Satinder Singh, Michael J Kearns, and Yishay Mansour. Nash convergence of gradient dynamics in general-sum games. In UAI, pages 541-548, 2000.","title":"Nash convergence of gradient dynamics in general-sum games","authors":"Satinder Singh, Michael J Kearns, Yishay Mansour","year":"2000","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Tuyls and Nowé, 2005] Karl Tuyls and Ann Nowé. Evolu- tionary game theory and multi-agent reinforcement learn- ing. The Knowledge Engineering Review, 20(1):63-90, 2005.","title":"Karl Tuyls and Ann Nowé. Evolutionary game theory and multi-agent reinforcement learning","authors":"Nowé Tuyls","year":"2005","venue":"The Knowledge Engineering Review","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Tuyls et al., 2006] Karl Tuyls, Pieter Jan'T Hoen, and Bram Vanschoenwinkel. An evolutionary dynamical analysis of multi-agent learning in iterated games. Autonomous Agents and Multi-Agent Systems, 12(1):115-153, 2006.","title":"An evolutionary dynamical analysis of multi-agent learning in iterated games","authors":"Tuyls","year":"2006","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Watkins and Dayan, 1992] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3):279- 292, 1992.","title":"Q-learning","authors":"Jch Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Wei et al., 2021] Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle-point optimization. In ICLR, 2021.","title":"Linear last-iterate convergence in constrained saddle-point optimization","authors":"Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, Haipeng Luo","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Zinkevich et al., 2007] Martin Zinkevich, Michael Johan- son, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. In NeurIPS, pages 1729-1736, 2007.","title":"Regret minimization in games with incomplete information","authors":"Martin Zinkevich, Michael Johanson, Michael Bowling, Carmelo Piccione","year":"2007","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Zinkevich, 2003] Martin Zinkevich. Online convex pro- gramming and generalized infinitesimal gradient ascent. In ICML, pages 928-936, 2003.","title":"Online convex programming and generalized infinitesimal gradient ascent","authors":"Martin Zinkevich","year":"2003","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}