{"type":"ScholarlyArticle","title":"A Successful Strategy for Multichannel Iterated Prisoner’s Dilemma","authors":[{"name":"Zhen Wang","slug":"zhen-wang","given":"Zhen","family":"Wang","affiliation":"School of Computer Science , Northwestern Polytechnical University; School of Cybersecurity , Northwestern Polytechnical University; School of Artificial Intelligence, OPtics and ElectroNics (iOPEN) , Northwestern Polytechnical University"},{"name":"Zhaoheng Cao","slug":"zhaoheng-cao","given":"Zhaoheng","family":"Cao","affiliation":"School of Computer Science , Northwestern Polytechnical University; School of Artificial Intelligence, OPtics and ElectroNics (iOPEN) , Northwestern Polytechnical University"},{"name":"Juan Shi","slug":"juan-shi","given":"Juan","family":"Shi","affiliation":"School of Automation , Northwestern Polytechnical University"},{"name":"Peican Zhu","slug":"peican-zhu","given":"Peican","family":"Zhu","affiliation":"School of Artificial Intelligence, OPtics and ElectroNics (iOPEN) , Northwestern Polytechnical University"},{"name":"Shuyue Hu","slug":"shuyue-hu","given":"Shuyue","family":"Hu","affiliation":"Shanghai Artificial Intelligence Laboratory"},{"name":"Chen Chu","slug":"chen-chu","given":"Chen","family":"Chu","affiliation":"School of Statistics and Mathematics , Yunnan University of Finance and Economics; School of Artificial Intelligence, OPtics and ElectroNics (iOPEN) , Northwestern Polytechnical University"}],"abstract":"Iterated prisoner’s dilemma (IPD) and its variants are fundamental models for understanding the evolution of cooperation in human society as well as AI systems. In this paper, we focus on multichannel IPD, and examine how an agent should behave to obtain generally high payoffs under this setting. \r\nWe propose a novel strategy that chooses to cooperate or defect by considering the difference in the cumulative number of defections between two agents.\r\nWe show that our proposed strategy is nice, retaliatory, and forgiving.\r\nMoreover, we analyze the performance of our proposed strategy across different scenarios, including the self-play settings with and without errors, as well as when facing various opponent strategies. In particular, we show that our proposed strategy is invincible and never loses to any opponent strategy in terms of the expected payoff. \r\nLast but not least, we empirically validate the evolutionary advantage of our strategy, and demonstrate its potential to serve as a catalyst for cooperation emergence.","doi":"10.24963/ijcai.2024/31","url":"https://ijcai.org/proceedings/2024/31","pdf":"https://ijcai.org/proceedings/2024/0031.pdf","markdown":"https://ijcai.org/proceedings/2024/31.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/31","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":31,"pages":{"first":274,"last":282},"keywords":["Agent-based and Multi-agent Systems: MAS: Coordination and cooperation","Game Theory and Economic Paradigms: GTEP: Noncooperative games","Agent-based and Multi-agent Systems: MAS: Agent-based simulation and emergence"],"videos":[],"references":[{"position":1,"raw":"[Akin, 2016] Ethan Akin. The iterated prisoner's dilemma: good strategies and their dynamics. Ergodic Theory, Ad- vances in Dynamical Systems, pages 77-107, 2016.","title":"The iterated prisoner's dilemma: good strategies and their dynamics","authors":"Ethan Akin, Akin","year":"2016","venue":"Ergodic Theory, Advances in Dynamical Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Antal et al., 2009] Tibor Antal, Hisashi Ohtsuki, John Wakeley, Peter D Taylor, and Martin A Nowak. Evolu- tion of cooperation by phenotypic similarity. Proceedings of the National Academy of Sciences, 106(21):8597-8600, 2009.","title":"Evolution of cooperation by phenotypic similarity","authors":"Antal","year":"2009","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Axelrod and Hamilton, 1981] Robert Axelrod and William D Hamilton. The evolution of cooperation. science, 211(4489):1390-1396, 1981.","title":"The evolution of cooperation","authors":"Robert Axelrod, William D Hamilton","year":"1981","venue":"science","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Baker, 2020] Bowen Baker. Emergent reciprocity and team formation from randomized uncertain social prefer- ences. Advances in neural information processing systems, 33:15786-15799, 2020.","title":"Emergent reciprocity and team formation from randomized uncertain social preferences","authors":"Bowen Baker, Baker","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bodnar et al., 2020] Cristian Bodnar, Ben Day, and Pietro Lió. Proximal distilled evolutionary reinforcement learn- ing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3283-3290, 2020.","title":"Proximal distilled evolutionary reinforcement learning","authors":"Cristian Bodnar, Ben Day, Pietro Lió","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chu et al., 2022] Chen Chu, Yong Li, Jinzhuo Liu, Shuyue Hu, Xuelong Li, and Zhen Wang. A formal model for mul- tiagent q-learning dynamics on regular graphs. In IJCAI, pages 194-200, 2022.","title":"A formal model for multiagent q-learning dynamics on regular graphs","authors":"Chen Chu, Yong Li, Jinzhuo Liu, Shuyue Hu, Xuelong Li, Zhen Wang","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/28"},{"position":7,"raw":"[Donahue et al., 2020] Kate Donahue, Oliver P Hauser, Mar- tin A Nowak, and Christian Hilbe. Evolving cooperation in multichannel games. Nature communications, 11(1):3885, 2020.","title":"Evolving cooperation in multichannel games","authors":"Kate Donahue, Oliver P Hauser, Martin A Nowak, Christian Hilbe","year":"2020","venue":"Nature communications","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Farrell and Maskin, 1989] Joseph Farrell and Eric Maskin. Renegotiation in repeated games. Games and economic behavior, 1(4):327-360, 1989.","title":"Renegotiation in repeated games","authors":"Maskin Farrell, Joseph Farrell, Eric Maskin","year":"1989","venue":"Games and economic behavior","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Foerster et al., 2018] Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. In Proceedings of the 17th International Con- ference on Autonomous Agents and MultiAgent Systems, pages 122-130, 2018.","title":"Learning with opponent-learning awareness","authors":"Jakob Foerster, Maruan Richard Y Chen, Shimon Al-Shedivat, Pieter Whiteson, Igor Abbeel, Mordatch","year":"2018","venue":"Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[García and van Veelen, 2016] Julián García and Matthijs van Veelen. In and out of equilibrium i: Evolution of strategies in repeated games with discounting. Journal of Economic Theory, 161:161-189, 2016.","title":"In and out of equilibrium i: Evolution of strategies in repeated games with discounting","authors":"García, Julián Van Veelen ;, Matthijs García, Van Veelen","year":"2016","venue":"Journal of Economic Theory","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hao et al., 2018] Dong Hao, Kai Li, and Tao Zhou. Payoff control in the iterated prisoner's dilemma. In Proceedings of the 27th International Joint Conference on Artificial In- telligence, pages 296-302, 2018.","title":"Payoff control in the iterated prisoner's dilemma","authors":"Dong Hao, Kai Li, Tao Zhou","year":"2018","venue":"Proceedings of the 27th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/41"},{"position":12,"raw":"[Hilbe et al., 2018] Christian Hilbe, Krishnendu Chatterjee, and Martin A Nowak. Partners and rivals in direct reci- procity. Nature human behaviour, 2(7):469-477, 2018.","title":"Partners and rivals in direct reciprocity","authors":"Hilbe","year":"2018","venue":"Nature human behaviour","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Leibo et al., 2017] Joel Z Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. Multi- agent reinforcement learning in sequential social dilem- mas. In Proceedings of the 16th Conference on Au- tonomous Agents and MultiAgent Systems, pages 464-473, 2017.","title":"Multiagent reinforcement learning in sequential social dilemmas","authors":"Vinicius Joel Z Leibo, Marc Zambaldi, Janusz Lanctot, Thore Marecki, Graepel","year":"2017","venue":"Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lerer and Peysakhovich, 2017] Adam Lerer and Alexander Peysakhovich. Maintaining cooperation in complex so- cial dilemmas using deep reinforcement learning. arXiv preprint arXiv:1707.01068, 2017.","title":"Maintaining cooperation in complex social dilemmas using deep reinforcement learning","authors":"Peysakhovich Lerer, Adam Lerer, Alexander Peysakhovich","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li and Hao, 2019] Kai Li and Dong Hao. Cooperation enforcement and collusion resistance in repeated public goods games. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2085-2092, 2019.","title":"Cooperation enforcement and collusion resistance in repeated public goods games","authors":"Li, Hao ; ] Kai Li, Dong Hao","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2022] Juan Li, Xiaowei Zhao, Bing Li, Char- lotte SL Rossetti, Christian Hilbe, and Haoxiang Xia. Evo- lution of cooperation through cumulative reciprocity. Na- ture Computational Science, 2(10):677-686, 2022.","title":"Evolution of cooperation through cumulative reciprocity","authors":"Juan Li, Xiaowei Zhao, Bing Li, Charlotte Sl Rossetti, Christian Hilbe, Haoxiang Xia","year":"2022","venue":"Nature Computational Science","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lu et al., 2022] Christopher Lu, Timon Willi, Christian A Schroeder De Witt, and Jakob Foerster. Model-free op- ponent shaping. In International Conference on Machine Learning, pages 14398-14411. PMLR, 2022.","title":"Model-free opponent shaping","authors":"Christopher Lu, Timon Willi, Christian A Schroeder De Witt, Jakob Foerster","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mailath and Samuelson, 2006] George J Mailath and Larry Samuelson. Repeated games and reputations: long-run relationships. Oxford university press, 2006.","title":"Repeated games and reputations: long-run relationships","authors":"J George, Larry Mailath, Samuelson","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Miller, 1985] Nicholas R Miller. Nice strategies finish first: A review of the evolution of cooperation. Politics and the Life Sciences, 4(1):86-91, 1985.","title":"Nice strategies finish first: A review of the evolution of cooperation","authors":"Nicholas R Miller","year":"1985","venue":"Politics and the Life Sciences","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mittal and Deb, 2009] Shashi Mittal and Kalyanmoy Deb. Optimal strategies of the iterated prisoner's dilemma prob- lem for multiple conflicting objectives. IEEE Transactions on Evolutionary Computation, 13(3):554-565, 2009.","title":"Optimal strategies of the iterated prisoner's dilemma problem for multiple conflicting objectives","authors":"Deb ; Mittal, Shashi Mittal, Kalyanmoy Deb","year":"2009","venue":"IEEE Transactions on Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nowak and Sigmund, 1992] Martin A Nowak and Karl Sig- mund. Tit for tat in heterogeneous populations. Nature, 355(6357):250-253, 1992.","title":"Tit for tat in heterogeneous populations","authors":"A Martin, Karl Nowak, Sigmund","year":"1992","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nowak and Sigmund, 1993] Martin Nowak and Karl Sig- mund. A strategy of win-stay, lose-shift that outper- forms tit-for-tat in the prisoner's dilemma game. Nature, 364(6432):56-58, 1993.","title":"A strategy of win-stay, lose-shift that outperforms tit-for-tat in the prisoner's dilemma game","authors":"Sigmund Nowak","year":"1993","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"William H Press and Freeman J Dyson. Iterated prisoner's dilemma contains strategies that dominate any evolutionary opponent. Proceedings of the National Academy of Sciences, 109(26):10409-10413, 2012.","title":"Iterated prisoner's dilemma contains strategies that dominate any evolutionary opponent","authors":"H William, Freeman J Press, Dyson","year":"2012","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Roughgarden, 2010] Tim Roughgarden. Algorithmic game theory. Communications of the ACM, 53(7):78-86, 2010.","title":"Algorithmic game theory","authors":"Tim Roughgarden","year":"2010","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Selten and Hammerstein, 1984] Reinhard Selten and Peter Hammerstein. Gaps in harley's argument on evolution- arily stable learning rules and in the logic of \"tit for tat\". Behavioral and Brain Sciences, 7(1):115-116, 1984.","title":"Gaps in harley's argument on evolutionarily stable learning rules and in the logic of \"tit for tat","authors":"Hammerstein Selten, Reinhard Selten, Peter Hammerstein","year":"1984","venue":"Behavioral and Brain Sciences","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Van Veelen, 2012] Matthijs Van Veelen. Robustness against indirect invasions. Games and Economic Behavior, 74(1):382-393, 2012.","title":"Robustness against indirect invasions","authors":"; Matthijs Van Veelen, Van Veelen","year":"2012","venue":"Games and Economic Behavior","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vinitsky et al., 2023] Eugene Vinitsky, Raphael Köster, John P Agapiou, Edgar A Duéñez-Guzmán, Alexan- der S Vezhnevets, and Joel Z Leibo. A learning agent that acquires social norms from public sanctions in de- centralized multi-agent settings. Collective Intelligence, 2(2):26339137231162025, 2023.","title":"A learning agent that acquires social norms from public sanctions in decentralized multi-agent settings","authors":"Eugene Vinitsky, Raphael Köster, John P Agapiou, Edgar A Duéñez-Guzmán, Alexander S Vezhnevets, Joel Z Leibo","year":"2023","venue":"Collective Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang and Lin, 2020] Shiheng Wang and Fangzhen Lin. Nice invincible strategy for the average-payoff ipd. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 2268-2275, 2020.","title":"Nice invincible strategy for the average-payoff ipd","authors":"Wang, Lin ;, Shiheng Wang, Fangzhen Lin","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2022] Zhen Wang, Chunjiang Mu, Shuyue Hu, Chen Chu, and Xuelong Li. Modelling the dynamics of regret minimization in large agent populations: a master equation approach. In IJCAI, pages 534-540, 2022.","title":"Modelling the dynamics of regret minimization in large agent populations: a master equation approach","authors":"Zhen Wang, Chunjiang Mu, Shuyue Hu, Chen Chu, Xuelong Li","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/76"},{"position":30,"raw":"[Zhao et al., 2022] Stephen Zhao, Chris Lu, Roger B Grosse, and Jakob Foerster. Proximal learning with opponent- learning awareness. Advances in Neural Information Pro- cessing Systems, 35:26324-26336, 2022.","title":"Proximal learning with opponentlearning awareness","authors":"Stephen Zhao, Chris Lu, Roger B Grosse, Jakob Foerster","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}