{"type":"ScholarlyArticle","title":"To Promote Full Cooperation in Social Dilemmas, Agents Need to Unlearn Loyalty","authors":[{"name":"Chin-wing Leung","slug":"chin-wing-leung","given":"Chin-wing","family":"Leung","affiliation":"Department of Computer Science , University of Warwick"},{"name":"Tom Lenaerts","slug":"tom-lenaerts","given":"Tom","family":"Lenaerts","affiliation":"Machine Learning Group , Université Libre de Bruxelles; Artificial Intelligence Lab , Vrije Universiteit Brussel; Center for Human-Compatible AI , UC Berkeley"},{"name":"Paolo Turrini","slug":"paolo-turrini","given":"Paolo","family":"Turrini","affiliation":"Department of Computer Science , University of Warwick"}],"abstract":"If given the choice, what strategy should agents use to switch partners in strategic social interactions?  While many analyses have been performed on specific switching heuristics, showing how and when these lead to more cooperation, no insights have been provided into which rule will actually be learnt by agents when given the freedom to do so.\r\nStarting from a baseline model that has demonstrated the potential of rewiring for cooperation, we provide answers to this question over the full spectrum of social dilemmas.  Multi-agent Q-learning with Boltzmann exploration is used to learn when to sever or maintain an association.  In both the Prisoner's Dilemma and the Stag Hunt games we observe that the Out-for-Tat rewiring rule, breaking ties with other agents choosing socially undesirable actions, becomes dominant, confirming at the same time that cooperation flourishes when rewiring is fast enough relative to imitation. Nonetheless, in the transitory region before full cooperation, a Stay strategy, keeping a connection at all costs, remains present, which shows that loyalty needs to be overcome for full cooperation to emerge.\r\nIn conclusion, individuals learn cooperation-promoting rewiring rules but need to overcome a kind of loyalty to achieve full cooperation in the full spectrum of social dilemmas.","doi":"10.24963/ijcai.2024/13","url":"https://ijcai.org/proceedings/2024/13","pdf":"https://ijcai.org/proceedings/2024/0013.pdf","markdown":"https://ijcai.org/proceedings/2024/13.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/13","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":13,"pages":{"first":111,"last":119},"keywords":["Agent-based and Multi-agent Systems: MAS: Agent-based simulation and emergence","Agent-based and Multi-agent Systems: MAS: Multi-agent learning","Agent-based and Multi-agent Systems: MAS: Coordination and cooperation","Agent-based and Multi-agent Systems: MAS: Agent societies"],"videos":[],"references":[{"position":1,"raw":"[Anastassacos et al., 2020] Nicolas Anastassacos, Stephen Hailes, and Mirco Musolesi. Partner selection for the emergence of cooperation in multi-agent systems using reinforcement learning. In The Thirty-Fourth AAAI Con- ference on Artificial Intelligence, AAAI 2020, The Thirty- Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7047-7054. AAAI Press, 2020.","title":"Partner selection for the emergence of cooperation in multi-agent systems using reinforcement learning","authors":"Anastassacos","year":"2020","venue":"The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bara et al., 2022] Jacques Bara, Paolo Turrini, and Giulia Andrighetto. Enabling imitation-based cooperation in dy- namic social networks. Auton. Agents Multi Agent Syst., 36(2):34, 2022.","title":"Enabling imitation-based cooperation in dynamic social networks","authors":"Jacques Bara, Paolo Turrini, Giulia Andrighetto","year":"2022","venue":"Auton. Agents Multi Agent Syst","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bloembergen et al., 2015] Daan Bloembergen, Karl Tuyls, Daniel Hennes, and Michael Kaisers. Evolutionary dy- namics of multi-agent learning: A survey. J. Artif. Intell. Res., 53:659-697, 2015.","title":"Evolutionary dynamics of multi-agent learning: A survey","authors":"Daan Bloembergen, Karl Tuyls, Daniel Hennes, Michael Kaisers","year":"2015","venue":"J. Artif. Intell. Res","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Börgers and Sarin, 1997] Tilman Börgers and Rajiv Sarin. Learning through reinforcement and replicator dynamics. Journal of Economic Theory, 77(1):1-14, 1997.","title":"Learning through reinforcement and replicator dynamics","authors":"Sarin Börgers, Tilman Börgers, Rajiv Sarin","year":"1997","venue":"Journal of Economic Theory","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Eshel and Cavalli-Sforza, 1982] Ilan Eshel and L. L. Cavalli-Sforza. Assortment of encounters and evolution of cooperativeness. Proceedings of the National Academy of Sciences, 79(4):1331-1335, 1982.","title":"Assortment of encounters and evolution of cooperativeness","authors":"Ilan Eshel, L L Cavalli-Sforza","year":"1982","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Foley et al., 2018] Michael Foley, Patrick Forber, Rory Smead, and Christoph Riedl. Conflict and convention in dynamic networks. Journal of The Royal Society Inter- face, 15(140):20170835, 2018.","title":"Conflict and convention in dynamic networks","authors":"Michael Foley, Patrick Forber, Rory Smead, Christoph Riedl","year":"2018","venue":"Journal of The Royal Society Interface","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fulker et al., 2021] Zachary Fulker, Patrick Forber, Rory Smead, and Christoph Riedl. Spite is contagious in dy- namic networks. Nature Communications, 12(1):260, 2021.","title":"Spite is contagious in dynamic networks","authors":"Zachary Fulker, Patrick Forber, Rory Smead, Christoph Riedl","year":"2021","venue":"Nature Communications","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gilbert, 1995] Nigel Gilbert. Emergence in social simula- tion. In Nigel Gilbert and Rosaria Conte, editors, Artificial Societies: The Computer Simulation Of Social Life. Rout- ledge, 1995.","title":"Emergence in social simulation","authors":"Nigel Gilbert","year":"1995","venue":"Artificial Societies: The Computer Simulation Of Social Life. Routledge","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Leung and Turrini, 2024] Chin-wing Leung and Paolo Tur- rini. Learning partner selection rules that sustain cooper- ation in social dilemmas with the option of opting out. In Mehdi Dastani, Jaime Simão Sichman, Natasha Alechina, and Virginia Dignum, editors, Proceedings of the 23rd In- ternational Conference on Autonomous Agents and Mul- tiagent Systems, AAMAS 2024, Auckland, New Zealand, May 6-10, 2024, pages 1110-1118. ACM, 2024.","title":"Learning partner selection rules that sustain cooperation in social dilemmas with the option of opting out","authors":"Turrini Leung","year":"2024","venue":"Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2024","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Nowak, 2006] Martin A. Nowak. Five rules for the evolu- tion of cooperation. Science, 314(5805):1560-1563, 2006.","title":"Five rules for the evolution of cooperation","authors":"Martin A Nowak","year":"2006","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ohtsuki et al., 2006] Hisashi Ohtsuki, Christoph Hauert, Erez Lieberman, and Martin A Nowak. A simple rule for the evolution of cooperation on graphs and social net- works. Nature, 441(7092):502-505, 2006.","title":"A simple rule for the evolution of cooperation on graphs and social networks","authors":"Hisashi Ohtsuki, Christoph Hauert, Erez Lieberman, Martin A Nowak","year":"2006","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Pacheco et al., 2006] Jorge M. Pacheco, Arne Traulsen, and Martin A. Nowak. Coevolution of strategy and structure in complex networks with dynamical linking. Phys. Rev. Lett., 97:258103, Dec 2006.","title":"Coevolution of strategy and structure in complex networks with dynamical linking","authors":"Jorge M Pacheco, Arne Traulsen, Martin A Nowak","year":"2006","venue":"Phys. Rev. Lett","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pérolat et al., 2017] Julien Pérolat, Joel Z. Leibo, Vinícius Flores Zambaldi, Charles Beattie, Karl Tuyls, and Thore Graepel. A multi-agent reinforcement learning model of common-pool resource appropriation. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Ro- man Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 3643-3652, 2017.","title":"A multi-agent reinforcement learning model of common-pool resource appropriation","authors":"Julien Pérolat, Joel Z Leibo, Vinícius Flores Zambaldi, Charles Beattie, Karl Tuyls, Thore Graepel","year":"2017","venue":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Perreau de Pinninck et al., 2010] Adrian Perreau de Pin- ninck, Carles Sierra, and Marco Schorlemmer. A multi- agent network for peer norm enforcement. Autonomous Agents and Multi-Agent Systems, 21(3):397-424, Nov 2010.","title":"A multiagent network for peer norm enforcement","authors":"Adrian Perreau De Pinninck, Carles Sierra, Marco Schorlemmer","year":"2010","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pujol et al., 2002] Josep M. Pujol, Ramon Sangüesa, and Jordi Delgado. Extracting reputation in multi agent sys- tems by means of social network topology. In Proceedings of the First International Joint Conference on Autonomous Agents and Multiagent Systems: Part 1, AAMAS '02, page 467-474, New York, NY, USA, 2002. Association for Computing Machinery.","title":"Extracting reputation in multi agent systems by means of social network topology","authors":"M Josep, Ramon Pujol, Jordi Sangüesa, Delgado","year":"2002","venue":"Proceedings of the First International Joint Conference on Autonomous Agents and Multiagent Systems: Part 1, AAMAS '02","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rand et al., 2011] David G. Rand, Samuel Arbesman, and Nicholas A. Christakis. Dynamic social networks promote cooperation in experiments with humans. Proceedings of the National Academy of Sciences, 108(48):19193-19198, 2011.","title":"Dynamic social networks promote cooperation in experiments with humans","authors":"David G Rand, Samuel Arbesman, Nicholas A Christakis","year":"2011","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Roca et al., 2009] Carlos P. Roca, José A. Cuesta, and An- gel Sánchez. Evolutionary game theory: Temporal and spatial effects beyond replicator dynamics. Physics of Life Reviews, 6(4):208-249, 2009.","title":"Angel Sánchez. Evolutionary game theory: Temporal and spatial effects beyond replicator dynamics","authors":"Carlos P Roca, José A Cuesta","year":"2009","venue":"Physics of Life Reviews","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sabater and Sierra, 2002] Jordi Sabater and Carles Sierra. Reputation and social network analysis in multi-agent sys- tems. In Proceedings of the first international joint confer- ence on Autonomous agents and multiagent systems part 1 -AAMAS '02, New York, New York, USA, 2002. ACM Press.","title":"Jordi Sabater and Carles Sierra. Reputation and social network analysis in multi-agent systems","authors":"Sierra Sabater","year":"2002","venue":"Proceedings of the first international joint conference on Autonomous agents and multiagent systems part 1 -AAMAS '02","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sabater-Mir et al., 2006] Jordi Sabater-Mir, Mario Paolucci, and Rosaria Conte. Repage: Reputation and image among limited autonomous partners. Journal of Artificial Societies and Social Simulation, 9(2):3, 2006.","title":"Repage: Reputation and image among limited autonomous partners","authors":"Sabater-Mir","year":"2006","venue":"Journal of Artificial Societies and Social Simulation","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Salazar et al., 2011] Norman Salazar, Juan A. Rodriguez- Aguilar, Josep Ll. Arcos, Ana Peleteiro, and Juan C. Burguillo-Rial. Emerging cooperation on complex net- works. In The 10th International Conference on Au- tonomous Agents and Multiagent Systems -Volume 2, AA- MAS '11, page 669-676, Richland, SC, 2011. Interna- tional Foundation for Autonomous Agents and Multiagent Systems.","title":"Emerging cooperation on complex networks","authors":"Norman Salazar, Juan A Rodriguez-Aguilar, Josep Ll, Ana Arcos, Juan C Peleteiro, Burguillo-Rial","year":"2011","venue":"The 10th International Conference on Au-tonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Santos et al., 2006a] Francisco C. Santos, Jorge M. Pacheco, and Tom Lenaerts. Cooperation prevails when individuals adjust their social ties. PLoS Comput. Biol., 2(10), 2006.","title":"Cooperation prevails when individuals adjust their social ties","authors":"Santos","year":"2006","venue":"PLoS Comput. Biol","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Santos et al., 2006b] Francisco C. Santos, Jorge M. Pacheco, and Tom Lenaerts. Evolutionary dynamics of social dilemmas in structured heterogeneous popula- tions. Proceedings of the National Academy of Sciences, 103(9):3490-3494, 2006.","title":"Evolutionary dynamics of social dilemmas in structured heterogeneous populations","authors":"Santos","year":"2006","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Santos et al., 2018] Fernando P. Santos, Jorge M. Pacheco, and Francisco C. Santos. Social norms of cooperation with costly reputation building. In Sheila A. McIlraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence, (AAAI- 18), the 30th innovative Applications of Artificial Intel- ligence (IAAI-18), and the 8th AAAI Symposium on Ed- ucational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 4727-4734. AAAI Press, 2018.","title":"Social norms of cooperation with costly reputation building","authors":"Fernando P Santos, Jorge M Pacheco, Francisco C Santos","year":"2018","venue":"AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Segbroeck et al., 2009] Sven Van Segbroeck, Francisco C. Santos, Ann Nowé, Jorge M. Pacheco, and Tom Lenaerts. The coevolution of loyalty and cooperation. In Proceed- ings of the IEEE Congress on Evolutionary Computation, CEC 2009, Trondheim, Norway, 18-21 May, 2009, pages 500-505. IEEE, 2009.","title":"The coevolution of loyalty and cooperation","authors":"Sven Van Segbroeck, Francisco C Santos, Ann Nowé, Jorge M Pacheco, Tom Lenaerts","year":"2009","venue":"Proceedings of the IEEE Congress on Evolutionary Computation, CEC 2009","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Segbroeck et al., 2010] Sven Van Segbroeck, Steven de Jong, Ann Nowé, Francisco C. Santos, and Tom Lenaerts. Learning to coordinate in complex networks. Adapt. Behav., 18(5):416-427, 2010.","title":"Learning to coordinate in complex networks","authors":"Sven Van Segbroeck, Steven De Jong, Ann Nowé, Francisco C Santos, Tom Lenaerts","year":"2010","venue":"Adapt. Behav","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Traulsen et al., 2006] Arne Traulsen, Martin A Nowak, and Jorge M Pacheco. Stochastic dynamics of invasion and fixation. Physical Review E, 74(1):011909, 2006.","title":"Stochastic dynamics of invasion and fixation","authors":"Arne Traulsen, Martin A Nowak, Jorge M Pacheco","year":"2006","venue":"Physical Review E","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2012] Jing Wang, Siddharth Suri, and Dun- can J. Watts. Cooperation and assortativity with dynamic partner updating. Proceedings of the National Academy of Sciences, 109(36):14363-14368, 2012.","title":"Cooperation and assortativity with dynamic partner updating","authors":"Jing Wang, Siddharth Suri, Duncan J Watts","year":"2012","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Watkins and Dayan, 1992] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3):279- 292, 1992.","title":"Q-learning","authors":"Jch Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2016] Bo-Yu Zhang, Song-Jia Fan, Cong Li, Xiu-Deng Zheng, Jian-Zhang Bao, Ross Cressman, and Yi Tao. Opting out against defection leads to stable co- existence with cooperation. Scientific reports, 6:35902, October 2016.","title":"Opting out against defection leads to stable coexistence with cooperation","authors":"Bo-Yu Zhang, Song-Jia Fan, Cong Li, Xiu-Deng Zheng, Jian-Zhang Bao, Ross Cressman, Yi Tao","year":"2016","venue":"Scientific reports","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zheng et al., 2017] Xiu-Deng Zheng, Cong Li, Jie-Ru Yu, Shi-Chang Wang, Song-Jia Fan, Bo-Yu Zhang, and Yi Tao. A simple rule of direct reciprocity leads to the stable co- existence of cooperation and defection in the prisoner's dilemma game. Journal of Theoretical Biology, 420:12- 17, 2017.","title":"A simple rule of direct reciprocity leads to the stable coexistence of cooperation and defection in the prisoner's dilemma game","authors":"Xiu-Deng Zheng, Cong Li, Jie-Ru Yu, Shi-Chang Wang, Song-Jia Fan, Bo-Yu Zhang, Yi Tao","year":"2017","venue":"Journal of Theoretical Biology","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}