{"type":"ScholarlyArticle","title":"Constructive Conflict-Driven Multi-Agent Reinforcement Learning for Strategic Diversity","authors":[{"name":"Yuxiang Mai","slug":"yuxiang-mai","given":"Yuxiang","family":"Mai","affiliation":"School of Artificial Intelligence , University of Chinese Academy of Sciences; CRISE , Institute of Automation , Chinese Academy of Sciences; The Key Laboratory of Cognition and Decision Intelligence for Complex Systems , Institute of Automation , Chinese Academy of Sciences"},{"name":"Qiyue Yin","slug":"qiyue-yin","given":"Qiyue","family":"Yin","affiliation":"School of Artificial Intelligence , University of Chinese Academy of Sciences; CRISE , Institute of Automation , Chinese Academy of Sciences; The Key Laboratory of Cognition and Decision Intelligence for Complex Systems , Institute of Automation , Chinese Academy of Sciences"},{"name":"Wancheng Ni","slug":"wancheng-ni","given":"Wancheng","family":"Ni","affiliation":"School of Artificial Intelligence , University of Chinese Academy of Sciences; CRISE , Institute of Automation , Chinese Academy of Sciences; The Key Laboratory of Cognition and Decision Intelligence for Complex Systems , Institute of Automation , Chinese Academy of Sciences"},{"name":"Pei Xu","slug":"pei-xu","given":"Pei","family":"Xu","affiliation":"CRISE , Institute of Automation , Chinese Academy of Sciences; The Key Laboratory of Cognition and Decision Intelligence for Complex Systems , Institute of Automation , Chinese Academy of Sciences"},{"name":"Kaiqi Huang","slug":"kaiqi-huang","given":"Kaiqi","family":"Huang","affiliation":"School of Artificial Intelligence , University of Chinese Academy of Sciences; CRISE , Institute of Automation , Chinese Academy of Sciences; The Key Laboratory of Cognition and Decision Intelligence for Complex Systems , Institute of Automation , Chinese Academy of Sciences"}],"abstract":"In recent years, diversity has emerged as a useful mechanism to enhance the efficiency of multi-agent reinforcement learning (MARL). However, existing methods predominantly focus on designing policies based on individual agent characteristics, often neglecting the interplay and mutual influence among agents during policy formation. To address this gap, we propose Competitive Diversity through Constructive Conflict (CoDiCon), a novel approach that incorporates competitive incentives into cooperative scenarios to encourage policy exchange and foster strategic diversity among agents. Drawing inspiration from sociological research, which highlights the benefits of moderate competition and constructive conflict in group decision-making, we design an intrinsic reward mechanism using ranking features to introduce competitive motivations. A centralized intrinsic reward module generates and distributes varying reward values to agents, ensuring an effective balance between competition and cooperation. By optimizing the parameterized centralized reward module to maximize environmental rewards, we reformulate the constrained bilevel optimization problem to align with the original task objectives. We evaluate our algorithm against state-of-the-art methods in the SMAC and GRF environments. Experimental results demonstrate that CoDiCon achieves superior performance, with competitive intrinsic rewards effectively promoting diverse and adaptive strategies among cooperative agents.","doi":"10.24963/ijcai.2025/24","url":"https://ijcai.org/proceedings/2025/24","pdf":"https://ijcai.org/proceedings/2025/0024.pdf","markdown":"https://ijcai.org/proceedings/2025/24.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/24","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":24,"pages":{"first":205,"last":213},"keywords":["Agent-based and Multi-agent Systems: MAS: Multi-agent learning","Machine Learning: ML: Reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Andrychowicz et al., 2016] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.","title":"Learning to learn by gradient descent by gradient descent","authors":"Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, Nando De Freitas","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Colby et al., 2015] Mitchell K Colby, Sepideh Kharaghani, Chris HolmesParker, and Kagan Tumer. Counterfactual exploration for improving multiagent learning. In AAMAS, pages 171-179, 2015.","title":"Counterfactual exploration for improving multiagent learning","authors":"Sepideh Mitchell K Colby, Chris Kharaghani, Kagan Holmesparker, Tumer","year":"2015","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Du et al., 2019] Yali Du, Lei Han, Meng Fang, Ji Liu, Tian- hong Dai, and Dacheng Tao. Liir: Learning individ- ual intrinsic reward in multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.","title":"Liir: Learning individual intrinsic reward in multi-agent reinforcement learning","authors":"Yali Du, Lei Han, Meng Fang, Ji Liu, Tianhong Dai, Dacheng Tao","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Foerster et al., 2018] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 32, 2018.","title":"Counterfactual multi-agent policy gradients","authors":"Foerster","year":"2018","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Haarnoja et al., 2019] Tuomas Haarnoja, Sehoon Ha, Au- rick Zhou, Jie Tan, George Tucker, and Sergey Levine. Learning to walk via deep reinforcement learning. In Antonio Bicchi, Hadas Kress-Gazit, and Seth Hutchin- son, editors, Robotics: Science and Systems XV, University of Freiburg, Freiburg im Breisgau, Germany, June 22-26, 2019, 2019.","title":"Learning to walk via deep reinforcement learning","authors":"Tuomas Haarnoja, Sehoon Ha, Aurick Zhou, Jie Tan, George Tucker, Sergey Levine","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hostallero et al., 2019] Wan Ju Kang David Earl Hostallero, Kyunghwan Son, Daewoo Kim, and Yung Yi Qtran. Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Pro- ceedings of the 31st International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 2019.","title":"Learning to factorize with transformation for cooperative multi-agent reinforcement learning","authors":"Wan Ju, Kang David, Earl Hostallero, Kyunghwan Son, Daewoo Kim, Yung Yi Qtran","year":"2019","venue":"Proceedings of the 31st International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2019] Yeping Hu, Alireza Nakhaei, Masayoshi Tomizuka, and Kikuo Fujimura. Interaction-aware deci- sion making with adaptive strategies under merging sce- narios. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 151-158. IEEE, 2019.","title":"Interaction-aware decision making with adaptive strategies under merging scenarios","authors":"Hu","year":"2019","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jiang and Lu, 2021] Jiechuan Jiang and Zongqing Lu. The emergence of individuality. In International Conference on Machine Learning, pages 4992-5001. PMLR, 2021.","title":"The emergence of individuality","authors":"Lu ; Jiang, Jiechuan Jiang, Zongqing Lu","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[King et al., 2009] Eden B King, Michelle R Hebl, and Daniel J Beal. Conflict and cooperation in diverse work- groups. Journal of Social Issues, 65(2):261-285, 2009.","title":"Conflict and cooperation in diverse workgroups","authors":"Michelle R Eden B King, Daniel J Hebl, Beal","year":"2009","venue":"Journal of Social Issues","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kirchmeyer and Cohen, 1992] Catherine Kirchmeyer and Aaron Cohen. Multicultural groups: Their performance and reactions with constructive conflict. Group & Organi- zation Management, 17(2):153-170, 1992.","title":"Multicultural groups: Their performance and reactions with constructive conflict","authors":"Cohen Kirchmeyer, Catherine Kirchmeyer, Aaron Cohen","year":"1992","venue":"Group & Organization Management","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2021] Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. Cel- ebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Information Processing Sys- tems, 34:3991-4002, 2021.","title":"Celebrating diversity in shared multi-agent reinforcement learning","authors":"Li","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competitive envi- ronments. Advances in neural information processing sys- tems, 30, 2017.","title":"Multiagent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, Pieter Openai, Igor Abbeel, Mordatch","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Oliehoek and Amato, 2016] Frans A Oliehoek and Christo- pher Amato. A concise introduction to decentralized POMDPs. Springer, 2016.","title":"A concise introduction to decentralized POMDPs","authors":"A Frans, Christopher Oliehoek, Amato","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rashid et al., 2020] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Fo- erster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178):1-51, 2020.","title":"Monotonic value function factorisation for deep multi-agent reinforcement learning","authors":"Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De, Gregory Witt, Jakob Farquhar, Shimon Foerster, Whiteson","year":"2020","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Samvelyan et al., 2019] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043, 2019.","title":"The starcraft multi-agent challenge","authors":"Tabish Mikayel Samvelyan, Christian Rashid, Schroeder De, Gregory Witt, Nantas Farquhar, Tim Gj Nardelli, Chia-Man Rudner, Hung, Jakob Philip Hs Torr, Shimon Foerster, Whiteson","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Santoro et al., 2016] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lill- icrap. Meta-learning with memory-augmented neural networks. In International conference on machine learning, pages 1842-1850. PMLR, 2016.","title":"Meta-learning with memory-augmented neural networks","authors":"Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, Timothy Lillicrap","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schulman, 2015] John Schulman. Trust region policy opti- mization. arXiv preprint arXiv:1502.05477, 2015.","title":"Trust region policy optimization","authors":"John Schulman","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sunehag et al., 2017] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296, 2017.","title":"Value-decomposition networks for cooperative multi-agent learning","authors":"Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech, Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Van der Maaten and Hinton, 2008] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.","title":"Visualizing data using t-sne","authors":"Van Der Maaten, Laurens Hinton, Geoffrey Van Der Maaten, Hinton","year":"2008","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2020] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. In International Conference on Learning Representations, 2020.","title":"Qplex: Duplex dueling multi-agent q-learning","authors":"Wang","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wiering, 2000] Marco A Wiering. Multi-agent reinforce- ment learning for traffic light control. In Machine Learn- ing: Proceedings of the Seventeenth International Confer- ence (ICML'2000), pages 1151-1158, 2000.","title":"Multi-agent reinforcement learning for traffic light control","authors":"A Marco, Wiering","year":"2000","venue":"Machine Learning: Proceedings of the Seventeenth International Conference (ICML'2000)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8:229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2018] Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. Ad- vances in neural information processing systems, 31, 2018.","title":"Meta-gradient reinforcement learning","authors":"Zhongwen Xu, David Hado P Van Hasselt, Silver","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yu et al., 2022] Chao Yu, Akash Velu, Eugene Vinitsky, Ji- axuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Sys- tems, 35:24611-24624, 2022.","title":"The surprising effectiveness of ppo in cooperative multi-agent games","authors":"Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, Yi Wu","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhou et al., 2020] Meng Zhou, Ziyu Liu, Pengwei Sui, Yix- uan Li, and Yuk Ying Chung. Learning implicit credit as- signment for cooperative multi-agent reinforcement learn- ing. arXiv preprint arXiv:2007.02529, 2020.","title":"Learning implicit credit assignment for cooperative multi-agent reinforcement learning","authors":"Meng Zhou, Ziyu Liu, Pengwei Sui, Yixuan Li, Yuk Ying, Chung","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}