{"type":"ScholarlyArticle","title":"Group-Aware Coordination Graph for Multi-Agent Reinforcement Learning","authors":[{"name":"Wei Duan","slug":"wei-duan","given":"Wei","family":"Duan","affiliation":"Australian Artificial Intelligence Institute (AAII) , University of Technology Sydney"},{"name":"Jie Lu","slug":"jie-lu","given":"Jie","family":"Lu","affiliation":"Australian Artificial Intelligence Institute (AAII) , University of Technology Sydney"},{"name":"Junyu Xuan","slug":"junyu-xuan","given":"Junyu","family":"Xuan","affiliation":"Australian Artificial Intelligence Institute (AAII) , University of Technology Sydney"}],"abstract":"Cooperative Multi-Agent Reinforcement Learning (MARL) necessitates seamless collaboration among agents, often represented by an underlying relation graph. Existing methods for learning this graph primarily focus on agent-pair relations, neglecting higher-order relationships. While several approaches attempt to extend cooperation modelling to encompass behaviour similarities within groups, they commonly fall short in concurrently learning the latent graph, thereby constraining the information exchange among partially observed agents. To overcome these limitations, we present a novel approach to infer the Group-Aware Coordination Graph (GACG), which is designed to capture both the cooperation between agent pairs based on current observations and group-level dependencies from behaviour patterns observed across trajectories. \r\nThis graph is further used in graph convolution for information exchange between agents during decision-making. To further ensure behavioural consistency among agents within the same group, we introduce a group distance loss, which promotes group cohesion and encourages specialization between groups. Our evaluations, conducted on StarCraft II micromanagement tasks, demonstrate GACG's superior performance. An ablation study further provides experimental evidence of the effectiveness of each component of our method.","doi":"10.24963/ijcai.2024/434","url":"https://ijcai.org/proceedings/2024/434","pdf":"https://ijcai.org/proceedings/2024/0434.pdf","markdown":"https://ijcai.org/proceedings/2024/434.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/434","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":434,"pages":{"first":3926,"last":3934},"keywords":["Machine Learning: ML: Multiagent Reinforcement Learning","Agent-based and Multi-agent Systems: MAS: Coordination and cooperation"],"videos":[],"references":[{"position":1,"raw":"[Boehmer et al., 2020] Wendelin Boehmer, Vitaly Kurin, and Shimon Whiteson. Deep coordination graphs. In Pro- ceedings of the 37th International Conference on Machine Learning (ICML 2020), Virtual Event, volume 119 of Pro- ceedings of Machine Learning Research, pages 980-991.","title":"Deep coordination graphs","authors":"Wendelin Boehmer, Shimon Vitaly Kurin, Whiteson","year":null,"venue":"Proceedings of the 37th International Conference on Machine Learning (ICML 2020), Virtual Event","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"PMLR, 2020.","title":null,"authors":"Pmlr","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cui et al., 2020] Jingjing Cui, Yuanwei Liu, and Arumugam Nallanathan. Multi-agent reinforcement learning-based re- source allocation for UAV networks. IEEE Trans. Wirel. Commun., 19(2):729-743, 2020.","title":"Multi-agent reinforcement learning-based resource allocation for UAV networks","authors":"Jingjing Cui, Yuanwei Liu, Arumugam Nallanathan","year":"2020","venue":"IEEE Trans. Wirel. Commun","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Duan et al., 2022] Wei Duan, Junyu Xuan, Maoying Qiao, and Jie Lu. Learning from the dark: Boosting graph convolutional neural networks with diverse negative sam- ples. In Thirty-Sixth AAAI Conference on Artificial In- telligence (AAAI 2022), Virtual Event, pages 6550-6558. AAAI Press, 2022.","title":"Learning from the dark: Boosting graph convolutional neural networks with diverse negative samples","authors":"Wei Duan, Junyu Xuan, Maoying Qiao, Jie Lu","year":"2022","venue":"Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI 2022), Virtual Event","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Duan et al., 2023] Wei Duan, Junyu Xuan, Maoying Qiao, and Jie Lu. Graph convolutional neural networks with di- verse negative samples via decomposed determinant point processes. IEEE Trans. Neural Networks Learn. Syst., 2023.","title":"Graph convolutional neural networks with diverse negative samples via decomposed determinant point processes","authors":"Wei Duan, Junyu Xuan, Maoying Qiao, Jie Lu","year":"2023","venue":"IEEE Trans. Neural Networks Learn. Syst","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Duan et al., 2024a] Wei Duan, Jie Lu, Yu Guang Wang, and Junyu Xuan. Layer-diverse negative sampling for graph neural networks. Transactions on Machine Learning Re- search, 2024.","title":"Layer-diverse negative sampling for graph neural networks","authors":"Duan","year":"2024","venue":"Transactions on Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Duan et al., 2024b] Wei Duan, Jie Lu, and Junyu Xuan. Inferring latent temporal sparse coordination graph for multi-agent reinforcement learning. CoRR, abs/2403.19253, 2024.","title":"Inferring latent temporal sparse coordination graph for multi-agent reinforcement learning","authors":"Duan","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Iqbal et al., 2021] Shariq Iqbal, Christian A. Schröder de Witt, Bei Peng, Wendelin Boehmer, Shimon White- son, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning (ICML 2021), 18-24 July, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 4596- 4606. PMLR, 2021.","title":"Randomized entity-wise factorization for multi-agent reinforcement learning","authors":"Shariq Iqbal, Christian A Schröder De Witt, Bei Peng, Wendelin Boehmer, Shimon Whiteson, Fei Sha","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning (ICML 2021)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiang et al., 2020] Jiechuan Jiang, Chen Dun, Tiejun Huang, and Zongqing Lu. Graph convolutional reinforce- ment learning. In 8th International Conference on Learn- ing Representations (ICLR 2020), Addis Ababa, Ethiopia, 2020.","title":"Graph convolutional reinforcement learning","authors":"Jiang","year":"2020","venue":"8th International Conference on Learning Representations (ICLR 2020)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2021] Sheng Li, Jayesh K. Gupta, Peter Morales, Ross E. Allen, and Mykel J. Kochenderfer. Deep implicit coordination graphs for multi-agent reinforcement learn- ing. In AAMAS '21: 20th International Conference on Au- tonomous Agents and Multiagent Systems (AAMAS 2021), Virtual Event, United Kingdom, pages 764-772. ACM, 2021.","title":"Deep implicit coordination graphs for multi-agent reinforcement learning","authors":"Sheng Li, Jayesh K Gupta, Peter Morales, Ross E Allen, Mykel J Kochenderfer","year":"2021","venue":"AAMAS '21: 20th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2021), Virtual Event","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2020a] Iou-Jen Liu, Raymond A Yeh, and Alexander G Schwing. Pic: permutation invariant critic for multi-agent deep reinforcement learning. In Proceedings of the 3rd Conference on Robot Learning (CoRL 2019), Osaka, Japan, pages 590-602, 2020.","title":"Pic: permutation invariant critic for multi-agent deep reinforcement learning","authors":"Liu","year":"2020","venue":"Proceedings of the 3rd Conference on Robot Learning (CoRL 2019)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2020b] Yong Liu, Weixun Wang, Yujing Hu, Jianye Hao, Xingguo Chen, and Yang Gao. Multi-agent game abstraction via graph attention neural network. In The Thirty-Fourth AAAI Conference on Artificial Intelli- gence (AAAI 2020), New York, NY, USA,, pages 7211- 7218. AAAI Press, 2020.","title":"Multi-agent game abstraction via graph attention neural network","authors":"Liu","year":"2020","venue":"The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI 2020)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Oliehoek and Amato, 2016] Frans A. Oliehoek and Christo- pher Amato. A Concise Introduction to Decentral- ized POMDPs. Springer Briefs in Intelligent Systems. Springer, 2016.","title":"A Concise Introduction to Decentralized POMDPs","authors":"Frans A Oliehoek, Christopher Amato","year":"2016","venue":"Springer Briefs in Intelligent Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Orr and Dutta, 2023] James Orr and Ayan Dutta. Multi- agent deep reinforcement learning for multi-robot appli- cations: A survey. Sensors, 23(7):3625, 2023.","title":"Multiagent deep reinforcement learning for multi-robot applications: A survey","authors":"Dutta Orr","year":"2023","venue":"Sensors","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pacchiano et al., 2020] Aldo Pacchiano, Jack Parker- Holder, Yunhao Tang, Krzysztof Choromanski, Anna Choromanska, and Michael Jordan. Learning to score behaviors for guided policy optimization. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, (ICML 2020), volume 119 of Proceedings of Machine Learning Research, pages 7445-7454, 13-18 Jul 2020.","title":"Learning to score behaviors for guided policy optimization","authors":"Aldo Pacchiano, Jack Parker-Holder, Yunhao Tang, Krzysztof Choromanski, Anna Choromanska, Michael Jordan","year":"2020","venue":"Proceedings of the 37th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Phan et al., 2021] Thomy Phan, Fabian Ritz, Lenz Belzner, Philipp Altmann, Thomas Gabor, and Claudia Linnhoff- Popien. VAST: value function factorization with variable agent sub-teams. In Advances in Neural Information Pro- cessing Systems 34: Annual Conference on Neural Infor- mation Processing Systems (NIPS 2021), December 6-14, virtual, pages 24018-24032, 2021.","title":"VAST: value function factorization with variable agent sub-teams","authors":"Thomy Phan, Fabian Ritz, Lenz Belzner, Philipp Altmann, Thomas Gabor, Claudia Linnhoff-Popien","year":"2021","venue":"Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems (NIPS 2021)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rashid et al., 2018] Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent rein- forcement learning. In Proceedings of the 35th Inter- national Conference on Machine Learning (ICML 2018), Stockholmsmässan, Stockholm, Sweden, volume 80, pages 4292-4301, 2018.","title":"QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning","authors":"Tabish Rashid, Mikayel Samvelyan, Christian Schröder De Witt, Gregory Farquhar, Jakob N Foerster, Shimon Whiteson","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning (ICML 2018), Stockholmsmässan","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rizk et al., 2019] Yara Rizk, Mariette Awad, and Ed- ward W. Tunstel. Cooperative heterogeneous multi-robot systems: A survey. ACM Comput. Surv., 52(2):29:1- 29:31, 2019.","title":"Cooperative heterogeneous multi-robot systems: A survey","authors":"Yara Rizk, Mariette Awad, Edward W Tunstel","year":"2019","venue":"ACM Comput. Surv","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Samvelyan et al., 2019a] Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob N. Foerster, and Shimon White- son. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS 2019), Montreal, QC, Canada,, pages 2186-2188. International Foundation for Autonomous Agents and Multiagent Systems, 2019.","title":"The starcraft multi-agent challenge","authors":"Samvelyan","year":"2019","venue":"Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS 2019)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Samvelyan et al., 2019b] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon White- son. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.04043, 2019.","title":"The StarCraft Multi-Agent Challenge","authors":"Samvelyan","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shao et al., 2022] Jianzhun Shao, Zhiqiang Lou, Hongchang Zhang, Yuhang Jiang, Shuncheng He, and Xiangyang Ji. Self-organized group for cooperative multi-agent reinforcement learning. In NeurIPS, 2022.","title":"Self-organized group for cooperative multi-agent reinforcement learning","authors":"Jianzhun Shao, Zhiqiang Lou, Hongchang Zhang, Yuhang Jiang, Shuncheng He, Xiangyang Ji","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sunehag et al., 2018] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Son- nerat, Joel Z. Leibo, Karl Tuyls, and Thore Grae- pel. Value-decomposition networks for cooperative multi- agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS 2018), Stockholm, Swe- den, pages 2085-2087, 2018.","title":"Value-decomposition networks for cooperative multiagent learning based on team reward","authors":"Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech, Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, Thore Graepel","year":"2018","venue":"Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS 2018)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tacchetti et al., 2019] Andrea Tacchetti, H. Francis Song, Pedro A. M. Mediano, Vinícius Flores Zambaldi, János Kramár, Neil C. Rabinowitz, Thore Graepel, Matthew M. Botvinick, and Peter W. Battaglia. Relational forward models for multi-agent learning. In 7th International Con- ference on Learning Representations (ICLR 2019), New Orleans, LA, USA, 2019.","title":"Relational forward models for multi-agent learning","authors":"Andrea Tacchetti, H Francis Song, Pedro A M Mediano, Vinícius Flores Zambaldi, János Kramár, Neil C Rabinowitz, Thore Graepel, Matthew M Botvinick, Peter W Battaglia","year":"2019","venue":"7th International Conference on Learning Representations (ICLR 2019)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2020a] Tonghan Wang, Heng Dong, Victor R. Lesser, and Chongjie Zhang. ROMA: multi-agent rein- forcement learning with emergent roles. In Proceedings of the 37th International Conference on Machine Learning (ICML 2020), Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 9876-9886, 2020.","title":"ROMA: multi-agent reinforcement learning with emergent roles","authors":"Wang","year":"2020","venue":"Proceedings of the 37th International Conference on Machine Learning (ICML 2020), Virtual Event","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2020b] Tonghan Wang, Jianhao Wang, Chongyi Zheng, and Chongjie Zhang. Learning nearly decomposable value functions via communication mini- mization. In 8th International Conference on Learning Representations (ICLR 2020), Addis Ababa, Ethiopia, 2020.","title":"Learning nearly decomposable value functions via communication minimization","authors":"Wang","year":"2020","venue":"8th International Conference on Learning Representations (ICLR 2020)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2022a] Min Wang, Libing Wu, Jianxin Li, and Liu He. Traffic signal control with reinforcement learning based on region-aware cooperative strategy. IEEE Trans. Intell. Transp. Syst., 23(7):6774-6785, 2022.","title":"Traffic signal control with reinforcement learning based on region-aware cooperative strategy","authors":"Wang","year":"2022","venue":"IEEE Trans. Intell. Transp. Syst","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2022b] Tonghan Wang, Liang Zeng, Weijun Dong, Qianlan Yang, Yang Yu, and Chongjie Zhang. Context-aware sparse deep coordination graphs. In The Tenth International Conference on Learning Representa- tions (ICLR 2022), Virtual Event. OpenReview.net, 2022.","title":"Context-aware sparse deep coordination graphs","authors":"Wang","year":"2022","venue":"The Tenth International Conference on Learning Representations (ICLR 2022), Virtual Event. OpenReview.net","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wu et al., 2021] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A com- prehensive survey on graph neural networks. IEEE Trans. Neural Networks Learn. Syst., 32(1):4-24, 2021.","title":"A comprehensive survey on graph neural networks","authors":"Wu","year":"2021","venue":"IEEE Trans. Neural Networks Learn. Syst","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2022] Qianlan Yang, Weijun Dong, Zhizhou Ren, Jianhao Wang, Tonghan Wang, and Chongjie Zhang. Self-organized polynomial-time coordination graphs. In International Conference on Machine Learning (ICML 2022), Baltimore, Maryland, USA, volume 162 of Pro- ceedings of Machine Learning Research, pages 24963- 24979. PMLR, 2022.","title":"Self-organized polynomial-time coordination graphs","authors":"Yang","year":"2022","venue":"International Conference on Machine Learning (ICML 2022)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zang et al., 2023] Yifan Zang, Jinmin He, Kai Li, Haobo Fu, QIANG FU, Junliang Xing, and Jian Cheng. Auto- matic grouping for efficient cooperative multi-agent rein- forcement learning. In Thirty-seventh Conference on Neu- ral Information Processing Systems, (NIPS 2023), 2023.","title":"Automatic grouping for efficient cooperative multi-agent rein-forcement learning","authors":"Yifan Zang, Jinmin He, Kai Li, Haobo Fu, Qiang Fu, Junliang Xing, Jian Cheng","year":"2023","venue":"Thirty-seventh Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}