{"type":"ScholarlyArticle","title":"Multi-Agent Concentrative Coordination with Decentralized Task Representation","authors":[{"name":"Lei Yuan","slug":"lei-yuan","given":"Lei","family":"Yuan","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University; Polixir Technologies"},{"name":"Chenghe Wang","slug":"chenghe-wang","given":"Chenghe","family":"Wang","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University"},{"name":"Jianhao Wang","slug":"jianhao-wang","given":"Jianhao","family":"Wang","affiliation":"Institute for Interdisciplinary Information Sciences , Tsinghua University"},{"name":"Fuxiang Zhang","slug":"fuxiang-zhang","given":"Fuxiang","family":"Zhang","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University"},{"name":"Feng Chen","slug":"feng-chen","given":"Feng","family":"Chen","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University"},{"name":"Cong Guan","slug":"cong-guan","given":"Cong","family":"Guan","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University"},{"name":"Zongzhang Zhang","slug":"zongzhang-zhang","given":"Zongzhang","family":"Zhang","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University"},{"name":"Chongjie Zhang","slug":"chongjie-zhang","given":"Chongjie","family":"Zhang","affiliation":"Institute for Interdisciplinary Information Sciences , Tsinghua University"},{"name":"Yang Yu","slug":"yang-yu","given":"Yang","family":"Yu","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University; Peng Cheng Laboratory"}],"abstract":"Value-based multi-agent reinforcement learning (MARL) methods hold the promise of promoting coordination in cooperative settings. Popular MARL methods mainly focus on the scalability or the representational capacity of value functions. Such a learning paradigm can reduce agents' uncertainties and promote coordination. However, they fail to leverage the task structure decomposability, which generally exists in real-world multi-agent systems (MASs), leading to a significant amount of time exploring the optimal policy in complex scenarios. To address this limitation, we propose a novel framework Multi-Agent Concentrative Coordination (MACC) based on task decomposition, with which an agent can implicitly form local groups to reduce the learning space to facilitate coordination. In MACC, agents first learn representations for subtasks from their local information and then implement an attention mechanism to concentrate on the most relevant ones. Thus, agents can pay targeted attention to specific subtasks and improve coordination. Extensive experiments on various complex multi-agent benchmarks demonstrate that MACC achieves remarkable performance compared to existing methods.","doi":"10.24963/ijcai.2022/85","url":"https://ijcai.org/proceedings/2022/85","pdf":"https://ijcai.org/proceedings/2022/0085.pdf","markdown":"https://ijcai.org/proceedings/2022/85.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/85","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":85,"pages":{"first":599,"last":605},"keywords":["Agent-based and Multi-agent Systems: Coordination and Cooperation","Agent-based and Multi-agent Systems: Mechanism Design","Agent-based and Multi-agent Systems: Agreement Technologies: Argumentation","Agent-based and Multi-agent Systems: Multi-agent Learning","Agent-based and Multi-agent Systems: Agreement Technologies: Negotiation and Contract-Based Systems"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986526","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984669","length_sec":315,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Boehmer et al., 2020] Wendelin Boehmer, Vitaly Kurin, and Shimon Whiteson. Deep coordination graphs. In ICML, pages 980-991, 2020.","title":"Deep coordination graphs","authors":"Wendelin Boehmer, Shimon Vitaly Kurin, Whiteson","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, pages 1724-1734, 2014.","title":"Learning phrase representations using RNN encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Du and Ding, 2021] Wei Du and Shifei Ding. A survey on multi-agent deep reinforcement learning: from the per- spective of challenges and applications. Artificial Intel- ligence Review, 54(5):3215-3238, 2021.","title":"A survey on multi-agent deep reinforcement learning: from the perspective of challenges and applications","authors":"Wei Du, Shifei Ding","year":"2021","venue":"Artificial Intelligence Review","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Du et al., 2021] Xin Du, Jiahai Wang, Siyuan Chen, and Zhiyue Liu. Multi-agent deep reinforcement learning with spatio-temporal feature fusion for traffic signal control. In ECML, pages 470-485, 2021.","title":"Multi-agent deep reinforcement learning with spatio-temporal feature fusion for traffic signal control","authors":"Xin Du, Jiahai Wang, Siyuan Chen, Zhiyue Liu","year":"2021","venue":"ECML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Kingma and Welling, 2014] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Krüger, 2020] Tobias Krüger. Locating the Source of a WLAN Signal Using Multiple Robots and Dec-POMDP Planning. PhD thesis, Universität Hamburg, 2020.","title":"Locating the Source of a WLAN Signal Using Multiple Robots and Dec-POMDP Planning","authors":"Tobias Krüger, Krüger","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environ- ments. In NIPS, 2017.","title":"Multi-agent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, Igor Mordatch","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Oliehoek et al., 2008a] Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate Q- value functions for decentralized POMDPs. Journal of Artificial Intelligence Research, 32:289-353, 2008.","title":"Optimal and approximate Qvalue functions for decentralized POMDPs","authors":"Oliehoek","year":"2008","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Oliehoek et al., 2008b] Frans A Oliehoek, Matthijs TJ Spaan, Nikos Vlassis, and Shimon Whiteson. Exploiting locality of interaction in factored Dec-POMDPs. In AA- MAS, pages 517-524, 2008.","title":"Exploiting locality of interaction in factored Dec-POMDPs","authors":"Oliehoek","year":"2008","venue":"AA-MAS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Pajarinen and Peltonen, 2011] Joni Pajarinen and Jaakko Peltonen. Efficient planning for factored infinite-horizon Dec-POMDPs. In IJCAI, pages 325-331, 2011.","title":"Efficient planning for factored infinite-horizon Dec-POMDPs","authors":"Peltonen Pajarinen, Joni Pajarinen, Jaakko Peltonen","year":"2011","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2011/64"},{"position":11,"raw":"[Papoudakis et al., 2021] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In NeurIPS, 2021.","title":"Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks","authors":"Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, Stefano V Albrecht","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Rashid et al., 2018] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. In ICML, pages 4295-4304, 2018.","title":"QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning","authors":"Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, Shimon Whiteson","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Samvelyan et al., 2019] Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob N. Foerster, and Shimon White- son. The Starcraft multi-agent challenge. In AAMAS, pages 2186-2188, 2019.","title":"The Starcraft multi-agent challenge","authors":"Tabish Mikayel Samvelyan, Christian Rashid, Gregory Schröder De Witt, Nantas Farquhar, Nardelli, G J Tim, Chia-Man Rudner, Hung, H S Philip, Jakob N Torr, Shimon Foerster, Whiteson","year":"2019","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sarkar and Debnath, 2012] Anirban Sarkar and Narayan C. Debnath. Measuring complexity of multi-agent system ar- chitecture. In INDIN, pages 998-1003, 2012.","title":"Measuring complexity of multi-agent system architecture","authors":"Anirban Sarkar, C Narayan, Debnath","year":"2012","venue":"INDIN","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Simon, 1991] Herbert A Simon. The architecture of com- plexity. In Facets of systems science, pages 457-476. Springer, 1991.","title":"The architecture of complexity","authors":"A Herbert, Simon","year":"1991","venue":"Facets of systems science","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sunehag et al., 2018] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning based on team reward. In AAMAS, pages 2085-2087, 2018.","title":"Value-decomposition networks for cooperative multi-agent learning based on team reward","authors":"Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech, Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls","year":"2018","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Van der Maaten and Hinton, 2008] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(11):2579-2605, 2008.","title":"Visualizing data using t-SNE","authors":"Van Der Maaten, Laurens Hinton, Geoffrey Van Der Maaten, Hinton","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2020] Jianhong Wang, Yuan Zhang, Tae-Kyun Kim, and Yunjie Gu. Shapley Q-value: A local reward approach to solve global reward games. In AAAI, pages 7285-7292, 2020.","title":"Shapley Q-value: A local reward approach to solve global reward games","authors":"Jianhong Wang, Yuan Zhang, Tae-Kyun Kim, Yunjie Gu","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2021a] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. QPLEX: Duplex dueling multi-agent Q-learning. In ICLR, 2021.","title":"QPLEX: Duplex dueling multi-agent Q-learning","authors":"Wang","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2021b] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. RODE: Learning roles to decompose multi-agent tasks. ICLR, 2021.","title":"RODE: Learning roles to decompose multi-agent tasks","authors":"Wang","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang and Lesser, 2011] Chongjie Zhang and Victor Lesser. Coordinated multi-agent reinforcement learning in networked distributed POMDPs. In AAAI, 2011.","title":"Chongjie Zhang and Victor Lesser. Coordinated multi-agent reinforcement learning in networked distributed POMDPs","authors":"Lesser Zhang","year":"2011","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2020] Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuandong Tian. Multi-agent collaboration via reward at- tribution decomposition. arXiv:2010.08531, 2020.","title":"Multi-agent collaboration via reward attribution decomposition","authors":"Zhang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang, 2011] Chongjie Zhang. Scaling multi-agent learn- ing in complex environments. PhD thesis, 2011.","title":"Scaling multi-agent learning in complex environments","authors":"Chongjie Zhang, Zhang","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou et al., 2020] Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng Chen, et al. Smarts: Scalable multi-agent reinforcement learning train- ing school for autonomous driving. arXiv:2010.09776, 2020.","title":"Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving","authors":"Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng Chen","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}