{"type":"ScholarlyArticle","title":"Efficient Multi-Agent Communication via Shapley Message Value","authors":[{"name":"Di Xue","slug":"di-xue","given":"Di","family":"Xue","affiliation":"State Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"Lei Yuan","slug":"lei-yuan","given":"Lei","family":"Yuan","affiliation":"State Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China; Polixir Technologies , Nanjing 210000 , China"},{"name":"Zongzhang Zhang","slug":"zongzhang-zhang","given":"Zongzhang","family":"Zhang","affiliation":"State Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"Yang Yu","slug":"yang-yu","given":"Yang","family":"Yu","affiliation":"State Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China; Peng Cheng Laboratory , Shenzhen , Guangdong , China"}],"abstract":"Utilizing messages from teammates is crucial in cooperative multi-agent tasks due to the partially observable nature of the environment. Naively asking messages from all teammates without pruning may confuse individual agents, hindering the learning process and impairing the whole system's performance. Most previous work either utilizes a gate or employs an attention mechanism to extract relatively important messages. However, they do not explicitly evaluate each message's value, failing to learn an efficient communication protocol in more complex scenarios. To tackle this issue, we model the teammates of an agent as a message coalition and calculate the Shapley Message Value (SMV) of each agent within it. SMV reflects the contribution of each message to an agent and redundant messages can be spotted in this way effectively. On top of that, we design a novel framework named Shapley Message Selector (SMS), which learns to predict the SMVs of teammates for an agent solely based on local information so that the agent can only query those teammates with positive SMVs. Empirically, we demonstrate that our method can prune redundant messages and achieve comparable or better performance in various multi-agent cooperative scenarios than full communication settings and existing strong baselines.","doi":"10.24963/ijcai.2022/82","url":"https://ijcai.org/proceedings/2022/82","pdf":"https://ijcai.org/proceedings/2022/0082.pdf","markdown":"https://ijcai.org/proceedings/2022/82.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/82","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":82,"pages":{"first":578,"last":584},"keywords":["Machine Learning: Deep Reinforcement Learning","Agent-based and Multi-agent Systems: Agent Communication","Agent-based and Multi-agent Systems: Multi-agent Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987054","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985197","length_sec":668,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chalkiadakis et al., 2011] Georgios Chalkiadakis, Edith Elkind, and Michael Wooldridge. Computational aspects of cooperative game theory. Synthesis Lectures on Artifi- cial Intelligence and Machine Learning, 5(6):1-168, 2011.","title":"Computational aspects of cooperative game theory","authors":"Georgios Chalkiadakis, Edith Elkind, Michael Wooldridge","year":"2011","venue":"Synthesis Lectures on Artificial Intelligence and Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Das et al., 2019] Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. TarMAC: Targeted multi-agent communi- cation. In ICML, pages 1538-1546, 2019.","title":"TarMAC: Targeted multi-agent communication","authors":"Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, Joelle Pineau","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ding et al., 2020] Ziluo Ding, Tiejun Huang, and Zongqing Lu. Learning individually inferred communication for multi-agent cooperation. In NeurIPS, pages 22069-22079, 2020.","title":"Learning individually inferred communication for multi-agent cooperation","authors":"Ding","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fatima et al., 2008] Shaheen S Fatima, Michael Wooldridge, and Nicholas R Jennings. A linear ap- proximation method for the Shapley value. Artificial Intelligence, 172(14):1673-1699, 2008.","title":"A linear approximation method for the Shapley value","authors":"Michael Shaheen S Fatima, Nicholas R Wooldridge, Jennings","year":"2008","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Foerster et al., 2018] Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI, pages 2974-2982, 2018.","title":"Counterfactual multi-agent policy gradients","authors":"Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, Shimon Whiteson","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gronauer and Diepold, 2021] Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: A sur- vey. Artificial Intelligence Review, pages 1-49, 2021.","title":"Multi-agent deep reinforcement learning: A survey","authors":"Diepold Gronauer, Sven Gronauer, Klaus Diepold","year":"2021","venue":"Artificial Intelligence Review","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2021] Jian Hu, Siyang Jiang, Seth Austin Hard- ing, Haibin Wu, and Shih-wei Liao. RIIT: Rethinking the importance of implementation tricks in multi-agent re- inforcement learning. arXiv preprint arXiv:2102.03479, 2021.","title":"RIIT: Rethinking the importance of implementation tricks in multi-agent reinforcement learning","authors":"Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, Shih-Wei Liao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jang et al., 2016] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In ICLR, 2016.","title":"Categorical reparameterization with gumbel-softmax","authors":"Eric Jang, Shixiang Gu, Ben Poole","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kraemer and Banerjee, 2016] Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomput- ing, 190:82-94, 2016.","title":"Multi-agent reinforcement learning as a rehearsal for decentralized planning","authors":"Landon Kraemer, Bikramjit Banerjee","year":"2016","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environ- ments. In NIPS, pages 6379-6390, 2017.","title":"Multi-agent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, Igor Mordatch","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mao et al., 2020] Hangyu Mao, Zhengchao Zhang, Zhen Xiao, Zhibo Gong, and Yan Ni. Learning agent commu- nication under limited bandwidth by message pruning. In AAAI, pages 5142-5149, 2020.","title":"Learning agent communication under limited bandwidth by message pruning","authors":"Mao","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Niu et al., 2021] Yaru Niu, Rohan Paleja, and Matthew Gombolay. Multi-agent graph-attention communication and teaming. In AAMAS, pages 964-973, 2021.","title":"Multi-agent graph-attention communication and teaming","authors":"Yaru Niu, Rohan Paleja, Matthew Gombolay","year":"2021","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Oliehoek and Amato, 2016] Frans A Oliehoek and Christo- pher Amato. A Concise Introduction to Decentralized POMDPs. Springer, 2016.","title":"A Concise Introduction to Decentralized POMDPs","authors":"A Frans, Christopher Oliehoek, Amato","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Papoudakis et al., 2021] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In NeurIPS, 2021.","title":"Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks","authors":"Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, Stefano V Albrecht","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rashid et al., 2018] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. In ICML, pages 4295-4304, 2018.","title":"QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning","authors":"Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, Shimon Whiteson","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Samvelyan et al., 2019] Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob N. Foerster, and Shimon White- son. The Starcraft multi-agent challenge. In AAMAS, pages 2186-2188, 2019.","title":"The Starcraft multi-agent challenge","authors":"Tabish Mikayel Samvelyan, Christian Rashid, Gregory Schröder De Witt, Nantas Farquhar, Nardelli, G J Tim, Chia-Man Rudner, Hung, H S Philip, Jakob N Torr, Shimon Foerster, Whiteson","year":"2019","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Singh et al., 2019] Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. In ICLR, 2019.","title":"Learning when to communicate at scale in multiagent cooperative and competitive tasks","authors":"Amanpreet Singh, Tushar Jain, Sainbayar Sukhbaatar","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1958, 2014.","title":"Dropout: A simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2020a] Jianhong Wang, Yuan Zhang, Tae- Kyun Kim, and Yunjie Gu. Shapley Q-value: A local re- ward approach to solve global reward games. In AAAI, pages 7285-7292, 2020.","title":"Shapley Q-value: A local reward approach to solve global reward games","authors":"Wang","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2020b] Tonghan Wang, Jianhao Wang, Chongyi Zheng, and Chongjie Zhang. Learning nearly decomposable value functions via communication minimization. In ICLR, 2020.","title":"Learning nearly decomposable value functions via communication minimization","authors":"Wang","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2021] Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. DOP: Off-policy multi-agent decomposed policy gradients. In ICLR, 2021.","title":"DOP: Off-policy multi-agent decomposed policy gradients","authors":"Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, Chongjie Zhang","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}