{"type":"ScholarlyArticle","title":"Scalable Communication for Multi-Agent Reinforcement Learning via Transformer-Based Email Mechanism","authors":[{"name":"Xudong Guo","slug":"xudong-guo","given":"Xudong","family":"Guo","affiliation":"Department of Automation , Tsinghua University"},{"name":"Daming Shi","slug":"daming-shi","given":"Daming","family":"Shi","affiliation":"Department of Automation , Tsinghua University"},{"name":"Wenhui Fan","slug":"wenhui-fan","given":"Wenhui","family":"Fan","affiliation":"Department of Automation , Tsinghua University"}],"abstract":"Communication can impressively improve cooperation in multi-agent reinforcement learning (MARL), especially for partially-observed tasks. However, existing works either broadcast the messages leading to information redundancy, or learn targeted communication by modeling all the other agents as targets, which is not scalable when the number of agents varies. In this work, to tackle the scalability problem of MARL communication for partially-observed tasks, we propose a novel framework Transformer-based Email Mechanism (TEM). The agents adopt local communication to send messages only to the ones that can be observed without modeling all the agents. Inspired by human cooperation with email forwarding, we design message chains to forward information to cooperate with the agents outside the observation range. We introduce Transformer to encode and decode the message chain to choose the next receiver selectively. Empirically, TEM outperforms the baselines on multiple cooperative MARL benchmarks. When the number of agents varies, TEM maintains superior performance without further training.","doi":"10.24963/ijcai.2023/15","url":"https://ijcai.org/proceedings/2023/15","pdf":"https://ijcai.org/proceedings/2023/0015.pdf","markdown":"https://ijcai.org/proceedings/2023/15.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/15","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":15,"pages":{"first":126,"last":134},"keywords":["Machine Learning: ML: Deep reinforcement learning","Agent-based and Multi-agent Systems: MAS: Coordination and cooperation","Agent-based and Multi-agent Systems: MAS: Agent communication","Robotics: ROB: Multi-robot systems"],"videos":[],"references":[{"position":1,"raw":"[Das et al., 2019] Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. TarMAC: Targeted Multi-Agent Commu- nication. In Proceedings of the 36th International Con- ference on Machine Learning, pages 1538-1546. PMLR, May 2019.","title":"TarMAC: Targeted Multi-Agent Communication","authors":"Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, Joelle Pineau","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ding et al., 2020] Ziluo Ding, Tiejun Huang, and Zongqing Lu. Learning Individually Inferred Communication for Multi-Agent Cooperation. In Advances in Neural Informa- tion Processing Systems, volume 33, pages 22069-22079, 2020.","title":"Learning Individually Inferred Communication for Multi-Agent Cooperation","authors":"Ding","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Foerster et al., 2016] Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to Communicate with Deep Multi-Agent Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 29, 2016.","title":"Learning to Communicate with Deep Multi-Agent Reinforcement Learning","authors":"Foerster","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Jiang and Lu, 2018] Jiechuan Jiang and Zongqing Lu. Learning Attentional Communication for Multi-Agent Co- operation. November 2018. arXiv: 1805.07733.","title":"Learning Attentional Communication for Multi-Agent Cooperation","authors":"Lu Jiang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Kim et al., 2019] Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi. Learning to Schedule Communication in Multi-agent Reinforcement Learning. February 2019. arXiv: 1902.01554.","title":"Learning to Schedule Communication in Multi-agent Reinforcement Learning","authors":"Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, Yung Yi","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kober et al., 2013] Jens Kober, J. Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238- 1274, September 2013.","title":"Reinforcement learning in robotics: A survey","authors":"Jens Kober, J Andrew Bagnell, Jan Peters","year":"2013","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lillicrap et al., 2019] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous con- trol with deep reinforcement learning. July 2019. arXiv: 1509.02971.","title":"Continuous control with deep reinforcement learning","authors":"Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lowe et al., 2020] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environ- ments. March 2020. arXiv:1706.02275.","title":"Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, Igor Mordatch","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[OpenAI et al., 2019] OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d O. Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with Large Scale Deep Reinforcement Learning, Decem- ber 2019. arXiv:1912.06680.","title":"Dota 2 with Large Scale Deep Reinforcement Learning","authors":"Openai","year":"2019","venue":"December 2019","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Peng et al., 2017] Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games, September 2017. arXiv:1703.10069.","title":"Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games","authors":"Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, Jun Wang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Rashid et al., 2018] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Fo- erster, and Shimon Whiteson. QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforce- ment Learning, June 2018. arXiv:1803.11485.","title":"QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning","authors":"Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, Shimon Whiteson","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Samvelyan et al., 2019] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge, December 2019. arXiv:1902.04043.","title":"The StarCraft Multi-Agent Challenge","authors":"Tabish Mikayel Samvelyan, Christian Rashid, Gregory Schroeder De Witt, Nantas Farquhar, Nardelli, G J Tim, Chia-Man Rudner, Hung, H S Philip, Jakob Torr, Shimon Foerster, Whiteson","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust Region Policy Optimization. In Proceedings of the 32nd International Conference on Machine Learning, pages 1889-1897. PMLR, June 2015.","title":"Trust Region Policy Optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal Policy Optimization Algorithms, August 2017. arXiv:1707.06347.","title":"Proximal Policy Optimization Algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Shalev-Shwartz et al., 2016] Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, Multi-Agent, Re- inforcement Learning for Autonomous Driving, October 2016. arXiv:1610.03295.","title":"Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving","authors":"Shai Shalev-Shwartz, Shaked Shammah, Amnon Shashua","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Singh et al., 2018] Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks, December 2018. arXiv:1812.09755.","title":"Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks","authors":"Amanpreet Singh, Tushar Jain, Sainbayar Sukhbaatar","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Son et al., 2019] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. QTRAN: Learning to Factorize with Transformation for Coopera- tive Multi-Agent Reinforcement Learning. In Proceedings of the 36th International Conference on Machine Learn- ing, pages 5887-5896. PMLR, May 2019.","title":"QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning","authors":"Son","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sukhbaatar et al., 2016] Sainbayar Sukhbaatar, arthur szlam, and Rob Fergus. Learning Multiagent Commu- nication with Backpropagation. In Advances in Neural Information Processing Systems, volume 29, 2016.","title":"Sainbayar Sukhbaatar, arthur szlam, and Rob Fergus. Learning Multiagent Communication with Backpropagation","authors":"Sukhbaatar","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Sys- tems, volume 30, 2017.","title":"Attention is All you Need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vinyals et al., 2019] Oriol Vinyals, Igor Babuschkin, Woj- ciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexan- der S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wün- sch, Katrina McKinney, Oliver Smith, Tom Schaul, Timo- thy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in Star- Craft II using multi-agent reinforcement learning. Nature, 575(7782):350-354, November 2019.","title":"Grandmaster level in Star-Craft II using multi-agent reinforcement learning","authors":"Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P Agapiou, Max Jaderberg, Alexander S Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina Mckinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, David Silver","year":"2019","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wei et al., 2019] Hua Wei, Nan Xu, Huichu Zhang, Guan- jie Zheng, Xinshi Zang, Chacha Chen, Weinan Zhang, Yanmin Zhu, Kai Xu, and Zhenhui Li. CoLight: Learn- ing Network-level Cooperation for Traffic Signal Control. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 1913- 1922, November 2019.","title":"CoLight: Learning Network-level Cooperation for Traffic Signal Control","authors":"Hua Wei, Nan Xu, Huichu Zhang, Guanjie Zheng, Xinshi Zang, Chacha Chen, Weinan Zhang, Yanmin Zhu, Kai Xu, Zhenhui Li","year":"2019","venue":"Proceedings of the 28th ACM International Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wen et al., 2022] Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-Agent Reinforcement Learn- ing is a Sequence Modeling Problem, May 2022. arXiv:2205.14953.","title":"Multi-Agent Reinforcement Learning is a Sequence Modeling Problem","authors":"Wen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yu et al., 2021] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games, July 2021. arXiv:2103.01955.","title":"The Surprising Effectiveness of PPO in Cooperative","authors":"Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, Yi Wu","year":"2021","venue":"Multi-Agent Games","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yuan et al., 2022] Lei Yuan, Jianhao Wang, Fuxiang Zhang, Chenghe Wang, ZongZhang Zhang, Yang Yu, and Chongjie Zhang. Multi-Agent Incentive Communication via Decentralized Teammate Modeling. Proceedings of the AAAI Conference on Artificial Intelligence, 36(9):9466- 9474, June 2022.","title":"Multi-Agent Incentive Communication via Decentralized Teammate Modeling","authors":"Lei Yuan, Jianhao Wang, Fuxiang Zhang, Chenghe Wang, Zongzhang Zhang, Yang Yu, Chongjie Zhang","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2019] Sai Qian Zhang, Qi Zhang, and Jieyu Lin. Efficient Communication in Multi-Agent Reinforce- ment Learning via Variance Based Control. In Advances in Neural Information Processing Systems, volume 32, 2019.","title":"Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control","authors":"Qi Sai Qian Zhang, Jieyu Zhang, Lin","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2020] Sai Qian Zhang, Qi Zhang, and Jieyu Lin. Succinct and Robust Multi-Agent Communication With Temporal Message Control. In Advances in Neu- ral Information Processing Systems, volume 33, pages 17271-17282, 2020.","title":"Succinct and Robust Multi-Agent Communication With Temporal Message Control","authors":"Qi Sai Qian Zhang, Jieyu Zhang, Lin","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}