{"type":"ScholarlyArticle","title":"MA2CL:Masked Attentive Contrastive Learning for Multi-Agent Reinforcement Learning","authors":[{"name":"Haolin Song","slug":"haolin-song","given":"Haolin","family":"Song","affiliation":"EEIS Department , University of Science and Technology of China"},{"name":"Mingxiao Feng","slug":"mingxiao-feng","given":"Mingxiao","family":"Feng","affiliation":"EEIS Department , University of Science and Technology of China"},{"name":"Wengang Zhou","slug":"wengang-zhou","given":"Wengang","family":"Zhou","affiliation":"EEIS Department , University of Science and Technology of China; Institute of Artificial Intelligence , Hefei Comprehensive National Science Center , Hefei , China"},{"name":"Houqiang Li","slug":"houqiang-li","given":"Houqiang","family":"Li","affiliation":"EEIS Department , University of Science and Technology of China; Institute of Artificial Intelligence , Hefei Comprehensive National Science Center , Hefei , China"}],"abstract":"Recent approaches have utilized self-supervised auxiliary tasks as representation learning to improve the performance and sample efficiency of vision-based reinforcement learning algorithms in single-agent settings. However, in multi-agent reinforcement learning (MARL), these techniques face challenges because each agent only receives partial observation from an environment influenced by others, resulting in correlated observations in the agent dimension. So it is necessary to consider agent-level information in representation learning for MARL. In this paper, we propose an effective framework called Multi-Agent Masked Attentive Contrastive Learning (MA2CL), which encourages learning representation to be both temporal and agent-level predictive by reconstructing the masked agent observation in latent space. Specifically, we use an attention reconstruction model for recovering and the model is trained via contrastive learning. MA2CL allows better utilization of contextual information at the agent level, facilitating the training of MARL agents for cooperation tasks. Extensive experiments demonstrate that our method significantly improves the performance and sample efficiency of different MARL algorithms and outperforms other methods in various vision-based and state-based scenarios.","doi":"10.24963/ijcai.2023/470","url":"https://ijcai.org/proceedings/2023/470","pdf":"https://ijcai.org/proceedings/2023/0470.pdf","markdown":"https://ijcai.org/proceedings/2023/470.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/470","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":470,"pages":{"first":4226,"last":4234},"keywords":["Machine Learning: ML: Representation learning","Machine Learning: ML: Deep reinforcement learning","Agent-based and Multi-agent Systems: MAS: Coordination and cooperation"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2020a] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In In- ternational conference on machine learning, pages 1597- 1607. PMLR, 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Chen","year":"2020","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2020b] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020.","title":"Improved baselines with momentum contrastive learning","authors":"Chen","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[de Witt et al., 2020] Christian Schroeder de Witt, Bei Peng, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Deep multi-agent rein- forcement learning for decentralized continuous coopera- tive control. arXiv preprint arXiv:2003.06709, 2020.","title":"Christian Schroeder de Witt","authors":"De Witt","year":"2020","venue":"Deep multi-agent reinforcement learning for decentralized continuous cooperative control","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Feng et al., 2023] Mingxiao Feng, Wengang Zhou, Yaodong Yang, and Houqiang Li. Joint-predictive representations for multi-agent reinforcement learn- ing, 2023. Paper rejected from the International Conference on Learning Representations (ICLR), https://openreview.net/forum?id=S80ioOGLpD9.","title":"Joint-predictive representations for multi-agent reinforcement learning","authors":"Feng","year":"2023","venue":"Paper rejected from the International Conference on Learning Representations (ICLR)","doi":null,"url":"https://openreview.net/forum?id=S80ioOGLpD9","internal_url":null},{"position":5,"raw":"[Grill et al., 2020] Jean-Bastien Grill, Florian Strub, Flo- rent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhao- han Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learn- ing. Advances in neural information processing systems, 33:21271-21284, 2020.","title":"Bootstrap your own latent-a new approach to self-supervised learning","authors":"Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Huang et al., 2022] Jiawei Huang, Jinglin Chen, Li Zhao, Tao Qin, Nan Jiang, and Tie-Yan Liu. Towards deployment-efficient reinforcement learning: Lower bound and optimality. arXiv preprint arXiv:2202.06450, 2022.","title":"Towards deployment-efficient reinforcement learning: Lower bound and optimality","authors":"Jiawei Huang, Jinglin Chen, Li Zhao, Tao Qin, Nan Jiang, Tie-Yan Liu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Laskin et al., 2020] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in neural information processing systems, 33:19884-19895, 2020.","title":"Reinforcement learning with augmented data","authors":"Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, Aravind Srinivas","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lee et al., 2020] Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor- critic: Deep reinforcement learning with a latent variable model. Advances in Neural Information Processing Sys- tems, 2020.","title":"Stochastic latent actorcritic: Deep reinforcement learning with a latent variable model","authors":"Alex X Lee, Anusha Nagabandi, Pieter Abbeel, Sergey Levine","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ma et al., 2022] Guozheng Ma, Zhen Wang, Zhecheng Yuan, Xueqian Wang, Bo Yuan, and Dacheng Tao. A comprehensive survey of data augmentation in visual re- inforcement learning. arXiv preprint arXiv:2210.04561, 2022.","title":"A comprehensive survey of data augmentation in visual reinforcement learning","authors":"Guozheng Ma, Zhen Wang, Zhecheng Yuan, Xueqian Wang, Bo Yuan, Dacheng Tao","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Oliehoek and Amato, 2016] Frans A Oliehoek and Christo- pher Amato. A concise introduction to decentralized POMDPs. Springer, 2016. [Oord et al., 2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748, 2018.","title":"A concise introduction to decentralized POMDPs","authors":"A Frans, Christopher Oliehoek, Amato","year":"2016","venue":"Representation learning with contrastive predictive coding","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Panerati et al., 2021] Jacopo Panerati, Hehui Zheng, SiQi Zhou, James Xu, Amanda Prorok, and Angela P Schoellig. Learning to fly-a gym environment with pybullet physics for reinforcement learning of multi-agent quadcopter con- trol. In 2021 IEEE/RSJ International Conference on In- telligent Robots and Systems (IROS), pages 7512-7519. IEEE, 2021.","title":"Learning to fly-a gym environment with pybullet physics for reinforcement learning of multi-agent quadcopter control","authors":"Jacopo Panerati, Hehui Zheng, Siqi Zhou, James Xu, Amanda Prorok, Angela P Schoellig","year":"2021","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Samvelyan et al., 2019] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043, 2019.","title":"The starcraft multi-agent challenge","authors":"Tabish Mikayel Samvelyan, Christian Rashid, Schroeder De, Gregory Witt, Nantas Farquhar, Tim Gj Nardelli, Chia-Man Rudner, Hung, Jakob Philip Hs Torr, Shimon Foerster, Whiteson","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Schwarzer et al., 2020] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learn- ing with self-predictive representations. arXiv preprint arXiv:2007.05929, 2020.","title":"Data-efficient reinforcement learning with self-predictive representations","authors":"Schwarzer","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Shang et al., 2021] Wenling Shang, Lasse Espeholt, Anton Raichuk, and Tim Salimans. Agent-centric representations for multi-agent reinforcement learning. arXiv preprint arXiv:2104.09402, 2021.","title":"Agent-centric representations for multi-agent reinforcement learning","authors":"Wenling Shang, Lasse Espeholt, Anton Raichuk, Tim Salimans","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Srinivas et al., 2020] Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised repre- sentations for reinforcement learning. arXiv preprint arXiv:2004.04136, 2020.","title":"Curl: Contrastive unsupervised representations for reinforcement learning","authors":"Aravind Srinivas, Michael Laskin, Pieter Abbeel","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wen et al., 2022] Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learn- ing is a sequence modeling problem. arXiv preprint arXiv:2205.14953, 2022.","title":"Multi-agent reinforcement learning is a sequence modeling problem","authors":"Wen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yarats et al., 2020] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2020.","title":"Image augmentation is all you need: Regularizing deep reinforcement learning from pixels","authors":"Denis Yarats, Ilya Kostrikov, Rob Fergus","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yarats et al., 2021] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 10674-10681, 2021.","title":"Improving sample efficiency in model-free reinforcement learning from images","authors":"Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, Rob Fergus","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ye et al., 2021] Weirui Ye, Shaohuai Liu, Thanard Kuru- tach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in Neural Information Pro- cessing Systems, 34:25476-25488, 2021.","title":"Mastering atari games with limited data","authors":"Ye","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yu et al., 2021a] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surpris- ing effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955, 2021.","title":"The surprising effectiveness of ppo in cooperative, multi-agent games","authors":"Yu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yu et al., 2021b] Tao Yu, Cuiling Lan, Wenjun Zeng, Mingxiao Feng, Zhizheng Zhang, and Zhibo Chen. Playvirtual: Augmenting cycle-consistent virtual trajecto- ries for reinforcement learning. Advances in Neural Infor- mation Processing Systems, 34:5276-5289, 2021.","title":"Playvirtual: Augmenting cycle-consistent virtual trajectories for reinforcement learning","authors":"Yu","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yu et al., 2022] Tao Yu, Zhizheng Zhang, Cuiling Lan, Zhibo Chen, and Yan Lu. Mask-based latent recon- struction for reinforcement learning. arXiv preprint arXiv:2201.12096, 2022.","title":"Mask-based latent reconstruction for reinforcement learning","authors":"Tao Yu, Zhizheng Zhang, Cuiling Lan, Zhibo Chen, Yan Lu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhao et al., 2022] Jian Zhao, Youpeng Zhao, Weixun Wang, Mingyu Yang, Xunhan Hu, Wengang Zhou, Jianye Hao, and Houqiang Li. Coach-assisted multi-agent reinforce- ment learning framework for unexpected crashed agents. Frontiers of Information Technology & Electronic Engi- neering, 23(7):1032-1042, 2022.","title":"Coach-assisted multi-agent reinforcement learning framework for unexpected crashed agents","authors":"Jian Zhao, Youpeng Zhao, Weixun Wang, Mingyu Yang, Xunhan Hu, Wengang Zhou, Jianye Hao, Houqiang Li","year":"2022","venue":"Frontiers of Information Technology & Electronic Engineering","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhu et al., 2022] Jinhua Zhu, Yingce Xia, Lijun Wu, Jia- jun Deng, Wengang Zhou, Tao Qin, Tie-Yan Liu, and Houqiang Li. Masked contrastive representation learning for reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.","title":"Masked contrastive representation learning for reinforcement learning","authors":"Jinhua Zhu, Yingce Xia, Lijun Wu, Jiajun Deng, Wengang Zhou, Tao Qin, Tie-Yan Liu, Houqiang Li","year":"2022","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}