{"type":"ScholarlyArticle","title":"Exploration via Joint Policy Diversity for Sparse-Reward Multi-Agent Tasks","authors":[{"name":"Pei Xu","slug":"pei-xu","given":"Pei","family":"Xu","affiliation":"School of Artificial Intelligence , University of Chinese Academy of Sciences; CRISE , Institute of Automation , Chinese Academy of Sciences"},{"name":"Junge Zhang","slug":"junge-zhang","given":"Junge","family":"Zhang","affiliation":"CRISE , Institute of Automation , Chinese Academy of Sciences"},{"name":"Kaiqi Huang","slug":"kaiqi-huang","given":"Kaiqi","family":"Huang","affiliation":"School of Artificial Intelligence , University of Chinese Academy of Sciences; CRISE , Institute of Automation , Chinese Academy of Sciences; CAS , Center for Excellence in Brain Science and Intelligence Technology"}],"abstract":"Exploration under sparse rewards is a key challenge for multi-agent reinforcement learning problems. Previous works argue that complex dynamics between agents and the huge exploration space in MARL scenarios amplify the vulnerability of classical count-based exploration methods when combined with agents parameterized by neural networks, resulting in inefficient exploration. In this paper, we show that introducing constrained joint policy diversity into a classical count-based method can significantly improve exploration when agents are parameterized by neural networks. Specifically, we propose a joint policy diversity to measure the difference between current joint policy and previous joint policies, and then use a filtering-based exploration constraint to further refine the joint policy diversity. Under the sparse-reward setting, we show that the proposed method significantly outperforms the state-of-the-art methods in the multiple-particle environment, the Google Research Football, and StarCraft II micromanagement tasks. To the best of our knowledge, on the hard 3s_vs_5z task which needs non-trivial strategies to defeat enemies, our method is the first to learn winning strategies without domain knowledge under the sparse-reward setting.","doi":"10.24963/ijcai.2023/37","url":"https://ijcai.org/proceedings/2023/37","pdf":"https://ijcai.org/proceedings/2023/0037.pdf","markdown":"https://ijcai.org/proceedings/2023/37.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/37","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":37,"pages":{"first":326,"last":334},"keywords":["Agent-based and Multi-agent Systems: MAS: Multi-agent learning","Machine Learning: ML: Deep reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Azar et al., 2017] Mohammad Gheshlaghi Azar, Ian Os- band, and Rémi Munos. Minimax regret bounds for re- inforcement learning. In International Conference on Ma- chine Learning, pages 263-272. PMLR, 2017.","title":"Minimax regret bounds for reinforcement learning","authors":"Mohammad Gheshlaghi Azar, Ian Osband, Rémi Munos","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Badia et al., 2020] Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, et al. Never give up: Learning directed exploration strategies. arXiv preprint arXiv:2002.06038, 2020.","title":"Never give up: Learning directed exploration strategies","authors":"Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bazzan, 2009] Ana LC Bazzan. Opportunities for multi- agent systems and multiagent reinforcement learning in traffic control. Autonomous Agents and Multi-Agent Sys- tems, 18(3):342-375, 2009.","title":"Opportunities for multiagent systems and multiagent reinforcement learning in traffic control","authors":"Ana Lc Bazzan","year":"2009","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bellemare et al., 2016] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Process- ing Systems, pages 1471-1479, 2016.","title":"Unifying count-based exploration and intrinsic motivation","authors":"Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, Remi Munos","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Burda et al., 2018a] Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. arXiv preprint arXiv:1808.04355, 2018.","title":"Large-scale study of curiosity-driven learning","authors":"Burda","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Burda et al., 2018b] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random net- work distillation. arXiv preprint arXiv:1810.12894, 2018.","title":"Exploration by random network distillation","authors":"Burda","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chenghao et al., 2021] Li Chenghao, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. Celebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.","title":"Celebrating diversity in shared multi-agent reinforcement learning","authors":"Li Chenghao, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, Chongjie Zhang","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fu et al., 2022] Wei Fu, Chao Yu, Zelai Xu, Jiaqi Yang, and Yi Wu. Revisiting some common practices in coopera- tive multi-agent reinforcement learning. arXiv preprint arXiv:2206.07505, 2022.","title":"Revisiting some common practices in cooperative multi-agent reinforcement learning","authors":"Wei Fu, Chao Yu, Zelai Xu, Jiaqi Yang, Yi Wu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hazan et al., 2019] Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Ma- chine Learning, pages 2681-2691. PMLR, 2019.","title":"Provably efficient maximum entropy exploration","authors":"Elad Hazan, Sham Kakade, Karan Singh, Abby Van Soest","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jin et al., 2018] Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is q-learning provably effi- cient? Advances in neural information processing systems, 31, 2018.","title":"Is q-learning provably efficient?","authors":"Jin","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kurach et al., 2020] Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lasse Espe- holt, Carlos Riquelme, Damien Vincent, Marcin Michal- ski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. In Proceed- ings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages 4501-4510, 2020.","title":"Google research football: A novel reinforcement learning environment","authors":"Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2021] Iou-Jen Liu, Unnat Jain, Raymond A Yeh, and Alexander Schwing. Cooperative exploration for multi-agent deep reinforcement learning. In Interna- tional Conference on Machine Learning, pages 6826- 6836. PMLR, 2021.","title":"Cooperative exploration for multi-agent deep reinforcement learning","authors":"Iou-Jen Liu, Unnat Jain, Raymond A Yeh, Alexander Schwing","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environ- ments. In Advances in Neural Information Processing Sys- tems, 2017.","title":"Multi-agent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, Igor Mordatch","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mahajan et al., 2019] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. arXiv preprint arXiv:1910.07483, 2019.","title":"Maven: Multi-agent variational exploration","authors":"Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, Shimon Whiteson","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pathak et al., 2017] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 16-17, 2017.","title":"Curiosity-driven exploration by self-supervised prediction","authors":"Deepak Pathak, Pulkit Agrawal, Alexei A Efros, Trevor Darrell","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rashid et al., 2018] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4295-4304. PMLR, 2018.","title":"Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning","authors":"Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, Shimon Whiteson","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rashid et al., 2019] Tabish Rashid, Bei Peng, Wendelin Boehmer, and Shimon Whiteson. Optimistic exploration even with a pessimistic initialisation. In International Conference on Learning Representations, 2019.","title":"Optimistic exploration even with a pessimistic initialisation","authors":"Tabish Rashid, Bei Peng, Wendelin Boehmer, Shimon Whiteson","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ryu et al., 2022] Heechang Ryu, Hayong Shin, and Jinkyoo Park. Remax: Relational representation for multi-agent exploration. In Proceedings of the 21st International Con- ference on Autonomous Agents and Multiagent Systems, pages 1137-1145, 2022.","title":"Remax: Relational representation for multi-agent exploration","authors":"Ryu","year":"2022","venue":"Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Samvelyan et al., 2019] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043, 2019.","title":"The starcraft multi-agent challenge","authors":"Tabish Mikayel Samvelyan, Christian Rashid, Schroeder De, Gregory Witt, Nantas Farquhar, Tim Gj Nardelli, Chia-Man Rudner, Hung, Jakob Philip Hs Torr, Shimon Foerster, Whiteson","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Strehl and Littman, 2008] Alexander L Strehl and Michael L Littman. An analysis of model-based in- terval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8):1309-1331, 2008.","title":"An analysis of model-based interval estimation for markov decision processes","authors":"L Alexander, Strehl, Michael L Littman","year":"2008","venue":"Journal of Computer and System Sciences","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Swamy et al., 2020] Gokul Swamy, Siddharth Reddy, Sergey Levine, and Anca D Dragan. Scaled autonomy: Enabling human operators to control robot fleets. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 5942-5948. IEEE, 2020.","title":"Scaled autonomy: Enabling human operators to control robot fleets","authors":"Gokul Swamy, Siddharth Reddy, Sergey Levine, Anca D Dragan","year":"2020","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2019] Tonghan Wang, Jianhao Wang, Yi Wu, and Chongjie Zhang. Influence-based multi-agent explo- ration. In International Conference on Learning Represen- tation, 2019.","title":"Influence-based multi-agent explo-ration","authors":"Wang","year":"2019","venue":"International Conference on Learning Representation","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu et al., 2021] Pei Xu, Qiyue Yin, Junge Zhang, and Kaiqi Huang. Deep reinforcement learning with part-aware ex- ploration bonus in video games. IEEE Transactions on Games, 14(4):644-653, 2021.","title":"Deep reinforcement learning with part-aware exploration bonus in video games","authors":"Pei Xu, Qiyue Yin, Junge Zhang, Kaiqi Huang","year":"2021","venue":"IEEE Transactions on Games","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zha et al., 2021] Daochen Zha, Wenye Ma, Lei Yuan, Xia Hu, and Ji Liu. Rank the episodes: A simple approach for exploration in procedurally-generated environments. arXiv preprint arXiv:2101.08152, 2021.","title":"Rank the episodes: A simple approach for exploration in procedurally-generated environments","authors":"Zha","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2021] Qizhen Zhang, Chris Lu, Animesh Garg, and Jakob Foerster. Centralized model and ex- ploration policy for multi-agent rl. arXiv preprint arXiv:2107.06434, 2021.","title":"Centralized model and exploration policy for multi-agent rl","authors":"Zhang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zheng et al., 2021] Lulu Zheng, Jiarui Chen, Jianhao Wang, Jiamin He, Yujing Hu, Yingfeng Chen, Changjie Fan, Yang Gao, and Chongjie Zhang. Episodic multi-agent reinforcement learning with curiosity-driven exploration. Advances in Neural Information Processing Systems, 34, 2021.","title":"Episodic multi-agent reinforcement learning with curiosity-driven exploration","authors":"Lulu Zheng, Jiarui Chen, Jianhao Wang, Jiamin He, Yujing Hu, Yingfeng Chen, Changjie Fan, Yang Gao, Chongjie Zhang","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}