{"type":"ScholarlyArticle","title":"Selective Learning for Sample-Efficient Training in Multi-Agent Sparse Reward Tasks (Extended Abstract)","authors":[{"name":"Xinning Chen","slug":"xinning-chen","given":"Xinning","family":"Chen","affiliation":"College of Computer Science and Electronic Engineering , Hunan University , China"},{"name":"Xuan Liu","slug":"xuan-liu","given":"Xuan","family":"Liu","affiliation":"College of Computer Science and Electronic Engineering , Hunan University , China"},{"name":"Yanwen Ba","slug":"yanwen-ba","given":"Yanwen","family":"Ba","affiliation":"College of Computer Science and Electronic Engineering , Hunan University , China"},{"name":"Shigeng Zhang","slug":"shigeng-zhang","given":"Shigeng","family":"Zhang","affiliation":"School of Computer Science and Engineering , Central South University , China"},{"name":"Bo Ding","slug":"bo-ding","given":"Bo","family":"Ding","affiliation":"School of Computer Science , National University of Defense Technology , China"},{"name":"Kenli Li","slug":"kenli-li","given":"Kenli","family":"Li","affiliation":"College of Computer Science and Electronic Engineering , Hunan University , China"}],"abstract":"Learning effective strategies in sparse reward tasks is one of the fundamental challenges in reinforcement learning. This becomes extremely difficult in multi-agent environments, as the concurrent learning of multiple agents induces the non-stationarity problem and a sharply increased joint state space. Existing works have attempted to promote multi-agent cooperation through experience sharing. However, learning from a large collection of shared experiences is inefficient as there are only a few high-value states in sparse reward tasks, which may instead lead to the curse of dimensionality in large-scale multi-agent systems. This paper focuses on sparse-reward multi-agent cooperative tasks and proposes an effective experience-sharing method, Multi-Agent Selective Learning (MASL), to boost sample-efficient training by reusing valuable experiences from other agents. MASL adopts a retrogression-based selection method to identify high-value traces of agents from the team rewards, based on which some recall traces are generated and shared among agents to motivate effective exploration. Moreover, MASL selectively considers information from other agents to cope with the non-stationarity issue while enabling efficient training for large-scale agents. Experimental results show that MASL significantly improves sample efficiency compared with state-of-the-art MARL algorithms in cooperative tasks with sparse rewards.","doi":"10.24963/ijcai.2024/927","url":"https://ijcai.org/proceedings/2024/927","pdf":"https://ijcai.org/proceedings/2024/0927.pdf","markdown":"https://ijcai.org/proceedings/2024/927.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/927","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":927,"pages":{"first":8384,"last":8388},"keywords":["Machine Learning: ML: Multiagent Reinforcement Learning","Agent-based and Multi-agent Systems: MAS: Coordination and cooperation"],"videos":[],"references":[{"position":1,"raw":"[Bellemare et al., 2016] Marc G. Bellemare, Sriram Srini- vasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count-based exploration and in- trinsic motivation. In Proc. Adv. Neural Inf. Process. Syst., pages 1471-1479, 2016.","title":"Unifying count-based exploration and intrinsic motivation","authors":"G Marc, Sriram Bellemare, Georg Srinivasan, Tom Ostrovski, David Schaul, Rémi Saxton, Munos","year":"2016","venue":"Proc. Adv. Neural Inf. Process. Syst","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2023] Xinning Chen, Xuan Liu, Yanwen Ba, Shigeng Zhang, Bo Ding, and Kenli Li. Selective learning for sample-efficient training in multi-agent sparse reward tasks. In 26th European Conference on Artificial Intelli- gence, volume 372 of Frontiers in Artificial Intelligence and Applications, pages 413-420. IOS Press, 2023.","title":"Selective learning for sample-efficient training in multi-agent sparse reward tasks","authors":"Xinning Chen, Xuan Liu, Yanwen Ba, Shigeng Zhang, Bo Ding, Kenli Li","year":"2023","venue":"Frontiers in Artificial Intelligence and Applications","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chu et al., 2020] Tianshu Chu, Sandeep Chinchali, and Sachin Katti. Multi-agent reinforcement learning for net- worked system control. In ICLR. OpenReview.net, 2020.","title":"Multi-agent reinforcement learning for networked system control","authors":"Tianshu Chu, Sandeep Chinchali, Sachin Katti","year":"2020","venue":"ICLR. OpenReview.net","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Foerster et al., 2018] Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proc. AAAI Conf. Artif. Intell., pages 2974-2982, 2018.","title":"Counterfactual multi-agent policy gradients","authors":"Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, Shimon Whiteson","year":"2018","venue":"Proc. AAAI Conf. Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goyal et al., 2019] Anirudh Goyal, Philemon Brakel, William Fedus, Soumye Singhal, Timothy P. Lillicrap, Sergey Levine, Hugo Larochelle, and Yoshua Ben- gio. Recall traces: Backtracking models for efficient reinforcement learning. In Proc. ICLR, 2019.","title":"Recall traces: Backtracking models for efficient reinforcement learning","authors":"Anirudh Goyal, Philemon Brakel, William Fedus, Soumye Singhal, Timothy P Lillicrap, Sergey Levine, Hugo Larochelle, Yoshua Bengio","year":"2019","venue":"Proc. ICLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hao et al., 2019] Xiaotian Hao, Weixun Wang, Jianye Hao, and Yaodong Yang. Independent generative adversarial self-imitation learning in cooperative multiagent systems. In Proc. AAMAS, pages 1315-1323, 2019.","title":"Independent generative adversarial self-imitation learning in cooperative multiagent systems","authors":"Xiaotian Hao, Weixun Wang, Jianye Hao, Yaodong Yang","year":"2019","venue":"Proc. AAMAS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hosu and Rebedea, 2016] Ionel-Alexandru Hosu and Tra- ian Rebedea. Playing atari games with deep reinforce- ment learning and human checkpoint replay. CoRR, abs/1607.05077, 2016.","title":"Playing atari games with deep reinforcement learning and human checkpoint replay","authors":"Ionel-Alexandru Hosu, Traian Rebedea","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Iqbal and Sha, 2019] Shariq Iqbal and Fei Sha. Actor- attention-critic for multi-agent reinforcement learning. In Proc. Int. Conf. Mach. Learn., 2019.","title":"Actorattention-critic for multi-agent reinforcement learning","authors":"Shariq Iqbal, Fei Sha","year":"2019","venue":"Proc. Int. Conf. Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiang and Lu, 2018] Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent coop- eration. In Proceedings of Conference on Neural Informa- tion Processing Systems, pages 7265-7275, 2018.","title":"Learning attentional communication for multi-agent cooperation","authors":"Lu ; Jiang, Jiechuan Jiang, Zongqing Lu","year":"2018","venue":"Proceedings of Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kim et al., 2019] Woojun Kim, Myungsik Cho, and Youngchul Sung. Message-dropout: An efficient training method for multi-agent deep reinforcement learning. In Proc. AAAI Conf. Artif. Intell., pages 6079-6086, 2019.","title":"Message-dropout: An efficient training method for multi-agent deep reinforcement learning","authors":"Kim","year":"2019","venue":"Proc. AAAI Conf. Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kubus et al., 2018] Daniel Kubus, Rania Rayyes, and Jochen J. Steil. Learning forward and inverse kinematics maps efficiently. In International Conference on Intelli- gent Robots and Systems, pages 5133-5140. IEEE, 2018.","title":"Learning forward and inverse kinematics maps efficiently","authors":"Daniel Kubus, Rania Rayyes, Jochen J Steil","year":"2018","venue":"International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2019] Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang, and Stuart J. Russell. Robust multi- agent reinforcement learning via minimax deep determin- istic policy gradient. In Proc. AAAI Conf. Artif. Intell., pages 4213-4220, 2019.","title":"Robust multiagent reinforcement learning via minimax deep deterministic policy gradient","authors":"Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang, Stuart J Russell","year":"2019","venue":"Proc. AAAI Conf. Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2020] Haoran Li, Qichao Zhang, and Dongbin Zhao. Deep reinforcement learning-based automatic ex- ploration for navigation in unknown environment. IEEE Trans. Neural Networks Learn. Syst., 31(6):2064-2076, 2020.","title":"Deep reinforcement learning-based automatic exploration for navigation in unknown environment","authors":"Haoran Li, Qichao Zhang, Dongbin Zhao","year":"2020","venue":"IEEE Trans. Neural Networks Learn. Syst","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lillicrap et al., 2016] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yu- val Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Proc. ICLR, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra","year":"2016","venue":"Proc. ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2020] Yong Liu, Weixun Wang, Yujing Hu, Jianye Hao, Xingguo Chen, and Yang Gao. Multi-agent game abstraction via graph attention neural network. In Proc. AAAI Conf. Artif. Intell., pages 7211-7218, 2020.","title":"Multi-agent game abstraction via graph attention neural network","authors":"Yong Liu, Weixun Wang, Yujing Hu, Jianye Hao, Xingguo Chen, Yang Gao","year":"2020","venue":"Proc. AAAI Conf. Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environ- ments. In Proc. Adv. Neural Inf. Process. Syst., pages 6379-6390, 2017.","title":"Multi-agent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, Igor Mordatch","year":"2017","venue":"Proc. Adv. Neural Inf. Process. Syst","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nguyen et al., 2020] Thanh Thi Nguyen, Ngoc Duy Nguyen, and Saeid Nahavandi. Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications. IEEE Trans. Cybern., 50(9):3826-3839, 2020.","title":"Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications","authors":"Thanh Thi Nguyen, Ngoc Duy Nguyen, Saeid Nahavandi","year":"2020","venue":"IEEE Trans. Cybern","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Pathak et al., ] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proc. Int. Conf. Mach. Learn., pages 2778-2787.","title":"Curiosity-driven exploration by self-supervised prediction","authors":"Pathak","year":null,"venue":"Proc. Int. Conf. Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Perrusquía et al., 2021] Adolfo Perrusquía, Wen Yu, and Xiaoou Li. Multi-agent reinforcement learning for redun- dant robot control in task-space. Int. J. Mach. Learn. Cy- bern., 12(1):231-241, 2021.","title":"Multi-agent reinforcement learning for redundant robot control in task-space","authors":"Adolfo Perrusquía, Wen Yu, Xiaoou Li","year":"2021","venue":"Int. J. Mach. Learn. Cybern","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Stadie et al., 2017] Bradly C. Stadie, Pieter Abbeel, and Ilya Sutskever. Third person imitation learning. In Proc. ICLR, 2017.","title":"Third person imitation learning","authors":"Bradly C Stadie, Pieter Abbeel, Ilya Sutskever","year":"2017","venue":"Proc. ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tang et al., 2017] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. #exploration: A study of count-based exploration for deep reinforcement learn- ing. In Proc. Adv. Neural Inf. Process. Syst., pages 2753- 2762, 2017.","title":"#exploration: A study of count-based exploration for deep reinforcement learning","authors":"Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, Pieter Abbeel","year":"2017","venue":"Proc. Adv. Neural Inf. Process. Syst","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yang et al., 2018] Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi- agent reinforcement learning. In Proc. Int. Conf. Mach. Learn., pages 5567-5576, 2018.","title":"Mean field multiagent reinforcement learning","authors":"Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, Jun Wang","year":"2018","venue":"Proc. Int. Conf. Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ziebart et al., 2008] Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proc. AAAI Conf. Artif. Intell., pages 1433-1438, 2008.","title":"Maximum entropy inverse reinforcement learning","authors":"Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey","year":"2008","venue":"Proc. AAAI Conf. Artif. Intell","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}