{"type":"ScholarlyArticle","title":"Improving Multi-agent Reinforcement Learning with Stable Prefix Policy","authors":[{"name":"Yue Deng","slug":"yue-deng","given":"Yue","family":"Deng","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Zirui Wang","slug":"zirui-wang","given":"Zirui","family":"Wang","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Yin Zhang","slug":"yin-zhang","given":"Yin","family":"Zhang","affiliation":"College of Computer Science and Technology , Zhejiang University"}],"abstract":"In multi-agent reinforcement learning (MARL), the epsilon-greedy method plays an important role in balancing exploration and exploitation during the decision-making process in value-based algorithms. However, the epsilon-greedy exploration process will introduce conservativeness when calculating the expected state value when the agents are more in need of exploitation during the approximate policy convergence, which may result in a suboptimal policy convergence. Besides, eliminating the epsilon-greedy algorithm leaves no exploration and may lead to unacceptable local optimal policies. To address this dilemma, we use the previously collected trajectories to construct a Monte-Carlo Trajectory Tree, so that an existing optimal template, a sequence of state prototypes, can be planned out. The agents start by following the planned template and act according to the policy without exploration, Stable Prefix Policy. The agents will adaptively dropout and begin to explore by following the epsilon-greedy method when the policy still needs exploration. We scale our approach to various value-based MARL methods and empirically verify our method in a cooperative MARL task, SMAC benchmarks. Experimental results demonstrate that our method achieves not only better performance but also faster convergence speed than baseline algorithms within early time steps.","doi":"10.24963/ijcai.2024/6","url":"https://ijcai.org/proceedings/2024/6","pdf":"https://ijcai.org/proceedings/2024/0006.pdf","markdown":"https://ijcai.org/proceedings/2024/6.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/6","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":6,"pages":{"first":49,"last":57},"keywords":["Agent-based and Multi-agent Systems: MAS: Multi-agent learning","Agent-based and Multi-agent Systems: MAS: Coordination and cooperation"],"videos":[],"references":[{"position":1,"raw":"[Agarwal et al., 2021] Alekh Agarwal, Sham M Kakade, Ja- son D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distri- bution shift. The Journal of Machine Learning Research, 22(1):4431-4506, 2021.","title":"On the theory of policy gradient methods: Optimality, approximation, and distribution shift","authors":"Alekh Agarwal, M Sham, Jason D Kakade, Gaurav Lee, Mahajan","year":"2021","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bu et al., 2020] Yuxuan Bu, Xuechen Chen, T. D. Kulka- rni, Amir Saeedi, and Joshua B. Tenenbaum. Individual- global-maximum: A new framework for multi-agent re- inforcement learning. arXiv preprint arXiv:2006.07689, 2020.","title":"Individualglobal-maximum: A new framework for multi-agent reinforcement learning","authors":"Yuxuan Bu, Xuechen Chen, T D Kulkarni, Amir Saeedi, Joshua B Tenenbaum","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cao et al., 2012] Yongcan Cao, Wenwu Yu, Wei Ren, and Guanrong Chen. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Trans- actions on Industrial informatics, 9(1):427-438, 2012.","title":"An overview of recent progress in the study of distributed multi-agent coordination","authors":"Cao","year":"2012","venue":"IEEE Transactions on Industrial informatics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Foerster et al., 2017] Jakob Foerster, Guillaume Assael, Michael White, Timothy P. Lillicrap, and John D. Schul- man. Learning to communicate with deep multi-agent re- inforcement learning. arXiv preprint arXiv:1706.03762, 2017.","title":"Learning to communicate with deep multi-agent reinforcement learning","authors":"Jakob Foerster, Guillaume Assael, Michael White, Timothy P Lillicrap, John D Schulman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Foerster et al., 2018] Jakob Foerster, Guillaume Assael, Michael White, Timothy P. Lillicrap, and John D. Schul- man. Counterfactual multi-agent policy gradients. arXiv preprint arXiv:1802.01561, 2018.","title":"Counterfactual multi-agent policy gradients","authors":"Jakob Foerster, Guillaume Assael, Michael White, Timothy P Lillicrap, John D Schulman","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hu et al., 2021] Jian Hu, Siyang Jiang, Seth Austin Hard- ing, Haibin Wu, and Shih wei Liao. Rethinking the imple- mentation tricks and monotonicity constraint in coopera- tive multi-agent reinforcement learning. 2021.","title":"Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning","authors":"Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, Shih Wei Liao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jin et al., 2018] Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is q-learning provably effi- cient? Advances in neural information processing systems, 31, 2018.","title":"Is q-learning provably efficient?","authors":"Jin","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Koenig and Simmons, 1993] Sven Koenig and Reid G Sim- mons. Complexity analysis of real-time reinforcement learning. In AAAI, volume 93, pages 99-105, 1993.","title":"Complexity analysis of real-time reinforcement learning","authors":"Simmons Koenig, Sven Koenig, Reid G Simmons","year":"1993","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competitive envi- ronments. Advances in neural information processing sys- tems, 30, 2017.","title":"Multiagent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, Pieter Openai, Igor Abbeel, Mordatch","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mahajan et al., 2019] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. Advances in Neural Information Processing Systems, 32, 2019.","title":"Maven: Multi-agent variational exploration","authors":"Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, Shimon Whiteson","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Oliehoek et al., 2016] Frans A Oliehoek, Christopher Am- ato, et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.","title":"A concise introduction to decentralized POMDPs","authors":"Christopher Frans A Oliehoek, Amato","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ouyang et al., 2017] Yi Ouyang, Mukul Gagrani, Ashutosh Nayyar, and Rahul Jain. Learning unknown markov deci- sion processes: A thompson sampling approach. Advances in neural information processing systems, 30, 2017.","title":"Learning unknown markov decision processes: A thompson sampling approach","authors":"Yi Ouyang, Mukul Gagrani, Ashutosh Nayyar, Rahul Jain","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Rashid et al., 2018] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder Witt, Gregory Farquhar, Jakob Fo- erster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learn- ing, pages 4292-4301, 2018.","title":"Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning","authors":"Tabish Rashid, Mikayel Samvelyan, Christian Schroeder Witt, Gregory Farquhar, Jakob Foerster, Shimon Whiteson","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rashid et al., 2020] Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expand- ing monotonic value function factorisation for deep multi- agent reinforcement learning. Advances in neural infor- mation processing systems, 33:10199-10210, 2020.","title":"Weighted qmix: Expanding monotonic value function factorisation for deep multiagent reinforcement learning","authors":"Tabish Rashid, Gregory Farquhar, Bei Peng, Shimon Whiteson","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Samvelyan et al., 2019] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon White- son. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.04043, 2019.","title":"The StarCraft Multi-Agent Challenge","authors":"Tabish Mikayel Samvelyan, Christian Rashid, Gregory Schroeder De Witt, Nantas Farquhar, Nardelli, G J Tim, Chia-Man Rudner, Hung, H S Philiph, Jakob Torr, Shimon Foerster, Whiteson","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Son et al., 2019] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learn- ing to factorize with transformation for cooperative multi- agent reinforcement learning. In International Conference on Machine Learning, pages 5887-5896. PMLR, 2019.","title":"Qtran: Learning to factorize with transformation for cooperative multiagent reinforcement learning","authors":"Son","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sunehag et al., 2017] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296, 2017.","title":"Value-decomposition networks for cooperative multi-agent learning","authors":"Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech, Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sutton and Barto, 1998] R. S. Sutton and A. G. Barto. Re- inforcement learning: An introduction. MIT press, 1998.","title":"Reinforcement learning: An introduction","authors":"R S Sutton, A G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Uchendu et al., 2023] Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et al. Jump-start reinforcement learning. In International Conference on Machine Learning, pages 34556-34583.","title":"Jump-start reinforcement learning","authors":"Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao","year":null,"venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"PMLR, 2023.","title":null,"authors":"Pmlr","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2019a] Tonghan Wang, Jianhao Wang, Yi Wu, and Chongjie Zhang. Influence-based multi-agent explo- ration. arXiv preprint arXiv:1910.05512, 2019.","title":"Influence-based multi-agent exploration","authors":"Wang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2019b] Tonghan Wang, Jianhao Wang, Chongyi Zheng, and Chongjie Zhang. Learning nearly decomposable value functions via communication minimization. arXiv preprint arXiv:1910.05366, 2019.","title":"Learning nearly decomposable value functions via communication minimization","authors":"Wang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2020a] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.01062, 2020.","title":"Qplex: Duplex dueling multi-agent q-learning","authors":"Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2020b] Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi-agent rein- forcement learning with emergent roles. arXiv preprint arXiv:2003.08039, 2020.","title":"Roma: Multi-agent reinforcement learning with emergent roles","authors":"Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2023] Jiangxing Wang, Deheng Ye, and Zongqing Lu. More centralized training, still decentral- ized execution: Multi-agent conditional policy factoriza- tion. In International Conference on Learning Represen- tations (ICLR), 2023.","title":"More centralized training, still decentralized execution: Multi-agent conditional policy factorization","authors":"Wang","year":"2023","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xie et al., 2022] Tengyang Xie, Dylan J Foster, Yu Bai, Nan Jiang, and Sham M Kakade. The role of coverage in online reinforcement learning. arXiv preprint arXiv:2210.04157, 2022.","title":"The role of coverage in online reinforcement learning","authors":"Xie","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yang et al., 2020] Yaodong Yang, Ying Wen, Jun Wang, Li- heng Chen, Kun Shao, David Mguni, and Weinan Zhang. Multi-agent determinantal q-learning. In International Conference on Machine Learning, pages 10757-10766.","title":"Multi-agent determinantal q-learning","authors":"Yaodong Yang, Ying Wen, Jun Wang, Liheng Chen, Kun Shao, David Mguni, Weinan Zhang","year":null,"venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"PMLR, 2020.","title":null,"authors":"Pmlr","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Ye et al., 2021] Weirui Ye, Shaohuai Liu, Thanard Kuru- tach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in Neural Information Pro- cessing Systems, 34:25476-25488, 2021.","title":"Mastering atari games with limited data","authors":"Ye","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yu et al., 2022] Chao Yu, Akash Velu, Eugene Vinitsky, Ji- axuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Sys- tems, 35:24611-24624, 2022.","title":"The surprising effectiveness of ppo in cooperative multi-agent games","authors":"Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, Yi Wu","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang and Lesser, 2011] Chongjie Zhang and Victor Lesser. Coordinated multi-agent reinforcement learning in networked distributed pomdps. In Twenty-Fifth AAAI Conference on Artificial Intelligence, 2011.","title":"Chongjie Zhang and Victor Lesser. Coordinated multi-agent reinforcement learning in networked distributed pomdps","authors":"Lesser Zhang","year":"2011","venue":"Twenty-Fifth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2021] Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu. Fop: Factoriz- ing optimal joint policy of maximum-entropy multi-agent reinforcement learning. In International Conference on Machine Learning, pages 12491-12500. PMLR, 2021.","title":"Fop: Factorizing optimal joint policy of maximum-entropy multi-agent reinforcement learning","authors":"Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, Zongqing Lu","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}