{"type":"ScholarlyArticle","title":"PTDE: Personalized Training with Distilled Execution for Multi-Agent Reinforcement Learning","authors":[{"name":"Yiqun Chen","slug":"yiqun-chen","given":"Yiqun","family":"Chen","affiliation":"Renmin University of China"},{"name":"Hangyu Mao","slug":"hangyu-mao","given":"Hangyu","family":"Mao","affiliation":""},{"name":"Jiaxin Mao","slug":"jiaxin-mao","given":"Jiaxin","family":"Mao","affiliation":"Renmin University of China"},{"name":"Shiguang Wu","slug":"shiguang-wu","given":"Shiguang","family":"Wu","affiliation":"Noah's Ark Lab , Huawei"},{"name":"Tianle Zhang","slug":"tianle-zhang","given":"Tianle","family":"Zhang","affiliation":"JD Explore Academy"},{"name":"Bin Zhang","slug":"bin-zhang","given":"Bin","family":"Zhang","affiliation":"Institute of Automation , Chinese Academy of Sciences"},{"name":"Wei Yang","slug":"wei-yang","given":"Wei","family":"Yang","affiliation":"Institute of Automation , Chinese Academy of Sciences"},{"name":"Hongxing Chang","slug":"hongxing-chang","given":"Hongxing","family":"Chang","affiliation":"Institute of Automation , Chinese Academy of Sciences"}],"abstract":"Centralized Training with Decentralized Execution (CTDE) has emerged as a widely adopted paradigm in multi-agent reinforcement learning, emphasizing the utilization of global information for learning an enhanced joint Q-function or centralized critic. In contrast, our investigation delves into harnessing global information to directly enhance individual Q-functions or individual actors. Notably, we discover that applying identical global information universally across all agents proves insufficient for optimal performance. Consequently, we advocate for the customization of global information tailored to each agent, creating agent-personalized global information to bolster overall performance. Furthermore, we introduce a novel paradigm named Personalized Training with Distilled Execution (PTDE), wherein agent-personalized global information is distilled into the agent's local information. This distilled information is then utilized during decentralized execution, resulting in minimal performance degradation. PTDE can be seamless integrated with state-of-the-art algorithms, leading to notable performance enhancements across diverse benchmarks, including the SMAC benchmark, Google Research Football (GRF) benchmark, and Learning to Rank (LTR) task.","doi":"10.24963/ijcai.2024/4","url":"https://ijcai.org/proceedings/2024/4","pdf":"https://ijcai.org/proceedings/2024/0004.pdf","markdown":"https://ijcai.org/proceedings/2024/4.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/4","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":4,"pages":{"first":31,"last":39},"keywords":["Agent-based and Multi-agent Systems: MAS: Multi-agent learning","Agent-based and Multi-agent Systems: MAS: Coordination and cooperation"],"videos":[],"references":[{"position":1,"raw":"[Chai et al., 2021] Jiajun Chai, Weifan Li, Yuanheng Zhu, Dongbin Zhao, Zhe Ma, Kewu Sun, and Jishiyu Ding. Unmas: Multiagent reinforcement learning for unshaped cooperative scenarios. IEEE Transactions on Neural Net- works and Learning Systems, pages 1-12, 2021.","title":"Unmas: Multiagent reinforcement learning for unshaped cooperative scenarios","authors":"Jiajun Chai, Weifan Li, Yuanheng Zhu, Dongbin Zhao, Zhe Ma, Kewu Sun, Jishiyu Ding","year":"2021","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2022] Yiqun Chen, Wei Yang, Tianle Zhang, Shiguang Wu, and Hongxing Chang. Commander-soldiers reinforcement learning for cooperative multi-agent sys- tems. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1-7. IEEE, 2022.","title":"Commander-soldiers reinforcement learning for cooperative multi-agent systems","authors":"Yiqun Chen, Wei Yang, Tianle Zhang, Shiguang Wu, Hongxing Chang","year":"2022","venue":"2022 International Joint Conference on Neural Networks (IJCNN)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2024] Yiqun Chen, Jiaxin Mao, Yi Zhang, De- hong Ma, Long Xia, Jun Fan, Daiting Shi, Zhicong Cheng, and Dawei Yin. Ma4div: Multi-agent reinforcement learning for search result diversification. arXiv preprint arXiv:2403.17421, 2024.","title":"Ma4div: Multi-agent reinforcement learning for search result diversification","authors":"Yiqun Chen, Jiaxin Mao, Yi Zhang, Dehong Ma, Long Xia, Jun Fan, Daiting Shi, Zhicong Cheng, Dawei Yin","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Foerster et al., 2018] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelli- gence, volume 32, 2018.","title":"Counterfactual multi-agent policy gradients","authors":"Foerster","year":"2018","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gou et al., 2021] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789-1819, 2021.","title":"Knowledge distillation: A survey","authors":"Jianping Gou, Baosheng Yu, Stephen J Maybank, Dacheng Tao","year":"2021","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Guss et al., 2021] William Hebgen Guss, Stephanie Milani, Nicholay Topin, Brandon Houghton, Sharada Mohanty, Andrew Melnik, Augustin Harter, Benoit Buschmaas, Bjarne Jaster, Christoph Berganski, Hangyu Mao, et al. Towards robust and domain agnostic reinforcement learn- ing competitions: Minerl 2020. In NeurIPS 2020 Compe- tition and Demonstration Track, pages 233-252. PMLR, 2021.","title":"Towards robust and domain agnostic reinforcement learning competitions: Minerl","authors":"Stephanie William Hebgen Guss, Nicholay Milani, Brandon Topin, Sharada Houghton, Andrew Mohanty, Augustin Melnik, Benoit Harter, Bjarne Buschmaas, Christoph Jaster, Hangyu Berganski, Mao","year":"2020","venue":"NeurIPS 2020 Competition and Demonstration Track","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Han et al., 2020] Ruihua Han, Shengduo Chen, and Qi Hao. Cooperative multi-robot navigation in dynamic environ- ment with deep reinforcement learning. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 448-454. IEEE, 2020.","title":"Cooperative multi-robot navigation in dynamic environment with deep reinforcement learning","authors":"Ruihua Han, Shengduo Chen, Qi Hao","year":"2020","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2(7), 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2021] Jian Hu, Siyang Jiang, Seth Austin Hard- ing, Haibin Wu, and Shih-wei Liao. Rethinking the im- plementation tricks and monotonicity constraint in coop- erative multi-agent reinforcement learning. arXiv e-prints, pages arXiv-2102, 2021.","title":"Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning","authors":"Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, Shih-Wei Liao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hu et al., 2024] Tianyi Hu, Zhiqiang Pu, Xiaolin Ai, Teng- hai Qiu, and Jianqiang Yi. Measuring policy distance for multi-agent reinforcement learning. arXiv preprint arXiv:2401.11257, 2024.","title":"Measuring policy distance for multi-agent reinforcement learning","authors":"Hu","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Iqbal and Sha, 2019] Shariq Iqbal and Fei Sha. Actor- attention-critic for multi-agent reinforcement learning. In International conference on machine learning, pages 2961-2970. PMLR, 2019.","title":"Actorattention-critic for multi-agent reinforcement learning","authors":"Shariq Iqbal, Fei Sha","year":"2019","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Järvelin and Kekäläinen, 2002] Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422-446, 2002.","title":"Cumulated gain-based evaluation of ir techniques","authors":"Järvelin, Kekäläinen, Kalervo Järvelin, Jaana Kekäläinen","year":"2002","venue":"ACM Transactions on Information Systems (TOIS)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kurach et al., 2020] Karol Kurach, Anton Raichuk, et al. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4501-4510, 2020.","title":"Google research football: A novel reinforcement learning environment","authors":"Karol Kurach, Anton Raichuk","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu and others, 2009] Tie-Yan Liu et al. Learning to rank for information retrieval. Foundations and Trends® in In- formation Retrieval, 3(3):225-331, 2009.","title":"Learning to rank for information retrieval","authors":"Others ; Tie-Yan Liu, Liu","year":"2009","venue":"Foundations and Trends® in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2021] Bo Liu, Qiang Liu, Peter Stone, Ani- mesh Garg, Yuke Zhu, and Anima Anandkumar. Coach- player multi-agent reinforcement learning for dynamic team composition. In International Conference on Ma- chine Learning, pages 6860-6870. PMLR, 2021.","title":"Coachplayer multi-agent reinforcement learning for dynamic team composition","authors":"Bo Liu, Qiang Liu, Peter Stone, Animesh Garg, Yuke Zhu, Anima Anandkumar","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Long et al., 2018] Pinxin Long, Tingxiang Fan, Xinyi Liao, Wenxi Liu, Hao Zhang, and Jia Pan. Towards optimally decentralized multi-robot collision avoidance via deep re- inforcement learning. In 2018 IEEE International Con- ference on Robotics and Automation (ICRA), pages 6252- 6259. IEEE, 2018.","title":"Towards optimally decentralized multi-robot collision avoidance via deep reinforcement learning","authors":"Long","year":"2018","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competitive envi- ronments. Advances in neural information processing sys- tems, 30, 2017.","title":"Multiagent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, Pieter Openai, Igor Abbeel, Mordatch","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mao et al., 2020] Hangyu Mao, Zhengchao Zhang, Zhen Xiao, Zhibo Gong, and Yan Ni. Learning agent commu- nication under limited bandwidth by message pruning. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pages 5142-5149, 2020.","title":"Learning agent communication under limited bandwidth by message pruning","authors":"Mao","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mao et al., 2021] Hangyu Mao, Chao Wang, Xiaotian Hao, Yihuan Mao, Yiming Lu, Chengjie Wu, Jianye Hao, Dong Li, and Pingzhong Tang. Seihai: A sample-efficient hierar- chical ai for the minerl competition. In International Con- ference on Distributed Artificial Intelligence, pages 38-51. Springer, 2021.","title":"Seihai: A sample-efficient hierarchical ai for the minerl competition","authors":"Hangyu Mao, Chao Wang, Xiaotian Hao, Yihuan Mao, Yiming Lu, Chengjie Wu, Jianye Hao, Dong Li, Pingzhong Tang","year":"2021","venue":"International Conference on Distributed Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Oliehoek and Amato, 2016] Frans A Oliehoek and Christo- pher Amato. A concise introduction to decentralized POMDPs. Springer, 2016.","title":"A concise introduction to decentralized POMDPs","authors":"A Frans, Christopher Oliehoek, Amato","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Qie et al., 2019] Han Qie, Dianxi Shi, Tianlong Shen, Xin- hai Xu, Yuan Li, and Liujing Wang. Joint optimiza- tion of multi-uav target assignment and path planning based on multi-agent reinforcement learning. IEEE ac- cess, 7:146264-146272, 2019.","title":"Joint optimization of multi-uav target assignment and path planning based on multi-agent reinforcement learning","authors":"Han Qie, Dianxi Shi, Tianlong Shen, Xinhai Xu, Yuan Li, Liujing Wang","year":"2019","venue":"IEEE access","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Qin and Liu, 2013] Tao Qin and Tie-Yan Liu. Introducing letor 4.0 datasets. arXiv preprint arXiv:1306.2597, 2013.","title":"Introducing letor 4.0 datasets","authors":"Liu Qin, Tao Qin, Tie-Yan Liu","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rashid et al., 2018] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International conference on machine learning, pages 4295-4304. PMLR, 2018.","title":"Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning","authors":"Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, Shimon Whiteson","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rusu et al., 2015] Andrei A Rusu, Sergio Gomez Col- menarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. arXiv preprint arXiv:1511.06295, 2015.","title":"Policy distillation","authors":"Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, Raia Hadsell","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Samvelyan et al., 2019] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon White- son. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.04043, 2019.","title":"The StarCraft Multi-Agent Challenge","authors":"Tabish Mikayel Samvelyan, Christian Rashid, Gregory Schroeder De Witt, Nantas Farquhar, Nardelli, G J Tim, Chia-Man Rudner, Hung, H S Philiph, Jakob Torr, Shimon Foerster, Whiteson","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sunehag et al., 2017] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296, 2017.","title":"Value-decomposition networks for cooperative multi-agent learning","authors":"Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech, Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2020] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.01062, 2020.","title":"Qplex: Duplex dueling multi-agent q-learning","authors":"Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yu et al., 2021] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surpris- ing effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955, 2021.","title":"The surprising effectiveness of ppo in cooperative, multi-agent games","authors":"Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, Yi Wu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2022] Bin Zhang, Yunpeng Bai, Zhiwei Xu, Dapeng Li, and Guoliang Fan. Efficient policy generation in multi-agent systems via hypergraph neural network. In International Conference on Neural Information Process- ing, pages 219-230. Springer, 2022.","title":"Efficient policy generation in multi-agent systems via hypergraph neural network","authors":"Bin Zhang, Yunpeng Bai, Zhiwei Xu, Dapeng Li, Guoliang Fan","year":"2022","venue":"International Conference on Neural Information Processing","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2023a] Bin Zhang, Lijuan Li, Zhiwei Xu, Dapeng Li, and Guoliang Fan. Inducing stackelberg equilibrium through spatio-temporal sequential decision- making in multi-agent reinforcement learning, 2023.","title":"Inducing stackelberg equilibrium through spatio-temporal sequential decisionmaking in multi-agent reinforcement learning","authors":"Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2023b] Bin Zhang, Hangyu Mao, Lijuan Li, Zhiwei Xu, Dapeng Li, Rui Zhao, and Guoliang Fan. Stackelberg decision transformer for asynchronous ac- tion coordination in multi-agent systems. arXiv preprint arXiv:2305.07856, 2023.","title":"Stackelberg decision transformer for asynchronous action coordination in multi-agent systems","authors":"Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhao et al., 2022] Jian Zhao, Xunhan Hu, Mingyu Yang, Wengang Zhou, Jiangcheng Zhu, and Houqiang Li. Ctds: Centralized teacher with decentralized student for multi-agent reinforcement learning. arXiv preprint arXiv:2203.08412, 2022.","title":"Ctds: Centralized teacher with decentralized student for multi-agent reinforcement learning","authors":"Jian Zhao, Xunhan Hu, Mingyu Yang, Wengang Zhou, Jiangcheng Zhu, Houqiang Li","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}