{"type":"ScholarlyArticle","title":"Off-Agent Trust Region Policy Optimization","authors":[{"name":"Ruiqing Chen","slug":"ruiqing-chen","given":"Ruiqing","family":"Chen","affiliation":"Institute for AI , Peking University , Beijing , China; ShanghaiTech University , Shanghai , China"},{"name":"Xiaoyuan Zhang","slug":"xiaoyuan-zhang","given":"Xiaoyuan","family":"Zhang","affiliation":"Institute for AI , Peking University , Beijing , China"},{"name":"Yali Du","slug":"yali-du","given":"Yali","family":"Du","affiliation":"King's College London , UK"},{"name":"Yifan Zhong","slug":"yifan-zhong","given":"Yifan","family":"Zhong","affiliation":"Institute for AI , Peking University , Beijing , China"},{"name":"Zheng Tian","slug":"zheng-tian","given":"Zheng","family":"Tian","affiliation":"ShanghaiTech University , Shanghai , China"},{"name":"Fanglei Sun","slug":"fanglei-sun","given":"Fanglei","family":"Sun","affiliation":"ShanghaiTech University , Shanghai , China"},{"name":"Yaodong Yang","slug":"yaodong-yang","given":"Yaodong","family":"Yang","affiliation":"Institute for AI , Peking University , Beijing , China"}],"abstract":"Leveraging the experiences of other agents offers a powerful mechanism to enhance policy optimization in multi-agent reinforcement learning (MARL). However, contemporary MARL algorithms often neglect experience sharing possibilities or adopt a simple approach via direct parameter sharing. Our work explores a refined off-agent learning framework that allows selective integration of experience from other agents to improve policy learning. Our investigation begins with a thorough assessment of current mechanisms for reusing experiences among heterogeneous agents, revealing that direct experience transfer may result in negative consequences. Moreover, even the experience of homogeneous agents requires modification before reusing. Our approach introduces off-agent adaptations to the multi-agent policy optimization methods, enabling effective and purposeful leverage of cross-agent experiences beyond conventional parameter sharing. Accompanying this, we provide a theoretical guarantee for an approximate monotonic improvement. Experiments conducted on the StarCraftII Multi-Agent Challenge (SMAC) and Google Research Football (GRF) demonstrate that our algorithms outperform state-of-the-art (SOTA) methods and achieve faster convergence, suggesting the viability of our approach for efficient experience reusing in MARL.","doi":"10.24963/ijcai.2024/420","url":"https://ijcai.org/proceedings/2024/420","pdf":"https://ijcai.org/proceedings/2024/0420.pdf","markdown":"https://ijcai.org/proceedings/2024/420.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/420","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":420,"pages":{"first":3798,"last":3806},"keywords":["Machine Learning: ML: Multiagent Reinforcement Learning","Machine Learning: ML: Partially observable reinforcement learning and POMDPs","Machine Learning: ML: Reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Adler and Blue, 2002] Jeffrey L Adler and Victor J Blue. A cooperative multi-agent transportation management and route guidance system. Transportation Research Part C: Emerging Technologies, 10(5-6):433-454, 2002.","title":"A cooperative multi-agent transportation management and route guidance system","authors":"L Jeffrey, Victor J Adler, Blue","year":"2002","venue":"Transportation Research Part C: Emerging Technologies","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baker et al., 2020] Bowen Baker, Ingmar Kanitscheider, Todor M. Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In ICLR, 2020.","title":"Emergent tool use from multi-agent autocurricula","authors":"Bowen Baker, Ingmar Kanitscheider, M Todor, Yi Markov, Glenn Wu, Bob Powell, Igor Mcgrew, Mordatch","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brown and Sandholm, 2019] Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, 365(6456):885-890, 2019.","title":"Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker","authors":"Sandholm Brown","year":"2019","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Busoniu et al., 2008] Lucian Busoniu, Robert Babuska, and Bart De Schutter. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, 38(2):156-172, 2008.","title":"A comprehensive survey of multiagent reinforcement learning","authors":"Lucian Busoniu, Robert Babuska, Bart De Schutter","year":"2008","venue":"IEEE Transactions on Systems, Man, and Cybernetics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Christianos et al., 2020] Filippos Christianos, Lukas Schäfer, and Stefano Albrecht. Shared experience actor- critic for multi-agent reinforcement learning. Advances in neural information processing systems, 33:10707-10717, 2020.","title":"Shared experience actorcritic for multi-agent reinforcement learning","authors":"Filippos Christianos, Lukas Schäfer, Stefano Albrecht","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Christianos et al., 2021] Filippos Christianos, Georgios Pa- poudakis, Muhammad A Rahman, and Stefano V Al- brecht. Scaling multi-agent reinforcement learning with selective parameter sharing. In International Conference on Machine Learning, pages 1989-1998. PMLR, 2021.","title":"Scaling multi-agent reinforcement learning with selective parameter sharing","authors":"Filippos Christianos, Georgios Papoudakis, Muhammad A Rahman, Stefano V Albrecht","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[de Witt et al., 2020] Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533, 2020.","title":"Christian Schroeder de Witt","authors":"De Witt","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fakoor et al., 2020] Rasool Fakoor, Pratik Chaudhari, and Alexander J Smola. P3o: Policy-on policy-off policy op- timization. In Uncertainty in Artificial Intelligence, pages 1017-1027. PMLR, 2020.","title":"P3o: Policy-on policy-off policy optimization","authors":"Rasool Fakoor, Pratik Chaudhari, Alexander J Smola","year":"2020","venue":"Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Foerster et al., 2018] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelli- gence, volume 32, 2018.","title":"Counterfactual multi-agent policy gradients","authors":"Foerster","year":"2018","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Glavic et al., 2017] Mevludin Glavic, Raphaël Fonteneau, and Damien Ernst. Reinforcement learning for electric power system decision and control: Past considerations and perspectives. IFAC-PapersOnLine, 50(1):6918-6927, 2017.","title":"Reinforcement learning for electric power system decision and control: Past considerations and perspectives","authors":"Glavic","year":"2017","venue":"IFAC-PapersOnLine","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on ma- chine learning, pages 1861-1870. PMLR, 2018.","title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","authors":"Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hussein et al., 2017] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learn- ing: A survey of learning methods. ACM Computing Sur- veys (CSUR), 50(2):1-35, 2017.","title":"Imitation learning: A survey of learning methods","authors":"Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, Chrisina Jayne","year":"2017","venue":"ACM Computing Surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jarrahi, 2018] Mohammad Hossein Jarrahi. Artificial in- telligence and the future of work: Human-ai symbiosis in organizational decision making. Business horizons, 61(4):577-586, 2018.","title":"Artificial intelligence and the future of work: Human-ai symbiosis in organizational decision making","authors":"Mohammad Hossein, Jarrahi","year":"2018","venue":"Business horizons","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kuba et al., 2021] Jakub Grudzien Kuba, Ruiqing Chen, Munning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. arXiv preprint arXiv:2109.11251, 2021.","title":"Trust region policy optimisation in multi-agent reinforcement learning","authors":"Ruiqing Jakub Grudzien Kuba, Munning Chen, Ying Wen, Fanglei Wen, Jun Sun, Yaodong Wang, Yang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kurach et al., 2020] Karol Kurach, Anton Raichuk, Piotr Stańczyk, Michał Zaj ąc, Olivier Bachem, Lasse Espe- holt, Carlos Riquelme, Damien Vincent, Marcin Michal- ski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. In Proceed- ings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages 4501-4510, 2020.","title":"Google research football: A novel reinforcement learning environment","authors":"Karol Kurach, Anton Raichuk, Piotr Stańczyk, Michał Zaj Ąc, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous con- trol with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competitive envi- ronments. Advances in neural information processing sys- tems, 30, 2017.","title":"Multiagent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, Pieter Openai, Igor Abbeel, Mordatch","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Oh et al., 2018] Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In Inter- national Conference on Machine Learning, pages 3878- 3887. PMLR, 2018.","title":"Self-imitation learning","authors":"Oh","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Oliehoek and Amato, 2016] Frans A Oliehoek and Christo- pher Amato. A concise introduction to decentralized POMDPs. Springer, 2016.","title":"A concise introduction to decentralized POMDPs","authors":"A Frans, Christopher Oliehoek, Amato","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Osa et al., 2018] Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. An algorithmic perspective on imitation learn- ing. Foundations and Trends® in Robotics, 7(1-2):1-179, 2018.","title":"An algorithmic perspective on imitation learning","authors":"Takayuki Osa, Joni Pajarinen, Gerhard Neumann, Andrew Bagnell, Pieter Abbeel, Jan Peters","year":"2018","venue":"Foundations and Trends® in Robotics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Peng et al., 2017] Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent bidirectionally-coordinated nets: Emergence of human-level coordination in learning to play starcraft combat games. arXiv preprint arXiv:1703.10069, 2017.","title":"Multiagent bidirectionally-coordinated nets: Emergence of human-level coordination in learning to play starcraft combat games","authors":"Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, Jun Wang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Qiu et al., 2023] Dawei Qiu, Yi Wang, Jianhong Wang, Ning Zhang, Goran Strbac, and Chongqing Kang. Resilience-oriented coordination of networked micro- grids: a shapley q-value learning approach. IEEE Trans- actions on Power Systems, pages 1-15, 2023.","title":"Resilience-oriented coordination of networked microgrids: a shapley q-value learning approach","authors":"Dawei Qiu, Yi Wang, Jianhong Wang, Ning Zhang, Goran Strbac, Chongqing Kang","year":"2023","venue":"IEEE Transactions on Power Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Queeney et al., 2021] James Queeney, Yannis Paschalidis, and Christos G Cassandras. Generalized proximal policy optimization with sample reuse. Advances in Neural In- formation Processing Systems, 34:11909-11919, 2021.","title":"Generalized proximal policy optimization with sample reuse","authors":"James Queeney, Yannis Paschalidis, Christos G Cassandras","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Samvelyan et al., 2019] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043, 2019.","title":"The starcraft multi-agent challenge","authors":"Tabish Mikayel Samvelyan, Christian Rashid, Schroeder De, Gregory Witt, Nantas Farquhar, Tim Gj Nardelli, Chia-Man Rudner, Hung, Jakob Philip Hs Torr, Shimon Foerster, Whiteson","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sun et al., 2023] Mingfei Sun, Sam Devlin, Jacob Beck, Katja Hofmann, and Shimon Whiteson. Trust region bounds for decentralized ppo under non-stationarity. In Proceedings of the 2023 International Conference on Au- tonomous Agents and Multiagent Systems, pages 5-13, 2023.","title":"Trust region bounds for decentralized ppo under non-stationarity","authors":"Sun","year":"2023","venue":"Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vinyals et al., 2019] Oriol Vinyals, Igor Babuschkin, Woj- ciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in star- craft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.","title":"Grandmaster level in starcraft ii using multi-agent reinforcement learning","authors":"Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev","year":"2019","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2016] Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with expe- rience replay. arXiv preprint arXiv:1611.01224, 2016.","title":"Sample efficient actor-critic with experience replay","authors":"Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, Nando De Freitas","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2023] Xihuai Wang, Zheng Tian, Ziyu Wan, Ying Wen, Jun Wang, and Weinan Zhang. Order mat- ters: Agent-by-agent policy optimization. arXiv preprint arXiv:2302.06205, 2023.","title":"Order matters: Agent-by-agent policy optimization","authors":"Xihuai Wang, Zheng Tian, Ziyu Wan, Ying Wen, Jun Wang, Weinan Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wu et al., 2021] Zifan Wu, Chao Yu, Deheng Ye, Junge Zhang, Hankz Hankui Zhuo, et al. Coordinated proximal policy optimization. Advances in Neural Information Pro- cessing Systems, 34:26437-26448, 2021.","title":"Coordinated proximal policy optimization","authors":"Zifan Wu, Chao Yu, Deheng Ye, Junge Zhang, Hankz Hankui Zhuo","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang and Wang, 2020] Yaodong Yang and Jun Wang. An overview of multi-agent reinforcement learning from game theoretical perspective. arXiv preprint arXiv:2011.00583, 2020.","title":"An overview of multi-agent reinforcement learning from game theoretical perspective","authors":"Yang, Wang ;, Yaodong Yang, Jun Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yu et al., 2021] C Yu, A Velu, E Yinitsky, et al. The sur- prising effectiveness of mappo in cooperative, multi-agent games [j/ol]. 2021.","title":"The surprising effectiveness of mappo in cooperative","authors":"Yu, Velu, Yinitsky","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zare et al., 2023] Maryam Zare, Parham M Kebria, Ab- bas Khosravi, and Saeid Nahavandi. A survey of imita- tion learning: Algorithms, recent developments, and chal- lenges. arXiv preprint arXiv:2309.02473, 2023.","title":"A survey of imitation learning: Algorithms, recent developments, and challenges","authors":"Maryam Zare, Abbas Parham M Kebria, Saeid Khosravi, Nahavandi","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang and Lesser, 2011] Chongjie Zhang and Victor Lesser. Coordinated multi-agent reinforcement learning in networked distributed pomdps. In Twenty-Fifth AAAI Conference on Artificial Intelligence, 2011.","title":"Chongjie Zhang and Victor Lesser. Coordinated multi-agent reinforcement learning in networked distributed pomdps","authors":"Lesser Zhang","year":"2011","venue":"Twenty-Fifth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhou et al., 2019] Ming Zhou, Yong Chen, Ying Wen, Yaodong Yang, Yufeng Su, Weinan Zhang, Dell Zhang, and Jun Wang. Factorized q-learning for large-scale multi- agent systems. In Proceedings of the First International Conference on Distributed Artificial Intelligence, pages 1- 7, 2019.","title":"Factorized q-learning for large-scale multiagent systems","authors":"Ming Zhou, Yong Chen, Ying Wen, Yaodong Yang, Yufeng Su, Weinan Zhang, Dell Zhang, Jun Wang","year":"2019","venue":"Proceedings of the First International Conference on Distributed Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhou et al., 2021] Ming Zhou, Ziyu Wan, Hanjing Wang, Muning Wen, Runzhe Wu, Ying Wen, Yaodong Yang, Weinan Zhang, and Jun Wang. Malib: A parallel framework for population-based multi-agent reinforce- ment learning. arXiv preprint arXiv:2106.07551, 2021.","title":"Malib: A parallel framework for population-based multi-agent reinforcement learning","authors":"Ming Zhou, Ziyu Wan, Hanjing Wang, Muning Wen, Runzhe Wu, Ying Wen, Yaodong Yang, Weinan Zhang, Jun Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}