{"type":"ScholarlyArticle","title":"Multi-Agent Advisor Q-Learning (Extended Abstract)","authors":[{"name":"Sriram Ganapathi Subramanian","slug":"sriram-ganapathi-subramanian","given":"Sriram","family":"Ganapathi Subramanian","affiliation":"Vector Institute; University of Waterloo"},{"name":"Matthew E. Taylor","slug":"matthew-e-taylor","given":"Matthew E.","family":"Taylor","affiliation":"University of Alberta; Alberta Machine Intelligence Institute (Amii)"},{"name":"Kate Larson","slug":"kate-larson","given":"Kate","family":"Larson","affiliation":"University of Waterloo"},{"name":"Mark Crowley","slug":"mark-crowley","given":"Mark","family":"Crowley","affiliation":"University of Waterloo"}],"abstract":"In the last decade, there have been significant advances in multi-agent reinforcement learning (MARL) but there are still numerous challenges, such as high sample complexity and slow convergence to stable policies, that need to be overcome before wide-spread deployment is possible. However, many real-world environments already, in practice,  deploy sub-optimal or heuristic approaches for generating policies. An interesting question that arises is how to best use such approaches as advisors to help improve reinforcement learning in multi-agent domains. We provide a principled framework for incorporating action recommendations from online sub-optimal advisors in multi-agent settings. We describe the problem of ADvising Multiple Intelligent Reinforcement Agents (ADMIRAL) in nonrestrictive general-sum stochastic game environments and present two novel Q-learning-based algorithms: ADMIRAL - Decision Making (ADMIRAL-DM) and ADMIRAL - Advisor Evaluation (ADMIRAL-AE), which allow us to improve learning by appropriately incorporating advice from an advisor (ADMIRAL-DM), and evaluate the effectiveness of an advisor (ADMIRAL-AE). We analyze the algorithms theoretically and provide fixed point guarantees regarding their learning in general-sum stochastic games. Furthermore, extensive experiments illustrate that these algorithms: can be used in a variety of environments, have performances that compare favourably to other related baselines, can scale to large state-action spaces, and are robust to poor advice from advisors.","doi":"10.24963/ijcai.2023/776","url":"https://ijcai.org/proceedings/2023/776","pdf":"https://ijcai.org/proceedings/2023/0776.pdf","markdown":"https://ijcai.org/proceedings/2023/776.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/776","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Journal Track","track":"Journal Track","number":776,"pages":{"first":6884,"last":6889},"keywords":["Machine Learning: ML: Reinforcement learning","Agent-based and Multi-agent Systems: MAS: Multi-agent learning","Machine Learning: ML: Deep reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"Bowling, 2000","title":null,"authors":"Bowling","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"Michael H. Bowling. Convergence Prob- lems of General-Sum Multiagent Reinforcement Learning. In Proceedings of the Seventeenth International Confer- ence on Machine Learning (ICML 2000), Stanford Uni- versity, Stanford, CA, USA, June 29 -July 2, 2000, pages 89-94.","title":"Convergence Problems of General-Sum Multiagent Reinforcement Learning","authors":"H Michael, Bowling","year":"2000","venue":"Proceedings of the Seventeenth International Conference on Machine Learning (ICML 2000)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"Morgan Kaufmann, 2000.","title":null,"authors":"Morgan Kaufmann","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dulac-Arnold et al., 2021] Gabriel Dulac-Arnold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: Definitions, Benchmarks and Analysis. Machine Learning, 1:1-50, 2021.","title":"Challenges of real-world reinforcement learning: Definitions, Benchmarks and Analysis","authors":"Gabriel Dulac-Arnold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, Todd Hester","year":"2021","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gao et al., 2018] Yang Gao, Huazhe Xu, Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement Learn- ing from Imperfect Demonstrations. In 6th International Conference on Learning Representations, (ICLR 2018), Vancouver, BC, Canada, April 30 -May 3, 2018, Work- shop Track Proceedings. OpenReview.net, 2018.","title":"Reinforcement Learning from Imperfect Demonstrations","authors":"Yang Gao, Huazhe Xu, Ji Lin, Fisher Yu, Sergey Levine, Trevor Darrell","year":"2018","venue":"6th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hadfield-Menell et al., 2016] Dylan Hadfield-Menell, Stu- art J. Russell, Pieter Abbeel, and Anca D. Dragan. Co- operative Inverse Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS 2016), Barcelona, Spain, December 5-10, 2016, pages 3909- 3917, 2016.","title":"Cooperative Inverse Reinforcement Learning","authors":"Hadfield-Menell","year":"2016","venue":"Advances in Neural Information Processing Systems (NeurIPS 2016)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hernandez-Leal et al., 2019] Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. A survey and critique of multiagent deep reinforcement learning. Journal of Autonomous Agents and Multi-Agent Systems (JAAMAS), 33(6):750-797, 2019.","title":"A survey and critique of multiagent deep reinforcement learning","authors":"Hernandez-Leal","year":"2019","venue":"Journal of Autonomous Agents and Multi-Agent Systems (JAAMAS)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hester et al., 2018] Todd Hester, Matej Vecerík, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Hor- gan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dulac-Arnold, John P. Agapiou, Joel Z. Leibo, and Au- drunas Gruslys. Deep Q-learning From Demonstrations. In Thirty-Second Conference of Association for the Ad- vancement of Artificial Intelligence (AAAI-18), New Or- leans, Louisiana, USA, February 2-7, 2018, pages 3223- 3230. AAAI Press, 2018.","title":"Deep Q-learning From Demonstrations","authors":"Todd Hester, Matej Vecerík, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dulac-Arnold, John P Agapiou, Joel Z Leibo, Audrunas Gruslys","year":"2018","venue":"Thirty-Second Conference of Association for the Advancement of Artificial Intelligence (AAAI-18)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu and Wellman, 2003] Junling Hu and Michael P Well- man. Nash Q-learning for general-sum stochastic games. Journal of Machine Learning Research (JMLR), 4(Nov):1039-1069, 2003.","title":"Junling Hu and Michael P Wellman. Nash Q-learning for general-sum stochastic games","authors":"Wellman Hu","year":"2003","venue":"Journal of Machine Learning Research (JMLR)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kakade, 2003] Sham Machandranath Kakade. On the sam- ple complexity of reinforcement learning. PhD thesis, UCL (University College London), 2003.","title":"Sham Machandranath Kakade. On the sample complexity of reinforcement learning","authors":"Kakade","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Le et al., 2017] Hoang Minh Le, Yisong Yue, Peter Carr, and Patrick Lucey. Coordinated Multi-Agent Imitation Learning. In Proceedings of the 34th International Con- ference on Machine Learning, (ICML 2017), Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1995-2003. PMLR, 2017.","title":"Coordinated Multi-Agent Imitation Learning","authors":"Minh Hoang, Yisong Le, Peter Yue, Patrick Carr, Lucey","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin et al., 2019] Xiaomin Lin, Stephen C Adams, and Pe- ter A Beling. Multi-agent inverse reinforcement learning for certain general-sum stochastic games. Journal of Arti- ficial Intelligence Research (JAIR), 66:473-502, 2019.","title":"Multi-agent inverse reinforcement learning for certain general-sum stochastic games","authors":"Xiaomin Lin, Stephen C Adams, Peter A Beling","year":"2019","venue":"Journal of Artificial Intelligence Research (JAIR)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environ- ments. In Advances in Neural Information Processing Sys- tems (NeurIPS 2017), Long Beach, CA, USA, December 4-9, 2017, pages 6379-6390, 2017.","title":"Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, Igor Mordatch","year":"2017","venue":"Advances in Neural Information Processing Systems (NeurIPS 2017)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nair et al., 2018] Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming Exploration in Reinforcement Learning with Demonstrations. In 2018 IEEE International Conference on Robotics and Automation, (ICRA 2018) Brisbane, Aus- tralia, May 21-25, 2018, pages 6292-6299. IEEE, 2018.","title":"Overcoming Exploration in Reinforcement Learning with Demonstrations","authors":"Ashvin Nair, Bob Mcgrew, Marcin Andrychowicz, Wojciech Zaremba, Pieter Abbeel","year":"2018","venue":"2018 IEEE International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Natarajan et al., 2010] Sriraam Natarajan, Gautam Kuna- puli, Kshitij Judah, Prasad Tadepalli, Kristian Kersting, and Jude W. Shavlik. Multi-Agent Inverse Reinforcement Learning. In The Ninth International Conference on Ma- chine Learning and Applications, (ICMLA 2010), Wash- ington, DC, USA, 12-14 December 2010, pages 395-400. IEEE Computer Society, 2010.","title":"Multi-Agent Inverse Reinforcement Learning","authors":"Natarajan","year":"2010","venue":"The Ninth International Conference on Machine Learning and Applications","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Peng et al., 2020] Peixi Peng, Junliang Xing, and Lili Cao. Hybrid Learning for Multi-agent Cooperation with Sub- optimal Demonstrations. In Proceedings of the Twenty- Ninth International Joint Conference on Artificial Intel- ligence, (IJCAI-2020), pages 3037-3043. International Joint Conferences on Artificial Intelligence Organization, 7 2020. Main track.","title":"Hybrid Learning for Multi-agent Cooperation with Suboptimal Demonstrations","authors":"Peng","year":"2020","venue":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Perez-Liebana et al., 2019] Diego Perez-Liebana, Raluca D Gaina, Olve Drageset, Ercüment Ilhan, Martin Balla, and Simon M Lucas. Analysis of statistical forward planning methods in pommerman. In AAAI Conference on Arti- ficial Intelligence and Interactive Digital Entertainment, volume 15, pages 66-72, 2019.","title":"Analysis of statistical forward planning methods in pommerman","authors":"Perez-Liebana","year":"2019","venue":"AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Resnick et al., 2018] Cinjon Resnick, Wes Eldridge, David Ha, Denny Britz, Jakob Foerster, Julian Togelius, Kyunghyun Cho, and Joan Bruna. Pommerman: A Multi- Agent Playground. In arXiv preprint arXiv:1809.07124, 2018.","title":"Pommerman: A Multi-Agent Playground","authors":"Resnick","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shapley, 1953] Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences, 39(10):1095-1100, 1953.","title":"Stochastic games","authors":"S Lloyd, Shapley","year":"1953","venue":"Proceedings of the national academy of sciences","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shoham and Leyton-Brown, 2008] Yoav Shoham and Kevin Leyton-Brown. Multiagent systems: Algorithmic, game-theoretic, and logical foundations. Cambridge University Press, 2008.","title":"Multiagent systems: Algorithmic, game-theoretic, and logical foundations","authors":"Yoav Shoham, Kevin Leyton-Brown","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Silva and Costa, 2019] Felipe Leno Da Silva and Anna He- lena Reali Costa. A Survey on Transfer Learning for Mul- tiagent Reinforcement Learning Systems. Journal of Arti- ficial Intelligence Research (JAIR), 64:645-703, 2019.","title":"A Survey on Transfer Learning for Multiagent Reinforcement Learning Systems","authors":"Felipe Leno, Da Silva, Anna Helena, Reali Costa","year":"2019","venue":"Journal of Artificial Intelligence Research (JAIR)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Subramanian et al., 2022] Sriram Ganapathi Subramanian, Matthew E. Taylor, Kate Larson, and Mark Crowley. Multi-Agent Advisor Q-Learning. Journal of Artificial In- telligence Research, 74:1-74, 2022.","title":"Multi-Agent Advisor Q-Learning","authors":"Sriram Ganapathi Subramanian, Matthew E Taylor, Kate Larson, Mark Crowley","year":"2022","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Introduction to Reinforcement Learning, volume 135. MIT Press Cambridge, 1998.","title":"Introduction to Reinforcement Learning","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang and Klabjan, 2018] Xingyu Wang and Diego Klab- jan. Competitive Multi-agent Inverse Reinforcement Learning with Sub-optimal Demonstrations. In Proceed- ings of the Thirty-fifth International Conference on Ma- chine Learning (ICML 2018), Stockholm, Sweden, July 10- 15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 5130-5138. PMLR, 2018.","title":"Xingyu Wang and Diego Klabjan. Competitive Multi-agent Inverse Reinforcement Learning with Sub-optimal Demonstrations","authors":"Klabjan Wang","year":"2018","venue":"Proceedings of the Thirty-fifth International Conference on Machine Learning (ICML 2018)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang and Taylor, 2017] Zhaodong Wang and Matthew E. Taylor. Improving Reinforcement Learning with Confidence-Based Demonstrations. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI 2017), Melbourne, Australia, August 19-25, 2017, pages 3027-3033. ijcai.org, 2017.","title":"Improving Reinforcement Learning with Confidence-Based Demonstrations","authors":"Wang, Taylor ;, Zhaodong Wang, Matthew E Taylor","year":"2017","venue":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI 2017)","doi":null,"url":null,"internal_url":"/proceedings/2017/422"},{"position":27,"raw":"[Watkins and Dayan, 1992] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine Learning, 8(3-4):279- 292, 1992.","title":"Q-learning","authors":"Jch Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yogeswaran and Ponnambalam, 2012] Mohan Yogeswaran and SG Ponnambalam. Reinforcement learning: Exploration-exploitation dilemma in multi-agent foraging task. Opsearch, 49(3):223-236, 2012.","title":"Mohan Yogeswaran and SG Ponnambalam. Reinforcement learning: Exploration-exploitation dilemma in multi-agent foraging task","authors":"Ponnambalam Yogeswaran","year":"2012","venue":"Opsearch","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yu et al., 2019] Lantao Yu, Jiaming Song, and Stefano Er- mon. Multi-Agent Adversarial Inverse Reinforcement Learning. In Proceedings of the Thirty-Sixth International Conference on Machine Learning, (ICML 2019), Long Beach, California, USA, 9-15 June 2019, volume 97 of Proceedings of Machine Learning Research, pages 7194- 7201. PMLR, 2019.","title":"Multi-Agent Adversarial Inverse Reinforcement Learning","authors":"Lantao Yu, Jiaming Song, Stefano Ermon","year":"2019","venue":"Proceedings of the Thirty-Sixth International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhou et al., 2020] Ming Zhou, Jun Luo, Julian Villela, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng Chen, Au- rora Chongxi Huang, Ying Wen, Kimia Hassanzadeh, Daniel Graves, Dong Chen, Zhengbang Zhu, Nhat M. Nguyen, Mohamed Elsayed, Kun Shao, Sanjeevan Ahi- lan, Baokuan Zhang, Jiannan Wu, Zhengang Fu, Kasra Rezaee, Peyman Yadmellat, Mohsen Rohani, Nico- las Perez Nieves, Yihan Ni, Seyedershad Banijamali, Alexander Imani Cowen-Rivers, Zheng Tian, Daniel Palenicek, Haitham Bou-Ammar, Hongbo Zhang, Wulong Liu, Jianye Hao, and Jun Wang. SMARTS: Scalable Multi-agent Reinforcement Learning Training School for Autonomous Driving. CoRR, abs/2010.09776, 2020.","title":"SMARTS: Scalable Multi-agent Reinforcement Learning Training School for Autonomous Driving","authors":"Ming Zhou, Jun Luo, Julian Villela, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng Chen, Aurora Chongxi Huang, Ying Wen, Kimia Hassanzadeh, Daniel Graves, Dong Chen, Zhengbang Zhu, M Nhat, Mohamed Nguyen, Kun Elsayed, Sanjeevan Shao, Baokuan Ahilan, Jiannan Zhang, Zhengang Wu, Kasra Fu, Peyman Rezaee, Mohsen Yadmellat, Nicolas Perez Rohani, Yihan Nieves, Seyedershad Ni, Alexander Banijamali, Zheng Imani Cowen-Rivers, Daniel Tian, Haitham Palenicek, Hongbo Bou-Ammar, Wulong Zhang, Jianye Liu, Jun Hao, Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}