{"type":"ScholarlyArticle","title":"Inducing Stackelberg Equilibrium through Spatio-Temporal Sequential Decision-Making in Multi-Agent Reinforcement Learning","authors":[{"name":"Bin Zhang","slug":"bin-zhang","given":"Bin","family":"Zhang","affiliation":"Institute of Automation , Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences {zhangbin2020 ,"},{"name":"Lijuan Li","slug":"lijuan-li","given":"Lijuan","family":"Li","affiliation":"Institute of Automation , Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences {zhangbin2020 ,"},{"name":"Zhiwei Xu","slug":"zhiwei-xu","given":"Zhiwei","family":"Xu","affiliation":"Institute of Automation , Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences {zhangbin2020 ,"},{"name":"Dapeng Li","slug":"dapeng-li","given":"Dapeng","family":"Li","affiliation":"Institute of Automation , Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences {zhangbin2020 ,"},{"name":"Guoliang Fan","slug":"guoliang-fan","given":"Guoliang","family":"Fan","affiliation":"Institute of Automation , Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences {zhangbin2020 ,"}],"abstract":"In multi-agent reinforcement learning (MARL), self-interested agents attempt to establish equilibrium and achieve coordination depending on game structure. However, existing MARL approaches are mostly bound by the simultaneous actions of all agents in the Markov game (MG) framework, and few works consider the formation of equilibrium strategies via asynchronous action coordination. In view of the advantages of Stackelberg equilibrium (SE) over Nash equilibrium, we construct a spatio-temporal sequential decision-making structure derived from the MG and propose an N-level policy model based on a conditional hypernetwork shared by all agents. This approach allows for asymmetric training with symmetric execution, with each agent responding optimally conditioned on the decisions made by superior agents. Agents can learn heterogeneous SE policies while still maintaining parameter sharing, which leads to reduced cost for learning and storage and enhanced scalability as the number of agents increases. Experiments demonstrate that our method effectively converges to the SE policies in repeated matrix game scenarios, and performs admirably in immensely complex settings including cooperative tasks and mixed tasks.","doi":"10.24963/ijcai.2023/40","url":"https://ijcai.org/proceedings/2023/40","pdf":"https://ijcai.org/proceedings/2023/0040.pdf","markdown":"https://ijcai.org/proceedings/2023/40.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/40","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":40,"pages":{"first":353,"last":361},"keywords":["Machine Learning: ML: Reinforcement learning","Game Theory and Economic Paradigms: GTEP: Noncooperative games","Agent-based and Multi-agent Systems: MAS: Multi-agent learning","Agent-based and Multi-agent Systems: MAS: Coordination and cooperation"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2021] Dong Chen, Zhaojian Li, Yongqiang Wang, Longsheng Jiang, and Yue Wang. Deep multi-agent reinforcement learning for highway on-ramp merging in mixed traffic. arXiv preprint arXiv:2105.05701, 2021.","title":"Deep multi-agent reinforcement learning for highway on-ramp merging in mixed traffic","authors":"Dong Chen, Zhaojian Li, Yongqiang Wang, Longsheng Jiang, Yue Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2022] Yiqun Chen, Hangyu Mao, Tianle Zhang, Shiguang Wu, Bin Zhang, Jianye Hao, Dong Li, Bin Wang, and Hongxing Chang. Ptde: Personalized train- ing with distillated execution for multi-agent reinforce- ment learning. arXiv preprint arXiv:2210.08872, 2022.","title":"Ptde: Personalized training with distillated execution for multi-agent reinforcement learning","authors":"Yiqun Chen, Hangyu Mao, Tianle Zhang, Shiguang Wu, Bin Zhang, Jianye Hao, Dong Li, Bin Wang, Hongxing Chang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Claus and Boutilier, 1998] Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems. In Jack Mostow and Chuck Rich, editors, Proceedings of the Fifteenth National Conference on Artificial Intelligence and Tenth Innovative Applications of Artificial Intelligence Conference, AAAI 98, IAAI 98, July 26-30, 1998, Madison, Wisconsin, USA, pages 746-752. AAAI Press / The MIT Press, 1998.","title":"The dynamics of reinforcement learning in cooperative multiagent systems","authors":"Claus, Boutilier, Caroline Claus, Craig Boutilier","year":"1998","venue":"Proceedings of the Fifteenth National Conference on Artificial Intelligence and Tenth Innovative Applications of Artificial Intelligence Conference, AAAI 98, IAAI 98","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Foerster et al., 2016] Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learn- ing. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Ad- vances in Neural Information Processing Systems 29: An- nual Conference on Neural Information Processing Sys- tems 2016, December 5-10, 2016, Barcelona, Spain, pages 2137-2145, 2016.","title":"Learning to communicate with deep multi-agent reinforcement learning","authors":"Jakob N Foerster, M Yannis, Nando Assael, Shimon De Freitas, Whiteson","year":"2016","venue":"Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hu and Wellman, 2003] Junling Hu and Michael P. Well- man. Nash q-learning for general-sum stochastic games. J. Mach. Learn. Res., 4:1039-1069, 2003.","title":"Wellman. Nash q-learning for general-sum stochastic games","authors":"Wellman Hu, Junling Hu, Michael P","year":"2003","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Iqbal and Sha, 2019] Shariq Iqbal and Fei Sha. Actor- attention-critic for multi-agent reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Ma- chine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2961-2970. PMLR, 2019.","title":"Actorattention-critic for multi-agent reinforcement learning","authors":"Shariq Iqbal, Fei Sha","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning, ICML 2019","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Könönen, 2004] Ville Könönen. Asymmetric multiagent re- inforcement learning. Web Intell. Agent Syst., 2(2):105- 121, 2004.","title":"Asymmetric multiagent reinforcement learning","authors":"Ville Könönen, Könönen","year":"2004","venue":"Web Intell. Agent Syst","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kuba et al., 2022] Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi- agent reinforcement learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Vir- tual Event, April 25-29, 2022. OpenReview.net, 2022.","title":"Trust region policy optimisation in multiagent reinforcement learning","authors":"Ruiqing Jakub Grudzien Kuba, Muning Chen, Ying Wen, Fanglei Wen, Jun Sun, Yaodong Wang, Yang","year":"2022","venue":"The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lillicrap et al., 2016] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yu- val Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Ben- gio and Yann LeCun, editors, 4th International Confer- ence on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceed- ings, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra","year":"2016","venue":"4th International Conference on Learning Representations, ICLR 2016","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Littman and Stone, 2001] Michael L Littman and Peter Stone. Leading best-response strategies in repeated games. In In Seventeenth Annual International Joint Conference on Artificial Intelligence Workshop on Economic Agents, Models, and Mechanisms. Citeseer, 2001.","title":"Leading best-response strategies in repeated games","authors":"L Michael, Peter Littman, Stone","year":"2001","venue":"Seventeenth Annual International Joint Conference on Artificial Intelligence Workshop on Economic Agents, Models, and Mechanisms","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Littman, 1994] Michael L. Littman. Markov games as a framework for multi-agent reinforcement learning. In William W. Cohen and Haym Hirsh, editors, Machine Learning, Proceedings of the Eleventh International Con- ference, Rutgers University, New Brunswick, NJ, USA, July 10-13, 1994, pages 157-163.","title":"Markov games as a framework for multi-agent reinforcement learning","authors":"L Michael, Littman","year":"1994","venue":"Machine Learning, Proceedings of the Eleventh International Conference","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Morgan Kaufmann, 1994.","title":null,"authors":"Morgan Kaufmann","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environ- ments. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vish- wanathan, and Roman Garnett, editors, Advances in Neu- ral Information Processing Systems 30: Annual Confer- ence on Neural Information Processing Systems 2017, De- cember 4-9, 2017, Long Beach, CA, USA, pages 6379- 6390, 2017.","title":"Multi-agent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, Igor Mordatch","year":"2017","venue":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Metz et al., 2017] Luke Metz, Julian Ibarz, Navdeep Jaitly, and James Davidson. Discrete sequential prediction of continuous actions for deep RL. arXiv preprint arXiv:1705.05035, 2017.","title":"Discrete sequential prediction of continuous actions for deep RL","authors":"Metz","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Peng et al., 2021] Bei Peng, Tabish Rashid, Chris- tian Schröder de Witt, Pierre-Alexandre Kamienny, Philip H. S. Torr, Wendelin Boehmer, and Shimon Whiteson. FACMAC: factored multi-agent centralised policy gra- dients. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 12208-12221, 2021.","title":"FACMAC: factored multi-agent centralised policy gradients","authors":"Bei Peng, Tabish Rashid, Christian Schröder De Witt, Pierre-Alexandre Kamienny, H S Philip, Wendelin Torr, Shimon Boehmer, Whiteson","year":"2021","venue":"Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rashid et al., 2018] Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Ma- chine Learning, ICML 2018, Stockholmsmässan, Stock- holm, Sweden, July 10-15, 2018, volume 80 of Proceed- ings of Machine Learning Research, pages 4292-4301.","title":"QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning","authors":"Tabish Rashid, Mikayel Samvelyan, Christian Schröder De Witt, Gregory Farquhar, Jakob N Foerster, Shimon Whiteson","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"PMLR, 2018.","title":null,"authors":"Pmlr","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tavakoli et al., 2018] Arash Tavakoli, Fabio Pardo, and Petar Kormushev. Action branching architectures for deep reinforcement learning. In Sheila A. McIlraith and Kil- ian Q. Weinberger, editors, Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence, (AAAI- 18), the 30th innovative Applications of Artificial Intel- ligence (IAAI-18), and the 8th AAAI Symposium on Ed- ucational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 4131-4138. AAAI Press, 2018.","title":"Action branching architectures for deep reinforcement learning","authors":"Arash Tavakoli, Fabio Pardo, Petar Kormushev","year":"2018","venue":"AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tian et al., 2019] Zheng Tian, Ying Wen, Zhichen Gong, Faiz Punakkath, Shihao Zou, and Jun Wang. A regular- ized opponent model with maximum entropy objective. In Sarit Kraus, editor, Proceedings of the Twenty-Eighth In- ternational Joint Conference on Artificial Intelligence, IJ- CAI 2019, Macao, China, August 10-16, 2019, pages 602- 608. ijcai.org, 2019.","title":"A regularized opponent model with maximum entropy objective","authors":"Zheng Tian, Ying Wen, Zhichen Gong, Faiz Punakkath, Shihao Zou, Jun Wang","year":"2019","venue":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJ-CAI 2019","doi":null,"url":null,"internal_url":"/proceedings/2019/85"},{"position":22,"raw":"[von Oswald et al., 2020] Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe. Con- tinual learning with hypernetworks. In 8th International Conference on Learning Representations, ICLR 2020, Ad- dis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.","title":"Continual learning with hypernetworks","authors":"Oswald Von","year":"2020","venue":"8th International Conference on Learning Representations, ICLR 2020, Addis","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Von Stackelberg, 2010] Heinrich Von Stackelberg. Market structure and equilibrium. Springer Science & Business Media, 2010.","title":"Heinrich Von Stackelberg. Market structure and equilibrium","authors":"Von Stackelberg","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2021] Zhiwei Xu, Bin Zhang, Yunpeng Bai, Dapeng Li, and Guoliang Fan. Learning to coordinate via multiple graph neural networks. In International Con- ference on Neural Information Processing, pages 52-63. Springer, 2021.","title":"Learning to coordinate via multiple graph neural networks","authors":"Zhiwei Xu, Bin Zhang, Yunpeng Bai, Dapeng Li, Guoliang Fan","year":"2021","venue":"International Conference on Neural Information Processing","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2018] Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi- agent reinforcement learning. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th Inter- national Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 5567-5576. PMLR, 2018.","title":"Mean field multiagent reinforcement learning","authors":"Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, Jun Wang","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yu et al., 2021] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surpris- ing effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955, 2021.","title":"The surprising effectiveness of ppo in cooperative, multi-agent games","authors":"Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, Yi Wu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2018] Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar. Fully decentral- ized multi-agent reinforcement learning with networked agents. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Ma- chine Learning, ICML 2018, Stockholmsmässan, Stock- holm, Sweden, July 10-15, 2018, volume 80 of Proceed- ings of Machine Learning Research, pages 5867-5876.","title":"Fully decentralized multi-agent reinforcement learning with networked agents","authors":"Zhang","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"PMLR, 2018.","title":null,"authors":"Pmlr","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2020] Haifeng Zhang, Weizhe Chen, Zeren Huang, Minne Li, Yaodong Yang, Weinan Zhang, and Jun Wang. Bi-level actor-critic for multi-agent coordination. In The Thirty-Fourth AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Thirty-Second Innovative Applica- tions of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Arti- ficial Intelligence, EAAI 2020, New York, NY, USA, Febru- ary 7-12, 2020, pages 7325-7332. AAAI Press, 2020.","title":"Bi-level actor-critic for multi-agent coordination","authors":"Haifeng Zhang, Weizhe Chen, Zeren Huang, Minne Li, Yaodong Yang, Weinan Zhang, Jun Wang","year":"2020","venue":"The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2023a] Bin Zhang, Yunpeng Bai, Zhiwei Xu, Dapeng Li, and Guoliang Fan. Efficient policy gener- ation in multi-agent systems via hypergraph neural net- work. In Neural Information Processing: 29th Interna- tional Conference, ICONIP 2022, Virtual Event, Novem- ber 22-26, 2022, Proceedings, Part II, pages 219-230. Springer, 2023.","title":"Efficient policy generation in multi-agent systems via hypergraph neural network","authors":"Zhang","year":"2022","venue":"Neural Information Processing: 29th International Conference, ICONIP 2022, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2023b] Bin Zhang, Zhiwei Xu, Yiqun Chen, Dapeng Li, Yunpeng Bai, Guoliang Fan, and Lijuan Li. Multi-agent hyper-attention policy optimization. In Neural Information Processing: 29th International Conference, ICONIP 2022, Virtual Event, November 22-26, 2022, Pro- ceedings, Part I, pages 76-87. Springer, 2023.","title":"Multi-agent hyper-attention policy optimization","authors":"Zhang","year":"2022","venue":"Neural Information Processing: 29th International Conference, ICONIP 2022, Virtual Event","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}