{"type":"ScholarlyArticle","title":"Safe Multi-agent Learning via Trapping Regions","authors":[{"name":"Aleksander Czechowski","slug":"aleksander-czechowski","given":"Aleksander","family":"Czechowski","affiliation":"Delft University of Technology"},{"name":"Frans A. Oliehoek","slug":"frans-a-oliehoek","given":"Frans A.","family":"Oliehoek","affiliation":"Delft University of Technology"}],"abstract":"One of the main challenges of multi-agent learning lies in establishing convergence of the algorithms, as, in general, a collection of individual, self-serving agents is not guaranteed to converge with their joint policy, when learning concurrently. This is in stark contrast to most single-agent environments, and sets a prohibitive barrier for deployment in practical applications, as it induces uncertainty in long term behavior of the system. In this work, we apply the concept of trapping regions, known from qualitative theory of dynamical systems, to create safety sets in the joint strategy space for decentralized learning. We propose a binary partitioning algorithm for verification that candidate sets form trapping regions in systems with known learning dynamics, and a heuristic sampling algorithm for scenarios where learning dynamics are not known. We demonstrate the applications to a regularized version of Dirac Generative Adversarial Network,  a four-intersection traffic control scenario run in a state of the art open-source microscopic traffic simulator SUMO, and a mathematical model of economic competition.","doi":"10.24963/ijcai.2023/10","url":"https://ijcai.org/proceedings/2023/10","pdf":"https://ijcai.org/proceedings/2023/0010.pdf","markdown":"https://ijcai.org/proceedings/2023/10.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/10","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":10,"pages":{"first":82,"last":90},"keywords":["Agent-based and Multi-agent Systems: MAS: Multi-agent learning"],"videos":[],"references":[{"position":1,"raw":"[Altman, 1999] Eitan Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999.","title":"Constrained Markov decision processes","authors":"Eitan Altman","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Berkenkamp et al., 2017] Felix Berkenkamp, Matteo Turchetta, Angela P. Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. In Proceedings of the 31st International Conference on Neural Information Processing Systems, page 908-919, 2017.","title":"Safe model-based reinforcement learning with stability guarantees","authors":"Felix Berkenkamp, Matteo Turchetta, Angela P Schoellig, Andreas Krause","year":"2017","venue":"Proceedings of the 31st International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bonatti, 2006] Christian Bonatti. Generic properties of dy- namical systems. Encyclopedia of Mathematical Physics, pages 494-502, 2006.","title":"Christian Bonatti. Generic properties of dynamical systems","authors":"Bonatti","year":"2006","venue":"Encyclopedia of Mathematical Physics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brown, 1951] George W Brown. Iterative solution of games by fictitious play. Activity analysis of production and allo- cation, 13(1):374-376, 1951.","title":"Iterative solution of games by fictitious play","authors":"George W Brown","year":"1951","venue":"Activity analysis of production and allocation","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chang et al., 2019] Ya-Chien Chang, Nima Roohi, and Si- cun Gao. Neural lyapunov control. In NeurIPS, vol- ume 32, 2019.","title":"Neural lyapunov control","authors":"Ya-Chien Chang, Nima Roohi, Sicun Gao","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Czechowski and Oliehoek, 2023] Aleksander Czechowski and Frans A. Oliehoek. Safety guarantees in multi-agent learning via trapping regions. arXiv, https://arxiv.org/abs/2302.13844, 2023.","title":"Safety guarantees in multi-agent learning via trapping regions","authors":"Aleksander Czechowski, Frans A Oliehoek","year":"2023","venue":null,"doi":null,"url":"https://arxiv.org/abs/2302.13844","internal_url":null},{"position":7,"raw":"[ElSayed-Aly et al., 2021] Ingy ElSayed-Aly, Suda Bharad- waj, Christopher Amato, Rüdiger Ehlers, Ufuk Topcu, and Lu Feng. Safe multi-agent reinforcement learning via shielding. AAMAS, 2021.","title":"Safe multi-agent reinforcement learning via shielding","authors":"Ingy Elsayed-Aly, Suda Bharadwaj, Christopher Amato, Rüdiger Ehlers, Ufuk Topcu, Lu Feng","year":"2021","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fuchs et al., 1980] Henry Fuchs, Zvi M Kedem, and Bruce F Naylor. On visible surface generation by a pri- ori tree structures. In Proceedings of the 7th annual con- ference on Computer graphics and interactive techniques, pages 124-133, 1980.","title":"On visible surface generation by a priori tree structures","authors":"Henry Fuchs, Zvi M Kedem, Bruce F Naylor","year":"1980","venue":"Proceedings of the 7th annual conference on Computer graphics and interactive techniques","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Garcıa and Fernández, 2015] Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning. JMLR, 16(1):1437-1480, 2015.","title":"Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning","authors":"Fernández Garcıa","year":"2015","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hart and Mas-Colell, 2003] Sergiu Hart and Andreu Mas- Colell. Uncoupled dynamics do not lead to nash equi- librium. American Economic Review, 93(5):1830-1836, December 2003.","title":"Uncoupled dynamics do not lead to nash equilibrium","authors":"Sergiu Hart, Andreu Mas-Colell","year":"2003","venue":"American Economic Review","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kleinberg et al., 2011] Robert D Kleinberg, Katrina Ligett, Georgios Piliouras, and Éva Tardos. Beyond the Nash equilibrium barrier. In ICS, pages 125-140, 2011.","title":"Beyond the Nash equilibrium barrier","authors":"Katrina Robert D Kleinberg, Georgios Ligett, Éva Piliouras, Tardos","year":"2011","venue":"ICS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kuyer et al., 2008] Lior Kuyer, Shimon Whiteson, Bram Bakker, and Nikos Vlassis. Multiagent reinforcement learning for urban traffic control using coordination graphs. In Joint European Conference on Machine Learn- ing and Knowledge Discovery in Databases, pages 656- 671. Springer, 2008.","title":"Multiagent reinforcement learning for urban traffic control using coordination graphs","authors":"Lior Kuyer, Shimon Whiteson, Bram Bakker, Nikos Vlassis","year":"2008","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lanctot et al., 2017] Marc Lanctot, Vinicius Zambaldi, Au- drunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game- theoretic approach to multiagent reinforcement learning. NIPS, page 4193-4206, 2017.","title":"A unified gametheoretic approach to multiagent reinforcement learning","authors":"Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, Thore Graepel","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lockhart et al., 2019] Edward Lockhart, Marc Lanctot, Julien Pérolat, Jean-Baptiste Lespiau, Dustin Morrill, Fin- barr Timbers, and Karl Tuyls. Computing approximate equilibria in sequential adversarial games by exploitabil- ity descent. In IJCAI, page 464-470, 2019.","title":"Computing approximate equilibria in sequential adversarial games by exploitability descent","authors":"Lockhart","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/66"},{"position":15,"raw":"[Lopez et al., 2018] Pablo Alvarez Lopez, Michael Behrisch, Laura Bieker-Walz, Jakob Erdmann, Yun- Pang Flötteröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wagner, and Evamarie Wießner. Microscopic traffic simulation using SUMO. In 2018 21st International Conference on Intelligent Transportation Systems, pages 2575-2582, 2018.","title":"Microscopic traffic simulation using SUMO","authors":"Lopez","year":"2018","venue":"21st International Conference on Intelligent Transportation Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environ- ments. NIPS, page 6382-6393, 2017.","title":"Multi-agent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, Igor Mordatch","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lu et al., 2021] Songtao Lu, Kaiqing Zhang, Tianyi Chen, Tamer Basar, Lior Horesh, Ria Vinod, Pin Yu Chen, et al. Decentralized policy gradient descent ascent for safe multi-agent reinforcement learning. In AAAI, volume 35, pages 8767-8775, 2021.","title":"Decentralized policy gradient descent ascent for safe multi-agent reinforcement learning","authors":"Songtao Lu, Kaiqing Zhang, Tianyi Chen, Tamer Basar, Lior Horesh, Ria Vinod, Pin Yu Chen","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Meiss, 2007] James D Meiss. Differential dynamical sys- tems. SIAM, 2007.","title":"Differential dynamical systems","authors":"James D Meiss","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mescheder et al., 2018] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In ICML, pages 3481-3490, 2018.","title":"Which training methods for gans do actually converge?","authors":"Mescheder","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. NIPS Deep Learning Workshop, 2013.","title":"Playing Atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":"NIPS Deep Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Oliehoek et al., 2006] Frans A Oliehoek, Edwin D De Jong, and Nikos Vlassis. The parallel nash memory for asym- metric games. In Proceedings of the 8th annual conference on genetic and evolutionary computation, pages 337-344, 2006.","title":"The parallel nash memory for asymmetric games","authors":"Edwin D De Frans A Oliehoek, Nikos Jong, Vlassis","year":"2006","venue":"Proceedings of the 8th annual conference on genetic and evolutionary computation","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ruan et al., 2018] Wenjie Ruan, Xiaowei Huang, and Marta Kwiatkowska. Reachability analysis of deep neural net- works with provable guarantees. IJCAI, page 2651-2659, 2018.","title":"Reachability analysis of deep neural networks with provable guarantees","authors":"Wenjie Ruan, Xiaowei Huang, Marta Kwiatkowska","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/368"},{"position":23,"raw":"[Sato et al., 2002] Yuzuru Sato, Eizo Akiyama, and J Doyne Farmer. Chaos in learning a simple two-person game. PNAS, 99(7):4748-4751, 2002.","title":"Chaos in learning a simple two-person game","authors":"Yuzuru Sato, Eizo Akiyama, J Doyne Farmer","year":"2002","venue":"PNAS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sutton et al., 1999] Richard S Sutton, David A McAllester, Satinder P Singh, Yishay Mansour, et al. Policy gradient methods for reinforcement learning with function approx- imation. In NIPS, volume 99, pages 1057-1063, 1999.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David A Richard S Sutton, Mcallester, P Satinder, Yishay Singh, Mansour","year":"1999","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2021] Xiaoyan Wang, Jun Peng, Shuqiu Li, and Bing Li. Formal reachability analysis for multi-agent reinforcement learning systems. IEEE Access, 9:45812- 45821, 2021.","title":"Formal reachability analysis for multi-agent reinforcement learning systems","authors":"Xiaoyan Wang, Jun Peng, Shuqiu Li, Bing Li","year":"2021","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Watkins and Dayan, 1992] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279- 292, 1992.","title":"Q-learning","authors":"Jch Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wills and Heath, 2004] Adrian G. Wills and William P. Heath. Barrier function based model predictive control. Automatica, 40(8):1415-1422, 2004.","title":"Barrier function based model predictive control","authors":"Adrian G Wills, William P Heath","year":"2004","venue":"Automatica","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2018] Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi- agent reinforcement learning. In ICML, pages 5571-5580, 2018.","title":"Mean field multiagent reinforcement learning","authors":"Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, Jun Wang","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2020] Yongliang Yang, Kyriakos G Vamvoudakis, and Hamidreza Modares. Safe rein- forcement learning for dynamical games. International Journal of Robust and Nonlinear Control, 30(9):3706- 3726, 2020.","title":"Safe reinforcement learning for dynamical games","authors":"Yongliang Yang, Kyriakos G Vamvoudakis, Hamidreza Modares","year":"2020","venue":"International Journal of Robust and Nonlinear Control","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}