{"type":"ScholarlyArticle","title":"Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training","authors":[{"name":"Peide Huang","slug":"peide-huang","given":"Peide","family":"Huang","affiliation":"Carnegie Mellon University"},{"name":"Mengdi Xu","slug":"mengdi-xu","given":"Mengdi","family":"Xu","affiliation":"Carnegie Mellon University"},{"name":"Fei Fang","slug":"fei-fang","given":"Fei","family":"Fang","affiliation":"Carnegie Mellon University"},{"name":"Ding Zhao","slug":"ding-zhao","given":"Ding","family":"Zhao","affiliation":"Carnegie Mellon University"}],"abstract":"Robust Reinforcement Learning (RL) focuses on improving performances under model errors or adversarial attacks, which facilitates the real-life deployment of RL agents. Robust Adversarial Reinforcement Learning (RARL) is one of the most popular frameworks for robust RL. However, most of the existing literature models RARL as a zero-sum simultaneous game with Nash equilibrium as the solution concept, which could overlook the sequential nature of RL deployments, produce overly conservative agents, and induce training instability. In this paper, we introduce a novel hierarchical formulation of robust RL -- a general-sum Stackelberg game model called RRL-Stack -- to formalize the sequential nature and provide extra flexibility for robust training. We develop the Stackelberg Policy Gradient algorithm to solve RRL-Stack, leveraging the Stackelberg learning dynamics by considering the adversary's response. Our method generates challenging yet solvable adversarial environments which benefit RL agents' robust learning. Our algorithm demonstrates better training stability and robustness against different testing conditions in the single-agent robotics control and multi-agent highway merging tasks.","doi":"10.24963/ijcai.2022/430","url":"https://ijcai.org/proceedings/2022/430","pdf":"https://ijcai.org/proceedings/2022/0430.pdf","markdown":"https://ijcai.org/proceedings/2022/430.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/430","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":430,"pages":{"first":3099,"last":3106},"keywords":["Machine Learning: Robustness","Machine Learning: Deep Reinforcement Learning","Agent-based and Multi-agent Systems: Multi-agent Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986677","length_sec":80,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984820","length_sec":353,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Abraham et al., 2012] Ralph Abraham, Jerrold E Marsden, and Tudor Ratiu. Manifolds, tensor analysis, and appli- cations, volume 75. Springer Science & Business Media, 2012.","title":"Manifolds, tensor analysis, and applications","authors":"Abraham","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Akkaya et al., 2019] Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113, 2019.","title":"Solving rubik's cube with a robot hand","authors":"Akkaya","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Başar and Olsder, 1998] Tamer Başar and Geert Jan Olsder. Dynamic noncooperative game theory. SIAM, 1998.","title":"Dynamic noncooperative game theory","authors":"Olsder Başar","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Lud- wig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2021] Baiming Chen, Mengdi Xu, Liang Li, and Ding Zhao. Delay-aware model-based reinforcement learn- ing for continuous control. Neurocomputing, 450:119-128, 2021.","title":"Delay-aware model-based reinforcement learning for continuous control","authors":"Chen","year":"2021","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dennis et al., 2020] Michael Dennis, Natasha Jaques, Eu- gene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero- shot transfer via unsupervised environment design. arXiv preprint arXiv:2012.02096, 2020.","title":"Emergent complexity and zeroshot transfer via unsupervised environment design","authors":"Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, Sergey Levine","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Désidéri, 2012] Jean-Antoine Désidéri. Multiple-gradient descent algorithm (mgda) for multiobjective optimization. Comptes Rendus Mathematique, 350(5-6):313-318, 2012.","title":"Multiple-gradient descent algorithm (mgda) for multiobjective optimization","authors":"Jean-Antoine Désidéri","year":"2012","venue":"Comptes Rendus Mathematique","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ding et al., 2021] Wenhao Ding, Baiming Chen, Bo Li, Kim Ji Eun, and Ding Zhao. Multimodal safety-critical scenarios generation for decision-making algorithms evalu- ation. IEEE Robotics and Automation Letters, 6(2):1551- 1558, 2021.","title":"Multimodal safety-critical scenarios generation for decision-making algorithms evaluation","authors":"Wenhao Ding, Baiming Chen, Bo Li, Kim Ji Eun, Ding Zhao","year":"2021","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fiez et al., 2020] Tanner Fiez, Benjamin Chasnov, and Lil- lian Ratliff. Implicit learning dynamics in stackelberg games: Equilibria characterization, convergence analysis, and empirical study. In International Conference on Ma- chine Learning, pages 3133-3144. PMLR, 2020.","title":"Implicit learning dynamics in stackelberg games: Equilibria characterization, convergence analysis, and empirical study","authors":"Fiez","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Foerster et al., 2017] Jakob N Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning aware- ness. arXiv preprint arXiv:1709.04326, 2017.","title":"Learning with opponent-learning awareness","authors":"Jakob N Foerster, Maruan Richard Y Chen, Shimon Al-Shedivat, Pieter Whiteson, Igor Abbeel, Mordatch","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jin et al., 2020] Chi Jin, Praneeth Netrapalli, and Michael Jordan. What is local optimality in nonconvex-nonconcave minimax optimization? In International Conference on Machine Learning, pages 4880-4889. PMLR, 2020.","title":"What is local optimality in nonconvex-nonconcave minimax optimization","authors":"Jin","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kamalaruban et al., 2020] Parameswaran Kamalaruban, Yu- Ting Huang, Ya-Ping Hsieh, Paul Rolland, Cheng Shi, and Volkan Cevher. Robust reinforcement learning via adver- sarial training with langevin dynamics. arXiv preprint arXiv:2002.06063, 2020.","title":"Robust reinforcement learning via adversarial training with langevin dynamics","authors":"Parameswaran Kamalaruban, Yu-Ting Huang, Ya-Ping Hsieh, Paul Rolland, Cheng Shi, Volkan Cevher","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Leurent, 2018] Edouard Leurent. An environment for au- tonomous driving decision-making. https://github.com/ eleurent/highway-env, 2018. Accessed: 2022-05-21.","title":"Edouard Leurent. An environment for autonomous driving decision-making","authors":"Leurent","year":"2018","venue":null,"doi":null,"url":"https://github.com/eleurent/highway-env,2018","internal_url":null},{"position":14,"raw":"[Li et al., 2019] Bin Li, Peng Luo, and Dewen Xiong. Equi- librium strategies for alpha-maxmin expected utility max- imization. SIAM Journal on Financial Mathematics, 10(2):394-429, 2019.","title":"Equilibrium strategies for alpha-maxmin expected utility maximization","authors":"Bin Li, Peng Luo, Dewen Xiong","year":"2019","venue":"SIAM Journal on Financial Mathematics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Martens et al., 2012] James Martens, Ilya Sutskever, and Kevin Swersky. Estimating the hessian by back- propagating curvature. arXiv preprint arXiv:1206.6464, 2012.","title":"Estimating the hessian by backpropagating curvature","authors":"James Martens, Ilya Sutskever, Kevin Swersky","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Morimoto and Doya, 2005] Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural computation, 17(2):335-359, 2005.","title":"Robust reinforcement learning","authors":"Doya Morimoto, Jun Morimoto, Kenji Doya","year":"2005","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pinto et al., 2017] Lerrel Pinto, James Davidson, Rahul Suk- thankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In International Conference on Machine Learning, pages 2817-2826. PMLR, 2017.","title":"Robust adversarial reinforcement learning","authors":"Pinto","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Saad and Schultz, 1986] Youcef Saad and Martin H Schultz. Gmres: A generalized minimal residual algorithm for solv- ing nonsymmetric linear systems. SIAM Journal on scien- tific and statistical computing, 7(3):856-869, 1986.","title":"Gmres: A generalized minimal residual algorithm for solving nonsymmetric linear systems","authors":"Schultz Saad, Youcef Saad, Martin H Schultz","year":"1986","venue":"SIAM Journal on scientific and statistical computing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schäfer et al., 2019] Florian Schäfer, Hongkai Zheng, and Anima Anandkumar. Implicit competitive regularization in gans. arXiv preprint arXiv:1910.05852, 2019.","title":"Implicit competitive regularization in gans","authors":"Florian Schäfer, Hongkai Zheng, Anima Anandkumar","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shapley, 1953] Lloyd S Shapley. Stochastic games. Pro- ceedings of the national academy of sciences, 39(10):1095- 1100, 1953.","title":"Stochastic games","authors":"S Lloyd, Shapley","year":"1953","venue":"Proceedings of the national academy of sciences","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shen et al., 2020] Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao. Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning, pages 8707-8718. PMLR, 2020.","title":"Deep reinforcement learning with robust and smooth policy","authors":"Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, Tuo Zhao","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shewchuk and others, 1994] Jonathan Richard Shewchuk et al. An introduction to the conjugate gradient method without the agonizing pain, 1994.","title":"An introduction to the conjugate gradient method without the agonizing pain","authors":"Jonathan Richard, Shewchuk","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tessler et al., 2019] Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applica- tions in continuous control. In International Conference on Machine Learning, pages 6215-6224. PMLR, 2019.","title":"Action robust reinforcement learning and applications in continuous control","authors":"Chen Tessler, Yonathan Efroni, Shie Mannor","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2020] Mengdi Xu, Wenhao Ding, Jiacheng Zhu, Zuxin Liu, Baiming Chen, and Ding Zhao. Task-agnostic online reinforcement learning with an infinite mixture of gaussian processes. Advances in Neural Information Pro- cessing Systems, 33:6429-6440, 2020.","title":"Task-agnostic online reinforcement learning with an infinite mixture of gaussian processes","authors":"Xu","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2021] Mengdi Xu, Peide Huang, Fengpei Li, Ji- acheng Zhu, Xuewei Qi, Kentaro Oguchi, Zhiyuan Huang, Henry Lam, and Ding Zhao. Accelerated policy evaluation: Learning adversarial environments with adaptive impor- tance sampling. arXiv preprint arXiv:2106.10566, 2021.","title":"Accelerated policy evaluation: Learning adversarial environments with adaptive importance sampling","authors":"Xu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yu et al., 2021] Jing Yu, Clement Gehring, Florian Schäfer, and Animashree Anandkumar. Robust reinforcement learn- ing: A constrained game-theoretic approach. In Learning for Dynamics and Control, pages 1242-1254. PMLR, 2021.","title":"Robust reinforcement learning: A constrained game-theoretic approach","authors":"Jing Yu, Clement Gehring, Florian Schäfer, Animashree Anandkumar","year":"2021","venue":"Learning for Dynamics and Control","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2019] K Zhang, Z Yang, and T Başar. Multi- agent reinforcement learning: A selective overview of the- ories and algorithms. arxiv e-prints, page. arXiv preprint arXiv:1911.10635, 2019.","title":"Multiagent reinforcement learning: A selective overview of theories and algorithms","authors":"Z Zhang, Yang, Başar","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2020] Kaiqing Zhang, Bin Hu, and Tamer Basar. On the stability and convergence of robust adversar- ial reinforcement learning: A case study on linear quadratic systems. Advances in Neural Information Processing Sys- tems, 33, 2020.","title":"On the stability and convergence of robust adversarial reinforcement learning: A case study on linear quadratic systems","authors":"Zhang","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2021] Guodong Zhang, Yuanhao Wang, Lau- rent Lessard, and Roger Grosse. Don't fix what ain't broke: Near-optimal local convergence of alternating gradient descent-ascent for minimax optimization. arXiv preprint arXiv:2102.09468, 2021.","title":"Don't fix what ain't broke: Near-optimal local convergence of alternating gradient descent-ascent for minimax optimization","authors":"Guodong Zhang, Yuanhao Wang, Laurent Lessard, Roger Grosse","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhao et al., 2020] Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforce- ment learning for robotics: a survey. In 2020 IEEE Sympo- sium Series on Computational Intelligence (SSCI), pages 737-744. IEEE, 2020.","title":"Sim-to-real transfer in deep reinforcement learning for robotics: a survey","authors":"Zhao","year":"2020","venue":"IEEE Symposium Series on Computational Intelligence (SSCI)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}