{"type":"ScholarlyArticle","title":"Penalized Proximal Policy Optimization for Safe Reinforcement Learning","authors":[{"name":"Linrui Zhang","slug":"linrui-zhang","given":"Linrui","family":"Zhang","affiliation":"Tsinghua University"},{"name":"Li Shen","slug":"li-shen","given":"Li","family":"Shen","affiliation":"JD Explore Academy"},{"name":"Long Yang","slug":"long-yang","given":"Long","family":"Yang","affiliation":"Peking University"},{"name":"Shixiang Chen","slug":"shixiang-chen","given":"Shixiang","family":"Chen","affiliation":"JD Explore Academy"},{"name":"Xueqian Wang","slug":"xueqian-wang","given":"Xueqian","family":"Wang","affiliation":"Tsinghua University; s internship at JD Ex- plore Academy , Beijing , China."},{"name":"Bo Yuan","slug":"bo-yuan","given":"Bo","family":"Yuan","affiliation":"Tsinghua University"},{"name":"Dacheng Tao","slug":"dacheng-tao","given":"Dacheng","family":"Tao","affiliation":"JD Explore Academy"}],"abstract":"Safe reinforcement learning aims to learn the optimal policy while satisfying safety constraints, which is essential in real-world applications. However, current algorithms still struggle for efficient policy updates with hard constraint satisfaction. In this paper, we propose Penalized Proximal Policy Optimization (P3O), which solves the cumbersome constrained policy iteration via a single minimization of an equivalent unconstrained problem. Specifically, P3O utilizes a simple yet effective penalty approach to eliminate cost constraints and removes the trust-region constraint by the clipped surrogate objective. We theoretically prove the exactness of the penalized method with a finite penalty factor and provide a worst-case analysis for approximate error when evaluated on sample trajectories. Moreover, we extend P3O to more challenging multi-constraint and multi-agent scenarios which are less studied in previous work. Extensive experiments show that P3O outperforms state-of-the-art algorithms with respect to both reward improvement and constraint satisfaction on a set of constrained locomotive tasks.","doi":"10.24963/ijcai.2022/520","url":"https://ijcai.org/proceedings/2022/520","pdf":"https://ijcai.org/proceedings/2022/0520.pdf","markdown":"https://ijcai.org/proceedings/2022/520.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/520","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":520,"pages":{"first":3744,"last":3750},"keywords":["Constraint Satisfaction and Optimization: Constraint Optimization","Machine Learning: Deep Reinforcement Learning","Agent-based and Multi-agent Systems: Multi-agent Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986607","length_sec":86,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984750","length_sec":358,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Achiam et al., 2017] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimiza- tion. In International Conference on Machine Learning, pages 22-31. PMLR, 2017.","title":"Constrained policy optimization","authors":"Achiam","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Altman, 1999] Eitan Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999.","title":"Constrained Markov decision processes","authors":"Eitan Altman","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chao et al., 2021] YU Chao, A VELU, E VINITSKY, et al. The surprising effectiveness of ppo in cooperative, multi- agent games. arXiv preprint arXiv:2103.01955, 2021.","title":"The surprising effectiveness of ppo in cooperative, multiagent games","authors":"A Yu Chao, E Velu, Vinitsky","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chow et al., 2017] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18(1):6070-6120, 2017.","title":"Risk-constrained reinforcement learning with percentile risk criteria","authors":"Chow","year":"2017","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ding et al., 2020] Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo R Jovanovic. Natural policy gradient primal-dual method for constrained markov decision pro- cesses. In NeurIPS, 2020.","title":"Natural policy gradient primal-dual method for constrained markov decision processes","authors":"Dongsheng Ding, Kaiqing Zhang, Tamer Basar, Mihailo R Jovanovic","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ding et al., 2021] Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, and Mihailo Jovanovic. Provably efficient safe exploration via primal-dual policy optimiza- tion. In International Conference on Artificial Intelligence and Statistics, pages 3304-3312. PMLR, 2021.","title":"Provably efficient safe exploration via primal-dual policy optimization","authors":"Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, Mihailo Jovanovic","year":"2021","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Garcıa and Fernández, 2015] Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforce- ment learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.","title":"Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning","authors":"Fernández Garcıa","year":"2015","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jiang and Li, 2016] Nan Jiang and Lihong Li. Doubly ro- bust off-policy value evaluation for reinforcement learn- ing. In International Conference on Machine Learning, pages 652-661. PMLR, 2016.","title":"Doubly robust off-policy value evaluation for reinforcement learning","authors":"Nan Jiang, Lihong Li","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Levine et al., 2016] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1373, 2016.","title":"End-to-end training of deep visuomotor policies","authors":"Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel","year":"2016","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2020] Yongshuai Liu, Jiaxin Ding, and Xin Liu. Ipo: Interior-point policy optimization under constraints. In Proceedings of the AAAI Conference on Artificial Intel- ligence, volume 34, pages 4940-4947, 2020.","title":"Ipo: Interior-point policy optimization under constraints","authors":"Liu","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2021] Yongshuai Liu, Avishai Halev, and Xin Liu. Policy learning with constraints in model-free rein- forcement learning: A survey. In Proceedings of the Thir- tieth International Joint Conference on Artificial Intelli- gence, 2021. [Paternain et al., 2019] Santiago Paternain, Luiz FO Cha- mon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Con- strained reinforcement learning has zero duality gap. arXiv preprint arXiv:1910.13393, 2019.","title":"Policy learning with constraints in model-free reinforcement learning: A survey","authors":"Liu","year":"2019","venue":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2021/614"},{"position":13,"raw":"[Ray et al., 2019] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep rein- forcement learning. arXiv preprint arXiv:1910.01708, 7, 2019.","title":"Benchmarking safe exploration in deep reinforcement learning","authors":"Alex Ray, Joshua Achiam, Dario Amodei","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-359, 2017.","title":"Mastering the game of go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton","year":"2017","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Stooke et al., 2020] Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learn- ing by pid lagrangian methods. In International Con- ference on Machine Learning, pages 9133-9143. PMLR, 2020.","title":"Responsive safety in reinforcement learning by pid lagrangian methods","authors":"Stooke","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tessler et al., 2018] Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv:1805.11074, 2018.","title":"Reward constrained policy optimization","authors":"Chen Tessler, Daniel J Mankowitz, Shie Mannor","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vinyals et al., 2019] Oriol Vinyals, Igor Babuschkin, Woj- ciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in star- craft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.","title":"Grandmaster level in starcraft ii using multi-agent reinforcement learning","authors":"Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev","year":"2019","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2020] Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection- based constrained policy optimization. arXiv preprint arXiv:2010.03152, 2020.","title":"Projectionbased constrained policy optimization","authors":"Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, Peter J Ramadge","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2020] Yiming Zhang, Quan Vuong, and Keith W Ross. First order constrained optimization in pol- icy space. arXiv preprint arXiv:2002.06506, 2020.","title":"First order constrained optimization in policy space","authors":"Yiming Zhang, Quan Vuong, Keith W Ross","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}