{"type":"ScholarlyArticle","title":"Exterior Penalty Policy Optimization with Penalty Metric Network under Constraints","authors":[{"name":"Shiqing Gao","slug":"shiqing-gao","given":"Shiqing","family":"Gao","affiliation":"University"},{"name":"Jiaxin Ding","slug":"jiaxin-ding","given":"Jiaxin","family":"Ding","affiliation":"University"},{"name":"Luoyi Fu","slug":"luoyi-fu","given":"Luoyi","family":"Fu","affiliation":"University"},{"name":"Xinbing Wang","slug":"xinbing-wang","given":"Xinbing","family":"Wang","affiliation":"University"},{"name":"Chenghu Zhou","slug":"chenghu-zhou","given":"Chenghu","family":"Zhou","affiliation":"University"}],"abstract":"In Constrained Reinforcement Learning (CRL), agents explore the environment to learn the optimal policy while satisfying constraints. The penalty function method has recently been studied as an effective approach for handling constraints, which imposes constraints penalties on the objective to transform the constrained problem into an unconstrained one. However, it is challenging to choose appropriate penalties that balance policy performance and constraint satisfaction efficiently. In this paper, we propose a theoretically guaranteed penalty function method, Exterior Penalty Policy Optimization (EPO), with adaptive penalties generated by a Penalty Metric Network (PMN). PMN responds appropriately to varying degrees of constraint violations, enabling efficient constraint satisfaction and safe exploration. We theoretically prove that EPO consistently improves constraint satisfaction with a convergence guarantee. We propose a new surrogate function and provide worst-case constraint violation and approximation error. In practice, we propose an effective smooth penalty function, which can be easily implemented with a first-order optimizer. Extensive experiments are conducted, showing that EPO outperforms the baselines in terms of policy performance and constraint satisfaction with a stable training process, particularly on complex tasks.","doi":"10.24963/ijcai.2024/443","url":"https://ijcai.org/proceedings/2024/443","pdf":"https://ijcai.org/proceedings/2024/0443.pdf","markdown":"https://ijcai.org/proceedings/2024/443.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/443","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":443,"pages":{"first":4007,"last":4015},"keywords":["Constraint Satisfaction and Optimization: CSO: Constraint optimization problems","Machine Learning: ML: Theory of deep learning","Machine Learning: ML: Reinforcement learning","Machine Learning: ML: Optimization"],"videos":[],"references":[{"position":1,"raw":"[Achiam et al., 2017] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimiza- tion. In International conference on machine learning, pages 22-31, 2017.","title":"Constrained policy optimization","authors":"Achiam","year":"2017","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Altman, 2021] Eitan Altman. Constrained Markov decision processes. Routledge, 2021.","title":"Constrained Markov decision processes","authors":"Eitan Altman","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Beutler and Ross, 1985] Frederick J Beutler and Keith W Ross. Optimal policies for controlled markov chains with a constraint. Journal of mathematical analysis and appli- cations, 112(1):236-252, 1985.","title":"Optimal policies for controlled markov chains with a constraint","authors":"J Frederick, Keith W Beutler, Ross","year":"1985","venue":"Journal of mathematical analysis and applications","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Beutler and Ross, 1986] Frederick J Beutler and Keith W Ross. Time-average optimal constrained semi-markov decision processes. Advances in Applied Probability, 18(2):341-359, 1986.","title":"Time-average optimal constrained semi-markov decision processes","authors":"J Frederick, Keith W Beutler, Ross","year":"1986","venue":"Advances in Applied Probability","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chow et al., 2017] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18(1):6070-6120, 2017.","title":"Risk-constrained reinforcement learning with percentile risk criteria","authors":"Chow","year":"2017","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chow et al., 2018] Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. A lyapunov-based approach to safe reinforcement learning. Advances in neural information processing systems, 31, 2018.","title":"A lyapunov-based approach to safe reinforcement learning","authors":"Chow","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chow et al., 2019] Yinlam Chow, Ofir Nachum, Alek- sandra Faust, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. Lyapunov-based safe policy optimization for continuous control. arXiv preprint arXiv:1901.10031, 2019.","title":"Lyapunov-based safe policy optimization for continuous control","authors":"Chow","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dai et al., 2023] Juntao Dai, Jiaming Ji, Long Yang, Qian Zheng, and Gang Pan. Augmented proximal policy op- timization for safe reinforcement learning. In Proceed- ings of the AAAI Conference on Artificial Intelligence, vol- ume 37, pages 7288-7295, 2023.","title":"Augmented proximal policy optimization for safe reinforcement learning","authors":"Juntao Dai, Jiaming Ji, Long Yang, Qian Zheng, Gang Pan","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dalal et al., 2018] Gal Dalal, Balázs Szörényi, Gugan Thoppe, and Shie Mannor. Finite sample analyses for td (0) with function approximation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.","title":"Finite sample analyses for td (0) with function approximation","authors":"Dalal","year":"2018","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ding et al., 2020] Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovic. Natural policy gradient primal-dual method for constrained markov decision pro- cesses. Advances in Neural Information Processing Sys- tems, 33:8378-8390, 2020.","title":"Natural policy gradient primal-dual method for constrained markov decision processes","authors":"Dongsheng Ding, Kaiqing Zhang, Tamer Basar, Mihailo Jovanovic","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ding et al., 2021] Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, and Mihailo Jovanovic. Provably efficient safe exploration via primal-dual policy optimiza- tion. In International Conference on Artificial Intelligence and Statistics, pages 3304-3312, 2021.","title":"Provably efficient safe exploration via primal-dual policy optimization","authors":"Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, Mihailo Jovanovic","year":"2021","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gu et al., 2016] Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International confer- ence on machine learning, pages 2829-2838, 2016.","title":"Continuous deep q-learning with model-based acceleration","authors":"Gu","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on ma- chine learning, pages 1861-1870, 2018.","title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","authors":"Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kakade and Langford, 2002] Sham Kakade and John Lang- ford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pages 267-274, 2002.","title":"Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning","authors":"Langford Kakade","year":"2002","venue":"Proceedings of the Nineteenth International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Levine et al., 2016] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1373, 2016.","title":"End-to-end training of deep visuomotor policies","authors":"Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel","year":"2016","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2020] Yongshuai Liu, Jiaxin Ding, and Xin Liu. Ipo: Interior-point policy optimization under constraints. In Proceedings of the AAAI conference on artificial intel- ligence, volume 34, pages 4940-4947, 2020.","title":"Ipo: Interior-point policy optimization under constraints","authors":"Liu","year":"2020","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In In- ternational conference on machine learning, pages 1928- 1937, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Paternain et al., 2019] Santiago Paternain, Luiz Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Con- strained reinforcement learning has zero duality gap. Ad- vances in Neural Information Processing Systems, 32, 2019.","title":"Constrained reinforcement learning has zero duality gap","authors":"Paternain","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Peters and Schaal, 2008] Jan Peters and Stefan Schaal. Re- inforcement learning of motor skills with policy gradients. Neural networks, 21(4):682-697, 2008.","title":"Reinforcement learning of motor skills with policy gradients","authors":"Schaal Peters, Jan Peters, Stefan Schaal","year":"2008","venue":"Neural networks","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ross and Varadarajan, 1989] Keith W Ross and Ravi Varadarajan. Markov decision processes with sample path constraints: the communicating case. Operations Research, 37(5):780-790, 1989.","title":"Markov decision processes with sample path constraints: the communicating case","authors":"W Keith, Ravi Ross, Varadarajan","year":"1989","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ross and Varadarajan, 1991] Keith W Ross and Ravi Varadarajan. Multichain markov decision processes with a sample path constraint: A decomposition approach. Mathematics of Operations Research, 16(1):195-207, 1991.","title":"Multichain markov decision processes with a sample path constraint: A decomposition approach","authors":"W Keith, Ravi Ross, Varadarajan","year":"1991","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ross, 1989] Keith W Ross. Randomized and past- dependent policies for markov decision processes with multiple constraints. Operations Research, 37(3):474- 477, 1989.","title":"Randomized and pastdependent policies for markov decision processes with multiple constraints","authors":"Ross Keith","year":"1989","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schulman et al., 2015] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advan- tage estimation. arXiv preprint arXiv:1506.02438, 2015.","title":"Highdimensional continuous control using generalized advantage estimation","authors":"John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-359, 2017.","title":"Mastering the game of go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton","year":"2017","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Stooke et al., 2020] Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learn- ing by pid lagrangian methods. In International Confer- ence on Machine Learning, pages 9133-9143, 2020.","title":"Responsive safety in reinforcement learning by pid lagrangian methods","authors":"Stooke","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tessler et al., 2018] Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv:1805.11074, 2018.","title":"Reward constrained policy optimization","authors":"Chen Tessler, Daniel J Mankowitz, Shie Mannor","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"IEEE/RSJ international conference on intelligent robots and systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Vinyals et al., 2019] Oriol Vinyals, Igor Babuschkin, Woj- ciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in star- craft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.","title":"Grandmaster level in starcraft ii using multi-agent reinforcement learning","authors":"Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev","year":"2019","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wang et al., 2016] Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with expe- rience replay. arXiv preprint arXiv:1611.01224, 2016.","title":"Sample efficient actor-critic with experience replay","authors":"Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, Nando De Freitas","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yang et al., 2020] Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection- based constrained policy optimization. arXiv preprint arXiv:2010.03152, 2020.","title":"Projectionbased constrained policy optimization","authors":"Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, Peter J Ramadge","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yang et al., 2022] Long Yang, Jiaming Ji, Juntao Dai, Lin- rui Zhang, Binbin Zhou, Pengfei Li, Yaodong Yang, and Gang Pan. Constrained update projection approach to safe policy optimization. arXiv preprint arXiv:2209.07089, 2022.","title":"Constrained update projection approach to safe policy optimization","authors":"Long Yang, Jiaming Ji, Juntao Dai, Linrui Zhang, Binbin Zhou, Pengfei Li, Yaodong Yang, Gang Pan","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Yu et al., 2019] Ming Yu, Zhuoran Yang, Mladen Kolar, and Zhaoran Wang. Convergent policy optimization for safe reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.","title":"Convergent policy optimization for safe reinforcement learning","authors":"Ming Yu, Zhuoran Yang, Mladen Kolar, Zhaoran Wang","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Yu et al., 2022] Haonan Yu, Wei Xu, and Haichao Zhang. Towards safe reinforcement learning with a safety editor policy. arXiv preprint arXiv:2201.12427, 2022.","title":"Towards safe reinforcement learning with a safety editor policy","authors":"Yu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhang et al., 2020] Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Sys- tems, 33:15338-15349, 2020.","title":"First order constrained optimization in policy space","authors":"Yiming Zhang, Quan Vuong, Keith Ross","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhang et al., 2022] Linrui Zhang, Li Shen, Long Yang, Shixiang Chen, Bo Yuan, Xueqian Wang, and Dacheng Tao. Penalized proximal policy optimization for safe re- inforcement learning. arXiv preprint arXiv:2205.11814, 2022.","title":"Penalized proximal policy optimization for safe reinforcement learning","authors":"Zhang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2022/520"},{"position":40,"raw":"[Zhang et al., 2023] Linrui Zhang, Qin Zhang, Li Shen, Bo Yuan, Xueqian Wang, and Dacheng Tao. Evaluating model-free reinforcement learning toward safety-critical tasks. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 37, pages 15313-15321, 2023.","title":"Evaluating model-free reinforcement learning toward safety-critical tasks","authors":"Zhang","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}