{"type":"ScholarlyArticle","title":"A Survey of Constraint Formulations in Safe Reinforcement Learning","authors":[{"name":"Akifumi Wachi","slug":"akifumi-wachi","given":"Akifumi","family":"Wachi","affiliation":""},{"name":"Xun Shen","slug":"xun-shen","given":"Xun","family":"Shen","affiliation":"Osaka University"},{"name":"Yanan Sui","slug":"yanan-sui","given":"Yanan","family":"Sui","affiliation":"Tsinghua University"}],"abstract":"Safety is critical when applying reinforcement learning (RL) to real-world problems. As a result, safe RL has emerged as a fundamental and powerful paradigm for optimizing an agent’s policy while incorporating notions of safety. A prevalent safe RL approach is based on a constrained criterion, which seeks to maximize the expected cumulative reward subject to specific safety constraints. Despite recent effort to enhance safety in RL, a systematic understanding of the field remains difficult. This challenge stems from the diversity of constraint representations and little exploration of their interrelations. To bridge this knowledge gap, we present a comprehensive review of representative constraint formulations, along with a curated selection of algorithms designed specifically for each formulation. In addition, we elucidate the theoretical underpinnings that reveal the mathematical mutual relations among common problem formulations. We conclude with a discussion of the current state and future directions of safe reinforcement learning research","doi":"10.24963/ijcai.2024/913","url":"https://ijcai.org/proceedings/2024/913","pdf":"https://ijcai.org/proceedings/2024/0913.pdf","markdown":"https://ijcai.org/proceedings/2024/913.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/913","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Survey Track","track":"Survey Track","number":913,"pages":{"first":8262,"last":8271},"keywords":["Planning and Scheduling: PS: Markov decisions processes","Machine Learning: ML: Reinforcement learning","AI Ethics, Trust, Fairness: ETF: Safety and robustness"],"videos":[],"references":[{"position":1,"raw":"[Achiam et al., 2017] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In ICML, 2017.","title":"Constrained policy optimization","authors":"Achiam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Afsar et al., 2022] M Mehdi Afsar, Trafford Crump, and Behrouz Far. Reinforcement learning based recommender systems: A sur- vey. ACM Computing Surveys, 55(7):1-38, 2022.","title":"Reinforcement learning based recommender systems: A survey","authors":"Trafford Mehdi Afsar, Behrouz Crump, Far","year":"2022","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Alshiekh et al., 2018] Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. In AAAI, 2018.","title":"Safe reinforcement learning via shielding","authors":"Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, Ufuk Topcu","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Altman, 1999] Eitan Altman. Constrained Markov decision pro- cesses. CRC Press, 1999.","title":"Constrained Markov decision processes","authors":"Eitan Altman","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Amani and Yang, 2022] Sanae Amani and Lin F Yang. Doubly pessimistic algorithms for strictly safe off-policy optimization. In CISS, 2022.","title":"Doubly pessimistic algorithms for strictly safe off-policy optimization","authors":"Sanae Amani, Lin F Yang","year":"2022","venue":"CISS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Amani et al., 2021] Sanae Amani, Christos Thrampoulidis, and Lin Yang. Safe reinforcement learning with linear function ap- proximation. In ICML, 2021.","title":"Safe reinforcement learning with linear function approximation","authors":"Sanae Amani, Christos Thrampoulidis, Lin Yang","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ames et al., 2016] Aaron D Ames, Xiangru Xu, Jessy W Grizzle, and Paulo Tabuada. Control barrier function based quadratic pro- grams for safety critical systems. IEEE Transactions on Auto- matic Control, 62(8):3861-3876, 2016.","title":"Control barrier function based quadratic programs for safety critical systems","authors":"Ames","year":"2016","venue":"IEEE Transactions on Automatic Control","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Amodei et al., 2016] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete prob- lems in AI safety. arXiv preprint arXiv:1606.06565, 2016.","title":"Concrete problems in AI safety","authors":"Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[As et al., 2021] Yarden As, Ilnura Usmanova, Sebastian Curi, and Andreas Krause. Constrained policy optimization via Bayesian world models. In ICLR, 2021.","title":"Constrained policy optimization via Bayesian world models","authors":"Yarden As, Ilnura Usmanova, Sebastian Curi, Andreas Krause","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Bai et al., 2022] Qinbo Bai, Amrit Singh Bedi, Mridul Agarwal, Alec Koppel, and Vaneet Aggarwal. Achieving zero constraint violation for constrained reinforcement learning via primal-dual approach. In AAAI, 2022.","title":"Achieving zero constraint violation for constrained reinforcement learning via primal-dual approach","authors":"Bai","year":"2022","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Berkenkamp et al., 2017] Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based rein- forcement learning with stability guarantees. In NeurIPS, 2017.","title":"Safe model-based reinforcement learning with stability guarantees","authors":"Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, Andreas Krause","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Bharadhwaj et al., 2021] Homanga Bharadhwaj, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Ani- mesh Garg. Conservative safety critics for exploration. In ICLR, 2021.","title":"Conservative safety critics for exploration","authors":"Bharadhwaj","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Borkar, 2002] Vivek S Borkar. Q-learning for risk-sensitive con- trol. Mathematics of operations research, 27(2):294-311, 2002.","title":"Q-learning for risk-sensitive control","authors":"S Vivek, Borkar","year":"2002","venue":"Mathematics of operations research","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Brunke et al., 2021] Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From learning-based con- trol to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5, 2021.","title":"Safe learning in robotics: From learning-based control to safe reinforcement learning","authors":"Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, Angela P Schoellig","year":"2021","venue":"Annual Review of Control, Robotics, and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Bura et al., 2022] Archana Bura, Aria HasanzadeZonuzy, Dileep Kalathil, Srinivas Shakkottai, and Jean-Francois Chamberland. DOPE: Doubly optimistic and pessimistic exploration for safe reinforcement learning. In NeurIPS, 2022.","title":"DOPE: Doubly optimistic and pessimistic exploration for safe reinforcement learning","authors":"Archana Bura, Aria Hasanzadezonuzy, Dileep Kalathil, Srinivas Shakkottai, Jean-Francois Chamberland","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Cheng et al., 2019] Richard Cheng, Gábor Orosz, Richard M Mur- ray, and Joel W Burdick. End-to-end safe reinforcement learn- ing through barrier functions for safety-critical continuous con- trol tasks. In AAAI, 2019.","title":"End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks","authors":"Richard Cheng, Gábor Orosz, Richard M Murray, Joel W Burdick","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Dai et al., 2023] Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. In ICLR, 2023.","title":"Safe RLHF: Safe reinforcement learning from human feedback","authors":"Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, Yaodong Yang","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Dalal et al., 2018] Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.08757, 2018.","title":"Safe exploration in continuous action spaces","authors":"Dalal","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ding et al., 2020] Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovic. Natural policy gradient primal-dual method for constrained Markov decision processes. In NeurIPS, 2020.","title":"Natural policy gradient primal-dual method for constrained Markov decision processes","authors":"Dongsheng Ding, Kaiqing Zhang, Tamer Basar, Mihailo Jovanovic","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ding et al., 2021] Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, and Mihailo Jovanovic. Provably efficient safe exploration via primal-dual policy optimization. In AISTAT, 2021.","title":"Provably efficient safe exploration via primal-dual policy optimization","authors":"Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, Mihailo Jovanovic","year":"2021","venue":"AISTAT","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Fisac et al., 2018] Jaime F Fisac, Anayo K Akametalu, Melanie N Zeilinger, Shahab Kaynama, Jeremy Gillula, and Claire J Tomlin. A general safety framework for learning-based control in uncer- tain robotic systems. IEEE Transactions on Automatic Control, 64(7):2737-2752, 2018.","title":"A general safety framework for learning-based control in uncertain robotic systems","authors":"Jaime F Fisac, K Anayo, Melanie N Akametalu, Shahab Zeilinger, Jeremy Kaynama, Claire J Gillula, Tomlin","year":"2018","venue":"IEEE Transactions on Automatic Control","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Fulton and Platzer, 2018] Nathan Fulton and André Platzer. Safe reinforcement learning via formal methods: Toward safe control through proof and learning. In AAAI, 2018.","title":"Safe reinforcement learning via formal methods: Toward safe control through proof and learning","authors":"Fulton, Platzer, Nathan Fulton, André Platzer","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Garcıa and Fernández, 2015] Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning. JMLR, 16(1):1437-1480, 2015.","title":"Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning","authors":"Fernández Garcıa","year":"2015","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Geibel, 2006] Peter Geibel. Reinforcement learning for MDPs with constraints. In ECML, 2006.","title":"Reinforcement learning for MDPs with constraints","authors":"Peter Geibel","year":"2006","venue":"ECML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Gu et al., 2022] Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and ap- plications. arXiv preprint arXiv:2205.10330, 2022.","title":"A review of safe reinforcement learning: Methods, theory and applications","authors":"Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, Alois Knoll","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Hambly et al., 2023] Ben Hambly, Renyuan Xu, and Huining Yang. Recent advances in reinforcement learning in finance. Mathematical Finance, 33(3):437-503, 2023.","title":"Recent advances in reinforcement learning in finance","authors":"Ben Hambly, Renyuan Xu, Huining Yang","year":"2023","venue":"Mathematical Finance","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Hasanbeig et al., 2020] Mohammadhosein Hasanbeig, Alessandro Abate, and Daniel Kroening. Cautious reinforcement learning with logical constraints. In AAMAS, 2020.","title":"Cautious reinforcement learning with logical constraints","authors":"Hasanbeig","year":"2020","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Heger, 1994] Matthias Heger. Consideration of risk in reinforce- ment learning. In ICML, 1994.","title":"Consideration of risk in reinforcement learning","authors":"Matthias Heger","year":"1994","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Ji et al., 2023] Jiaming Ji, Jiayi Zhou, Borong Zhang, et al. Om- niSafe: An infrastructure for accelerating safe reinforcement learning research. arXiv preprint arXiv:2305.09304, 2023.","title":"Om-niSafe: An infrastructure for accelerating safe reinforcement learning research","authors":"Jiaming Ji, Jiayi Zhou, Borong Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Jia et al., 2020] Yan Jia, John Burden, Tom Lawton, et al. Safe reinforcement learning for sepsis treatment. In IEEE ICHI, 2020.","title":"Safe reinforcement learning for sepsis treatment","authors":"Yan Jia, John Burden, Tom Lawton","year":"2020","venue":"IEEE ICHI","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Kim et al., 2020] Youngmin Kim, Richard Allmendinger, and Manuel López-Ibáñez. Safe learning and optimization tech- niques: Towards a survey of the state of the art. In TAILOR, 2020.","title":"Safe learning and optimization techniques: Towards a survey of the state of the art","authors":"Youngmin Kim, Richard Allmendinger, Manuel López-Ibáñez","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Kordabad et al., 2022] Arash Bahari Kordabad, Rafael Wis- niewski, and Sebastien Gros. Safe reinforcement learning using Wasserstein distributionally robust MPC and chance constraint. IEEE Access, 10:130058-130067, 2022.","title":"Safe reinforcement learning using Wasserstein distributionally robust MPC and chance constraint","authors":"Kordabad","year":"2022","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Le et al., 2019] Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In ICML, 2019.","title":"Batch policy learning under constraints","authors":"Hoang Le, Cameron Voloshin, Yisong Yue","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Lee et al., 2021] Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim, and Arthur Guez. COptiDICE: Offline constrained reinforcement learning via stationary distribution correction estimation. In ICLR, 2021.","title":"COptiDICE: Offline constrained reinforcement learning via stationary distribution correction estimation","authors":"Jongmin Lee, Cosmin Paduraru, J Daniel, Nicolas Mankowitz, Doina Heess, Kee-Eung Precup, Arthur Kim, Guez","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Levine et al., 2016] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor poli- cies. JMLR, 17(1):1334-1373, 2016.","title":"End-to-end training of deep visuomotor policies","authors":"Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel","year":"2016","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Levine et al., 2020] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, re- view, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.","title":"Offline reinforcement learning: Tutorial, review, and perspectives on open problems","authors":"Sergey Levine, Aviral Kumar, George Tucker, Justin Fu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Li and Bastani, 2020] Shuo Li and Osbert Bastani. Robust model predictive shielding for safe reinforcement learning with stochas- tic dynamics. In ICRA, 2020.","title":"Robust model predictive shielding for safe reinforcement learning with stochastic dynamics","authors":"Bastani ; ] Shuo Li, Osbert Li, Bastani","year":"2020","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Li et al., 2019] Yuanlong Li, Yonggang Wen, Dacheng Tao, and Kyle Guan. Transforming cooling optimization for green data center via deep reinforcement learning. IEEE transactions on cybernetics, 50(5):2002-2013, 2019.","title":"Transforming cooling optimization for green data center via deep reinforcement learning","authors":"Li","year":"2019","venue":"IEEE transactions on cybernetics","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Liu et al., 2020] Yongshuai Liu, Jiaxin Ding, and Xin Liu. IPO: Interior-point policy optimization under constraints. In AAAI, 2020.","title":"IPO: Interior-point policy optimization under constraints","authors":"Yongshuai Liu, Jiaxin Ding, Xin Liu","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Liu et al., 2021] Yongshuai Liu, Avishai Halev, and Xin Liu. Pol- icy learning with constraints in model-free reinforcement learn- ing: A survey. In IJCAI, 2021.","title":"Policy learning with constraints in model-free reinforcement learning: A survey","authors":"Liu","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/614"},{"position":41,"raw":"[Liu et al., 2022] Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao. Constrained variational policy optimization for safe reinforcement learning. In ICML, 2022.","title":"Constrained variational policy optimization for safe reinforcement learning","authors":"Liu","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Liu et al., 2023a] Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, et al. Datasets and bench- marks for offline safe reinforcement learning. arXiv preprint arXiv:2306.09303, 2023.","title":"Datasets and benchmarks for offline safe reinforcement learning","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Liu et al., 2023b] Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. In ICML, 2023.","title":"Constrained decision transformer for offline safe reinforcement learning","authors":"Liu","year":"2023","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Meng and Khushi, 2019] Terry Lingze Meng and Matloob Khushi. Reinforcement learning in financial markets. Data, 4(3):110, 2019.","title":"Reinforcement learning in financial markets","authors":"Khushi Meng","year":"2019","venue":"Data","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Moldovan and Abbeel, 2012] Teodor Mihai Moldovan and Pieter Abbeel. Safe exploration in markov decision processes. In ICML, 2012.","title":"Safe exploration in markov decision processes","authors":"Abbeel Moldovan, Teodor Mihai Moldovan, Pieter Abbeel","year":"2012","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Mowbray et al., 2022] Max Mowbray, Panagiotis Petsagkourakis, Ehecatl Antonio del Rio-Chanona, and Dongda Zhang. Safe chance constrained reinforcement learning for batch process con- trol. Computers & chemical engineering, 157:107630, 2022.","title":"Safe chance constrained reinforcement learning for batch process control","authors":"Mowbray","year":"2022","venue":"Computers & chemical engineering","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Ono et al., 2015] Masahiro Ono, Marco Pavone, Yoshiaki Kuwata, and J Balaram. Chance-constrained dynamic programming with application to risk-aware robotic space exploration. Autonomous Robots, 39(4):555-571, 2015.","title":"Chance-constrained dynamic programming with application to risk-aware robotic space exploration","authors":"Masahiro Ono, Marco Pavone, Yoshiaki Kuwata, Balaram","year":"2015","venue":"Autonomous Robots","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Ouyang et al., 2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, et al. Training language models to follow instructions with human feedback. In NeurIPS, 2022.","title":"Training language models to follow instructions with human feedback","authors":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Pfrommer et al., 2022] Samuel Pfrommer, Tanmay Gautam, Alec Zhou, and Somayeh Sojoudi. Safe reinforcement learning with chance-constrained model predictive control. In L4DC, 2022.","title":"Safe reinforcement learning with chance-constrained model predictive control","authors":"Pfrommer","year":"2022","venue":"L4DC","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Pham et al., 2018] Tu-Hoa Pham, Giovanni De Magistris, and Ryuki Tachibana. Optlayer-practical constrained optimization for deep reinforcement learning in the real world. In IEEE ICRA, 2018.","title":"Optlayer-practical constrained optimization for deep reinforcement learning in the real world","authors":"Tu-Hoa Pham, Giovanni De Magistris, Ryuki Tachibana","year":"2018","venue":"IEEE ICRA","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Ray et al., 2019] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708, 2019.","title":"Benchmarking safe exploration in deep reinforcement learning","authors":"Alex Ray, Joshua Achiam, Dario Amodei","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Shalev-Shwartz et al., 2016] Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, rein- forcement learning for autonomous driving. arXiv preprint arXiv:1610.03295, 2016.","title":"Safe, multi-agent, reinforcement learning for autonomous driving","authors":"Shai Shalev-Shwartz, Shaked Shammah, Amnon Shashua","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Sharpe, 1966] William F Sharpe. Mutual fund performance. The Journal of business, 39(1):119-138, 1966.","title":"Mutual fund performance","authors":"F William, Sharpe","year":"1966","venue":"The Journal of business","doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Shi et al., 2023] Ming Shi, Yingbin Liang, and Ness Shroff. A near-optimal algorithm for safe reinforcement learning under in- stantaneous hard constraints. In ICML, 2023.","title":"A near-optimal algorithm for safe reinforcement learning under instantaneous hard constraints","authors":"Ming Shi, Yingbin Liang, Ness Shroff","year":"2023","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":57,"raw":"[Sootla et al., 2022a] Aivar Sootla, Alexander Cowen-Rivers, Jun Wang, and Haitham Bou Ammar. Enhancing safe exploration using safety state augmentation. In NeurIPS, 2022.","title":"Enhancing safe exploration using safety state augmentation","authors":"Sootla","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":58,"raw":"[Sootla et al., 2022b] Aivar Sootla, Alexander I Cowen-Rivers, Taher Jafferjee, Ziyan Wang, David H Mguni, Jun Wang, and Haitham Ammar. Sauté RL: Almost surely safe reinforcement learning using state augmentation. In ICML, 2022.","title":"Sauté RL: Almost surely safe reinforcement learning using state augmentation","authors":"Sootla","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":59,"raw":"[Stooke et al., 2020] Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by PID La- grangian methods. In ICML, 2020.","title":"Responsive safety in reinforcement learning by PID Lagrangian methods","authors":"Adam Stooke, Joshua Achiam, Pieter Abbeel","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":60,"raw":"[Sui et al., 2015] Yanan Sui, Alkis Gotovos, Joel Burdick, and An- dreas Krause. Safe exploration for optimization with Gaussian processes. In ICML, 2015.","title":"Safe exploration for optimization with Gaussian processes","authors":"Sui","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":61,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 1998.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":62,"raw":"[Tamar et al., 2012] Aviv Tamar, Dotan Di Castro, and Shie Man- nor. Policy gradients with variance related risk criteria. In ICML, 2012.","title":"Policy gradients with variance related risk criteria","authors":"Tamar","year":"2012","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":63,"raw":"[Tessler et al., 2019] Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. In ICLR, 2019.","title":"Reward constrained policy optimization","authors":"Chen Tessler, Daniel J Mankowitz, Shie Mannor","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":64,"raw":"[Thananjeyan et al., 2021] Brijen Thananjeyan, Ashwin Balakr- ishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery RL: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 6(3):4915-4922, 2021.","title":"Recovery RL: Safe reinforcement learning with learned recovery zones","authors":"Thananjeyan","year":"2021","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":65,"raw":"[Thomas et al., 2021] Garrett Thomas, Yuping Luo, and Tengyu Ma. Safe reinforcement learning by imagining the near future. In NeurIPS, 2021.","title":"Safe reinforcement learning by imagining the near future","authors":"Garrett Thomas, Yuping Luo, Tengyu Ma","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":66,"raw":"[Turchetta et al., 2016] Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. Safe exploration in finite Markov decision pro- cesses with Gaussian processes. In NeurIPS, 2016.","title":"Safe exploration in finite Markov decision processes with Gaussian processes","authors":"Matteo Turchetta, Felix Berkenkamp, Andreas Krause","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":67,"raw":"[Turchetta et al., 2020] Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, and Alekh Agarwal. Safe reinforcement learning via curriculum induction. In NeurIPS, 2020.","title":"Safe reinforcement learning via curriculum induction","authors":"Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, Alekh Agarwal","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":68,"raw":"[Wachi and Sui, 2020] Akifumi Wachi and Yanan Sui. Safe rein- forcement learning in constrained Markov decision processes. In ICML, 2020.","title":"Safe reinforcement learning in constrained Markov decision processes","authors":"Sui ; ] Akifumi Wachi, Yanan Wachi, Sui","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":69,"raw":"[Wachi et al., 2018] Akifumi Wachi, Yanan Sui, Yisong Yue, and Masahiro Ono. Safe exploration and optimization of constrained MDPs using Gaussian processes. In AAAI, 2018.","title":"Safe exploration and optimization of constrained MDPs using Gaussian processes","authors":"Akifumi Wachi, Yanan Sui, Yisong Yue, Masahiro Ono","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":70,"raw":"[Wachi et al., 2023] Akifumi Wachi, Wataru Hashimoto, Xun Shen, and Kazumune Hashimoto. Safe exploration in reinforce- ment learning: A generalized formulation and algorithms. In NeurIPS, 2023.","title":"Safe exploration in reinforcement learning: A generalized formulation and algorithms","authors":"Akifumi Wachi, Wataru Hashimoto, Xun Shen, Kazumune Hashimoto","year":"2023","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":71,"raw":"[Wang et al., 2023] Yixuan Wang, Simon Sinong Zhan, Ruochen Jiao, Zhilu Wang, Wanxin Jin, Zhuoran Yang, Zhaoran Wang, Chao Huang, and Qi Zhu. Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic en- vironments. In ICML, 2023.","title":"Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments","authors":"Yixuan Wang, Simon Sinong Zhan, Ruochen Jiao, Zhilu Wang, Wanxin Jin, Zhuoran Yang, Zhaoran Wang, Chao Huang, Qi Zhu","year":"2023","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":72,"raw":"[Wu et al., 2021] Runzhe Wu, Yufeng Zhang, Zhuoran Yang, and Zhaoran Wang. Offline constrained multi-objective reinforce- ment learning via pessimistic dual value iteration. In NeurIPS, 2021.","title":"Offline constrained multi-objective reinforcement learning via pessimistic dual value iteration","authors":"Runzhe Wu, Yufeng Zhang, Zhuoran Yang, Zhaoran Wang","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":73,"raw":"[Wurman et al., 2022] Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 602(7896):223-228, 2022.","title":"Outracing champion gran turismo drivers with deep reinforcement learning","authors":"Samuel Peter R Wurman, Kenta Barrett, James Kawamoto, Kaushik Macglashan, Thomas J Subramanian, Roberto Walsh, Alisa Capobianco, Franziska Devlic, Eckert","year":"2022","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":74,"raw":"[Xu et al., 2021] Tengyu Xu, Yingbin Liang, and Guanghui Lan. CRPO: A new approach for safe reinforcement learning with con- vergence guarantee. In ICML, 2021.","title":"CRPO: A new approach for safe reinforcement learning with convergence guarantee","authors":"Xu","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":75,"raw":"[Xu et al., 2022] Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized Q-learning for safe offline reinforcement learning. In AAAI, 2022.","title":"Constraints penalized Q-learning for safe offline reinforcement learning","authors":"Haoran Xu, Xianyuan Zhan, Xiangyu Zhu","year":"2022","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":76,"raw":"[Yang et al., 2020] Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-based constrained policy optimization. In ICLR, 2020.","title":"Projection-based constrained policy optimization","authors":"Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, Peter J Ramadge","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":77,"raw":"[Yang et al., 2021] Tsung-Yen Yang, Michael Y Hu, Yinlam Chow, Peter J Ramadge, and Karthik Narasimhan. Safe reinforcement learning with natural language constraints. In NeurIPS, 2021.","title":"Safe reinforcement learning with natural language constraints","authors":"Tsung-Yen Yang, Michael Y Hu, Yinlam Chow, Peter J Ramadge, Karthik Narasimhan","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":78,"raw":"[Yu et al., 2021] Chao Yu, Jiming Liu, Shamim Nemati, and Gu- osheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1-36, 2021.","title":"Reinforcement learning in healthcare: A survey","authors":"Chao Yu, Jiming Liu, Shamim Nemati, Guosheng Yin","year":"2021","venue":"ACM Computing Surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":79,"raw":"[Yu et al., 2022] Haonan Yu, Wei Xu, and Haichao Zhang. To- wards safe reinforcement learning with a safety editor policy. In NeurIPS, 2022.","title":"Towards safe reinforcement learning with a safety editor policy","authors":"Haonan Yu, Wei Xu, Haichao Zhang","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":80,"raw":"[Zhang et al., 2020] Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. In NeurIPS, 2020.","title":"First order constrained optimization in policy space","authors":"Yiming Zhang, Quan Vuong, Keith Ross","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":81,"raw":"[Zhang et al., 2022] Linrui Zhang, Li Shen, Long Yang, Shixiang Chen, Bo Yuan, Xueqian Wang, and Dacheng Tao. Penalized proximal policy optimization for safe reinforcement learning. In IJCAI, 2022.","title":"Penalized proximal policy optimization for safe reinforcement learning","authors":"Linrui Zhang, Li Shen, Long Yang, Shixiang Chen, Bo Yuan, Xueqian Wang, Dacheng Tao","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/520"},{"position":82,"raw":"[Zhao et al., 2023a] Weiye Zhao, Rui Chen, Yifan Sun, Tianhao Wei, and Changliu Liu. State-wise constrained policy optimiza- tion. arXiv preprint arXiv:2306.12594, 2023.","title":"State-wise constrained policy optimization","authors":"Zhao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":83,"raw":"[Zhao et al., 2023b] Weiye Zhao, Tairan He, Rui Chen, Tianhao Wei, and Changliu Liu. State-wise safe reinforcement learning: A survey. In IJCAI, 2023.","title":"State-wise safe reinforcement learning: A survey","authors":"Zhao","year":"2023","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2023/763"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}