{"type":"ScholarlyArticle","title":"Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Risk","authors":[{"name":"ChengYang Ying","slug":"chengyang-ying","given":"ChengYang","family":"Ying","affiliation":"Department of Computer Science & Technology , Institute for AI , BNRist Center , Tsinghua-Bosch Joint ML Center , THBI Lab , Tsinghua University ;"},{"name":"Xinning Zhou","slug":"xinning-zhou","given":"Xinning","family":"Zhou","affiliation":"Department of Computer Science & Technology , Institute for AI , BNRist Center , Tsinghua-Bosch Joint ML Center , THBI Lab , Tsinghua University ;"},{"name":"Hang Su","slug":"hang-su","given":"Hang","family":"Su","affiliation":"Department of Computer Science & Technology , Institute for AI , BNRist Center , Tsinghua-Bosch Joint ML Center , THBI Lab , Tsinghua University ;; Peng Cheng Laboratory; Tsinghua University-China Mobile Communications Group Co., Ltd. Joint Institute"},{"name":"Dong Yan","slug":"dong-yan","given":"Dong","family":"Yan","affiliation":"Department of Computer Science & Technology , Institute for AI , BNRist Center , Tsinghua-Bosch Joint ML Center , THBI Lab , Tsinghua University ;"},{"name":"Ning Chen","slug":"ning-chen","given":"Ning","family":"Chen","affiliation":"Department of Computer Science & Technology , Institute for AI , BNRist Center , Tsinghua-Bosch Joint ML Center , THBI Lab , Tsinghua University ;"},{"name":"Jun Zhu","slug":"jun-zhu","given":"Jun","family":"Zhu","affiliation":"Department of Computer Science & Technology , Institute for AI , BNRist Center , Tsinghua-Bosch Joint ML Center , THBI Lab , Tsinghua University ;; Peng Cheng Laboratory; Tsinghua University-China Mobile Communications Group Co., Ltd. Joint Institute"}],"abstract":"Though deep reinforcement learning (DRL) has obtained substantial success, it may encounter catastrophic failures due to the intrinsic uncertainty of both transition and observation. Most of the existing methods for safe reinforcement learning can only handle transition disturbance or observation disturbance since these two kinds of disturbance affect different parts of the agent;   besides, the popular worst-case return may lead to overly pessimistic policies. To address these issues, we first theoretically prove that the performance degradation under transition disturbance and observation disturbance depends on a novel metric of Value Function Range (VFR), which corresponds to the gap in the value function between the best state and the worst state. Based on the analysis, we adopt conditional value-at-risk (CVaR) as an assessment of risk and propose a novel reinforcement learning algorithm of CVaR-Proximal-Policy-Optimization (CPPO) which formalizes the risk-sensitive constrained optimization problem by keeping its CVaR under a given threshold. Experimental results show that CPPO achieves a higher cumulative reward and is more robust against both observation and transition disturbances on a series of continuous control tasks in MuJoCo.","doi":"10.24963/ijcai.2022/510","url":"https://ijcai.org/proceedings/2022/510","pdf":"https://ijcai.org/proceedings/2022/0510.pdf","markdown":"https://ijcai.org/proceedings/2022/510.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/510","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":510,"pages":{"first":3673,"last":3680},"keywords":["Machine Learning: Robustness","Machine Learning: Deep Reinforcement Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986808","length_sec":90,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984951","length_sec":230,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Alexander and Baptista, 2004] Gordon J Alexander and Alexandre M Baptista. A comparison of var and cvar constraints on portfolio selection with the mean-variance model. Management Science, 50(9):1261-1273, 2004.","title":"A comparison of var and cvar constraints on portfolio selection with the mean-variance model","authors":"J Gordon, Alexandre M Alexander, Baptista","year":"2004","venue":"Management Science","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alexander et al., 2006] Siddharth Alexander, Thomas F Coleman, and Yuying Li. Minimizing cvar and var for a portfolio of derivatives. Journal of Banking & Finance (JBF), 30(2):583-605, 2006.","title":"Minimizing cvar and var for a portfolio of derivatives","authors":"Siddharth Alexander, Thomas F Coleman, Yuying Li","year":"2006","venue":"Journal of Banking & Finance (JBF)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bertsekas, 1997] Dimitri P Bertsekas. Nonlinear program- ming. Journal of the Operational Research Society (JORS), 48(3):334-334, 1997.","title":"Nonlinear programming","authors":"Dimitri P Bertsekas","year":"1997","venue":"Journal of the Operational Research Society (JORS)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chow and Ghavamzadeh, 2014] Yinlam Chow and Moham- mad Ghavamzadeh. Algorithms for cvar optimization in mdps. In Proceedings of the 27th International Confer- ence on Neural Information Processing Systems (NeurIPS), pages 3509-3517, 2014.","title":"Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for cvar optimization in mdps","authors":"Ghavamzadeh Chow","year":"2014","venue":"Proceedings of the 27th International Conference on Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chow et al., 2015] Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: a cvar optimization approach. Advances in Neural Information Processing Systems (NeurIPS), 28:1522-1530, 2015.","title":"Risk-sensitive and robust decisionmaking: a cvar optimization approach","authors":"Chow","year":"2015","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chow et al., 2017] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research (JMLR), 18(1):6070-6120, 2017.","title":"Risk-constrained reinforcement learning with percentile risk criteria","authors":"Chow","year":"2017","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dabney et al., 2018] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International con- ference on machine learning, pages 1096-1105. PMLR, 2018.","title":"Implicit quantile networks for distributional reinforcement learning","authors":"Dabney","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Garcıa and Fernández, 2015] Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research (JMLR), 16(1):1437-1480, 2015.","title":"Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning","authors":"Fernández Garcıa","year":"2015","venue":"Journal of Machine Learning Research (JMLR)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Heger, 1994] Matthias Heger. Consideration of risk in re- inforcement learning. In Machine Learning Proceedings 1994, pages 105-111. Elsevier, 1994.","title":"Consideration of risk in reinforcement learning","authors":"Matthias Heger","year":"1994","venue":"Machine Learning Proceedings 1994","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2017] Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversar- ial attacks on neural network policies. arXiv preprint arXiv:1702.02284, 2017.","title":"Adversarial attacks on neural network policies","authors":"Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, Pieter Abbeel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kakade and Langford, 2002] Sham Kakade and John Lang- ford. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference on Machine Learning (ICML). Citeseer, 2002.","title":"Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning","authors":"Langford Kakade","year":"2002","venue":"Proc. 19th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kendall et al., 2019] Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in a day. In 2019 International Conference on Robotics and Automation (ICRA), pages 8248-8254. IEEE, 2019.","title":"Learning to drive in a day","authors":"Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, Amar Shah","year":"2019","venue":"2019 International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Inter- national Conference on Learning Representations (ICLR), 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ma et al., 2020] Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. Dsac: Distributional soft ac- tor critic for risk-sensitive reinforcement learning. arXiv preprint arXiv:2004.14547, 2020.","title":"Dsac: Distributional soft actor critic for risk-sensitive reinforcement learning","authors":"Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang, Qianchuan Zhao","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin A Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nilim and El Ghaoui, 2005] Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-798, 2005.","title":"Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices","authors":"El Nilim, Ghaoui","year":"2005","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Prashanth, 2014] LA Prashanth. Policy gradients for cvar- constrained mdps. In International Conference on Algorith- mic Learning Theory, pages 155-169. Springer, 2014.","title":"Policy gradients for cvarconstrained mdps","authors":"; La Prashanth, Prashanth","year":"2014","venue":"International Conference on Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rigter et al., 2021] Marc Rigter, Bruno Lacerda, and Nick Hawes. Risk-averse bayes-adaptive reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.","title":"Risk-averse bayes-adaptive reinforcement learning","authors":"Rigter","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning (ICML), pages 1889-1897. PMLR, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"International conference on machine learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J. Maddi- son, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Pan- neershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Tim- othy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, Demis Hassabis","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sutton et al., 2000] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approx- imation. In Advances in neural information processing systems (NeurIPS), pages 1057-1063, 2000.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David A Richard S Sutton, Mcallester, P Satinder, Yishay Singh, Mansour","year":"2000","venue":"Advances in neural information processing systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tamar et al., 2013] Aviv Tamar, Huan Xu, and Shie Mannor. Scaling up robust mdps by reinforcement learning. arXiv preprint arXiv:1306.6189, 2013.","title":"Scaling up robust mdps by reinforcement learning","authors":"Tamar","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tamar et al., 2015] Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the cvar via sampling. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.","title":"Optimizing the cvar via sampling","authors":"Tamar","year":"2015","venue":"Twenty-Ninth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tang et al., 2020] Yichuan Charlie Tang, Jian Zhang, and Ruslan Salakhutdinov. Worst cases policy gradients. In Conference on Robot Learning, pages 1078-1093. PMLR, 2020.","title":"Worst cases policy gradients","authors":"Charlie Yichuan, Jian Tang, Ruslan Zhang, Salakhutdinov","year":"2020","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wiesemann et al., 2013] Wolfram Wiesemann, Daniel Kuhn, and Berc ¸Rustem. Robust markov decision processes. Math- ematics of Operations Research, 38(1):153-183, 2013.","title":"Robust markov decision processes","authors":"Wiesemann","year":"2013","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2021] Qisong Yang, Thiago D Simão, Simon H Tindemans, and Matthijs TJ Spaan. Wcsac: Worst-case soft actor critic for safety-constrained reinforcement learning. In Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence. AAAI Press, online, 2021.","title":"Wcsac: Worst-case soft actor critic for safety-constrained reinforcement learning","authors":"Yang","year":"2021","venue":"Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2020] Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adver- sarial perturbations on state observations. Advances in Neu- ral Information Processing Systems (NeurIPS), 33:21024- 21037, 2020.","title":"Robust deep reinforcement learning against adversarial perturbations on state observations","authors":"Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, Cho-Jui Hsieh","year":"2020","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}