{"type":"ScholarlyArticle","title":"Policy Learning with Constraints in Model-free Reinforcement Learning: A Survey","authors":[{"name":"Yongshuai Liu","slug":"yongshuai-liu","given":"Yongshuai","family":"Liu","affiliation":""},{"name":"Avishai Halev","slug":"avishai-halev","given":"Avishai","family":"Halev","affiliation":""},{"name":"Xin Liu","slug":"xin-liu","given":"Xin","family":"Liu","affiliation":""}],"abstract":"Reinforcement Learning (RL) algorithms have had tremendous success in simulated domains. These algorithms, however, often cannot be directly applied to physical systems, especially in cases where there are constraints to satisfy (e.g. to ensure safety or limit resource consumption). In standard RL, the agent is incentivized to explore any policy with the sole goal of maximizing reward; in the real world, however, ensuring satisfaction of certain constraints in the process is also necessary and essential. In this article, we overview existing approaches addressing constraints in model-free reinforcement learning. We model the problem of learning with constraints as a Constrained Markov Decision Process and consider two main types of constraints: cumulative and instantaneous. We summarize existing approaches and discuss their pros and cons. To evaluate policy performance under constraints, we introduce a set of standard benchmarks and metrics. We also summarize limitations of current methods and present open questions for future research.","doi":"10.24963/ijcai.2021/614","url":"https://ijcai.org/proceedings/2021/614","pdf":"https://ijcai.org/proceedings/2021/0614.pdf","markdown":"https://ijcai.org/proceedings/2021/614.md","bibtex":"https://ijcai.org/proceedings/2021/bibtex/614","volume":{"slug":"2021","title":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2021","url":"https://ijcai.org/proceedings/2021/","year":2021,"location":"Montreal","dates":"19-27 August 2021","editors":"Prof. Zhi-Hua Zhou","isbn":"978-0-9992411-9-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2021"},"section":"Survey Track","track":"Survey Track","number":614,"pages":{"first":4508,"last":4515},"keywords":["Machine learning: General","Constraints and SAT: General","Planning and scheduling: General"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}