---
title: "Model-Based Reinforcement Learning for Infinite-Horizon Discounted Constrained Markov Decision Processes"
authors:
  - "Aria HasanzadeZonuzy"
  - "Dileep Kalathil"
  - "Srinivas Shakkottai"
doi: "10.24963/ijcai.2021/347"
url: "https://ijcai.org/proceedings/2021/347"
pdf: "https://ijcai.org/proceedings/2021/0347.pdf"
venue: "Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence"
year: 2021
pages: "2519-2525"
track: "Main Track"
keywords:
  - "Machine Learning: Reinforcement Learning"
  - "Planning and Scheduling: Markov Decisions Processes"
---

# Model-Based Reinforcement Learning for Infinite-Horizon Discounted Constrained Markov Decision Processes

Aria HasanzadeZonuzy, Dileep Kalathil, Srinivas Shakkottai

*Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence*, pages 2519-2525. DOI: [10.24963/ijcai.2021/347](https://doi.org/10.24963/ijcai.2021/347)

## Abstract

In many real-world reinforcement learning (RL) problems, in addition to maximizing the  objective, the learning agent has to  maintain some necessary safety constraints.  We formulate the problem of learning a safe policy as  an  infinite-horizon discounted Constrained Markov Decision Process (CMDP)  with an unknown transition probability matrix, where the safety requirements are modeled as   constraints on expected cumulative costs.  We propose two model-based constrained reinforcement learning (CRL) algorithms for learning a safe policy, namely, (i) GM-CRL algorithm,  where the algorithm has access to a generative model, and (ii) UC-CRL  algorithm,  where the algorithm learns the model using an upper confidence style online exploration method.   We characterize the sample complexity of these algorithms, i.e., the the number of samples needed to ensure a desired level of accuracy with high probability, both with respect to objective maximization and constraint satisfaction.