{"type":"ScholarlyArticle","title":"Constrained Bayesian Reinforcement Learning via Approximate Linear Programming","authors":[{"name":"Jongmin Lee","slug":"jongmin-lee","given":"Jongmin","family":"Lee","affiliation":"† School of Computing , KAIST , Republic of Korea"},{"name":"Youngsoo Jang","slug":"youngsoo-jang","given":"Youngsoo","family":"Jang","affiliation":"† School of Computing , KAIST , Republic of Korea"},{"name":"Pascal Poupart","slug":"pascal-poupart","given":"Pascal","family":"Poupart","affiliation":"‡ David R. Cheriton School of Computer Science , University of Waterloo , Canada"},{"name":"Kee-Eung Kim","slug":"kee-eung-kim","given":"Kee-Eung","family":"Kim","affiliation":"† School of Computing , KAIST , Republic of Korea"}],"abstract":"In this paper, we consider the safe learning scenario where we need to restrict the exploratory behavior of a reinforcement learning agent. Specifically, we treat the problem as a form of Bayesian reinforcement learning in an environment that is modeled as a constrained MDP (CMDP) where the cost function penalizes undesirable situations. We propose a model-based Bayesian reinforcement learning (BRL) algorithm for such an environment, eliciting risk-sensitive exploration in a principled way. Our algorithm efficiently solves the constrained BRL problem by approximate linear programming, and generates a finite state controller in an off-line manner. We provide theoretical guarantees and demonstrate empirically that our approach outperforms the state of the art.","doi":"10.24963/ijcai.2017/290","url":"https://ijcai.org/proceedings/2017/290","pdf":"https://ijcai.org/proceedings/2017/0290.pdf","markdown":"https://ijcai.org/proceedings/2017/290.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/290","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":290,"pages":{"first":2088,"last":2095},"keywords":["Uncertainty in AI: Markov Decision Processes","Planning and Scheduling: POMDPs","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Altman, 1999] Eitan Altman. Constrained Markov Deci- sion Processes. Chapman and Hall, 1999.","title":"Constrained Markov Decision Processes","authors":"Eitan Altman","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Araya-López et al., 2012] Mauricio Araya-López, Vincent Thomas, and Olivier Buffet. Near-optimal BRL using op- timistic local transition. In Proceedings of the 29th Inter- national Conference on Machine Learning, pages 97-104, 2012.","title":"Near-optimal BRL using optimistic local transition","authors":"Mauricio Araya-López, Vincent Thomas, Olivier Buffet","year":"2012","venue":"Proceedings of the 29th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Asmuth et al., 2009] John Asmuth, Lihong Li, Michael L. Littman, Ali Nouri, and David Wingate. A Bayesian sam- pling approach to exploration in reinforcement learning. In Proceedings of the Twenty-Fifth Conference on Uncer- tainty in Artificial Intelligence, pages 19-26, 2009.","title":"A Bayesian sampling approach to exploration in reinforcement learning","authors":"John Asmuth, Lihong Li, Michael L Littman, Ali Nouri, David Wingate","year":"2009","venue":"Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dearden et al., 1998] Richard Dearden, Nir Friedman, and Stuart Russell. Bayesian Q-learning. In Proceedings of the 15th National Conference on Artificial Intelligence, pages 761-768, 1998.","title":"Bayesian Q-learning","authors":"Richard Dearden, Nir Friedman, Stuart Russell","year":"1998","venue":"Proceedings of the 15th National Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dearden et al., 1999] Richard Dearden, Nir Friedman, and David Andre. Model based Bayesian exploration. In Pro- ceedings of the Fifteenth Conference on Uncertainty in Ar- tificial Intelligence, pages 150-159, 1999.","title":"Model based Bayesian exploration","authors":"Richard Dearden, Nir Friedman, David Andre","year":"1999","venue":"Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Duff, 2002] Michael O'Gordon Duff. Optimal learning: Computational procedures for Bayes-adaptive Markov de- cision processes. PhD thesis, University of Massachusetts Amherst, 2002.","title":"Optimal learning: Computational procedures for Bayes-adaptive Markov decision processes","authors":"O'gordon Michael, Duff","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[García and Fernández, 2012] Javier García and Fernando Fernández. Safe exploration of state and action spaces in reinforcement learning. Journal of Artificial Intelligence Research, 45(1):515-564, 2012.","title":"Javier García and Fernando Fernández. Safe exploration of state and action spaces in reinforcement learning","authors":"Fernández García","year":"2012","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hans et al., 2008] Alexander Hans, Daniel Schneega, An- ton M. Schäfer, and Steffen Udluft. Safe exploration for reinforcement learning. In Proceedings of the European Symposium on Artificial Neural Network, pages 143-148, 2008.","title":"Safe exploration for reinforcement learning","authors":"Alexander Hans, Daniel Schneega, Anton M Schäfer, Steffen Udluft","year":"2008","venue":"Proceedings of the European Symposium on Artificial Neural Network","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Howard and Matheson, 1972] Ronald A. Howard and James E. Matheson. Risk-sensitive markov decision processes. Management Science, 18(7):356-369, 1972.","title":"Risk-sensitive markov decision processes","authors":"Ronald A Howard, James E Matheson","year":"1972","venue":"Management Science","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hsu et al., 2007] David Hsu, Wee Sun Lee, and Nan Rong. What makes some POMDP problems easy to approxi- mate? In Advances in Neural Information Processing Sys- tems 20, pages 689-696, 2007.","title":"What makes some POMDP problems easy to approximate?","authors":"David Hsu, Wee Sun Lee, Nan Rong","year":"2007","venue":"Advances in Neural Information Processing Systems 20","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Iyengar, 2005] Garud N. Iyengar. Robust dynamic program- ming. Mathematics of Operations Research, 30(2):257- 280, 2005.","title":"Robust dynamic programming","authors":"N Garud, Iyengar","year":"2005","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim et al., 2011] Dongho Kim, Jaesong Lee, Kee-Eung Kim, and Pascal Poupart. Point-based value iteration for constrained pomdps. In Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, pages 1968-1974, 2011.","title":"Point-based value iteration for constrained pomdps","authors":"Dongho Kim, Jaesong Lee, Kee-Eung Kim, Pascal Poupart","year":"2011","venue":"Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2011/329"},{"position":13,"raw":"[Kim et al., 2012] Dongho Kim, Kee-Eung Kim, and Pascal Poupart. Cost-sensitive exploration in Bayesian reinforce- ment learning. In Advances in Neural Information Pro- cessing Systems 25, pages 3068-3076, 2012.","title":"Cost-sensitive exploration in Bayesian reinforcement learning","authors":"Dongho Kim, Kee-Eung Kim, Pascal Poupart","year":"2012","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kolter and Ng, 2009] J. Zico Kolter and Andrew Y. Ng. Near-Bayesian exploration in polynomial time. In Pro- ceedings of the 26th Annual International Conference on Machine Learning, pages 513-520, 2009.","title":"Near-Bayesian exploration in polynomial time","authors":"Kolter, ] J Zico Ng, Andrew Y Kolter, Ng","year":"2009","venue":"Proceedings of the 26th Annual International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mihatsch and Neuneier, 2002] Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement learning. Machine Learning, 49(2):267-290, 2002.","title":"Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement learning","authors":"Neuneier Mihatsch","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Minka, 1998] Thomas P. Minka. Bayesian linear regression. Technical report, Microsoft research, 1998.","title":"Bayesian linear regression","authors":"P Thomas, Minka","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nilim and El Ghaoui, 2005] Arnab Nilim and Laurent El Ghaoui. Robust control of Markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-798, 2005.","title":"Robust control of Markov decision processes with uncertain transition matrices","authors":"Arnab Nilim, Laurent El Ghaoui","year":"2005","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Poupart et al., 2006] Pascal Poupart, Nikos Vlassis, Jesse Hoey, and Kevin Regan. An analytic solution to dis- crete Bayesian reinforcement learning. In Proceedings of the 23rd International Conference on Machine Learning, pages 697-704, 2006.","title":"An analytic solution to discrete Bayesian reinforcement learning","authors":"Pascal Poupart, Nikos Vlassis, Jesse Hoey, Kevin Regan","year":"2006","venue":"Proceedings of the 23rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Poupart et al., 2015] Pascal Poupart, Aarti Malhotra, Pei Pei, Kee-Eung Kim, Bongseok Goh, and Michael Bowl- ing. Approximate linear programming for constrained par- tially observable Markov decision processes. In Proceed- ings of the Twenty-Ninth AAAI Conference on Artificial In- telligence, pages 3342-3348, 2015.","title":"Approximate linear programming for constrained partially observable Markov decision processes","authors":"Pascal Poupart, Aarti Malhotra, Pei Pei, Kee-Eung Kim, Bongseok Goh, Michael Bowling","year":"2015","venue":"Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Strens, 2000] Malcolm J. A. Strens. A Bayesian framework for reinforcement learning. In Proceedings of the Sev- enteenth International Conference on Machine Learning, pages 943-950, 2000.","title":"A Bayesian framework for reinforcement learning","authors":"J A Malcolm, Strens","year":"2000","venue":"Proceedings of the Seventeenth International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.","title":"Reinforcement Learning: An Introduction","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tziortziotis et al., 2013] Nikolaos Tziortziotis, Chris- tos Dimitrakakis, and Konstantinos Blekas. Linear Bayesian reinforcement learning. In Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, 2013.","title":"Linear Bayesian reinforcement learning","authors":"Nikolaos Tziortziotis, Christos Dimitrakakis, Konstantinos Blekas","year":"2013","venue":"Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2013/255"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}