{"type":"ScholarlyArticle","title":"Structure Learning for Safe Policy Improvement","authors":[{"name":"Thiago D. Simão","slug":"thiago-d-simao","given":"Thiago D.","family":"Simão","affiliation":"Delft University of Technology , The Netherlands"},{"name":"Matthijs T. J. Spaan","slug":"matthijs-t-j-spaan","given":"Matthijs T. J.","family":"Spaan","affiliation":"Delft University of Technology , The Netherlands"}],"abstract":"We investigate how Safe Policy Improvement (SPI) algorithms can exploit the structure of factored Markov decision processes when such structure is unknown a priori. To facilitate the application of reinforcement learning in the real world, SPI provides probabilistic guarantees that policy changes in a running process will improve the performance of this process. However, current SPI algorithms have requirements that might be impractical, such as: (i) availability of a large amount of historical data, or (ii) prior knowledge of the underlying structure. To overcome these limitations we enhance a Factored SPI (FSPI) algorithm with different structure learning methods. The resulting algorithms need fewer samples to improve the policy and require weaker prior knowledge assumptions. In well-factorized domains, the proposed algorithms improve performance significantly compared to a flat SPI algorithm, demonstrating a sample complexity closer to an FSPI algorithm that knows the structure. This indicates that the combination of FSPI and structure learning algorithms is a promising solution to real-world problems involving many variables.","doi":"10.24963/ijcai.2019/479","url":"https://ijcai.org/proceedings/2019/479","pdf":"https://ijcai.org/proceedings/2019/0479.pdf","markdown":"https://ijcai.org/proceedings/2019/479.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/479","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":479,"pages":{"first":3453,"last":3459},"keywords":["Machine Learning: Reinforcement Learning","Planning and Scheduling: Model-Based Reasoning","Planning and Scheduling: Planning under Uncertainty"],"videos":[],"references":[{"position":1,"raw":"[Boutilier et al., 1995] Craig Boutilier, Richard Dearden, and Moisés Goldszmidt. Exploiting Structure in Policy Construction. In Proc. Int. Joint Conf. on Artificial Intelli- gence, pages 1104-1113.","title":"Exploiting Structure in Policy Construction","authors":"Craig Boutilier, Richard Dearden, Moisés Goldszmidt","year":null,"venue":"Proc. Int. Joint Conf. on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/1995/143"},{"position":2,"raw":"Morgan Kaufmann, 1995.","title":null,"authors":"Morgan Kaufmann","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brafman and Tennenholtz, 2002] Ronen I. Brafman and Moshe Tennenholtz. R-MAX -A General Polynomial Time Algorithm for Near-Optimal Reinforcement Learn- ing. Journal of Machine Learning Research, 3:213-231, 2002.","title":"R-MAX -A General Polynomial Time Algorithm for Near-Optimal Reinforcement Learning","authors":"Ronen I Brafman, Moshe Tennenholtz","year":"2002","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":"/proceedings/2001/130"},{"position":4,"raw":"[Chakraborty and Stone, 2011] Doran Chakraborty and Pe- ter Stone. Structure Learning in Ergodic Factored MDPs without Knowledge of the Transition Function's In- Degree. In Proc. of International Conference on Machine Learning, pages 737-744.","title":"Structure Learning in Ergodic Factored MDPs without Knowledge of the Transition Function's In-Degree","authors":"Doran Chakraborty, Peter Stone","year":null,"venue":"Proc. of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Degris et al., 2006] Thomas Degris, Olivier Sigaud, and Pierre-Henri Wuillemin. Learning the Structure of Fac- tored Markov Decision Processes in Reinforcement Learn- ing Problems. In Proc. of International Conference on Ma- chine Learning, volume 148 of ACM International Confer- ence Proceeding Series, pages 257-264. ACM, 2006.","title":"Learning the Structure of Factored Markov Decision Processes in Reinforcement Learning Problems","authors":"Thomas Degris, Olivier Sigaud, Pierre-Henri Wuillemin","year":"2006","venue":"Proc. of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Delage and Mannor, 2010] Erick Delage and Shie Man- nor. Percentile Optimization for Markov Decision Pro- cesses with Parameter Uncertainty. Operations Research, 58(1):203-213, 2010.","title":"Erick Delage and Shie Mannor. Percentile Optimization for Markov Decision Processes with Parameter Uncertainty","authors":"Mannor Delage","year":"2010","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dietterich, 1998] Thomas G Dietterich. The MAXQ Method for Hierarchical Reinforcement Learning. In Proc. of International Conference on Machine Learning, vol- ume 98, pages 118-126, 1998.","title":"The MAXQ Method for Hierarchical Reinforcement Learning","authors":"G Thomas, Dietterich","year":"1998","venue":"Proc. of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Diuk et al., 2009] Carlos Diuk, Lihong Li, and Bethany R. Leffler. The Adaptive k-meteorologists Problem and Its Application to Structure Learning and Feature Selection in Reinforcement Learning. In Proc. of International Confer- ence on Machine Learning, pages 249-256. ACM, 2009.","title":"The Adaptive k-meteorologists Problem and Its Application to Structure Learning and Feature Selection in Reinforcement Learning","authors":"Carlos Diuk, Lihong Li, Bethany R Leffler","year":"2009","venue":"Proc. of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[García and Fernández, 2015] Javier García and Fernando Fernández. A Comprehensive Survey on Safe Reinforce- ment Learning. Journal of Machine Learning Research, 16:1437-1480, 2015.","title":"Javier García and Fernando Fernández. A Comprehensive Survey on Safe Reinforcement Learning","authors":"Fernández García","year":"2015","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guestrin et al., 2002] Carlos Guestrin, Relu Patrascu, and Dale Schuurmans. Algorithm-Directed Exploration for Model-Based Reinforcement Learning in Factored MDPs. In Proc. of International Conference on Machine Learn- ing, pages 235-242.","title":"Algorithm-Directed Exploration for Model-Based Reinforcement Learning in Factored MDPs","authors":"Carlos Guestrin, Relu Patrascu, Dale Schuurmans","year":null,"venue":"Proc. of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Morgan Kaufmann, 2002.","title":null,"authors":"Morgan Kaufmann","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Guestrin et al., 2003] Carlos Guestrin, Daphne Koller, Ronald Parr, and Shobha Venkataraman. Efficient Solu- tion Algorithms for Factored MDPs. Journal of Artificial Intelligence Research, 19:399-468, 2003.","title":"Efficient Solution Algorithms for Factored MDPs","authors":"Carlos Guestrin, Daphne Koller, Ronald Parr, Shobha Venkataraman","year":"2003","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hallak et al., 2015] Assaf Hallak, Franc ¸ois Schnitzler, Tim- othy Arthur Mann, and Shie Mannor. Off-policy Model- based Learning under Unknown Factored Dynamics. In Proc. of International Conference on Machine Learning, pages 711-719. JMLR.org, 2015.","title":"Off-policy Modelbased Learning under Unknown Factored Dynamics","authors":"Assaf Hallak, Timothy Franc ¸ois Schnitzler, Shie Arthur Mann, Mannor","year":"2015","venue":"Proc. of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Laroche et al., 2019] Romain Laroche, Paul Trichelair, and Rémi Tachet des Combes. Safe Policy Improvement with Baseline Bootstrapping. In Proc. of International Confer- ence on Machine Learning, 2019.","title":"Safe Policy Improvement with Baseline Bootstrapping","authors":"Romain Laroche, Paul Trichelair, Rémi Tachet Des Combes","year":"2019","venue":"Proc. of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2011] Lihong Li, Michael L. Littman, Thomas J. Walsh, and Alexander L. Strehl. Knows What It Knows: A Framework For Self-Aware Learning. Machine Learning, 82(3):399-443, 2011.","title":"Knows What It Knows: A Framework For Self-Aware Learning","authors":"Lihong Li, Michael L Littman, Thomas J Walsh, Alexander L Strehl","year":"2011","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Petrik et al., 2016] Marek Petrik, Mohammad Ghavamzadeh, and Yinlam Chow. Safe Policy Im- provement by Minimizing Robust Baseline Regret. In Advances in Neural Information Processing Systems 29, pages 2298-2306.","title":"Safe Policy Improvement by Minimizing Robust Baseline Regret","authors":"Marek Petrik, Mohammad Ghavamzadeh, Yinlam Chow","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"Curran Associates, Inc., 2016.","title":null,"authors":"Curran Associates, Inc","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Puterman, 1994] Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming.","title":"Markov Decision Processes: Discrete Stochastic Dynamic Programming","authors":"Martin L Puterman","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Simão and Spaan, 2019] Thiago D. Simão and Matthijs T. J. Spaan. Safe Policy Improvement with Baseline Bootstrap- ping in Factored Environments. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence, 2019.","title":"Safe Policy Improvement with Baseline Bootstrapping in Factored Environments","authors":"D Thiago, Simão, T J Matthijs, Spaan","year":"2019","venue":"Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Strehl et al., 2007] Alexander L. Strehl, Carlos Diuk, and Michael L. Littman. Efficient Structure Learning in Factored-State MDPs. In Proceedings of the Twenty- Second AAAI Conference on Artificial Intelligence, pages 645-650. AAAI Press, 2007.","title":"Efficient Structure Learning in Factored-State MDPs","authors":"Alexander L Strehl, Carlos Diuk, Michael L Littman","year":"2007","venue":"Proceedings of the Twenty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Strehl, 2007] Alexander L Strehl. Model-Based Reinforce- ment Learning in Factored-State MDPs. In 2007 IEEE International Symposium on Approximate Dynamic Pro- gramming and Reinforcement Learning, pages 103-110. IEEE, 2007.","title":"Model-Based Reinforcement Learning in Factored-State MDPs","authors":"L Alexander, Strehl","year":"2007","venue":"IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Thomas et al., 2015] Philip S. Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High Confidence Policy Improvement. In Proc. of International Conference on Machine Learning, pages 2380-2388.","title":"High Confidence Policy Improvement","authors":"Philip S Thomas, Georgios Theocharous, Mohammad Ghavamzadeh","year":null,"venue":"Proc. of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"JMLR.org, 2015.","title":"JMLR","authors":null,"year":"2015","venue":"JMLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}