{"type":"ScholarlyArticle","title":"Optimal Decision Tree Policies for Markov Decision Processes","authors":[{"name":"Daniël Vos","slug":"daniel-vos","given":"Daniël","family":"Vos","affiliation":"Delft University of Technology"},{"name":"Sicco Verwer","slug":"sicco-verwer","given":"Sicco","family":"Verwer","affiliation":"Delft University of Technology"}],"abstract":"Interpretability of reinforcement learning policies is essential for many real-world tasks but learning such interpretable policies is a hard problem. Particularly, rule-based policies such as decision trees and rules lists are difficult to optimize due to their non-differentiability. While existing techniques can learn verifiable decision tree policies, there is no guarantee that the learners generate a policy that performs optimally. In this work, we study the optimization of size-limited decision trees for Markov Decision Processes (MPDs) and propose OMDTs: Optimal MDP Decision Trees. Given a user-defined size limit and MDP formulation, OMDT directly maximizes the expected discounted return for the decision tree using Mixed-Integer Linear Programming. By training optimal tree policies for different MDPs we empirically study the optimality gap for existing imitation learning techniques and find that they perform sub-optimally. We show that this is due to an inherent shortcoming of imitation learning, namely that complex policies cannot be represented using size-limited trees. In such cases, it is better to directly optimize the tree for expected return. While there is generally a trade-off between the performance and interpretability of machine learning models, we find that on small MDPs, depth 3 OMDTs often perform close to optimally.","doi":"10.24963/ijcai.2023/606","url":"https://ijcai.org/proceedings/2023/606","pdf":"https://ijcai.org/proceedings/2023/0606.pdf","markdown":"https://ijcai.org/proceedings/2023/606.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/606","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Planning and Scheduling","track":"Main Track","number":606,"pages":{"first":5457,"last":5465},"keywords":["Planning and Scheduling: PS: Markov decisions processes","Search: S: Combinatorial search and optimisation","Machine Learning: ML: Explainable/Interpretable machine learning"],"videos":[],"references":[{"position":1,"raw":"[Aghaei et al., 2019] Sina Aghaei, Mohammad Javad Azizi, and Phebe Vayanos. Learning optimal and fair decision trees for non-discriminative decision-making. In Proceed- ings of the AAAI Conference on Artificial Intelligence, vol- ume 33, pages 1418-1426, 2019.","title":"Learning optimal and fair decision trees for non-discriminative decision-making","authors":"Aghaei","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aglin et al., 2020] Gaël Aglin, Siegfried Nijssen, and Pierre Schaus. Learning optimal decision trees using caching branch-and-bound search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3146-3153, 2020.","title":"Learning optimal decision trees using caching branch-and-bound search","authors":"Siegfried Gaël Aglin, Pierre Nijssen, Schaus","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Aglin et al., 2021] Gaël Aglin, Siegfried Nijssen, and Pierre Schaus. PyDL8.5: a library for learning optimal decision trees. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artifi- cial Intelligence, pages 5222-5224, 2021.","title":"PyDL8.5: a library for learning optimal decision trees","authors":"Siegfried Gaël Aglin, Pierre Nijssen, Schaus","year":"2021","venue":"Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2020/750"},{"position":4,"raw":"[Ashok et al., 2020] Pranav Ashok, Mathias Jackermeier, Pushpak Jagtap, Jan Křetínskỳ, Maximilian Weininger, and Majid Zamani. dtcontrol: Decision tree learning algo- rithms for controller representation. In Proceedings of the 23rd international conference on hybrid systems: Compu- tation and control, pages 1-7, 2020.","title":"dtcontrol: Decision tree learning algorithms for controller representation","authors":"Pranav Ashok, Mathias Jackermeier, Pushpak Jagtap, Jan Křetínskỳ, Maximilian Weininger, Majid Zamani","year":"2020","venue":"Proceedings of the 23rd international conference on hybrid systems: Computation and control","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bastani et al., 2018] Osbert Bastani, Yewen Pu, and Ar- mando Solar-Lezama. Verifiable reinforcement learning via policy extraction. Advances in neural information pro- cessing systems, 31, 2018.","title":"Verifiable reinforcement learning via policy extraction","authors":"Osbert Bastani, Yewen Pu, Armando Solar-Lezama","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bellman, 1957] Richard Bellman. A Markovian decision process. Journal of mathematics and mechanics, pages 679-684, 1957.","title":"A Markovian decision process","authors":"Richard Bellman","year":"1957","venue":"Journal of mathematics and mechanics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bertsimas and Dunn, 2017] Dimitris Bertsimas and Jack Dunn. Optimal classification trees. Machine Learning, 106(7):1039-1082, 2017.","title":"Optimal classification trees","authors":"Dimitris Bertsimas, Jack Dunn","year":"2017","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Brázdil et al., 2015] Tomáš Brázdil, Krishnendu Chatterjee, Martin Chmelík, Andreas Fellner, and Jan Křetínskỳ. Counterexample explanation by learning small strategies in Markov decision processes. In International Conference on Computer Aided Verification, pages 158-177. Springer, 2015.","title":"Counterexample explanation by learning small strategies in Markov decision processes","authors":"Brázdil","year":"2015","venue":"International Conference on Computer Aided Verification","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Breiman et al., 1984] L Breiman, JH Friedman, R Olshen, and CJ Stone. Classification and Regression Trees. Wadsworth, 1984.","title":"Classification and Regression Trees","authors":"Breiman, R Friedman, Olshen, Stone","year":"1984","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Demirović et al., 2020] Emir Demirović, Anna Lukina, Emmanuel Hebrard, Jeffrey Chan, James Bailey, Christo- pher Leckie, Kotagiri Ramamohanarao, and Peter J Stuckey. MurTree: optimal classification trees via dynamic programming and search. arXiv preprint arXiv:2007.12652, 2020.","title":"MurTree: optimal classification trees via dynamic programming and search","authors":"Demirović","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Glanois et al., 2021] Claire Glanois, Paul Weng, Matthieu Zimmer, Dong Li, Tianpei Yang, Jianye Hao, and Wulong Liu. A survey on interpretable reinforcement learning. arXiv preprint arXiv:2112.13112, 2021.","title":"A survey on interpretable reinforcement learning","authors":"Claire Glanois, Paul Weng, Matthieu Zimmer, Dong Li, Tianpei Yang, Jianye Hao, Wulong Liu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gunning and Aha, 2019] David Gunning and David Aha. DARPA's explainable artificial intelligence (XAI) pro- gram. AI magazine, 40(2):44-58, 2019.","title":"DARPA's explainable artificial intelligence (XAI) program","authors":"David Gunning, David Aha","year":"2019","venue":"AI magazine","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gupta et al., 2015] Ujjwal Das Gupta, Erik Talvitie, and Michael Bowling. Policy tree: Adaptive representation for policy gradient. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.","title":"Policy tree: Adaptive representation for policy gradient","authors":"Ujjwal Das Gupta, Erik Talvitie, Michael Bowling","year":"2015","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hu et al., 2020] Hao Hu, Mohamed Siala, Emmanuel He- brard, and Marie-José Huguet. Learning optimal decision trees with MaxSAT and its integration in AdaBoost. In IJCAI-PRICAI 2020, 29th International Joint Conference on Artificial Intelligence and the 17th Pacific Rim Interna- tional Conference on Artificial Intelligence, 2020.","title":"Learning optimal decision trees with MaxSAT and its integration in AdaBoost","authors":"Hao Hu, Mohamed Siala, Emmanuel Hebrard, Marie-José Huguet","year":"2020","venue":"IJCAI-PRICAI 2020, 29th International Joint Conference on Artificial Intelligence and the 17th Pacific Rim International Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2020/163"},{"position":15,"raw":"[Kearns, 1996] Michael Kearns. Boosting theory towards practice: Recent developments in decision tree induction and the weak learning framework. In Proceedings of the national conference on artificial intelligence, pages 1337- 1339, 1996.","title":"Boosting theory towards practice: Recent developments in decision tree induction and the weak learning framework","authors":"Michael Kearns","year":"1996","venue":"Proceedings of the national conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Likmeta et al., 2020] Amarildo Likmeta, Alberto Maria Metelli, Andrea Tirinzoni, Riccardo Giol, Marcello Restelli, and Danilo Romano. Combining reinforcement learning with rule-based controllers for transparent and general decision-making in autonomous driving. Robotics and Autonomous Systems, 131:103568, 2020.","title":"Combining reinforcement learning with rule-based controllers for transparent and general decision-making in autonomous driving","authors":"Amarildo Likmeta, Alberto Maria Metelli, Andrea Tirinzoni, Riccardo Giol, Marcello Restelli, Danilo Romano","year":"2020","venue":"Robotics and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2020] Jimmy Lin, Chudi Zhong, Diane Hu, Cyn- thia Rudin, and Margo Seltzer. Generalized and scalable optimal sparse decision trees. In International Conference on Machine Learning, pages 6150-6160. PMLR, 2020.","title":"Generalized and scalable optimal sparse decision trees","authors":"Jimmy Lin, Chudi Zhong, Diane Hu, Cynthia Rudin, Margo Seltzer","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lipton, 2018] Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of in- terpretability is both important and slippery. Queue, 16(3):31-57, 2018.","title":"The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery","authors":"Zachary C Lipton","year":"2018","venue":"Queue","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lundberg and Lee, 2017] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.","title":"A unified approach to interpreting model predictions","authors":"M Scott, Su-In Lundberg, Lee","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Molnar, 2020] Christoph Molnar. Interpretable machine learning. Lulu. com, 2020.","title":"Interpretable machine learning","authors":"Christoph Molnar","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Narodytska et al., 2018] Nina Narodytska, Alexey Ignatiev, Filipe Pereira, Joao Marques-Silva, and I Ras. Learning optimal decision trees with sat. In IJCAI, pages 1362- 1368, 2018.","title":"Learning optimal decision trees with sat","authors":"Nina Narodytska, Alexey Ignatiev, Filipe Pereira, Joao Marques-Silva, Ras","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/189"},{"position":22,"raw":"[Nijssen and Fromont, 2007] Siegfried Nijssen and Elisa Fromont. Mining optimal decision trees from itemset lat- tices. In Proceedings of the 13th ACM SIGKDD interna- tional conference on Knowledge discovery and data min- ing, pages 530-539, 2007.","title":"Mining optimal decision trees from itemset lattices","authors":"Fromont Nijssen, Siegfried Nijssen, Elisa Fromont","year":"2007","venue":"Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Puterman, 2014] Martin L Puterman. Markov decision pro- cesses: discrete stochastic dynamic programming. John Wiley & Sons, 2014.","title":"Markov decision processes: discrete stochastic dynamic programming","authors":"Martin L Puterman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Quinlan, 1986] J. Ross Quinlan. Induction of decision trees. Machine learning, 1(1):81-106, 1986.","title":"Induction of decision trees","authors":"J, Ross Quinlan","year":"1986","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ribeiro et al., 2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. \"Why should I trust you?\" Explain- ing the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowl- edge discovery and data mining, pages 1135-1144, 2016.","title":"Why should I trust you?\" Explaining the predictions of any classifier","authors":"Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin","year":"2016","venue":"Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ross et al., 2011] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceed- ings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Work- shop and Conference Proceedings, 2011.","title":"A reduction of imitation learning and structured prediction to no-regret online learning","authors":"Stéphane Ross, Geoffrey Gordon, Drew Bagnell","year":"2011","venue":"Proceedings of the fourteenth international conference on artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Rudin, 2019] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-215, 2019.","title":"Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead","authors":"Cynthia Rudin","year":"2019","venue":"Nature Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Schidler and Szeider, 2021] André Schidler and Stefan Szeider. SAT-based decision tree learning for large data sets. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3904-3912, 2021.","title":"SAT-based decision tree learning for large data sets","authors":"Szeider Schidler, André Schidler, Stefan Szeider","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Schrittwieser et al., 2020] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature, 588(7839):604-609, 2020.","title":"Mastering Atari, Go, chess and shogi by planning with a learned model","authors":"Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel","year":"2020","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Silva et al., 2020] Andrew Silva, Matthew Gombolay, Tay- lor Killian, Ivan Jimenez, and Sung-Hyun Son. Optimiza- tion methods for interpretable differentiable decision trees applied to reinforcement learning. In International confer- ence on artificial intelligence and statistics, pages 1855- 1865. PMLR, 2020.","title":"Optimization methods for interpretable differentiable decision trees applied to reinforcement learning","authors":"Andrew Silva, Matthew Gombolay, Taylor Killian, Ivan Jimenez, Sung-Hyun Son","year":"2020","venue":"International conference on artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Topin et al., 2021] Nicholay Topin, Stephanie Milani, Fei Fang, and Manuela Veloso. Iterative Bounding MDPs: Learning interpretable policies via non-interpretable meth- ods. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9923-9931, 2021.","title":"Iterative Bounding MDPs: Learning interpretable policies via non-interpretable methods","authors":"Topin","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Verhaeghe et al., 2020] Hélene Verhaeghe, Siegfried Ni- jssen, Gilles Pesant, Claude-Guy Quimper, and Pierre Schaus. Learning optimal decision trees using constraint programming. Constraints, 25(3):226-250, 2020.","title":"Learning optimal decision trees using constraint programming","authors":"Verhaeghe","year":"2020","venue":"Constraints","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Verwer and Zhang, 2017] Sicco Verwer and Yingqian Zhang. Learning decision trees with flexible constraints and objectives using integer optimization. In International Conference on AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems, pages 94-103. Springer, 2017.","title":"Learning decision trees with flexible constraints and objectives using integer optimization","authors":"Zhang Verwer, Sicco Verwer, Yingqian Zhang","year":"2017","venue":"International Conference on AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Verwer and Zhang, 2019] Sicco Verwer and Yingqian Zhang. Learning optimal classification trees using a binary linear program formulation. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 1625-1632, 2019.","title":"Learning optimal classification trees using a binary linear program formulation","authors":"Zhang Verwer, Sicco Verwer, Yingqian Zhang","year":"2019","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Vos and Verwer, 2022] Daniël Vos and Sicco Verwer. Ro- bust optimal classification trees against adversarial exam- ples. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8520-8528, 2022.","title":"Robust optimal classification trees against adversarial examples","authors":"Verwer Vos, Daniël Vos, Sicco Verwer","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}