{"type":"ScholarlyArticle","title":"Learning Optimal Decision Trees with MaxSAT and its Integration in AdaBoost","authors":[{"name":"Hao Hu","slug":"hao-hu","given":"Hao","family":"Hu","affiliation":"LAAS-CNRS , Université de Toulouse , CNRS , INSA , Toulouse , France"},{"name":"Mohamed Siala","slug":"mohamed-siala","given":"Mohamed","family":"Siala","affiliation":"LAAS-CNRS , Université de Toulouse , CNRS , INSA , Toulouse , France"},{"name":"Emmanuel Hebrard","slug":"emmanuel-hebrard","given":"Emmanuel","family":"Hebrard","affiliation":"LAAS-CNRS , Université de Toulouse , CNRS , INSA , Toulouse , France"},{"name":"Marie-José Huguet","slug":"marie-jose-huguet","given":"Marie-José","family":"Huguet","affiliation":"LAAS-CNRS , Université de Toulouse , CNRS , INSA , Toulouse , France"}],"abstract":"Recently, several exact methods to compute decision trees have been introduced. On the one hand, these approaches can find optimal trees for various objective functions including total size, depth or accuracy on the training set and therefore.  On the other hand, these methods are not yet widely used in practice and classic heuristics are often still the methods of choice.\r\n\r\nIn this paper we show how the SAT model proposed by [Narodytska et.al 2018] can be lifted to a MaxSAT approach, making it much more practically relevant. \r\nIn particular, it scales to much larger data sets; the objective function can easily be adapted to take into account combinations of size, depth and accuracy on the training set; and the fine-grained control of the objective function it offers makes it particularly well suited for boosting.\r\n\r\nOur experiments show promising results.\r\nIn particular, we show that the prediction quality of  our approach often exceeds state of the art heuristics.  We also show that the MaxSAT formulation is well adapted for boosting using the well-known AdaBoost Algorithm.","doi":"10.24963/ijcai.2020/163","url":"https://ijcai.org/proceedings/2020/163","pdf":"https://ijcai.org/proceedings/2020/0163.pdf","markdown":"https://ijcai.org/proceedings/2020/163.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/163","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Constraints and SAT","track":"Main track","number":163,"pages":{"first":1170,"last":1176},"keywords":["Heuristic Search and Game Playing: Combinatorial Search and Optimisation","Constraints and SAT: MaxSAT, MinSAT","Constraints and SAT: Constraint Optimization"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24667/file_24667.m3u8","length_sec":292,"provider":"hls"}],"references":[{"position":1,"raw":"[Aglin et al., 2020] Gael Aglin, Siegfried Nijssen, and Pierre Schaus. Learning optimal decision trees using caching branch-and-bound search. In Proceedings of the Thirty- Fourth Conference on Artificial Intelligence (AAAI), New York, USA, February 2020.","title":"Learning optimal decision trees using caching branch-and-bound search","authors":"Aglin","year":"2020","venue":"Proceedings of the Thirty-Fourth Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Avellaneda, 2020] Florent Avellaneda. Efficient inference of optimal decision trees. In Proceedings of the Thirty- Fourth Conference on Artificial Intelligence (AAAI), New York, USA, February 2020.","title":"Efficient inference of optimal decision trees","authors":"Florent Avellaneda","year":"2020","venue":"Proceedings of the Thirty-Fourth Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bennett, 1994] Kristin Bennett. Global tree optimization: A non-greedy decision tree algorithm. In Computing Science and Statistics, pages 156-160, 1994.","title":"Global tree optimization: A non-greedy decision tree algorithm","authors":"Kristin Bennett","year":"1994","venue":"Computing Science and Statistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Berg et al., 2019] Jeremias Berg, Emir Demirović, and Pe- ter J. Stuckey. Core-boosted linear search for incomplete maxsat. In Proceedings of the 16th International Confer- ence on the Integration of Constraint Programming, Ar- tificial Intelligence, and Operations Research -CPAIOR, pages 39-56, June 4-7, 2019.","title":"Core-boosted linear search for incomplete maxsat","authors":"Jeremias Berg, Emir Demirović, Peter J Stuckey","year":"2019","venue":"Proceedings of the 16th International Conference on the Integration of Constraint Programming, Artificial Intelligence, and Operations Research -CPAIOR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bertsimas and Dunn, 2017] Dimitris Bertsimas and Jack Dunn. Optimal classification trees. Machine Learning, 106(7):1039-1082, 2017.","title":"Optimal classification trees","authors":"Dimitris Bertsimas, Jack Dunn","year":"2017","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bessiere et al., 2009] Christian Bessiere, Emmanuel He- brard, and Barry O'Sullivan. Minimising decision tree size as combinatorial optimisation. In Proceedings of the of 15th International Conference on Principles and Practice of Constraint Programming -CP, pages 173-187, Septem- ber 20-24, 2009.","title":"Christian Bessiere, Emmanuel Hebrard, and Barry O'Sullivan. Minimising decision tree size as combinatorial optimisation","authors":"Bessiere","year":"2009","venue":"Proceedings of the of 15th International Conference on Principles and Practice of Constraint Programming -CP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Biere et al., 2009] Armin Biere, Marijn Heule, Hans van Maaren, and Toby Walsh, editors. Handbook of Satisfi- ability, volume 185 of Frontiers in Artificial Intelligence and Applications, 2009.","title":"Handbook of Satisfiability","authors":"Biere","year":"2009","venue":"Frontiers in Artificial Intelligence and Applications","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Breiman et al., 1984] Leo Breiman, J. H. Friedman, R. A. Olshen, , and C. J. Stone. Classification and Regression Trees. Chapman & Hall/CRC, 1st edition, 1984.","title":"Classification and Regression Trees","authors":"J H Leo Breiman, R A Friedman, C J Olshen, Stone","year":"1984","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Breiman, 1996] Leo Breiman. Bagging predictors. Machine Learning, 24(2):123-140, 1996.","title":"Bagging predictors","authors":"Breiman","year":"1996","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Breiman, 2001] Leo Breiman. Random forests. Machine Learning, 45(1):5-32, 2001.","title":"Leo Breiman. Random forests","authors":"Breiman","year":"2001","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Demirović and Stuckey, 2019] Emir Demirović and Peter J. Stuckey. Techniques inspired by local search for incom- plete maxsat and the linear algorithm: Varying resolution and solution-guided search. In Thomas Schiex and Simon de Givry, editors, Principles and Practice of Constraint Programming -25th International Conference, CP 2019, Stamford, CT, USA, September 30 -October 4, 2019, Pro- ceedings, volume 11802 of Lecture Notes in Computer Sci- ence, pages 177-194. Springer, 2019.","title":"Techniques inspired by local search for incomplete maxsat and the linear algorithm: Varying resolution and solution-guided search","authors":"Stuckey Demirović, Emir Demirović, Peter J Stuckey","year":"2019","venue":"Principles and Practice of Constraint Programming -25th International Conference, CP 2019","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Freund and Schapire, 1997] Yoav Freund and Robert E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Com- puter and System Sciences, 55(1):119-139, 1997.","title":"A decision-theoretic generalization of on-line learning and an application to boosting","authors":"Yoav Freund, Robert E Schapire","year":"1997","venue":"Journal of Computer and System Sciences","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hu et al., 2019] Xiyang Hu, Cynthia Rudin, and Margo Seltzer. Optimal sparse decision trees. In Advances in Neural Information Processing Systems 32: Annual Con- ference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14, pages 7265-7273, December 2019.","title":"Optimal sparse decision trees","authors":"Hu","year":"2019","venue":"Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ignatiev et al., 2019] Alexey Ignatiev, Antonio Morgado, and Joao Marques-Silva. Rc2: an efficient maxsat solver. Journal on Satisfiability, Boolean Modeling and Compu- tation, 11:53-64, 09 2019.","title":"Rc2: an efficient maxsat solver","authors":"Alexey Ignatiev, Antonio Morgado, Joao Marques-Silva","year":"2019","venue":"Journal on Satisfiability, Boolean Modeling and Computation","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Narodytska et al., 2018] Nina Narodytska, Alexey Ignatiev, Filipe Pereira, and João Marques-Silva. Learning optimal decision trees with SAT. In Proceedings of the Twenty- Seventh International Joint Conference on Artificial Intel- ligence -IJCAI, pages 1362-1368, July 13-19, 2018.","title":"Learning optimal decision trees with SAT","authors":"Nina Narodytska, Alexey Ignatiev, Filipe Pereira, João Marques-Silva","year":"2018","venue":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence -IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/189"},{"position":16,"raw":"[Nijssen and Fromont, 2007] Siegfried Nijssen and Elisa Fromont. Mining optimal decision trees from itemset lat- tices. In Proceedings of the 13th ACM SIGKDD Inter- national Conference on Knowledge Discovery and Data Mining -KDD, page 530-539, August 12-15, 2007.","title":"Mining optimal decision trees from itemset lattices","authors":"Fromont Nijssen, Siegfried Nijssen, Elisa Fromont","year":"2007","venue":"Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining -KDD","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pedregosa et al., 2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blon- del, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.","title":"Scikit-learn: Machine learning in Python","authors":"F Pedregosa, G Varoquaux, A Gramfort, V Michel, B Thirion, O Grisel, M Blondel, P Prettenhofer, R Weiss, V Dubourg, J Vanderplas, A Passos, D Cournapeau, M Brucher, M Perrot, E Duchesnay","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Quinlan, 1986] J. R. Quinlan. Induction of decision trees. Machine Learning, 1(1):81-106, March 1986.","title":"Induction of decision trees","authors":"J R Quinlan","year":"1986","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Quinlan, 1993] J. Ross Quinlan. C4.5: programs for ma- chine learning. Morgan Kaufmann Publishers Inc., 1993.","title":"C4.5: programs for machine learning","authors":"J, Ross Quinlan","year":"1993","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Verhaeghe et al., 2019] Hélène Verhaeghe, Siegfried Ni- jssen, Gilles Pesant, Claude-Guy Quimper, and Pierre Schaus. Learning optimal decision trees using constraint programming. In Proceedings of the 25th International Conference on Principles and Practice of Constraint Pro- gramming -CP, Sept 30 -Oct 4, 2019.","title":"Learning optimal decision trees using constraint programming","authors":"Hélène Verhaeghe, Siegfried Nijssen, Gilles Pesant, Claude-Guy Quimper, Pierre Schaus","year":"2019","venue":"Proceedings of the 25th International Conference on Principles and Practice of Constraint Programming -CP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Verwer and Zhang, 2019] Sicco Verwer and Yingqian Zhang. Learning optimal classification trees using a binary linear program formulation. In The Thirty-Third National Conference on Artificial Intelligence (AAAI), pages 1625-1632, 2019.","title":"Learning optimal classification trees using a binary linear program formulation","authors":"Zhang Verwer, Sicco Verwer, Yingqian Zhang","year":"2019","venue":"The Thirty-Third National Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhou and Feng, 2017] Zhi-Hua Zhou and Ji Feng. Deep for- est: Towards an alternative to deep neural networks. In Proceedings of the 26th International Joint Conference on Artificial Intelligence -IJCAI, page 3553-3559, August, 19-25, 2017.","title":"Deep forest: Towards an alternative to deep neural networks","authors":"Feng ; ] Zhi-Hua Zhou, Ji Zhou, Feng","year":"2017","venue":"Proceedings of the 26th International Joint Conference on Artificial Intelligence -IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/497"},{"position":23,"raw":"[Zhou, 2012] Zhi-Hua Zhou. Ensemble Methods: Founda- tions and Algorithms. Chapman & Hall/CRC, 1st edition, 2012.","title":"Ensemble Methods: Foundations and Algorithms","authors":"Zhi-Hua Zhou","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}