{"type":"ScholarlyArticle","title":"Unbiased Gradient Boosting Decision Tree with Unbiased Feature Importance","authors":[{"name":"Zheyu Zhang","slug":"zheyu-zhang","given":"Zheyu","family":"Zhang","affiliation":"Institute for Interdisciplinary Information Sciences (IIIS) , Tsinghua University"},{"name":"Tianping Zhang","slug":"tianping-zhang","given":"Tianping","family":"Zhang","affiliation":"Institute for Interdisciplinary Information Sciences (IIIS) , Tsinghua University"},{"name":"Jian Li","slug":"jian-li","given":"Jian","family":"Li","affiliation":"Institute for Interdisciplinary Information Sciences (IIIS) , Tsinghua University"}],"abstract":"Gradient Boosting Decision Tree (GBDT) has achieved remarkable success in a wide variety of applications. The split finding algorithm, which determines the tree construction process, is one of the most crucial components of GBDT. However, the split finding algorithm has long been criticized for its bias towards features with a large number of potential splits. This bias introduces severe interpretability and overfitting issues in GBDT. To this end, we provide a fine-grained analysis of bias in GBDT and demonstrate that the bias originates from 1) the systematic bias in the gain estimation of each split and 2) the bias in the split finding algorithm resulting from the use of the same data to evaluate the split improvement and determine the best split. Based on the analysis, we propose unbiased gain, a new unbiased measurement of gain importance using out-of-bag samples. Moreover, we incorporate the unbiased property into the split finding algorithm and develop UnbiasedGBM to solve the overfitting issue of GBDT. We assess the performance of UnbiasedGBM and unbiased gain in a large-scale empirical study comprising 60 datasets and show that: 1) UnbiasedGBM exhibits better performance than popular GBDT implementations such as LightGBM, XGBoost, and Catboost on average on the 60 datasets and 2) unbiased gain achieves better average performance in feature selection than popular feature importance methods.","doi":"10.24963/ijcai.2023/515","url":"https://ijcai.org/proceedings/2023/515","pdf":"https://ijcai.org/proceedings/2023/0515.pdf","markdown":"https://ijcai.org/proceedings/2023/515.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/515","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":515,"pages":{"first":4629,"last":4637},"keywords":["Machine Learning: ML: Classification","Machine Learning: ML: Applications"],"videos":[],"references":[{"position":1,"raw":"[Akiba et al., 2019] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623-2631, 2019.","title":"Optuna: A next-generation hyperparameter optimization framework","authors":"Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, Masanori Koyama","year":"2019","venue":"Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Boulesteix et al., 2012] Anne-Laure Boulesteix, Andreas Bender, Justo Lorenzo Bermejo, and Carolin Strobl. Ran- dom forest gini importance favours snps with large minor allele frequency: impact, sources and recommendations. Briefings in Bioinformatics, 13(3):292-304, 2012.","title":"Random forest gini importance favours snps with large minor allele frequency: impact, sources and recommendations","authors":"Anne-Laure Boulesteix, Andreas Bender, Justo Lorenzo Bermejo, Carolin Strobl","year":"2012","venue":"Briefings in Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Breiman et al., 1984] Leo Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone. Classification and Regression Trees. Wadsworth, 1984.","title":"Classification and Regression Trees","authors":"J H Leo Breiman, R A Friedman, C J Olshen, Stone","year":"1984","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Breiman, 2001] Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.","title":"Leo Breiman. Random forests","authors":"Breiman","year":"2001","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen and Guestrin, 2016] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowl- edge discovery and data mining, pages 785-794, 2016.","title":"Xgboost: A scalable tree boosting system","authors":"Tianqi Chen, Carlos Guestrin","year":"2016","venue":"Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Demsar, 2006] Janez Demsar. Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res., 7:1-30, 2006.","title":"Statistical comparisons of classifiers over multiple data sets","authors":"Janez Demsar","year":"2006","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dorogush et al., 2018] Anna Veronika Dorogush, Vasily Er- shov, and Andrey Gulin. Catboost: gradient boost- ing with categorical features support. arXiv preprint arXiv:1810.11363, 2018.","title":"Catboost: gradient boosting with categorical features support","authors":"Anna Veronika Dorogush, Vasily Ershov, Andrey Gulin","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dua and Graff, 2017] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.","title":"Dheeru Dua and Casey Graff. UCI machine learning repository","authors":"Graff Dua","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Feurer et al., 2020] Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. Auto-sklearn 2.0: Hands-free automl via meta-learning. arXiv preprint arXiv:2007.04074, 2020.","title":"Auto-sklearn 2.0: Hands-free automl via meta-learning","authors":"Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, Frank Hutter","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Forman and others, 2003] George Forman et al. An exten- sive empirical study of feature selection metrics for text classification. J. Mach. Learn. Res., 3(Mar):1289-1305, 2003.","title":"An extensive empirical study of feature selection metrics for text classification","authors":"George Forman","year":"2003","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Friedman, 2001] Jerome H Friedman. Greedy function ap- proximation: a gradient boosting machine. Annals of statistics, pages 1189-1232, 2001.","title":"Greedy function approximation: a gradient boosting machine","authors":"Jerome H Friedman","year":"2001","venue":"Annals of statistics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gorishniy et al., 2021] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural In- formation Processing Systems, 34:18932-18943, 2021.","title":"Revisiting deep learning models for tabular data","authors":"Gorishniy","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Grinsztajn et al., 2022] Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still out- perform deep learning on tabular data? arXiv preprint arXiv:2207.08815, 2022.","title":"Why do tree-based models still outperform deep learning on tabular data","authors":"Léo Grinsztajn, Edouard Oyallon, Gaël Varoquaux","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hastie et al., 2001] Trevor Hastie, Jerome H. Friedman, and Robert Tibshirani. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Series in Statistics. Springer, 2001.","title":"The Elements of Statistical Learning: Data Mining, Inference, and Prediction","authors":"Trevor Hastie, Jerome H Friedman, Robert Tibshirani","year":"2001","venue":"Springer Series in Statistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ke et al., 2017] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boost- ing decision tree. Advances in neural information process- ing systems, 30, 2017.","title":"Lightgbm: A highly efficient gradient boosting decision tree","authors":"Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, Tie-Yan Liu","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kim and Loh, 2001] Hyunjoong Kim and Wei-Yin Loh. Classification trees with unbiased multiway splits. Journal of the American Statistical Association, 96(454):589-604, 2001.","title":"Classification trees with unbiased multiway splits","authors":"Loh ; ] Hyunjoong Kim, Wei-Yin Kim, Loh","year":"2001","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2019] Xiao Li, Yu Wang, Sumanta Basu, Karl Kumbier, and Bin Yu. A debiased mdi feature importance measure for random forests. Advances in Neural Informa- tion Processing Systems, 32, 2019.","title":"A debiased mdi feature importance measure for random forests","authors":"Xiao Li, Yu Wang, Sumanta Basu, Karl Kumbier, Bin Yu","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li, 2012] Ping Li. Robust logitboost and adaptive base class (abc) logitboost. arXiv preprint arXiv:1203.3491, 2012.","title":"Robust logitboost and adaptive base class (abc) logitboost","authors":"Ping Li","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Loh and Shih, 1997] Wei-Yin Loh and Yu-Shan Shih. Split selection methods for classification trees. Statistica sinica, pages 815-840, 1997.","title":"Split selection methods for classification trees","authors":"Wei-Yin Loh, Yu-Shan Shih","year":"1997","venue":"Statistica sinica","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Loh, 2009] Wei-Yin Loh. Improving the precision of clas- sification trees. The Annals of Applied Statistics, pages 1710-1737, 2009.","title":"Improving the precision of classification trees","authors":"Wei-Yin Loh","year":"2009","venue":"The Annals of Applied Statistics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lundberg et al., 2018] Scott M Lundberg, Gabriel G Erion, and Su-In Lee. Consistent individualized feature attribu- tion for tree ensembles. arXiv preprint arXiv:1802.03888, 2018.","title":"Consistent individualized feature attribution for tree ensembles","authors":"Scott M Lundberg, Su-In Gabriel G Erion, Lee","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nembrini et al., 2018] Stefano Nembrini, Inke R König, and Marvin N Wright. The revival of the gini importance? Bioinformatics, 34(21):3711-3718, 2018.","title":"The revival of the gini importance?","authors":"Stefano Nembrini, Marvin N Inke R König, Wright","year":"2018","venue":"Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Nicodemus, 2011] Kristin K Nicodemus. On the stabil- ity and ranking of predictors from random forest vari- able importance measures. Briefings in bioinformatics, 12(4):369-373, 2011.","title":"On the stability and ranking of predictors from random forest variable importance measures","authors":"Kristin K Nicodemus","year":"2011","venue":"Briefings in bioinformatics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Quinlan, 1986] J. Ross Quinlan. Induction of decision trees. Mach. Learn., 1(1):81-106, 1986.","title":"Induction of decision trees","authors":"J, Ross Quinlan","year":"1986","venue":"Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sandri and Zuccolotto, 2008] Marco Sandri and Paola Zuc- colotto. A bias correction algorithm for the gini vari- able importance measure in classification trees. Journal of Computational and Graphical Statistics, 17(3):611-628, 2008.","title":"A bias correction algorithm for the gini variable importance measure in classification trees","authors":"Marco Sandri, Paola Zuccolotto","year":"2008","venue":"Journal of Computational and Graphical Statistics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shwartz-Ziv and Armon, 2022] Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84-90, 2022.","title":"Tabular data: Deep learning is not all you need","authors":"Shwartz-Ziv, Armon ; Ravid Shwartz-Ziv, Amitai Armon","year":"2022","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Strobl et al., 2007] Carolin Strobl, Anne-Laure Boulesteix, Achim Zeileis, and Torsten Hothorn. Bias in random forest variable importance measures: Illustrations, sources and a solution. BMC bioinformatics, 8(1):1-21, 2007.","title":"Bias in random forest variable importance measures: Illustrations, sources and a solution","authors":"Carolin Strobl, Anne-Laure Boulesteix, Achim Zeileis, Torsten Hothorn","year":"2007","venue":"BMC bioinformatics","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Vanschoren et al., 2013] Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luís Torgo. Openml: networked science in machine learning. SIGKDD Explor., 15(2):49- 60, 2013.","title":"Openml: networked science in machine learning","authors":"Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, Luís Torgo","year":"2013","venue":"SIGKDD Explor","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wistuba et al., 2015] Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme. Learning hyperparameter opti- mization initializations. In 2015 IEEE international con- ference on data science and advanced analytics (DSAA), pages 1-10. IEEE, 2015.","title":"Learning hyperparameter optimization initializations","authors":"Martin Wistuba, Nicolas Schilling, Lars Schmidt-Thieme","year":"2015","venue":"IEEE international conference on data science and advanced analytics (DSAA)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhou and Hooker, 2021] Zhengze Zhou and Giles Hooker. Unbiased measurement of feature importance in tree- based methods. ACM Transactions on Knowledge Discov- ery from Data (TKDD), 15(2):1-21, 2021.","title":"Unbiased measurement of feature importance in treebased methods","authors":"Hooker Zhou, Zhengze Zhou, Giles Hooker","year":"2021","venue":"ACM Transactions on Knowledge Discovery from Data (TKDD)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}