{"type":"ScholarlyArticle","title":"Gradient Boosting with Piece-Wise Linear Regression Trees","authors":[{"name":"Yu Shi","slug":"yu-shi","given":"Yu","family":"Shi","affiliation":"Institute for Interdisciplinary Information Sciences , Tsinghua University , Beijing , China"},{"name":"Jian Li","slug":"jian-li","given":"Jian","family":"Li","affiliation":"Institute for Interdisciplinary Information Sciences , Tsinghua University , Beijing , China"},{"name":"Zhize Li","slug":"zhize-li","given":"Zhize","family":"Li","affiliation":"Institute for Interdisciplinary Information Sciences , Tsinghua University , Beijing , China"}],"abstract":"Gradient Boosted Decision Trees (GBDT) is a very successful ensemble learning algorithm widely used across a variety of applications. Recently, several variants of GBDT training algorithms and implementations have been designed and heavily optimized in some very popular open sourced toolkits including XGBoost, LightGBM and CatBoost. \r\nIn this paper, we show that both the accuracy and efficiency of GBDT can be further enhanced by using more complex base learners. Specifically, we extend gradient boosting to use  piecewise linear regression trees (PL Trees), instead of piecewise constant regression trees, as base learners. We show that PL Trees can accelerate convergence of GBDT and improve the  accuracy. We also propose some optimization tricks to substantially reduce the training time of PL Trees, with little sacrifice of accuracy. Moreover, we propose several implementation techniques to speedup our algorithm on modern computer architectures with powerful Single Instruction Multiple Data (SIMD) parallelism. The experimental results show that GBDT with PL Trees can provide very competitive testing accuracy with comparable or less training time.","doi":"10.24963/ijcai.2019/476","url":"https://ijcai.org/proceedings/2019/476","pdf":"https://ijcai.org/proceedings/2019/0476.pdf","markdown":"https://ijcai.org/proceedings/2019/476.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/476","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":476,"pages":{"first":3432,"last":3438},"keywords":["Machine Learning: Ensemble Methods","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Breiman, 2017] Leo Breiman. Classification and Regres- sion Trees. Routledge, 2017.","title":"Classification and Regression Trees","authors":"Leo Breiman, Breiman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen and Guestrin, 2016] Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785-794. ACM, 2016.","title":"XGBoost: A scalable tree boosting system","authors":"Tianqi Chen, Carlos Guestrin","year":"2016","venue":"Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2012] Tianqi Chen, Linpeng Tang, Qin Liu, Diyi Yang, Saining Xie, Xuezhi Cao, Chunyang Wu, En- peng Yao, Zhengyang Liu, Zhansheng Jiang, et al. Com- bining factorization model and additive forest for collabo- rative followee recommendation. KDD CUP, 2012.","title":"Combining factorization model and additive forest for collaborative followee recommendation","authors":"Tianqi Chen, Linpeng Tang, Qin Liu, Diyi Yang, Saining Xie, Xuezhi Cao, Chunyang Wu, Enpeng Yao, Zhengyang Liu, Zhansheng Jiang","year":"2012","venue":"KDD CUP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dobra and Gehrke, 2002] Alin Dobra and Johannes Gehrke. Secret: a scalable linear regression tree algorithm. In Pro- ceedings of the 8th ACM SIGKDD International Confer- ence on Knowledge Discovery and Data Mining, pages 481-487. ACM, 2002.","title":"Secret: a scalable linear regression tree algorithm","authors":"Alin Dobra, Johannes Gehrke","year":"2002","venue":"Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Espasa et al., 1998] Roger Espasa, Mateo Valero, and James E Smith. Vector architectures: past, present and fu- ture. In Proceedings of the 12th International Conference on Supercomputing, pages 425-432. ACM, 1998.","title":"Vector architectures: past, present and future","authors":"Espasa","year":"1998","venue":"Proceedings of the 12th International Conference on Supercomputing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Friedman et al., 2000] Jerome Friedman, Trevor Hastie, Robert Tibshirani, et al. Additive logistic regression: a sta- tistical view of boosting (with discussion and a rejoinder by the authors). The Annals of Statistics, 28(2):337-407, 2000.","title":"Additive logistic regression: a statistical view of boosting (with discussion and a rejoinder by the authors)","authors":"Friedman","year":"2000","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Friedman, 1979] Jerome H Friedman. A tree-structured ap- proach to nonparametric multiple regression. Smoothing Techniques for Curve Estimation, 757:5-22, 1979.","title":"A tree-structured approach to nonparametric multiple regression","authors":"Jerome H Friedman","year":"1979","venue":"Smoothing Techniques for Curve Estimation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Friedman, 2001] Jerome H Friedman. Greedy function ap- proximation: a gradient boosting machine. Annals of Statistics, pages 1189-1232, 2001.","title":"Greedy function approximation: a gradient boosting machine","authors":"Jerome H Friedman","year":"2001","venue":"Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hall et al., 2009] Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H Witten. The weka data mining software: an update. ACM SIGKDD Explorations Newsletter, 11(1):10-18, 2009.","title":"The weka data mining software: an update","authors":"Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, Ian H Witten","year":"2009","venue":"ACM SIGKDD Explorations Newsletter","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ke et al., 2017] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boost- ing decision tree. In Advances in Neural Information Pro- cessing Systems, pages 3149-3157, 2017.","title":"Lightgbm: A highly efficient gradient boosting decision tree","authors":"Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, Tie-Yan Liu","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kuhn et al., 2012] Max Kuhn, Steve Weston, Chris Keefer, and Nathan Coulter. Cubist models for regression. R pack- age Vignette R package version 0.0, 18, 2012.","title":"Cubist models for regression. R package Vignette R package version 0","authors":"Kuhn","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Prokhorenkova et al., 2018] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categor- ical features. In Advances in Neural Information Process- ing Systems, pages 6639-6649, 2018.","title":"Catboost: unbiased boosting with categorical features","authors":"Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, Andrey Gulin","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Quinlan and others, 1992] John R Quinlan et al. Learning with continuous classes. In 5th Australian Joint Confer- ence on Artificial Intelligence, volume 92, pages 343-348. World Scientific, 1992.","title":"Learning with continuous classes","authors":"John R Quinlan","year":"1992","venue":"5th Australian Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Quinlan, 1986] J. Ross Quinlan. Induction of decision trees. Machine Learning, 1(1):81-106, 1986.","title":"Induction of decision trees","authors":"J, Ross Quinlan","year":"1986","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Quinlan, 2014] J Ross Quinlan. C4. 5: programs for ma- chine learning. Elsevier, 2014.","title":"C4. 5: programs for machine learning","authors":"Quinlan Ross","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sun et al., 2014] Peng Sun, Tong Zhang, and Jie Zhou. A convergence rate analysis for logitboost, mart and their variant. In ICML, pages 1251-1259, 2014.","title":"A convergence rate analysis for logitboost, mart and their variant","authors":"Peng Sun, Tong Zhang, Jie Zhou","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tyree et al., 2011] Stephen Tyree, Kilian Q Weinberger, Kunal Agrawal, and Jennifer Paykin. Parallel boosted regression trees for web search ranking. In Proceedings of the 20th International Conference on World Wide Web, pages 387-396. ACM, 2011.","title":"Parallel boosted regression trees for web search ranking","authors":"Stephen Tyree, Q Kilian, Kunal Weinberger, Jennifer Agrawal, Paykin","year":"2011","venue":"Proceedings of the 20th International Conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vens and Blockeel, 2006] Celine Vens and Hendrik Block- eel. A simple regression based heuristic for learning model trees. Intelligent Data Analysis, 10(3):215-236, 2006.","title":"Celine Vens and Hendrik Blockeel. A simple regression based heuristic for learning model trees","authors":"Blockeel Vens","year":"2006","venue":"Intelligent Data Analysis","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vogel et al., 2007] David S Vogel, Ognian Asparouhov, and Tobias Scheffer. Scalable look-ahead linear regression trees. In Proceedings of the 13th ACM SIGKDD Inter- national Conference on Knowledge Discovery and Data Mining, pages 757-764. ACM, 2007.","title":"Scalable look-ahead linear regression trees","authors":"S David, Ognian Vogel, Tobias Asparouhov, Scheffer","year":"2007","venue":"Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang and Hastie, 2014] Jianqiang C Wang and Trevor Hastie. Boosted varying-coefficient regression models for product demand prediction. Journal of Computational and Graphical Statistics, 23(2):361-382, 2014.","title":"Boosted varying-coefficient regression models for product demand prediction","authors":"C Jianqiang, Trevor Wang, Hastie","year":"2014","venue":"Journal of Computational and Graphical Statistics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang and Witten, 1997] Y. Wang and I. H. Witten. Induc- tion of model trees for predicting continuous classes. In Poster papers of the 9th European Conference on Machine Learning. Springer, 1997.","title":"Induction of model trees for predicting continuous classes","authors":"Y Wang, I H Witten","year":"1997","venue":"Poster papers of the 9th European Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2014] Endong Wang, Qing Zhang, Bo Shen, Guangyong Zhang, Xiaowei Lu, Qing Wu, and Yajuan Wang. Intel math kernel library. In High-Performance Computing on the Intel R Xeon Phi TM , pages 167-188. Springer, 2014.","title":"Intel math kernel library","authors":"Endong Wang, Qing Zhang, Bo Shen, Guangyong Zhang, Xiaowei Lu, Qing Wu, Yajuan Wang","year":"2014","venue":"High-Performance Computing on the Intel R Xeon Phi TM","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wen et al., 2018] Zeyi Wen, Bingsheng He, Ramamoha- narao Kotagiri, Shengliang Lu, and Jiashuai Shi. Efficient gradient boosted decision tree training on GPUs. In 2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 234-243. IEEE, 2018.","title":"Efficient gradient boosted decision tree training on GPUs","authors":"Zeyi Wen, Bingsheng He, Ramamohanarao Kotagiri, Shengliang Lu, Jiashuai Shi","year":"2018","venue":"IEEE International Parallel and Distributed Processing Symposium (IPDPS)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2018] Huan Zhang, Si Si, and Cho-Jui Hsieh. GPU-acceleration for large-scale tree boosting. In SysML Conference, 2018.","title":"GPU-acceleration for large-scale tree boosting","authors":"Huan Zhang, Si Si, Cho-Jui Hsieh","year":"2018","venue":"SysML Conference","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}