{"type":"ScholarlyArticle","title":"AugBoost: Gradient Boosting Enhanced with Step-Wise Feature Augmentation","authors":[{"name":"Philip Tannor","slug":"philip-tannor","given":"Philip","family":"Tannor","affiliation":"Tel-Aviv University"},{"name":"Lior Rokach","slug":"lior-rokach","given":"Lior","family":"Rokach","affiliation":"Ben-Gurion University of the Negev"}],"abstract":"Gradient Boosted Decision Trees (GBDT) is a widely used machine learning algorithm, which obtains state-of-the-art results on many machine learning tasks. In this paper we introduce a method for obtaining better results, by augmenting the features in the dataset between the iterations of GBDT. We explore a number of augmentation methods: training an Artificial Neural Network (ANN) and extracting features from it's last hidden layer (supervised), and rotating the feature-space using unsupervised methods such as PCA or Random Projection (RP). These variations on GBDT were tested on 20 classification tasks, on which all of them outperformed GBDT and previous related work.","doi":"10.24963/ijcai.2019/493","url":"https://ijcai.org/proceedings/2019/493","pdf":"https://ijcai.org/proceedings/2019/0493.pdf","markdown":"https://ijcai.org/proceedings/2019/493.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/493","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":493,"pages":{"first":3555,"last":3561},"keywords":["Machine Learning: Dimensionality Reduction and Manifold Learning","Machine Learning: Ensemble Methods","Machine Learning: Classification","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Alcala-Fdez et al., 2011] Jesus Alcala-Fdez, Alberto Fer- nandez, Julian Luengo, Joaquin Derrac, and Salvador Gar- cia. Keel data-mining software tool: Data set repository, integration of algorithms and experimental analysis frame- work. Multiple-Valued Logic and Soft Computing, 17(2- 3):255-287, 2011.","title":"Jesus Alcala-Fdez, Alberto Fernandez, Julian Luengo, Joaquin Derrac, and Salvador Garcia. Keel data-mining software tool: Data set repository, integration of algorithms and experimental analysis framework","authors":"Alcala-Fdez","year":"2011","venue":"Multiple-Valued Logic and Soft Computing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Breiman, 2001] Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.","title":"Leo Breiman. Random forests","authors":"Breiman","year":"2001","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Burges, 2010] Christopher JC Burges. From ranknet to lambdarank to lambdamart: An overview. Learning, 11(23-581):81, 2010.","title":"From ranknet to lambdarank to lambdamart: An overview","authors":"J C Christopher, Burges","year":"2010","venue":"Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Casale et al., 2011] Pierluigi Casale, Oriol Pujol, and Petia Radeva. Embedding random projections in regularized gradient boosting machines. In Ensembles in Machine Learning Applications, pages 201-216. Springer, 2011.","title":"Embedding random projections in regularized gradient boosting machines","authors":"Pierluigi Casale, Oriol Pujol, Petia Radeva","year":"2011","venue":"Ensembles in Machine Learning Applications","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen and Guestrin, 2016] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowl- edge discovery and data mining, pages 785-794. ACM, 2016.","title":"Xgboost: A scalable tree boosting system","authors":"Tianqi Chen, Carlos Guestrin","year":"2016","venue":"Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2018] Hugh Chen, Scott Lundberg, and Su- In Lee. Hybrid gradient boosting trees and neuralnet- works for forecasting operating room data. arXiv preprint arXiv:1801.07384, 2018.","title":"Hybrid gradient boosting trees and neuralnetworks for forecasting operating room data","authors":"Hugh Chen, Scott Lundberg, Su-In Lee","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Demšar, 2006] Janez Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research, 7(Jan):1-30, 2006.","title":"Janez Demšar. Statistical comparisons of classifiers over multiple data sets","authors":"Demšar","year":"2006","venue":"Journal of Machine learning research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dheeru and Karra Taniskidou, 2017] Dua Dheeru and Efi Karra Taniskidou. UCI machine learning repository, 2017.","title":"Dua Dheeru and Efi Karra Taniskidou. UCI machine learning repository","authors":"Karra Dheeru, Taniskidou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Freund and Schapire, 1997] Yoav Freund and Robert E Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of com- puter and system sciences, 55(1):119-139, 1997.","title":"A decision-theoretic generalization of on-line learning and an application to boosting","authors":"Schapire Freund, Yoav Freund, Robert E Schapire","year":"1997","venue":"Journal of computer and system sciences","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Friedman, 2001] Jerome H Friedman. Greedy function ap- proximation: a gradient boosting machine. Annals of statistics, pages 1189-1232, 2001.","title":"Greedy function approximation: a gradient boosting machine","authors":"Jerome H Friedman","year":"2001","venue":"Annals of statistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Joly, 2017] Arnaud Joly. Exploiting random projections and sparsity with random forests and gradient boost- ing methods-application to multi-label and multi-output learning, random forest model compression and leveraging input sparsity. arXiv preprint arXiv:1704.08067, 2017.","title":"Exploiting random projections and sparsity with random forests and gradient boosting methods-application to multi-label and multi-output learning, random forest model compression and leveraging input sparsity","authors":"Arnaud Joly","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ke et al., 2017] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boost- ing decision tree. In Advances in Neural Information Pro- cessing Systems, pages 3146-3154, 2017.","title":"Lightgbm: A highly efficient gradient boosting decision tree","authors":"Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, Tie-Yan Liu","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kuncheva and Rodríguez, 2007] Ludmila I Kuncheva and Juan J Rodríguez. An experimental study on rotation forest ensembles. In International workshop on multiple classi- fier systems, pages 459-468. Springer, 2007.","title":"An experimental study on rotation forest ensembles","authors":"I Ludmila, Juan J Kuncheva, Rodríguez","year":"2007","venue":"International workshop on multiple classifier systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li, 2012] Ping Li. Robust logitboost and adaptive base class (abc) logitboost. arXiv preprint arXiv:1203.3491, 2012.","title":"Robust logitboost and adaptive base class (abc) logitboost","authors":"Ping Li","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pan et al., 2010] Sinno Jialin Pan, Qiang Yang, et al. A sur- vey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345-1359, 2010.","title":"A survey on transfer learning","authors":"Qiang Sinno Jialin Pan, Yang","year":"2010","venue":"IEEE Transactions on knowledge and data engineering","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pedregosa et al., 2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blon- del, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.","title":"Scikit-learn: Machine learning in Python","authors":"F Pedregosa, G Varoquaux, A Gramfort, V Michel, B Thirion, O Grisel, M Blondel, P Prettenhofer, R Weiss, V Dubourg, J Vanderplas, A Passos, D Cournapeau, M Brucher, M Perrot, E Duchesnay","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Prokhorenkova et al., 2017] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categor- ical features. arXiv preprint arXiv:1706.09516, 2017.","title":"Catboost: unbiased boosting with categorical features","authors":"Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, Andrey Gulin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Richardson et al., 2007] Matthew Richardson, Ewa Domi- nowska, and Robert Ragno. Predicting clicks: estimating the click-through rate for new ads. In Proceedings of the 16th international conference on World Wide Web, pages 521-530. ACM, 2007.","title":"Predicting clicks: estimating the click-through rate for new ads","authors":"Matthew Richardson, Ewa Dominowska, Robert Ragno","year":"2007","venue":"Proceedings of the 16th international conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rodriguez et al., 2006] Juan José Rodriguez, Ludmila I Kuncheva, and Carlos J Alonso. Rotation forest: A new classifier ensemble method. IEEE transactions on pattern analysis and machine intelligence, 28(10):1619- 1630, 2006.","title":"Rotation forest: A new classifier ensemble method","authors":"Rodriguez","year":"2006","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schclar and Rokach, 2009] Alon Schclar and Lior Rokach. Random projection ensemble classifiers. In International Conference on Enterprise Information Systems, pages 309-316. Springer, 2009.","title":"Random projection ensemble classifiers","authors":"Alon Schclar, Lior Rokach","year":"2009","venue":"International Conference on Enterprise Information Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Taieb and Hyndman, 2014] Souhaib Ben Taieb and Rob J Hyndman. A gradient boosting approach to the kaggle load forecasting competition. International journal of forecasting, 30(2):382-394, 2014.","title":"A gradient boosting approach to the kaggle load forecasting competition","authors":"Hyndman ; Taieb, Ben Souhaib, Rob J Taieb, Hyndman","year":"2014","venue":"International journal of forecasting","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang and Zhang, 2008] Chun-Xia Zhang and Jiang-She Zhang. Rotboost: A technique for combining rotation for- est and adaboost. Pattern recognition letters, 29(10):1524- 1536, 2008.","title":"Rotboost: A technique for combining rotation forest and adaboost","authors":"Chun-Xia Zhang, Jiang-She Zhang","year":"2008","venue":"Pattern recognition letters","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}