{"type":"ScholarlyArticle","title":"Speeding up Very Fast Decision Tree with Low Computational Cost","authors":[{"name":"Jian Sun","slug":"jian-sun","given":"Jian","family":"Sun","affiliation":"School of Software , Tsinghua University , China"},{"name":"Hongyu Jia","slug":"hongyu-jia","given":"Hongyu","family":"Jia","affiliation":"School of Software , Tsinghua University , China"},{"name":"Bo Hu","slug":"bo-hu","given":"Bo","family":"Hu","affiliation":"Beijing University of Posts and Telecommunications , China"},{"name":"Xiao Huang","slug":"xiao-huang","given":"Xiao","family":"Huang","affiliation":"School of Software , Tsinghua University , China"},{"name":"Hao Zhang","slug":"hao-zhang","given":"Hao","family":"Zhang","affiliation":"School of Software , Tsinghua University , China"},{"name":"Hai Wan","slug":"hai-wan","given":"Hai","family":"Wan","affiliation":"School of Software , Tsinghua University , China"},{"name":"Xibin Zhao","slug":"xibin-zhao","given":"Xibin","family":"Zhao","affiliation":"School of Software , Tsinghua University , China"}],"abstract":"Very Fast Decision Tree (VFDT) is one of the most widely used online decision tree induction algorithms, and it provides high classification accuracy with theoretical guarantees. In VFDT, the split-attempt operation is essential for leaf-split. It is computation-intensive since it computes the heuristic measure of all attributes of a leaf. To reduce split-attempts, VFDT tries to split at constant intervals (for example, every 200 examples). However, this mechanism introduces split-delay for split can only happen at fixed intervals, which slows down the growth of VFDT and finally lowers accuracy. To address this problem, we first devise an online incremental algorithm that computes the heuristic measure of an attribute with a much lower computational cost. Then a subset of attributes is carefully selected to find a potential split timing using this algorithm. A split-attempt will be carried out once the timing is verified. By the whole process, computational cost and split-delay are lowered significantly. Comprehensive experiments are conducted using multiple synthetic and real datasets. Compared with state-of-the-art algorithms, our method reduces split-attempts by about 5 to 10 times on average with much lower split-delay, which makes our algorithm run faster and more accurate.","doi":"10.24963/ijcai.2020/177","url":"https://ijcai.org/proceedings/2020/177","pdf":"https://ijcai.org/proceedings/2020/0177.pdf","markdown":"https://ijcai.org/proceedings/2020/177.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/177","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Data Mining","track":"Main track","number":177,"pages":{"first":1272,"last":1278},"keywords":["Machine Learning: Classification","Data Mining: Mining Data Streams","Machine Learning: Online Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24154/file_24154.m3u8","length_sec":300,"provider":"hls"}],"references":[{"position":1,"raw":"[Agrawal et al., 1993] Rakesh Agrawal, Tomasz Imielinski, and Arun Swami. Database mining: A performance per- spective. IEEE transactions on knowledge and data engi- neering, 5(6):914-925, 1993.","title":"Database mining: A performance perspective","authors":"Tomasz Rakesh Agrawal, Arun Imielinski, Swami","year":"1993","venue":"IEEE transactions on knowledge and data engineering","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bifet and Gavaldà, 2009] Albert Bifet and Ricard Gavaldà. Adaptive learning from evolving data streams. In Inter- national Symposium on Intelligent Data Analysis, pages 249-260. Springer, 2009.","title":"Albert Bifet and Ricard Gavaldà. Adaptive learning from evolving data streams","authors":"Gavaldà Bifet","year":"2009","venue":"International Symposium on Intelligent Data Analysis","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bifet et al., 2010] Albert Bifet, Geoff Holmes, Richard Kirkby, and Bernhard Pfahringer. Moa: Massive on- line analysis. Journal of Machine Learning Research, 11(May):1601-1604, 2010.","title":"Moa: Massive online analysis","authors":"Bifet","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bifet et al., 2017] Albert Bifet, Jiajin Zhang, Wei Fan, Cheng He, Jianfeng Zhang, Jianfeng Qian, Geoff Holmes, and Bernhard Pfahringer. Extremely fast decision tree mining for evolving data streams. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowl- edge Discovery and Data Mining, pages 1733-1742. ACM, 2017.","title":"Extremely fast decision tree mining for evolving data streams","authors":"Albert Bifet, Jiajin Zhang, Wei Fan, Cheng He, Jianfeng Zhang, Jianfeng Qian, Geoff Holmes, Bernhard Pfahringer","year":"2017","venue":"Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Breiman et al., 1984] L Breiman, JH Friedman, R Olshen, and CJ Stone. Classification and regression trees. 1984.","title":"Classification and regression trees","authors":"Breiman, R Friedman, Olshen, Stone","year":"1984","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Das et al., 2019] Ariyam Das, Jin Wang, Sahil M Gandhi, Jae Lee, Wei Wang, and Carlo Zaniolo. Learn smart with less: building better online decision trees with fewer train- ing examples. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 2209- 2215. AAAI Press, 2019.","title":"Learn smart with less: building better online decision trees with fewer training examples","authors":"Das","year":"2019","venue":"Proceedings of the 28th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2019/306"},{"position":7,"raw":"[Dobre and Xhafa, 2014] Ciprian Dobre and Fatos Xhafa. Intelligent services for big data science. Future Genera- tion Computer Systems, 37:267-281, 2014.","title":"Intelligent services for big data science","authors":"Ciprian Dobre, Fatos Xhafa","year":"2014","venue":"Future Generation Computer Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Domingos and Hulten, 2000] Pedro Domingos and Geoff Hulten. Mining high-speed data streams. In Kdd, vol- ume 2, page 4, 2000.","title":"Mining high-speed data streams","authors":"Domingos, Hulten, Pedro Domingos, Geoff Hulten","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gama et al., 2003] João Gama, Ricardo Rocha, and Pedro Medas. Accurate decision trees for mining high-speed data streams. In Proceedings of the ninth ACM SIGKDD in- ternational conference on Knowledge discovery and data mining, pages 523-528. ACM, 2003.","title":"Accurate decision trees for mining high-speed data streams","authors":"João Gama, Ricardo Rocha, Pedro Medas","year":"2003","venue":"Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gama et al., 2004] Joao Gama, Pedro Medas, and Ricardo Rocha. Forest trees for on-line data. In Proceedings of the 2004 ACM symposium on Applied computing, pages 632-636. ACM, 2004.","title":"Forest trees for on-line data","authors":"Joao Gama, Pedro Medas, Ricardo Rocha","year":"2004","venue":"Proceedings of the 2004 ACM symposium on Applied computing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[García-Martín, 2017] Eva García-Martín. Extraction and energy efficient processing of streaming data. PhD thesis, Blekinge Tekniska Högskola, 2017.","title":"Extraction and energy efficient processing of streaming data","authors":"Eva García-Martín","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hoeffding, 1994] Wassily Hoeffding. Probability inequal- ities for sums of bounded random variables. In The Collected Works of Wassily Hoeffding, pages 409-426. Springer, 1994.","title":"Wassily Hoeffding. Probability inequalities for sums of bounded random variables","authors":"Hoeffding","year":"1994","venue":"The Collected Works of Wassily Hoeffding","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hulten et al., 2001] Geoff Hulten, Laurie Spencer, and Pe- dro Domingos. Mining time-changing data streams. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pages 97-106. ACM, 2001.","title":"Mining time-changing data streams","authors":"Geoff Hulten, Laurie Spencer, Pedro Domingos","year":"2001","venue":"Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ikonomovska et al., 2011a] Elena Ikonomovska, João Gama, and Sašo Džeroski. Learning model trees from evolving data streams. Data mining and knowledge discovery, 23(1):128-168, 2011.","title":"Elena Ikonomovska, João Gama, and Sašo Džeroski. Learning model trees from evolving data streams","authors":"Ikonomovska","year":"2011","venue":"Data mining and knowledge discovery","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ikonomovska et al., 2011b] Elena Ikonomovska, Joao Gama, Bernard Zenko, and Saso Dzeroski. Speeding-up hoeffding-based regression trees with options. In Pro- ceedings of the 28th International Conference on Machine Learning (ICML-11), pages 537-544.","title":"Speeding-up hoeffding-based regression trees with options","authors":"Ikonomovska","year":"2011","venue":"Proceedings of the 28th International Conference on Machine Learning (ICML-11)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kleiner et al., 2012] Ariel Kleiner, Ameet Talwalkar, Pur- namrita Sarkar, and Michael I Jordan. The big data boot- strap. 2012.","title":"The big data bootstrap","authors":"Ariel Kleiner, Ameet Talwalkar, Purnamrita Sarkar, Michael I Jordan","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Loosli et al., 2007] Gaëlle Loosli, Stéphane Canu, and Léon Bottou. Training invariant support vector machines using selective sampling. Large scale kernel machines, 2, 2007.","title":"Training invariant support vector machines using selective sampling","authors":"Gaëlle Loosli, Stéphane Canu, Léon Bottou","year":"2007","venue":"Large scale kernel machines","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Losing et al., 2018] Viktor Losing, Heiko Wersing, and Barbara Hammer. Enhancing very fast decision trees with local split-time predictions. In 2018 IEEE Interna- tional Conference on Data Mining (ICDM), pages 287- 296. IEEE, 2018.","title":"Enhancing very fast decision trees with local split-time predictions","authors":"Losing","year":"2018","venue":"IEEE International Conference on Data Mining (ICDM)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Manapragada et al., 2018] Chaitanya Manapragada, Geof- frey I Webb, and Mahsa Salehi. Extremely fast decision tree. In Proceedings of the 24th ACM SIGKDD Interna- tional Conference on Knowledge Discovery & Data Min- ing, pages 1953-1962. ACM, 2018.","title":"Extremely fast decision tree","authors":"Chaitanya Manapragada, Geoffrey I Webb, Mahsa Salehi","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Montiel et al., 2018] Jacob Montiel, Jesse Read, Albert Bifet, and Talel Abdessalem. Scikit-multiflow: a multi- output streaming framework. The Journal of Machine Learning Research, 19(1):2915-2914, 2018.","title":"Talel Abdessalem. Scikit-multiflow: a multioutput streaming framework","authors":"Montiel","year":"2018","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Quinlan, 1986] J. Ross Quinlan. Induction of decision trees. Machine learning, 1(1):81-106, 1986.","title":"Induction of decision trees","authors":"J, Ross Quinlan","year":"1986","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Quinlan, 1993] J Ross Quinlan. C 4.5: Programs for ma- chine learning. The Morgan Kaufmann Series in Machine Learning, San Mateo, CA: Morgan Kaufmann,-c1993, 1993.","title":"C 4.5: Programs for machine learning","authors":"Quinlan Ross","year":"1993","venue":"The Morgan Kaufmann Series in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schlimmer and Granger, 1986] Jeffrey C Schlimmer and Richard H Granger. Incremental learning from noisy data. Machine learning, 1(3):317-354, 1986.","title":"Incremental learning from noisy data","authors":"C Jeffrey, Richard H Schlimmer, Granger","year":"1986","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sovdat, 2014] Blaz Sovdat. Updating formulas and algo- rithms for computing entropy and gini index from time- changing data streams. arXiv preprint arXiv:1403.6348, 2014.","title":"Updating formulas and algorithms for computing entropy and gini index from timechanging data streams","authors":"Blaz Sovdat","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Street and Kim, 2001] W Nick Street and YongSeog Kim. A streaming ensemble algorithm (sea) for large-scale clas- sification. In Proceedings of the seventh ACM SIGKDD in- ternational conference on Knowledge discovery and data mining, pages 377-382. ACM, 2001.","title":"A streaming ensemble algorithm (sea) for large-scale classification","authors":"Nick Street, Yongseog Kim","year":"2001","venue":"Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Widmer and Kubat, 1996] Gerhard Widmer and Miroslav Kubat. Learning in the presence of concept drift and hid- den contexts. Machine learning, 23(1):69-101, 1996.","title":"Learning in the presence of concept drift and hidden contexts","authors":"Kubat Widmer, Gerhard Widmer, Miroslav Kubat","year":"1996","venue":"Machine learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}