{"type":"ScholarlyArticle","title":"Learn Smart with Less: Building Better Online Decision Trees with Fewer Training Examples","authors":[{"name":"Ariyam Das","slug":"ariyam-das","given":"Ariyam","family":"Das","affiliation":"Department of Computer Science , University of California , Los Angeles"},{"name":"Jin Wang","slug":"jin-wang","given":"Jin","family":"Wang","affiliation":"Department of Computer Science , University of California , Los Angeles"},{"name":"Sahil M. Gandhi","slug":"sahil-m-gandhi","given":"Sahil M.","family":"Gandhi","affiliation":"Department of Computer Science , University of California , Los Angeles"},{"name":"Jae Lee","slug":"jae-lee","given":"Jae","family":"Lee","affiliation":"Department of Computer Science , University of California , Los Angeles"},{"name":"Wei Wang","slug":"wei-wang","given":"Wei","family":"Wang","affiliation":"Department of Computer Science , University of California , Los Angeles"},{"name":"Carlo Zaniolo","slug":"carlo-zaniolo","given":"Carlo","family":"Zaniolo","affiliation":"Department of Computer Science , University of California , Los Angeles"}],"abstract":"Online  decision  tree  models  are  extensively  used in  many  industrial  machine  learning  applications for real-time classification tasks. These models are highly accurate, scalable and easy to use in practice. The Very Fast Decision Tree (VFDT) is the classic  online  decision  tree  induction  model  that has been widely adopted due to its theoretical guarantees  as  well  as  competitive  performance.  However, VFDT and its variants solely rely on conservative statistical measures like Hoeffding bound to incrementally grow the tree. This makes these models  extremely  circumspect  and  limits  their  ability to  learn  fast.  In  this  paper,  we  efficiently  employ statistical  resampling  techniques  to  build  an online tree faster using fewer examples. We first theoretically show that a naive implementation of resampling techniques like non-parametric bootstrap does not scale due to large memory and computational overheads. We  mitigate  this  by  proposing  a  robust  memory-efficient bootstrap simulation heuristic (Mem-ES) that  successfully  expedites  the  learning  process. Experimental  results  on  both  synthetic  data  and large-scale real world datasets demonstrate the efficiency  and  effectiveness  of  our  proposed  technique.","doi":"10.24963/ijcai.2019/306","url":"https://ijcai.org/proceedings/2019/306","pdf":"https://ijcai.org/proceedings/2019/0306.pdf","markdown":"https://ijcai.org/proceedings/2019/306.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/306","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":306,"pages":{"first":2209,"last":2215},"keywords":["Machine Learning: Data Mining","Machine Learning: Online Learning","Machine Learning: Time-series;Data Streams","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Abuzaid et al., 2016] F. Abuzaid, J. K. Bradley, F. T. Liang, A. Feng, L. Yang, M. Zaharia, and A. Talwalkar. Yg- gdrasil: An optimized system for training deep decision trees at scale. In NIPS, pages 3817-3825. 2016.","title":"Yggdrasil: An optimized system for training deep decision trees at scale","authors":"F Abuzaid, J K Bradley, F T Liang, A Feng, L Yang, M Zaharia, A Talwalkar","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aggarwal, 2006] Charu C. Aggarwal. Data Streams: Mod- els and Algorithms (Advances in Database Systems). Springer-Verlag, 2006.","title":"Data Streams: Models and Algorithms","authors":"C Charu, Aggarwal","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ao et al., 2018] X. Ao, P. Luo, J. Wang, F. Zhuang, and Q. He. Mining precise-positioning episode rules from event sequences. IEEE TKDE, 2018.","title":"Mining precise-positioning episode rules from event sequences","authors":"Ao","year":"2018","venue":"IEEE TKDE","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ao et al., 2019] X. Ao, H. Shi, J. Wang, L. Zuo, H. Li, and Q. He. Large-scale frequent episode mining from complex event sequences with hierarchies. ACM TIST, 2019.","title":"Large-scale frequent episode mining from complex event sequences with hierarchies","authors":"Ao","year":"2019","venue":"ACM TIST","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bickel and Freedman, 1981] Peter J. Bickel and David A. Freedman. Some asymptotic theory for the bootstrap. The Annals of Statistics, 9(6):1196-1217, 11 1981.","title":"Some asymptotic theory for the bootstrap","authors":"Peter J Bickel, David A Freedman","year":"1981","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bickel et al., 2012] Peter J Bickel, Friedrich Götze, and Willem R van Zwet. Resampling fewer than n observa- tions: gains, losses, and remedies for losses. In Selected works of Willem van Zwet, pages 267-297. Springer, 2012.","title":"Resampling fewer than n observations: gains, losses, and remedies for losses","authors":"J Peter, Friedrich Bickel, Willem R Götze, Van Zwet","year":"2012","venue":"Selected works of Willem van Zwet","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bifet et al., 2010] A. Bifet, G. Holmes, R. Kirkby, and B. Pfahringer. MOA: massive online analysis. Journal of Machine Learning Research, 11:1601-1604, 2010.","title":"MOA: massive online analysis","authors":"A Bifet, G Holmes, R Kirkby, B Pfahringer","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bifet et al., 2017] A. Bifet, J. Zhang, W. Fan, C. He, J. Zhang, J. Qian, G. Holmes, and B. Pfahringer. Ex- tremely fast decision tree mining for evolving data streams. In SIGKDD, pages 1733-1742, 2017.","title":"Extremely fast decision tree mining for evolving data streams","authors":"A Bifet, J Zhang, W Fan, C He, J Zhang, J Qian, G Holmes, B Pfahringer","year":"2017","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Breiman et al., 1984] Leo Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone. Classification and Regression Trees. Statistics/Probability Series. Wadsworth Publishing Company, Belmont, California, U.S.A., 1984.","title":"Classification and Regression Trees","authors":"J H Leo Breiman, R A Friedman, C J Olshen, Stone","year":"1984","venue":"Statistics/Probability Series","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen and He, 2014] Tianqi Chen and Tong He. Higgs bo- son discovery with boosted trees. In HEPML@NIPS, pages 69-80, 2014.","title":"Higgs boson discovery with boosted trees","authors":"Tianqi Chen, Tong He","year":"2014","venue":"HEPML@NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Das and Zaniolo, 2016] Ariyam Das and Carlo Zaniolo. Fast lossless frequent itemset mining in data streams us- ing crucial patterns. In SDM, pages 576-584, 2016.","title":"Ariyam Das and Carlo Zaniolo. Fast lossless frequent itemset mining in data streams using crucial patterns","authors":"Zaniolo Das","year":"2016","venue":"SDM","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Domingos and Hulten, 2000] Pedro M. Domingos and Ge- off Hulten. Mining high-speed data streams. In SIGKDD, pages 71-80, 2000.","title":"Mining high-speed data streams","authors":"Pedro M Domingos, Geoff Hulten","year":"2000","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Fan, 2004] Wei Fan. Systematic data selection to mine concept-drifting data streams. In SIGKDD, 2004.","title":"Systematic data selection to mine concept-drifting data streams","authors":"Wei Fan","year":"2004","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Freitas, 2014] Alex A. Freitas. Comprehensible classifica- tion models: A position paper. SIGKDD Explor. Newsl., 15(1):1-10, 2014.","title":"Comprehensible classification models: A position paper","authors":"Alex A Freitas","year":"2014","venue":"SIGKDD Explor. Newsl","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gine and Zinn, 1990] Evarist Gine and Joel Zinn. Boot- strapping general empirical measures. The Annals of Prob- ability, 18(2):851-869, 04 1990.","title":"Bootstrapping general empirical measures","authors":"Gine, Zinn, Evarist Gine, Joel Zinn","year":"1990","venue":"The Annals of Probability","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Gu et al., 2016] Jiaqi Gu, Jin Wang, and Carlo Zaniolo. Ranking support for matched patterns over complex event streams: The CEPR system. In ICDE, 2016. Haghi et al., 2017]","title":"Ranking support for matched patterns over complex event streams: The CEPR system","authors":"Jiaqi Gu, Jin Wang, Carlo Zaniolo","year":"2016","venue":"ICDE","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"Mostafa Haghi, Kerstin Thurow, and Regina Stoll. Wearable devices in medical internet of things: scientific research and commercially available de- vices. Healthcare informatics research, 23(1):4-15, 2017.","title":"Wearable devices in medical internet of things: scientific research and commercially available devices","authors":"Mostafa Haghi, Kerstin Thurow, Regina Stoll","year":"2017","venue":"Healthcare informatics research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Hall, 2013] Peter Hall. The bootstrap and Edgeworth ex- pansion. Springer Science & Business Media, 2013.","title":"The bootstrap and Edgeworth expansion","authors":"Peter Hall","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Hulten et al., 2001] Geoff Hulten, Laurie Spencer, and Pe- dro M. Domingos. Mining time-changing data streams. In SIGKDD, pages 97-106, 2001.","title":"Mining time-changing data streams","authors":"Geoff Hulten, Laurie Spencer, Pedro M Domingos","year":"2001","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Jin and Agrawal, 2003a] Ruoming Jin and Gagan Agrawal. Communication and memory efficient parallel decision tree construction. In SDM, pages 119-129, 2003.","title":"Communication and memory efficient parallel decision tree construction","authors":"Jin, Agrawal ;, Ruoming Jin, Gagan Agrawal","year":"2003","venue":"SDM","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Jin and Agrawal, 2003b] Ruoming Jin and Gagan Agrawal. Efficient decision tree construction on streaming data. In SIGKDD, pages 571-576, 2003.","title":"Efficient decision tree construction on streaming data","authors":"Jin, Agrawal ;, Ruoming Jin, Gagan Agrawal","year":"2003","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Kargupta et al., 2002] H. Kargupta, B.-H. Park, S. Pittie, L. Liu, D. Kushraj, and K. Sarkar. Mobimine: Monitor- ing the stock market from a pda. SIGKDD Explor. Newsl., 3(2):37-46, 2002.","title":"Mobimine: Monitoring the stock market from a pda","authors":"H Kargupta, B.-H Park, S Pittie, L Liu, D Kushraj, K Sarkar","year":"2002","venue":"SIGKDD Explor. Newsl","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Kargupta et al., 2004] H. Kargupta, R. Bhargava, K. Liu, M. Powers, P. Blair, S. Bushra, J. Dull, K. Sarkar, M. Klein, M. Vasa, and D. Handy. Vedas: A mobile and distributed data stream mining system for real-time vehicle monitoring. In SDM, 2004.","title":"Vedas: A mobile and distributed data stream mining system for real-time vehicle monitoring","authors":"H Kargupta, R Bhargava, K Liu, M Powers, P Blair, S Bushra, J Dull, K Sarkar, M Klein, M Vasa, D Handy","year":"2004","venue":"SDM","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Kingsford and Salzberg, 2008] Carl Kingsford and Steven L. Salzberg. What are decision trees? Na- ture Biotechnology, 26(1011), 2008.","title":"What are decision trees?","authors":"Carl Kingsford, Steven L Salzberg","year":"1011","venue":"Nature Biotechnology","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Kleiner et al., 2012] Ariel Kleiner, Ameet Talwalkar, Pur- namrita Sarkar, and Michael I. Jordan. The big data boot- strap. In ICML, 2012.","title":"The big data bootstrap","authors":"Ariel Kleiner, Ameet Talwalkar, Purnamrita Sarkar, Michael I Jordan","year":"2012","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Kwapisz et al., 2011] Jennifer R. Kwapisz, Gary M. Weiss, and Samuel A. Moore. Activity recognition using cell phone accelerometers. SIGKDD Explor. Newsl., 12(2):74- 82, 2011.","title":"Activity recognition using cell phone accelerometers","authors":"Jennifer R Kwapisz, Gary M Weiss, Samuel A Moore","year":"2011","venue":"SIGKDD Explor. Newsl","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Losing et al., 2018] Viktor Losing, Heiko Wersing, and Barbara Hammer. Enhancing very fast decision trees with local split-time predictions. In ICDM, pages 287-296, 2018.","title":"Enhancing very fast decision trees with local split-time predictions","authors":"Losing","year":"2018","venue":"ICDM","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Manapragada et al., 2018] Chaitanya Manapragada, Geof- frey I. Webb, and Mahsa Salehi. Extremely fast decision tree. In SIGKDD, pages 1953-1962, 2018.","title":"Extremely fast decision tree","authors":"Chaitanya Manapragada, Geoffrey I Webb, Mahsa Salehi","year":"2018","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Politis et al., 1999] Dimitris Politis, Joseph P Romano, and Michael Wolf. Weak convergence of dependent empir- ical measures with application to subsampling in func- tion spaces. Journal of statistical planning and inference, 79(2):179-190, 1999.","title":"Weak convergence of dependent empirical measures with application to subsampling in function spaces","authors":"Dimitris Politis, Joseph P Romano, Michael Wolf","year":"1999","venue":"Journal of statistical planning and inference","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Quinlan, 1993] J. Ross Quinlan. C4.5: Programs for Ma- chine Learning. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1993.","title":"C4.5: Programs for Machine Learning","authors":"J, Ross Quinlan","year":"1993","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2003] Haixun Wang, Wei Fan, Philip S. Yu, and Jiawei Han. Mining concept-drifting data streams using ensemble classifiers. In SIGKDD, pages 226-235, 2003.","title":"Mining concept-drifting data streams using ensemble classifiers","authors":"Haixun Wang, Wei Fan, Philip S Yu, Jiawei Han","year":"2003","venue":"SIGKDD","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}