{"type":"ScholarlyArticle","title":"Request-and-Reverify: Hierarchical Hypothesis Testing for Concept Drift Detection with Expensive Labels","authors":[{"name":"Shujian Yu","slug":"shujian-yu","given":"Shujian","family":"Yu","affiliation":"Nokia Bell Labs , Murray Hill , NJ , USA; Dept. of Electrical and Computer Engineering , University of Florida , Gainesville , FL , USA"},{"name":"Xiaoyang Wang","slug":"xiaoyang-wang","given":"Xiaoyang","family":"Wang","affiliation":"Nokia Bell Labs , Murray Hill , NJ , USA"},{"name":"José C. Príncipe","slug":"jose-c-principe","given":"José C.","family":"Príncipe","affiliation":"Dept. of Electrical and Computer Engineering , University of Florida , Gainesville , FL , USA"}],"abstract":"One important assumption underlying common classification models is the stationarity of the data. However, in real-world streaming applications, the data concept indicated by the joint distribution of feature and label is not stationary but drifting over time. Concept drift detection aims to detect such drifts and adapt the model so as to mitigate any deterioration in the model's predictive performance. Unfortunately, most existing concept drift detection methods rely on a strong and over-optimistic condition that the true labels are available immediately for all already classified instances. In this paper, a novel Hierarchical Hypothesis Testing framework with Request-and-Reverify strategy is developed to detect concept drifts by requesting labels only when necessary. Two methods, namely Hierarchical Hypothesis Testing with Classification Uncertainty (HHT-CU) and Hierarchical Hypothesis Testing with Attribute-wise \"Goodness-of-fit\" (HHT-AG), are proposed respectively under the novel framework. In experiments with benchmark datasets, our methods demonstrate overwhelming advantages over state-of-the-art unsupervised drift detectors. More importantly, our methods even outperform DDM (the widely used supervised drift detector) when we use significantly fewer labels.","doi":"10.24963/ijcai.2018/421","url":"https://ijcai.org/proceedings/2018/421","pdf":"https://ijcai.org/proceedings/2018/0421.pdf","markdown":"https://ijcai.org/proceedings/2018/421.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/421","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":421,"pages":{"first":3033,"last":3039},"keywords":["Machine Learning: Time-series;Data Streams","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Alippi et al., 2017] Cesare Alippi, Giacomo Boracchi, and Manuel Roveri. Hierarchical change-detection tests. IEEE Trans. Neural Netw. Learn. Syst., 28(2):246-258, 2017.","title":"Hierarchical change-detection tests","authors":"Cesare Alippi, Giacomo Boracchi, Manuel Roveri","year":"2017","venue":"IEEE Trans. Neural Netw. Learn. Syst","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baena-García et al., 2006] Manuel Baena-García, José del Campo-Ávila, et al. Early drift detection method. In Int. Workshop Knowledge Discovery from Data Streams, 2006.","title":"Early drift detection method","authors":"Manuel Baena-García, José Del Campo-Ávila","year":"2006","venue":"Int. Workshop Knowledge Discovery from Data Streams","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bishop, 2006] Christopher M Bishop. Pattern Recognition and Machine Learning. Springer, 2006.","title":"Pattern Recognition and Machine Learning","authors":"M Christopher, Bishop","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brzezinski and Stefanowski, 2014] Dariusz Brzezinski and Jerzy Stefanowski. Prequential auc for classifier evalua- tion and drift detection in evolving data streams. In Work- shop New Frontiers in Mining Complex Patterns, 2014.","title":"Prequential auc for classifier evaluation and drift detection in evolving data streams","authors":"Dariusz Brzezinski, Jerzy Stefanowski","year":"2014","venue":"Workshop New Frontiers in Mining Complex Patterns","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ditzler and Polikar, 2011] Greg Ditzler and Robi Polikar. Hellinger distance based drift detection for nonstationary environments. In IEEE Symp. CIDUE, pages 41-48, 2011.","title":"Hellinger distance based drift detection for nonstationary environments","authors":"Greg Ditzler, Robi Polikar","year":"2011","venue":"IEEE Symp. CIDUE","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ditzler et al., 2015] Greg Ditzler, Manuel Roveri, et al. Learning in nonstationary environments: A survey. IEEE Comput. Intell. Mag., 10(4):12-25, 2015.","title":"Learning in nonstationary environments: A survey","authors":"Greg Ditzler, Manuel Roveri","year":"2015","venue":"IEEE Comput. Intell. Mag","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dries and Rückert, 2009] Anton Dries and Ulrich Rückert. Adaptive concept drift detection. Statistical Anal. Data Mining: The ASA Data Sci. J., 2(5-6):311-327, 2009.","title":"Adaptive concept drift detection","authors":"Rückert Dries, Anton Dries, Ulrich Rückert","year":"2009","venue":"Statistical Anal. Data Mining: The ASA Data Sci. J","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dyer et al., 2014] Karl B Dyer, Robert Capo, and Robi Po- likar. Compose: A semisupervised learning framework for initially labeled nonstationary streaming data. IEEE Trans. Neural Netw. Learn. Syst., 25(1):12-26, 2014.","title":"Compose: A semisupervised learning framework for initially labeled nonstationary streaming data","authors":"Karl B Dyer, Robert Capo, Robi Polikar","year":"2014","venue":"IEEE Trans. Neural Netw. Learn. Syst","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Frías-Blanco et al., 2015] Isvani Frías-Blanco, José del Campo-Ávila, et al. Online and non-parametric drift de- tection methods based on hoeffding's bounds. IEEE Trans. Knowl. Data Eng., 27(3):810-823, 2015.","title":"Isvani Frías-Blanco, José del Campo-Ávila, et al. Online and non-parametric drift detection methods based on hoeffding's bounds","authors":"Frías-Blanco","year":"2015","venue":"IEEE Trans. Knowl. Data Eng","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gama et al., 2004] Joao Gama, Pedro Medas, et al. Learn- ing with drift detection. In Brazilian Symp. on Artificial Intelligence, pages 286-295. Springer, 2004.","title":"Learning with drift detection","authors":"Joao Gama, Pedro Medas","year":"2004","venue":"Brazilian Symp. on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gonc ¸alves et al., 2014] Paulo Gonc ¸alves, Silas de Car- valho Santos, et al. A comparative study on concept drift detectors. Expert Syst. Appl., 41(18):8144-8156, 2014.","title":"Paulo Gonc ¸alves, Silas de Carvalho Santos, et al. A comparative study on concept drift detectors","authors":"Gonc ¸alves","year":"2014","venue":"Expert Syst. Appl","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hoeffding, 1963] Wassily Hoeffding. Probability inequali- ties for sums of bounded random variables. Journal of the American statistical association, 58(301):13-30, 1963.","title":"Wassily Hoeffding. Probability inequalities for sums of bounded random variables","authors":"Hoeffding","year":"1963","venue":"Journal of the American statistical association","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hoens et al., 2012] T. Hoens, R. Polikar, and N. Chawla. Learning from streaming data with concept drift and im- balance. Progress in Artificial Intell., 1(1):89-101, 2012.","title":"Learning from streaming data with concept drift and imbalance","authors":"T Hoens, R Polikar, N Chawla","year":"2012","venue":"Progress in Artificial Intell","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kaluža et al., 2010] Boštjan Kaluža, Violeta Mirchevska, et al. An agent-based approach to care in independent liv- ing. Ambient intelligence, pages 177-186, 2010.","title":"An agent-based approach to care in independent living","authors":"Boštjan Kaluža, Violeta Mirchevska","year":"2010","venue":"Ambient intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Katakis et al., 2008] I.","title":null,"authors":"Katakis","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"Katakis, G. Tsoumakas, and I. Vla- havas. An ensemble of classifiers for coping with recurring contexts in data streams. In ECAI, pages 763-764, 2008.","title":"An ensemble of classifiers for coping with recurring contexts in data streams","authors":"G Katakis, I Tsoumakas, Vlahavas","year":"2008","venue":"ECAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kelly et al., 1999] Mark G Kelly, David J Hand, and Niall M Adams. The impact of changing populations on classifier performance. In KDD, pages 367-371, 1999.","title":"The impact of changing populations on classifier performance","authors":"G Mark, David J Kelly, Niall M Hand, Adams","year":"1999","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kifer et al., 2004] Daniel Kifer, Shai Ben-David, and Jo- hannes Gehrke. Detecting change in data streams. In Int. Conf. on Very Large Data Bases, pages 180-191, 2004.","title":"Detecting change in data streams","authors":"Daniel Kifer, Shai Ben-David, Johannes Gehrke","year":"2004","venue":"Int. Conf. on Very Large Data Bases","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kim and Park, 2017] Youngin Kim and Cheong Hee Park. An efficient concept drift detection method for stream- ing data under limited labeling. IEICE Trans. Inf. Syst., 100(10):2537-2546, 2017.","title":"An efficient concept drift detection method for streaming data under limited labeling","authors":"Park ; Kim, Youngin Kim, Cheong Hee, Park","year":"2017","venue":"IEICE Trans. Inf. Syst","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Krawczyk et al., 2017] Bartosz Krawczyk, Leandro L Minku, et al. Ensemble learning for data stream analysis: a survey. Information Fusion, 37:132-156, 2017.","title":"Ensemble learning for data stream analysis: a survey","authors":"Bartosz Krawczyk, Leandro L Minku","year":"2017","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lindstrom et al., 2011] Patrick Lindstrom, Brian Namee, and Sarah Delany. Drift detection using uncertainty dis- tribution divergence. In ICDM Workshops. IEEE, 2011.","title":"Drift detection using uncertainty distribution divergence","authors":"Patrick Lindstrom, Brian Namee, Sarah Delany","year":"2011","venue":"ICDM Workshops. IEEE","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Montgomery, 2009] Douglas Montgomery. Introduction to statistical quality control. John Wiley & Sons, 2009.","title":"Introduction to statistical quality control","authors":"Douglas Montgomery","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Peacock, 1983] JA Peacock. Two-dimensional goodness- of-fit testing in astronomy. Monthly Notices of the Royal Astronomical Society, 202(3):615-627, 1983.","title":"Two-dimensional goodnessof-fit testing in astronomy","authors":"; Ja Peacock, Peacock","year":"1983","venue":"Monthly Notices of the Royal Astronomical Society","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Reis et al., 2016] Denis dos Reis, Peter Flach, et al. Fast unsupervised online drift detection using incremental kolmogorov-smirnov test. In KDD, 2016.","title":"Fast unsupervised online drift detection using incremental kolmogorov-smirnov test","authors":"Peter Denis Dos Reis, Flach","year":"2016","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sethi and Kantardzic, 2017] T. Sethi and M. Kantardzic. On the reliable detection of concept drift from streaming un- labeled data. Expert Syst. Appl., 82:77-99, 2017.","title":"On the reliable detection of concept drift from streaming unlabeled data","authors":"T Sethi, M Kantardzic","year":"2017","venue":"Expert Syst. Appl","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sklar, 1959] M Sklar. Fonctions de répartition à n dimen- sions et leurs marges. Publ. Inst. Statist. Univ. Paris, 8:229-231, 1959.","title":"Fonctions de répartition à n dimensions et leurs marges","authors":"Sklar","year":"1959","venue":"Publ. Inst. Statist. Univ","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sobolewski and Wozniak, 2013] Piotr Sobolewski and Michal Wozniak. Concept drift detection and model selection with simulated recurrence and ensembles of statistical detectors. J. UCS, 19(4):462-483, 2013.","title":"Concept drift detection and model selection with simulated recurrence and ensembles of statistical detectors","authors":"Piotr Sobolewski, Michal Wozniak","year":"2013","venue":"J. UCS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Souza et al., 2015] Vinícius Souza, Diego Silva, et al. Data stream classification guided by clustering on nonstationary environments and extreme verification latency. In SIAM Int. Conf. Data Mining, pages 873-881, 2015.","title":"Data stream classification guided by clustering on nonstationary environments and extreme verification latency","authors":"Vinícius Souza, Diego Silva","year":"2015","venue":"SIAM Int. Conf. Data Mining","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tsymbal, 2004] Alexey Tsymbal. The problem of concept drift: definitions and related work. Computer Science De- partment, Trinity College Dublin, 106(2), 2004.","title":"The problem of concept drift: definitions and related work","authors":"Alexey Tsymbal","year":"2004","venue":"Computer Science Department, Trinity College Dublin","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang and Abraham, 2015] Heng Wang and Zubin Abra- ham. Concept drift detection for streaming data. In IJCNN, pages 1-9. IEEE, 2015.","title":"Concept drift detection for streaming data","authors":"Heng Wang, Zubin Abraham","year":"2015","venue":"IJCNN","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2013] Shuo Wang, Leandro L. Minku, et al. Concept drift detection for online class imbalance learn- ing. In IJCNN, pages 1-10. IEEE, 2013.","title":"Concept drift detection for online class imbalance learning","authors":"Shuo Wang, Leandro L Minku","year":"2013","venue":"IJCNN","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Widmer and Kubat, 1993] G. Widmer and M. Kubat. Effec- tive learning in dynamic environments by explicit context tracking. In ECML, pages 227-243. Springer, 1993.","title":"Effective learning in dynamic environments by explicit context tracking","authors":"G Widmer, M Kubat","year":"1993","venue":"ECML","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yu and Abraham, 2017] Shujian Yu and Z. Abraham. Con- cept drift detection with hierarchical hypothesis testing. In SIAM Int. Conf. Data Mining, pages 768-776, 2017.","title":"Concept drift detection with hierarchical hypothesis testing","authors":"Abraham ; Shujian Yu, Z Yu, Abraham","year":"2017","venue":"SIAM Int. Conf. Data Mining","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Žliobaite et al., 2014] Indre Žliobaite, Albert Bifet, et al. Active learning with drifting streaming data. IEEE Trans. Neural Netw. Learn. Syst., 25(1):27-39, 2014.","title":"Active learning with drifting streaming data","authors":"Žliobaite","year":"2014","venue":"IEEE Trans. Neural Netw. Learn. Syst","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Žliobaite, 2010] Indre Žliobaite. Change with delayed la- beling: When is it detectable? In ICDM Workshops, pages 843-850. IEEE, 2010.","title":"Change with delayed labeling: When is it detectable?","authors":"Indre Žliobaite, Žliobaite","year":"2010","venue":"ICDM Workshops","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}