{"type":"ScholarlyArticle","title":"Measuring the Discrepancy between Conditional Distributions: Methods, Properties and Applications","authors":[{"name":"Shujian Yu","slug":"shujian-yu","given":"Shujian","family":"Yu","affiliation":"NEC Labs Europe , 69115 Heidelberg , Germany"},{"name":"Ammar Shaker","slug":"ammar-shaker","given":"Ammar","family":"Shaker","affiliation":"NEC Labs Europe , 69115 Heidelberg , Germany"},{"name":"Francesco Alesiani","slug":"francesco-alesiani","given":"Francesco","family":"Alesiani","affiliation":"NEC Labs Europe , 69115 Heidelberg , Germany"},{"name":"Jose Principe","slug":"jose-principe","given":"Jose","family":"Principe","affiliation":"University of Florida , Gainesville , FL 32611 , USA"}],"abstract":"We propose a simple yet powerful test statistic to quantify the discrepancy between two conditional distributions. The new statistic avoids the explicit estimation of the underlying distributions in high-dimensional space and it operates on the cone of symmetric positive semidefinite (SPS) matrix using the Bregman matrix divergence. Moreover, it inherits the merits of the correntropy function to explicitly incorporate high-order statistics in the data. We present the properties of our new statistic and illustrate its connections to prior art. We finally show the applications of our new statistic on three different machine learning problems, namely the multi-task learning over graphs, the concept drift detection, and the information-theoretic feature selection, to demonstrate its utility and advantage. Code of our statistic is available at https://bit.ly/BregmanCorrentropy.","doi":"10.24963/ijcai.2020/385","url":"https://ijcai.org/proceedings/2020/385","pdf":"https://ijcai.org/proceedings/2020/0385.pdf","markdown":"https://ijcai.org/proceedings/2020/385.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/385","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":385,"pages":{"first":2777,"last":2784},"keywords":["Data Mining: Theoretical Foundations","Machine Learning: Time-series;Data Streams","Machine Learning: Transfer, Adaptation, Multi-task Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25218/file_25218.m3u8","length_sec":291,"provider":"hls"}],"references":[{"position":1,"raw":"[Anderson et al., 1994] Niall H Anderson, Peter Hall, and D Michael Titterington. Two-sample test statistics for measuring discrepancies between two multivariate prob- ability density functions using kernel-based density es- timates. Journal of Multivariate Analysis, 50(1):41-54, 1994.","title":"Two-sample test statistics for measuring discrepancies between two multivariate probability density functions using kernel-based density estimates","authors":"Peter Niall H Anderson, D Hall, Michael Titterington","year":"1994","venue":"Journal of Multivariate Analysis","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baena-Garcıa et al., 2006] Manuel Baena-Garcıa, José del Campo-Ávila, Raúl Fidalgo, Albert Bifet, R Gavalda, and R Morales-Bueno. Early drift detection method. In Fourth international workshop on knowledge discovery from data streams, volume 6, pages 77-86, 2006.","title":"Early drift detection method","authors":"Manuel Baena-Garcıa, José Del Campo-Ávila, Raúl Fidalgo, Albert Bifet, R Gavalda, R Morales-Bueno","year":"2006","venue":"Fourth international workshop on knowledge discovery from data streams","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Banerjee et al., 2005] Arindam Banerjee, Srujana Merugu, Inderjit S Dhillon, and Joydeep Ghosh. Clustering with bregman divergences. JMLR, 6(Oct):1705-1749, 2005. (a) breast (b) waveform Figure 4: Validation accuracy on (a) breast and (b) waveform data sets. The number of samples and the feature dimensionality for each data set are listed in the title. The value beside each method in the legend indicates the average rank in that data set.","title":") breast and (b) waveform data sets. The number of samples and the feature dimensionality for each data set are listed in the title","authors":"Arindam Banerjee, Srujana Merugu, Joydeep Inderjit S Dhillon, Ghosh","year":"2005","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Battiti, 1994] Roberto Battiti. Using mutual information for selecting features in supervised neural net learning. IEEE TNNLS, 5(4):537-550, 1994.","title":"Using mutual information for selecting features in supervised neural net learning","authors":"Roberto Battiti","year":"1994","venue":"IEEE TNNLS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brown et al., 2012] Gavin Brown, Adam Pocock, Ming-Jie Zhao, and Mikel Luján. Conditional likelihood maximisa- tion: a unifying framework for information theoretic fea- ture selection. JMLR, 13(Jan):27-66, 2012.","title":"Conditional likelihood maximisation: a unifying framework for information theoretic feature selection","authors":"Gavin Brown, Adam Pocock, Ming-Jie Zhao, Mikel Luján","year":"2012","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2017] Badong Chen, , et al. Kernel risk- sensitive loss: definition, properties and application to robust adaptive filtering. IEEE TSP, 65(11):2888-2901, 2017.","title":"Kernel risksensitive loss: definition, properties and application to robust adaptive filtering","authors":"Badong Chen","year":"2017","venue":"IEEE TSP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cover and Thomas, 2012] Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 2012.","title":"Elements of information theory","authors":"M Thomas, Joy A Cover, Thomas","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[dos Reis et al., 2016] Denis Moreira dos Reis, Peter Flach, Stan Matwin, and Gustavo Batista. Fast unsupervised on- line drift detection using incremental kolmogorov-smirnov test. In SIGKDD, pages 1545-1554. ACM, 2016.","title":"Fast unsupervised online drift detection using incremental kolmogorov-smirnov test","authors":"Denis Moreira Dos Reis, Peter Flach, Stan Matwin, Gustavo Batista","year":"2016","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dries and Rückert, 2009] Anton Dries and Ulrich Rückert. Adaptive concept drift detection. In SDM, pages 235-246. SIAM, 2009.","title":"Adaptive concept drift detection","authors":"Rückert Dries, Anton Dries, Ulrich Rückert","year":"2009","venue":"SDM","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Duchi, 2007] John Duchi. Derivations for linear algebra and optimization. Berkeley, California, 3, 2007.","title":"Derivations for linear algebra and optimization","authors":"John Duchi","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fan et al., 2006] Yanqin Fan, Qi Li, and Insik Min. A non- parametric bootstrap test of conditional distributions. E- conometric Theory, 22(4):587-613, 2006.","title":"A nonparametric bootstrap test of conditional distributions","authors":"Yanqin Fan, Qi Li, Insik Min","year":"2006","venue":"Econometric Theory","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Franc ¸ois et al., 2007] Damien Franc ¸ois, Fabrice Rossi, Vin- cent Wertz, and Michel Verleysen. Resampling methods for parameter-free and robust feature selection with mutual information. Neurocomputing, 70(7-9):1276-1288, 2007.","title":"Resampling methods for parameter-free and robust feature selection with mutual information","authors":"Damien Franc ¸ois, Fabrice Rossi, Vincent Wertz, Michel Verleysen","year":"2007","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Frías-Blanco et al., 2014] Isvani Frías-Blanco, José del Campo-Ávila, Gonzalo Ramos-Jimenez, Rafael Morales- Bueno, Agustín Ortiz-Díaz, and Yailé Caballero-Mota. Online and non-parametric drift detection methods based on hoeffding bounds. IEEE TKDE, 27(3):810-823, 2014.","title":"Isvani Frías-Blanco, José del Campo-Ávila, Gonzalo Ramos-Jimenez, Rafael Morales-Bueno, Agustín Ortiz-Díaz, and Yailé Caballero-Mota. Online and non-parametric drift detection methods based on hoeffding bounds","authors":"Frías-Blanco","year":"2014","venue":"IEEE TKDE","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gama et al., 2004] Joao Gama, Pedro Medas, Gladys Castillo, and Pedro Rodrigues. Learning with drift de- tection. In Brazilian symposium on artificial intelligence, pages 286-295. Springer, 2004.","title":"Learning with drift detection","authors":"Joao Gama, Pedro Medas, Gladys Castillo, Pedro Rodrigues","year":"2004","venue":"Brazilian symposium on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gama et al., 2014] João Gama, Indrė Žliobaitė, Albert B- ifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. A survey on concept drift adaptation. ACM computing sur- veys (CSUR), 46(4):44, 2014.","title":"A survey on concept drift adaptation","authors":"João Gama, Indrė Žliobaitė, Albert Bifet, Mykola Pechenizkiy, Abdelhamid Bouchachia","year":"2014","venue":"ACM computing surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Gonc ¸alves et al., 2016] André R Gonc ¸alves, Fernando J Von Zuben, and Arindam Banerjee. Multi-task sparse structure learning with gaussian copula models. JMLR, 17(1):1205-1234, 2016.","title":"Multi-task sparse structure learning with gaussian copula models","authors":"Fernando J André R Gonc ¸alves, Arindam Von Zuben, Banerjee","year":"2016","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Gretton et al., 2012a] Arthur Gretton, , et al. Optimal kernel choice for large-scale two-sample tests. In NeurIPS, pages 1205-1213, 2012.","title":"Optimal kernel choice for large-scale two-sample tests","authors":"Gretton","year":"2012","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Gretton et al., 2012b] Arthur Gretton, Karsten M Borg- wardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. JMLR, 13(Mar):723- 773, 2012.","title":"A kernel two-sample test","authors":"Gretton","year":"2012","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Harel et al., 2014] Maayan Harel, Shie Mannor, Ran El- Yaniv, and Koby Crammer. Concept drift detection through resampling. In ICML, pages 1009-1017, 2014.","title":"Concept drift detection through resampling","authors":"Maayan Harel, Shie Mannor, Ran El-Yaniv, Koby Crammer","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[He et al., 2019] Xiao He, Francesco Alesiani, and Ammar Shaker. Efficient and scalable multi-task regression on massive number of tasks. In AAAI, volume 33, pages 3763-3770, 2019.","title":"Efficient and scalable multi-task regression on massive number of tasks","authors":"Xiao He, Francesco Alesiani, Ammar Shaker","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kirkpatrick et al., 2017] James Kirkpatrick, , et al. Over- coming catastrophic forgetting in neural networks. PNAS, 114(13):3521-3526, 2017.","title":"Overcoming catastrophic forgetting in neural networks","authors":"James Kirkpatrick","year":"2017","venue":"PNAS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Kulis et al., 2009] Brian Kulis, Mátyás A Sustik, and Inder- jit S Dhillon. Low-rank kernel learning with bregman ma- trix divergences. JMLR, 10(Feb):341-376, 2009.","title":"Low-rank kernel learning with bregman matrix divergences","authors":"Brian Kulis, Mátyás A Sustik, Inderjit S Dhillon","year":"2009","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Lee and Park, 2006] Young Kyung Lee and Byeong U Park. Estimation of kullback-leibler divergence by local likeli- hood. Annals of the Institute of Statistical Mathematics, 58(2):327-340, 2006.","title":"Estimation of kullback-leibler divergence by local likelihood","authors":"Lee, Park ; Young, Kyung Lee, Byeong U Park","year":"2006","venue":"Annals of the Institute of Statistical Mathematics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Lewis, 1992] David D Lewis. Feature selection and feature extraction for text categorization. In Proceedings of the workshop on Speech and Natural Language, pages 212- 217. Association for Computational Linguistics, 1992.","title":"Feature selection and feature extraction for text categorization","authors":"Lewis David","year":"1992","venue":"Proceedings of the workshop on Speech and Natural Language","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Liu et al., 2007] Weifeng Liu, Puskal P Pokharel, and José C Príncipe. Correntropy: Properties and appli- cations in non-gaussian signal processing. IEEE TSP, 55(11):5286-5298, 2007.","title":"Correntropy: Properties and applications in non-gaussian signal processing","authors":"Weifeng Liu, José C Pokharel, Príncipe","year":"2007","venue":"IEEE TSP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Nagler and Czado, 2016] Thomas Nagler and Claudia Cza- do. Evading the curse of dimensionality in nonparametric density estimation with simplified vine copulas. Journal of Multivariate Analysis, 151:69-89, 2016.","title":"Evading the curse of dimensionality in nonparametric density estimation with simplified vine copulas","authors":"Thomas Nagler, Claudia Czado","year":"2016","venue":"Journal of Multivariate Analysis","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Nielsen and Chuang, 2011] Michael A. Nielsen and Isaac L. Chuang. Quantum Computation and Quantum Informa- tion. Cambridge University Press, 10th edition, 2011.","title":"Quantum Computation and Quantum Information","authors":"Michael A Nielsen, Isaac L Chuang","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Pardo, 2005] Leandro Pardo. Statistical inference based on divergence measures. Chapman and Hall/CRC, 2005.","title":"Statistical inference based on divergence measures","authors":"Leandro Pardo","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Rao et al., 2011] Murali Rao, Sohan Seth, Jianwu Xu, Yun- mei Chen, Hemant Tagare, and José C Príncipe. A test of independence based on a generalized correlation function. Signal Processing, 91(1):15-27, 2011.","title":"A test of independence based on a generalized correlation function","authors":"Murali Rao, Sohan Seth, Jianwu Xu, Yunmei Chen, Hemant Tagare, José C Príncipe","year":"2011","venue":"Signal Processing","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Ren et al., 2016] Yong Ren, Jun Zhu, Jialian Li, and Yucen Luo. Conditional generative moment-matching networks. In NeurIPS, pages 2928-2936, 2016.","title":"Conditional generative moment-matching networks","authors":"Yong Ren, Jun Zhu, Jialian Li, Yucen Luo","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Santamaría et al., 2006] Ignacio Santamaría, Puskal P Pokharel, and José Carlos Principe. Generalized corre- lation function: definition, properties, and application to blind equalization. IEEE TSP, 54(6):2187-2197, 2006.","title":"Generalized correlation function: definition, properties, and application to blind equalization","authors":"Ignacio Santamaría, José Pokharel, Carlos Principe","year":"2006","venue":"IEEE TSP","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Sethi and Kantardzic, 2017] Tegjyot Singh Sethi and Mehmed Kantardzic. On the reliable detection of concept drift from streaming unlabeled data. Expert Systems with Applications, 82:77-99, 2017.","title":"Tegjyot Singh Sethi and Mehmed Kantardzic. On the reliable detection of concept drift from streaming unlabeled data","authors":"Kantardzic Sethi","year":"2017","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Silverman, 1986] B. W. Silverman. Density Estimation for Statistics and Data Analysis. Chapman & Hall, 1986.","title":"Density Estimation for Statistics and Data Analysis","authors":"B W Silverman","year":"1986","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Tsanas et al., 2009] Athanasios Tsanas, Max A Little, Patrick E McSharry, and Lorraine O Ramig. Accurate tele- monitoring of parkinson's disease progression by noninva- sive speech tests. IEEE TBME, 57(4):884-893, 2009.","title":"Accurate telemonitoring of parkinson's disease progression by noninvasive speech tests","authors":"Athanasios Tsanas, Max A Little, Patrick E Mcsharry, Lorraine O Ramig","year":"2009","venue":"IEEE TBME","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wang et al., 2009] Qing Wang, Sanjeev R Kulkarni, and Sergio Verdú. Divergence estimation for multidimension- al densities via k-nearest-neighbor distances. IEEE TIT, 55(5):2392-2405, 2009.","title":"Divergence estimation for multidimensional densities via k-nearest-neighbor distances","authors":"Qing Wang, Sanjeev R Kulkarni, Sergio Verdú","year":"2009","venue":"IEEE TIT","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Yu and Abraham, 2017] Shujian Yu and Zubin Abraham. Concept drift detection with hierarchical hypothesis test- ing. In SDM, pages 768-776. SIAM, 2017.","title":"Concept drift detection with hierarchical hypothesis testing","authors":"Abraham ; Yu, Shujian Yu, Zubin Abraham","year":"2017","venue":"SDM","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhang and Yeung, 2010] Yu Zhang and Dit-Yan Yeung. A convex formulation for learning task relationships in multi-task learning. In UAI, pages 733-742, 2010.","title":"A convex formulation for learning task relationships in multi-task learning","authors":"Yeung ; ] Yu Zhang, Dit-Yan Zhang, Yeung","year":"2010","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zheng, 2000] John Xu Zheng. A consistent test of con- ditional parametric distributions. Econometric Theory, 16(5):667-691, 2000.","title":"A consistent test of conditional parametric distributions","authors":"John Xu, Zheng","year":"2000","venue":"Econometric Theory","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}