{"type":"ScholarlyArticle","title":"De-biasing Covariance-Regularized Discriminant Analysis","authors":[{"name":"Haoyi Xiong","slug":"haoyi-xiong","given":"Haoyi","family":"Xiong","affiliation":"† Baidu Inc. , Beijing , China; + National Engineering Laboratory of Deep Learning Technology and Application , Beijing , China"},{"name":"Wei Cheng","slug":"wei-cheng","given":"Wei","family":"Cheng","affiliation":"§ NEC Laboratories America , NJ , United States"},{"name":"Yanjie Fu","slug":"yanjie-fu","given":"Yanjie","family":"Fu","affiliation":"‡ Missouri University of Science and Technology , MO , United States; † Baidu Inc. , Beijing , China"},{"name":"Wenqing Hu","slug":"wenqing-hu","given":"Wenqing","family":"Hu","affiliation":"‡ Missouri University of Science and Technology , MO , United States"},{"name":"Jiang Bian","slug":"jiang-bian","given":"Jiang","family":"Bian","affiliation":"‡ Missouri University of Science and Technology , MO , United States; † Baidu Inc. , Beijing , China"},{"name":"Zhishan Guo","slug":"zhishan-guo","given":"Zhishan","family":"Guo","affiliation":"‡ Missouri University of Science and Technology , MO , United States"}],"abstract":"Fisher's Linear Discriminant Analysis (FLD) is a well-known technique for linear classification, feature extraction and dimension reduction. The empirical FLD relies on two key estimations from the data -- the mean vector for each class and the (inverse) covariance matrix. To improve the accuracy of FLD under the High Dimension Low Sample Size (HDLSS) settings, Covariance-Regularized FLD (CRLD) has been proposed to use shrunken covariance estimators, such as Graphical Lasso, to strike a balance between biases and variances. Though CRLD could obtain better classification accuracy, it usually incurs bias and converges to the optimal result with a slower asymptotic rate. Inspired by the recent progress in de-biased Lasso, we propose a novel FLD classifier, DBLD, which improves classification accuracy of CRLD through de-biasing. Theoretical analysis shows that DBLD possesses better asymptotic properties than CRLD. We conduct experiments on both synthetic datasets and real application datasets to confirm the correctness of our theoretical analysis and demonstrate the superiority of DBLD over classical FLD, CRLD and other downstream competitors under HDLSS settings.","doi":"10.24963/ijcai.2018/401","url":"https://ijcai.org/proceedings/2018/401","pdf":"https://ijcai.org/proceedings/2018/0401.pdf","markdown":"https://ijcai.org/proceedings/2018/401.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/401","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":401,"pages":{"first":2889,"last":2897},"keywords":["Machine Learning: Data Mining","Machine Learning Applications: Applications of Supervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Bickel and Levina, 2008] Peter J Bickel and Elizaveta Lev- ina. Regularized estimation of large covariance matrices. The Annals of Statistics, pages 199-227, 2008.","title":"Regularized estimation of large covariance matrices","authors":"Levina ; Bickel, J Peter, Elizaveta Bickel, Levina","year":"2008","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bickel et al., 2004] Peter J Bickel, Elizaveta Levina, et al. Some theory for fisher's linear discriminant function,naive bayes', and some alternatives when there are many more variables than observations. Bernoulli, 10(6):989-1010, 2004.","title":"Some theory for fisher's linear discriminant function,naive bayes', and some alternatives when there are many more variables than observations","authors":"J Peter, Elizaveta Bickel, Levina","year":"2004","venue":"Bernoulli","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cai and Liu, 2011] Tony Cai and Weidong Liu. A direct estimation approach to sparse linear discriminant anal- ysis. Journal of the American Statistical Association, 106(496):1566-1577, 2011.","title":"A direct estimation approach to sparse linear discriminant analysis","authors":"Tony Cai, Weidong Liu","year":"2011","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cai et al., 2016] T Tony Cai, Zhao Ren, Harrison H Zhou, et al. Estimating structured high-dimensional covariance and precision matrices: Optimal rates and adaptive estima- tion. Electronic Journal of Statistics, 10(1):1-59, 2016.","title":"Estimating structured high-dimensional covariance and precision matrices: Optimal rates and adaptive estimation","authors":"Tony Cai, Zhao Ren, Harrison H Zhou","year":"2016","venue":"Electronic Journal of Statistics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[DasGupta, 2008] Anirban DasGupta. Asymptotic theory of statistics and probability. Springer Science & Business Media, 2008.","title":"Asymptotic theory of statistics and probability","authors":"Anirban Dasgupta","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Duda et al., 2001] Richard O. Duda, Peter E. Hart, and David G. Stork. Pattern Classification (2nd Ed). Wiley, 2001.","title":"Pattern Classification","authors":"Richard O Duda, Peter E Hart, David G Stork","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Durrant and Kabán, 2015] Robert J Durrant and Ata Kabán. Random projections as regularizers: learning a linear dis- criminant from fewer observations than dimensions. Ma- chine Learning, 99(2):257-286, 2015.","title":"Random projections as regularizers: learning a linear discriminant from fewer observations than dimensions","authors":"J Robert, Ata Durrant, Kabán","year":"2015","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Friedman et al., 2008] Jerome Friedman, Trevor Hastie, and Robert Tibshirani. Sparse inverse covariance estimation with the graphical lasso. Biostatistics, 9(3):432-441, 2008.","title":"Sparse inverse covariance estimation with the graphical lasso","authors":"Jerome Friedman, Trevor Hastie, Robert Tibshirani","year":"2008","venue":"Biostatistics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jankova et al., 2015] Jana Jankova, Sara van de Geer, et al. Confidence intervals for high-dimensional inverse covari- ance estimation. Electronic Journal of Statistics, 9(1):1205- 1229, 2015.","title":"Confidence intervals for high-dimensional inverse covariance estimation","authors":"Jana Jankova, Sara Van De Geer","year":"2015","venue":"Electronic Journal of Statistics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Javanmard and Montanari, 2014] Adel Javanmard and An- drea Montanari. Confidence intervals and hypothesis test- ing for high-dimensional regression. Journal of Machine Learning Research, 15(1):2869-2909, 2014.","title":"Adel Javanmard and Andrea Montanari. Confidence intervals and hypothesis testing for high-dimensional regression","authors":"Montanari Javanmard","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kaski and Peltonen, 2003] Samuel Kaski and Jaakko Pelto- nen. Informative discriminant analysis. In ICML, pages 329-336, 2003.","title":"Informative discriminant analysis","authors":"Peltonen Kaski, Samuel Kaski, Jaakko Peltonen","year":"2003","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krzanowski et al., 1995] WJ Krzanowski, Philip Jonathan, WV McCarthy, and MR Thomas. Discriminant analysis with singular covariance matrices: methods and applica- tions to spectroscopic data. Applied Statistics, pages 101- 115, 1995.","title":"Discriminant analysis with singular covariance matrices: methods and applications to spectroscopic data","authors":"Krzanowski","year":"1995","venue":"Applied Statistics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kulis and others, 2013] Brian Kulis et al. Metric learning: A survey. Foundations and Trends in Machine Learning, 5(4):287-364, 2013.","title":"Metric learning: A survey","authors":"Brian Kulis","year":"2013","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin, 2017] Chih-Jen Lin. Libsvm data: Classification (binary class). https://www.csie.ntu.edu. tw/ ˜cjlin/libsvmtools/datasets/binary. html, 2017.","title":"Libsvm data: Classification (binary class)","authors":"Chih-Jen Lin","year":"2017","venue":null,"doi":null,"url":"https://www.csie.ntu.edu.tw/˜cjlin/libsvmtools/datasets/binary.html","internal_url":null},{"position":15,"raw":"[Marčenko and Pastur, 1967] Vladimir A Marčenko and Leonid A Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik, 1(4):457, 1967.","title":"Distribution of eigenvalues for some sets of random matrices","authors":"A Vladimir, Leonid A Marčenko, Pastur","year":"1967","venue":"Mathematics of the USSR-Sbornik","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Peck and Van Ness, 1982] Roger Peck and John Van Ness. The use of shrinkage estimators in linear discriminant anal- ysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, (5):530-537, 1982.","title":"The use of shrinkage estimators in linear discriminant analysis","authors":"Van Peck, Roger Ness, John Peck, Van Ness","year":"1982","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rothman et al., 2010] Adam J Rothman, Elizaveta Levina, and Ji Zhu. Sparse multivariate regression with covari- ance estimation. Journal of Computational and Graphical Statistics, 19(4):947-962, 2010.","title":"Sparse multivariate regression with covariance estimation","authors":"Elizaveta Adam J Rothman, Ji Levina, Zhu","year":"2010","venue":"Journal of Computational and Graphical Statistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shao et al., 2011] Jun Shao, Yazhen Wang, Xinwei Deng, Sijian Wang, et al. Sparse linear discriminant analysis by thresholding for high dimensional data. The Annals of Statistics, 39(2):1241-1265, 2011.","title":"Sparse linear discriminant analysis by thresholding for high dimensional data","authors":"Jun Shao, Yazhen Wang, Xinwei Deng, Sijian Wang","year":"2011","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tibshirani et al., 2002] Robert Tibshirani, Trevor Hastie, Balasubramanian Narasimhan, and Gilbert Chu. Diagnosis of multiple cancer types by shrunken centroids of gene ex- pression. Proceedings of the National Academy of Sciences, 99(10):6567-6572, 2002.","title":"Diagnosis of multiple cancer types by shrunken centroids of gene expression","authors":"Robert Tibshirani, Trevor Hastie, Balasubramanian Narasimhan, Gilbert Chu","year":"2002","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Witten and Tibshirani, 2009] Daniela M Witten and Robert Tibshirani. Covariance-regularized regression and classi- fication for high dimensional problems. Journal of the Royal Statistical Society: Series B (Statistical Methodol- ogy), 71(3):615-636, 2009.","title":"Covariance-regularized regression and classification for high dimensional problems","authors":"M Daniela, Robert Witten, Tibshirani","year":"2009","venue":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Witten et al., 2011] Daniela M Witten, Jerome H Friedman, and Noah Simon. New insights and faster computations for the graphical lasso. Journal of Computational and Graphical Statistics, 20(4):892-900, 2011.","title":"New insights and faster computations for the graphical lasso","authors":"Jerome H Daniela M Witten, Noah Friedman, Simon","year":"2011","venue":"Journal of Computational and Graphical Statistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ye et al., 2004] Jieping Ye, Ravi Janardan, and Qi Li. Two- dimensional linear discriminant analysis. In NIPS, pages 1569-1576, Cambridge, MA, USA, 2004.","title":"Twodimensional linear discriminant analysis","authors":"Ye","year":"2004","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang and others, 2003] Zhihua Zhang et al. Learning met- rics via discriminant kernels and multidimensional scaling: Toward expected euclidean representation. In ICML, vol- ume 2, pages 872-879, 2003.","title":"Learning metrics via discriminant kernels and multidimensional scaling: Toward expected euclidean representation","authors":"Zhihua Zhang","year":"2003","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang and Zhang, 2014] Cun-Hui Zhang and Stephanie S Zhang. Confidence intervals for low dimensional param- eters in high dimensional linear models. Journal of the Royal Statistical Society: Series B (Statistical Methodol- ogy), 76(1):217-242, 2014.","title":"Confidence intervals for low dimensional parameters in high dimensional linear models","authors":"Cun-Hui Zhang, Stephanie S Zhang","year":"2014","venue":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2015] Jinghe Zhang, Haoyi Xiong, Yu Huang, Hao Wu, Kevin Leach, and Laura E. Barnes. MSEQ: Early detection of anxiety and depression via temporal orders of diagnoses in electronic health data. In BigData. IEEE, 2015.","title":"MSEQ: Early detection of anxiety and depression via temporal orders of diagnoses in electronic health data","authors":"Zhang","year":"2015","venue":"BigData. IEEE","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zollanvari and Dougherty, 2013] Amin Zollanvari and Ed- ward R Dougherty. Random matrix theory in pattern clas- sification: An application to error estimation. In 2013 Asilomar Conference on Signals, Systems and Computers, 2013.","title":"Amin Zollanvari and Edward R Dougherty. Random matrix theory in pattern classification: An application to error estimation","authors":"Dougherty Zollanvari","year":"2013","venue":"Asilomar Conference on Signals, Systems and Computers","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}