{"type":"ScholarlyArticle","title":"Quadratic Sparse Gaussian Graphical Model Estimation Method for Massive Variables","authors":[{"name":"Jiaqi Zhang","slug":"jiaqi-zhang","given":"Jiaqi","family":"Zhang","affiliation":"College of Software Engineering , Southeast University , China"},{"name":"Meng Wang","slug":"meng-wang","given":"Meng","family":"Wang","affiliation":"College of Software Engineering , Southeast University , China; School of Computer Science and Engineering , Southeast University , China; School of Artificial Intelligence , Southeast University , China"},{"name":"Qinchi Li","slug":"qinchi-li","given":"Qinchi","family":"Li","affiliation":"Electrical Engineering and Automation , YOUPEI College , Yancheng Institute of Technology , China"},{"name":"Sen Wang","slug":"sen-wang","given":"Sen","family":"Wang","affiliation":"School of Information Technology and Electrical Engineering , The University of Queensland , Australia"},{"name":"Xiaojun Chang","slug":"xiaojun-chang","given":"Xiaojun","family":"Chang","affiliation":"Department of Data Science & AI , Monash University , Australia"},{"name":"Beilun Wang","slug":"beilun-wang","given":"Beilun","family":"Wang","affiliation":"College of Software Engineering , Southeast University , China; School of Computer Science and Engineering , Southeast University , China; School of Artificial Intelligence , Southeast University , China"}],"abstract":"We consider the problem of estimating a sparse Gaussian Graphical Model with a special graph topological structure and more than a million variables. Most previous scalable estimators still contain expensive calculation steps (e.g., matrix inversion or Hessian matrix calculation) and become infeasible in high-dimensional scenarios, where p (number of variables) is larger than n (number of samples). To overcome this challenge, we propose a novel method, called Fast and Scalable Inverse Covariance Estimator by Thresholding (FST). FST first obtains a graph structure by applying a generalized threshold to the sample covariance matrix. Then, it solves multiple block-wise subproblems via element-wise thresholding. By using matrix thresholding instead of matrix inversion as the computational bottleneck, FST reduces its computational complexity to a much lower order of magnitude (O(p2)). We show that FST obtains the same sharp convergence rate O(√(log max{p, n}/n) as other state-of-the-art methods. We validate the method empirically, on multiple simulated datasets and one real-world dataset, and show that FST is two times faster than the four baselines while achieving a lower error rate under both Frobenius-norm and max-norm.","doi":"10.24963/ijcai.2020/410","url":"https://ijcai.org/proceedings/2020/410","pdf":"https://ijcai.org/proceedings/2020/0410.pdf","markdown":"https://ijcai.org/proceedings/2020/410.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/410","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":410,"pages":{"first":2964,"last":2972},"keywords":["Machine Learning: Big data; Scalability","Machine Learning: Learning Graphical Models","Data Mining: Mining Graphs, Semi Structured Data, Complex Data"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24933/file_24933.m3u8","length_sec":291,"provider":"hls"}],"references":[{"position":1,"raw":"[Banerjee et al., 2008] Onureena Banerjee, Laurent El Ghaoui, and Alexandre d'Aspremont. Model selection through sparse maximum likelihood estimation for multivariate gaussian or binary data. Journal of Machine learning research, 9(Mar):485-516, 2008.","title":"Onureena Banerjee, Laurent El Ghaoui, and Alexandre d'Aspremont. Model selection through sparse maximum likelihood estimation for multivariate gaussian or binary data","authors":"Banerjee","year":"2008","venue":"Journal of Machine learning research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bickel et al., 2008] Peter J Bickel, Elizaveta Levina, et al. Covariance regularization by thresholding. The Annals of Statistics, 36(6):2577-2604, 2008.","title":"Covariance regularization by thresholding","authors":"J Peter, Elizaveta Bickel, Levina","year":"2008","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bollhöfer and Schenk, 2016] Matthias Bollhöfer and Olaf Schenk. Large-scale sparse inverse covariance matrix esti- mation. 2016.","title":"Large-scale sparse inverse covariance matrix estimation","authors":"Schenk Bollhöfer, Matthias Bollhöfer, Olaf Schenk","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cai et al., 2011] Tony Cai, Weidong Liu, and Xi Luo. A constrained l1 minimization approach to sparse precision matrix estimation. Journal of the American Statistical Association, 106(494):594-607, 2011.","title":"A constrained l1 minimization approach to sparse precision matrix estimation","authors":"Tony Cai, Weidong Liu, Xi Luo","year":"2011","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Carvalho et al., 2008] Carlos M Carvalho, Jeffrey Chang, Joseph E Lucas, Joseph R Nevins, Quanli Wang, and Mike West. High-dimensional sparse factor modeling: applica- tions in gene expression genomics. Journal of the American Statistical Association, 103(484):1438-1456, 2008.","title":"High-dimensional sparse factor modeling: applications in gene expression genomics","authors":"Carlos M Carvalho, Jeffrey Chang, Joseph E Lucas, Joseph R Nevins, Quanli Wang, Mike West","year":"2008","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Eftekhari et al., 2018] Aryan Eftekhari, Matthias Bollhofer, and Olaf Schenk. Distributed memory sparse inverse covari- ance matrix estimation on high-performance computing ar- chitectures. In Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, page 20. IEEE Press, 2018.","title":"Distributed memory sparse inverse covariance matrix estimation on high-performance computing architectures","authors":"Aryan Eftekhari, Matthias Bollhofer, Olaf Schenk","year":"2018","venue":"Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Friedman et al., 2008] Jerome Friedman, Trevor Hastie, and Robert Tibshirani. Sparse inverse covariance estimation with the graphical lasso. Biostatistics, 9(3):432-441, 2008.","title":"Sparse inverse covariance estimation with the graphical lasso","authors":"Jerome Friedman, Trevor Hastie, Robert Tibshirani","year":"2008","venue":"Biostatistics","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hsieh et al., 2013] Cho-Jui Hsieh, Mátyás A Sustik, Inder- jit S Dhillon, Pradeep K Ravikumar, and Russell Poldrack. Big & quic: Sparse inverse covariance estimation for a million variables. In Advances in neural information pro- cessing systems, pages 3165-3173, 2013.","title":"Big & quic: Sparse inverse covariance estimation for a million variables","authors":"Cho-Jui Hsieh, Mátyás A Sustik, Pradeep K Inderjit S Dhillon, Russell Ravikumar, Poldrack","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hsieh et al., 2014] Cho-Jui Hsieh, Mátyás A Sustik, Inder- jit S Dhillon, and Pradeep Ravikumar. Quic: quadratic ap- proximation for sparse inverse covariance estimation. The Journal of Machine Learning Research, 15(1):2911-2947, 2014.","title":"Quic: quadratic approximation for sparse inverse covariance estimation","authors":"Cho-Jui Hsieh, Mátyás A Sustik, Pradeep Inderjit S Dhillon, Ravikumar","year":"2014","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ideker and Krogan, 2012] Trey Ideker and Nevan J Krogan. Differential network biology. Molecular systems biology, 8(1):565, 2012.","title":"Differential network biology","authors":"Krogan Ideker, Trey Ideker, Nevan J Krogan","year":"2012","venue":"Molecular systems biology","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Koanantakool et al., 2017] Penporn Koanantakool, Alnur Ali, Ariful Azad, Aydin Buluc, Dmitriy Morozov, Leonid Oliker, Katherine Yelick, and Sang-Yun Oh. Communication-avoiding optimization methods for dis- tributed massive-scale sparse inverse covariance estimation. arXiv preprint arXiv:1710.10769, 2017.","title":"Communication-avoiding optimization methods for distributed massive-scale sparse inverse covariance estimation","authors":"Koanantakool","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lauritzen, 1996] S. L. Lauritzen. Graphical Models. Oxford University Press, 1996.","title":"Graphical Models","authors":"S L Lauritzen","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mazumder and Hastie, 2012a] Rahul Mazumder and Trevor Hastie. Exact covariance thresholding into connected com- ponents for large-scale graphical lasso. Journal of Machine Learning Research, 13(Mar):781-794, 2012.","title":"Exact covariance thresholding into connected components for large-scale graphical lasso","authors":"Hastie Mazumder, Rahul Mazumder, Trevor Hastie","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mazumder and Hastie, 2012b] Rahul Mazumder and Trevor Hastie. Exact covariance thresholding into connected com- ponents for large-scale graphical lasso. Journal of Machine Learning Research, 13(Mar):781-794, 2012.","title":"Exact covariance thresholding into connected components for large-scale graphical lasso","authors":"Hastie Mazumder, Rahul Mazumder, Trevor Hastie","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Newman, 2004] Mark EJ Newman. Fast algorithm for de- tecting community structure in networks. Physical review E, 69(6):066133, 2004.","title":"Fast algorithm for detecting community structure in networks","authors":"Mark Ej Newman","year":"2004","venue":"Physical review E","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Perin et al., 2013] Rodrigo Perin, Martin Telefont, and Henry Markram. Computing the size and number of neu- ronal clusters in local circuits. Frontiers in neuroanatomy, 7:1, 2013.","title":"Computing the size and number of neuronal clusters in local circuits","authors":"Rodrigo Perin, Martin Telefont, Henry Markram","year":"2013","venue":"Frontiers in neuroanatomy","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ravikumar et al., 2011] Pradeep Ravikumar, Martin J Wain- wright, Garvesh Raskutti, Bin Yu, et al. High-dimensional covariance estimation by minimizing l1-penalized log- determinant divergence. Electronic Journal of Statistics, 5:935-980, 2011.","title":"High-dimensional covariance estimation by minimizing l1-penalized logdeterminant divergence","authors":"Pradeep Ravikumar, Martin J Wainwright, Garvesh Raskutti, Bin Yu","year":"2011","venue":"Electronic Journal of Statistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rothman et al., 2009] Adam J Rothman, Elizaveta Levina, and Ji Zhu. Generalized thresholding of large covariance matrices. Journal of the American Statistical Association, 104(485):177-186, 2009.","title":"Generalized thresholding of large covariance matrices","authors":"Elizaveta Adam J Rothman, Ji Levina, Zhu","year":"2009","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shimamura et al., 2007] Teppei Shimamura, Seiya Imoto, Rui Yamaguchi, and Satoru Miyano. Weighted lasso in graphical gaussian modeling for large gene network esti- mation based on microarray data. In Genome Informatics 2007: Genome Informatics Series Vol. 19, pages 142-153. World Scientific, 2007.","title":"Weighted lasso in graphical gaussian modeling for large gene network estimation based on microarray data","authors":"Teppei Shimamura, Seiya Imoto, Rui Yamaguchi, Satoru Miyano","year":"2007","venue":"Genome Informatics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sojoudi, 2016] Somayeh Sojoudi. Equivalence of graphical lasso and thresholding for sparse graphs. The Journal of Machine Learning Research, 17(1):3943-3963, 2016.","title":"Somayeh Sojoudi. Equivalence of graphical lasso and thresholding for sparse graphs","authors":"Sojoudi","year":"2016","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tso et al., 2018] Ivy F Tso, Saige Rutherford, Yu Fang, Mike Angstadt, and Stephan F Taylor. The \"social brain\" is highly sensitive to the mere presence of social informa- tion: An automated meta-analysis and an independent study. PloS one, 13(5):e0196503, 2018.","title":"The \"social brain\" is highly sensitive to the mere presence of social information: An automated meta-analysis and an independent study","authors":"Tso","year":"2018","venue":"PloS one","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Van De Vijver et al., 2002] Marc J Van De Vijver, Yudong D He, Laura J Van't Veer, Hongyue Dai, Augustinus AM Hart, Dorien W Voskuil, George J Schreiber, Johannes L Peterse, Chris Roberts, Matthew J Marton, et al. A gene-expression signature as a predictor of survival in breast cancer. New England Journal of Medicine, 347(25):1999-2009, 2002.","title":"A gene-expression signature as a predictor of survival in breast cancer","authors":"Van De Vijver","year":"2002","venue":"New England Journal of Medicine","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Witten et al., 2011] Daniela M Witten, Jerome H Friedman, and Noah Simon. New insights and faster computations for the graphical lasso. Journal of Computational and Graphical Statistics, 20(4):892-900, 2011.","title":"New insights and faster computations for the graphical lasso","authors":"Jerome H Daniela M Witten, Noah Friedman, Simon","year":"2011","venue":"Journal of Computational and Graphical Statistics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang et al., 2014a] Eunho Yang, Aurelie Lozano, and Pradeep Ravikumar. Elementary estimators for sparse co- variance matrices and other structured moments. In Inter- national conference on machine learning, pages 397-405, 2014.","title":"Elementary estimators for sparse covariance matrices and other structured moments","authors":"Yang","year":"2014","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2014b] Eunho Yang, Aurélie C Lozano, and Pradeep K Ravikumar. Elementary estimators for graphi- cal models. In Advances in neural information processing systems, pages 2159-2167, 2014.","title":"Elementary estimators for graphical models","authors":"Yang","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yuan and Lin, 2007] Ming Yuan and Yi Lin. Model selection and estimation in the gaussian graphical model. Biometrika, 94(1):19-35, 2007.","title":"Model selection and estimation in the gaussian graphical model","authors":"Ming Yuan, Yi Lin","year":"2007","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2018] Richard Zhang, Salar Fattahi, and So- mayeh Sojoudi. Large-scale sparse inverse covariance es- timation via thresholding and max-det matrix completion. In International Conference on Machine Learning, pages 5761-5770, 2018.","title":"Large-scale sparse inverse covariance estimation via thresholding and max-det matrix completion","authors":"Richard Zhang, Salar Fattahi, Somayeh Sojoudi","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}