{"type":"ScholarlyArticle","title":"Efficient Regularization Parameter Selection for Latent Variable Graphical Models via Bi-Level Optimization","authors":[{"name":"Joachim Giesen","slug":"joachim-giesen","given":"Joachim","family":"Giesen","affiliation":"Friedrich-Schiller-Universität Jena"},{"name":"Frank Nussbaum","slug":"frank-nussbaum","given":"Frank","family":"Nussbaum","affiliation":"Friedrich-Schiller-Universität Jena"},{"name":"Christopher Schneider","slug":"christopher-schneider","given":"Christopher","family":"Schneider","affiliation":"Ernst-Abbe-Hochschule Jena"}],"abstract":"Latent variable graphical models are an extension of Gaussian graphical models that decompose the precision matrix into a sparse and a low-rank component. These models can be learned with theoretical guarantees from data via a semidefinite program. This program features two regularization terms, one for promoting sparsity and one for promoting a low rank. In practice, however, it is not straightforward to learn a good model since the model highly depends on the regularization parameters that control the relative weight of the loss function and the two regularization terms. Selecting good regularization parameters can be modeled as a bi-level optimization problem, where the upper level optimizes some form of generalization error and the lower level provides a description of the solution gamut. The solution gamut is the set of feasible solutions for all possible values of the regularization parameters. In practice, it is often not feasible to describe the solution gamut efficiently. Hence, algorithmic schemes for approximating solution gamuts have been\r\ndevised. One such scheme is Benson's generic vector optimization algorithm that comes with approximation guarantees. So far Benson's algorithm has not been used in conjunction with semidefinite programs like the latent variable graphical Lasso. Here, we develop an adaptive variant of Benson's algorithm for the semidefinite case and show that\r\nit keeps the known approximation and run time guarantees. Furthermore, Benson's algorithm turns out to be practically more efficient for the latent variable graphical model than the existing solution gamut approximation scheme on a wide range of data sets.","doi":"10.24963/ijcai.2019/330","url":"https://ijcai.org/proceedings/2019/330","pdf":"https://ijcai.org/proceedings/2019/0330.pdf","markdown":"https://ijcai.org/proceedings/2019/330.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/330","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":330,"pages":{"first":2378,"last":2384},"keywords":["Machine Learning: Learning Graphical Models","Machine Learning: Feature Selection ; Learning Sparse Models","Machine Learning: Unsupervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Benson, 1998] Harold P. Benson. An Outer Approximation Algorithm for Generating All Efficient Extreme Points in the Outcome Set of a Multiple Objective Linear Program- ming Problem. Journal of Global Optimization, 13(1):1- 24, 1998.","title":"An Outer Approximation Algorithm for Generating All Efficient Extreme Points in the Outcome Set of a Multiple Objective Linear Programming Problem","authors":"Harold P Benson","year":"1998","venue":"Journal of Global Optimization","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blechschmidt et al., 2015] Katharina Blechschmidt, Joachim Giesen, and Sören Laue. Tracking Approximate Solutions of Parameterized Optimization Problems over Multi-Dimensional (Hyper-)Parameter Domains. In International Conference on Machine Learning (ICML), pages 438-447, 2015.","title":"Tracking Approximate Solutions of Parameterized Optimization Problems over Multi-Dimensional (Hyper-)Parameter Domains","authors":"Katharina Blechschmidt, Joachim Giesen, Sören Laue","year":"2015","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bremner et al., 1998] David Bremner, Komei Fukuda, and Ambros Marzetta. Primal-Dual Methods for Vertex and Facet Enumeration. Discrete & Computational Geometry, 20(3):333-357, 1998.","title":"Primal-Dual Methods for Vertex and Facet Enumeration","authors":"David Bremner, Komei Fukuda, Ambros Marzetta","year":"1998","venue":"Discrete & Computational Geometry","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Candès and Recht, 2009] Emmanuel J. Candès and Ben- jamin Recht. Exact Matrix Completion via Convex Op- timization. Foundations of Computational Mathematics, 9(6):717-772, 2009.","title":"Exact Matrix Completion via Convex Optimization","authors":"Emmanuel J Candès, Benjamin Recht","year":"2009","venue":"Foundations of Computational Mathematics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Candès et al., 2011] Emmanuel J. Candès, Xiaodong Li, Yi Ma, and John Wright. Robust Principal Component Analysis? Journal of the ACM, 58(3):11:1-11:37, 2011.","title":"Robust Principal Component Analysis","authors":"Emmanuel J Candès, Xiaodong Li, Yi Ma, John Wright","year":"2011","venue":"Journal of the ACM","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chandrasekaran et al., 2012] Venkat Chandrasekaran, Pablo A. Parrilo, and Alan S. Willsky. Latent Variable Graphical Model Selection via Convex Optimization. The Annals of Statistics, 40(4):1935-1967, 2012.","title":"Latent Variable Graphical Model Selection via Convex Optimization","authors":"Venkat Chandrasekaran, Pablo A Parrilo, Alan S Willsky","year":"2012","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2011] Jianhui Chen, Jiayu Zhou, and Jieping Ye. Integrating Low-Rank and Group-Sparse Structures for Robust Multi-Task Learning. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 42-50, 2011.","title":"Integrating Low-Rank and Group-Sparse Structures for Robust Multi-Task Learning","authors":"Jianhui Chen, Jiayu Zhou, Jieping Ye","year":"2011","venue":"Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2017] Fen Chen, Peng Zhao, Ting Feng Tang, and Yan Zhou. Fast Low-Rank Decomposition Model- Based Hyperspectral Image Classification Method. IEEE Geoscience Remote Sensing Letters, 14(2):169-173, 2017.","title":"Fast Low-Rank Decomposition Model-Based Hyperspectral Image Classification Method","authors":"Fen Chen, Peng Zhao, Ting Feng Tang, Yan Zhou","year":"2017","venue":"IEEE Geoscience Remote Sensing Letters","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dempe, 2002] Stephan Dempe. Foundations of Bilevel Pro- gramming. Springer, 2002.","title":"Foundations of Bilevel Programming","authors":"Stephan Dempe","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dua and Karra Taniskidou, 2017] Dheeru Dua and Efi Karra Taniskidou. UCI Machine Learning Repository, 2017.","title":"Dheeru Dua and Efi Karra Taniskidou","authors":"Karra Dua, Taniskidou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dudík et al., 2004] Miroslav Dudík, Steven J. Phillips, and Robert E. Schapire. Performance Guarantees for Regular- ized Maximum Entropy Density Estimation. In Confer- ence on Learning Theory (COLT), pages 472-486, 2004.","title":"Performance Guarantees for Regularized Maximum Entropy Density Estimation","authors":"Dudík","year":"2004","venue":"Conference on Learning Theory (COLT)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Giesen et al., 2019] Joachim Giesen, Andreas Löhne, Sören Laue, and Christopher Schneider. Using Benson's Algo- rithm for Regularization Parameter Tracking. In Proceed- ings of the Thirty-Third AAAI Conference on Artificial In- telligence (AAAI), 2019.","title":"Using Benson's Algorithm for Regularization Parameter Tracking","authors":"Joachim Giesen, Andreas Löhne, Sören Laue, Christopher Schneider","year":"2019","venue":"Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Giraud and Tsybakov, 2012] Christophe Giraud and Alexandre Tsybakov. Discussion: Latent Variable Graphi- cal Model Selection via Convex Optimization. The Annals of Statistics, 40(4):1996-2004, 2012.","title":"Discussion: Latent Variable Graphical Model Selection via Convex Optimization","authors":"Christophe Giraud, Alexandre Tsybakov","year":"2012","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Giraud et al., 2012] Christophe Giraud, Sylvie Huet, and Nicolas Verzelen. Graph Selection with GGMselect. Sta- tistical Applications in Genetics and Molecular Biology, 11(3), 2012.","title":"Graph Selection with GGMselect","authors":"Christophe Giraud, Sylvie Huet, Nicolas Verzelen","year":"2012","venue":"Statistical Applications in Genetics and Molecular Biology","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hansen et al., 1992] Pierre Hansen, Brigitte Jaumard, and Gilles Savard. New Branch-and-Bound Rules for Linear Bilevel Programming. SIAM Journal on Scientific and Sta- tistical Computing, 13(5):1194-1217, 1992.","title":"New Branch-and-Bound Rules for Linear Bilevel Programming","authors":"Hansen","year":"1992","venue":"SIAM Journal on Scientific and Statistical Computing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Higuera et al., 2015] Clara Higuera, Katheleen J. Gardiner, and Krzysztof J. Cios. Self-Organizing Feature Maps Identify Proteins Critical to Learning in a Mouse Model of Down Syndrome. PloS ONE, 10(6):1-28, 2015.","title":"Self-Organizing Feature Maps Identify Proteins Critical to Learning in a Mouse Model of Down Syndrome","authors":"Clara Higuera, Katheleen J Gardiner, Krzysztof J Cios","year":"2015","venue":"PloS ONE","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kamenev, 1994] George K. Kamenev. Analysis of an Algo- rithm for Approximating Convex Bodies. Computational Mathematics and Mathematical Physics, 34(4):521-528, 1994.","title":"Analysis of an Algorithm for Approximating Convex Bodies","authors":"George K Kamenev","year":"1994","venue":"Computational Mathematics and Mathematical Physics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Löhne et al., 2014] Andreas Löhne, Birgit Rudloff, and Firdevs Ulus. Primal and Dual Approximation Algorithms for Convex Vector Optimization Problems. Journal of Global Optimization, 60(4):713-736, 2014.","title":"Primal and Dual Approximation Algorithms for Convex Vector Optimization Problems","authors":"Löhne","year":"2014","venue":"Journal of Global Optimization","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ma et al., 2013] Shiqian Ma, Lingzhou Xue, and Hui Zou. Alternating Direction Methods for Latent Variable Gaus- sian Graphical Model Selection. Neural Computation, 25(8):2172-2198, 2013.","title":"Alternating Direction Methods for Latent Variable Gaussian Graphical Model Selection","authors":"Ma","year":"2013","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Song et al., 2007] Le Song, Alexander J. Smola, Karsten M. Borgwardt, and Arthur Gretton. Colored Maximum Vari- ance Unfolding. In Advances in Neural Information Pro- cessing Systems (NIPS), pages 1385-1392, 2007.","title":"Colored Maximum Variance Unfolding","authors":"Le Song, Alexander J Smola, Karsten M Borgwardt, Arthur Gretton","year":"2007","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Spearman, 1904] Charles Spearman. \"General Intelli- gence,\" Objectively Determined and Measured. American Journal of Psychology, 15:201-293, 1904.","title":"General Intelligence,\" Objectively Determined and Measured","authors":"Charles Spearman, Spearman","year":"1904","venue":"American Journal of Psychology","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tsanas et al., 2014] Athanasios Tsanas, Max A. Little, Cyn- thia Fox, and Lorraine O. Ramig. Objective Automatic Assessment of Rehabilitative Speech Treatment in Parkin- son's Disease. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 22(1):181-190, 2014.","title":"Objective Automatic Assessment of Rehabilitative Speech Treatment in Parkinson's Disease","authors":"Athanasios Tsanas, Max A Little, Cynthia Fox, Lorraine O Ramig","year":"2014","venue":"IEEE Transactions on Neural Systems and Rehabilitation Engineering","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[von Stackelberg, 2010] Heinrich von Stackelberg. Market Structure and Equilibrium. Springer Berlin Heidelberg, 2010.","title":"Heinrich von Stackelberg. Market Structure and Equilibrium","authors":"Von Stackelberg","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yuan and Lin, 2007] Ming Yuan and Yi Lin. Model Se- lection and Estimation in the Gaussian Graphical Model. Biometrika, 94(1):19-35, 2007.","title":"Model Selection and Estimation in the Gaussian Graphical Model","authors":"Lin ; Yuan, Ming Yuan, Yi Lin","year":"2007","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou et al., 2014] Fang Zhou, Q. Claire, and Ross D. King. Predicting the Geographical Origin of Music. In IEEE International Conference on Data Mining (ICDM), pages 1115-1120, 2014.","title":"Predicting the Geographical Origin of Music","authors":"Fang Zhou, Q Claire, Ross D King","year":"2014","venue":"IEEE International Conference on Data Mining (ICDM)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}