{"type":"ScholarlyArticle","title":"Learning With Subquadratic Regularization : A Primal-Dual Approach","authors":[{"name":"Raman Sankaran","slug":"raman-sankaran","given":"Raman","family":"Sankaran","affiliation":"Indian Institute of Science , Bengaluru"},{"name":"Francis Bach","slug":"francis-bach","given":"Francis","family":"Bach","affiliation":"INRIA -Ecole Normale Supérieure -PSL Research University , Paris"},{"name":"Chiranjib Bhattacharyya","slug":"chiranjib-bhattacharyya","given":"Chiranjib","family":"Bhattacharyya","affiliation":"Indian Institute of Science , Bengaluru"}],"abstract":"Subquadratic norms have been studied recently in the context of structured sparsity, which has been shown to be more beneficial than  conventional regularizers in applications such as image denoising, compressed sensing, banded covariance estimation, etc. While existing works have been successful in learning structured sparse models such as trees, graphs, their associated optimization procedures have been inefficient because of hard-to-evaluate proximal operators of the norms. In this paper, we study the computational aspects of learning with subquadratic norms in a general setup. Our main contributions are two proximal-operator based algorithms ADMM-η and CP-η, which generically apply to these learning problems with convex loss functions, and achieve a proven rate of convergence of O(1/T) after T iterations. These algorithms are derived in a primal-dual framework, which have not been examined for subquadratic norms.  We illustrate the efficiency of the algorithms developed in the context of tree-structured sparsity, where they comprehensively outperform relevant baselines.","doi":"10.24963/ijcai.2020/272","url":"https://ijcai.org/proceedings/2020/272","pdf":"https://ijcai.org/proceedings/2020/0272.pdf","markdown":"https://ijcai.org/proceedings/2020/272.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/272","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":272,"pages":{"first":1963,"last":1969},"keywords":["Data Mining: Feature Extraction, Selection and Dimensionality Reduction","Machine Learning: Feature Selection; Learning Sparse Models"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/26663/file_26663.m3u8","length_sec":890,"provider":"hls"}],"references":[{"position":1,"raw":"[Argyriou et al., 2008] A. Argyriou, T. Evgeniou, and M. Pontil. Convex multi-task feature learning. Machine Learning, 73(3):243-272, 2008.","title":"Convex multi-task feature learning","authors":"A Argyriou, T Evgeniou, M Pontil","year":"2008","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bach et al., 2011] F. Bach, R. Jenatton, J. Mairal, and G. Obozinski. Optimization with sparsity-inducing penal- ties. Foundations and Trends in Machine Learning, 4(1):1-106, 2011.","title":"Optimization with sparsity-inducing penalties","authors":"F Bach, R Jenatton, J Mairal, G Obozinski","year":"2011","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bach, 2010] F. Bach. Structured sparsity-inducing norms through submodular functions. In Advances in Neural In- formation Processing Systems (NIPS), 2010.","title":"Structured sparsity-inducing norms through submodular functions","authors":"F Bach","year":"2010","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bach, 2011a] F. Bach. Learning with submodular func- tions: A convex optimization perspective. Foundations and Trends in Machine Learning, 6-2-3:145-373, 2011.","title":"Learning with submodular functions: A convex optimization perspective","authors":"; F Bach, Bach","year":"2011","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bach, 2011b] F. Bach. Shaping level sets with submodular functions. In Advances in Neural Information Processing Systems (NIPS), 2011.","title":"Shaping level sets with submodular functions","authors":"; F Bach, Bach","year":"2011","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Baldassarre et al., 2012] Luca Baldassarre, Jean Morales, Andreas Argyriou, and Massimiliano Pontil. A general framework for structured sparsity via proximal optimiza- tion. In Artificial Intelligence and Statistics, pages 82-90, 2012.","title":"A general framework for structured sparsity via proximal optimization","authors":"Luca Baldassarre, Jean Morales, Andreas Argyriou, Massimiliano Pontil","year":"2012","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Beck and Teboulle, 2009] A. Beck and M. Teboulle. A fast iterative shrinkage-thresholding algorithm for linear in- verse problems. SIAM J. Img. Sci., 2(1):183-202, March 2009.","title":"A fast iterative shrinkage-thresholding algorithm for linear inverse problems","authors":"A Beck, M Teboulle","year":"2009","venue":"SIAM J. Img. Sci","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Best and Chakravarti, 1990] M. J. Best and N. Chakravarti. Active set algorithms for isotonic regression: a unifying framework. Mathematical Programming, 47(1):425-439, 1990.","title":"Active set algorithms for isotonic regression: a unifying framework","authors":"M J Best, N Chakravarti","year":"1990","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Canu and Grandvalet, 1999] S. Canu and Y. Grandvalet. Outcomes of the equivalence of adaptive ridge with least absolute shrinkage. Adv. NIPS, 1999.","title":"Outcomes of the equivalence of adaptive ridge with least absolute shrinkage","authors":"S Canu, Y Grandvalet","year":"1999","venue":"Adv. NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chambolle and Pock, 2011] A. Chambolle and T. Pock. A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis., 40(1):120- 145, May 2011.","title":"A first-order primal-dual algorithm for convex problems with applications to imaging","authors":"A Chambolle, T Pock","year":"2011","venue":"J. Math. Imaging Vis","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Daubechies et al., 2010] I. Daubechies, R. DeVore, M. For- nasier, and C. S. Güntürk. Iteratively reweighted least squares minimization for sparse recovery. Communica- tions on Pure and Applied Mathematics, 63(1):1-38, 2010.","title":"Iteratively reweighted least squares minimization for sparse recovery","authors":"Daubechies","year":"2010","venue":"Communications on Pure and Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jenatton et al., 2011a] R. Jenatton, A. Gramfort, V. Michel, G. Obozinski, F. Bach, and B. Thirion. Multi-scale mining of fmri data with hierarchical structured sparsity. Pattern Recognition in NeuroImaging (PRNI), 2011.","title":"Multi-scale mining of fmri data with hierarchical structured sparsity","authors":"Jenatton","year":"2011","venue":"Pattern Recognition in NeuroImaging","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jenatton et al., 2011b] R. Jenatton, J. Mairal, G. Obozinski, and F. Bach. Proximal methods for hierarchical sparse coding. J. Mach. Learn. Res., 12:2297-2334, July 2011.","title":"Proximal methods for hierarchical sparse coding","authors":"Jenatton","year":"2011","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kloft et al., 2011] M. Kloft, U. Brefeld, S. Sonnenburg, and A. Zien. l p -norm multiple kernel learning. Journal of Ma- chine Learning Research, 12:953-997, Mar 2011.","title":"l p -norm multiple kernel learning","authors":"M Kloft, U Brefeld, S Sonnenburg, A Zien","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kyrillidis, 2016] Kyrillidis. Structured sparsity: discrete and convex approaches. Foundations and Trends in Ma- chine Learning, 2016.","title":"Kyrillidis. Structured sparsity: discrete and convex approaches","authors":"Kyrillidis","year":"2016","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lee et al., 2006] Su-In Lee, Honglak Lee, Pieter Abbeel, and Andrew Y. Ng. Efficient l1 regularized logistic regres- sion. In Neural Information Processing Systems, 2006.","title":"Efficient l1 regularized logistic regression","authors":"Lee","year":"2006","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mairal et al., 2014] J. Mairal, F. Bach, and J. Ponce. Sparse modeling for image and vision processing. Foundations and Trends in Computer Graphics and Vision, 8(2-3):85- 283, 2014.","title":"Sparse modeling for image and vision processing","authors":"J Mairal, F Bach, J Ponce","year":"2014","venue":"Foundations and Trends in Computer Graphics and Vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[McDonald et al., 2016] Andrew M McDonald, Massimil- iano Pontil, and Dimitris Stamos. New perspectives on k-support and cluster norms. Journal of Machine Learn- ing Research, 17(155):1-38, 2016.","title":"New perspectives on k-support and cluster norms","authors":"Massimiliano Andrew M Mcdonald, Dimitris Pontil, Stamos","year":"2016","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Micchelli et al., 2013] C. Micchelli, J. Morales, and M. Pontil. Regularizers for structured sparsity. Advances in Computational Mathematics, 38(3):455-489, 2013.","title":"Regularizers for structured sparsity","authors":"C Micchelli, J Morales, M Pontil","year":"2013","venue":"Advances in Computational Mathematics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nesterov, 2007] Y. Nesterov. Gradient methods for mini- mizing composite objective function. CORE Discussion Paper, 2007.","title":"Gradient methods for minimizing composite objective function","authors":"Y Nesterov","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nesterov, 2013] Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2013.","title":"Introductory lectures on convex optimization: A basic course","authors":"Yurii Nesterov","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Obozinski and Bach, 2016] Guillaume Obozinski and Fran- cis Bach. A unified perspective on convex structured spar- sity: Hierarchical, symmetric, submodular norms and be- yond. Technical report, HAL-01412385, December 2016.","title":"Guillaume Obozinski and Francis Bach. A unified perspective on convex structured sparsity: Hierarchical, symmetric, submodular norms and beyond","authors":"Bach Obozinski","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Obozinski et al., 2011] G. Obozinski, L. Jacob, and J.P. Vert. Group lasso with overlaps: The latent group lasso approach. ArXiv preprint:1110.0413v1, 2011.","title":"Group lasso with overlaps: The latent group lasso approach","authors":"G Obozinski, L Jacob, J P Vert","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pardalos and Xue, 1999] Panos M Pardalos and Guoliang Xue. Algorithms for a class of isotonic regression prob- lems. Algorithmica, 23(3):211-222, 1999.","title":"Algorithms for a class of isotonic regression problems","authors":"M Panos, Guoliang Pardalos, Xue","year":"1999","venue":"Algorithmica","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Parikh and Boyd, 2014] N. Parikh and S. Boyd. Proximal algorithms. Foundations and Trends in Optimization, 1(3):123-231, 2014.","title":"Proximal algorithms","authors":"N Parikh, S Boyd","year":"2014","venue":"Foundations and Trends in Optimization","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sankaran et al., 2017] Raman Sankaran, Francis Bach, and Chiranjib Bhattacharya. Identifying groups of strongly correlated variables through smoothed ordered weighted l 1-norms. In Artificial Intelligence and Statistics, pages 1123-1131, 2017.","title":"Identifying groups of strongly correlated variables through smoothed ordered weighted l 1-norms","authors":"Raman Sankaran, Francis Bach, Chiranjib Bhattacharya","year":"2017","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Scheinberg et al., 2014] Katya Scheinberg, Donald Gold- farb, and Xi Bai. Fast first-order methods for composite convex optimization with backtracking. Foundations of Computational Mathematics, 14(3):389-417, 2014.","title":"Fast first-order methods for composite convex optimization with backtracking","authors":"Katya Scheinberg, Donald Goldfarb, Xi Bai","year":"2014","venue":"Foundations of Computational Mathematics","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tibshirani, 1994] R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B, 58:267-288, 1994.","title":"Regression shrinkage and selection via the lasso","authors":"R Tibshirani","year":"1994","venue":"Journal of the Royal Statistical Society, Series B","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Vapnik, 2000] V. Vapnik. The Nature of Statistical Learning Theory. Springer-Verlag, 2000.","title":"The Nature of Statistical Learning Theory","authors":"V Vapnik","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[W. Kienzle, 2006] K. Chellapilla W. Kienzle. Personal- ized handwriting recognition via biased regularization. In Proceedings of the International Conference on Machine Learning(ICML), 2006.","title":"Personalized handwriting recognition via biased regularization","authors":"K, Chellapilla W Kienzle","year":"2006","venue":"Proceedings of the International Conference on Machine Learning(ICML)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yan and Bien, 2015] Xiaohan Yan and Jacob Bien. Hierar- chical sparse modeling: A choice of two group lasso for- mulations. arXiv preprint arXiv:1512.01631, 2015.","title":"Hierarchical sparse modeling: A choice of two group lasso formulations","authors":"Xiaohan Yan, Jacob Bien","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}