{"type":"ScholarlyArticle","title":"Faster Training Algorithms for Structured Sparsity-Inducing Norm","authors":[{"name":"Bin Gu","slug":"bin-gu","given":"Bin","family":"Gu","affiliation":"* School of Computer & Software , Nanjing University of Information Science & Technology , P.R.China; † Department of Electrical & Computer Engineering , University of Pittsburgh , USA"},{"name":"Xingwang Ju","slug":"xingwang-ju","given":"Xingwang","family":"Ju","affiliation":"* School of Computer & Software , Nanjing University of Information Science & Technology , P.R.China"},{"name":"Xiang Li","slug":"xiang-li","given":"Xiang","family":"Li","affiliation":"‡ Computer Science Department , University of Western Ontario , Canada"},{"name":"Guansheng Zheng","slug":"guansheng-zheng","given":"Guansheng","family":"Zheng","affiliation":"* School of Computer & Software , Nanjing University of Information Science & Technology , P.R.China"}],"abstract":"Structured-sparsity regularization is popular for sparse learning because of its flexibility of encoding the feature structures. This paper considers a generalized version of structured-sparsity regularization (especially for $l_1/l_{\\infty}$ norm) with arbitrary group overlap. Due to the group overlap, it is time-consuming to solve the associated proximal operator. Although Mairal~\\shortcite{mairal2010network} have proposed a  network-flow  algorithm to solve the proximal operator, it is still time-consuming especially in the high-dimensional setting. To address this challenge, in this paper, we have developed a more efficient solution for $l_1/l_{\\infty}$ group lasso with arbitrary group overlap using an Inexact Proximal-Gradient method. In each iteration, our algorithm only requires to calculate an inexact solution to the proximal sub-problem, which can be done efficiently. On the theoretic side, the proposed algorithm enjoys the same global convergence rate as the exact proximal methods. Experiments demonstrate that our algorithm is much more efficient than network-flow algorithm, while retaining the similar generalization performance.","doi":"10.24963/ijcai.2018/299","url":"https://ijcai.org/proceedings/2018/299","pdf":"https://ijcai.org/proceedings/2018/0299.pdf","markdown":"https://ijcai.org/proceedings/2018/299.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/299","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":299,"pages":{"first":2163,"last":2169},"keywords":["Constraints and SAT: Constraint Optimisation","Constraints and SAT: Constraints and Data Mining ; Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Bach et al., 2012] Francis Bach, Rodolphe Jenatton, Julien Mairal, Guillaume Obozinski, et al. Optimization with sparsity-inducing penalties. Foundations and Trends in Machine Learning, 4(1):1-106, 2012.","title":"Optimization with sparsity-inducing penalties","authors":"Francis Bach, Rodolphe Jenatton, Julien Mairal, Guillaume Obozinski","year":"2012","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Beck and Teboulle, 2009] Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for lin- ear inverse problems. SIAM journal on imaging sciences, 2(1):183-202, 2009.","title":"A fast iterative shrinkage-thresholding algorithm for linear inverse problems","authors":"Amir Beck, Marc Teboulle","year":"2009","venue":"SIAM journal on imaging sciences","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Boyd and Vandenberghe, 2004] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge universi- ty press, 2004.","title":"Convex optimization","authors":"Vandenberghe Boyd, Stephen Boyd, Lieven Vandenberghe","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Efron et al., 2004] Bradley Efron, Trevor Hastie, Iain John- stone, Robert Tibshirani, et al. Least angle regression. The Annals of statistics, 32(2):407-499, 2004.","title":"Least angle regression","authors":"Bradley Efron, Trevor Hastie, Iain Johnstone, Robert Tibshirani","year":"2004","venue":"The Annals of statistics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Friedman et al., 2008] Jerome Friedman, Trevor Hastie, and Robert Tibshirani. Sparse inverse covariance estima- tion with the graphical lasso. Biostatistics, 9(3):432-441, 2008.","title":"Sparse inverse covariance estimation with the graphical lasso","authors":"Jerome Friedman, Trevor Hastie, Robert Tibshirani","year":"2008","venue":"Biostatistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gu and Ling, 2015] Bin Gu and Charles Ling. A new gen- eralized error path algorithm for model selection. In Inter- national Conference on Machine Learning, pages 2549- 2558, 2015.","title":"A new generalized error path algorithm for model selection","authors":"Ling ; ] Bin Gu, Charles Gu, Ling","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gu et al., 2017a] Bin Gu, Guodong Liu, and Heng Huang. Groups-keeping solution path algorithm for sparse regres- sion with automatic feature grouping. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 185-193. ACM, 2017.","title":"Groups-keeping solution path algorithm for sparse regression with automatic feature grouping","authors":"Gu","year":"2017","venue":"Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gu et al., 2017b] Bin Gu, Victor S Sheng, Keng Yeow Tay, Walter Romano, and Shuo Li. Cross validation through two-dimensional solution surface for cost-sensitive svm. IEEE transactions on pattern analysis and machine intel- ligence, 39(6):1103-1121, 2017.","title":"Cross validation through two-dimensional solution surface for cost-sensitive svm","authors":"Gu","year":"2017","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gu et al., 2018a] Bin Gu, Zhouyuan Huo, and Heng Huang. Asynchronous doubly stochastic group regularized learn- ing. In International Conference on Artificial Intelligence and Statistics, pages 1791-1800, 2018.","title":"Asynchronous doubly stochastic group regularized learning","authors":"Gu","year":"2018","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gu et al., 2018b] Bin Gu, Xin Miao, Zhouyuan Huo, and Heng Huang. Asynchronous doubly stochastic sparse ker- nel learning. In AAAI, 2018.","title":"Asynchronous doubly stochastic sparse kernel learning","authors":"Gu","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gu et al., 2018c] Bin Gu, De Wang, Zhouyuan Huo, and Heng Huang. Inexact proximal gradient methods for non- convex and non-smooth optimization. In AAAI, 2018.","title":"Inexact proximal gradient methods for nonconvex and non-smooth optimization","authors":"Gu","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hegde et al., 2015] Chinmay Hegde, Piotr Indyk, and Lud- wig Schmidt. A nearly-linear time framework for graph- structured sparsity. In International Conference on Ma- chine Learning, pages 928-937, 2015.","title":"A nearly-linear time framework for graphstructured sparsity","authors":"Chinmay Hegde, Piotr Indyk, Ludwig Schmidt","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huo et al., 2018] Zhouyuan Huo, Bin Gu, Ji Liu, and Heng Huang. Accelerated method for stochastic composition optimization with nonsmooth regularization. In AAAI, 2018.","title":"Accelerated method for stochastic composition optimization with nonsmooth regularization","authors":"Huo","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jacob et al., 2009] Laurent Jacob, Guillaume Obozinski, and Jean-Philippe Vert. Group lasso with overlap and graph lasso. In Proceedings of the 26th annual internation- al conference on machine learning, pages 433-440. ACM, 2009.","title":"Group lasso with overlap and graph lasso","authors":"Laurent Jacob, Guillaume Obozinski, Jean-Philippe Vert","year":"2009","venue":"Proceedings of the 26th annual international conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jenatton et al., 2010] Rodolphe Jenatton, Julien Mairal, Guillaume Obozinski, and Francis R Bach. Proximal methods for sparse hierarchical dictionary learning. In ICML, number 2010, pages 487-494. Citeseer, 2010.","title":"Proximal methods for sparse hierarchical dictionary learning","authors":"Jenatton","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kim and Xing, 2010] Seyoung Kim and Eric P Xing. Tree- guided group lasso for multi-task regression with struc- tured sparsity. 2010.","title":"Treeguided group lasso for multi-task regression with structured sparsity","authors":"Xing ; Kim, Seyoung Kim, Eric P Xing","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kohavi and others, 1995] Ron Kohavi et al. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Ijcai, volume 14, pages 1137-1145.","title":"A study of cross-validation and bootstrap for accuracy estimation and model selection","authors":"Ron Kohavi, Kohavi","year":"1995","venue":"Ijcai","doi":null,"url":null,"internal_url":"/proceedings/1995/147"},{"position":18,"raw":"[Mairal et al., 2010] Julien Mairal, Rodolphe Jenatton, Fran- cis R Bach, and Guillaume R Obozinski. Network flow algorithms for structured sparsity. In Advances in Neural Information Processing Systems, pages 1558-1566, 2010.","title":"Network flow algorithms for structured sparsity","authors":"Julien Mairal, Rodolphe Jenatton, Francis R Bach, Guillaume R Obozinski","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nesterov and others, 2007] Yurii Nesterov et al. Gradien- t methods for minimizing composite objective function, 2007.","title":"Gradient methods for minimizing composite objective function","authors":"Yurii Nesterov","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Neumaier and Groeneveld, 1998] Arnold Neumaier and Eildert Groeneveld. Restricted maximum likelihood esti- mation of covariances in sparse linear models. Genetics Selection Evolution, 30(1):3, 1998.","title":"Restricted maximum likelihood estimation of covariances in sparse linear models","authors":"Groeneveld Neumaier, Arnold Neumaier, Eildert Groeneveld","year":"1998","venue":"Genetics Selection Evolution","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Quattoni et al., 2008] Ariadna Quattoni, Michael Collins, and Trevor Darrell. Transfer learning for image classifi- cation with sparse prototype representations. In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, pages 1-8. IEEE, 2008.","title":"Transfer learning for image classification with sparse prototype representations","authors":"Ariadna Quattoni, Michael Collins, Trevor Darrell","year":"2008","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Quattoni et al., 2009] Ariadna Quattoni, Xavier Carreras, Michael Collins, and Trevor Darrell. An efficient pro- jection for l 1, ∞ regularization. In Proceedings of the 26th Annual International Conference on Machine Learn- ing, pages 857-864. ACM, 2009.","title":"An efficient projection for l 1, ∞ regularization","authors":"Ariadna Quattoni, Xavier Carreras, Michael Collins, Trevor Darrell","year":"2009","venue":"Proceedings of the 26th Annual International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schmidt et al., 2011] Mark Schmidt, Nicolas L Roux, and Francis R Bach. Convergence rates of inexact proximal- gradient methods for convex optimization. In Advances in neural information processing systems, pages 1458-1466, 2011.","title":"Convergence rates of inexact proximalgradient methods for convex optimization","authors":"Mark Schmidt, Nicolas L Roux, Francis R Bach","year":"2011","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Villa et al., 2014] Silvia Villa, Lorenzo Rosasco, Sofia Mosci, and Alessandro Verri. Proximal methods for the latent group lasso penalty. Computational Optimization and Applications, 58(2):381-407, 2014.","title":"Proximal methods for the latent group lasso penalty","authors":"Silvia Villa, Lorenzo Rosasco, Sofia Mosci, Alessandro Verri","year":"2014","venue":"Computational Optimization and Applications","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu et al., 2012] Guoshen Yu, Guillermo Sapiro, and Stéphane Mallat. Solving inverse problems with piece- wise linear estimators: From gaussian mixture models to structured sparsity. IEEE Transactions on Image Processing, 21(5):2481-2499, 2012.","title":"Solving inverse problems with piecewise linear estimators: From gaussian mixture models to structured sparsity","authors":"Yu","year":"2012","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yuan et al., 2011] Lei Yuan, Jun Liu, and Jieping Ye. Effi- cient methods for overlapping group lasso. In Advances in Neural Information Processing Systems, pages 352-360, 2011.","title":"Efficient methods for overlapping group lasso","authors":"Lei Yuan, Jun Liu, Jieping Ye","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhong and Kwok, 2012] Leon Wenliang Zhong and James T Kwok. Efficient sparse modeling with automatic feature grouping. IEEE transactions on neural networks and learning systems, 23(9):1436-1447, 2012.","title":"Efficient sparse modeling with automatic feature grouping","authors":"Kwok ; ] Zhong, Wenliang Leon, James T Zhong, Kwok","year":"2012","venue":"IEEE transactions on neural networks and learning systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}