{"type":"ScholarlyArticle","title":"Doubly Sparse Asynchronous Learning for Stochastic Composite Optimization","authors":[{"name":"Runxue Bao","slug":"runxue-bao","given":"Runxue","family":"Bao","affiliation":"Electrical and Computer Engineering Department , University of Pittsburgh , PA , USA"},{"name":"Xidong Wu","slug":"xidong-wu","given":"Xidong","family":"Wu","affiliation":"Electrical and Computer Engineering Department , University of Pittsburgh , PA , USA"},{"name":"Wenhan Xian","slug":"wenhan-xian","given":"Wenhan","family":"Xian","affiliation":"Electrical and Computer Engineering Department , University of Pittsburgh , PA , USA"},{"name":"Heng Huang","slug":"heng-huang","given":"Heng","family":"Huang","affiliation":"Electrical and Computer Engineering Department , University of Pittsburgh , PA , USA"}],"abstract":"Parallel optimization has become popular for large-scale learning in the past decades. However, existing methods suffer from huge computational costs, memory usage, and communication burden in high-dimensional scenarios. To address the challenges, we propose a new accelerated doubly sparse asynchronous learning (DSAL) method for stochastic composite optimization, under which two algorithms are proposed on shared-memory and distributed-memory architecture respectively, which only conducts gradient descent on the nonzero coordinates (data sparsity) and active set (model sparsity). The proposed algorithm can converge much faster and achieve significant speedup by simultaneously enjoying the sparsity of the model and data. Moreover, by sending the gradients on the active set only, communication costs are dramatically reduced. Theoretically, we prove that the proposed method achieves the linear convergence rate with lower overall complexity and can achieve the model identification in a finite number of iterations almost surely. Finally, extensive experimental results on benchmark datasets confirm the superiority of our proposed method.","doi":"10.24963/ijcai.2022/266","url":"https://ijcai.org/proceedings/2022/266","pdf":"https://ijcai.org/proceedings/2022/0266.pdf","markdown":"https://ijcai.org/proceedings/2022/266.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/266","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Data Mining","track":"Main Track","number":266,"pages":{"first":1916,"last":1922},"keywords":["Data Mining: Parallel, Distributed and Cloud-based High Performance Mining","Machine Learning: Optimisation","Data Mining: Big Data and Scalability","Machine Learning: Feature Extraction, Selection and Dimensionality Reduction"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986951","length_sec":79,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985094","length_sec":320,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Agarwal and Duchi, 2012] Alekh Agarwal and John C Duchi. Distributed delayed stochastic optimization. In IEEE CDC, 2012.","title":"Distributed delayed stochastic optimization","authors":"Alekh Agarwal, John C Duchi","year":"2012","venue":"IEEE CDC","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bao et al., 2019] Runxue Bao, Bin Gu, and Heng Huang. Efficient approximate solution path algorithm for ordered weighted l 1-norm with accuracy guarantee. In IEEE ICDM, 2019.","title":"Efficient approximate solution path algorithm for ordered weighted l 1-norm with accuracy guarantee","authors":"Bao","year":"2019","venue":"IEEE ICDM","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bao et al., 2020] Runxue Bao, Bin Gu, and Heng Huang. Fast oscar and owl with safe screening rules. In ICML, 2020.","title":"Fast oscar and owl with safe screening rules","authors":"Bao","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bogdan et al., 2015] Małgorzata Bogdan, Ewout Van Den Berg, Chiara Sabatti, et al. Slope-adaptive variable selection via convex optimization. Ann. Appl. Stat, 2015.","title":"Slope-adaptive variable selection via convex optimization","authors":"Małgorzata Bogdan, Ewout Van Den, Chiara Berg, Sabatti","year":"2015","venue":"Ann. Appl. Stat","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chang and Lin, 2011] Chih-Chung Chang and Chih-Jen Lin. Libsvm: A library for support vector machines. ACM TIST, 2011.","title":"Libsvm: A library for support vector machines","authors":"Chang, Lin ; Chih-Chung Chang, Chih-Jen Lin","year":"2011","venue":"ACM TIST","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dean et al., 2012] Jeffrey Dean, Greg Corrado, Rajat Monga, et al. Large scale distributed deep networks. In NeurIPS, 2012.","title":"Large scale distributed deep networks","authors":"Jeffrey Dean, Greg Corrado, Rajat Monga","year":"2012","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ghaoui et al., 2010] Laurent El Ghaoui, Vivian Viallon, and Tarek Rabbani. Safe feature elimination in sparse super- vised learning. Technical report, 2010.","title":"Safe feature elimination in sparse supervised learning","authors":"Laurent El Ghaoui, Vivian Viallon, Tarek Rabbani","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gui and Li, 2005] Jiang Gui and Hongzhe Li. Penalized cox regression analysis in the high-dimensional and low- sample size settings, with applications to microarray gene expression data. Bioinformatics, 2005.","title":"Penalized cox regression analysis in the high-dimensional and lowsample size settings, with applications to microarray gene expression data","authors":"Gui, Li ; ] Jiang, Gui, Hongzhe Li","year":"2005","venue":"Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Joulani et al., 2019] Pooria Joulani, András György, and Csaba Szepesvari. Think out of the \"box\": Generically- constrained asynchronous composite optimization and hedging. NeurIPS, 2019.","title":"Think out of the \"box\": Genericallyconstrained asynchronous composite optimization and hedging","authors":"Joulani","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Langford et al., 2009] John Langford, Alexander J Smola, and Martin Zinkevich. Slow learners are fast. In NeurIPS, 2009.","title":"Slow learners are fast","authors":"John Langford, Alexander J Smola, Martin Zinkevich","year":"2009","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Leblond et al., 2017] Rémi Leblond, Fabian Pedregosa, and Simon Lacoste-Julien. Asaga: asynchronous parallel saga. In AISTATS, 2017.","title":"Asaga: asynchronous parallel saga","authors":"Rémi Leblond, Fabian Pedregosa, Simon Lacoste-Julien","year":"2017","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Leblond et al., 2018] Rémi Leblond, Fabian Pedregosa, and Simon Lacoste-Julien. Improved asynchronous parallel optimization analysis for stochastic incremental methods. JMLR, 2018.","title":"Improved asynchronous parallel optimization analysis for stochastic incremental methods","authors":"Rémi Leblond, Fabian Pedregosa, Simon Lacoste-Julien","year":"2018","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2016a] Qingyang Li, Shuang Qiu, Shuiwang Ji, et al. Parallel lasso screening for big data optimization. In ACM SIGKDD, 2016.","title":"Parallel lasso screening for big data optimization","authors":"Li","year":"2016","venue":"ACM SIGKDD","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2016b] Yitan Li, Linli Xu, Xiaowei Zhong, and Qing Ling. Make workers work harder: decoupled asyn- chronous proximal stochastic gradient descent. arXiv preprint arXiv:1605.06619, 2016.","title":"Make workers work harder: decoupled asynchronous proximal stochastic gradient descent","authors":"Li","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lian et al., 2015] Xiangru Lian, Yijun Huang, Yuncheng Li, and Ji Liu. Asynchronous parallel stochastic gradient for nonconvex optimization. In NeurIPS, 2015.","title":"Asynchronous parallel stochastic gradient for nonconvex optimization","authors":"Lian","year":"2015","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Meng et al., 2017] Qi Meng, Wei Chen, Jingcheng Yu, et al. Asynchronous stochastic proximal optimization al- gorithms with variance reduction. In AAAI, 2017.","title":"Asynchronous stochastic proximal optimization algorithms with variance reduction","authors":"Qi Meng, Wei Chen, Jingcheng Yu","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ndiaye et al., 2017] Eugene Ndiaye, Olivier Fercoq, Alexandre Gramfort, et al. Gap safe screening rules for sparsity enforcing penalties. JMLR, 2017.","title":"Gap safe screening rules for sparsity enforcing penalties","authors":"Ndiaye","year":"2017","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ng, 2004] Andrew Y Ng. Feature selection, l 1 vs. l 2 regu- larization, and rotational invariance. In ICML, 2004.","title":"Feature selection, l 1 vs. l 2 regularization, and rotational invariance","authors":"Y Andrew, Ng","year":"2004","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nguyen et al., 2018] Lam Nguyen, Phuong Ha Nguyen, Marten Dijk, et al. Sgd and hogwild! convergence without the bounded gradients assumption. In ICML, 2018.","title":"Sgd and hogwild! convergence without the bounded gradients assumption","authors":"Lam Nguyen, Phuong Ha Nguyen, Marten Dijk","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pedregosa et al., 2017] Fabian Pedregosa, Rémi Leblond, and Simon Lacoste-Julien. Breaking the nonsmooth bar- rier: A scalable parallel method for composite optimiza- tion. In NeurIPS, 2017.","title":"Breaking the nonsmooth barrier: A scalable parallel method for composite optimization","authors":"Fabian Pedregosa, Rémi Leblond, Simon Lacoste-Julien","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Recht et al., 2011] Benjamin Recht, Christopher Ré, Stephen J Wright, et al. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In NeurIPS, 2011.","title":"Hogwild: A lock-free approach to parallelizing stochastic gradient descent","authors":"Benjamin Recht, Christopher Ré, Stephen J Wright","year":"2011","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Reddi et al., 2015] Sashank J Reddi, Ahmed Hefny, Suvrit Sra, et al. On variance reduction in stochastic gradient descent and its asynchronous variants. In NeurIPS, 2015.","title":"On variance reduction in stochastic gradient descent and its asynchronous variants","authors":"J Sashank, Ahmed Reddi, Suvrit Hefny, Sra","year":"2015","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Stich et al., 2021] Sebastian Stich, Amirkeivan Mo- htashami, and Martin Jaggi. Critical parameters for scalable distributed learning with large batches and asynchronous updates. In AISTATS, 2021.","title":"Critical parameters for scalable distributed learning with large batches and asynchronous updates","authors":"Sebastian Stich, Amirkeivan Mohtashami, Martin Jaggi","year":"2021","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tibshirani and others, 2013] Ryan J Tibshirani et al. The lasso problem and uniqueness. Electron. J. Stat., 2013.","title":"The lasso problem and uniqueness","authors":"J Ryan, Tibshirani","year":"2013","venue":"Electron. J. Stat","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tibshirani et al., 2012] Robert Tibshirani, Jacob Bien, Jerome Friedman, et al. Strong rules for discarding predictors in lasso-type problems. JRSS, 2012.","title":"Strong rules for discarding predictors in lasso-type problems","authors":"Robert Tibshirani, Jacob Bien, Jerome Friedman","year":"2012","venue":"JRSS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tibshirani, 1996] Robert Tibshirani. Regression shrinkage and selection via the lasso. JRSS, 1996.","title":"Regression shrinkage and selection via the lasso","authors":"Robert Tibshirani","year":"1996","venue":"JRSS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2012] Gang Wang, Derek Hoiem, and David Forsyth. Learning image similarity from flickr groups us- ing fast kernel machines. IEEE TPAMI, 2012.","title":"Learning image similarity from flickr groups using fast kernel machines","authors":"Wang","year":"2012","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wu et al., 2013] Xindong Wu, Xingquan Zhu, Gong-Qing Wu, and Wei Ding. Data mining with big data. IEEE TKDE, 2013.","title":"Data mining with big data","authors":"Xindong Wu, Xingquan Zhu, Gong-Qing Wu, Wei Ding","year":"2013","venue":"IEEE TKDE","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xiao and Zhang, 2014] Lin Xiao and Tong Zhang. A prox- imal stochastic gradient method with progressive variance reduction. SIOPT, 2014.","title":"A proximal stochastic gradient method with progressive variance reduction","authors":"Lin Xiao, Tong Zhang","year":"2014","venue":"SIOPT","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yuan and Lin, 2006] Ming Yuan and Yi Lin. Model selec- tion and estimation in regression with grouped variables. JRSS, 2006.","title":"Model selection and estimation in regression with grouped variables","authors":"Lin ; Yuan, Ming Yuan, Yi Lin","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang and Kwok, 2014] Ruiliang Zhang and James Kwok. Asynchronous distributed admm for consensus optimiza- tion. In ICML, 2014.","title":"Asynchronous distributed admm for consensus optimization","authors":"Kwok Zhang, Ruiliang Zhang, James Kwok","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2016] Ruiliang Zhang, Shuai Zheng, and James T Kwok. Asynchronous distributed semi-stochastic gradient optimization. In AAAI, 2016.","title":"Asynchronous distributed semi-stochastic gradient optimization","authors":"Zhang","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhao and Li, 2016] Shen-Yi Zhao and Wu-Jun Li. Fast asynchronous parallel stochastic gradient descent: A lock- free approach with convergence guarantee. In AAAI, 2016.","title":"Fast asynchronous parallel stochastic gradient descent: A lockfree approach with convergence guarantee","authors":"Li ; ] Shen-Yi Zhao, Wu-Jun Zhao, Li","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhou et al., 2018] Kaiwen Zhou, Fanhua Shang, and James Cheng. A simple stochastic variance reduced algorithm with fast convergence rates. In ICML, 2018.","title":"A simple stochastic variance reduced algorithm with fast convergence rates","authors":"Kaiwen Zhou, Fanhua Shang, James Cheng","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}