{"type":"ScholarlyArticle","title":"Distributed Accelerated Proximal Coordinate Gradient Methods","authors":[{"name":"Yong Ren","slug":"yong-ren","given":"Yong","family":"Ren","affiliation":"Center for Bio-Inspired Computing Research State Key Lab for Intell. Tech. & Systems Dept. of Comp. Sci. & Tech. , TNList Lab , Tsinghua University"},{"name":"Jun Zhu","slug":"jun-zhu","given":"Jun","family":"Zhu","affiliation":"Center for Bio-Inspired Computing Research State Key Lab for Intell. Tech. & Systems Dept. of Comp. Sci. & Tech. , TNList Lab , Tsinghua University"}],"abstract":"We develop a general accelerated proximal coordinate descent algorithm in distributed settings (Dis- APCG) for the optimization problem that minimizes the sum of two convex functions: the first part f is smooth with a gradient oracle, and the other one Ψ is separable with respect to blocks of coordinate and has a simple known structure (e.g., L1 norm). Our algorithm gets new accelerated convergence rate in the case that f is strongly con- vex by making use of modern parallel structures, and includes previous non-strongly case as a special case. We further present efficient implementations to avoid full-dimensional operations in each step, significantly reducing the computation cost. Experiments on the regularized empirical risk minimization problem demonstrate the effectiveness of our algorithm and match our theoretical findings.","doi":"10.24963/ijcai.2017/370","url":"https://ijcai.org/proceedings/2017/370","pdf":"https://ijcai.org/proceedings/2017/0370.pdf","markdown":"https://ijcai.org/proceedings/2017/370.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/370","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":370,"pages":{"first":2655,"last":2661},"keywords":["Machine Learning: Machine Learning","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Bradley et al., 2011] Joseph Bradley, Aapo Kyrola, Danny Bickson, and Carlos Guestrin. Parallel coordinate descent for l1-regularized loss minimization. ICML, 2011.","title":"Parallel coordinate descent for l1-regularized loss minimization","authors":"Joseph Bradley, Aapo Kyrola, Danny Bickson, Carlos Guestrin","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dunner et al., 2016] Celestine Dunner, Simone Forte, Mar- tin Takac, and Martin Jaggi. Primal-dual rates and certifi- cates. ICML, 2016.","title":"Primal-dual rates and certificates","authors":"Celestine Dunner, Simone Forte, Martin Takac, Martin Jaggi","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fan and Lin, 2011] Rong-En Fan and Chih-Jen Lin. Lib- svm data: Classification, regression and multi-label. URL: http://www.csie.ntu.edu.tw/ cjlin/libsvmtools/datasets, 2011.","title":"Libsvm data: Classification, regression and multi-label","authors":"Rong-En Fan, Chih-Jen Lin","year":null,"venue":null,"doi":null,"url":"http://www.csie.ntu.edu.tw/cjlin/libsvmtools/datasets,2011","internal_url":null},{"position":4,"raw":"[Fercoq and Richtarik, 2015] Olivier Fercoq and Peter Richtarik. Accelerated, parallel and proximal coordinate descent. SIAM Journal on Optimization, 2015.","title":"Olivier Fercoq and Peter Richtarik. Accelerated, parallel and proximal coordinate descent","authors":"Richtarik Fercoq","year":"2015","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Lin et al., 2014] Qihang Lin, Zhaosong Lu, and Lin Xiao. An accelerated proximal coordinate gradient method and its application to regularized empirical risk minimization. NIPS, 2014.","title":"An accelerated proximal coordinate gradient method and its application to regularized empirical risk minimization","authors":"Lin","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ma et al., 2015] Chenxin Ma, Virginia Smith, Martin Jaggi, Michael Jordan, Peter Richtarik, and Martin Takac. Adding vs. averaging in distributed primal-dual optimiza- tion. ICML, 2015.","title":"Adding vs. averaging in distributed primal-dual optimization","authors":"Ma","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Necoara and Clipici, 2013] Ion Necoara and Dragos Clipici. Efficient parallel coordinate descent algorithm for convex optimization problems with separable constraints: application to distributed mpc. Journal of Process Control, 2013.","title":"Ion Necoara and Dragos Clipici. Efficient parallel coordinate descent algorithm for convex optimization problems with separable constraints: application to distributed mpc","authors":"Clipici Necoara","year":"2013","venue":"Journal of Process Control","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Nesterov, 1983] Yurii Nesterov. A method of solving a con- vex programming problem with convergence rate o(1/k 2). Soviet Mathematics Doklady, 1983.","title":"A method of solving a convex programming problem with convergence rate o(1/k 2)","authors":"Yurii Nesterov","year":"1983","venue":"Soviet Mathematics Doklady","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Nesterov, 2012] Yurii Nesterov. Efficiency of coordinate de- scent methods on huge-scale optimization problems. SIAM Journal on Optimization, 2012.","title":"Efficiency of coordinate descent methods on huge-scale optimization problems","authors":"Yurii Nesterov","year":"2012","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Richtarik and Takac, 2013a] Peter Richtarik and Martin Takac. Distributed coordinate descent method for learn- ing with big data. arXiv:1310.2059, 2013.","title":"Distributed coordinate descent method for learning with big data","authors":"Takac Richtarik, Peter Richtarik, Martin Takac","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Richtarik and Takac, 2013b] Peter Richtarik and Martin Takac. Parallel coordinate descent methods for big data optimization. arXiv:1212.0873v2, 2013.","title":"Parallel coordinate descent methods for big data optimization","authors":"Takac Richtarik, Peter Richtarik, Martin Takac","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Shalev-Shwartz and Zhang, 2014] Shai Shalev-Shwartz and Tong Zhang. Accelerated proximal stochastic dual coorid- inate ascent for regularized loss minimization. ICML, 2014.","title":"Accelerated proximal stochastic dual cooridinate ascent for regularized loss minimization","authors":"Shalev-Shwartz Zhang, Shai Shalev-Shwartz, Tong Zhang","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Takac and Richtarik, 2015] Martin Takac and Pe- ter Richtarik. Distributed mini-batch sdca. arXiv:1507.08322v1, 2015.","title":"Distributed mini-batch sdca","authors":"Martin Takac, Peter Richtarik","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Wright, 2015] Stephen Wright. Coordinate descent algo- rithms. Mathematical Programming 151.1, 3-34., 2015.","title":"Coordinate descent algorithms","authors":"Stephen Wright","year":"2015","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Xiao and Lu, 2013] Lin Xiao and Zhaosong Lu. On the complexity analysis of randomized block-coordinate de- scent methods. arXiv:1304.4723, 2013.","title":"On the complexity analysis of randomized block-coordinate descent methods","authors":"Lin Xiao, Zhaosong Lu","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Yang, 2013] Tianbao Yang. Trading computation for com- munication: Distributed stochastic dual coordinate ascent. NIPS, 2013.","title":"Trading computation for communication: Distributed stochastic dual coordinate ascent","authors":"Tianbao Yang","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}