{"type":"ScholarlyArticle","title":"Distributed Primal-Dual Optimization for Non-uniformly Distributed Data","authors":[{"name":"Minhao Cheng","slug":"minhao-cheng","given":"Minhao","family":"Cheng","affiliation":"Department of Computer Science , University of California , Davis"},{"name":"Cho-Jui Hsieh","slug":"cho-jui-hsieh","given":"Cho-Jui","family":"Hsieh","affiliation":"Department of Computer Science , University of California , Davis; Department of Statistics , University of California , Davis"}],"abstract":"Distributed primal-dual optimization has received many focuses in the past few years. In this framework, training samples are stored in multiple machines. At each round, all the machines conduct a sequence of updates based on their local data, and then the local updates are synchronized and merged to obtain the update to the global model. All the previous approaches merge the local updates by averaging all of them with a uniform weight. However, in many real world applications data are not uniformly distributed on each machine, so the uniform weight is inadequate to capture the heterogeneity of local updates. To resolve this issue, we propose a better way to merge local updates in the primal-dual optimization framework. Instead of using a single weight for all the local updates, we develop a computational efficient algorithm to automatically choose the optimal weights for each machine. Furthermore, we propose an efficient way to estimate the duality gap of the merged update by exploiting the structure of the objective function, and this leads to an efficient line search algorithm based on the reduction of duality gap. Combining these two ideas, our algorithm is much faster and more scalable  than existing methods on real world problems.","doi":"10.24963/ijcai.2018/280","url":"https://ijcai.org/proceedings/2018/280","pdf":"https://ijcai.org/proceedings/2018/0280.pdf","markdown":"https://ijcai.org/proceedings/2018/280.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/280","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":280,"pages":{"first":2028,"last":2034},"keywords":["Machine Learning: Machine Learning","Machine Learning Applications: Big data ; Scalability"],"videos":[],"references":[{"position":1,"raw":"[Agarwal and Duchi, 2011] Alekh Agarwal and John C Duchi. Distributed delayed stochastic optimization. In Ad- vances in Neural Information Processing Systems, pages 873-881, 2011.","title":"Distributed delayed stochastic optimization","authors":"Alekh Agarwal, John C Duchi","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2016] Jianmin Chen, Xinghao Pan, Rajat Monga, Samy Bengio, and Rafal Jozefowicz. Re- visiting distributed synchronous sgd. arXiv preprint arXiv:1604.00981, 2016.","title":"Revisiting distributed synchronous sgd","authors":"Jianmin Chen, Xinghao Pan, Rajat Monga, Samy Bengio, Rafal Jozefowicz","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dekel et al., 2012] Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, and Lin Xiao. Optimal distributed online predic- tion using mini-batches. Journal of Machine Learning Re- search, 13(Jan):165-202, 2012.","title":"Optimal distributed online prediction using mini-batches","authors":"Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, Lin Xiao","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hsieh et al., 2008] Cho-Jui Hsieh, Kai-Wei Chang, Chih- Jen Lin, S. Sathiya Keerthi, and Sellamanickam Sun- dararajan. A dual coordinate descent method for large- scale linear SVM. In Proceedings of the Twenty Fifth Inter- national Conference on Machine Learning (ICML), 2008.","title":"and Sellamanickam Sundararajan. A dual coordinate descent method for largescale linear SVM","authors":"Cho-Jui Hsieh, Kai-Wei Chang, Chih-Jen Lin, S Sathiya Keerthi","year":"2008","venue":"Proceedings of the Twenty Fifth International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jaggi et al., 2014] Martin Jaggi, Virginia Smith, Mar- tin Takáč, Jonathan Terhorst, Thomas Hofmann, and Michael I Jordan. Communication-efficient distributed dual coordinate ascent. In NIPS. 2014.","title":"Communication-efficient distributed dual coordinate ascent","authors":"Martin Jaggi, Virginia Smith, Martin Takáč, Jonathan Terhorst, Thomas Hofmann, Michael I Jordan","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Konečnỳ et al., 2015] Jakub Konečnỳ, Brendan McMahan, and Daniel Ramage. Federated optimization: Distributed optimization beyond the datacenter. arXiv preprint arXiv:1511.03575, 2015.","title":"Federated optimization: Distributed optimization beyond the datacenter","authors":"Jakub Konečnỳ, Brendan Mcmahan, Daniel Ramage","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Konečnỳ et al., 2016] Jakub Konečnỳ, H Brendan McMa- han, Daniel Ramage, and Peter Richtárik. Federated op- timization: Distributed machine learning for on-device in- telligence. arXiv preprint arXiv:1610.02527, 2016.","title":"Federated optimization: Distributed machine learning for on-device intelligence","authors":"Jakub Konečnỳ, Brendan Mcmahan, Daniel Ramage, Peter Richtárik","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lee and Roth, 2015] C.-P. Lee and D. Roth. Distributed box-constrained quadratic optimization for dual linear SVM. In ICML, 2015.","title":"Distributed box-constrained quadratic optimization for dual linear SVM","authors":"C.-P Lee, D Roth","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lian et al., 2017] Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentral- ized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient de- scent. In Advances in Neural Information Processing Sys- tems, pages 5336-5346, 2017.","title":"Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent","authors":"Lian","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ma et al., 2015] C. Ma, V. Smith, M. Jaggi, M. I. Jordan, P. Richtárik, and M. Takáč. Adding vs. averaging in dis- tributed primal-dual optimization. In ICML, 2015.","title":"Adding vs. averaging in distributed primal-dual optimization","authors":"C Ma, V Smith, M Jaggi, M I Jordan, P Richtárik, M Takáč","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[McMahan et al., 2016] H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al. Communication- efficient learning of deep networks from decentralized data. arXiv preprint arXiv:1602.05629, 2016.","title":"Communicationefficient learning of deep networks from decentralized data","authors":"Eider H Brendan Mcmahan, Daniel Moore, Seth Ramage, Hampson","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Shalev-Shwartz and Zhang, 2013] Shai Shalev-Shwartz and Tong Zhang. Stochastic dual coordinate ascent methods for regularized loss minimization. Journal of Machine Learning Research, 14:567-599, 2013.","title":"Stochastic dual coordinate ascent methods for regularized loss minimization","authors":"Shalev-Shwartz Zhang, ; ] Shai Shalev-Shwartz, Tong Zhang","year":"2013","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Tropp and others, 2015] Joel A Tropp et al. An introduc- tion to matrix concentration inequalities. Foundations and Trends R in Machine Learning, 8(1-2):1-230, 2015.","title":"An introduction to matrix concentration inequalities","authors":"Joel A Tropp","year":"2015","venue":"Foundations and Trends R in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Yang, 2013] T. Yang. Trading computation for communi- cation: Distributed stochastic dual coordinate ascent. In NIPS, 2013.","title":"Trading computation for communication: Distributed stochastic dual coordinate ascent","authors":"T Yang","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Zheng et al., 2017] Shun Zheng, Jialei Wang, Fen Xia, Wei Xu, and Tong Zhang. A general distributed dual coordinate optimization framework for regularized loss minimiza- tion. Journal of Machine Learning Research, 18(115):1- 52, 2017.","title":"A general distributed dual coordinate optimization framework for regularized loss minimization","authors":"Zheng","year":"2017","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}