{"type":"ScholarlyArticle","title":"Fast Stochastic Variance Reduced ADMM for Stochastic Composition Optimization","authors":[{"name":"Yue Yu","slug":"yue-yu","given":"Yue","family":"Yu","affiliation":"Institute for Interdisciplinary Information Sciences , Tsinghua University"},{"name":"Longbo Huang","slug":"longbo-huang","given":"Longbo","family":"Huang","affiliation":"Institute for Interdisciplinary Information Sciences , Tsinghua University"}],"abstract":"We consider the  stochastic composition optimization problem  proposed in \\cite{wang2017stochastic}, which  has applications ranging from estimation to statistical and machine learning. We propose the first ADMM based algorithm named com SVR ADMM, and show that com SVR ADMM converges linearly for  strongly convex and Lipschitz smooth objectives, and has a convergence rate of $O(\\logS/S)$, which improves upon the $O(S^{-4/9})$ rate in \\cite{wang2016accelerating} when the objective is convex and Lipschitz smooth. Moreover, com SVR ADMM possesses a rate of $O(1/\\sqrt{S})$ when the objective is convex but without Lipschitz smoothness.  We also conduct experiments  and show that it outperforms existing algorithms.","doi":"10.24963/ijcai.2017/470","url":"https://ijcai.org/proceedings/2017/470","pdf":"https://ijcai.org/proceedings/2017/0470.pdf","markdown":"https://ijcai.org/proceedings/2017/470.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/470","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":470,"pages":{"first":3364,"last":3370},"keywords":["Machine Learning: Machine Learning","Machine Learning: Learning Theory"],"videos":[],"references":[{"position":1,"raw":"[Alexander and Baptista, 2004] Gordon J Alexander and Alexandre M Baptista. A comparison of var and cvar constraints on portfolio selection with the mean-variance model. Management science, 50(9):1261-1273, 2004.","title":"A comparison of var and cvar constraints on portfolio selection with the mean-variance model","authors":"J Gordon, Alexandre M Alexander, Baptista","year":"2004","venue":"Management science","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Allen-Zhu, 2016] Zeyuan Allen-Zhu. Katyusha: Acceler- ated variance reduction for faster sgd. ArXiv e-prints, abs/1603.05953, 2016.","title":"Katyusha: Accelerated variance reduction for faster sgd","authors":"Allen-Zhu ; Zeyuan Allen-Zhu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Boyd et al., 2011] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. Distributed opti- mization and statistical learning via the alternating direc- tion method of multipliers. Foundations and Trends R in Machine Learning, 3(1):1-122, 2011.","title":"Distributed optimization and statistical learning via the alternating direction method of multipliers","authors":"Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein","year":"2011","venue":"Foundations and Trends R in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dai et al., 2016] Bo Dai, Niao He, Yunpeng Pan, Byron Boots, and Le Song. Learning from conditional dis- tributions via dual kernel embeddings. arXiv preprint arXiv:1607.04579, 2016.","title":"Learning from conditional distributions via dual kernel embeddings","authors":"Dai","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gabay and Mercier, 1976] Daniel Gabay and Bertrand Mercier. A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Computers & Mathematics with Applications, 2(1):17-40, 1976.","title":"A dual algorithm for the solution of nonlinear variational problems via finite element approximation","authors":"Daniel Gabay, Bertrand Mercier","year":"1976","venue":"Computers & Mathematics with Applications","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Glowinski and Marroco, 1975] Roland Glowinski and A Marroco. Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de dirichlet non linéaires. Revue franc ¸aise d'automatique, informatique, recherche opérationnelle. Analyse numérique, 9(2):41-76, 1975.","title":"Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de dirichlet non linéaires","authors":"Marroco Glowinski","year":"1975","venue":"Revue franc ¸aise d'automatique, informatique, recherche opérationnelle. Analyse numérique","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Golub and Van Loan, 2012] Gene H Golub and Charles F Van Loan. Matrix computations, volume 3. JHU Press, 2012.","title":"Matrix computations","authors":"Van Golub, ; Loan, H Gene, Charles F Golub, Van Loan","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hien et al., 2016] Le Thi Khanh Hien, Canyi Lu, Huan Xu, and Jiashi Feng. Accelerated stochastic mirror descent al- gorithms for composite non-strongly convex optimization. arXiv preprint arXiv:1605.06892, 2016.","title":"Accelerated stochastic mirror descent algorithms for composite non-strongly convex optimization","authors":"Hien","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Johnson and Zhang, 2013] Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in Neural Information Processing Systems, pages 315-323, 2013.","title":"Accelerating stochastic gradient descent using predictive variance reduction","authors":"Rie Johnson, Tong Zhang","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lian et al., 2016] Xiangru Lian, Mengdi Wang, and Ji Liu. Finite-sum composition optimization via variance reduced gradient descent. arXiv preprint arXiv:1610.04674, 2016.","title":"Finite-sum composition optimization via variance reduced gradient descent","authors":"Lian","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ouyang et al., 2013] Hua Ouyang, Niao He, Long Tran, and Alexander G Gray. Stochastic alternating direction method of multipliers. ICML (1), 28:80-88, 2013.","title":"Stochastic alternating direction method of multipliers","authors":"Ouyang","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Qian and Zhou, 2016] Chao Zhang Hui Qian and Ze- bang Shen Tengfei Zhou. Accelerated stochastic admm for empirical risk minimization. arXiv preprint arXiv:1611.04074, 2016.","title":"Accelerated stochastic admm for empirical risk minimization","authors":"Chao Zhang, Hui Qian, Zebang Shen, Tengfei Zhou","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Shapiro et al., 2014] Alexander Shapiro, Darinka Dentcheva, et al. Lectures on stochastic programming: modeling and theory, volume 16. Siam, 2014.","title":"Lectures on stochastic programming: modeling and theory","authors":"Shapiro","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, vol- ume 1. MIT press Cambridge, 1998.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Suzuki and others, 2013] Taiji Suzuki et al. Dual averag- ing and proximal gradient descent for online alternating direction multiplier method. In ICML (1), pages 392-400, 2013.","title":"Dual averaging and proximal gradient descent for online alternating direction multiplier method","authors":"Others ; Taiji Suzuki, Suzuki","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wang and Banerjee, 2013] Huahua Wang and Arindam Banerjee. Online alternating direction method (longer ver- sion). arXiv preprint arXiv:1306.3721, 2013.","title":"Online alternating direction method (longer version","authors":"Wang, Banerjee, Huahua Wang, Arindam Banerjee","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2016a] Mengdi Wang, Ethan X Fang, and Han Liu. Stochastic compositional gradient descent: Algo- rithms for minimizing compositions of expected-value functions. Mathematical Programming, 161(1-2):419- 449, 2016.","title":"Stochastic compositional gradient descent: Algorithms for minimizing compositions of expected-value functions","authors":"Wang","year":"2016","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2016b] Mengdi Wang, Ji Liu, and Ethan X. Fang. Accelerating stochastic composition optimization. In Advances In Neural Information Processing Systems, pages 1714-1722, 2016.","title":"Accelerating stochastic composition optimization","authors":"Wang","year":"2016","venue":"Advances In Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yu and Huang, 2017] Yue Yu and Longbo Huang. Fast stochastic variance reduced admm for stochastic composi- tion optimization. arXiv preprint arXiv:1705.04138, 2017.","title":"Fast stochastic variance reduced admm for stochastic composition optimization","authors":"Yue Yu, Longbo Huang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhang et al., 2011] Xiaoqun Zhang, Martin Burger, and Stanley Osher. A unified primal-dual algorithm framework based on bregman iteration. Journal of Scientific Comput- ing, 46(1):20-46, 2011.","title":"A unified primal-dual algorithm framework based on bregman iteration","authors":"Zhang","year":"2011","venue":"Journal of Scientific Computing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zheng and Kwok, 2016a] Zheng and Kwok. Fast-and-light stochastic admm. arXiv preprint arXiv:1604.07070, 2016.","title":"Fast-and-light stochastic admm","authors":"Kwok ; Zheng, Zheng, Kwok","year":"2016","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2016/343"},{"position":22,"raw":"[Zheng and Kwok, 2016b] Shuai Zheng and James T Kwok. Stochastic variance-reduced admm. arXiv preprint arXiv:1604.07070, 2016.","title":"Stochastic variance-reduced admm","authors":"Zheng, Shuai Kwok ; B], James T Zheng, Kwok","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}