{"type":"ScholarlyArticle","title":"A Generic Approach for Accelerating Stochastic Zeroth-Order Convex Optimization","authors":[{"name":"Xiaotian Yu","slug":"xiaotian-yu","given":"Xiaotian","family":"Yu","affiliation":"Department of Computer Science and Engineering The Chinese University of Hong Kong , Shatin , N .T. , Hong Kong; Shenzhen Key Laboratory of Rich Media Big Data Analytics and Applications , Shenzhen Research Institute , The Chinese University of Hong Kong , Shenzhen , China"},{"name":"Irwin King","slug":"irwin-king","given":"Irwin","family":"King","affiliation":"Department of Computer Science and Engineering The Chinese University of Hong Kong , Shatin , N .T. , Hong Kong; Shenzhen Key Laboratory of Rich Media Big Data Analytics and Applications , Shenzhen Research Institute , The Chinese University of Hong Kong , Shenzhen , China"},{"name":"Michael R. Lyu","slug":"michael-r-lyu","given":"Michael R.","family":"Lyu","affiliation":"Department of Computer Science and Engineering The Chinese University of Hong Kong , Shatin , N .T. , Hong Kong; Shenzhen Key Laboratory of Rich Media Big Data Analytics and Applications , Shenzhen Research Institute , The Chinese University of Hong Kong , Shenzhen , China"},{"name":"Tianbao Yang","slug":"tianbao-yang","given":"Tianbao","family":"Yang","affiliation":"Department of Computer Science , The University of Iowa , Iowa City , IA 52242 , USA"}],"abstract":"In this paper, we propose a generic approach for accelerating the convergence of existing algorithms to solve the problem of stochastic zeroth-order convex optimization (SZCO). Standard techniques for accelerating the convergence of stochastic zeroth-order algorithms are by exploring multiple functional evaluations (e.g., two-point evaluations), or by exploiting global conditions of the problem (e.g., smoothness and strong convexity). Nevertheless, these classic acceleration techniques are necessarily restricting the applicability of newly developed algorithms. The key of our proposed generic approach is to explore a local growth condition  (or called local error bound condition) of the objective function in SZCO. The benefits of the proposed acceleration technique are: (i) it is applicable to both settings with one-point evaluation and two-point evaluations; (ii) it does not necessarily require strong convexity or smoothness condition of the objective function; (iii) it yields an improvement on convergence for a broad family of problems. Empirical studies in various settings demonstrate the effectiveness of the proposed acceleration approach.","doi":"10.24963/ijcai.2018/422","url":"https://ijcai.org/proceedings/2018/422","pdf":"https://ijcai.org/proceedings/2018/0422.pdf","markdown":"https://ijcai.org/proceedings/2018/422.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/422","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":422,"pages":{"first":3040,"last":3046},"keywords":["Machine Learning: Machine Learning","Uncertainty in AI: Uncertainty in AI"],"videos":[],"references":[{"position":1,"raw":"[Agarwal et al., 2010] Alekh Agarwal, Ofer Dekel, and Lin Xiao. Optimal algorithms for online convex optimization with multi-point bandit feedback. In COLT, pages 28-40, 2010.","title":"Optimal algorithms for online convex optimization with multi-point bandit feedback","authors":"Alekh Agarwal, Ofer Dekel, Lin Xiao","year":"2010","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bolte et al., 2015] Jérôme Bolte, Trong Phong Nguyen, Juan Peypouquet, and Bruce Suter. From error bounds to the complexity of first-order descent methods for convex functions. CoRR, abs/1510.08234, 2015.","title":"From error bounds to the complexity of first-order descent methods for convex functions","authors":"Jérôme Bolte, Trong Phong Nguyen, Juan Peypouquet, Bruce Suter","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2011] Po-Lung Chen, Chen-Tse Tsai, Yao-Nan Chen, Ku-Chun Chou, Chun-Liang Li, Cheng-Hao Tsai, Kuan-Wei Wu, Yu-Cheng Chou, Chung-Yi Li, Wei-Shih Lin, et al. A linear ensemble of individual and blended models for music rating prediction. In KDD-Cup, pages 21-60, 2011.","title":"A linear ensemble of individual and blended models for music rating prediction","authors":"Chen","year":"2011","venue":"KDD-Cup","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dani et al., 2008] Varsha Dani, Sham M Kakade, and Thomas P Hayes. The price of bandit information for on- line optimization. In NIPS, pages 345-352, 2008.","title":"The price of bandit information for online optimization","authors":"Varsha Dani, M Sham, Thomas P Kakade, Hayes","year":"2008","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Duchi et al., 2015] John C Duchi, Michael I Jordan, Mar- tin J Wainwright, and Andre Wibisono. Optimal rates for zero-order convex optimization: The power of two func- tion evaluations. IEEE Transactions on Information Theo- ry, 61(5):2788-2806, 2015.","title":"Optimal rates for zero-order convex optimization: The power of two function evaluations","authors":"John C Duchi, Martin J Michael I Jordan, Andre Wainwright, Wibisono","year":"2015","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Flaxman et al., 2005] Abraham D Flaxman, Adam Tauman Kalai, and H Brendan McMahan. Online convex optimiza- tion in the bandit setting: gradient descent without a gra- dient. In SODA, pages 385-394, 2005.","title":"Online convex optimization in the bandit setting: gradient descent without a gradient","authors":"Adam Abraham D Flaxman, Brendan Tauman Kalai, Mcmahan","year":"2005","venue":"SODA","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hazan and Kale, 2014] Elad Hazan and Satyen Kale. Be- yond the regret minimization barrier: optimal algorithm- s for stochastic strongly-convex optimization. Journal of Machine Learning Research, 15(1):2489-2512, 2014.","title":"Beyond the regret minimization barrier: optimal algorithms for stochastic strongly-convex optimization","authors":"Kale ; ] Elad Hazan, Satyen Hazan, Kale","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lian et al., 2016] Xiangru Lian, Huan Zhang, Cho-Jui H- sieh, Yijun Huang, and Ji Liu. A comprehensive linear speedup analysis for asynchronous stochastic parallel op- timization from zeroth-order to first-order. In NIPS, pages 3054-3062, 2016.","title":"A comprehensive linear speedup analysis for asynchronous stochastic parallel optimization from zeroth-order to first-order","authors":"Lian","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Nakamura et al., 2017] Nathan Nakamura, Jason Seepaul, Joseph B Kadane, and B Reeja-Jayan. Design for low- temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and In- dustry, 2017. [Nemirovski et al., 2009] Arkadi Nemirovski, Anatoli Judit- sky, Guanghui Lan, and Alexander Shapiro. Robust s- tochastic approximation approach to stochastic program- ming. SIAM Journal on Optimization, 19(4):1574-1609, 2009.","title":"Design for lowtemperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry","authors":"Nathan Nakamura, Jason Seepaul, Joseph B Kadane, B Reeja-Jayan, ; Nemirovski","year":"2009","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Nesterov and Spokoiny, 2017] Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527-566, 2017.","title":"Random gradient-free minimization of convex functions","authors":"Yurii Nesterov, Vladimir Spokoiny","year":"2017","venue":"Foundations of Computational Mathematics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Shamir, 2013] Ohad Shamir. On the complexity of ban- dit and derivative-free stochastic convex optimization. In COLT, pages 3-24, 2013.","title":"On the complexity of bandit and derivative-free stochastic convex optimization","authors":"Ohad Shamir","year":"2013","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Shamir, 2017] Ohad Shamir. An optimal algorithm for bandit and zero-order convex optimization with two- point feedback. Journal of Machine Learning Research, 18(52):1-11, 2017.","title":"An optimal algorithm for bandit and zero-order convex optimization with twopoint feedback","authors":"Ohad Shamir","year":"2017","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Sokolov et al., 2016] Artem Sokolov, Julia Kreutzer, Stefan Riezler, and Christopher Lo. Stochastic structured predic- tion under bandit feedback. In NIPS, pages 1489-1497, 2016.","title":"Stochastic structured prediction under bandit feedback","authors":"Artem Sokolov, Julia Kreutzer, Stefan Riezler, Christopher Lo","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Wang et al., 2017] Yining Wang, Simon Du, Sivaraman Balakrishnan, and Aarti Singh. Stochastic zeroth-order optimization in high dimensions. arXiv preprint arX- iv:1710.10551, 2017.","title":"Stochastic zeroth-order optimization in high dimensions","authors":"Yining Wang, Simon Du, Sivaraman Balakrishnan, Aarti Singh","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Wibisono et al., 2012] Andre Wibisono, Martin J Wain- wright, Michael I Jordan, and John C Duchi. Finite sam- ple convergence rates of zero-order stochastic optimiza- tion methods. In NIPS, pages 1439-1447, 2012.","title":"Finite sample convergence rates of zero-order stochastic optimization methods","authors":"Andre Wibisono, Martin J Wainwright, Michael I Jordan, John C Duchi","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Xu et al., 2016] Yi Xu, Qihang Lin, and Tianbao Yang. Ac- celerated stochastic subgradient methods under local error bound condition. arXiv preprint arXiv:1607.01027, 2016.","title":"Accelerated stochastic subgradient methods under local error bound condition","authors":"Yi Xu, Qihang Lin, Tianbao Yang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Xu et al., 2017] Yi Xu, Qihang Lin, and Tianbao Yang. S- tochastic convex optimization: Faster local growth implies faster global convergence. In ICML, pages 3821-3830, 2017.","title":"Stochastic convex optimization: Faster local growth implies faster global convergence","authors":"Yi Xu, Qihang Lin, Tianbao Yang","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Yang and Lin, 2015] Tianbao Yang and Qihang Lin. Rs- g: Beating subgradient method without smoothness and strong convexity. arXiv preprint arXiv:1512.03107, 2015.","title":"Rsg: Beating subgradient method without smoothness and strong convexity","authors":"Yang, Lin ;, Tianbao Yang, Qihang Lin","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zadorozhnyi et al., 2016] Oleksandr Zadorozhnyi, Gun- thard Benecke, Stephan Mandt, Tobias Scheffer, and Mar- ius Kloft. Huber-norm regularization for linear prediction models. In ECML, pages 714-730, 2016.","title":"Huber-norm regularization for linear prediction models","authors":"Oleksandr Zadorozhnyi, Gunthard Benecke, Stephan Mandt, Tobias Scheffer, Marius Kloft","year":"2016","venue":"ECML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zinkevich, 2003] Martin Zinkevich. Online convex pro- gramming and generalized infinitesimal gradient ascent. In ICML, pages 928-936, 2003.","title":"Online convex programming and generalized infinitesimal gradient ascent","authors":"Martin Zinkevich","year":"2003","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}