{"type":"ScholarlyArticle","title":"Zeroth-Order Stochastic Alternating Direction Method of Multipliers for Nonconvex Nonsmooth Optimization","authors":[{"name":"Feihu Huang","slug":"feihu-huang","given":"Feihu","family":"Huang","affiliation":"Department of Electrical & Computer Engineering , University of Pittsburgh , USA"},{"name":"Shangqian Gao","slug":"shangqian-gao","given":"Shangqian","family":"Gao","affiliation":"Department of Electrical & Computer Engineering , University of Pittsburgh , USA"},{"name":"Songcan Chen","slug":"songcan-chen","given":"Songcan","family":"Chen","affiliation":"College of Computer Science & Technology , Nanjing University of Aeronautics and Astronautics; MIIT Key Laboratory of Pattern Analysis & Machine Intelligence , China"},{"name":"Heng Huang","slug":"heng-huang","given":"Heng","family":"Huang","affiliation":"Department of Electrical & Computer Engineering , University of Pittsburgh , USA; JD Finance America Corporation"}],"abstract":"Alternating direction method of multipliers (ADMM) is a popular optimization tool for the composite and constrained problems in machine learning. However, in many machine learning problems such as black-box learning and bandit feedback,\r\nADMM could fail because the explicit gradients of these problems are difficult or even infeasible to obtain. Zeroth-order (gradient-free) methods can effectively solve these problems due to that the objective function values\r\nare only required in the optimization. Recently, though there exist a few zeroth-order ADMM methods, they build on the convexity of objective function. Clearly, these existing zeroth-order methods are limited in many applications.\r\nIn the paper, thus, we propose a class of fast zeroth-order stochastic ADMM methods (\\emph{i.e.}, ZO-SVRG-ADMM and ZO-SAGA-ADMM) for solving nonconvex problems with multiple nonsmooth penalties, based on the\r\ncoordinate smoothing gradient estimator. Moreover, we prove that both the ZO-SVRG-ADMM and ZO-SAGA-ADMM have convergence rate of $O(1/T)$, where $T$ denotes the number of iterations. In particular, our methods not only reach the best convergence rate of $O(1/T)$ for the nonconvex optimization, but also are able to effectively solve many complex machine learning problems with multiple regularized penalties and constraints.\r\nFinally, we conduct the experiments of black-box binary classification and structured adversarial attack on black-box deep neural network to validate the efficiency of our algorithms.","doi":"10.24963/ijcai.2019/354","url":"https://ijcai.org/proceedings/2019/354","pdf":"https://ijcai.org/proceedings/2019/0354.pdf","markdown":"https://ijcai.org/proceedings/2019/354.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/354","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":354,"pages":{"first":2549,"last":2555},"keywords":["Machine Learning: Adversarial Machine Learning","Machine Learning Applications: Big data ; Scalability"],"videos":[],"references":[{"position":1,"raw":"[Agarwal et al., 2010] Alekh Agarwal, Ofer Dekel, and Lin Xiao. Optimal algorithms for online convex optimization with multi-point bandit feedback. In COLT, pages 28-40.","title":"Optimal algorithms for online convex optimization with multi-point bandit feedback","authors":"Alekh Agarwal, Ofer Dekel, Lin Xiao","year":null,"venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Beck and Teboulle, 2009] Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for lin- ear inverse problems. SIAM journal on imaging sciences, 2(1):183-202, 2009.","title":"A fast iterative shrinkage-thresholding algorithm for linear inverse problems","authors":"Amir Beck, Marc Teboulle","year":"2009","venue":"SIAM journal on imaging sciences","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Boyd et al., 2011] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. Distributed opti- mization and statistical learning via the alternating direc- tion method of multipliers. Foundations and Trends R ⃝ in Machine Learning, 3(1):1-122, 2011.","title":"Distributed optimization and statistical learning via the alternating direction method of multipliers","authors":"Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein","year":"2011","venue":"Foundations and Trends R ⃝ in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2017] Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order opti- mization based black-box attacks to deep neural networks without training substitute models. In Workshop on Artifi- cial Intelligence and Security, pages 15-26. ACM, 2017.","title":"Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models","authors":"Chen","year":"2017","venue":"Workshop on Artificial Intelligence and Security","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Defazio et al., 2014] Aaron Defazio, Francis Bach, and Si- mon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. In NIPS, pages 1646-1654, 2014.","title":"Saga: A fast incremental gradient method with support for non-strongly convex composite objectives","authors":"Aaron Defazio, Francis Bach, Simon Lacoste-Julien","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Duchi et al., 2015] John C Duchi, Michael I Jordan, Mar- tin J Wainwright, and Andre Wibisono. Optimal rates for zero-order convex optimization: The power of two func- tion evaluations. IEEE TIT, 61(5):2788-2806, 2015.","title":"Optimal rates for zero-order convex optimization: The power of two function evaluations","authors":"John C Duchi, Martin J Michael I Jordan, Andre Wainwright, Wibisono","year":"2015","venue":"IEEE TIT","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gabay and Mercier, 1976] Daniel Gabay and Bertrand Mercier. A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Computers & Mathematics with Applications, 2(1):17-40, 1976.","title":"A dual algorithm for the solution of nonlinear variational problems via finite element approximation","authors":"Daniel Gabay, Bertrand Mercier","year":"1976","venue":"Computers & Mathematics with Applications","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gao et al., 2018] Xiang Gao, Bo Jiang, and Shuzhong Zhang. On the information-adaptive variants of the admm: an iteration complexity perspective. Journal of Scientific Computing, 76(1):327-363, 2018.","title":"On the information-adaptive variants of the admm: an iteration complexity perspective","authors":"Xiang Gao, Bo Jiang, Shuzhong Zhang","year":"2018","venue":"Journal of Scientific Computing","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ghadimi and Lan, 2013] Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for non- convex stochastic programming. SIAM Journal on Opti- mization, 23(4):2341-2368, 2013.","title":"Stochastic first-and zeroth-order methods for nonconvex stochastic programming","authors":"Saeed Ghadimi, Guanghui Lan","year":"2013","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ghadimi et al., 2016] Saeed Ghadimi, Guanghui Lan, and Hongchao Zhang. Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming, 155(1-2):267-305, 2016.","title":"Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization","authors":"Saeed Ghadimi, Guanghui Lan, Hongchao Zhang","year":"2016","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gu et al., 2018] Bin Gu, Zhouyuan Huo, Cheng Deng, and Heng Huang. Faster derivative-free stochastic algorithm for shared memory machines. In ICML, pages 1807-1816, 2018.","title":"Faster derivative-free stochastic algorithm for shared memory machines","authors":"Bin Gu, Zhouyuan Huo, Cheng Deng, Heng Huang","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2011] Ran He, Wei-Shi Zheng, and Bao-Gang Hu. Maximum correntropy criterion for robust face recogni- tion. IEEE TPAMI, 33(8):1561-1576, 2011.","title":"Maximum correntropy criterion for robust face recognition","authors":"Ran He, Wei-Shi Zheng, Bao-Gang Hu","year":"2011","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang and Chen, 2015] Feihu Huang and Songcan Chen. Joint learning of multiple sparse matrix gaussian graph- ical models. IEEE transactions on neural networks and learning systems, 26(11):2606-2620, 2015.","title":"Joint learning of multiple sparse matrix gaussian graphical models","authors":"Chen ; Huang, Feihu Huang, Songcan Chen","year":"2015","venue":"IEEE transactions on neural networks and learning systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Huang et al., 2016] Feihu Huang, Songcan Chen, and Zhaosong Lu. Stochastic alternating direction method of multipliers with variance reduction for nonconvex opti- mization. arXiv preprint arXiv:1610.02758, 2016.","title":"Stochastic alternating direction method of multipliers with variance reduction for nonconvex optimization","authors":"Huang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Huang et al., 2019a] Feihu Huang, Songcan Chen, and Heng Huang. Faster stochastic alternating direction method of multipliers for nonconvex optimization. In ICML, pages 2839-2848, 2019.","title":"Faster stochastic alternating direction method of multipliers for nonconvex optimization","authors":"Huang","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Huang et al., 2019b] Feihu Huang, Bin Gu, Zhouyuan Huo, Songcan Chen, and Heng Huang. Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In AAAI, 2019.","title":"Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization","authors":"Huang","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jiang et al., 2019] Bo Jiang, Tianyi Lin, Shiqian Ma, and Shuzhong Zhang. Structured nonconvex and nonsmooth optimization: algorithms and iteration complexity anal- ysis. Computational Optimization and Applications, 72(1):115-157, 2019.","title":"Structured nonconvex and nonsmooth optimization: algorithms and iteration complexity analysis","authors":"Bo Jiang, Tianyi Lin, Shiqian Ma, Shuzhong Zhang","year":"2019","venue":"Computational Optimization and Applications","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Johnson and Zhang, 2013] Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In NIPS, pages 315-323, 2013.","title":"Accelerating stochastic gradient descent using predictive variance reduction","authors":"Rie Johnson, Tong Zhang","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kim et al., 2009] Seyoung Kim, Kyung-Ah Sohn, and Er- ic P Xing. A multivariate regression approach to associa- tion analysis of a quantitative trait network. Bioinformat- ics, 25(12):i204-i212, 2009.","title":"A multivariate regression approach to association analysis of a quantitative trait network","authors":"Kim","year":"2009","venue":"Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2018a] Sijia Liu, Jie Chen, Pin-Yu Chen, and Alfred Hero. Zeroth-order online alternating direction method of multipliers: Convergence analysis and applica- tions. In AISTATS, volume 84, pages 288-297, 2018.","title":"Zeroth-order online alternating direction method of multipliers: Convergence analysis and applications","authors":"Liu","year":"2018","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2018b] Sijia Liu, Bhavya Kailkhura, Pin-Yu Chen, Paishun Ting, Shiyu Chang, and Lisa Amini. Zeroth-order stochastic variance reduction for nonconvex optimization. In NIPS, pages 3731-3741, 2018.","title":"Zeroth-order stochastic variance reduction for nonconvex optimization","authors":"Liu","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nesterov and Spokoiny, 2017] Yurii Nesterov and Vladimir G. Spokoiny. Random gradient-free minimiza- tion of convex functions. Foundations of Computational Mathematics, 17:527-566, 2017.","title":"Random gradient-free minimization of convex functions","authors":"Yurii Nesterov, Vladimir G Spokoiny","year":"2017","venue":"Foundations of Computational Mathematics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ouyang et al., 2013] Hua Ouyang, Niao He, Long Tran, and Alexander G Gray. Stochastic alternating direction method of multipliers. ICML, 28:80-88, 2013.","title":"Stochastic alternating direction method of multipliers","authors":"Hua Ouyang, Niao He, Long Tran, Alexander G Gray","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Suzuki, 2014] Taiji Suzuki. Stochastic dual coordinate as- cent with alternating direction method of multipliers. In ICML, pages 736-744, 2014.","title":"Stochastic dual coordinate ascent with alternating direction method of multipliers","authors":"Taiji Suzuki, Suzuki","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Taylor et al., 2016] Gavin Taylor, Ryan Burmeister, Zheng Xu, Bharat Singh, Ankit Patel, and Tom Goldstein. Train- ing neural networks without gradients: a scalable admm approach. In ICML, pages 2722-2731, 2016.","title":"Training neural networks without gradients: a scalable admm approach","authors":"Gavin Taylor, Ryan Burmeister, Zheng Xu, Bharat Singh, Ankit Patel, Tom Goldstein","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2015] Fenghui Wang, Wenfei Cao, and Zong- ben Xu. Convergence of multi-block bregman admm for nonconvex composite problems. arXiv preprint arX- iv:1505.03063, 2015.","title":"Convergence of multi-block bregman admm for nonconvex composite problems","authors":"Fenghui Wang, Wenfei Cao, Zongben Xu","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2018] Kaidi Xu, Sijia Liu, Pu Zhao, Pin-Yu Chen, Huan Zhang, Deniz Erdogmus, Yanzhi Wang, and Xue Lin. Structured adversarial attack: Towards general imple- mentation and better interpretability. arXiv preprint arX- iv:1808.01664, 2018.","title":"Structured adversarial attack: Towards general implementation and better interpretability","authors":"Kaidi Xu, Sijia Liu, Pu Zhao, Pin-Yu Chen, Huan Zhang, Deniz Erdogmus, Yanzhi Wang, Xue Lin","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zheng and Kwok, 2016] Shuai Zheng and James T Kwok. Fast-and-light stochastic admm. In IJCAI, pages 2407- 2613, 2016.","title":"Fast-and-light stochastic admm","authors":"Shuai Zheng, James T Kwok","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}