{"type":"ScholarlyArticle","title":"Proximal Gradient Algorithm with Momentum and Flexible Parameter Restart for Nonconvex Optimization","authors":[{"name":"Yi Zhou","slug":"yi-zhou","given":"Yi","family":"Zhou","affiliation":"University of Utah"},{"name":"Zhe Wang","slug":"zhe-wang","given":"Zhe","family":"Wang","affiliation":"The Ohio State University"},{"name":"Kaiyi Ji","slug":"kaiyi-ji","given":"Kaiyi","family":"Ji","affiliation":"The Ohio State University"},{"name":"Yingbin Liang","slug":"yingbin-liang","given":"Yingbin","family":"Liang","affiliation":"The Ohio State University"},{"name":"Vahid Tarokh","slug":"vahid-tarokh","given":"Vahid","family":"Tarokh","affiliation":"Duke University"}],"abstract":"Various types of parameter restart schemes have been proposed for proximal gradient algorithm with momentum to facilitate their convergence in convex optimization. However, under parameter restart, the convergence of proximal gradient algorithm with momentum remains obscure in nonconvex optimization. \r\n In this paper, we propose a novel proximal gradient algorithm with momentum and parameter restart for solving nonconvex and nonsmooth problems. Our algorithm is designed to 1) allow for adopting flexible parameter restart schemes that cover many existing ones; 2) have a global sub-linear convergence rate in nonconvex and nonsmooth optimization; and 3) have guaranteed convergence to a critical point and have various types of asymptotic convergence rates depending on the parameterization of local geometry in nonconvex and nonsmooth optimization. Numerical experiments demonstrate the convergence and effectiveness of our proposed algorithm.","doi":"10.24963/ijcai.2020/201","url":"https://ijcai.org/proceedings/2020/201","pdf":"https://ijcai.org/proceedings/2020/0201.pdf","markdown":"https://ijcai.org/proceedings/2020/201.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/201","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Data Mining","track":"Main track","number":201,"pages":{"first":1445,"last":1451},"keywords":["Data Mining: Theoretical Foundations","Constraints and SAT: Constraint Optimization"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24027/file_24027.m3u8","length_sec":290,"provider":"hls"}],"references":[{"position":1,"raw":"[Attouch and Bolte, 2009] H. Attouch and J. Bolte. On the conver- gence of the proximal algorithm for nonsmooth functions involv- ing analytic features. Mathematical Programming, 116(1-2):5- 16, 2009.","title":"On the convergence of the proximal algorithm for nonsmooth functions involving analytic features","authors":"H Attouch, J Bolte","year":"2009","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Attouch et al., 2010] H. Attouch, J. Bolte, P. Redont, and A. Soubeyran. Proximal alternating minimization and projec- tion methods for nonconvex problems: An approach based on the Kurdyka-Łojasiewicz inequality. Mathematics of Operations Research, 35(2):438-457, 2010.","title":"Proximal alternating minimization and projection methods for nonconvex problems: An approach based on the Kurdyka-Łojasiewicz inequality","authors":"H Attouch, J Bolte, P Redont, A Soubeyran","year":"2010","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Attouch et al., 2013] H. Attouch, J. Bolte, and B. Svaiter. Con- vergence of descent methods for semi-algebraic and tame prob- lems: proximal algorithms, forward-backward splitting, and reg- ularized Gauss-Seidel methods. Mathematical Programming, 137(1):91-129, Feb 2013.","title":"Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods","authors":"H Attouch, J Bolte, B Svaiter","year":"2013","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Beck and Teboulle, 2009] A. Beck and M. Teboulle. A fast iter- ative shrinkage-thresholding algorithm for linear inverse prob- lems. SIAM Journal on Imaging Sciences, 2(1):183-202, March 2009.","title":"A fast iterative shrinkage-thresholding algorithm for linear inverse problems","authors":"A Beck, M Teboulle","year":"2009","venue":"SIAM Journal on Imaging Sciences","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bolte et al., 2007] J. Bolte, A. Daniilidis, and A. Lewis. The Łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems. SIAM Journal on Optimization, 17:1205-1223, 2007.","title":"The Łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems","authors":"J Bolte, A Daniilidis, A Lewis","year":"2007","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bolte et al., 2014] J. Bolte, S. Sabach, and M. Teboulle. Proxi- mal alternating linearized minimization for nonconvex and non- smooth problems. Mathematical Programming, 146(1-2):459- 494, 2014.","title":"Proximal alternating linearized minimization for nonconvex and nonsmooth problems","authors":"J Bolte, S Sabach, M Teboulle","year":"2014","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chang and Lin, 2011] C. Chang and C. Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(3):1-27, 2011.","title":"LIBSVM: A library for support vector machines","authors":"C Chang, C Lin","year":"2011","venue":"ACM Transactions on Intelligent Systems and Technology","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fercoq and Qu, 2016] O. Fercoq and Z. Qu. Restarting acceler- ated gradient methods with a rough strong convexity estimate. ArXiv:1609.07358v1, Sep 2016.","title":"Restarting accelerated gradient methods with a rough strong convexity estimate","authors":"O Fercoq, Z Qu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fercoq and Qu, 2017] O. Fercoq and Z. Qu. Adaptive restart of accelerated gradient methods under local quadratic growth con- dition. arXiv:1709.02300v1, Sep 2017.","title":"Adaptive restart of accelerated gradient methods under local quadratic growth condition","authors":"O Fercoq, Z Qu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ghadimi and Lan, 2016] S. Ghadimi and G. Lan. Accelerated gradient methods for nonconvex nonlinear and stochastic pro- gramming. Mathematical Programming, 156(1-2):59-99, March 2016.","title":"Accelerated gradient methods for nonconvex nonlinear and stochastic programming","authors":"S Ghadimi, G Lan","year":"2016","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ghadimi et al., 2016] S. Ghadimi, G. Lan, and H. Zhang. Mini- batch stochastic approximation methods for nonconvex stochas- tic composite optimization. Mathematical Programming, 155(1):267-305, Jan 2016.","title":"Minibatch stochastic approximation methods for nonconvex stochastic composite optimization","authors":"S Ghadimi, G Lan, H Zhang","year":"2016","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Giselsson and Boyd, 2014] P. Giselsson and S. Boyd. Monotonic- ity and restart in fast gradient methods. In Proc. IEEE Conference on Decision and Control (CDC), pages 5058-5063, Dec 2014.","title":"Monotonicity and restart in fast gradient methods","authors":"P Giselsson, S Boyd","year":"2014","venue":"Proc. IEEE Conference on Decision and Control (CDC)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kim and Fessler, 2018] D. Kim and J.A. Fessler. Adaptive restart of the optimized gradient method for convex optimization. Jour- nal of Optimization Theory and Applications, 178(1):240-263, Jul 2018.","title":"Adaptive restart of the optimized gradient method for convex optimization","authors":"D Kim, J A Fessler","year":"2018","venue":"Journal of Optimization Theory and Applications","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li and Lin, 2015] H. Li and Z. Lin. Accelerated proximal gradi- ent methods for nonconvex programming. In Proc. International Conference on Neural Information Processing Systems (NIPS), pages 379-387, 2015.","title":"Accelerated proximal gradient methods for nonconvex programming","authors":"H Li, Z Lin","year":"2015","venue":"Proc. International Conference on Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2017] Q. Li, Y. Zhou, Y. Liang, and P. K. Varshney. Con- vergence analysis of proximal gradient with momentum for non- convex optimization. In Proc. International Conference on Ma- chine Learning (ICML, pages 2111-2119, Aug 2017.","title":"Convergence analysis of proximal gradient with momentum for nonconvex optimization","authors":"Li","year":"2017","venue":"Proc. International Conference on Machine Learning (ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liang and Schonlieb, 2018] J. Liang and C. Schonlieb. Improving FISTA: Faster, Smarter and Greedier. arXiv:1811.01430v2, Nov 2018.","title":"Improving FISTA: Faster, Smarter and Greedier","authors":"J Liang, C Schonlieb","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liang et al., 2016] Jingwei Liang, Jalal M. Fadili, and Gabriel Peyré. A multi-step inertial forward-backward splitting method for non-convex optimization. In Proc. Neural Information Pro- cessing Systems (NIPS), pages 4042-4050, 2016.","title":"A multi-step inertial forward-backward splitting method for non-convex optimization","authors":"Jingwei Liang, Jalal M Fadili, Gabriel Peyré","year":"2016","venue":"Proc. Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lin and Xiao, 2015] Q. Lin and L. Xiao. An adaptive acceler- ated proximal gradient method and its homotopy continuation for sparse optimization. Computational Optimization and Applica- tions, 60(3):633-674, Apr 2015.","title":"An adaptive accelerated proximal gradient method and its homotopy continuation for sparse optimization","authors":"Q Lin, L Xiao","year":"2015","venue":"Computational Optimization and Applications","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu and Yang, 2017] M. Liu and T. Yang. Adaptive accelerated gradient converging method under holderian error bound condi- tion. In Proc. Advances in Neural Information Processing Sys- tems (NIPS), pages 3104-3114. 2017.","title":"Adaptive accelerated gradient converging method under holderian error bound condition","authors":"M Liu, T Yang","year":"2017","venue":"Proc. Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nesterov, 2007] Y. Nesterov. Gradient methods for minimizing composite objective function. Core discussion papers, 2007.","title":"Gradient methods for minimizing composite objective function","authors":"Y Nesterov","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nesterov, 2014] Y. Nesterov. Introductory Lectures on Convex Op- timization: A Basic Course. Springer, 2014.","title":"Introductory Lectures on Convex Optimization: A Basic Course","authors":"Y Nesterov","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ochs, 2018] P. Ochs. Local convergence of the heavy-ball method and iPiano for non-convex optimization. Journal of Optimization Theory and Applications, 177(1):153-180, Apr 2018.","title":"Local convergence of the heavy-ball method and iPiano for non-convex optimization","authors":"P Ochs","year":"2018","venue":"Journal of Optimization Theory and Applications","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[O'Donoghue and Candès, 2015] B. O'Donoghue and E. Candès. Adaptive restart for accelerated gradient schemes. Foundations of Computational Mathematics, 15(3):715-732, Jun 2015.","title":"Adaptive restart for accelerated gradient schemes","authors":"B O'donoghue, E Candès","year":"2015","venue":"Foundations of Computational Mathematics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Renegar and Grimmer, 2018] J. Renegar and B. Grimmer. A sim- ple nearly-optimal restart scheme for speeding-up first order methods. arXiv:1803.00151v1, Mar 2018.","title":"A simple nearly-optimal restart scheme for speeding-up first order methods","authors":"Renegar, ] J Grimmer, B Renegar, Grimmer","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rockafellar and Wets, 1997] R.T. Rockafellar and R.J.B. Wets. Variational Analysis. Springer, 1997.","title":"Variational Analysis","authors":"R T Rockafellar, R J B Wets","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Roulet and dAspremont, 2017] V. Roulet and A. dAspremont. Sharpness, restart and acceleration. In Proc. Advances in Neu- ral Information Processing Systems (NIPS), pages 1119-1129. 2017.","title":"Sharpness, restart and acceleration","authors":"V Roulet, A Daspremont","year":"2017","venue":"Proc. Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tseng, 2010] P. Tseng. Approximation accuracy, gradient meth- ods, and error bound for structured convex optimization. Mathe- matical Programming, 125(2):263-295, Oct 2010.","title":"Approximation accuracy, gradient methods, and error bound for structured convex optimization","authors":"P Tseng","year":"2010","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2019] G. Wang, G. B. Giannakis, and J. Chen. Learn- ing ReLU networks on linearly separable data: Algorithm, op- timality, and generalization. IEEE Transactions on Signal Pro- cessing, 67(9):2357-2370, 2019.","title":"Learning ReLU networks on linearly separable data: Algorithm, optimality, and generalization","authors":"G Wang, G B Giannakis, J Chen","year":"2019","venue":"IEEE Transactions on Signal Processing","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2016] Y. Zhou, Y. Yu, W. Dai, Y. Liang, and P. Xing. On convergence of model parallel proximal gradient algorithm for stale synchronous parallel system. In Proc. International Conference on Artificial Intelligence and Statistics (AISTATS, pages 713-722, May 2016.","title":"On convergence of model parallel proximal gradient algorithm for stale synchronous parallel system","authors":"Y Zhou, Y Yu, W Dai, Y Liang, P Xing","year":"2016","venue":"Proc. International Conference on Artificial Intelligence and Statistics (AISTATS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhou et al., 2018a] Y. Zhou, Z. Wang, and Y. Liang. Convergence of cubic regularization for nonconvex optimization under KŁ property. In Proc. Advances in Neural Information Processing Systems (NeurIPS), 2018.","title":"Convergence of cubic regularization for nonconvex optimization under KŁ property","authors":"Zhou","year":"2018","venue":"Proc. Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhou et al., 2018b] Y. Zhou, Y. Yu, W. Dai, Y. Liang, and E. P. Xing. Distributed proximal gradient algorithm for partially asyn- chronous computer clusters. Journal of Machine Learning Re- search (JMLR), 19(19):1-32, 2018.","title":"Distributed proximal gradient algorithm for partially asynchronous computer clusters","authors":"Zhou","year":"2018","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}