{"type":"ScholarlyArticle","title":"Generalized Majorization-Minimization for Non-Convex Optimization","authors":[{"name":"Hu Zhang","slug":"hu-zhang","given":"Hu","family":"Zhang","affiliation":"School of Computer Science , University of Technology Sydney , Australia"},{"name":"Pan Zhou","slug":"pan-zhou","given":"Pan","family":"Zhou","affiliation":"Dept. ECE , National University of Singapore , Singapore"},{"name":"Yi Yang","slug":"yi-yang","given":"Yi","family":"Yang","affiliation":"School of Computer Science , University of Technology Sydney , Australia; Baidu Research"},{"name":"Jiashi Feng","slug":"jiashi-feng","given":"Jiashi","family":"Feng","affiliation":"Dept. ECE , National University of Singapore , Singapore"}],"abstract":"Majorization-Minimization (MM) algorithms optimize an objective function by iteratively minimizing its majorizing surrogate and offer attractively fast convergence rate for convex problems. However, their convergence behaviors for non-convex problems remain unclear. In this paper, we propose a novel MM surrogate function from strictly upper bounding the objective to bounding the objective in expectation. With this generalized surrogate conception, we develop a new optimization algorithm, termed SPI-MM, that leverages the recent proposed SPIDER for more efficient non-convex optimization. We prove that for finite-sum problems, the SPI-MM algorithm converges to an stationary point within deterministic and lower stochastic gradient complexity. To our best knowledge, this work gives the first non-asymptotic convergence analysis for MM-alike algorithms in general non-convex optimization. Extensive empirical studies on non-convex logistic regression and sparse PCA demonstrate the advantageous efficiency of the proposed algorithm and validate our theoretical results.","doi":"10.24963/ijcai.2019/591","url":"https://ijcai.org/proceedings/2019/591","pdf":"https://ijcai.org/proceedings/2019/0591.pdf","markdown":"https://ijcai.org/proceedings/2019/591.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/591","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":591,"pages":{"first":4257,"last":4263},"keywords":["Computer Vision: Big Data and Large Scale Methods","Machine Learning: Data Mining"],"videos":[],"references":[{"position":1,"raw":"[Agarwal and Bottou, 2014] Alekh Agarwal and Leon Bot- tou. A lower bound for the optimization of finite sums. arXiv preprint arXiv:1410.0723, 2014.","title":"Alekh Agarwal and Leon Bottou. A lower bound for the optimization of finite sums","authors":"Bottou Agarwal","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bietti and Mairal, 2017] Alberto Bietti and Julien Mairal. Stochastic optimization with variance reduction for infinite datasets with finite sum structure. In Advances in Neural Information Processing Systems, pages 1623-1633, 2017.","title":"Stochastic optimization with variance reduction for infinite datasets with finite sum structure","authors":"Alberto Bietti, Julien Mairal","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Borwein and Lewis, 2010] Jonathan Borwein and Adrian S Lewis. Convex analysis and nonlinear optimization: the- ory and examples. Springer Science & Business Media, 2010.","title":"Convex analysis and nonlinear optimization: theory and examples","authors":"Lewis ; Borwein, Jonathan Borwein, Adrian S Lewis","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cappé and Moulines, 2009] Olivier Cappé and Eric Moulines. On-line expectation-maximization algorithm for latent data models. Journal of the Royal Statistical So- ciety: Series B (Statistical Methodology), 71(3):593-613, 2009.","title":"On-line expectation-maximization algorithm for latent data models","authors":"Moulines Cappé, Olivier Cappé, Eric Moulines","year":"2009","venue":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chang and Lin, 2011] Chih-Chung Chang and Chih-Jen Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1-27:27, 2011. Software available at http://www. csie.ntu.edu.tw/ ∼ cjlin/libsvm.","title":"LIBSVM: A library for support vector machines","authors":"Chang, Lin ; Chih-Chung Chang, Chih-Jen Lin","year":"2011","venue":"ACM Transactions on Intelligent Systems and Technology","doi":null,"url":"http://www.csie.ntu.edu.tw/∼cjlin/libsvm","internal_url":null},{"position":6,"raw":"[Defazio et al., 2014] Aaron Defazio, Francis Bach, and Si- mon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. In Advances in neural information processing systems, pages 1646-1654, 2014.","title":"Saga: A fast incremental gradient method with support for non-strongly convex composite objectives","authors":"Aaron Defazio, Francis Bach, Simon Lacoste-Julien","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Draper and Smith, 2014] Norman R Draper and Harry Smith. Applied regression analysis, volume 326. John Wiley & Sons, 2014.","title":"Applied regression analysis","authors":"R Norman, Harry Draper, Smith","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fang et al., 2018] Cong Fang, Chris Junchi Li, Zhouchen Lin, and Tong Zhang. Spider: Near-optimal non-convex optimization via stochastic path-integrated differential es- timator. In Advances in Neural Information Processing Systems, pages 687-697, 2018.","title":"Spider: Near-optimal non-convex optimization via stochastic path-integrated differential estimator","authors":"Cong Fang, Chris Junchi Li, Zhouchen Lin, Tong Zhang","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Feng et al., 2013] Jiashi Feng, Huan Xu, and Shuicheng Yan. Online robust pca via stochastic optimization. In Ad- vances in Neural Information Processing Systems, pages 404-412, 2013.","title":"Online robust pca via stochastic optimization","authors":"Feng","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gasso et al., 2009] Gilles Gasso, Alain Rakotomamonjy, and Stéphane Canu. Recovering sparse signals with a cer- tain family of nonconvex penalties and dc programming. IEEE Transactions on Signal Processing, 57(12):4686- 4698, 2009.","title":"Recovering sparse signals with a certain family of nonconvex penalties and dc programming","authors":"Gilles Gasso, Alain Rakotomamonjy, Stéphane Canu","year":"2009","venue":"IEEE Transactions on Signal Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Johnson and Zhang, 2013] Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in neural information pro- cessing systems, pages 315-323, 2013.","title":"Accelerating stochastic gradient descent using predictive variance reduction","authors":"Rie Johnson, Tong Zhang","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kang et al., 2015] Yangyang Kang, Zhihua Zhang, and Wu- Jun Li. On the global convergence of majorization min- imization algorithms for nonconvex optimization prob- lems. arXiv preprint arXiv:1504.07791, 2015.","title":"On the global convergence of majorization minimization algorithms for nonconvex optimization problems","authors":"Yangyang Kang, Zhihua Zhang, Wu-Jun Li","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lange et al., 2000] Kenneth Lange, David R Hunter, and Il- soon Yang. Optimization transfer using surrogate objec- tive functions. Journal of computational and graphical statistics, 9(1):1-20, 2000.","title":"Optimization transfer using surrogate objective functions","authors":"Lange","year":"2000","venue":"Journal of computational and graphical statistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Ge- offrey Hinton. Deep learning. nature, 521(7553):436, 2015.","title":"Deep learning","authors":"Yann Lecun, Yoshua Bengio, Geoffrey Hinton","year":"2015","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mairal, 2013a] Julien Mairal. Optimization with first-order surrogate functions. In International Conference on Ma- chine Learning, pages 783-791, 2013.","title":"Optimization with first-order surrogate functions","authors":"Julien Mairal, Mairal","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mairal, 2013b] Julien Mairal. Stochastic majorization- minimization algorithms for large-scale optimization. In Advances in Neural Information Processing Systems, pages 2283-2291, 2013.","title":"Stochastic majorizationminimization algorithms for large-scale optimization","authors":"Julien Mairal, Mairal","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mairal, 2015] Julien Mairal. Incremental majorization- minimization optimization with application to large-scale machine learning. SIAM Journal on Optimization, 25(2):829-855, 2015.","title":"Incremental majorizationminimization optimization with application to large-scale machine learning","authors":"Julien Mairal","year":"2015","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nesterov, 1998] Yurii Nesterov. Introductory lectures on convex programming volume i: Basic course. Lecture notes, 1998.","title":"Introductory lectures on convex programming volume i: Basic course","authors":"Yurii Nesterov","year":"1998","venue":"Lecture notes","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nguyen et al., 2017] Lam M Nguyen, Jie Liu, Katya Scheinberg, and Martin Takáč. Sarah: A novel method for machine learning problems using stochastic recursive gradient. In Proceedings of the 34th International Confer- ence on Machine Learning-Volume 70, pages 2613-2621. JMLR. org, 2017.","title":"Sarah: A novel method for machine learning problems using stochastic recursive gradient","authors":"Jie Lam M Nguyen, Katya Liu, Martin Scheinberg, Takáč","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Parizi et al., 2015] Sobhan Naderi Parizi, Kun He, Stan Sclaroff, and Pedro Felzenszwalb. Gener- alized majorization-minimization. arXiv preprint arXiv:1506.07613, 2015.","title":"Generalized majorization-minimization","authors":"Sobhan Naderi Parizi, Kun He, Stan Sclaroff, Pedro Felzenszwalb","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Razaviyayn et al., 2016] Meisam Razaviyayn, Maziar San- jabi, and Zhi-Quan Luo. A stochastic successive min- imization method for nonsmooth nonconvex optimiza- tion with applications to transceiver design in wireless communication networks. Mathematical Programming, 157(2):515-545, 2016.","title":"A stochastic successive minimization method for nonsmooth nonconvex optimization with applications to transceiver design in wireless communication networks","authors":"Razaviyayn","year":"2016","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shalev-Shwartz and Zhang, 2013] Shai Shalev-Shwartz and Tong Zhang. Stochastic dual coordinate ascent methods for regularized loss minimization. Journal of Machine Learning Research, 14(Feb):567-599, 2013.","title":"Stochastic dual coordinate ascent methods for regularized loss minimization","authors":"Shalev-Shwartz Zhang, ; ] Shai Shalev-Shwartz, Tong Zhang","year":"2013","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vaida, 2005] Florin Vaida. Parameter convergence for em and mm algorithms. Statistica Sinica, 15(3):831, 2005.","title":"Parameter convergence for em and mm algorithms","authors":"Florin Vaida","year":"2005","venue":"Statistica Sinica","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2018] Zhe Wang, Kaiyi Ji, Yi Zhou, Yingbin Liang, and Vahid Tarokh. Spiderboost: A class of faster variance-reduced algorithms for nonconvex optimization. arXiv preprint arXiv:1810.10690, 2018.","title":"Spiderboost: A class of faster variance-reduced algorithms for nonconvex optimization","authors":"Zhe Wang, Kaiyi Ji, Yi Zhou, Yingbin Liang, Vahid Tarokh","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2016] Chen Xu, Zhouchen Lin, Zhenyu Zhao, and Hongbin Zha. Relaxed majorization-minimization for non-smooth and non-convex optimization. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.","title":"Relaxed majorization-minimization for non-smooth and non-convex optimization","authors":"Chen Xu, Zhouchen Lin, Zhenyu Zhao, Hongbin Zha","year":"2016","venue":"Thirtieth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zou et al., 2006] Hui Zou, Trevor Hastie, and Robert Tib- shirani. Sparse principal component analysis. Journal of computational and graphical statistics, 15(2):265-286, 2006.","title":"Sparse principal component analysis","authors":"Hui Zou, Trevor Hastie, Robert Tibshirani","year":"2006","venue":"Journal of computational and graphical statistics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}