{"type":"ScholarlyArticle","title":"Asynchronous Stochastic Frank-Wolfe Algorithms for Non-Convex Optimization","authors":[{"name":"Bin Gu","slug":"bin-gu","given":"Bin","family":"Gu","affiliation":"JD Finance America Corporation"},{"name":"Wenhan Xian","slug":"wenhan-xian","given":"Wenhan","family":"Xian","affiliation":"Department of Electrical & Computer Engineering , University of Pittsburgh , USA"},{"name":"Heng Huang","slug":"heng-huang","given":"Heng","family":"Huang","affiliation":"Department of Electrical & Computer Engineering , University of Pittsburgh , USA; JD Finance America Corporation"}],"abstract":"Asynchronous parallel stochastic optimization for non-convex  problems  becomes more and more   important in machine learning especially due to the popularity of deep learning. The Frank-Wolfe (a.k.a. conditional gradient) algorithms  has regained much interest  because of  its projection-free property and the ability of handling structured constraints. However,  our understanding of  asynchronous stochastic Frank-Wolfe algorithms is  extremely limited especially in the non-convex setting. To address this challenging problem, in this paper, we propose our  asynchronous stochastic  Frank-Wolfe algorithm (AsySFW) and  its variance reduction version (AsySVFW) for solving the constrained non-convex optimization problems.  More importantly, we  prove the fast convergence rates  of   AsySFW and AsySVFW in the non-convex setting. To the best of our knowledge, AsySFW and AsySVFW  are the first asynchronous parallel stochastic algorithms with convergence guarantees for solving the constrained  non-convex optimization problems. The  experimental  results on real high-dimensional gray-scale images   not only confirm the  fast convergence  of   our algorithms, but also show  a near-linear speedup  on a parallel system with shared memory due to the lock-free implementation.","doi":"10.24963/ijcai.2019/104","url":"https://ijcai.org/proceedings/2019/104","pdf":"https://ijcai.org/proceedings/2019/0104.pdf","markdown":"https://ijcai.org/proceedings/2019/104.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/104","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":104,"pages":{"first":737,"last":743},"keywords":["Machine Learning Applications: Big data ; Scalability","Computer Vision: Big Data and Large Scale Methods"],"videos":[],"references":[{"position":1,"raw":"[Allen-Zhu and Li, 2018] Zeyuan Allen-Zhu and Yuanzhi Li. Neon2: Finding local minima via first-order oracles. In Advances in Neural Information Processing Systems, pages 3720-3730, 2018.","title":"Neon2: Finding local minima via first-order oracles","authors":"Allen-Zhu And Li, ; ] Zeyuan Allen-Zhu, Yuanzhi Li","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen and Wang, 2018] Hong Chen and Yulong Wang. Kernel-based sparse regression with the correntropy- induced loss. Applied and Computational Harmonic Anal- ysis, 44(1):144-164, 2018.","title":"Kernel-based sparse regression with the correntropyinduced loss","authors":"Chen, Wang ;, Hong Chen, Yulong Wang","year":"2018","venue":"Applied and Computational Harmonic Analysis","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dudik et al., 2012] Miroslav Dudik, Zaid Harchaoui, and Jérôme Malick. Lifted coordinate descent for learning with trace-norm regularization. In Artificial Intelligence and Statistics, pages 327-336, 2012.","title":"Lifted coordinate descent for learning with trace-norm regularization","authors":"Dudik","year":"2012","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Feng et al., 2015] Yunlong Feng, Xiaolin Huang, Lei Shi, Yuning Yang, and Johan AK Suykens. Learning with the maximum correntropy criterion induced losses for regres- sion. Journal of Machine Learning Research, 16:993- 1034, 2015.","title":"Learning with the maximum correntropy criterion induced losses for regression","authors":"Yunlong Feng, Xiaolin Huang, Lei Shi, Yuning Yang, Johan Ak Suykens","year":"2015","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Frank and Wolfe, 1956] Marguerite Frank and Philip Wolfe. An algorithm for quadratic programming. Naval Research Logistics (NRL), 3(1-2):95-110, 1956.","title":"An algorithm for quadratic programming","authors":"Wolfe Frank, Marguerite Frank, Philip Wolfe","year":"1956","venue":"Naval Research Logistics (NRL)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Garber and Hazan, 2015] Dan Garber and Elad Hazan. Faster rates for the frank-wolfe method over strongly- convex sets. In ICML, volume 15, pages 541-549, 2015.","title":"Faster rates for the frank-wolfe method over stronglyconvex sets","authors":"Hazan ; Garber, Dan Garber, Elad Hazan","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goldfarb et al., 2017] Donald Goldfarb, Garud Iyengar, and Chaoxu Zhou. Linear convergence of stochastic frank wolfe variants. arXiv preprint arXiv:1703.07269, 2017.","title":"Linear convergence of stochastic frank wolfe variants","authors":"Donald Goldfarb, Garud Iyengar, Chaoxu Zhou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gu and Huo, 2018] Bin Gu and Zhouyuan Huo. Asyn- chronous doubly stochastic group regularized learning. In International Conference on Artificial Intelligence and S- tatistics (AISTATS 2018), 2018.","title":"Asynchronous doubly stochastic group regularized learning","authors":"Huo Gu, Bin Gu, Zhouyuan Huo","year":"2018","venue":"International Conference on Artificial Intelligence and Statistics (AISTATS 2018)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gu et al., 2017] Bin Gu, Guodong Liu, and Heng Huang. Groups-keeping solution path algorithm for sparse regres- sion with automatic feature grouping. In KDD, pages 185- 193. ACM, 2017.","title":"Groups-keeping solution path algorithm for sparse regression with automatic feature grouping","authors":"Bin Gu, Guodong Liu, Heng Huang","year":"2017","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gu et al., 2018a] Bin Gu, Zhouyuan Huo, Cheng Deng, and Heng Huang. Faster derivative-free stochastic algorithm for shared memory machines. In International Conference on Machine Learning, pages 1807-1816, 2018.","title":"Faster derivative-free stochastic algorithm for shared memory machines","authors":"Gu","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gu et al., 2018b] Bin Gu, De Wang, Zhouyuan Huo, and Heng Huang. Inexact proximal gradient methods for non- convex and non-smooth optimization. In AAAI, pages 3093-3100, 2018.","title":"Inexact proximal gradient methods for nonconvex and non-smooth optimization","authors":"Gu","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hazan and Luo, 2016] Elad Hazan and Haipeng Luo. Variance-reduced and projection-free stochastic optimiza- tion. In International Conference on Machine Learning, pages 1263-1271, 2016.","title":"Variance-reduced and projection-free stochastic optimization","authors":"Elad Hazan, Haipeng Luo","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hsieh et al., 2015] Cho-Jui Hsieh, Nagarajan Natarajan, and Inderjit S Dhillon. Pu learning for matrix completion. In ICML, pages 2445-2453, 2015.","title":"Pu learning for matrix completion","authors":"Cho-Jui Hsieh, Nagarajan Natarajan, Inderjit S Dhillon","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Huo et al., 2018] Zhouyuan Huo, Bin Gu, Ji Liu, and Heng Huang. Accelerated method for stochastic composition optimization with nonsmooth regularization. In Thirty- Second AAAI Conference on Artificial Intelligence, 2018. [Jaggi, 2013] Martin Jaggi. Revisiting frank-wolfe: Projection-free sparse convex optimization. In ICML (1), pages 427-435, 2013.","title":"Accelerated method for stochastic composition optimization with nonsmooth regularization","authors":"Huo","year":"2013","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lacoste-Julien and Jaggi, 2015] Simon Lacoste-Julien and Martin Jaggi. On the global linear convergence of frank- wolfe optimization variants. In Advances in Neural Infor- mation Processing Systems, pages 496-504, 2015.","title":"On the global linear convergence of frankwolfe optimization variants","authors":"Jaggi Lacoste-Julien, Simon Lacoste-Julien, Martin Jaggi","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lacoste-Julien et al., 2013] Simon Lacoste-Julien, Martin Jaggi, Mark Schmidt, and Patrick Pletscher. Block- coordinate frank-wolfe optimization for structural svm- s. In ICML 2013 International Conference on Machine Learning, pages 53-61, 2013.","title":"Blockcoordinate frank-wolfe optimization for structural svms","authors":"Simon Lacoste-Julien, Martin Jaggi, Mark Schmidt, Patrick Pletscher","year":"2013","venue":"ICML 2013 International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lacoste-Julien, 2016] Simon Lacoste-Julien. Convergence rate of frank-wolfe for non-convex objectives. arXiv preprint arXiv:1607.00345, 2016.","title":"Convergence rate of frank-wolfe for non-convex objectives","authors":"Simon Lacoste-Julien","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lan, 2013] Guanghui Lan. The complexity of large-scale convex programming under a linear optimization oracle. arXiv preprint arXiv:1309.5550, 2013.","title":"The complexity of large-scale convex programming under a linear optimization oracle","authors":"Lan ;, Guanghui Lan","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lei et al., 2017] Lihua Lei, Cheng Ju, Jianbo Chen, and Michael I Jordan. Non-convex finite-sum optimization via scsg methods. In Advances in Neural Information Process- ing Systems, pages 2348-2358, 2017.","title":"Non-convex finite-sum optimization via scsg methods","authors":"Lihua Lei, Cheng Ju, Jianbo Chen, Michael I Jordan","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mania et al., 2017] Horia Mania, Xinghao Pan, Dimitris Pa- pailiopoulos, Benjamin Recht, Kannan Ramchandran, and Michael I Jordan. Perturbed iterate analysis for asyn- chronous stochastic optimization. SIAM Journal on Op- timization, 27(4):2202-2229, 2017.","title":"Perturbed iterate analysis for asynchronous stochastic optimization","authors":"Horia Mania, Xinghao Pan, Dimitris Papailiopoulos, Benjamin Recht, Kannan Ramchandran, Michael I Jordan","year":"2017","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Moharrer and Ioannidis, 2017] Armin Moharrer and Stratis Ioannidis. Distributing frank-wolfe via map-reduce. In 2017 IEEE International Conference on Data Mining, ICDM 2017, New Orleans, LA, USA, November 18-21, 2017, pages 317-326, 2017.","title":"Distributing frank-wolfe via map-reduce","authors":"Ioannidis Moharrer, Armin Moharrer, Stratis Ioannidis","year":"2017","venue":"IEEE International Conference on Data Mining, ICDM 2017","doi":null,"url":null,"internal_url":"/proceedings/2018/748"},{"position":22,"raw":"[Recht et al., 2011] Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in neural information processing systems, pages 693-701, 2011.","title":"Hogwild: A lock-free approach to parallelizing stochastic gradient descent","authors":"Benjamin Recht, Christopher Re, Stephen Wright, Feng Niu","year":"2011","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Reddi et al., 2016] Sashank J Reddi, Suvrit Sra, Barnabás Póczos, and Alex Smola. Stochastic frank-wolfe meth- ods for nonconvex optimization. In Communication, Con- trol, and Computing (Allerton), 2016 54th Annual Allerton Conference on, pages 1244-1251. IEEE, 2016.","title":"Stochastic frank-wolfe methods for nonconvex optimization","authors":"J Sashank, Suvrit Reddi, Barnabás Sra, Alex Póczos, Smola","year":"2016","venue":"Communication, Control, and Computing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2016] Yu-Xiang Wang, Veeranjaneyulu Sad- hanala, Wei Dai, Willie Neiswanger, Suvrit Sra, and Er- ic Xing. Parallel and distributed block-coordinate frank- wolfe algorithms. In International Conference on Machine Learning, pages 1548-1557, 2016.","title":"Parallel and distributed block-coordinate frankwolfe algorithms","authors":"Yu-Xiang Wang, Veeranjaneyulu Sadhanala, Wei Dai, Willie Neiswanger, Suvrit Sra, Eric Xing","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2018] Yuning Yang, Yunlong Feng, and Jo- han AK Suykens. Correntropy based matrix completion. Entropy, 20(3):171, 2018.","title":"Correntropy based matrix completion","authors":"Yang","year":"2018","venue":"Entropy","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2017] Wenpeng Zhang, Peilin Zhao, Wenwu Zhu, Steven CH Hoi, and Tong Zhang. Projection-free dis- tributed online learning in networks. In International Con- ference on Machine Learning, pages 4054-4062, 2017.","title":"Projection-free distributed online learning in networks","authors":"Zhang","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}