{"type":"ScholarlyArticle","title":"Accelerated Incremental Gradient Descent using Momentum Acceleration with Scaling Factor","authors":[{"name":"Yuanyuan Liu","slug":"yuanyuan-liu","given":"Yuanyuan","family":"Liu","affiliation":"Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education; School of Artificial Intelligence , Xidian University , China"},{"name":"Fanhua Shang","slug":"fanhua-shang","given":"Fanhua","family":"Shang","affiliation":"Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education; School of Artificial Intelligence , Xidian University , China"},{"name":"Licheng Jiao","slug":"licheng-jiao","given":"Licheng","family":"Jiao","affiliation":"Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education; School of Artificial Intelligence , Xidian University , China"}],"abstract":"Recently, research on variance reduced incremental gradient descent methods (e.g., SAGA) has made exciting progress (e.g., linear convergence for strongly convex (SC) problems). However, existing accelerated methods (e.g., point-SAGA) suffer from drawbacks such as inflexibility. In this paper, we design a novel and simple momentum to accelerate the classical SAGA algorithm, and propose a direct accelerated incremental gradient descent algorithm. In particular, our theoretical result shows that our algorithm attains a best known oracle complexity for strongly convex problems and an improved convergence rate for the case of n>=L/\\mu. We also give experimental results justifying our theoretical results and showing the effectiveness of our algorithm.","doi":"10.24963/ijcai.2019/422","url":"https://ijcai.org/proceedings/2019/422","pdf":"https://ijcai.org/proceedings/2019/0422.pdf","markdown":"https://ijcai.org/proceedings/2019/422.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/422","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":422,"pages":{"first":3045,"last":3051},"keywords":["Machine Learning Applications: Other Applications","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Allen-Zhu and Hazan, 2016] Zeyuan Allen-Zhu and Elad Hazan. Optimal black-box reductions between optimization objectives. In NIPS, 2016.","title":"Optimal black-box reductions between optimization objectives","authors":"Allen-Zhu Hazan ; Zeyuan Allen-Zhu, Elad Hazan","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Allen-Zhu and Yuan, 2016] Zeyuan Allen-Zhu and Yang Yuan. Improved SVRG for non-strongly-convex or sum-of-non- convex objectives. In ICML, pages 1080-1089, 2016.","title":"Improved SVRG for non-strongly-convex or sum-of-nonconvex objectives","authors":"Allen-Zhu Yuan ; Zeyuan Allen-Zhu, Yang Yuan","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Allen-Zhu, 2018] Zeyuan Allen-Zhu. Katyusha: The first direct acceleration of stochastic gradient methods. J. Mach. Learn. Res., 18:1-51, 2018.","title":"Katyusha: The first direct acceleration of stochastic gradient methods","authors":"Allen-Zhu ; Zeyuan Allen-Zhu","year":"2018","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Beck and Teboulle, 2009] Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci., 2(1):183-202, 2009.","title":"A fast iterative shrinkage-thresholding algorithm for linear inverse problems","authors":"Amir Beck, Marc Teboulle","year":"2009","venue":"SIAM J. Imaging Sci","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Defazio et al., 2014a] Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. In NIPS, pages 1646-1654, 2014.","title":"SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives","authors":"Defazio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Defazio et al., 2014b] Aaron Defazio, Tiberio S. Caetano, and Justin Domke. Finito: A faster, permutable incremental gradi- ent method for big data problems. In ICML, pages 1125-1133, 2014.","title":"Finito: A faster, permutable incremental gradient method for big data problems","authors":"Defazio","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Defazio, 2016] Aaron Defazio. A simple practical accelerated method for finite sums. In NIPS, pages 676-684, 2016.","title":"A simple practical accelerated method for finite sums","authors":"Aaron Defazio","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Frostig et al., 2015] Roy Frostig, Rong Ge, Sham M. Kakade, and Aaron Sidford. Un-regularizing: approximate proximal point and faster stochastic algorithms for empirical risk mini- mization. In ICML, pages 2540-2548, 2015.","title":"Un-regularizing: approximate proximal point and faster stochastic algorithms for empirical risk minimization","authors":"Roy Frostig, Rong Ge, M Sham, Aaron Kakade, Sidford","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hien et al., 2019] Le Thi Khanh Hien, Cuong V. Nguyen, Huan Xu, Canyi Lu, and Jiashi Feng. Accelerated stochastic mir- ror descent algorithms for composite non-strongly convex opti- mization. J. Optimiz. Theory App., 2019.","title":"Accelerated stochastic mirror descent algorithms for composite non-strongly convex optimization","authors":"Le Thi, Khanh Hien, Cuong V Nguyen, Huan Xu, Canyi Lu, Jiashi Feng","year":"2019","venue":"J. Optimiz. Theory App","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Johnson and Zhang, 2013] Rie Johnson and Tong Zhang. Accel- erating stochastic gradient descent using predictive variance re- duction. In NIPS, pages 315-323, 2013.","title":"Accelerating stochastic gradient descent using predictive variance reduction","authors":"Rie Johnson, Tong Zhang","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim et al., 2009] Seyoung Kim, Kyung-Ah Sohn, and Eric P. X- ing. A multivariate regression approach to association analysis of a quantitative trait network. Bioinformatics, 25:i204-i212, 2009.","title":"Xing. A multivariate regression approach to association analysis of a quantitative trait network","authors":"Seyoung Kim, Kyung-Ah Sohn, Eric P","year":"2009","venue":"Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin et al., 2015] Hongzhou Lin, Julien Mairal, and Zaid Har- chaoui. A universal catalyst for first-order optimization. In NIPS, pages 3366-3374, 2015.","title":"A universal catalyst for first-order optimization","authors":"Hongzhou Lin, Julien Mairal, Zaid Harchaoui","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2017] Yuanyuan Liu, Fanhua Shang, and James Cheng. Accelerated variance reduced stochastic ADMM. In AAAI, pages 2287-2293, 2017.","title":"Accelerated variance reduced stochastic ADMM","authors":"Yuanyuan Liu, Fanhua Shang, James Cheng","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mahdavi et al., 2013] Mehrdad Mahdavi, Lijun Zhang, and Rong Jin. Mixed optimization for smooth functions. In NIPS, pages 674-682, 2013.","title":"Mixed optimization for smooth functions","authors":"Mehrdad Mahdavi, Lijun Zhang, Rong Jin","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Murata and Suzuki, 2017] Tomoya Murata and Taiji Suzuki. Doubly accelerated stochastic variance reduced dual averaging method for regularized empirical risk minimization. In NIPS, pages 608-617, 2017.","title":"Doubly accelerated stochastic variance reduced dual averaging method for regularized empirical risk minimization","authors":"Suzuki Murata, Tomoya Murata, Taiji Suzuki","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nitanda, 2014] Atsushi Nitanda. Stochastic proximal gradien- t descent with acceleration techniques. In NIPS, pages 1574- 1582, 2014.","title":"Stochastic proximal gradient descent with acceleration techniques","authors":"Atsushi Nitanda","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Roux et al., 2012] Nicolas Le Roux, Mark Schmidt, and Francis Bach. A stochastic gradient method with an exponential conver- gence rate for finite training sets. In NIPS, pages 2672-2680, 2012.","title":"A stochastic gradient method with an exponential convergence rate for finite training sets","authors":"Nicolas Le Roux, Mark Schmidt, Francis Bach","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shalev-Shwartz and Zhang, 2013] Shai Shalev-Shwartz and Tong Zhang. Stochastic dual coordinate ascent methods for regularized loss minimization. J. Mach. Learn. Res., 14:567-599, 2013.","title":"Stochastic dual coordinate ascent methods for regularized loss minimization","authors":"Shalev-Shwartz Zhang, ; ] Shai Shalev-Shwartz, Tong Zhang","year":"2013","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shang et al., 2017] Fanhua Shang, Yuanyuan Liu, James Cheng, and Jiacheng Zhuo. Fast stochastic variance reduced gradien- t method with momentum acceleration for machine learning. arXiv:1703.07948, 2017.","title":"Fast stochastic variance reduced gradient method with momentum acceleration for machine learning","authors":"Fanhua Shang, Yuanyuan Liu, James Cheng, Jiacheng Zhuo","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shang et al., 2018a] Fanhua Shang, Licheng Jiao, Kaiwen Zhou, James Cheng, Yan Ren, and Yufei Jin. ASVRG: Accelerated proximal SVRG. In P. Mach. Learn. Res., pages 815-830, 2018.","title":"ASVRG: Accelerated proximal SVRG","authors":"Shang","year":"2018","venue":"P. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shang et al., 2018b] Fanhua Shang, Yuanyuan Liu, James Cheng, K. W. Ng, and Yuichi Yoshida. Guaranteed sufficient decrease for stochastic variance reduced gradient optimization. In AISTATS, pages 1027-1036, 2018.","title":"Guaranteed sufficient decrease for stochastic variance reduced gradient optimization","authors":"Shang","year":"2018","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shang et al., 2019] Fanhua Shang, Kaiwen Zhou, Hongying Li- u, James Cheng, Ivor Tsang, Lijun Zhang, Dacheng Tao, and Licheng Jiao. VR-SGD: A simple stochastic variance reduction method for machine learning. IEEE Trans. Knowl. Data Eng., 2019.","title":"VR-SGD: A simple stochastic variance reduction method for machine learning","authors":"Fanhua Shang, Kaiwen Zhou, Hongying Liu, James Cheng, Ivor Tsang, Lijun Zhang, Dacheng Tao, Licheng Jiao","year":"2019","venue":"IEEE Trans. Knowl. Data Eng","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tibshirani and Taylor, 2011] Ryan J. Tibshirani and Jonathan Taylor. The solution path of the generalized lasso. Annals of Statistics, 39(3):1335-1371, 2011.","title":"The solution path of the generalized lasso","authors":"Ryan J Tibshirani, Jonathan Taylor","year":"2011","venue":"Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Woodworth and Srebro, 2016] Blake Woodworth and Nathan S- rebro. Tight complexity bounds for optimizing composite ob- jectives. In NIPS, 2016.","title":"Tight complexity bounds for optimizing composite objectives","authors":"Srebro Woodworth, Blake Woodworth, Nathan Srebro","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xiao and Zhang, 2014] Lin Xiao and Tong Zhang. A proximal stochastic gradient method with progressive variance reduction. SIAM J. Optim., 24(4):2057-2075, 2014.","title":"A proximal stochastic gradient method with progressive variance reduction","authors":"Lin Xiao, Tong Zhang","year":"2014","venue":"SIAM J. Optim","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2013] Lijun Zhang, Mehrdad Mahdavi, and Rong Jin. Linear convergence with condition number independent access of full gradients. In NIPS, pages 980-988, 2013.","title":"Linear convergence with condition number independent access of full gradients","authors":"Lijun Zhang, Mehrdad Mahdavi, Rong Jin","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhou et al., 2018] Kaiwen Zhou, Fanhua Shang, and James Cheng. A simple stochastic variance reduced algorithm with fast convergence rates. In ICML, pages 5975-5984, 2018.","title":"A simple stochastic variance reduced algorithm with fast convergence rates","authors":"Kaiwen Zhou, Fanhua Shang, James Cheng","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou et al., 2019] Kaiwen Zhou, Qinghua Ding, Fanhua Shang, James Cheng, Danli Li, and Zhi-Quan Luo. Direct accelera- tion of SAGA using sampled negative momentum. In AISTATS, 2019.","title":"Direct acceleration of SAGA using sampled negative momentum","authors":"Kaiwen Zhou, Qinghua Ding, Fanhua Shang, James Cheng, Danli Li, Zhi-Quan Luo","year":"2019","venue":"AISTATS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}