{"type":"ScholarlyArticle","title":"pbSGD: Powered Stochastic Gradient Descent Methods for Accelerated Non-Convex Optimization","authors":[{"name":"Beitong Zhou","slug":"beitong-zhou","given":"Beitong","family":"Zhou","affiliation":"School of Artificial Intelligence and Automation , Huazhong University of Science and Technology"},{"name":"Jun Liu","slug":"jun-liu","given":"Jun","family":"Liu","affiliation":"Department of Applied Mathematics , University of Waterloo"},{"name":"Weigao Sun","slug":"weigao-sun","given":"Weigao","family":"Sun","affiliation":"School of Artificial Intelligence and Automation , Huazhong University of Science and Technology"},{"name":"Ruijuan Chen","slug":"ruijuan-chen","given":"Ruijuan","family":"Chen","affiliation":"School of Artificial Intelligence and Automation , Huazhong University of Science and Technology"},{"name":"Claire Tomlin","slug":"claire-tomlin","given":"Claire","family":"Tomlin","affiliation":"Department of Electrical Engineering and Computer Sciences , UC Berkeley"},{"name":"Ye Yuan","slug":"ye-yuan","given":"Ye","family":"Yuan","affiliation":"School of Artificial Intelligence and Automation , Huazhong University of Science and Technology"}],"abstract":"We propose a novel technique for improving the stochastic gradient descent (SGD) method to train deep networks, which we term pbSGD. The proposed pbSGD method simply raises the stochastic gradient to a certain power elementwise during iterations and introduces only one additional parameter, namely, the power exponent (when it equals to 1, pbSGD reduces to SGD). We further propose pbSGD with momentum, which we term pbSGDM. The main results of this paper present comprehensive experiments on popular deep learning models and benchmark datasets. Empirical results show that the proposed pbSGD and pbSGDM obtain faster initial training speed than adaptive gradient methods,  comparable generalization ability with SGD, and improved robustness to hyper-parameter selection and vanishing gradients. pbSGD is essentially a gradient modifier via a nonlinear transformation. As such, it is orthogonal and complementary to other techniques for accelerating gradient-based optimization such as learning rate schedules. Finally, we show convergence rate analysis for both pbSGD and pbSGDM methods. The theoretical rates of convergence match the best known theoretical rates of convergence for SGD and SGDM methods on nonconvex functions.","doi":"10.24963/ijcai.2020/451","url":"https://ijcai.org/proceedings/2020/451","pdf":"https://ijcai.org/proceedings/2020/0451.pdf","markdown":"https://ijcai.org/proceedings/2020/451.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/451","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":451,"pages":{"first":3258,"last":3266},"keywords":["Machine Learning: Deep Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24564/file_24564.m3u8","length_sec":291,"provider":"hls"}],"references":[{"position":1,"raw":"[Bello et al., 2017] Irwan Bello, Barret Zoph, Vijay Vasude- van, and Quoc V Le. Neural optimizer search with rein- forcement learning. In Proc. of ICML, pages 459-468, 2017.","title":"Neural optimizer search with reinforcement learning","authors":"Irwan Bello, Barret Zoph, Vijay Vasudevan, Quoc V Le","year":"2017","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bernstein et al., 2018] Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Anima Anandkumar. signSGD: Compressed optimisation for non-convex prob- lems. arXiv:1802.04434, 2018.","title":"signSGD: Compressed optimisation for non-convex problems","authors":"Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, Anima Anandkumar","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018] Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of a class of Adam-type algorithms for non-convex optimization. arXiv:1808.02941, 2018.","title":"On the convergence of a class of Adam-type algorithms for non-convex optimization","authors":"Chen","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Duchi et al., 2011] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121-2159, 2011.","title":"Adaptive subgradient methods for online learning and stochastic optimization","authors":"John Duchi, Elad Hazan, Yoram Singer","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fang et al., 2019] Cong Fang, Zhouchen Lin, and Tong Zhang. Sharp analysis for nonconvex SGD escaping from saddle points. arXiv:1902.00247, 2019.","title":"Sharp analysis for nonconvex SGD escaping from saddle points","authors":"Cong Fang, Zhouchen Lin, Tong Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ge et al., 2015] Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points-online stochastic gra- dient for tensor decomposition. In Proc. of COLT, pages 797-842, 2015.","title":"Escaping from saddle points-online stochastic gradient for tensor decomposition","authors":"Rong Ge, Furong Huang, Chi Jin, Yang Yuan","year":"2015","venue":"Proc. of COLT","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proc. of CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proc. of CVPR, pages 4700- 4708, 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jin et al., 2019] Chi Jin, Praneeth Netrapalli, Rong Ge, Sham M Kakade, and Michael I Jordan. Stochas- tic gradient descent escapes saddle points efficiently. arXiv:1902.04811, 2019.","title":"Stochastic gradient descent escapes saddle points efficiently","authors":"Jin","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Keskar et al., 2016] Nitish Shirish Keskar, Dheevatsa Mudi- gere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learn- ing: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.","title":"On large-batch training for deep learning: Generalization gap and sharp minima","authors":"Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak, Peter Tang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Proc. of ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Ben- gio, Patrick Haffner, et al. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Loshchilov and Hutter, 2016] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv:1608.03983, 2016.","title":"Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts","authors":"Hutter Loshchilov","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Luo et al., 2018] Liangchen Luo, Wenhao Huang, Qi Zeng, Zaiqing Nie, and Xu Sun. Learning personalized end-to- end goal-oriented dialog. arXiv:1811.04604, 2018.","title":"Learning personalized end-toend goal-oriented dialog","authors":"Luo","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mazumdar and Ratliff, 2018] Eric Mazumdar and Lillian J. Ratliff. On the convergence of gradient-based learning in continuous games. arXiv:1804.05464, 2018.","title":"On the convergence of gradient-based learning in continuous games","authors":"Ratliff Mazumdar, Eric Mazumdar, Lillian J Ratliff","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nesterov, 1983] Yurii E Nesterov. A method for solving the convex programming problem with convergence rate o(1/k 2 )). In Dokl. akad. nauk Sssr, volume 269, pages 543-547, 1983.","title":"A method for solving the convex programming problem with convergence rate o(1/k 2 ))","authors":"E Yurii, Nesterov","year":"1983","venue":"Dokl. akad. nauk Sssr","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Polyak, 1964] Boris T Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4(5):1-17, 1964.","title":"Some methods of speeding up the convergence of iteration methods","authors":"T Boris, Polyak","year":"1964","venue":"USSR Computational Mathematics and Mathematical Physics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Qian, 1999] Ning Qian. On the momentum term in gradient descent learning algorithms. Neural Networks, 12(1):145- 151, 1999.","title":"On the momentum term in gradient descent learning algorithms","authors":"Ning Qian","year":"1999","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Reddi et al., 2018] Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of Adam and beyond. In Proc. of ICLR, 2018.","title":"On the convergence of Adam and beyond","authors":"J Sashank, Satyen Reddi, Sanjiv Kale, Kumar","year":"2018","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Robbins and Monro, 1951] Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathematical Statistics, 22(3):400-407, 1951.","title":"Herbert Robbins and Sutton Monro. A stochastic approximation method","authors":"Monro Robbins","year":"1951","venue":"The Annals of Mathematical Statistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, et al. Imagenet large scale visual recognition challenge. Interna- tional Journal of Computer Vision, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng","year":"2015","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Smith and Topin, 2017] Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. arXiv:1708.07120, 2017.","title":"Super-convergence: Very fast training of neural networks using large learning rates","authors":"N Leslie, Nicholay Smith, Topin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tieleman and Hinton, 2012] Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-RMSProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 4(2):26-31, 2012.","title":"Lecture 6.5-RMSProp: Divide the gradient by a running average of its recent magnitude","authors":"Hinton Tieleman, Tijmen Tieleman, Geoffrey Hinton","year":"2012","venue":"COURSERA: Neural Networks for Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wilson et al., 2017] Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Proc. of NeurIPS, pages 4148-4158, 2017.","title":"The marginal value of adaptive gradient methods in machine learning","authors":"Rebecca Ashia C Wilson, Mitchell Roelofs, Nati Stern, Benjamin Srebro, Recht","year":"2017","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wu and He, 2018] Yuxin Wu and Kaiming He. Group nor- malization. In Proc. of ECCV, pages 3-19, 2018.","title":"Group normalization","authors":"He Wu, Yuxin Wu, Kaiming He","year":"2018","venue":"Proc. of ECCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xie et al., 2017] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proc. of CVPR, pages 1492-1500, 2017.","title":"Aggregated residual transformations for deep neural networks","authors":"Xie","year":"2017","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yan et al., 2018] Yan Yan, Tianbao Yang, Zhe Li, Qihang Lin, and Yi Yang. A unified analysis of stochastic momen- tum methods for deep learning. Proc. of IJCAI, 2018.","title":"A unified analysis of stochastic momentum methods for deep learning","authors":"Yan Yan, Tianbao Yang, Zhe Li, Qihang Lin, Yi Yang","year":"2018","venue":"Proc. of IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/410"},{"position":29,"raw":"[Yang et al., 2019] Tao Yang, Xinlei Yi, Junfeng Wu, Ye Yuan, Di Wu, Ziyang Meng, Yiguang Hong, Hong Wang, Zongli Lin, and Karl H. Johansson. A survey of distributed optimization. Annual Reviews in Control, 47:278 -305, 2019.","title":"A survey of distributed optimization","authors":"Tao Yang, Xinlei Yi, Junfeng Wu, Ye Yuan, Di Wu, Ziyang Meng, Yiguang Hong, Hong Wang, Zongli Lin, Karl H Johansson","year":"2019","venue":"Annual Reviews in Control","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yuan et al., 2019] Ye Yuan, Mu Li, Jun Liu, and Claire Tom- lin. On the Powerball method: variants of descent methods for accelerated optimization. IEEE Control Systems Letters, 3(3):601-606, 2019.","title":"On the Powerball method: variants of descent methods for accelerated optimization","authors":"Ye Yuan, Mu Li, Jun Liu, Claire Tomlin","year":"2019","venue":"IEEE Control Systems Letters","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zagoruyko and Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv:1605.07146, 2016.","title":"Wide residual networks","authors":"Sergey Zagoruyko, Nikos Komodakis","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}