{"type":"ScholarlyArticle","title":"Taming the Noisy Gradient: Train Deep Neural Networks with Small Batch Sizes","authors":[{"name":"Yikai Zhang","slug":"yikai-zhang","given":"Yikai","family":"Zhang","affiliation":"Department of Computer Science , Rutgers University"},{"name":"Hui Qu","slug":"hui-qu","given":"Hui","family":"Qu","affiliation":"Department of Computer Science , Rutgers University"},{"name":"Chao Chen","slug":"chao-chen","given":"Chao","family":"Chen","affiliation":"Departments of Biomedical Informatics , Stony Brook University"},{"name":"Dimitris Metaxas","slug":"dimitris-metaxas","given":"Dimitris","family":"Metaxas","affiliation":"Department of Computer Science , Rutgers University"}],"abstract":"Deep learning architectures are usually proposed with millions of parameters, resulting in a memory issue when training deep neural networks with stochastic gradient descent type methods using large batch sizes. However, training with small batch sizes tends to produce low quality solution due to the large variance of stochastic gradients. In this paper, we tackle this problem by proposing a new framework for training deep neural network with small batches/noisy gradient. During optimization, our method iteratively applies a proximal type regularizer to make loss function strongly convex. Such regularizer stablizes the gradient, leading to better training performance. We prove that our algorithm achieves comparable convergence rate as vanilla SGD even with small batch size. Our framework is simple to implement and can be potentially combined with many existing optimization algorithms. Empirical results show that our method outperforms SGD and Adam when batch size is small.  Our implementation is available at  https://github.com/huiqu18/TRAlgorithm.","doi":"10.24963/ijcai.2019/604","url":"https://ijcai.org/proceedings/2019/604","pdf":"https://ijcai.org/proceedings/2019/0604.pdf","markdown":"https://ijcai.org/proceedings/2019/604.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/604","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":604,"pages":{"first":4348,"last":4354},"keywords":["Machine Learning: Deep Learning","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Allen-Zhu, 2018] Zeyuan Allen-Zhu. How to make the gra- dients small stochastically: Even faster convex and non- convex sgd. In Advances in Neural Information Process- ing Systems, pages 1165-1175, 2018.","title":"How to make the gradients small stochastically: Even faster convex and nonconvex sgd","authors":"Allen-Zhu ; Zeyuan Allen-Zhu","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bottou, 2010] Léon Bottou. Large-scale machine learn- ing with stochastic gradient descent. In Proceedings of COMPSTAT'2010, pages 177-186. Springer, 2010.","title":"Large-scale machine learning with stochastic gradient descent","authors":"Léon Bottou, Bottou","year":"2010","venue":"Proceedings of COMPSTAT'2010","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bottou, 2012] Léon Bottou. Stochastic gradient descent tricks. In Neural networks: Tricks of the trade, pages 421- 436. Springer, 2012.","title":"Stochastic gradient descent tricks","authors":"Léon Bottou, Bottou","year":"2012","venue":"Neural networks: Tricks of the trade","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chaudhari et al., 2017] Pratik Chaudhari, Anna Choroman- ska, S. Soatto, Yann LeCun, C. Baldassi, C. Borgs, J. Chayes, Levent Sagun, and R. Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. In Interna- tional Conference on Learning Representations (ICLR), 2017.","title":"Entropy-sgd: Biasing gradient descent into wide valleys","authors":"Pratik Chaudhari, Anna Choromanska, S Soatto, C Yann Lecun, C Baldassi, J Borgs, Levent Chayes, R Sagun, Zecchina","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cordts et al., 2016] Marius Cordts, Mohamed Omran, Se- bastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene under- standing. In CVPR, pages 3213-3223, 2016.","title":"The cityscapes dataset for semantic urban scene understanding","authors":"Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, Bernt Schiele","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ge et al., 2015] Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points-online stochastic gra- dient for tensor decomposition. In Conference on Learning Theory, pages 797-842, 2015.","title":"Escaping from saddle points-online stochastic gradient for tensor decomposition","authors":"Rong Ge, Furong Huang, Chi Jin, Yang Yuan","year":"2015","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hardt et al., 2016] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochas- tic gradient descent. In International Conference on Ma- chine Learning, pages 1225-1234, 2016.","title":"Train faster, generalize better: Stability of stochastic gradient descent","authors":"Moritz Hardt, Ben Recht, Yoram Singer","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hoerl and Kennard, 1970] Arthur E Hoerl and Robert W Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55-67, 1970.","title":"Ridge regression: Biased estimation for nonorthogonal problems","authors":"E Arthur, Robert W Hoerl, Kennard","year":"1970","venue":"Technometrics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 4700-4708, 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Isola et al., 2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with con- ditional adversarial networks. In CVPR, pages 1125-1134, 2017.","title":"Image-to-image translation with conditional adversarial networks","authors":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A Efros","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jin et al., 2017] Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M Kakade, and Michael I Jordan. How to escape saddle points efficiently. In Proceedings of the 34th In- ternational Conference on Machine Learning-Volume 70, pages 1724-1732. JMLR. org, 2017.","title":"How to escape saddle points efficiently","authors":"Jin","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Johnson and Zhang, 2013] Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in neural information pro- cessing systems, pages 315-323, 2013.","title":"Accelerating stochastic gradient descent using predictive variance reduction","authors":"Rie Johnson, Tong Zhang","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Le et al., 2011] Quoc V Le, Jiquan Ngiam, Adam Coates, Abhik Lahiri, Bobby Prochnow, and Andrew Y Ng. On optimization methods for deep learning. In Proceedings of the 28th International Conference on International Con- ference on Machine Learning, pages 265-272. Omnipress, 2011.","title":"On optimization methods for deep learning","authors":"Jiquan Quoc V Le, Adam Ngiam, Abhik Coates, Bobby Lahiri, Andrew Y Prochnow, Ng","year":"2011","venue":"Proceedings of the 28th International Conference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lei and Jordan, 2017] Lihua Lei and Michael Jordan. Less than a single pass: Stochastically controlled stochastic gra- dient. In Artificial Intelligence and Statistics, pages 148- 156, 2017.","title":"Less than a single pass: Stochastically controlled stochastic gradient","authors":"Jordan ; Lei, Lihua Lei, Michael Jordan","year":"2017","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2015] Hongzhou Lin, Julien Mairal, and Zaid Harchaoui. A universal catalyst for first-order optimiza- tion. In Advances in Neural Information Processing Sys- tems, pages 3384-3392, 2015.","title":"A universal catalyst for first-order optimization","authors":"Hongzhou Lin, Julien Mairal, Zaid Harchaoui","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In CVPR, pages 3431-3440, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Jonathan Long, Evan Shelhamer, Trevor Darrell","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Parikh et al., 2014] Neal Parikh, Stephen Boyd, et al. Prox- imal algorithms. Foundations and Trends R in Optimiza- tion, 1(3):127-239, 2014.","title":"Proximal algorithms","authors":"Neal Parikh, Stephen Boyd","year":"2014","venue":"Foundations and Trends R in Optimization","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rakhlin et al., 2012] Alexander Rakhlin, Ohad Shamir, and Karthik Sridharan. Making gradient descent optimal for strongly convex stochastic optimization. In Proceedings of the 29th International Coference on International Confer- ence on Machine Learning, pages 1571-1578. Omnipress, 2012.","title":"Making gradient descent optimal for strongly convex stochastic optimization","authors":"Alexander Rakhlin, Ohad Shamir, Karthik Sridharan","year":"2012","venue":"Proceedings of the 29th International Coference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Reddi et al., 2016] Sashank J Reddi, Ahmed Hefny, Suvrit Sra, Barnabas Poczos, and Alex Smola. Stochastic vari- ance reduction for nonconvex optimization. In Interna- tional conference on machine learning, pages 314-323, 2016.","title":"Stochastic variance reduction for nonconvex optimization","authors":"J Sashank, Ahmed Reddi, Suvrit Hefny, Barnabas Sra, Alex Poczos, Smola","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Reddi et al., 2018] Sashank J Reddi, Satyen Kale, and San- jiv Kumar. On the convergence of adam and beyond. In ICLR, 2018.","title":"On the convergence of adam and beyond","authors":"J Sashank, Satyen Reddi, Sanjiv Kale, Kumar","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ruder, 2016] Sebastian Ruder. An overview of gradi- ent descent optimization algorithms. arXiv preprint arXiv:1609.04747, 2016.","title":"An overview of gradient descent optimization algorithms","authors":"Sebastian Ruder","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shamir and Zhang, 2013] Ohad Shamir and Tong Zhang. Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes. In International Conference on Machine Learning, pages 71- 79, 2013.","title":"Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes","authors":"Zhang ; ] Ohad Shamir, Tong Shamir, Zhang","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2018] Ting-Chun Wang, Ming-Yu Liu, Jun- Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipula- tion with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8798-8807, 2018.","title":"High-resolution image synthesis and semantic manipulation with conditional gans","authors":"Ting-Chun, Ming-Yu Wang, Jun-Yan Liu, Andrew Zhu, Jan Tao, Bryan Kautz, Catanzaro","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}