{"type":"ScholarlyArticle","title":"Nostalgic Adam: Weighting More of the Past Gradients When Designing the Adaptive Learning Rate","authors":[{"name":"Haiwen Huang","slug":"haiwen-huang","given":"Haiwen","family":"Huang","affiliation":"School of Mathematical Sciences , Peking University"},{"name":"Chang Wang","slug":"chang-wang","given":"Chang","family":"Wang","affiliation":"School of Mathematical Sciences , Peking University"},{"name":"Bin Dong","slug":"bin-dong","given":"Bin","family":"Dong","affiliation":"Beijing International Center for Mathematical Research , Peking University; Center for Data Science , Peking University; Laboratory for Biomedical Image Analysis , Beijing Institute of Big Data Research , Beijing"}],"abstract":"First-order optimization algorithms have been proven prominent in deep learning. In particu- lar, algorithms such as RMSProp and Adam are extremely popular. However, recent works have pointed out the lack of “long-term memory” in Adam-like algorithms, which could hamper their performance and lead to divergence. In our study, we observe that there are benefits of weighting more of the past gradients when designing the adaptive learning rate. We therefore propose an algorithm called the Nostalgic Adam (NosAdam) with theoretically guaranteed convergence at the best known convergence rate. NosAdam can be regarded as a fix to the non-convergence issue of Adam in alternative to the recent work of [Reddi et al., 2018]. Our preliminary numerical experiments show that NosAdam is a promising alternative al- gorithm to Adam. The proofs, code and other supplementary materials are already released.","doi":"10.24963/ijcai.2019/355","url":"https://ijcai.org/proceedings/2019/355","pdf":"https://ijcai.org/proceedings/2019/0355.pdf","markdown":"https://ijcai.org/proceedings/2019/355.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/355","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":355,"pages":{"first":2556,"last":2562},"keywords":["Computer Vision: Statistical Methods and Machine Learning","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Balles and Hennig, 2018] Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients. In Proceedings of the 35th Inter- national Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, pages 413-422, 2018.","title":"Dissecting adam: The sign, magnitude and variance of stochastic gradients","authors":"Hennig Balles, Lukas Balles, Philipp Hennig","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen and Gu, 2019] Jinghui Chen and Quanquan Gu. Padam: Closing the generalization gap of adaptive gradi- ent methods in training deep neural networks, 2019.","title":"Padam: Closing the generalization gap of adaptive gradient methods in training deep neural networks","authors":"Jinghui Chen, Quanquan Gu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Duchi et al., 2010] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learn- ing and stochastic optimization. Technical Report UCB/EECS-2010-24, EECS Department, University of California, Berkeley, Mar 2010.","title":"Adaptive subgradient methods for online learning and stochastic optimization","authors":"John Duchi, Elad Hazan, Yoram Singer","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CoRR, abs/1512.03385, 2015.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Keskar and Socher, 2017] Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from adam to SGD. CoRR, abs/1712.07628, 2017.","title":"Improving generalization performance by switching from adam to SGD","authors":"Nitish Shirish, Keskar, Richard Socher","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kingma and Ba, 2014] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"Krizhevsky et al., ] Alex Krizhevsky, Vinod Nair, and Ge- offrey Hinton. Cifar-10 (canadian institute for advanced research).","title":"Cifar-10","authors":"Krizhevsky","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"Y. Lecun. The mnist database of handwritten dig- its. http://yann.lecun.com/exdb/mnist/.","title":"The mnist database of handwritten digits","authors":"Y Lecun","year":null,"venue":null,"doi":null,"url":"http://yann.lecun.com/exdb/mnist/","internal_url":null},{"position":9,"raw":"[Li et al., 2018] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Neural Information Processing Systems, 2018.","title":"Visualizing the loss landscape of neural nets","authors":"Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, Tom Goldstein","year":"2018","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Luo et al., 2019] Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. In Proceedings of the 7th Interna- tional Conference on Learning Representations, New Or- leans, Louisiana, May 2019.","title":"Adaptive gradient methods with dynamic bound of learning rate","authors":"Luo","year":"2019","venue":"Proceedings of the 7th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Nesterov, 1983] Y. Nesterov. A method for unconstrained convex minimization problem with the rate of convergence o(1/k 2 ). Doklady AN USSR, 269:543-547, 1983.","title":"A method for unconstrained convex minimization problem with the rate of convergence o(1/k 2 )","authors":"Y Nesterov","year":"1983","venue":"Doklady AN USSR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Polyak, 1964] Boris Polyak. Some methods of speeding up the convergence of iteration methods. 4:1-17, 12 1964.","title":"Some methods of speeding up the convergence of iteration methods","authors":"Boris Polyak, Polyak","year":"1964","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Reddi et al., 2018] Sashank J. Reddi, Satyen Kale, and San- jiv Kumar. On the convergence of adam and beyond. In International Conference on Learning Representations, 2018.","title":"On the convergence of adam and beyond","authors":"J Sashank, Satyen Reddi, Sanjiv Kale, Kumar","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Tieleman and Hinton, 2012] T. Tieleman and G. Hinton. Lecture 6.5-RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.","title":"Lecture 6.5-RmsProp: Divide the gradient by a running average of its recent magnitude","authors":"T Tieleman, G Hinton","year":"2012","venue":"COURSERA: Neural Networks for Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Wilson et al., 2017] Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wal- lach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4148-4158.","title":"The marginal value of adaptive gradient methods in machine learning","authors":"Rebecca Ashia C Wilson, Mitchell Roelofs, Nati Stern, Benjamin Srebro, Recht","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"Curran Associates, Inc., 2017.","title":null,"authors":"Curran Associates, Inc","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Zagoruyko and Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. CoRR, abs/1605.07146, 2016.","title":"Wide residual networks","authors":"Sergey Zagoruyko, Nikos Komodakis","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Zeiler, 2012] Matthew D. Zeiler. ADADELTA: an adaptive learning rate method. CoRR, abs/1212.5701, 2012.","title":"ADADELTA: an adaptive learning rate method","authors":"D Matthew, Zeiler","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zhou et al., 2018] Zhiming Zhou, Qingru Zhang, Guan- song Lu, Hongwei Wang, Weinan Zhang, and Yong Yu. Adashift: Decorrelation and convergence of adaptive learning rate methods. CoRR, abs/1810.00143, 2018.","title":"Adashift: Decorrelation and convergence of adaptive learning rate methods","authors":"Zhiming Zhou, Qingru Zhang, Guansong Lu, Hongwei Wang, Weinan Zhang, Yong Yu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zinkevich, 2003] Martin Zinkevich. Online convex pro- gramming and generalized infinitesimal gradient ascent. ICML'03, pages 928-935. AAAI Press, 2003.","title":"Online convex programming and generalized infinitesimal gradient ascent","authors":"Martin Zinkevich","year":"2003","venue":"ICML'03","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}