{"type":"ScholarlyArticle","title":"Marthe: Scheduling the Learning Rate Via Online Hypergradients","authors":[{"name":"Michele Donini","slug":"michele-donini","given":"Michele","family":"Donini","affiliation":""},{"name":"Luca Franceschi","slug":"luca-franceschi","given":"Luca","family":"Franceschi","affiliation":"University College London , London , UK; Istituto Italiano di Tecnologia , Genova , Italy"},{"name":"Orchid Majumder","slug":"orchid-majumder","given":"Orchid","family":"Majumder","affiliation":"University College London , London , UK; Istituto Italiano di Tecnologia , Genova , Italy"},{"name":"Massimiliano Pontil","slug":"massimiliano-pontil","given":"Massimiliano","family":"Pontil","affiliation":"Università di Firenze , Firenze , Italy"},{"name":"Paolo Frasconi","slug":"paolo-frasconi","given":"Paolo","family":"Frasconi","affiliation":""}],"abstract":"We study the problem of fitting task-specific learning rate schedules from the perspective of hyperparameter optimization, aiming at good generalization. We describe the structure of the gradient of a validation error w.r.t. the learning rate schedule -- the hypergradient. Based on this, we introduce MARTHE, a novel online algorithm guided by cheap approximations of the hypergradient that uses past information from the optimization trajectory to simulate future behaviour. It interpolates between two recent techniques, RTHO (Franceschi et al., 2017) and HD (Baydin et al. 2018), and is able to produce learning rate schedules that are more stable leading to models that generalize better.","doi":"10.24963/ijcai.2020/293","url":"https://ijcai.org/proceedings/2020/293","pdf":"https://ijcai.org/proceedings/2020/0293.pdf","markdown":"https://ijcai.org/proceedings/2020/293.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/293","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":293,"pages":{"first":2119,"last":2125},"keywords":["Machine Learning: Deep Learning","Machine Learning: Online Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25032/file_25032.m3u8","length_sec":292,"provider":"hls"}],"references":[{"position":1,"raw":"[Agarwal et al., 2017] N. Agarwal, B. Bullins, and E. Hazan. Second-order stochastic optimization for machine learning in linear time. Journal of Machine Learning Research, 18(1):4148-4187, 2017.","title":"Second-order stochastic optimization for machine learning in linear time","authors":"N Agarwal, B Bullins, E Hazan","year":"2017","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Almeida et al., 1999] L.B. Almeida, T. Langlois, J.D. Ama- ral, and A. Plakhov. Parameter adaptation in stochastic op- timization. In On-line learning in neural networks, pages 111-134. Cambridge University Press, 1999.","title":"Parameter adaptation in stochastic optimization","authors":"L B Almeida, T Langlois, J D Amaral, A Plakhov","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Andrychowicz et al., 2016] M. Andrychowicz, M. Denil, S. Gomez, M.W. Hoffman, D. Pfau, T. Schaul, and N. de Freitas. Learning to learn by gradient descent by gradient descent. In NeurIPS, pages 3981-3989, 2016.","title":"Learning to learn by gradient descent by gradient descent","authors":"M Andrychowicz, M Denil, S Gomez, M W Hoffman, D Pfau, T Schaul, N De Freitas","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Baydin et al., 2018] A.G. Baydin, R. Cornish, D. Martínez- Rubio, M. Schmidt, and F.D. Wood. Online learning rate adaptation with hypergradient descent. In ICLR, 2018.","title":"Online learning rate adaptation with hypergradient descent","authors":"A G Baydin, R Cornish, D Martínez-Rubio, M Schmidt, F D Wood","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Beatson and Adams, 2019] A. Beatson and R.P. Adams. Ef- ficient optimization of loops and limits with randomized telescoping sums. ICML, 2019.","title":"Efficient optimization of loops and limits with randomized telescoping sums","authors":"A Beatson, R P Adams","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bengio et al., 1994] Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Trans. on Neural Networks, 5(2):157-166, 1994.","title":"Learning long-term dependencies with gradient descent is difficult","authors":"Y Bengio, P Simard, P Frasconi","year":"1994","venue":"IEEE Trans. on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bengio, 2012] Y. Bengio. Practical recommendations for gradient-based training of deep architectures. In Neural networks: Tricks of the trade, pages 437-478. Springer, 2012.","title":"Practical recommendations for gradient-based training of deep architectures","authors":"Y Bengio","year":"2012","venue":"Neural networks: Tricks of the trade","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Franceschi et al., 2017] L. Franceschi, M. Donini, P. Fras- coni, and M. Pontil. Forward and reverse gradient-based hyperparameter optimization. In ICML, pages 1165-1173, 2017.","title":"Forward and reverse gradient-based hyperparameter optimization","authors":"L Franceschi, M Donini, P Frasconi, M Pontil","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Franceschi et al., 2018] L. Franceschi, P. Frasconi, S. Salzo, and M. Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In ICML, pages 1568- 1577, 2018.","title":"Bilevel programming for hyperparameter optimization and meta-learning","authors":"L Franceschi, P Frasconi, S Salzo, M Pontil","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Glorot and Bengio, 2010] X. Glorot and Y. Bengio. Under- standing the difficulty of training deep feedforward neural networks. In AISTATS, pages 249-256, 2010.","title":"Understanding the difficulty of training deep feedforward neural networks","authors":"Bengio Glorot, X Glorot, Y Bengio","year":"2010","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Griewank and Faure, 2002] A. Griewank and C. Faure. Re- duced functions, gradients and hessians from fixed-point iterations for state equations. Numerical Algorithms, 30(2):113-139, 2002.","title":"Reduced functions, gradients and hessians from fixed-point iterations for state equations","authors":"A Griewank, C Faure","year":"2002","venue":"Numerical Algorithms","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Griewank and Walther, 2008] A. Griewank and A. Walther. Evaluating derivatives: principles and techniques of algo- rithmic differentiation, volume 105. Siam, 2008.","title":"Evaluating derivatives: principles and techniques of algorithmic differentiation","authors":"A Griewank, A Walther","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2016] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, June 2016.","title":"Deep residual learning for image recognition","authors":"K He, X Zhang, S Ren, J Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hutter et al., 2019] F. Hutter, L. Kotthoff, and J. Van- schoren. Automatic Machine Learning: Methods, Systems, Challenges. Springer, 2019.","title":"Vanschoren","authors":"F Hutter, L Kotthoff, J","year":"2019","venue":"Automatic Machine Learning: Methods, Systems, Challenges","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ioffe and Szegedy, 2015] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by re- ducing internal covariate shift. In ICML, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"S Ioffe, C Szegedy","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Krizhevsky and Hinton, 2009] A. Krizhevsky and G. Hin- ton. Learning multiple layers of features from tiny images. Technical report, Univ. Toronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"A Krizhevsky, G Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[LeCun et al., 1998] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, et al. Gradient-based learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278- 2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Y Lecun, L Bottou, Y Bengio, P Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Loshchilov and Hutter, 2017] I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. ICLR, 2017.","title":"Sgdr: Stochastic gradient descent with warm restarts","authors":"Hutter Loshchilov, I Loshchilov, F Hutter","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Luketina et al., 2016] J. Luketina, M. Berglund, K. Greff, and T. Raiko. Scalable gradient-based tuning of con- tinuous regularization hyperparameters. In ICML, pages 2952-2960, 2016.","title":"Scalable gradient-based tuning of continuous regularization hyperparameters","authors":"J Luketina, M Berglund, K Greff, T Raiko","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[MacKay et al., 2019] M. MacKay, P. Vicol, J. Lorraine, D. Duvenaud, and R. Grosse. Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions. AISTATS, 2019.","title":"Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions","authors":"M Mackay, P Vicol, J Lorraine, D Duvenaud, R Grosse","year":"2019","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Maclaurin et al., 2015] D. Maclaurin, D. Duvenaud, and R.P. Adams. Gradient-based hyperparameter optimization through reversible learning. In ICML, 2015.","title":"Gradient-based hyperparameter optimization through reversible learning","authors":"D Maclaurin, D Duvenaud, R P Adams","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Orabona and Pál, 2016] F. Orabona and D. Pál. Coin bet- ting and parameter-free online learning. In NeurIPS, pages 577-585, 2016.","title":"Coin betting and parameter-free online learning","authors":"F Orabona, D Pál","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pearlmutter, 1994] B.A. Pearlmutter. Fast exact multiplica- tion by the hessian. Neural computation, 6(1):147-160, 1994.","title":"Fast exact multiplication by the hessian","authors":"B A Pearlmutter","year":"1994","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pedregosa, 2016] F. Pedregosa. Hyperparameter optimiza- tion with approximate gradient. In ICML, 2016.","title":"Hyperparameter optimization with approximate gradient","authors":"F Pedregosa","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Pineda, 1988] F.J. Pineda. Generalization of back propaga- tion to recurrent and higher order neural networks. In Neu- ral information processing systems, pages 602-611, 1988.","title":"Generalization of back propagation to recurrent and higher order neural networks","authors":"F J Pineda","year":"1988","venue":"Neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Schaul et al., 2013] T. Schaul, S. Zhang, and Y. LeCun. No more pesky learning rates. In ICML, pages 343-351, 2013.","title":"No more pesky learning rates","authors":"T Schaul, S Zhang, Y Lecun","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Schraudolph, 1999] N.N. Schraudolph. Local gain adapta- tion in stochastic gradient descent. In ICANN, volume 2, pages 569-574. IET, 1999.","title":"Local gain adaptation in stochastic gradient descent","authors":"N N Schraudolph","year":"1999","venue":"ICANN","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Simonyan and Zisserman, 2014] K. Simonyan and A. Zis- serman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"K Simonyan, A Zisserman","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wichrowska et al., 2017] O. Wichrowska, N. Mah- eswaranathan, M.W. Hoffman, S.G. Colmenarejo, M. Denil, N. Freitas, and J. Sohl-Dickstein. Learned optimizers that scale and generalize. In ICML, pages 3751-3760, 2017.","title":"Learned optimizers that scale and generalize","authors":"O Wichrowska, N Maheswaranathan, M W Hoffman, S G Colmenarejo, M Denil, N Freitas, J Sohl-Dickstein","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wu et al., 2018] Y. Wu, M. Ren, R. Liao, and R. Grosse. Understanding short-horizon bias in stochastic meta- optimization. In ICLR, 2018.","title":"Understanding short-horizon bias in stochastic metaoptimization","authors":"Y Wu, M Ren, R Liao, R Grosse","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}