{"type":"ScholarlyArticle","title":"Adaptive Learning Rate via Covariance Matrix Based Preconditioning for Deep Neural Networks","authors":[{"name":"Yasutoshi Ida","slug":"yasutoshi-ida","given":"Yasutoshi","family":"Ida","affiliation":"NTT Software Innovation Center , 3-9-11 Midori-cho Musashino-shi , Tokyo , 180-8585 , Japan"},{"name":"Yasuhiro Fujiwara","slug":"yasuhiro-fujiwara","given":"Yasuhiro","family":"Fujiwara","affiliation":"NTT Software Innovation Center , 3-9-11 Midori-cho Musashino-shi , Tokyo , 180-8585 , Japan"},{"name":"Sotetsu Iwamura","slug":"sotetsu-iwamura","given":"Sotetsu","family":"Iwamura","affiliation":"NTT Software Innovation Center , 3-9-11 Midori-cho Musashino-shi , Tokyo , 180-8585 , Japan"}],"abstract":"Adaptive learning rate algorithms such as RMSProp are widely used for training deep neural networks.\r\nRMSProp offers efficient training since it uses first order gradients to approximate Hessian-based preconditioning.\r\nHowever, since the first order gradients include noise caused by stochastic optimization, the approximation may be inaccurate.\r\nIn this paper, we propose a novel adaptive learning rate algorithm called SDProp.\r\nIts key idea is effective handling of the noise by preconditioning based on covariance matrix.\r\nFor various neural networks, our approach is more efficient and effective than RMSProp and its variant.","doi":"10.24963/ijcai.2017/267","url":"https://ijcai.org/proceedings/2017/267","pdf":"https://ijcai.org/proceedings/2017/0267.pdf","markdown":"https://ijcai.org/proceedings/2017/267.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/267","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":267,"pages":{"first":1923,"last":1929},"keywords":["Machine Learning: Machine Learning","Machine Learning: Deep Learning","Machine Learning: Neural Networks","Robotics and Vision: Vision and Perception"],"videos":[],"references":[{"position":1,"raw":"[Dauphin et al., 2014] Yann Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and Attacking the Saddle Point Problem in High-dimensional Non-convex Opti- mization. In NIPS, pages 2933-2941, 2014.","title":"Identifying and Attacking the Saddle Point Problem in High-dimensional Non-convex Optimization","authors":"Yann Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, Yoshua Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dauphin et al., 2015] Yann Dauphin, Harm de Vries, and Yoshua Bengio. Equilibrated Adaptive Learning Rates for Non-convex Optimization. In NIPS, pages 1504-1512, 2015.","title":"Equilibrated Adaptive Learning Rates for Non-convex Optimization","authors":"Yann Dauphin, Harm De Vries, Yoshua Bengio","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dekel et al., 2012] Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, and Lin Xiao. Optimal Distributed Online Pre- diction using Mini-batches. Journal of Machine Learning Research, 13:165-202, 2012.","title":"Optimal Distributed Online Prediction using Mini-batches","authors":"Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, Lin Xiao","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Elman, 1990] Jeffrey Elman. Finding Structure in Time. Cognitive Science, 14(2):179-211, 1990.","title":"Finding Structure in Time","authors":"Jeffrey Elman","year":"1990","venue":"Cognitive Science","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fujiwara et al., 2016a] Yasuhiro Fujiwara, Yasutoshi Ida, Junya Arai, Mai Nishimura, and Sotetsu Iwamura. Fast Algorithm for the Lasso based L1-Graph Construction. In proceedings of the Very Large Database Endow- ment(PVLDB), 10(3):229-240, 2016.","title":"Fast Algorithm for the Lasso based L1-Graph Construction","authors":"Fujiwara","year":"2016","venue":"proceedings of the Very Large Database Endowment(PVLDB)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fujiwara et al., 2016b] Yasuhiro Fujiwara, Yasutoshi Ida, Hiroaki Shiokawa, and Sotetsu Iwamura. Fast Lasso Al- gorithm via Selective Coordinate Descent. In AAAI, pages 1561-1567, 2016.","title":"Fast Lasso Algorithm via Selective Coordinate Descent","authors":"Fujiwara","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fukuda et al., 2014] Yukikatsu Fukuda, Yasutoshi Ida, Takashi Matsumoto, Naohiro Takemura, and Kaoru Sakatani. A Bayesian Algorithm for Anxiety Index Pre- diction based on Cerebral Blood Oxygenation in the Pre- frontal Cortex Measured by Near Infrared Spectroscopy. IEEE journal of translational engineering in health and medicine, 2:1-10, 2014.","title":"A Bayesian Algorithm for Anxiety Index Prediction based on Cerebral Blood Oxygenation in the Prefrontal Cortex Measured by Near Infrared Spectroscopy","authors":"Fukuda","year":"2014","venue":"IEEE journal of translational engineering in health and medicine","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Halko et al., 2011] Nathan Halko, Per-Gunnar Martinsson, and Joel Tropp. Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions. SIAM review, 53(2):217-288, 2011.","title":"Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions","authors":"Halko","year":"2011","venue":"SIAM review","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ida et al., 2013] Yasutoshi Ida, Takuma Nakamura, and Takashi Matsumoto. Domain-dependent/independent Topic Switching Model for Online Reviews with Numeri- cal Ratings. In CIKM, pages 229-238, 2013.","title":"Domain-dependent/independent Topic Switching Model for Online Reviews with Numerical Ratings","authors":"Yasutoshi Ida, Takuma Nakamura, Takashi Matsumoto","year":"2013","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Karpathy et al., 2015] Andrej Karpathy, Justin Johnson, and Fei-Fei Li. Visualizing and Understanding Recurrent Networks. arXiv preprint arXiv:1506.02078, 2015.","title":"Visualizing and Understanding Recurrent Networks","authors":"Andrej Karpathy, Justin Johnson, Fei-Fei Li","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. Inter- national Conference in Learning Representations (ICLR), 2014.","title":"Adam: A Method for Stochastic Optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":"International Conference in Learning Representations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning Multiple Layers of Features from Tiny Images, 2009.","title":"Learning Multiple Layers of Features from Tiny Images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based Learning Ap- plied to Document Recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based Learning Applied to Document Recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Miyashita et al., 2013] Hiroki Miyashita, Takuma Naka- mura, Yasutoshi Ida, Takashi Matsumoto, and Takashi Kaburagi. Nonparametric Bayes-based Heterogeneous Drosophila Melanogaster Gene Regulatory Network Infer- ence: T-process Regression. In International Conference on Artificial Intelligence and Applications, pages 51-58, 2013.","title":"Nonparametric Bayes-based Heterogeneous Drosophila Melanogaster Gene Regulatory Network Inference: T-process Regression","authors":"Hiroki Miyashita, Takuma Nakamura, Yasutoshi Ida, Takashi Matsumoto, Takashi Kaburagi","year":"2013","venue":"International Conference on Artificial Intelligence and Applications","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Neelakantan et al., 2015] Arvind Neelakantan, Luke Vilnis, Quoc V Le, Ilya Sutskever, Lukasz Kaiser, Karol Ku- rach, and James Martens. Adding Gradient Noise Im- proves Learning for Very Deep Networks. arXiv preprint arXiv:1511.06807, 2015.","title":"Adding Gradient Noise Improves Learning for Very Deep Networks","authors":"Arvind Neelakantan, Luke Vilnis, V Quoc, Ilya Le, Lukasz Sutskever, Karol Kaiser, James Kurach, Martens","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pascanu et al., 2013] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the Difficulty of Training Recurrent Neural Networks. In ICML, pages 1310-1318, 2013.","title":"On the Difficulty of Training Recurrent Neural Networks","authors":"Razvan Pascanu, Tomas Mikolov, Yoshua Bengio","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Robbins and Monro, 1951] Herbert Robbins and Sutton Monro. A Stochastic Approximation Method. The Annals of Mathematical Statistics, pages 400-407, 1951.","title":"A Stochastic Approximation Method","authors":"Monro Robbins, Herbert Robbins, Sutton Monro","year":"1951","venue":"The Annals of Mathematical Statistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sermanet et al., 2012] Pierre Sermanet, Sandhya Chintala, and Yann LeCun. Convolutional Neural Networks Applied to House Numbers Digit Classification. In International Conference on Pattern Recognition (ICPR), pages 3288- 3291, 2012.","title":"Convolutional Neural Networks Applied to House Numbers Digit Classification","authors":"Sermanet","year":"2012","venue":"International Conference on Pattern Recognition (ICPR)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sra et al., 2012] Suvrit Sra, Sebastian Nowozin, and Stephen Wright. Optimization for Machine Learning. Mit Press, 2012.","title":"Suvrit Sra, Sebastian Nowozin, and Stephen Wright. Optimization for Machine Learning","authors":"Sra","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tieleman and Hinton, 2012] Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the Gradient by a Running Average of its Recent Magnitude. COURSERA: Neural Networks for Machine Learning, 2012.","title":"Lecture 6.5-rmsprop: Divide the Gradient by a Running Average of its Recent Magnitude","authors":"Hinton Tieleman, Tijmen Tieleman, Geoffrey Hinton","year":"2012","venue":"COURSERA: Neural Networks for Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zeiler, 2012] Matthew Zeiler. ADADELTA: An Adaptive Learning Rate Method. arXiv preprint arXiv:1212.5701, 2012.","title":"ADADELTA: An Adaptive Learning Rate Method","authors":"Matthew Zeiler","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}