Proceedings

Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves

Tobias Domhan, Jost Tobias Springenberg, Frank Hutter

Download BibTeX Cite

Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence. Special Track on Machine Learning. Pages 3460-3467.

Abstract

Deep neural networks (DNNs) show very strong performance on many machine learning problems, but they are very sensitive to the setting of their hyperparameters. Automated hyperparameter optimization methods have recently been shown to yield settings competitive with those found by human experts, but their widespread adoption is hampered by the fact that they require more computational resources than human experts. Humans have one advantage: when they evaluate a poor hyperparameter setting they can quickly detect (after a few SGD steps) that the resulting network performs poorly and terminate the corresponding evaluation to save time. Here, we mimic this early termination of bad runs based on a probabilistic model that extrapolates performance from the first part of a learning curve. Experiments with different neural network architectures show that our resulting approach speeds up state-of-the-art hyperparameter optimization methods for DNNs roughly twofold, enabling them to find DNN settings that yield better performance than those chosen by human experts.

Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. 2015. Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-15. IJCAI, 3460–3467. https://ijcai.org/proceedings/2015/487
Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards