Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves

1 Introduction

Deep neural networks (DNNs) trained via backpropagation currently constitute the state-of-the-art for many classification problems, such as object recognition from images [Krizhevsky et al., 2012; Donahue et al., 2014] or speech recognition from audio data (see [Deng et al., 2013] for a recent review). Unfortunately, they are also very sensitive to the setting of their hyperparameters [Montavon et al., 2012]. While good settings are hard to find by non-experts, automatic hyperparameter optimization methods have recently been shown to yield performance competitive with human experts, and in some cases even surpassed them [Bergstra et al., 2011; Snoek et al., 2012; Dahl et al., 2013; Bergstra et al., 2013].

However, fitting large DNNs is computationally expensive and the time overhead of automated hyperparameter optimization hampers its widespread adoption. Instead, many human deep learning experts still perform manual hyperparameter search, relying on a "bag of tricks" to determine model hyperparameters and learning rates for stochastic gradient descent (SGD) [Montavon et al., 2012]. Using this acquired knowledge they can often tell after a few SGD steps whether the training procedure will converge to a model with competitive performance or not. To save time, they then prematurely terminate runs expected to perform poorly, allowing them to make more rapid progress than automated methods (which train even poor models until the end).

In this work, we mimic this early termination of bad runs with the help of a probabilistic model that extrapolates performance from the first part of a learning curve to its remainder, enabling us to automatically identify and terminate bad runs to save time. After discussing related work on hyperparameter optimization and studies of learning curves (Section 2), we introduce our probabilistic approach for extrapolating learning curves and show how to use it to devise a predictive termination criterion that can be readily combined with any hyperparameter optimization method (Section 3). Experiments with different neural network architectures on the prominent object recognition benchmarks CIFAR-10, CIFAR-100 and MNIST show that predictive termination speeds up current hyperparameter optimization methods for DNNs by roughly a factor of two, enabling them to find DNN settings that yield better performance than those chosen by human experts (Section 4).

2 Foundations and RelatedWork

We first review modern hyperparameter optimization methods and previous attempts to model learning curves.

2.1 Hyperparameter OptimizationMethods

Given a machine learning algorithm A having hyperparameters λ1, . . . , λn with respective domains Λ1, . . . , Λn, we define its hyperparameter space as Λ = Λ1 × · · · × Λn. For each hyperparameter setting λ ∈ Λ, we use Aλ to denote the learning algorithm A using this setting. We further use l(λ) = L(Aλ,Dtrain,Dvalid) to denote the validation loss (e.g., misclassification rate) that Aλ achieves on data Dvalid when trained on Dtrain. The hyperparameter optimization problem is then to find λ ∈ Λ minimizing l(λ).

For decades, the de-facto standard for hyperparameter optimization in machine learning has been a simple grid search.

Other approaches proposed over the years include racing algorithms [Maron and Moore, 1994] and gradient search [Bengio, 2000]. Recently, it has been shown that a simple random search can perform much better than grid search, particularly for high-dimensional problems with low intrinsic dimensionality [Bergstra and Bengio, 2012]. More sophisticated Bayesian optimization methods perform even better and have yielded new state-of-the-art results for several datasets [Bergstra et al., 2011; Hutter et al., 2011; Snoek et al., 2012; Bergstra et al., 2013].

Bayesian Optimization (see, e.g., [Jones et al., 1998; Brochu et al., 2010]) constructs a probabilistic model M of f based on point evaluations of f and any available prior information, uses model M to select subsequent configurations λ to evaluate, updates M based on the new measured performance at λ, and iterates.

The three most popular implementations of Bayesian optimization are Spearmint [Snoek et al., 2012], which uses a Gaussian process (GP) [Rasmussen and Williams, 2006] model for M; SMAC [Hutter et al., 2011], which uses random forests [Breiman, 2001] modified to yield an uncertainty estimate [Hutter et al., 2014]; and the Tree Parzen Estimator (TPE) [Bergstra et al., 2011], which constructs a density estimate over good and bad instantiations of each hyperparameter to build M. Eggensperger et al. [2013] empirically compared these three systems, concluding that Spearmint's GP-based approach performs best for problems with few numerical (and no other) hyperparameters, and that SMAC's and TPE's tree-based approach performs best for high-dimensional and partly discrete hyperparameter optimization problems, as they occur in optimizing DNNs. We therefore use SMAC and TPE in this study.

2.2 Modeling Learning Curves

The term learning curve appears in the literature for describing two different phenomena: (1) the performance of an iterative machine learning algorithm as a function of its training time or number of iterations; and (2) the performance of a machine learning algorithm as a function of the size of the dataset it has available for training. While our work concerns learning curves of type 1, we describe related work on modelling both types of learning curves.

Learning curves of type 1 are very popular for visualizing the concept of overfitting: while performance on the training set tends to improve over time, test performance often degrades eventually. The study of these learning curves has led to early stopping heuristics aiming to terminate training before overfitting occurs (see, e.g., [Yao et al., 2007]). We note that the goal behind our new predictive termination criterion is different: we predict validation performance and terminate a run when it is unlikely to beat the performance of the best model we have encountered so far.

In parallel to our work, Swersky et al. [2014] devised a GPbased Bayesian optimization method that includes a learning curve model. They used this model for temporarily pausing the training of machine learning models, in order to explore several promising hyperparameter configurations for a short time and resume training on the most promising models later on. Swersky et al. [2014] successfully applied this technique to matrix factorization, online Latent Dirichlet Allocation (LDA) and logistic regression. However, so far it does not work well for deep neural networks, possibly since it is limited to one particular parametric learning curve model that may not describe learning curves of deep networks well. 1

Learning curves of type 2 have been studied to extrapolate performance from smaller to larger datasets. In early work, Frey and Fisher [1999] estimated the amount of data needed by a decision tree to achieve a desired accuracy using linear, logarithmic, exponential and power law parametric models. Subsequent work predicted the performance of multiple machine learning algorithms using a total of 6 parametric models: a power law model with two and three parameters, a logarithmic model, the vapor pressure model, the MorganMercer-Flodin (MMF) model, and the Weibull model [Gu et al., 2001]. More recently, e.g., Kolachina et al. [2012] predicted how a statistical machine translation system would perform if more data was available; they used 6 parametric models and concluded that the three parameter power law is most suitable for their task.

All these approaches for extrapolating learning curves have in common that they use maximum likelihood fits of each parametric model by itself. In contrast to the probabilistic approach we propose in this work, the curve models are thus neither combined to increase their representative power nor do they account for uncertainty in the data and model parameters.

3 Extrapolation of Learning Curves

In this paper, we focus on learning curves that describe the performance of DNNs as a function of the number of stochastic gradient descent (SGD) steps. We measure performance as classification accuracy on a validation set.

3.1 Learning Curve Model

In this section, we describe how we probabilistically extrapolate from a short initial portion of a learning curve to a later point. When running SGD on DNNs we measure validation performance in regular intervals. Let y1:n denote the observed performance values for the first n intervals. In our problem setup, we observe y1:n and aim to predict performance ym after a large number of intervals m ≫ n. We solve this problem using a probabilistic model.

Parametric Learning Curve Models

Our basic approach is to model the partially observed learning curve y1:n by a set of parametric model families {f1, . . . , fK}. Each of these parametric functions fk is described through a set of parameters θk. Assuming additive Gaussian noise ϵ ∼ N(0, σ2 ), we can use each fk to model performance at time step t as yt = fk(t|θ)+ϵ; the probability of a single observation yt under model fk is hence given as

We chose a large set of parametric curve models from the literature whose shape coincides with our prior knowledge about the form of learning curves: They are typically increasing, saturating functions; for example functions from the power law or the sigmoidal family. In total we considered K = 11 different model families; Figure 1 shows an example for how each of these functions would model a typical learning curve and also provides their parametric formulas. We note that all of these models capture certain aspects of learning curves, but that no single model can describe all learning curves by itself, motivating us to combine the models in a probabilistic framework.

1 Based on personal communication with the authors.

A Weighted Probabilistic Learning Curve Model

Instead of selecting an individual model we combine all K models into a single, more powerful, model. This combined model is given by a weighted linear combination:

where the new combined parameter vector

comprises a weight wk for each model, the individual model parameters θk, and the noise variance σ2 , and yt = fcomb(t|ξ) + ϵ.

Given this model, a simple approach would be to find a maximum likelihood estimate for all parameters. However, this would not properly model the uncertainty in the model parameters. Since our predictive termination criterion aims at only terminating runs that are highly unlikely to improve on the best run observed so far we need to model uncertainty as truthfully as possible and will hence adopt a Bayesian perspective, predicting values ym using Markov Chain Monte Carlo (MCMC) inference.

To enable such probabilistic inference we also need to place a prior probability on all parameters. It would be simplest to choose an uninformative prior, such that

However, with such an uninformative prior we put positive probability mass on parameterizations that yield learning curves which decrease after some time. To avoid this situation, we explicitly encode our knowledge into the prior that well-behaved learning curves are increasing saturating functions. We also restrict the weights to be positive only, allowing us to interpret each individual model as a non-negative additive component of a learning curve. This has the nice side effect that the predicted curve will only be flat once all models have flattened out.

Concretely, we define a new prior distribution over ξ, which encodes the above intuition, as

and for all k

The p(θk) and p(σ2 ) are still set to be uninformative, but are mentioned for the sake of completeness. The indicator function ✶(fcomb(1|ξ) < fcomb(m|ξ)) ensures that no pathological model that decreases from the initial value to the point of prediction m gets any probability mass. Finally, the prior on the weights p(wk) ensures weights are never negative. Figure 2 visualizes the problem this modified prior solves: while the uninformative prior yields learning curves that decrease over time (Figure 2a), our new prior yields increasing saturating curves (Figure 2b).

With these definitions in place we can finally perform MCMC sampling over the joint parameter and weight space ξ by drawing S samples ξ1, . . . , ξS from the posterior

where P(y1:n|ξ) for the model combination is given as P(y1:n|ξ) = Πnt=1N(yt; fcomb(t|ξ), σ2 ). To initialize the sampling procedure, we set all model parameters θk to their (per-model) maximum likelihood estimate. The model weights are initialized uniformly, that is wk = 1K . The noise parameter is also initialized to its maximum likelihood esti-

A sample approximation for ym with m > n can then be formed as

More importantly, since we also have an estimate of the parameter uncertainty and the predictive distribution P (ym|y1:n, ξ) is a Gaussian for each fixed ξ, we can estimate the probability that ym exceeds a certain value ˆ y as

where Φ(·; µ, σ2 ) is the cumulative distribution function of the Gaussian with mean µ and variance σ2 .

We note that the entire learning curve extrapolation process is robust and fully automated, with MCMC sampling taking care of all free parameters.

3.2 Speeding up Hyperparameter Optimization

We use our predictive models to speed up hyperparameter optimizers as follows. Firstly, while the hyperparameter optimizer is running we keep track of the best performance ˆ y found so far (we initialize ˆ y to -∞). Each time the optimizer queries the performance l(λ) of a hyperparameter setting λ we train a DNN using λ as usual, except that we terminate this run early if our extrapolation model predicts the network to eventually yield worse performance than ˆ y. More precisely, at regular intervals i during SGD training we measure and save validation set performance yi. There are emax epochs, k such intervals per epoch, and every p epochs, we gather the performance values y1:n of the n intervals so far and run MCMC to probabilistically extrapolate performance to the final step m = k×emax. We then consider the predicted probability P(ym ≥ ˆ y|y1:n) that the network, after training for m intervals, will exceed the performance ˆ y. If this probability is above a threshold δ then training continues as usual for the next p epochs. Otherwise, training is terminated and we return the expected validation error 1-E[ym|y1:n] (where E[ym|y1:n] is the expected accuracy from Equation 8) to the hyperparameter optimizer in lieu of the real (yet unknown) loss. We dub this procedure the predictive termination criterion. It is agnostic to the precise hyperparameter optimizer used and we will evaluate its performance using two different state-of-the-art optimizers.

We also note that, importantly, the network training does not need to be paused while the termination criterion is run: we simply run MCMC sampling on the available CPU cores while the network training continues on the GPU.

4 Experiments

To test our predictive termination criterion under a wide variety of different conditions, we performed experiments on a broad range of DNN architectures and datasets, carrying out a combined search over architectures and hyperparameters using two state-of-the-art hyperparameter optimization methods.

4.1 Experimental Setup

We used three popular datasets concerning object recognition from small-sized images: the image recognition datasets CIFAR-10 and CIFAR-100 [Krizhevsky, 2009] and the well known MNIST dataset [LeCun et al., 1989]. CIFAR-10 and CIFAR-100 each contain 50,000 training and 10,000 test RGB images with 32 × 32 pixels that were taken from a subset of the 80-million tiny images database. While CIFAR-10 contains images from 10 categories, CIFAR-100 contains images from 100 categories and thus contains 10 times fewer examples per class. The MNIST dataset is a classic object recognition dataset consisting of 60,000 training and 10,000

test images with 28 × 28 pixels depicting hand-written digits to be classified into 10 digit classes.

For performing the hyperparameter search on CIFAR-10 and CIFAR-100, we randomly split the training data into training and validation sets containing 40,000 and 10,000 examples, respectively. Likewise, for MNIST, we split the training data into a training set containing 50,000 examples and a validation set containing 10,000 examples. We used the deep learning framework CAFFE [Jia et al., 2014] to train DNNs on a single GPU per run. We further used the hyperparameter optimization toolbox HPOLIB [Eggensperger et al., 2013] in combination with our implementation of the predictive termination criterion based on learning curve prediction, using the MCMC sampler EMCEE [Foreman-Mackey et al., 2013].

For the predictive termination criterion we set the threshold to δ = 0.05 in all experiments, that is, we stopped training a network if our extrapolation model was 95% certain that it would not improve over the best known performance ˆ y when fully trained. We ran the predictive termination criterion every p = 30 epochs. The number of intervals k per SGD epoch at which we evaluate validation performance was chosen separately for each architecture to reflect the cost of computing predictions on the validation data; we used k = 10 for fully connected networks and k = 2 for convolutional networks.

4.2 Fully Connected Networks

In the first experiment, we trained fully connected networks for classification on a preprocessed variant of CIFAR-10 and on MNIST. To make training a fully connected network on CIFAR-10 feasible we used the same pipeline as Swersky et al. [2013], who followed the approach from Coates et al. [2011] to create preprocessed CIFAR-10 features that act as a fixed convolutional layer while keeping the required computation time manageable. The pipeline first runs unsupervised k-means (with 400 centroids) on small patches of 6 × 6 pixels that are randomly extracted from the CIFAR-10 training set. It then builds a feature vector by convolving each image in CIFAR-10 with the centroids and averaging the responses over parts of the image. After this preprocessing step, the network contains only fully connected layers to classify the preprocessed data. We evaluated the benefits that our predictive termination criterion yields in combination with three different hyperparameter optimizers: SMAC, TPE, and random search (all described in Section 2.1). Each hyperparameter optimizer had a budget of evaluating a total of 100 networks. The maximum number of epochs emax was 285.

In the MNIST experiment we fed the raw 784 pixel values to the fully connected networks. Our setup is thus comparable to most results on fully connected networks from the literature, e.g., the recent results on training dropout networks to classify MNIST [Srivastava et al., 2014]. Training a single network on MNIST required between 5 and 20 minutes, and the hyperparameter optimizers had a fixed budget of evaluating a total of 500 networks.

DNN Hyperparameters

The hyperparameters for the fully connected network control several architectural choices and hyperparameters related to the optimization procedure. They include global hyperpa-

Network hyperparameters

rameters (which apply to the whole network) and per-layer hyperparameters; since the number of layers is a hyperparameter itself, all hyperparameters of layer i are conditional on the number of layers being at least i. Both our hyperparameter optimizers SMAC and TPE can natively handle such conditional hyperparameters to solve the combined architecture search and hyperparameter optimization problem. We used stochastic gradient descent with momentum in all experiments. The learning rate was either fixed or changed according to the inv schedule 2 . All units in the network use rectified linear activation functions, and a softmax layer with dimensionality 10 is placed at the end to predict the 10 classes. Weights are either initialized with Gaussian noise or with the method proposed by Glorot and Bengio [2010]. Biases on the other hand are either initialized to zero or to a constant value. Dropout is optionally also applied to the input of the network. Table 1 details all hyperparameters, along with their ranges and the default values used to start the search; in total, the hyperparameter space to be searched has 10(network hyperparams) + 6(layers) × 7(hyperparams per layer) = 52 hyperparameters.

Results for preprocessed CIFAR-10

Figures 3a and 3b illustrate the speedups that our predictive termination yielded for training fully connected networks on preprocessed CIFAR-10 with SMAC and TPE. We ran each hyperparameter optimizer 5 times with different random seeds and show means and standard deviations of their validation errors across these runs. While all optimizers achieved strong performance for this architecture (around 20% validation error), their computational time requirements to achieve this level of performance differed greatly. As Figure 3a and Figure 3b show, our predictive termination criterion sped up both SMAC and TPE by at least a factor of two for reaching the same validation error as without it. Overall, the average time needed per hyperparamter optimization run was reduced from 40 to 18 hours. After this experiment, we applied the best models found by each optimizer to the test data to compute the classification error. These results-together with a comparison to random search and previous attempts for using k-means preprocessed CIFAR-10-are given in Table 2, showing that all optimizers we used found configurations with slightly better performance than the previously published results for this architecture, with SMAC yielding the overall best result for this experiment.

2 The inv schedule is defined as αt = α0(1+γt)-p , where α0 is the initial learning rate. In order to be able to set bounds intuitively, instead of parameterizing γ directly we make the half-life of α a new hyperparameter, which for a given p can be transformed back into γ.

Figure 4 shows the effect of our predictive termination criterion on the different DNN training runs: the predictive termination criterion successfully terminated runs that do not reach top performance but rather converge slowly to mediocre results. The figure also shows that it was possible to terminate many poor runs quite early.

Results for MNIST

Figure 3c illustrates the speedups that predictive termination yielded for training fully connected networks on MNIST with SMAC. We only ran SMAC (10 runs) for this experiment since it had yielded the best results for CIFAR-10 (cf. Table 2). Consistent with the results on CIFAR-10, SMAC found networks with good performance with and without predictive termination (reaching approximately 1% validation error) and was much faster when using the predictive termination criterion: it reached 1% validation error in about 60% of the time it took a standard SMAC run to achieve the same performance.

4.3 Small Convolutional Neural Networks

To study the generality of our learning curve extrapolation models, next, we turned to the problem of optimizing convolutional neural networks (CNNs), which constitute the stateof-the-art for visual object recognition (see [Krizhevsky et al., 2012; Jarrett et al., 2009] for recent explanations of convolutional layers). For this experiment we used the CIFAR10 and CIFAR-100 datasets. The images from both datasets were preprocessed using a whitening transform, following the practice outlined by Goodfellow et al. [2013]. Other than that the CNNs were trained on the raw pixel images.

Small CNN Hyperparameters

At the core, our CNNs no longer contain fully connected layers but rather use convolutional layers only, followed by a softmax classification layer. These layers extract features by convolving the input image or-for deeper layers-the output of the previous layer with a set of filters. Convolutional layers are regularly followed by dimensionality reduction steps (or pooling steps) which reduce the spatial dimensionality of

(c) With predictive termination

Figure 4: Comparison of learning curves for fully connected networks on CIFAR-10 with and without our predictive termination criterion. Best viewed in color. The plots contain all learning curves from the 10 runs of SMAC and TPE.

the feature map. In our first experiment with small CNNs, we model the hyperparameter space after the prominent architecture from Krizhevsky et al. [2012]. Concretely, we build a hyperparameter space that mimics the layers-18pct config from cuda-convnet 3 . Table 3 summarizes this hyperparameter space. In contrast to the experiments with fully connected networks, we now parameterized the number of layers indirectly, by choosing the number of pooling layers (2 - 3). A convolutional layer is then always placed between these pooling layers. Each pooling layer always halves the input size by using max-pooling. The convolutional kernel size in each layer of our small CNNs is set to 5 × 5. These restrictions also make training quite fast (approximately 30 minutes per network). Overall, our small CNNs contain 5 network hyperparameters and 3 layer hyperparameters (conditioned on the number of convolutional layers used), resulting in a total of 5 + 3 × 3 = 14 configurable hyperparameters.

Results for Small CNNs on CIFAR-10

We optimized the small CNN hyperparameter space using 10 runs of both SMAC and TPE, with and without predictive termination. Each run had a maximum budget of evaluating 150 configurations. The maximum number of epochs emax was 100. While all optimizers eventually found configurations with good validation performance of around 20% error, SMAC gave slightly better results on average (19.4%±0.2% error vs. 20%± 0.4% for TPE). We thus only present the results for SMAC here due to space constraints.

Figure 5a shows that predictive termination again sped up SMAC by a factor of at least two for finding the same validation error as without it. As shown in Figure 5b, predictive termination again consistently stopped bad runs early. When the best resulting configuration in terms of validation error was re-trained on the complete CIFAR-10 training dataset, it achieved a test error of 17.2% - slightly better than the baseline model (layers-18pct with 18% test error). A complete comparison of the test error for the best configuration found by the different optimizers is given in Table 4 (top).

3 http://code.google.com/p/cuda-convnet/

CIFAR-10 classification error

Results for Small CNNs on CIFAR-100

For CIFAR-100, we again optimized the same small CNN hyperparameter space as for CIFAR-10. For this experiment, we only used SMAC (10 runs with and without predictive termination) since it gave the best results in our experiments on CIFAR-10. As for CIFAR-10, each hyperparameter optimization run had a budget of evaluating 150 configurations. Figure 5c gives the results of this experiment. SMAC found configurations with approximately 43% validation error both with and without the predictive termination criterion, but was substantially faster with predictive termination and reached the same quality almost two times faster. Using the best configuration found with and without predictive termination to re-train on the full CIFAR-100 training data yielded slightly better test set performance for predictive termination (41.90% vs. 42.21%); in comparison, adapting the layers-18pct to CIFAR-100 yields 45% test error.

4.4 Large Convolutional Networks

Finally, to test our learning curve prediction on state-of-theart CNN models, we optimized the hyperparameters of a family of large convolutional networks on CIFAR-10.

Large CNN Hyperparameters

To model these larger CNNs, we re-use the hyperparameter space from Table 3 and alter it in several key aspects. Firstly, following the recently proposed All-CNN network [Springenberg et al., 2015], we replaced max-pooling with additional convolutional layers with stride two (each of these layers is also configurable using the convolutional layer hyperparameters from Table 3 (bottom)). Secondly, we no longer fixed the number of convolutional layers between dimensionality reduction (pooling) steps but made it an additional network hyperparameter with range 1 - 3. Our large CNNs are thus considerably deeper than our small CNNs. We further allowed more units in each convolutional layer (between 64 to 192) and changed the kernel size to 3 × 3. The other notable difference to the small CNNs is that the output of the last dimensionality reduction layer is not fed directly to a softmax classifier but rather sent through an additional 3 × 3 and a final one-by-one convolutional layer followed by the softmax layer (whose output is averaged over the remaining spatial dimensions). This setup is in accordance with the network structure from Springenberg et al. [2015]. Overall, our large CNNs have many hyperparameters due to the additional convolutional layers and the newly parameterized pooling layers: 6(network hyperparams)+3(layer hyperparams)× [3(reduction steps) × (3conv. layers + 1reduction layer) + 2final layers] = 48 hyperparameters.

Results for Large CNNs on CIFAR-10

We tested our predictive termination criterion on the configuration space for large CNNs on CIFAR-10. Due to the large time costs attached to such an experiment-training a single configuration for the All-CNN on CIFAR-10 takes between 6 and 12 hours on a NVIDIA Titan GPU-we restricted ourselves to a single run of SMAC on this hyperparameter configuration space, using 4 GPUs in parallel. We trained all networks for a maximum of 800 epochs and evaluated 100 configurations. Table 5 compares the performance of the best network resulting from this experiment to the state-of-the-art for CIFAR-10 without data augmentation. The best model found by our approach performs comparably with the ALL-CNN results, slightly outperforming the best previously reported results. While the total runtime for this experiment with pre-

CIFAR-10 classification error

dictive termination was approximately 8 days on 4 GPUs, the optimization run without predictive termination would have taken more than 20 days on 4 GPUs.

5 Conclusion

We presented a method for speeding up the hyperparameter search for deep neural networks by automatically detecting and terminating underperforming hyperparameter evaluations. For this purpose, we introduced a probabilistic learning curve model that-like human experts-can extrapolate performance from only a few steps of stochastic gradient descent and terminate the training of models that are expected to yield poor performance. Our method is agnostic to the hyperparameter optimizer used, and in our experiments for optimizing various network architectures on several benchmarks it consistently sped up two state-of-the-art hyperparameter optimizers by a factor of roughly two, leading to state-of-the-art results on the CIFAR-10 dataset without data augmentation. The code for our learning curve prediction models and its integration into the CAFFE framework is publicly available at https://github.com/automl/pylearningcurvepredictor.

Acknowledgements

This work was supported by the German Research Foundation (DFG), under Priority Programme Autonomous Learning (SPP 1527, grant HU 1900/3-1) and under the BrainLinksBrainTools Cluster of Excellence (grant number EXC 1086).