{"type":"ScholarlyArticle","title":"Hyper-parameter Tuning under a Budget Constraint","authors":[{"name":"Zhiyun Lu","slug":"zhiyun-lu","given":"Zhiyun","family":"Lu","affiliation":"University of Southern California"},{"name":"Liyu Chen","slug":"liyu-chen","given":"Liyu","family":"Chen","affiliation":"University of Southern California"},{"name":"Chao-Kai Chiang","slug":"chao-kai-chiang","given":"Chao-Kai","family":"Chiang","affiliation":"Appier Inc ."},{"name":"Fei Sha","slug":"fei-sha","given":"Fei","family":"Sha","affiliation":"Google AI"}],"abstract":"Hyper-parameter tuning is of crucial importance for real-world machine learning applications. While existing works mainly focus on speeding up the tuning process, we propose to study the problem of hyper-parameter tuning under a budget constraint, which is a more realistic scenario in developing large-scale systems. We formulate the task into a sequential decision making problem and propose a solution, which uses a Bayesian belief model to predict future performances, and an action-value function to plan and select the next configuration to run. With long term prediction and planning capability, our method is able to early stop unpromising configurations, and adapt the tuning behaviors to different constraints.  Experiment results show that our method outperforms existing algorithms, including the-state-of-the-art one, on real-world tuning tasks across a range of different budgets.","doi":"10.24963/ijcai.2019/796","url":"https://ijcai.org/proceedings/2019/796","pdf":"https://ijcai.org/proceedings/2019/0796.pdf","markdown":"https://ijcai.org/proceedings/2019/796.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/796","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Uncertainty in AI","track":"Main track","number":796,"pages":{"first":5744,"last":5750},"keywords":["Machine Learning: Probabilistic Machine Learning","Uncertainty in AI: Uncertainty in AI","Machine Learning Applications: Big data ; Scalability"],"videos":[],"references":[{"position":1,"raw":"Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc V Le. Neural optimizer search with reinforcement learning. arXiv preprint arXiv:1709.07417, 2017.","title":"Neural optimizer search with reinforcement learning","authors":"Irwan Bello, Barret Zoph, Vijay Vasudevan, Quoc V Le","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"Yutian Chen, Matthew W Hoffman, Sergio Gomez Col- menarejo, Misha Denil, Timothy P Lillicrap, and Nando de Freitas. Learning to learn for global optimization of black box functions. arXiv preprint arXiv:1611.03824, 2016.","title":"Learning to learn for global optimization of black box functions","authors":"Yutian Chen, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Timothy P Lillicrap, Nando De Freitas","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimization at scale. arXiv preprint arXiv:1807.01774, 2018.","title":"Bohb: Robust and efficient hyperparameter optimization at scale","authors":"Stefan Falkner, Aaron Klein, Frank Hutter","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"Luca Franceschi, Michele Donini, Paolo Frasconi, and Massi- miliano Pontil. Forward and reverse gradient-based hyper- parameter optimization. arXiv preprint arXiv:1703.01785, 2017.","title":"Forward and reverse gradient-based hyperparameter optimization","authors":"Luca Franceschi, Michele Donini, Paolo Frasconi, Massimiliano Pontil","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"Luca Franceschi, Paolo Frasconi, Saverio Salzo, and Mas- similano Pontil. Bilevel programming for hyperparam- eter optimization and meta-learning. arXiv preprint arXiv:1806.04910, 2018.","title":"Bilevel programming for hyperparameter optimization and meta-learning","authors":"Luca Franceschi, Paolo Frasconi, Saverio Salzo, Massimilano Pontil","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"Elad Hazan, Adam Klivans, and Yang Yuan. Hyperparam- eter optimization: a spectral approach. arXiv preprint arXiv:1706.00764, 2017.","title":"Hyperparameter optimization: a spectral approach","authors":"Elad Hazan, Adam Klivans, Yang Yuan","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"Ronald A Howard. Information value theory. IEEE Trans- actions on systems science and cybernetics, 2(1):22-26, 1966.","title":"Information value theory","authors":"A Ronald, Howard","year":"1966","venue":"IEEE Transactions on systems science and cybernetics","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"Max Jaderberg, Valentin Dalibard, Simon Osindero, Wo- jciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv:1711.09846, 2017.","title":"Population based training of neural networks","authors":"Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Ar- tificial Intelligence and Statistics, pages 240-248, 2016.","title":"Non-stochastic best arm identification and hyperparameter optimization","authors":"Kevin Jamieson, Ameet Talwalkar","year":"2016","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. Fast bayesian optimization of machine learning hyperparameters on large datasets. In Artificial Intelligence and Statistics, pages 528-536, 2017.","title":"Fast bayesian optimization of machine learning hyperparameters on large datasets","authors":"Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, Frank Hutter","year":"2017","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Remi Lam, Karen Willcox, and David H Wolpert. Bayesian optimization with a finite budget: An approximate dy- namic programming approach. In Advances in Neural In- formation Processing Systems, pages 883-891, 2016.","title":"Bayesian optimization with a finite budget: An approximate dynamic programming approach","authors":"Remi Lam, Karen Willcox, David H Wolpert","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Ros- tamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. arXiv preprint arXiv:1603.06560, 2016.","title":"Hyperband: A novel bandit-based approach to hyperparameter optimization","authors":"Lisha Li, Kevin Jamieson, Giulia Desalvo, Afshin Rostamizadeh, Ameet Talwalkar","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimization through re- versible learning. In International Conference on Machine Learning, pages 2113-2122, 2015.","title":"Gradient-based hyperparameter optimization through reversible learning","authors":"Dougal Maclaurin, David Duvenaud, Ryan Adams","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"Robert Nishihara, David Lopez-Paz, and Léon Bottou. No regret bound for extreme bandits. In AISTATS, pages 259- 267, 2016.","title":"No regret bound for extreme bandits","authors":"Robert Nishihara, David Lopez-Paz, Léon Bottou","year":"2016","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"Ashish Sabharwal, Horst Samulowitz, and Gerald Tesauro. Selecting near-optimal learners via incremental data allo- cation. In Thirtieth AAAI Conference on Artificial Intelli- gence, 2016.","title":"Selecting near-optimal learners via incremental data allocation","authors":"Ashish Sabharwal, Horst Samulowitz, Gerald Tesauro","year":"2016","venue":"Thirtieth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando de Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1):148-175, 2016.","title":"Taking the human out of the loop: A review of bayesian optimization","authors":"Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, Nando De Freitas","year":"2016","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pages 2951-2959, 2012.","title":"Practical bayesian optimization of machine learning algorithms","authors":"Jasper Snoek, Hugo Larochelle, Ryan P Adams","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Kevin Swersky, Jasper Snoek, and Ryan Prescott Adams. Freeze-thaw bayesian optimization. arXiv preprint arXiv:1406.3896, 2014.","title":"Freeze-thaw bayesian optimization","authors":"Kevin Swersky, Jasper Snoek, Ryan Prescott, Adams","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}