{"type":"ScholarlyArticle","title":"c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimization","authors":[{"name":"Shuhei Watanabe","slug":"shuhei-watanabe","given":"Shuhei","family":"Watanabe","affiliation":"Department of Computer Science , University of Freiburg , Germany"},{"name":"Frank Hutter","slug":"frank-hutter","given":"Frank","family":"Hutter","affiliation":"Department of Computer Science , University of Freiburg , Germany"}],"abstract":"Hyperparameter optimization (HPO) is crucial for strong performance of deep learning algorithms and real-world applications often impose some constraints, such as memory usage, or latency on top of the performance requirement. In this work, we propose constrained TPE (c-TPE), an extension of the widely-used versatile Bayesian optimization method, tree-structured Parzen estimator (TPE), to handle these constraints. Our proposed extension goes beyond a simple combination of an existing acquisition function and the original TPE, and instead includes modifications that address issues that cause poor performance. We thoroughly analyze these modifications both empirically and theoretically, providing insights into how they effectively overcome these challenges. In the experiments, we demonstrate that c-TPE exhibits the best average rank performance among existing methods with statistical significance on 81 expensive HPO with inequality constraints. Due to the lack of baselines, we only discuss the applicability of our method to hard-constrained optimization in Appendix D. See https://arxiv.org/abs/2211.14411 for the latest version with Appendix.","doi":"10.24963/ijcai.2023/486","url":"https://ijcai.org/proceedings/2023/486","pdf":"https://ijcai.org/proceedings/2023/0486.pdf","markdown":"https://ijcai.org/proceedings/2023/486.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/486","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":486,"pages":{"first":4371,"last":4379},"keywords":["Machine Learning: ML: Hyperparameter optimization","Machine Learning: ML: Automated machine learning"],"videos":[],"references":[{"position":1,"raw":"[Addison et al., 2022] H. Addison, KS. inversion, H. Ryan, and C. Ted. Happywhale -whale and dolphin identifica- tion. Kaggle, 2022.","title":"Happywhale -whale and dolphin identification","authors":"H Addison, H Ks. Inversion, C Ryan, Ted","year":"2022","venue":"Kaggle","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Akiba et al., 2019] T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: A next-generation hyperparam- eter optimization framework. In International Conference on Knowledge Discovery & Data Mining, 2019.","title":"Optuna: A next-generation hyperparameter optimization framework","authors":"T Akiba, S Sano, T Yanase, T Ohta, M Koyama","year":"2019","venue":"International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Alina et al., 2019] JE. Alina, C. Phil, B. Rodrigo, and G. Victor. Open images 2019 -object detection. Kaggle, 2019.","title":"Open images 2019 -object detection","authors":"Je, C Alina, B Phil, G Rodrigo, Victor","year":"2019","venue":"Kaggle","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Arnold and Hansen, 2012] D. Arnold and N. Hansen. A (1+1)-CMA-ES for constrained optimisation. In Genetic and Evolutionary Computation Conference, 2012.","title":"A (1+1)-CMA-ES for constrained optimisation","authors":"D Arnold, N Hansen","year":"2012","venue":"Genetic and Evolutionary Computation Conference","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Awad et al., 2021] N. Awad, N. Mallik, and F. Hutter. DEHB: Evolutionary hyberband for scalable, robust and efficient hyperparameter optimization. arXiv:2105.09821, 2021.","title":"DEHB: Evolutionary hyberband for scalable, robust and efficient hyperparameter optimization","authors":"N Awad, N Mallik, F Hutter","year":"2021","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2021/296"},{"position":6,"raw":"[Bergstra and Bengio, 2012] J. Bergstra and Y. Bengio. Ran- dom search for hyper-parameter optimization. Journal of Machine Learning Research, 13(2), 2012.","title":"Random search for hyper-parameter optimization","authors":"J Bergstra, Y Bengio","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bergstra et al., 2011] J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for hyper-parameter optimiza- tion. In Advances in Neural Information Processing Sys- tems, 2011.","title":"Algorithms for hyper-parameter optimization","authors":"J Bergstra, R Bardenet, Y Bengio, B Kégl","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bergstra et al., 2013] J. Bergstra, D. Yamins, and D. Cox. Making a science of model search: Hyperparameter op- timization in hundreds of dimensions for vision architec- tures. In International Conference on Machine Learning, 2013.","title":"Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures","authors":"J Bergstra, D Yamins, D Cox","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Brochu et al., 2010] E. Brochu, V. Cora, and N. de Fre- itas. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hi- erarchical reinforcement learning. arXiv:1012.2599, 2010.","title":"A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning","authors":"E Brochu, V Cora, N De Freitas","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen et al., 2018] Y. Chen, A. Huang, Z. Wang, I. Antonoglou, J. Schrittwieser, D. Silver, and N. de Fre- itas. Bayesian optimization in alphago. arXiv:1812.06855, 2018.","title":"Bayesian optimization in alphago","authors":"Y Chen, A Huang, Z Wang, I Antonoglou, J Schrittwieser, D Silver, N De Freitas","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Deb et al., 2002] K. Deb, A. Pratap, S. Agarwal, and T. Me- yarivan. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computa- tion, 6(2):182-197, 2002.","title":"A fast and elitist multiobjective genetic algorithm: NSGA-II","authors":"K Deb, A Pratap, S Agarwal, T Meyarivan","year":"2002","venue":"IEEE Transactions on Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Dong and Yang, 2020] X. Dong and Y. Yang. NAS-Bench- 201: Extending the scope of reproducible neural architec- ture search. arXiv:2001.00326, 2020.","title":"NAS-Bench-201: Extending the scope of reproducible neural architecture search","authors":"Dong, Yang ;, X Dong, Y Yang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Eriksson and Poloczek, 2021] D. Eriksson and M. Poloczek. Scalable constrained Bayesian opti- mization. In International Conference on Artificial Intelligence and Statistics, 2021.","title":"Scalable constrained Bayesian optimization","authors":"D Eriksson, M Poloczek","year":"2021","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Falkner et al., 2018] S. Falkner, A. Klein, and F. Hutter. BOHB: Robust and efficient hyperparameter optimization at scale. In International Conference on Machine Learn- ing, 2018.","title":"BOHB: Robust and efficient hyperparameter optimization at scale","authors":"S Falkner, A Klein, F Hutter","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gardner et al., 2014] J. Gardner, M. Kusner, ZE. Xu, K. Weinberger, and J. Cunningham. Bayesian optimiza- tion with inequality constraints. In International Confer- ence on Machine Learning, 2014.","title":"Bayesian optimization with inequality constraints","authors":"J Gardner, M Kusner, Z E Xu, K Weinberger, J Cunningham","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Garnett, 2022] R. Garnett. Bayesian Optimization. Cam- bridge University Press, 2022.","title":"Bayesian Optimization","authors":"R Garnett","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Garrido-Merchán et al., 2023] EC.","title":"EC","authors":"Garrido-Merchán","year":"2023","venue":"EC","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Garrido-Merchán, D. Fernández-Sánchez, and D. Hernández-Lobato. Parallel predictive entropy search for multi-objective Bayesian optimization with constraints applied to the tuning of machine learning algorithms. Expert Systems with Applications, 215, 2023.","title":"Parallel predictive entropy search for multi-objective Bayesian optimization with constraints applied to the tuning of machine learning algorithms","authors":"D Garrido-Merchán, D Fernández-Sánchez, Hernández-Lobato","year":"2023","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Gelbart et al., 2014] M. Gelbart, J. Snoek, and R. Adams. Bayesian optimization with unknown constraints. arXiv:1403.5607, 2014.","title":"Bayesian optimization with unknown constraints","authors":"M Gelbart, J Snoek, R Adams","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Jones et al., 1998] D. Jones, M. Schonlau, and W. Welch. Efficient global optimization of expensive black-box func- tions. Journal of Global Optimization, 13(4):455-492, 1998.","title":"Efficient global optimization of expensive black-box functions","authors":"D Jones, M Schonlau, W Welch","year":"1998","venue":"Journal of Global Optimization","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Klein and Hutter, 2019] A. Klein and F. Hutter. Tabular benchmarks for joint architecture and hyperparameter op- timization. arXiv:1905.04970, 2019.","title":"Tabular benchmarks for joint architecture and hyperparameter optimization","authors":"A Klein, F Hutter","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Kushner, 1964] HJ. Kushner. A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. Joint Automatic Control Conference, 1964.","title":"A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise","authors":"Hj, Kushner","year":"1964","venue":"Joint Automatic Control Conference","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Letham et al., 2019] B. Letham, B. Karrer, G. Ottoni, and E. Bakshy. Constrained Bayesian optimization with noisy experiments. Bayesian Analysis, 14(2):495-519, 2019.","title":"Constrained Bayesian optimization with noisy experiments","authors":"B Letham, B Karrer, G Ottoni, E Bakshy","year":"2019","venue":"Bayesian Analysis","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liaw et al., 2018] R. Liaw, E. Liang, R. Nishihara, P. Moritz, J. Gonzalez, and I. Stoica. Tune: A re- search platform for distributed model selection and train- ing. arXiv:1807.05118, 2018.","title":"Tune: A research platform for distributed model selection and training","authors":"R Liaw, E Liang, R Nishihara, P Moritz, J Gonzalez, I Stoica","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Lobato et al., 2015] JH. Lobato, M. Gelbart, M. Hoffman, R. Adams, and Z. Ghahramani. Predictive entropy search for bayesian optimization with unknown constraints. In International Conference on Machine Learning, 2015.","title":"Predictive entropy search for bayesian optimization with unknown constraints","authors":"Lobato","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Loshchilov and Hutter, 2016] I. Loshchilov and F. Hutter. CMA-ES for hyperparameter optimization of deep neural networks. arXiv:1604.07269, 2016.","title":"CMA-ES for hyperparameter optimization of deep neural networks","authors":"Hutter Loshchilov, I Loshchilov, F Hutter","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Loshchilov et al., 2013] I. Loshchilov, M. Schoenauer, and M. Sebag. Bi-population CMA-ES agorithms with surro- gate models and line searches. In Genetic and Evolution- ary Computation Conference, 2013.","title":"Bi-population CMA-ES agorithms with surrogate models and line searches","authors":"Loshchilov","year":"2013","venue":"Genetic and Evolutionary Computation Conference","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Melis et al., 2018] G. Melis, C. Dyer, and P. Blunsom. On the state of the art of evaluation in neural language models. In International Conference on Learning Representations, 2018.","title":"On the state of the art of evaluation in neural language models","authors":"G Melis, C Dyer, P Blunsom","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Montes et al., 2006] EM. Montes, J. Velázquez-Reyes, and CA. Coello. Modified differential evolution for con- strained optimization. In International Conference on Evo- lutionary Computation, 2006.","title":"Modified differential evolution for constrained optimization","authors":"E M Montes, J Velázquez-Reyes, C A Coello","year":"2006","venue":"International Conference on Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Nardi et al., 2019] L. Nardi, D. Koeplinger, and K. Oluko- tun. Practical design space exploration. In Interna- tional Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems, pages 347- 358. IEEE, 2019.","title":"Practical design space exploration","authors":"L Nardi, D Koeplinger, K Olukotun","year":"2019","venue":"International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Perrone et al., 2019] V. Perrone, I. Shcherbatyi, R. Jenat- ton, C. Archambeau, and M. Seeger. Constrained Bayesian optimization with max-value entropy search. arXiv:1910.07003, 2019.","title":"Constrained Bayesian optimization with max-value entropy search","authors":"V Perrone, I Shcherbatyi, R Jenatton, C Archambeau, M Seeger","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Shahriari et al., 2016] B. Shahriari, K. Swersky, Z. Wang, R. Adams, and N. de Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1):148-175, 2016.","title":"Taking the human out of the loop: A review of Bayesian optimization","authors":"B Shahriari, K Swersky, Z Wang, R Adams, N De Freitas","year":"2016","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Watanabe, 2023] S. Watanabe. Tree-structured Parzen esti- mator: Understanding its algorithm components and their roles for better empirical performance. arXiv:2304.11127, 2023.","title":"Tree-structured Parzen estimator: Understanding its algorithm components and their roles for better empirical performance","authors":"S Watanabe","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Ying et al., 2019] C. Ying, A. Klein, E. Christiansen, E. Real, K. Murphy, and F. Hutter. NAS-Bench-101: To- wards reproducible neural architecture search. In Interna- tional Conference on Machine Learning, 2019.","title":"NAS-Bench-101: Towards reproducible neural architecture search","authors":"C Ying, A Klein, E Christiansen, E Real, K Murphy, F Hutter","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}