{"type":"ScholarlyArticle","title":"Best Arm Identification with Retroactively Increased Sampling Budget for More Resource-Efficient HPO","authors":[{"name":"Jasmin Brandt","slug":"jasmin-brandt","given":"Jasmin","family":"Brandt","affiliation":"Paderborn University , Germany"},{"name":"Marcel Wever","slug":"marcel-wever","given":"Marcel","family":"Wever","affiliation":"LMU Munich , Germany; Munich Center for Machine Learning , Germany"},{"name":"Viktor Bengs","slug":"viktor-bengs","given":"Viktor","family":"Bengs","affiliation":"LMU Munich , Germany; Munich Center for Machine Learning , Germany"},{"name":"Eyke Hüllermeier","slug":"eyke-hullermeier","given":"Eyke","family":"Hüllermeier","affiliation":"LMU Munich , Germany; Munich Center for Machine Learning , Germany"}],"abstract":"Hyperparameter optimization (HPO) is indispensable for achieving optimal performance in machine learning tasks. A popular class of methods in this regard is based on Successive Halving (SHA), which casts HPO into a pure-exploration multi-armed bandit problem under finite sampling budget constraints. This is accomplished by considering hyperparameter configurations as arms and rewards as the negative validation losses. While enjoying theoretical guarantees as well as working well in practice, SHA comes, however, with several hyperparameters itself, one of which is the maximum budget that can be allocated to evaluate a single arm (hyperparameter configuration). Although there are already solutions to this meta hyperparameter optimization problem, such as the doubling trick or asynchronous extensions of SHA, these are either practically inefficient or lack theoretical guarantees. In this paper, we propose incremental SHA (iSHA), a synchronous extension of SHA, allowing to increase the maximum budget a posteriori while still enjoying theoretical guarantees. Our empirical analysis of HPO problems corroborates our theoretical findings and shows that iSHA is more resource-efficient than existing SHA-based approaches.","doi":"10.24963/ijcai.2024/414","url":"https://ijcai.org/proceedings/2024/414","pdf":"https://ijcai.org/proceedings/2024/0414.pdf","markdown":"https://ijcai.org/proceedings/2024/414.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/414","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":414,"pages":{"first":3742,"last":3750},"keywords":["Machine Learning: ML: Multi-armed bandits","Machine Learning: ML: Hyperparameter optimization","Machine Learning: ML: Incremental learning"],"videos":[],"references":[{"position":1,"raw":"[Abbasi-Yadkori et al., 2018] Yasin Abbasi-Yadkori, Peter Bartlett, Victor Gabillon, Alan Malek, and Michal Valko. Best of Both Worlds: Stochastic & Adversarial Best-arm Identification. In Proceedings of the 31st Conference On Learning Theory, volume 75 of Proceedings of Machine Learning Research, pages 918-949. PMLR, 06-09 Jul 2018.","title":"Best of Both Worlds: Stochastic & Adversarial Best-arm Identification","authors":"Yasin Abbasi-Yadkori, Peter Bartlett, Victor Gabillon, Alan Malek, Michal Valko","year":"2018","venue":"Proceedings of the 31st Conference On Learning Theory","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Awad et al., 2021] Noor H. Awad, Neeratyoy Mallik, and Frank Hutter. DEHB: Evolutionary Hyberband for Scal- able, Robust and Efficient Hyperparameter Optimization. In Proceedings of the 30th International Joint Conference on Artificial Intelligence, pages 2147-2153, 2021.","title":"DEHB: Evolutionary Hyberband for Scalable, Robust and Efficient Hyperparameter Optimization","authors":"H Noor, Neeratyoy Awad, Frank Mallik, Hutter","year":"2021","venue":"Proceedings of the 30th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2021/296"},{"position":3,"raw":"[Azizi et al., 2022] Mohammad Javad Azizi, Branislav Kve- ton, and Mohammad Ghavamzadeh. Fixed-Budget Best- Arm Identification in Structured Bandits. In Proceedings of the Thirty-First International Joint Conference on Artifi- cial Intelligence, IJCAI, pages 2798-2804. ijcai.org, 2022.","title":"Fixed-Budget Best-Arm Identification in Structured Bandits","authors":"Mohammad Javad Azizi, Branislav Kveton, Mohammad Ghavamzadeh","year":"2022","venue":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/388"},{"position":4,"raw":"[Bergstra et al., 2011] James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for Hyper-Parameter Optimization. In Advances in Neural Information Processing Systems, pages 2546-2554, 2011.","title":"Algorithms for Hyper-Parameter Optimization","authors":"James Bergstra, Rémi Bardenet, Yoshua Bengio, Balázs Kégl","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bischl et al., 2023] Bernd Bischl, Martin Binder, Michel Lang, Tobias Pielok, Jakob Richter, Stefan Coors, Janek Thomas, Theresa Ullmann, Marc Becker, Anne-Laure Boulesteix, et al. Hyperparameter Optimization: Foun- dations, Algorithms, Best Practices and Open Challenges. Wiley Interdisciplinary Reviews: Data Mining and Knowl- edge Discovery, 13, 2023.","title":"Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges","authors":"Bernd Bischl, Martin Binder, Michel Lang, Tobias Pielok, Jakob Richter, Stefan Coors, Janek Thomas, Theresa Ullmann, Marc Becker, Anne-Laure Boulesteix","year":"2023","venue":"Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bohdal et al., 2023] Ondrej Bohdal, Lukas Balles, Martin Wistuba, Beyza Ermis, Cédric Archambeau, and Giovanni Zappella. PASHA: Efficient HPO and NAS with Progres- sive Resource Allocation. In The 11th International Con- ference on Learning Representations, 2023.","title":"PASHA: Efficient HPO and NAS with Progressive Resource Allocation","authors":"Bohdal","year":"2023","venue":"The 11th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Brandt et al., 2022] Jasmin Brandt, Viktor Bengs, Björn Haddenhorst, and Eyke Hüllermeier. Finding Optimal Arms in Non-stochastic Combinatorial Bandits with Semi- bandit Feedback and Finite Budget. In Advances in Neural Information Processing Systems, 2022.","title":"Finding Optimal Arms in Non-stochastic Combinatorial Bandits with Semibandit Feedback and Finite Budget","authors":"Jasmin Brandt, Viktor Bengs, Björn Haddenhorst, Eyke Hüllermeier","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Brandt et al., 2023] Jasmin Brandt, Elias Schede, Björn Haddenhorst, Viktor Bengs, Eyke Hüllermeier, and Kevin Tierney. AC-Band: A Combinatorial Bandit-Based Ap- proach to Algorithm Configuration. Proceedings of the AAAI Conference on Artificial Intelligence, 37(10):12355- 12363, 2023.","title":"AC-Band: A Combinatorial Bandit-Based Approach to Algorithm Configuration","authors":"Jasmin Brandt, Elias Schede, Björn Haddenhorst, Viktor Bengs, Eyke Hüllermeier, Kevin Tierney","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Carpentier and Valko, 2015] Alexandra Carpentier and Michal Valko. Simple regret for infinitely many armed bandits. In Proceedings of the 32nd International Conference on Machine Learning, ICML, volume 37 of JMLR Workshop and Conference Proceedings, pages 1133-1141. JMLR.org, 2015.","title":"Simple regret for infinitely many armed bandits","authors":"Valko Carpentier, Alexandra Carpentier, Michal Valko","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning, ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Falkner et al., 2018] Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and Efficient Hyperparame- ter Optimization at Scale. In Proceedings of the 35th Inter- national Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1436- 1445. PMLR, 2018.","title":"BOHB: Robust and Efficient Hyperparameter Optimization at Scale","authors":"Stefan Falkner, Aaron Klein, Frank Hutter","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Feurer and Hutter, 2019] Matthias Feurer and Frank Hutter. Hyperparameter Optimization. In Automated Machine Learning -Methods, Systems, Challenges, The Springer Series on Challenges in Machine Learning, pages 3-33. Springer, 2019.","title":"Hyperparameter Optimization","authors":"Hutter Feurer, Matthias Feurer, Frank Hutter","year":"2019","venue":"Automated Machine Learning -Methods, Systems, Challenges, The Springer Series on Challenges in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Frazier, 2018] Peter I. Frazier. A Tutorial on Bayesian Op- timization. CoRR, abs/1807.02811, 2018.","title":"A Tutorial on Bayesian Optimization","authors":"I Peter, Frazier","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gong and Sellke, 2023] Xiao-Yue Gong and Mark Sellke. Asymptotically Optimal Pure Exploration for Infinite- Armed Bandits. CoRR, abs/2306.01995, 2023.","title":"Asymptotically Optimal Pure Exploration for Infinite-Armed Bandits","authors":"Xiao-Yue Gong, Mark Sellke","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hutter et al., 2011] Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. Sequential Model-Based Optimiza- tion for General Algorithm Configuration. In Learning and Intelligent Optimization -5th International Confer- ence, volume 6683 of Lecture Notes in Computer Science, pages 507-523. Springer, 2011.","title":"Sequential Model-Based Optimization for General Algorithm Configuration","authors":"Frank Hutter, H Holger, Kevin Hoos, Leyton-Brown","year":"2011","venue":"Learning and Intelligent Optimization -5th International Conference","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jamieson and Talwalkar, 2016a] Kevin Jamieson and Ameet Talwalkar. Non-stochastic Best Arm Identification and Hyperparameter Optimization. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pages 240-248. PMLR, 2016.","title":"Non-stochastic Best Arm Identification and Hyperparameter Optimization","authors":"Talwalkar Jamieson, Kevin Jamieson, Ameet Talwalkar","year":"2016","venue":"Proceedings of the 19th International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jamieson and Talwalkar, 2016b] Kevin G. Jamieson and Ameet Talwalkar. Non-stochastic Best Arm Identifica- tion and Hyperparameter Optimization. In Proceedings of the 19th International Conference on Artificial Intelli- gence and Statistics, AISTATS, volume 51 of JMLR Work- shop and Conference Proceedings, pages 240-248, 2016.","title":"Non-stochastic Best Arm Identification and Hyperparameter Optimization","authors":"G Jamieson And Talwalkar ; B] Kevin, Ameet Jamieson, Talwalkar","year":"2016","venue":"Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Karnin et al., 2013] Zohar Karnin, Tomer Koren, and Oren Somekh. Almost Optimal Exploration in Multi-Armed Bandits. In International Conference on Machine Learn- ing, pages 1238-1246. PMLR, 2013.","title":"Almost Optimal Exploration in Multi-Armed Bandits","authors":"Karnin","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kato et al., 2022] Masahiro Kato, Kaito Ariu, Masaaki Imaizumi, Masatoshi Uehara, Masahiro Nomura, and Chao Qin. Optimal Fixed-Budget Best Arm Identification using the Augmented Inverse Probability Weighting Es- timator in Two-Armed Gaussian Bandits with Unknown Variances. CoRR, abs/2201.04469, 2022.","title":"Optimal Fixed-Budget Best Arm Identification using the Augmented Inverse Probability Weighting Estimator in Two-Armed Gaussian Bandits with Unknown Variances","authors":"Masahiro Kato, Kaito Ariu, Masaaki Imaizumi, Masatoshi Uehara, Masahiro Nomura, Chao Qin","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lattimore and Szepesvári, 2020] Tor Lattimore and Csaba Szepesvári. Bandit Algorithms. Cambridge University Press, 2020.","title":"Tor Lattimore and Csaba Szepesvári. Bandit Algorithms","authors":"Szepesvári Lattimore","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2018] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyper- band: A Novel Bandit-Based Approach to Hyperparame- ter Optimization. Journal of Machine Learning Research, 18(185):1-52, 2018.","title":"Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization","authors":"Lisha Li, Kevin Jamieson, Giulia Desalvo, Afshin Rostamizadeh, Ameet Talwalkar","year":"2018","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2020] Liam Li, Kevin Jamieson, Afshin Ros- tamizadeh, Ekaterina Gonina, Jonathan Ben-Tzur, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. A System for Massively Parallel Hyperparameter Tuning. Proceed- ings of Machine Learning and Systems, 2:230-246, 2020.","title":"A System for Massively Parallel Hyperparameter Tuning","authors":"Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-Tzur, Moritz Hardt, Benjamin Recht, Ameet Talwalkar","year":"2020","venue":"Proceedings of Machine Learning and Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mallik et al., 2023] Neeratyoy Mallik, Edward Bergman, Carl Hvarfner, Danny Stoll, Maciej Janowski, Marius Lin- dauer, Luigi Nardi, and Frank Hutter. PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learn- ing. arXiv preprint arXiv:2306.12370, 2023.","title":"PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning","authors":"Mallik","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mendes et al., 2021] Pedro Mendes, Maria Casimiro, and Paolo Romano. HyperJump: Accelerating HyperBand via Risk Modelling. arXiv preprint arXiv:2108.02479, 2021.","title":"HyperJump: Accelerating HyperBand via Risk Modelling","authors":"Pedro Mendes, Maria Casimiro, Paolo Romano","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pfisterer et al., 2022] Florian Pfisterer, Lennart Schneider, Julia Moosbauer, Martin Binder, and Bernd Bischl. YAHPO Gym -An Efficient Multi-Objective Multi- Fidelity Benchmark for Hyperparameter Optimization. In International Conference on Automated Machine Learn- ing, AutoML 2022, volume 188 of Proceedings of Machine Learning Research, pages 3/1-39. PMLR, 2022.","title":"YAHPO Gym -An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimization","authors":"Florian Pfisterer, Lennart Schneider, Julia Moosbauer, Martin Binder, Bernd Bischl","year":"2022","venue":"International Conference on Automated Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rice, 1976] John R. Rice. The Algorithm Selection Prob- lem. Advances in Computers, 15:65-118, 1976.","title":"The Algorithm Selection Problem","authors":"John R Rice","year":"1976","venue":"Advances in Computers","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shen, 2019] Cong Shen. Universal Best Arm Identification. IEEE Trans. Signal Process., 67(17):4464-4478, 2019.","title":"Universal Best Arm Identification","authors":"Cong Shen","year":"2019","venue":"IEEE Trans. Signal Process","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tornede et al., 2023] Tanja Tornede, Alexander Tornede, Jonas Hanselle, Felix Mohr, Marcel Wever, and Eyke Hüllermeier. Towards green automated machine learning: Status quo and future directions. Journal of Artificial In- telligence Research, 77:427-457, 2023.","title":"Towards green automated machine learning: Status quo and future directions","authors":"Tanja Tornede, Alexander Tornede, Jonas Hanselle, Felix Mohr, Marcel Wever, Eyke Hüllermeier","year":"2023","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}