{"type":"ScholarlyArticle","title":"PED-ANOVA: Efficiently Quantifying Hyperparameter Importance in Arbitrary Subspaces","authors":[{"name":"Shuhei Watanabe","slug":"shuhei-watanabe","given":"Shuhei","family":"Watanabe","affiliation":"Department of Computer Science , University of Freiburg , Germany"},{"name":"Archit Bansal","slug":"archit-bansal","given":"Archit","family":"Bansal","affiliation":"Department of Computer Science , University of Freiburg , Germany"},{"name":"Frank Hutter","slug":"frank-hutter","given":"Frank","family":"Hutter","affiliation":"Department of Computer Science , University of Freiburg , Germany"}],"abstract":"The recent rise in popularity of Hyperparameter Optimization (HPO) for deep learning has highlighted the role that good hyperparameter (HP) space design can play in training strong models. In turn, designing a good HP space is critically dependent on understanding the role of different HPs. This motivates research on HP Importance (HPI), e.g., with the popular method of functional ANOVA (f-ANOVA). However, the original f-ANOVA formulation is inapplicable to the subspaces most relevant to algorithm designers, such as those defined by top performance. To overcome this issue, we derive a novel formulation of f-ANOVA for arbitrary subspaces and propose an algorithm that uses Pearson divergence (PED) to enable a closed-form calculation of HPI. We demonstrate that this new algorithm, dubbed PED-ANOVA, is able to successfully identify important HPs in different subspaces while also being extremely computationally efficient. See https://arxiv.org/abs/2304.10255 for the latest version with Appendix.","doi":"10.24963/ijcai.2023/488","url":"https://ijcai.org/proceedings/2023/488","pdf":"https://ijcai.org/proceedings/2023/0488.pdf","markdown":"https://ijcai.org/proceedings/2023/488.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/488","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":488,"pages":{"first":4389,"last":4396},"keywords":["Machine Learning: ML: Hyperparameter optimization","Machine Learning: ML: Automated machine learning"],"videos":[],"references":[{"position":1,"raw":"[Akiba et al., 2019] T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: A next-generation hyperparam- eter optimization framework. In International Conference on Knowledge Discovery & Data Mining, 2019.","title":"Optuna: A next-generation hyperparameter optimization framework","authors":"T Akiba, S Sano, T Yanase, T Ohta, M Koyama","year":"2019","venue":"International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bansal et al., 2022] A. Bansal, D. Stoll, M. Janowski, A. Zela, and F. Hutter. JAHS-Bench-201: A foundation for research on joint architecture and hyperparameter search. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track, 2022.","title":"JAHS-Bench-201: A foundation for research on joint architecture and hyperparameter search","authors":"A Bansal, D Stoll, M Janowski, A Zela, F Hutter","year":"2022","venue":"Advances in Neural Information Processing Systems Datasets and Benchmarks Track","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bergstra and Bengio, 2012] J. Bergstra and Y. Bengio. Ran- dom search for hyper-parameter optimization. Journal of Machine Learning Research, 13, 2012.","title":"Random search for hyper-parameter optimization","authors":"J Bergstra, Y Bengio","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bergstra et al., 2011] J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for hyper-parameter optimiza- tion. In Advances in Neural Information Processing Sys- tems, 2011.","title":"Algorithms for hyper-parameter optimization","authors":"J Bergstra, R Bardenet, Y Bengio, B Kégl","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Biedenkapp et al., 2017] A. Biedenkapp, M. Lindauer, K. Eggensperger, F. Hutter, C. Fawcett, and H. Hoos. Effi- cient parameter importance analysis via ablation with sur- rogates. In Association for the Advancement of Artificial Intelligence, 2017.","title":"Efficient parameter importance analysis via ablation with surrogates","authors":"A Biedenkapp, M Lindauer, K Eggensperger, F Hutter, C Fawcett, H Hoos","year":"2017","venue":"Association for the Advancement of Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Biedenkapp et al., 2018] A. Biedenkapp, J. Marben, M. Lindauer, and F. Hutter. CAVE: Configuration as- sessment, visualization and evaluation. In International Conference on Learning and Intelligent Optimization, 2018.","title":"CAVE: Configuration assessment, visualization and evaluation","authors":"A Biedenkapp, J Marben, M Lindauer, F Hutter","year":"2018","venue":"International Conference on Learning and Intelligent Optimization","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2018] Y. Chen, A. Huang, Z. Wang, I. Antonoglou, J. Schrittwieser, D. Silver, and N. de Freitas. Bayesian optimization in AlphaGo. arXiv:1812.06855, 2018.","title":"Bayesian optimization in AlphaGo","authors":"Y Chen, A Huang, Z Wang, I Antonoglou, J Schrittwieser, D Silver, N De Freitas","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Henderson et al., 2018] P. Henderson, R. Islam, P. Bach- man, J. Pineau, D. Precup, and D. Meger. Deep reinforce- ment learning that matters. In Association for the Advance- ment of Artificial Intelligence, 2018.","title":"Deep reinforcement learning that matters","authors":"P Henderson, R Islam, P Bachman, J Pineau, D Precup, D Meger","year":"2018","venue":"Association for the Advancement of Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hooker, 2007] G. Hooker. Generalized functional ANOVA diagnostics for high-dimensional functions of dependent variables. Journal of Computational and Graphical Statis- tics, 16, 2007.","title":"Generalized functional ANOVA diagnostics for high-dimensional functions of dependent variables","authors":"G Hooker","year":"2007","venue":"Journal of Computational and Graphical Statistics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hutter et al., 2014] F. Hutter, H. Hoos, and K. Leyton- Brown. An efficient approach for assessing hyperparam- eter importance. In International Conference on Machine Learning, 2014.","title":"An efficient approach for assessing hyperparameter importance","authors":"F Hutter, H Hoos, K Leyton-Brown","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lindauer et al., 2022] M. Lindauer, K. Eggensperger, M. Feurer, A. Biedenkapp, D. Deng, C. Benjamins, T. Ruhkopf, R. Sass, and F. Hutter. SMAC3: A versatile bayesian optimization package for Hyperparameter Op- timization. Journal of Machine Learning Research, 23, 2022.","title":"SMAC3: A versatile bayesian optimization package for Hyperparameter Optimization","authors":"M Lindauer, K Eggensperger, M Feurer, A Biedenkapp, D Deng, C Benjamins, T Ruhkopf, R Sass, F Hutter","year":"2022","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Melis et al., 2018] G. Melis, C. Dyer, and P. Blunsom. On the state of the art of evaluation in neural language models. In International Conference on Learning Representations, 2018.","title":"On the state of the art of evaluation in neural language models","authors":"G Melis, C Dyer, P Blunsom","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"K. Pearson. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be rea- sonably supposed to have arisen from random sampling. Philosophical Magazine and Journal of Science, 50, 1900.","title":"On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling","authors":"K Pearson","year":"1900","venue":"Philosophical Magazine and Journal of Science","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Perrone et al., 2019] V. Perrone, H. Shen, MW. Seeger, C. Archambeau, and R. Jenatton. Learning search spaces for Bayesian optimization: Another view of hyperparame- ter transfer learning. Advances in Neural Information Pro- cessing Systems, 2019.","title":"Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning","authors":"V Perrone, H Shen, M W Seeger, C Archambeau, R Jenatton","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sass et al., 2022] R. Sass, E. Bergman, A. Biedenkapp, F. Hutter, and M. Lindauer. DeepCAVE: An inter- active analysis tool for automated machine learning. arXiv:2206.03493, 2022.","title":"DeepCAVE: An interactive analysis tool for automated machine learning","authors":"R Sass, E Bergman, A Biedenkapp, F Hutter, M Lindauer","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Snoek et al., 2012] J. Snoek, H. Larochelle, and R. Adams. Practical bayesian optimization of machine learning algo- rithms. In Advances in Neural Information Processing Sys- tems, 2012.","title":"Practical bayesian optimization of machine learning algorithms","authors":"J Snoek, H Larochelle, R Adams","year":"2012","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Watanabe, 2023] S. Watanabe. Tree-structured Parzen esti- mator: Understanding its algorithm components and their roles for better empirical performance. arXiv:2304.11127, 2023.","title":"Tree-structured Parzen estimator: Understanding its algorithm components and their roles for better empirical performance","authors":"S Watanabe","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Zimmer et al., 2021] L. Zimmer, M. Lindauer, and F. Hut- ter. Auto-Pytorch: Multi-fidelity metalearning for efficient and robust AutoDL. Transactions on Pattern Analysis and Machine Intelligence, 43, 2021.","title":"Auto-Pytorch: Multi-fidelity metalearning for efficient and robust AutoDL","authors":"L Zimmer, M Lindauer, F Hutter","year":"2021","venue":"Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}