{"type":"ScholarlyArticle","title":"Mixed-Variable Bayesian Optimization","authors":[{"name":"Erik Daxberger","slug":"erik-daxberger","given":"Erik","family":"Daxberger","affiliation":"Department of Engineering , University of Cambridge; Max Planck Institute for Intelligent Systems , Tübingen"},{"name":"Anastasia Makarova","slug":"anastasia-makarova","given":"Anastasia","family":"Makarova","affiliation":"Department of Computer Science , ETH Zurich"},{"name":"Matteo Turchetta","slug":"matteo-turchetta","given":"Matteo","family":"Turchetta","affiliation":"Max Planck Institute for Intelligent Systems , Tübingen; Department of Computer Science , ETH Zurich"},{"name":"Andreas Krause","slug":"andreas-krause","given":"Andreas","family":"Krause","affiliation":"Department of Computer Science , ETH Zurich"}],"abstract":"The optimization of expensive to evaluate, black-box, mixed-variable functions, i.e. functions that have continuous and discrete inputs, is a difficult and yet pervasive problem in science and engineering. In Bayesian optimization (BO), special cases of this problem that consider fully continuous or fully discrete domains have been widely studied. However, few methods exist for mixed-variable domains and none of them can handle discrete constraints that arise in many real-world applications. In this paper, we introduce MiVaBo, a novel BO algorithm for the efficient optimization of mixed-variable functions combining a linear surrogate model based on expressive feature representations with Thompson sampling. We propose an effective method to optimize its acquisition function, a challenging problem for mixed-variable domains, making MiVaBo the first BO method that can handle complex constraints over the discrete variables. Moreover, we provide the first convergence analysis of a mixed-variable BO algorithm. Finally, we show that MiVaBo is significantly more sample efficient than state-of-the-art mixed-variable BO algorithms on several hyperparameter tuning tasks, including the tuning of deep generative models.","doi":"10.24963/ijcai.2020/365","url":"https://ijcai.org/proceedings/2020/365","pdf":"https://ijcai.org/proceedings/2020/0365.pdf","markdown":"https://ijcai.org/proceedings/2020/365.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/365","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":365,"pages":{"first":2633,"last":2639},"keywords":["Machine Learning: Bayesian Optimization","Machine Learning: Probabilistic Machine Learning","Machine Learning: Active Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25555/file_25555.m3u8","length_sec":298,"provider":"hls"}],"references":[{"position":1,"raw":"[Abeille et al., 2017] Marc Abeille, Alessandro Lazaric, et al. Linear Thompson sampling revisited. EJS, 2017.","title":"Linear Thompson sampling revisited","authors":"Marc Abeille, Alessandro Lazaric","year":"2017","venue":"EJS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baptista and Poloczek, 2018] Ricardo Baptista and Matthias Poloczek. Bayesian optimization of combi- natorial structures. In ICML, 2018.","title":"Bayesian optimization of combinatorial structures","authors":"Ricardo Baptista, Matthias Poloczek","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bergstra et al., 2011] James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameter optimization. In NIPS, 2011.","title":"Algorithms for hyperparameter optimization","authors":"Rémi James S Bergstra, Yoshua Bardenet, Balázs Bengio, Kégl","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Boros and Hammer, 2002] Endre Boros and Peter L Ham- mer. Pseudo-boolean optimization. Discrete applied math- ematics, 123(1-3):155-225, 2002.","title":"Pseudo-boolean optimization","authors":"Endre Boros, Peter L Hammer","year":"2002","venue":"Discrete applied mathematics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Burda et al., 2016] Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In ICLR, 2016.","title":"Importance weighted autoencoders","authors":"Yuri Burda, Roger Grosse, Ruslan Salakhutdinov","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen and Guestrin, 2016] Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In KDD, 2016.","title":"XGBoost: A scalable tree boosting system","authors":"Tianqi Chen, Carlos Guestrin","year":"2016","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Eggensperger et al., 2015] Katharina Eggensperger, Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Effi- cient benchmarking of hyperparameter optimizers via sur- rogates. In AAAI, 2015.","title":"Efficient benchmarking of hyperparameter optimizers via surrogates","authors":"Katharina Eggensperger, Frank Hutter, Kevin Holger H Hoos, Leyton-Brown","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Falkner et al., 2018] Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparam- eter optimization at scale. In ICML, 2018.","title":"BOHB: Robust and efficient hyperparameter optimization at scale","authors":"Stefan Falkner, Aaron Klein, Frank Hutter","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gardner et al., 2014] Jacob R. Gardner, Matt J. Kusner, Zhixiang Xu, Kilian Q. Weinberger, and John P. Cunning- ham. Bayesian optimization with inequality constraints. In ICML, 2014.","title":"Bayesian optimization with inequality constraints","authors":"Jacob R Gardner, Matt J Kusner, Zhixiang Xu, Kilian Q Weinberger, John P Cunningham","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Garrido-Merchán and Hernández-Lobato, 2018] Eduardo C Garrido-Merchán and Daniel Hernández-Lobato. Dealing with categorical and integer-valued variables in bayesian optimization with gaussian processes. CoRR, 2018.","title":"Dealing with categorical and integer-valued variables in bayesian optimization with gaussian processes","authors":"Hernández-Lobato ; Garrido-Merchán, C Eduardo, Daniel Garrido-Merchán, Hernández-Lobato","year":"2018","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hastie, 2017] Trevor J Hastie. Generalized additive models. In Statistical models in S. Routledge, 2017.","title":"Generalized additive models","authors":"Trevor J Hastie","year":"2017","venue":"Statistical models in S","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hazan et al., 2017] Elad Hazan, Adam Klivans, and Yang Yuan. Hyperparameter optimization: A spectral approach. In ICRL, 2017.","title":"Hyperparameter optimization: A spectral approach","authors":"Elad Hazan, Adam Klivans, Yang Yuan","year":"2017","venue":"ICRL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hernández-Lobato et al., 2015] José Miguel Hernández- Lobato, Michael A Gelbart, Matthew W Hoffman, Ryan P Adams, and Zoubin Ghahramani. Predictive entropy search for bayesian optimization with unknown con- straints. JMLR, 2015.","title":"Predictive entropy search for bayesian optimization with unknown constraints","authors":"Hernández-Lobato","year":"2015","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hutter et al., 2011] Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Sequential model-based optimiza- tion for general algorithm configuration. In LION, 2011.","title":"Sequential model-based optimization for general algorithm configuration","authors":"Frank Hutter, Kevin Holger H Hoos, Leyton-Brown","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jenatton et al., 2017] Rodolphe Jenatton, Cedric Archam- beau, Javier González, and Matthias Seeger. Bayesian optimization with tree-structured dependencies. In ICML, 2017.","title":"Bayesian optimization with tree-structured dependencies","authors":"Rodolphe Jenatton, Cedric Archambeau, Javier González, Matthias Seeger","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kingma and Welling, 2014] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Krause et al., 2008] Andreas Krause, Ajit Singh, and Car- los Guestrin. Near-optimal sensor placements in Gaussian processes: Theory, efficient algorithms and empirical stud- ies. JMLR, 2008.","title":"Near-optimal sensor placements in Gaussian processes: Theory, efficient algorithms and empirical studies","authors":"Andreas Krause, Ajit Singh, Carlos Guestrin","year":"2008","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2018] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter opti- mization. JMLR, 2018.","title":"Hyperband: A novel bandit-based approach to hyperparameter optimization","authors":"Lisha Li, Kevin Jamieson, Giulia Desalvo, Afshin Rostamizadeh, Ameet Talwalkar","year":"2018","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Močkus, 1975] Jonas Močkus. On Bayesian methods for seeking the extremum. In Optimization Techniques, 1975.","title":"Jonas Močkus. On Bayesian methods for seeking the extremum","authors":"Močkus","year":"1975","venue":"Optimization Techniques","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mutný and Krause, 2018] Mojmir Mutný and Andreas Krause. Efficient high dimensional bayesian optimization with additivity and quadrature fourier features. In NIPS, 2018.","title":"Mojmir Mutný and Andreas Krause. Efficient high dimensional bayesian optimization with additivity and quadrature fourier features","authors":"Krause Mutný","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Negoescu et al., 2011] Diana M Negoescu, Peter I Frazier, and Warren B Powell. The knowledge-gradient algorithm for sequencing experiments in drug discovery. INFORMS, 2011.","title":"The knowledge-gradient algorithm for sequencing experiments in drug discovery","authors":"Diana M Negoescu, Peter I Frazier, Warren B Powell","year":"2011","venue":"INFORMS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Oh et al., 2019] Changyong Oh, Jakub M. Tomczak, Efstra- tios Gavves, and Max Welling. Combinatorial bayesian optimization using graph representations. NeurIPS, 2019.","title":"Combinatorial bayesian optimization using graph representations","authors":"Changyong Oh, M Jakub, Efstratios Tomczak, Max Gavves, Welling","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rahimi and Recht, 2008] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In NIPS, 2008.","title":"Random features for large-scale kernel machines","authors":"Recht Rahimi, Ali Rahimi, Benjamin Recht","year":"2008","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rolland et al., 2018] Paul Rolland, Jonathan Scarlett, Il- ija Bogunovic, and Volkan Cevher. High-dimensional Bayesian optimization via additive models with overlap- ping groups. In AISTATS, 2018.","title":"High-dimensional Bayesian optimization via additive models with overlapping groups","authors":"Paul Rolland, Jonathan Scarlett, Ilija Bogunovic, Volkan Cevher","year":"2018","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Salimans et al., 2015] Tim Salimans, Diederik P Kingma, and Max Welling. Markov chain monte carlo and varia- tional inference: Bridging the gap. In ICML, 2015.","title":"Markov chain monte carlo and variational inference: Bridging the gap","authors":"Tim Salimans, P Diederik, Max Kingma, Welling","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shahriari et al., 2016] Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. Taking the human out of the loop: A review of Bayesian optimiza- tion. IEEE, 2016.","title":"Taking the human out of the loop: A review of Bayesian optimization","authors":"Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, Nando De Freitas","year":"2016","venue":"IEEE","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sontag et al., 2011] David Sontag, Amir Globerson, and Tommi Jaakkola. Introduction to dual composition for in- ference. In Optimization for Machine Learning. 2011.","title":"Introduction to dual composition for inference","authors":"David Sontag, Tommi Amir Globerson, Jaakkola","year":"2011","venue":"Optimization for Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Srinivas et al., 2010] Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experi- mental design. In ICML, 2010.","title":"Gaussian process optimization in the bandit setting: No regret and experimental design","authors":"Niranjan Srinivas, Andreas Krause, Matthias Sham M Kakade, Seeger","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sui et al., 2015] Yanan Sui, Alkis Gotovos, Joel Burdick, and Andreas Krause. Safe exploration for optimization with gaussian processes. In ICML, 2015.","title":"Safe exploration for optimization with gaussian processes","authors":"Sui","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Thompson, 1933] William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 1933.","title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples","authors":"William R Thompson","year":"1933","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Vanschoren et al., 2014] Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo. OpenML: networked science in machine learning. ACM SIGKDD, 15(2):49-60, 2014.","title":"OpenML: networked science in machine learning","authors":"Vanschoren","year":"2014","venue":"ACM SIGKDD","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Williams and Rasmussen, 2006] Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for ma- chine learning. MIT Press Cambridge, MA, 2006.","title":"Gaussian processes for machine learning","authors":"K I Christopher, Carl Edward Williams, Rasmussen","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}