{"type":"ScholarlyArticle","title":"High Dimensional Bayesian Optimization using Dropout","authors":[{"name":"Cheng Li","slug":"cheng-li","given":"Cheng","family":"Li","affiliation":"Centre for Pattern Recognition and Data Analytics (PRaDA) , Deakin University , Australia"},{"name":"Sunil Gupta","slug":"sunil-gupta","given":"Sunil","family":"Gupta","affiliation":"Centre for Pattern Recognition and Data Analytics (PRaDA) , Deakin University , Australia"},{"name":"Santu Rana","slug":"santu-rana","given":"Santu","family":"Rana","affiliation":"Centre for Pattern Recognition and Data Analytics (PRaDA) , Deakin University , Australia"},{"name":"Vu Nguyen","slug":"vu-nguyen","given":"Vu","family":"Nguyen","affiliation":"Centre for Pattern Recognition and Data Analytics (PRaDA) , Deakin University , Australia"},{"name":"Svetha Venkatesh","slug":"svetha-venkatesh","given":"Svetha","family":"Venkatesh","affiliation":"Centre for Pattern Recognition and Data Analytics (PRaDA) , Deakin University , Australia"},{"name":"Alistair Shilton","slug":"alistair-shilton","given":"Alistair","family":"Shilton","affiliation":"Centre for Pattern Recognition and Data Analytics (PRaDA) , Deakin University , Australia"}],"abstract":"Scaling Bayesian optimization to high dimensions is challenging task as the global optimization of high-dimensional acquisition function can be expensive and often infeasible. Existing methods depend either on limited “active” variables or the additive form of the objective function. We propose a new method for high-dimensional Bayesian optimization, that uses a drop-out strategy to optimize only a subset of variables at each iteration. We derive theoretical bounds for the regret and show how it can inform the derivation of our algorithm. We demonstrate the efficacy of our algorithms for optimization on two benchmark functions and two real-world applications - training cascade classifiers and optimizing alloy composition.","doi":"10.24963/ijcai.2017/291","url":"https://ijcai.org/proceedings/2017/291","pdf":"https://ijcai.org/proceedings/2017/0291.pdf","markdown":"https://ijcai.org/proceedings/2017/291.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/291","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":291,"pages":{"first":2096,"last":2102},"keywords":["Machine Learning: Machine Learning","Constraints and Satisfiability: Constraint Optimisation","Machine Learning: Active Learning"],"videos":[],"references":[{"position":1,"raw":"[Djolonga et al., 2013] Josip Djolonga, Andreas Krause, and Volkan Cevher. High-dimensional gaussian process ban- dits. In Advances in Neural Information Processing Sys- tems, pages 1025-1033, 2013.","title":"High-dimensional gaussian process bandits","authors":"Djolonga","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Jones et al., 1993] D. R. Jones, C. D. Perttunen, and B. E. Stuckman. Lipschitzian optimization without the lipschitz constant. Journal of Optimization Theory and Applica- tions, 79(1):157-181, 1993.","title":"Lipschitzian optimization without the lipschitz constant","authors":"D R Jones, C D Perttunen, B E Stuckman","year":"1993","venue":"Journal of Optimization Theory and Applications","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Jones et al., 1998] Donald R. Jones, Matthias Schonlau, and William J. Welch. Efficient global optimization of expen- sive black-box functions. Journal of Global Optimization, 13(4):455-492, 1998.","title":"Efficient global optimization of expensive black-box functions","authors":"Donald R Jones, Matthias Schonlau, William J Welch","year":"1998","venue":"Journal of Global Optimization","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Kandasamy et al., 2015] Kirthevasan Kandasamy, Jeff G. Schneider, and Barnabs Pczos. High Dimensional Bayesian Optimisation and Bandits via Additive Models. In ICML, volume 37, pages 295-304, 2015.","title":"High Dimensional Bayesian Optimisation and Bandits via Additive Models","authors":"Kandasamy","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Koch et al., 1999] T. W. Koch, P. N.and Simpson, J. K. Allen, and F. Mistree. Statistical approximations for mul- tidisciplinary design optimization: The problem of size. Journal of Aircraft, 36(1):275-286, 1999.","title":"Statistical approximations for multidisciplinary design optimization: The problem of size","authors":"T W Koch, P N Simpson, J K Allen, F Mistree","year":"1999","venue":"Journal of Aircraft","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Li et al., 2016] C. Li, K. Kandasamy, B. Poczos, and J. Schneider. High dimensional bayesian optimization via restricted projection pursuit models. In AISTATS, pages 1-9, 2016.","title":"High dimensional bayesian optimization via restricted projection pursuit models","authors":"C Li, K Kandasamy, B Poczos, J Schneider","year":"2016","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Mockus et al., 1978] J. Mockus, V. Tiesis, and A. Zilin- skas. The application of bayesian methods for seeking the extremum. Towards Global Optimisation, (2):117-129, 1978.","title":"The application of bayesian methods for seeking the extremum","authors":"J Mockus, V Tiesis, A Zilinskas","year":"1978","venue":"Towards Global Optimisation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Nesterov, 2012] Yurii Nesterov. Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems. SIAM Journal on Optimization, 22(2), 2012.","title":"Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems","authors":"Yurii Nesterov","year":"2012","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Nguyen et al., 2016] V. Nguyen, S. Rana, S. K. Gupta, C. Li, and S. Venkatesh. Budgeted batch bayesian opti- mization. In 2016 IEEE 16th International Conference on Data Mining (ICDM), pages 1107-1112, Dec 2016.","title":"Budgeted batch bayesian optimization","authors":"V Nguyen, S Rana, S K Gupta, C Li, S Venkatesh","year":"2016","venue":"IEEE 16th International Conference on Data Mining (ICDM)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Qian et al., 2016] Hong Qian, Yi-Qi Hu, and Yang Yu. Derivative-free optimization of high-dimensional non- convex functions by sequential random embeddings. In Proceedings of the 25th International Joint Conference on Artificial Intelligence, 2016.","title":"Derivative-free optimization of high-dimensional nonconvex functions by sequential random embeddings","authors":"Qian","year":"2016","venue":"Proceedings of the 25th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Rana et al., 2017] Santu Rana, Cheng Li, Sunil Gupta, Vu Nguyen, and Svetha Venkatesh. High dimensional bayesian optimization with elastic gaussian process. In International Conference on Machine learning, Sydney, 2017.","title":"High dimensional bayesian optimization with elastic gaussian process","authors":"Santu Rana, Cheng Li, Sunil Gupta, Vu Nguyen, Svetha Venkatesh","year":"2017","venue":"International Conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Rasmussen and Williams, 2005] Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2005.","title":"Gaussian Processes for Machine Learning","authors":"Carl Edward Rasmussen, Christopher K I Williams","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Snoek et al., 2012] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of ma- chine learning algorithms. In NIPS, pages 2951-2959, 2012.","title":"Practical bayesian optimization of machine learning algorithms","authors":"Snoek","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Srinivas et al., 2010] Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process op- timization in the bandit setting: No regret and experimen- tal design. In ICML, 2010.","title":"Gaussian process optimization in the bandit setting: No regret and experimental design","authors":"Niranjan Srinivas, Andreas Krause, Sham Kakade, Matthias Seeger","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1929-1958, January 2014.","title":"Dropout: A simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ulmasov et al., 2016] Doniyor Ulmasov, Caroline Baroukh, Benoit Chachuat, Marc P. Deisenroth, and Ruth Mis- ener. Proceedings of the european symposium on com- puter aided process engineering. In Bayesian Optimiza- tion with Dimension Scheduling: Application to Biologi- cal Systems, 2016.","title":"Doniyor Ulmasov","authors":"Ulmasov","year":"2016","venue":"Bayesian Optimization with Dimension Scheduling: Application to Biological Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Viola and Jones, 2001] Paul Viola and Michael Jones. Rapid object detection using a boosted cascade of sim- ple features. In Computer Vision and Pattern Recognition, pages 511-518, 2001.","title":"Rapid object detection using a boosted cascade of simple features","authors":"Jones Viola, Paul Viola, Michael Jones","year":"2001","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2013] Ziyu Wang, Masrour Zoghi, Frank Hut- ter, David Matheson, and Nando De Freitas. Bayesian op- timization in high dimensions via random embeddings. In IJCAI, pages 1778-1784, 2013.","title":"Bayesian optimization in high dimensions via random embeddings","authors":"Ziyu Wang, Masrour Zoghi, Frank Hutter, David Matheson, Nando De Freitas","year":"2013","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2013/263"},{"position":19,"raw":"[Xue et al., 2016] Dezhen Xue, Prasanna V. Balachandran, John Hogden, James Theiler, Deqing Xue, and Turab Lookman. Accelerated search for materials with targeted properties by adaptive design. Nature communications, 7, 2016.","title":"Accelerated search for materials with targeted properties by adaptive design","authors":"Dezhen Xue, V Prasanna, John Balachandran, James Hogden, Deqing Theiler, Turab Xue, Lookman","year":"2016","venue":"Nature communications","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}