{"type":"ScholarlyArticle","title":"High Dimensional Bayesian Optimization via Supervised Dimension Reduction","authors":[{"name":"Miao Zhang","slug":"miao-zhang","given":"Miao","family":"Zhang","affiliation":"School of Information and Electronics , Beijing Institute of Technology , China; Faculty of Engineering and Information Technology , University of Technology Sydney , Australia"},{"name":"Huiqi Li","slug":"huiqi-li","given":"Huiqi","family":"Li","affiliation":"School of Information and Electronics , Beijing Institute of Technology , China"},{"name":"Steven Su","slug":"steven-su","given":"Steven","family":"Su","affiliation":"Faculty of Engineering and Information Technology , University of Technology Sydney , Australia"}],"abstract":"Bayesian optimization (BO) has been broadly applied to computational expensive problems, but it is still challenging to extend BO to high dimensions. Existing works are usually under strict assumption of an additive or a linear embedding structure for objective functions. This paper directly introduces a supervised dimension reduction method, Sliced Inverse Regression (SIR), to high dimensional Bayesian optimization, which could effectively learn the intrinsic sub-structure of objective function during the optimization. Furthermore, a kernel trick is developed to reduce computational complexity and learn nonlinear subset of the unknowing function when applying SIR to extremely high dimensional BO. We present several computational benefits and derive theoretical regret bounds of our algorithm. Extensive experiments on synthetic examples and two real applications demonstrate the superiority of our algorithms for high dimensional Bayesian optimization.","doi":"10.24963/ijcai.2019/596","url":"https://ijcai.org/proceedings/2019/596","pdf":"https://ijcai.org/proceedings/2019/0596.pdf","markdown":"https://ijcai.org/proceedings/2019/596.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/596","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":596,"pages":{"first":4292,"last":4298},"keywords":["Machine Learning: Kernel Methods","Computer Vision: Statistical Methods and Machine Learning","Heuristic Search and Game Playing: Heuristic Search and Machine Learning","Machine Learning: Probabilistic Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Baptista and Poloczek, 2018] Ricardo Baptista and Matthias Poloczek. Bayesian optimization of com- binatorial structures. In International Conference on Machine Learning, pages 462-471, 2018.","title":"Bayesian optimization of combinatorial structures","authors":"Ricardo Baptista, Matthias Poloczek","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bergstra et al., 2011] James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameter optimization. In Advances in neural informa- tion processing systems, pages 2546-2554, 2011.","title":"Algorithms for hyperparameter optimization","authors":"Rémi James S Bergstra, Yoshua Bardenet, Balázs Bengio, Kégl","year":"2011","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2012] Bo Chen, Rui M. Castro, and Andreas Krause. Joint optimization and variable selection of high- dimensional gaussian processes. pages 1423-1430, 2012.","title":"Joint optimization and variable selection of highdimensional gaussian processes","authors":"Bo Chen, Rui M Castro, Andreas Krause","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cunningham and Ghahramani, 2015] John P Cunningham and Zoubin Ghahramani. Linear dimensionality reduction: Survey, insights, and generalizations. The Journal of Ma- chine Learning Research, 16(1):2859-2900, 2015.","title":"Linear dimensionality reduction: Survey, insights, and generalizations","authors":"P John, Zoubin Cunningham, Ghahramani","year":"2015","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deisenroth et al., 2015] Marc Peter Deisenroth, Dieter Fox, and Carl Edward Rasmussen. Gaussian processes for data-efficient learning in robotics and control. IEEE transactions on pattern analysis and machine intelligence, 37(2):408-423, 2015.","title":"Gaussian processes for data-efficient learning in robotics and control","authors":"Marc Peter Deisenroth, Dieter Fox, Carl Edward Rasmussen","year":"2015","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Djolonga et al., 2013] Josip Djolonga, Andreas Krause, and Volkan Cevher. High-dimensional gaussian process ban- dits. In Advances in Neural Information Processing Sys- tems, pages 1025-1033, 2013.","title":"High-dimensional gaussian process bandits","authors":"Djolonga","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hutter et al., 2014] Frank Hutter, Holger Hoos, and Kevin Leyton-Brown. An efficient approach for assessing hy- perparameter importance. In International Conference on Machine Learning, pages 754-762, 2014.","title":"An efficient approach for assessing hyperparameter importance","authors":"Frank Hutter, Holger Hoos, Kevin Leyton-Brown","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kandasamy et al., 2015] Kirthevasan Kandasamy, Jeff Schneider, and Barnabás Póczos. High dimensional bayesian optimisation and bandits via additive models. In International Conference on Machine Learning, pages 295-304, 2015.","title":"High dimensional bayesian optimisation and bandits via additive models","authors":"Kandasamy","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kandasamy et al., 2018] Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing. Neural architecture search with bayesian optimisa- tion and optimal transport. In Advances in Neural Infor- mation Processing Systems, pages 2020-2029, 2018.","title":"Neural architecture search with bayesian optimisation and optimal transport","authors":"Kandasamy","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2016] Chun-Liang Li, Kirthevasan Kandasamy, Barnabás Póczos, and Jeff Schneider. High dimensional bayesian optimization via restricted projection pursuit models. In Artificial Intelligence and Statistics, pages 884-892, 2016.","title":"High dimensional bayesian optimization via restricted projection pursuit models","authors":"Chun-Liang Li, Kirthevasan Kandasamy, Barnabás Póczos, Jeff Schneider","year":"2016","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2017] Cheng Li, Sunil Gupta, Santu Rana, Vu Nguyen, Svetha Venkatesh, and Alistair Shilton. High dimensional bayesian optimization using dropout. In Pro- ceedings of the 26th International Joint Conference on Ar- tificial Intelligence, pages 2096-2102. AAAI Press, 2017.","title":"High dimensional bayesian optimization using dropout","authors":"Cheng Li, Sunil Gupta, Santu Rana, Svetha Vu Nguyen, Alistair Venkatesh, Shilton","year":"2017","venue":"Proceedings of the 26th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2017/291"},{"position":12,"raw":"[Li, 1991] Ker-Chau Li. Sliced inverse regression for dimen- sion reduction. Journal of the American Statistical Asso- ciation, 86(414):316-327, 1991.","title":"Sliced inverse regression for dimension reduction","authors":"Ker-Chau Li","year":"1991","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lizotte, 2008] Daniel James Lizotte. Practical bayesian op- timization. PhD thesis, University of Alberta, 2008.","title":"Practical bayesian optimization","authors":"Daniel James, Lizotte","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rasmussen and Williams, 2004] Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, Cambridge, Mas- sachusetts, 2004.","title":"Gaussian Processes for Machine Learning","authors":"Williams ; Rasmussen, Carl Edward Rasmussen, Christopher K I Williams","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rolland et al., 2018] Paul Rolland, Jonathan Scarlett, Il- ija Bogunovic, and Volkan Cevher. High-dimensional bayesian optimization via additive models with overlap- ping groups. In International Conference on Artificial In- telligence and Statistics, pages 298-307, 2018.","title":"High-dimensional bayesian optimization via additive models with overlapping groups","authors":"Paul Rolland, Jonathan Scarlett, Ilija Bogunovic, Volkan Cevher","year":"2018","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Snoek et al., 2012] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of ma- chine learning algorithms. In Advances in neural informa- tion processing systems, pages 2951-2959, 2012.","title":"Practical bayesian optimization of machine learning algorithms","authors":"Snoek","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Srinivas et al., 2012] Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias W Seeger. Information- theoretic regret bounds for gaussian process optimization in the bandit setting. IEEE Transactions on Information Theory, 58(5):3250-3265, 2012.","title":"Informationtheoretic regret bounds for gaussian process optimization in the bandit setting","authors":"Niranjan Srinivas, Andreas Krause, Matthias W Sham M Kakade, Seeger","year":"2012","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tan and Mukherjee, 2018] Zilong Tan and Sayan Mukher- jee. Subspace-induced gaussian processes. arXiv preprint arXiv:1802.07528, 2018.","title":"Zilong Tan and Sayan Mukherjee. Subspace-induced gaussian processes","authors":"Mukherjee Tan","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ulmasov et al., 2016] Doniyor Ulmasov, Caroline Baroukh, Benoit Chachuat, Marc Peter Deisenroth, and Ruth Mis- ener. Bayesian optimization with dimension scheduling: Application to biological systems. In Computer Aided Chemical Engineering, volume 38, pages 1051-1056.","title":"Bayesian optimization with dimension scheduling: Application to biological systems","authors":"Ulmasov","year":"2016","venue":"Computer Aided Chemical Engineering","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2013] Ziyu Wang, Masrour Zoghi, Frank Hut- ter, David Matheson, Nando De Freitas, et al. Bayesian optimization in high dimensions via random embeddings. In IJCAI, pages 1778-1784, 2013.","title":"Bayesian optimization in high dimensions via random embeddings","authors":"Ziyu Wang, Masrour Zoghi, Frank Hutter, David Matheson, Nando De Freitas","year":"2013","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2013/263"},{"position":21,"raw":"[Wang et al., 2017] Zi Wang, Chengtao Li, Stefanie Jegelka, and Pushmeet Kohli. Batched high-dimensional bayesian optimization via structural kernel learning. In Interna- tional Conference on Machine Learning, pages 3656- 3664, 2017.","title":"Batched high-dimensional bayesian optimization via structural kernel learning","authors":"Zi Wang, Chengtao Li, Stefanie Jegelka, Pushmeet Kohli","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Westervelt et al., 2007] Eric R Westervelt, Christine Chevallereau, Jun Ho Choi, Benjamin Morris, and Jessy W Grizzle. Feedback control of dynamic bipedal robot locomotion. CRC press, 2007.","title":"Feedback control of dynamic bipedal robot locomotion","authors":"Christine Eric R Westervelt, Jun Chevallereau, Benjamin Ho Choi, Jessy W Morris, Grizzle","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2007] Qiang Wu, F Liang, and S Mukherjee. Regularized sliced inverse regression for kernel models. Technical report, Technical Report 07-25, ISDS, Duke Univ, 2007.","title":"Regularized sliced inverse regression for kernel models","authors":"Qiang Wu, F Liang, Mukherjee","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu, 2008] Han-Ming Wu. Kernel sliced inverse regression with applications to classification. Journal of Computa- tional and Graphical Statistics, 17(3):590-610, 2008.","title":"Kernel sliced inverse regression with applications to classification","authors":"Han-Ming Wu","year":"2008","venue":"Journal of Computational and Graphical Statistics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yeh et al., 2009] Yi-Ren Yeh, Su-Yun Huang, and Yuh-Jye Lee. Nonlinear dimension reduction with kernel sliced in- verse regression. IEEE Transactions on Knowledge and Data Engineering, 21(11):1590-1603, 2009.","title":"Nonlinear dimension reduction with kernel sliced inverse regression","authors":"Yi-Ren Yeh, Su-Yun Huang, Yuh-Jye Lee","year":"2009","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}