{"type":"ScholarlyArticle","title":"Experienced Optimization with Reusable Directional Model for Hyper-Parameter Search","authors":[{"name":"Yi-Qi Hu","slug":"yi-qi-hu","given":"Yi-Qi","family":"Hu","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"Yang Yu","slug":"yang-yu","given":"Yang","family":"Yu","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"Zhi-Hua Zhou","slug":"zhi-hua-zhou","given":"Zhi-Hua","family":"Zhou","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"}],"abstract":"Hyper-parameter selection is a crucial yet difficult issue in machine learning. For this problem, derivative-free optimization has being playing an irreplaceable role. However, derivative-free optimization commonly requires a lot of hyper-parameter samples, while each sample could have a high cost for hyper-parameter selection due to the costly evaluation of a learning model. To tackle this issue, in this paper, we propose an experienced optimization approach, i.e., learning how to optimize better from a set of historical optimization processes. From the historical optimization processes on previous datasets, a directional model is trained to predict the direction of the next good hyper-parameter. The directional model is then reused to guide the optimization in learning new datasets. We implement this mechanism within a state-of-the-art derivative-free optimization method SRacos, and conduct experiments on  learning the hyper-parameters of heterogeneous ensembles and neural network architectures. Experimental results verify that the proposed approach can significantly improve the learning accuracy within a limited hyper-parameter sample budget.","doi":"10.24963/ijcai.2018/315","url":"https://ijcai.org/proceedings/2018/315","pdf":"https://ijcai.org/proceedings/2018/0315.pdf","markdown":"https://ijcai.org/proceedings/2018/315.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/315","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":315,"pages":{"first":2276,"last":2282},"keywords":["Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Adankon and Cheriet, 2009] Mathias M. Adankon and Mo- hamed Cheriet. Model selection for the LS-SVM. Appli- cation to handwriting recognition. Pattern Recognition, 42(12):3264-3270, 2009.","title":"Model selection for the LS-SVM. Application to handwriting recognition","authors":"Mathias M Adankon, Mohamed Cheriet","year":"2009","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baker et al., 2017] Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network ar- chitectures using reinforcement learning. In Proceedings of 5th International Conference on Learning Representa- tions (ICLR'17), 2017.","title":"Designing neural network architectures using reinforcement learning","authors":"Bowen Baker, Otkrist Gupta, Nikhil Naik, Ramesh Raskar","year":"2017","venue":"Proceedings of 5th International Conference on Learning Representations (ICLR'17)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio, 2000] Yoshua Bengio. Gradient-based opti- mization of hyperparameters. Neural computation, 12(8):1889-1900, 2000.","title":"Gradient-based optimization of hyperparameters","authors":"Yoshua Bengio","year":"2000","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bergstra and Bengio, 2012] James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimiza- tion. Journal of Machine Learning Research, 13:281-305, 2012.","title":"Random search for hyper-parameter optimization","authors":"Bengio Bergstra, James Bergstra, Yoshua Bengio","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bergstra et al., 2011] James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameter optimization. In Advances in Neural Infor- mation Processing Systems (NIPS'11), pages 2546-2554, 2011.","title":"Algorithms for hyperparameter optimization","authors":"Rémi James S Bergstra, Yoshua Bardenet, Balázs Bengio, Kégl","year":"2011","venue":"Advances in Neural Information Processing Systems (NIPS'11)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Biem, 2003] Alain Biem. A model selection criterion for classification: Application to hmm topology optimiza- tion. In Proceedings of the 7th International Conference on Document Analysis and Recognition, pages 104-108, 2003.","title":"A model selection criterion for classification: Application to hmm topology optimization","authors":"Alain Biem","year":"2003","venue":"Proceedings of the 7th International Conference on Document Analysis and Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Brazdil et al., 2003] Pavel B Brazdil, Carlos Soares, and Joaquim Pinto Da Costa. Ranking learning algorithms: Using IBL and meta-learning on accuracy and time results. Machine Learning, 50(3):251-277, 2003.","title":"Ranking learning algorithms: Using IBL and meta-learning on accuracy and time results","authors":"Carlos Pavel B Brazdil, Joaquim Soares, Da Pinto, Costa","year":"2003","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Feurer et al., 2015] Matthias Feurer, Aaron Klein, Katha- rina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems (NIPS'15), pages 2962-2970, 2015.","title":"Efficient and robust automated machine learning","authors":"Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, Frank Hutter","year":"2015","venue":"Advances in Neural Information Processing Systems (NIPS'15)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hall et al., 2009] Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Wit- ten. The WEKA data mining software: an update. ACM SIGKDD Explorations Newsletter, 11(1):10-18, 2009.","title":"Witten. The WEKA data mining software: an update","authors":"Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, Ian H","year":"2009","venue":"ACM SIGKDD Explorations Newsletter","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hansen et al., 2003] Nikolaus Hansen, Sibylle D Müller, and Petros Koumoutsakos. Reducing the time complex- ity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evolutionary Computation, 11(1):1-18, 2003.","title":"Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES)","authors":"Nikolaus Hansen, Sibylle D Müller, Petros Koumoutsakos","year":"2003","venue":"Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hu et al., 2017] Yi-Qi Hu, Hong Qian, and Yang Yu. Se- quential classification-based optimization for direct policy search. In Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI'17), pages 2029-2035, 2017.","title":"Sequential classification-based optimization for direct policy search","authors":"Yi-Qi Hu, Hong Qian, Yang Yu","year":"2017","venue":"Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI'17)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hutter et al., 2011] Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. Sequential model-based optimiza- tion for general algorithm configuration. LION, 5:507- 523, 2011.","title":"Sequential model-based optimization for general algorithm configuration","authors":"Frank Hutter, H Holger, Kevin Hoos, Leyton-Brown","year":"2011","venue":"LION","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Lecun","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lindauer and Hutter, 2018] Marius Lindauer and Frank Hutter. Warmstarting of model-based algorithm config- uration. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI'18), pages 1355-1362, 2018.","title":"Warmstarting of model-based algorithm configuration","authors":"Hutter Lindauer, Marius Lindauer, Frank Hutter","year":"2018","venue":"Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI'18)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Munos, 2011] Rémi Munos. Optimistic optimization of a deterministic function without the knowledge of its smoothness. In Advances in Neural Information Process- ing Systems (NIPS'11), pages 783-791, 2011.","title":"Optimistic optimization of a deterministic function without the knowledge of its smoothness","authors":"Rémi Munos, Munos","year":"2011","venue":"Advances in Neural Information Processing Systems (NIPS'11)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Netzer et al., 2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learn- ing. In NIPS workshop on Deep Learning and Unsuper- vised Feature Learning, volume 2011, page 5, 2011.","title":"Reading digits in natural images with unsupervised feature learning","authors":"Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, Andrew Y Ng","year":"2011","venue":"NIPS workshop on Deep Learning and Unsupervised Feature Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Orr and Müller, 2012] Genevieve B Orr and Klaus-Robert Müller. Neural Networks: Tricks of the Trade -Second Edition. Springer, 2012.","title":"Neural Networks: Tricks of the Trade -Second Edition","authors":"Genevieve B Orr, Klaus-Robert Müller","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pedregosa et al., 2011] Fabian Pedregosa, Gael Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Pas- sos, David Cournapeau, Matthieu Brucher, Matthieu Per- rot, and Edouard Duchesnay. Scikit-learn: Machine learn- ing in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.","title":"Scikit-learn: Machine learning in Python","authors":"Fabian Pedregosa, Gael Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, Edouard Duchesnay","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shahriari et al., 2015] Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando De Freitas. Taking the human out of the loop: A review of Bayesian optimiza- tion. Proceedings of the IEEE, 104(1):148-175, 2015.","title":"Taking the human out of the loop: A review of Bayesian optimization","authors":"Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, Nando De Freitas","year":"2015","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Springenberg et al., 2015] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Ried- miller. Striving for simplicity: The all convolutional net. In Proceedings of 3rd International Conference on Learning Representations (ICLR'15), 2015.","title":"Striving for simplicity: The all convolutional net","authors":"Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, Martin Riedmiller","year":"2015","venue":"Proceedings of 3rd International Conference on Learning Representations (ICLR'15)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Swersky et al., 2013] Kevin Swersky, Jasper Snoek, and Ryan P. Adams. Multi-task bayesian optimization. In Advances in Neural Information Processing Systems (NIPS'13), pages 2004-2012, 2013.","title":"Multi-task bayesian optimization","authors":"Kevin Swersky, Jasper Snoek, Ryan P Adams","year":"2013","venue":"Advances in Neural Information Processing Systems (NIPS'13)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Thornton et al., 2013] Chris Thornton, Frank Hutter, Hol- ger H Hoos, and Kevin Leyton-Brown. Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discov- ery and Data Mining (SIGKDD'13), pages 847-855, 2013.","title":"Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms","authors":"Chris Thornton, Frank Hutter, Kevin Holger H Hoos, Leyton-Brown","year":"2013","venue":"Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (SIGKDD'13)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yu et al., 2016] Yang Yu, Hong Qian, and Yi-Qi Hu. Derivative-free optimization via classification. In Proceed- ings of the 30th AAAI Conference on Artificial Intelligence (AAAI'16), pages 2286-2292, 2016.","title":"Derivative-free optimization via classification","authors":"Yang Yu, Hong Qian, Yi-Qi Hu","year":"2016","venue":"Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI'16)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhou, 2016] Zhi-Hua Zhou. Learnware: On the future of machine learning. Frontiers of Computer Science, 10(4):589-590, 2016.","title":"Learnware: On the future of machine learning","authors":"Zhi-Hua Zhou","year":"2016","venue":"Frontiers of Computer Science","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}