{"type":"ScholarlyArticle","title":"Speeding Up Multi-Objective Hyperparameter Optimization by Task Similarity-Based Meta-Learning for the Tree-Structured Parzen Estimator","authors":[{"name":"Shuhei Watanabe","slug":"shuhei-watanabe","given":"Shuhei","family":"Watanabe","affiliation":"Department of Computer Science , University of Freiburg , Germany"},{"name":"Noor Awad","slug":"noor-awad","given":"Noor","family":"Awad","affiliation":"Department of Computer Science , University of Freiburg , Germany"},{"name":"Masaki Onishi","slug":"masaki-onishi","given":"Masaki","family":"Onishi","affiliation":"Artificial Intelligence Research Center , AIST , Tokyo , Japan"},{"name":"Frank Hutter","slug":"frank-hutter","given":"Frank","family":"Hutter","affiliation":"Department of Computer Science , University of Freiburg , Germany"}],"abstract":"Hyperparameter optimization (HPO) is a vital step in improving performance in deep learning (DL). Practitioners are often faced with the trade-off between multiple criteria, such as accuracy and latency. Given the high computational needs of DL and the growing demand for efficient HPO, the acceleration of multi-objective (MO) optimization becomes ever more important. Despite the significant body of work on meta-learning for HPO, existing methods are inapplicable to MO tree-structured Parzen estimator (MO-TPE), a simple yet powerful MO-HPO algorithm. In this paper, we extend TPE’s acquisition function to the meta-learning setting using a task similarity defined by the overlap of top domains between tasks. We also theoretically analyze and address the limitations of our task similarity. In the experiments, we demonstrate that our method speeds up MO-TPE on tabular HPO benchmarks and attains state-of-the-art performance. Our method was also validated externally by winning the AutoML 2022 competition on “Multiobjective Hyperparameter Optimization for Transformers”. See https://arxiv.org/abs/2212.06751 for the latest version with Appendix.","doi":"10.24963/ijcai.2023/487","url":"https://ijcai.org/proceedings/2023/487","pdf":"https://ijcai.org/proceedings/2023/0487.pdf","markdown":"https://ijcai.org/proceedings/2023/487.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/487","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":487,"pages":{"first":4380,"last":4388},"keywords":["Machine Learning: ML: Hyperparameter optimization","Machine Learning: ML: Automated machine learning","Machine Learning: ML: Meta-learning"],"videos":[],"references":[{"position":1,"raw":"[Bader and Zitzler, 2011] J. Bader and E. Zitzler. HypE: An algorithm for fast hypervolume-based many-objective op- timization. Evolutionary Computation, 19, 2011.","title":"HypE: An algorithm for fast hypervolume-based many-objective optimization","authors":"Zitzler Bader, ] J Bader, E Zitzler","year":"2011","venue":"Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bergstra and Bengio, 2012] J. Bergstra and Y. Bengio. Ran- dom search for hyper-parameter optimization. Journal of Machine Learning Research, 13(2), 2012.","title":"Random search for hyper-parameter optimization","authors":"J Bergstra, Y Bengio","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bergstra et al., 2011] J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for hyper-parameter optimiza- tion. In Advances in Neural Information Processing Sys- tems, 2011.","title":"Algorithms for hyper-parameter optimization","authors":"J Bergstra, R Bardenet, Y Bengio, B Kégl","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bergstra et al., 2013] J. Bergstra, D. Yamins, and D. Cox. Making a science of model search: Hyperparameter op- timization in hundreds of dimensions for vision architec- tures. In International Conference on Machine Learning, 2013.","title":"Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures","authors":"J Bergstra, D Yamins, D Cox","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brochu et al., 2010] E. Brochu, V. Cora, and N. de Fre- itas. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hi- erarchical reinforcement learning. arXiv:1012.2599, 2010.","title":"A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning","authors":"E Brochu, V Cora, N De Freitas","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Candelieri et al., 2022] A. Candelieri, A. Ponti, and F. Archetti. Fair and green hyperparameter optimization via multi-objective and multiple information source Bayesian optimization. arXiv:2205.08835, 2022.","title":"Fair and green hyperparameter optimization via multi-objective and multiple information source Bayesian optimization","authors":"A Candelieri, A Ponti, F Archetti","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2018] Y. Chen, A. Huang, Z. Wang, I. Antonoglou, J. Schrittwieser, D. Silver, and N. de Freitas. Bayesian optimization in AlphaGo. arXiv:1812.06855, 2018.","title":"Bayesian optimization in AlphaGo","authors":"Y Chen, A Huang, Z Wang, I Antonoglou, J Schrittwieser, D Silver, N De Freitas","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chugh et al., 2016] T. Chugh, Y. Jin, K. Miettinen, J. Haka- nen, and K. Sindhya. A surrogate-assisted reference vector guided evolutionary algorithm for computationally expen- sive many-objective optimization. Evolutionary Computa- tion, 22, 2016.","title":"A surrogate-assisted reference vector guided evolutionary algorithm for computationally expensive many-objective optimization","authors":"T Chugh, Y Jin, K Miettinen, J Hakanen, K Sindhya","year":"2016","venue":"Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Deb et al., 2002] K. Deb, A. Pratap, S. Agarwal, and T. Me- yarivan. A fast and elitist multiobjective genetic algorithm: NSGA-II. Evolutionary Computation, 6, 2002.","title":"A fast and elitist multiobjective genetic algorithm: NSGA-II","authors":"K Deb, A Pratap, S Agarwal, T Meyarivan","year":"2002","venue":"Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Eggensperger et al., 2021] K. Eggensperger, P. Müller, N. Mallik, M. Feurer, R. Sass, A. Klein, N. Awad, M. Lin- dauer, and F. Hutter. HPOBench: A collection of repro- ducible multi-fidelity benchmark problems for HPO. In Advances in Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.","title":"HPOBench: A collection of reproducible multi-fidelity benchmark problems for HPO","authors":"K Eggensperger, P Müller, N Mallik, M Feurer, R Sass, A Klein, N Awad, M Lindauer, F Hutter","year":"2021","venue":"Advances in Neural Information Processing Systems Track on Datasets and Benchmarks","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Emmerich et al., 2011] MTM. Emmerich, AH. Deutz, and JW. Klinkenberg. Hypervolume-based expected improve- ment: Monotonicity properties and exact computation. In Congress of Evolutionary Computation, 2011.","title":"Hypervolume-based expected improvement: Monotonicity properties and exact computation","authors":"Emmerich","year":"2011","venue":"Congress of Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Falkner et al., 2018] S. Falkner, A. Klein, and F. Hutter. BOHB: Robust and efficient hyperparameter optimization at scale. In International Conference on Machine Learn- ing, 2018.","title":"BOHB: Robust and efficient hyperparameter optimization at scale","authors":"S Falkner, A Klein, F Hutter","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Feurer et al., 2015] M. Feurer, JT. Springenberg, and F. Hutter. Initializing Bayesian hyperparameter optimiza- tion via meta-learning. In AAAI Conference on Artificial Intelligence, 2015.","title":"Initializing Bayesian hyperparameter optimization via meta-learning","authors":"M Feurer, J T Springenberg, F Hutter","year":"2015","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Feurer et al., 2018] M. Feurer, B. Letham, F. Hutter, and E. Bakshy. Practical transfer learning for Bayesian opti- mization. arXiv:1802.02219, 2018.","title":"Practical transfer learning for Bayesian optimization","authors":"M Feurer, B Letham, F Hutter, E Bakshy","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Fonseca and Fleming, 1996] CM. Fonseca and PJ. Flem- ing. On the performance assessment and comparison of stochastic multiobjective optimizers. In International Conference on Parallel Problem Solving from Nature, 1996.","title":"On the performance assessment and comparison of stochastic multiobjective optimizers","authors":"C M Fonseca, P J Fleming","year":"1996","venue":"International Conference on Parallel Problem Solving from Nature","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Garnett, 2022] R. Garnett. Bayesian Optimization. Cam- bridge University Press, 2022.","title":"Bayesian Optimization","authors":"R Garnett","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Guo et al., 2018] D. Guo, Y. Jin, J. Ding, and T. Chai. Het- erogeneous ensemble-based infill criterion for evolution- ary multiobjective optimization of expensive problems. Transactions on Cybernetics, 49, 2018.","title":"Heterogeneous ensemble-based infill criterion for evolutionary multiobjective optimization of expensive problems","authors":"D Guo, Y Jin, J Ding, T Chai","year":"2018","venue":"Transactions on Cybernetics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Henderson et al., 2018] P. Henderson, R. Islam, P. Bach- man, J. Pineau, D. Precup, and D. Meger. Deep reinforce- ment learning that matters. In AAAI Conference on Artifi- cial Intelligence, 2018.","title":"Deep reinforcement learning that matters","authors":"P Henderson, R Islam, P Bachman, J Pineau, D Precup, D Meger","year":"2018","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Hernández-Lobato et al., 2016] D. Hernández-Lobato, J. Hernandez-Lobato, A. Shah, and R. Adams. Predictive entropy search for multi-objective Bayesian optimization. In International Conference on Machine Learning, 2016.","title":"Predictive entropy search for multi-objective Bayesian optimization","authors":"Hernández-Lobato","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Jones et al., 1998] D. Jones, M. Schonlau, and W. Welch. Efficient global optimization of expensive black-box func- tions. Journal of Global Optimization, 13, 1998.","title":"Efficient global optimization of expensive black-box functions","authors":"D Jones, M Schonlau, W Welch","year":"1998","venue":"Journal of Global Optimization","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Klein and Hutter, 2019] A. Klein and F. Hutter. Tabular benchmarks for joint architecture and hyperparameter op- timization. arXiv:1905.04970, 2019.","title":"Tabular benchmarks for joint architecture and hyperparameter optimization","authors":"A Klein, F Hutter","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Knowles, 2006] J. Knowles. ParEGO: A hybrid algorithm with on-line landscape approximation for expensive mul- tiobjective optimization problems. Evolutionary Compu- tation, 10, 2006.","title":"ParEGO: A hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems","authors":"J Knowles","year":"2006","venue":"Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Kushner, 1964] HJ. Kushner. A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. Joint Automatic Control Conference, 1964.","title":"A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise","authors":"Hj, Kushner","year":"1964","venue":"Joint Automatic Control Conference","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Nomura et al., 2021] M. Nomura, S. Watanabe, Y. Aki- moto, Y. Ozaki, and M. Onishi. Warm starting CMA-ES for hyperparameter optimization. In AAAI Conference on Artificial Intelligence, 2021.","title":"Warm starting CMA-ES for hyperparameter optimization","authors":"M Nomura, S Watanabe, Y Akimoto, Y Ozaki, M Onishi","year":"2021","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ozaki et al., 2020] Y. Ozaki, Y. Tanigaki, S. Watanabe, and M. Onishi. Multiobjective tree-structured Parzen estimator for computationally expensive optimization problems. In Genetic and Evolutionary Computation Conference, 2020.","title":"Multiobjective tree-structured Parzen estimator for computationally expensive optimization problems","authors":"Y Ozaki, Y Tanigaki, S Watanabe, M Onishi","year":"2020","venue":"Genetic and Evolutionary Computation Conference","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ozaki et al., 2022] Y. Ozaki, Y. Tanigaki, S. Watanabe, M. Nomura, and M. Onishi. Multiobjective tree-structured Parzen estimator. Journal of Artificial Intelligence Re- search, 73, 2022.","title":"Multiobjective tree-structured Parzen estimator","authors":"Y Ozaki, Y Tanigaki, S Watanabe, M Nomura, M Onishi","year":"2022","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Pan et al., 2018] L. Pan, C. He, Y. Tian, H. Wang, X. Zhang, and Y. Jin. A classification-based surrogate-assisted evolu- tionary algorithm for expensive many-objective optimiza- tion. Evolutionary Computation, 23, 2018.","title":"A classification-based surrogate-assisted evolutionary algorithm for expensive many-objective optimization","authors":"L Pan, C He, Y Tian, H Wang, X Zhang, Y Jin","year":"2018","venue":"Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Perrone et al., 2018] V. Perrone, R. Jenatton, M. Seeger, and C. Archambeau. Scalable hyperparameter transfer learning. In Advances in Neural Information Processing Systems, 2018.","title":"Scalable hyperparameter transfer learning","authors":"V Perrone, R Jenatton, M Seeger, C Archambeau","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Perrone et al., 2019] V. Perrone, H. Shen, MW. Seeger, C. Archambeau, and R. Jenatton. Learning search spaces for Bayesian optimization: Another view of hyperparame- ter transfer learning. Advances in Neural Information Pro- cessing Systems, 2019.","title":"Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning","authors":"V Perrone, H Shen, M W Seeger, C Archambeau, R Jenatton","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Ponweiser et al., 2008] W. Ponweiser, T. Wagner, D. Bier- mann, and M. Vincze. Multiobjective optimization on a limited budget of evaluations using model-assisted S- metric selection. In International Conference on Parallel Problem Solving from Nature, 2008.","title":"Multiobjective optimization on a limited budget of evaluations using model-assisted Smetric selection","authors":"W Ponweiser, T Wagner, D Biermann, M Vincze","year":"2008","venue":"International Conference on Parallel Problem Solving from Nature","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Salinas et al., 2020] D. Salinas, H. Shen, and V. Perrone. A quantile-based approach for hyperparameter transfer learning. In International Conference on Machine Learn- ing, 2020.","title":"A quantile-based approach for hyperparameter transfer learning","authors":"D Salinas, H Shen, V Perrone","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Schmucker et al., 2020] R. Schmucker, M. Donini, V. Per- rone, MB. Zafar, and C. Archambeau. Multi-objective multi-fidelity hyperparameter optimization with applica- tion to fairness. In Meta-Learning Workshop at Advances in Neural Information Processing Systems, 2020.","title":"Multi-objective multi-fidelity hyperparameter optimization with application to fairness","authors":"R Schmucker, M Donini, V Perrone, M B Zafar, C Archambeau","year":"2020","venue":"Meta-Learning Workshop at Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Shahriari et al., 2016] B. Shahriari, K. Swersky, Z. Wang, R. Adams, and N. de Freitas. Taking the human out of the loop: A review of Bayesian optimization. IEEE, 104, 2016.","title":"Taking the human out of the loop: A review of Bayesian optimization","authors":"B Shahriari, K Swersky, Z Wang, R Adams, N De Freitas","year":"2016","venue":"IEEE","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Song et al., 2022] J. Song, L. Yu, W. Neiswanger, and S. Er- mon. A general recipe for likelihood-free Bayesian opti- mization. In International Conference on Machine Learn- ing, 2022.","title":"A general recipe for likelihood-free Bayesian optimization","authors":"J Song, L Yu, W Neiswanger, S Ermon","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Springenberg et al., 2016] JT. Springenberg, A. Klein, S. Falkner, and F. Hutter. Bayesian optimization with ro- bust Bayesian neural networks. In Advances in Neural In- formation Processing Systems, 2016.","title":"Bayesian optimization with robust Bayesian neural networks","authors":"Jt, A Springenberg, S Klein, F Falkner, Hutter","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Swersky et al., 2013] K. Swersky, J. Snoek, and R. Adams. Multi-task Bayesian optimization. In Advances in Neural Information Processing Systems, 2013.","title":"Multi-task Bayesian optimization","authors":"K Swersky, J Snoek, R Adams","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Vanschoren, 2019] J. Vanschoren. Meta-learning. In Auto- mated Machine Learning, pages 35-61. Springer, 2019.","title":"Meta-learning","authors":"J Vanschoren","year":"2019","venue":"Automated Machine Learning","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Volpp et al., 2020] M. Volpp, LP. Fröhlich, K. Fischer, A. Doerr, S. Falkner, F. Hutter, and C. Daniel. Meta- learning acquisition functions for transfer learning in Bayesian optimization. In International Conference on Learning Representations, 2020.","title":"Metalearning acquisition functions for transfer learning in Bayesian optimization","authors":"M Volpp, L P Fröhlich, K Fischer, A Doerr, S Falkner, F Hutter, C Daniel","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Wang and Jegelka, 2017] Z. Wang and S. Jegelka. Max- value entropy search for efficient Bayesian optimization. In International Conference on Machine Learning, 2017.","title":"Maxvalue entropy search for efficient Bayesian optimization","authors":"Z Wang, S Jegelka","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Watanabe and Hutter, 2022] S. Watanabe and F. Hutter. c- TPE: Generalizing tree-structured Parzen estimator with inequality constraints for continuous and categorical hy- perparameter optimization. Gaussian Processes, Spa- tiotemporal Modeling, and Decision-making Systems Workshop at Advances in Neural Information Processing Systems, 2022.","title":"c-TPE: Generalizing tree-structured Parzen estimator with inequality constraints for continuous and categorical hyperparameter optimization","authors":"S Watanabe, F Hutter","year":"2022","venue":"Gaussian Processes, Spatiotemporal Modeling, and Decision-making Systems Workshop at Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Watanabe and Hutter, 2023] S. Watanabe and F. Hutter. c- TPE: Tree-structured Parzen estimator with inequality constraints for expensive hyperparameter optimization. arXiv:2211.14411, 2023.","title":"c-TPE: Tree-structured Parzen estimator with inequality constraints for expensive hyperparameter optimization","authors":"S Watanabe, F Hutter","year":"2023","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2023/486"},{"position":42,"raw":"[Watanabe et al., 2023] S. Watanabe, A. Bansal, and F. Hut- ter. PED-ANOVA: Efficiently quantifying hyperparame- ter importance in arbitrary subspaces. arXiv:2304.10255, 2023.","title":"PED-ANOVA: Efficiently quantifying hyperparameter importance in arbitrary subspaces","authors":"S Watanabe, A Bansal, F Hutter","year":"2023","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2023/488"},{"position":43,"raw":"[Watanabe, 2023a] S. Watanabe. Python tool for visualizing variability of Pareto fronts over multiple runs. 2023.","title":"Python tool for visualizing variability of Pareto fronts over multiple runs","authors":"; S Watanabe, Watanabe","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Watanabe, 2023b] S. Watanabe. Tree-structured Parzen estimator: Understanding its algorithm components and their roles for better empirical performance. arXiv:2304.11127, 2023.","title":"Tree-structured Parzen estimator: Understanding its algorithm components and their roles for better empirical performance","authors":"; S Watanabe, Watanabe","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Wistuba et al., 2015] M. Wistuba, N. Schilling, and L. Schmidt-Thieme. Hyperparameter search space pruning-a new component for sequential model-based hyperparameter optimization. In Joint European Confer- ence on Machine Learning and Knowledge Discovery in Databases, 2015.","title":"Hyperparameter search space pruning-a new component for sequential model-based hyperparameter optimization","authors":"M Wistuba, N Schilling, L Schmidt-Thieme","year":"2015","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Wistuba et al., 2016] M. Wistuba, N. Schilling, and L. Schmidt-Thieme. Two-stage transfer surrogate model for automatic hyperparameter optimization. In Joint Eu- ropean Conference on Machine Learning and Knowledge Discovery in Databases, 2016.","title":"Two-stage transfer surrogate model for automatic hyperparameter optimization","authors":"M Wistuba, N Schilling, L Schmidt-Thieme","year":"2016","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Zhang and Duh, 2020] X. Zhang and K. Duh. Reproducible and efficient benchmarks for hyperparameter optimization of neural machine translation systems. Transactions of the Association for Computational Linguistics, 8, 2020.","title":"Reproducible and efficient benchmarks for hyperparameter optimization of neural machine translation systems","authors":"Duh Zhang, X Zhang, K Duh","year":"2020","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Zhang and Li, 2007] Q. Zhang and H. Li. MOEA/D: A mul- tiobjective evolutionary algorithm based on decomposi- tion. Evolutionary Computation, 11, 2007.","title":"MOEA/D: A multiobjective evolutionary algorithm based on decomposition","authors":"Li ; Zhang, ] Q Zhang, H Li","year":"2007","venue":"Evolutionary Computation","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}