{"type":"ScholarlyArticle","title":"Efficiency Through Procrastination: Approximately Optimal Algorithm Configuration with Runtime Guarantees","authors":[{"name":"Robert Kleinberg","slug":"robert-kleinberg","given":"Robert","family":"Kleinberg","affiliation":"Dept. of Computer Science Cornell University"},{"name":"Kevin Leyton-Brown","slug":"kevin-leyton-brown","given":"Kevin","family":"Leyton-Brown","affiliation":"Dept. of Computer Science University of British Columbia"},{"name":"Brendan Lucier","slug":"brendan-lucier","given":"Brendan","family":"Lucier","affiliation":"Microsoft Research"}],"abstract":"Algorithm configuration methods have achieved much practical success, but to date have not been backed by meaningful performance guarantees. We address this gap with a new algorithm configuration framework, Structured Procrastination. With high probability and nearly as quickly as possible in the worst case, our framework finds an algorithm configuration that provably achieves near optimal performance.  Moreover, its running time requirements asymptotically dominate those of existing methods.","doi":"10.24963/ijcai.2017/281","url":"https://ijcai.org/proceedings/2017/281","pdf":"https://ijcai.org/proceedings/2017/0281.pdf","markdown":"https://ijcai.org/proceedings/2017/281.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/281","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":281,"pages":{"first":2023,"last":2031},"keywords":["Machine Learning: Learning Theory","Constraints and Satisfiability: Solvers and Tools","Constraints and Satisfiability: Evaluation and Analysis","Constraints and Satisfiability: Constraints and Satisfiability"],"videos":[],"references":[{"position":1,"raw":"[Ansótegui et al., 2009] C. Ansótegui, M. Sellmann, and K. Tierney. A gender-based genetic algorithm for automatic configuration of algorithms. In Principles and Practice of Constraint Programming (CP), pages 142-157, 2009.","title":"A gender-based genetic algorithm for automatic configuration of algorithms","authors":"C Ansótegui, M Sellmann, K Tierney","year":"2009","venue":"Principles and Practice of Constraint Programming (CP)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ansótegui et al., 2015] C. Ansótegui, Y. Malitsky, M. Sell- mann, and K. Tierney. Model-based genetic algorithms for algorithm configuration. In International Joint Conference on Artificial Intelligence (IJCAI), pages 733-739, 2015.","title":"Model-based genetic algorithms for algorithm configuration","authors":"C Ansótegui, Y Malitsky, M Sellmann, K Tierney","year":"2015","venue":"International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2015/109"},{"position":3,"raw":"[Auer et al., 2002] P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Ma- chine learning, 47(2-3):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"P Auer, N Cesa-Bianchi, P Fischer","year":"2002","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Badanidiyuru et al., 2013] A. Badanidiyuru, R. Kleinberg, and A. Slivkins. Bandits with knapsacks. In Foundations of Computer Science (FOCS), pages 207-216, 2013.","title":"Bandits with knapsacks","authors":"A Badanidiyuru, R Kleinberg, A Slivkins","year":"2013","venue":"Foundations of Computer Science (FOCS)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bergstra et al., 2011] J. S. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems (NIPS), pages 2546-2554, 2011.","title":"Algorithms for hyper-parameter optimization","authors":"J S Bergstra, R Bardenet, Y Bengio, B Kégl","year":"2011","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Birattari et al., 2002] M. Birattari, T. Sttzle, L. Paquete, and K. Varrentrapp. A racing algorithm for configuring meta- heuristics. In Genetic and Evolutionary Computation Con- ference (GECCO), pages 11-18, 2002.","title":"A racing algorithm for configuring metaheuristics","authors":"M Birattari, T Sttzle, L Paquete, K Varrentrapp","year":"2002","venue":"Genetic and Evolutionary Computation Conference (GECCO)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bubeck et al., 2011] S. Bubeck, R. Munos, G. Stoltz, and C. Szepesvári. X-armed bandits. Journal of Machine Learn- ing Research, 12(May):1655-1695, 2011.","title":"X-armed bandits","authors":"S Bubeck, R Munos, G Stoltz, C Szepesvári","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bubeck et al., 2012] S. Bubeck, N. Cesa-Bianchi, et al. Re- gret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends in Machine Learning, 5(1):1-122, 2012.","title":"Regret analysis of stochastic and nonstochastic multi-armed bandit problems","authors":"S Bubeck, N Cesa-Bianchi","year":"2012","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cesa-Bianchi and Lugosi, 2012] N. Cesa-Bianchi and G. Lu- gosi. Combinatorial bandits. Journal of Computer and System Sciences, 78(5):1404-1422, 2012.","title":"Combinatorial bandits","authors":"N Cesa-Bianchi, G Lugosi","year":"2012","venue":"Journal of Computer and System Sciences","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chaudhuri et al., 2009] K. Chaudhuri, Y. Freund, and D. J. Hsu. A parameter-free hedging algorithm. In Advances in Neural Information Processing Systems (NIPS), pages 297-305, 2009.","title":"A parameter-free hedging algorithm","authors":"K Chaudhuri, Y Freund, D J Hsu","year":"2009","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dean and Voss, 1999] A. Dean and D. Voss. Design and analysis of experiments. Springer-Verlag NY, 1999.","title":"Design and analysis of experiments","authors":"A Dean, D Voss","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ganchev et al., 2010] K. Ganchev, Y. Nevmyvaka, M. Kearns, and J. W. Vaughan. Censored explo- ration and the dark pool problem. Communications of the ACM, 53(5):99-107, 2010.","title":"Censored exploration and the dark pool problem","authors":"K Ganchev, Y Nevmyvaka, M Kearns, J W Vaughan","year":"2010","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Guha and Munagala, 2007] S. Guha and K. Munagala. Ap- proximation algorithms for budgeted learning problems. In ACM Symposium on Theory of Computing (STOC), pages 104-113, 2007.","title":"Approximation algorithms for budgeted learning problems","authors":"S Guha, K Munagala","year":"2007","venue":"ACM Symposium on Theory of Computing (STOC)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hutter et al., 2007] F. Hutter, H. Hoos, and T. Stützle. Au- tomatic algorithm configuration based on local search. In AAAI Conference on Artificial Intelligence, pages 1152- 1157, 2007.","title":"Automatic algorithm configuration based on local search","authors":"F Hutter, H Hoos, T Stützle","year":"2007","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hutter et al., 2009] F. Hutter, H. Hoos, K. Leyton-Brown, and T. Stützle. ParamILS: An automatic algorithm con- figuration framework. Journal of Artificial Intelligence Research, 36:267-306, 2009.","title":"ParamILS: An automatic algorithm configuration framework","authors":"F Hutter, H Hoos, K Leyton-Brown, T Stützle","year":"2009","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hutter et al., 2011a] F. Hutter, H. Hoos, and K. Leyton- Brown. Bayesian optimization with censored response data. In NIPS workshop on Bayesian Optimization, Sequential Experimental Design, and Bandits (BayesOpt'11), 2011.","title":"Bayesian optimization with censored response data","authors":"Hutter","year":"2011","venue":"NIPS workshop on Bayesian Optimization, Sequential Experimental Design, and Bandits (BayesOpt'11)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hutter et al., 2011b] F. Hutter, H. Hoos, and K. Leyton- Brown. Sequential model-based optimization for general algorithm configuration. In Conference on Learning and Intelligent Optimization (LION), pages 507-523, 2011.","title":"Sequential model-based optimization for general algorithm configuration","authors":"Hutter","year":"2011","venue":"Conference on Learning and Intelligent Optimization (LION)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Improbable Research, 2011] Improbable Research. The 2011 Ig Nobel prize winners. http://www.improbable. com/ig/winners/#ig2011, 2011.","title":"Improbable Research. The 2011 Ig Nobel prize winners","authors":null,"year":"2011","venue":null,"doi":null,"url":"http://www.improbable.com/ig/winners/#ig2011,2011","internal_url":null},{"position":19,"raw":"[Kandasamy et al., 2016] K. Kandasamy, G. Dasarathy, B. Poczos, and J. Schneider. The multi-fidelity multi-armed bandit. In Advances in Neural Information Processing Sys- tems (NIPS), pages 1777-1785, 2016.","title":"The multi-fidelity multi-armed bandit","authors":"K Kandasamy, G Dasarathy, B Poczos, J Schneider","year":"2016","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kleinberg et al., 2008] R. Kleinberg, A. Slivkins, and E. Up- fal. Multi-armed bandits in metric spaces. In ACM Sympo- sium on Theory of Computing, pages 681-690, 2008.","title":"Multi-armed bandits in metric spaces","authors":"R Kleinberg, A Slivkins, E","year":"2008","venue":"ACM Symposium on Theory of Computing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kleinberg, 2006] R. Kleinberg. Anytime algorithms for multi-armed bandit problems. In ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 928-936, 2006.","title":"Anytime algorithms for multi-armed bandit problems","authors":"R Kleinberg","year":"2006","venue":"ACM-SIAM Symposium on Discrete Algorithms (SODA)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li et al., 2016] L. Li, K. Jamieson, G. DeSalvo, A. Ros- tamizadeh, and A. Talwalkar. Hyperband: A novel bandit- based approach to hyperparameter optimization. arXiv preprint arXiv:1603.06560, 2016.","title":"Hyperband: A novel banditbased approach to hyperparameter optimization","authors":"L Li, K Jamieson, G Desalvo, A Rostamizadeh, A Talwalkar","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[López-Ibáñez et al., 2011] M. López-Ibáñez, J. Dubois- Lacoste, T. Stützle, and M. Birattari. The irace package, iterated race for automatic algorithm configuration. Techni- cal report, IRIDIA, Université Libre de Bruxelles, 2011.","title":"The irace package, iterated race for automatic algorithm configuration","authors":"M López-Ibáñez, J Dubois-Lacoste, T Stützle, M Birattari","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Munos, 2014] R. Munos. From bandits to Monte-Carlo tree search: The optimistic principle applied to optimization and planning. Foundations and Trends in Machine Learning, 7(1):1-129, 2014.","title":"From bandits to Monte-Carlo tree search: The optimistic principle applied to optimization and planning","authors":"R Munos","year":"2014","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Perry, 1996] J. Perry. How to procrastinate and still get things done. http://www.chronicle.com/article/ How-to-ProcrastinateStill/93959, 1996.","title":"How to procrastinate and still get things done","authors":"J Perry","year":"1996","venue":null,"doi":null,"url":"http://www.chronicle.com/article/How-to-ProcrastinateStill/93959","internal_url":null},{"position":26,"raw":"[Shahriari et al., 2016] B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, and N. de Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1):148-175, 2016.","title":"Taking the human out of the loop: A review of Bayesian optimization","authors":"B Shahriari, K Swersky, Z Wang, R P Adams, N De Freitas","year":"2016","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Srinivas et al., 2012] N. Srinivas, A. Krause, S. M. Kakade, and M. W. Seeger. Information-theoretic regret bounds for Gaussian process optimization in the bandit setting. IEEE Transactions on Information Theory, 58(5):3250- 3265, 2012.","title":"Information-theoretic regret bounds for Gaussian process optimization in the bandit setting","authors":"N Srinivas, A Krause, S M Kakade, M W Seeger","year":"2012","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Thornton et al., 2013] C. Thornton, F. Hutter, H. Hoos, and K. Leyton-Brown. Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms. In Conference on Knowledge Discovery and Data mining (KDD), pages 847-855, 2013.","title":"Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms","authors":"C Thornton, F Hutter, H Hoos, K Leyton-Brown","year":"2013","venue":"Conference on Knowledge Discovery and Data mining (KDD)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tran-Thanh et al., 2012] L. Tran-Thanh, A. Chapman, A. Rogers, and N. R. Jennings. Knapsack based optimal policies for budget-limited multi-armed bandits. In AAAI Conference on Artificial Intelligence, 2012.","title":"Knapsack based optimal policies for budget-limited multi-armed bandits","authors":"L Tran-Thanh, A Chapman, A Rogers, N R Jennings","year":"2012","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}