{"type":"ScholarlyArticle","title":"Neural Networks for Predicting Algorithm Runtime Distributions","authors":[{"name":"Katharina Eggensperger","slug":"katharina-eggensperger","given":"Katharina","family":"Eggensperger","affiliation":"University of Freiburg"},{"name":"Marius Lindauer","slug":"marius-lindauer","given":"Marius","family":"Lindauer","affiliation":"University of Freiburg"},{"name":"Frank Hutter","slug":"frank-hutter","given":"Frank","family":"Hutter","affiliation":"University of Freiburg"}],"abstract":"Many state-of-the-art algorithms for solving hard combinatorial problems in artificial intelligence (AI) include elements of stochasticity that lead to high variations in runtime, even for a fixed problem instance. Knowledge about the resulting runtime distributions (RTDs) of algorithms on given problem instances can be exploited in various meta-algorithmic procedures, such as algorithm selection, portfolios, and randomized restarts. Previous work has shown that machine learning can be used to individually predict mean, median and variance of RTDs. To establish a new state-of-the-art in predicting RTDs, we demonstrate that the parameters of an RTD should be learned jointly and that neural networks can do this well by directly optimizing the likelihood of an RTD given runtime observations. In an empirical study involving five algorithms for SAT solving and AI planning, we show that neural networks predict the true RTDs of unseen instances better than\r\nprevious methods, and can even do so when only few runtime observations are available per training instance.","doi":"10.24963/ijcai.2018/200","url":"https://ijcai.org/proceedings/2018/200","pdf":"https://ijcai.org/proceedings/2018/0200.pdf","markdown":"https://ijcai.org/proceedings/2018/200.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/200","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Heuristic Search and Game Playing","track":"Main track","number":200,"pages":{"first":1442,"last":1448},"keywords":["Constraints and SAT: Constraints: Evaluation and Analysis","Constraints and SAT: Constraints: Solvers and Tools","Heuristic Search and Game Playing: Meta-Reasoning and Meta-heuristics","Constraints and SAT: SAT: Evaluation and Analysis","Constraints and SAT: SAT: Solvers and Tools","Heuristic Search and Game Playing: Heuristic Search and Machine Learning","Constraints and SAT: Constraints and Data Mining ; Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Arbelaez et al., 2016] A. Arbelaez, C. Truchet, and B. O'Sullivan. Learning sequential and parallel runtime distributions for random- ized algorithms. In Proc. of ICTAI'16, pages 655-662, 2016.","title":"Learning sequential and parallel runtime distributions for randomized algorithms","authors":"A Arbelaez, C Truchet, B O'sullivan","year":"2016","venue":"Proc. of ICTAI'16","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Babić and Hutter, 2007] D. Babić and F. Hutter. Spear theorem prover, 2007. Solver description, SAT competition.","title":"Spear theorem prover","authors":"D Babić, F Hutter","year":"2007","venue":"Solver description, SAT competition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Balint and Schöning, 2012] A. Balint and U. Schöning. Choosing probability distributions for stochastic local search and the role of make versus break. In Proc. of SAT'12, pages 16-29, 2012.","title":"Choosing probability distributions for stochastic local search and the role of make versus break","authors":"A Balint, U Schöning","year":"2012","venue":"Proc. of SAT'12","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Biere, 2014] A. Biere. Yet another local search solver and lingeling and friends entering the SAT competition 2014. In Proc. of SAT Competition 2014, pages 39-40, 2014.","title":"Yet another local search solver and lingeling and friends entering the SAT competition 2014","authors":"A Biere","year":"2014","venue":"Proc. of SAT Competition 2014","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brewer, 1995] E. Brewer. High-level optimization via automated statistical modeling. In ACM SIGPLAN Notices, pages 80-91, 1995.","title":"High-level optimization via automated statistical modeling","authors":"E Brewer","year":"1995","venue":"ACM SIGPLAN Notices","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chollet et al., 2015] Chollet et al. Keras. https://github.com/ fchollet/keras, 2015.","title":null,"authors":"Chollet","year":null,"venue":null,"doi":null,"url":"https://github.com/fchollet/keras,2015","internal_url":null},{"position":7,"raw":"[Eggensperger et al., 2018] K. Eggensperger, M. Lindauer, H. Hoos, F. Hutter, and K Leyton-Brown. Efficient benchmarking of algo- rithm configuration procedures via model-based surrogates. Ma- chine Learning, 107:15-41, 2018.","title":"Efficient benchmarking of algorithm configuration procedures via model-based surrogates","authors":"K Eggensperger, M Lindauer, H Hoos, F Hutter, K Leyton-Brown","year":"2018","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fawcett et al., 2014] C. Fawcett, M. Vallati, F. Hutter, J. Hoffmann, H. Hoos, and K. Leyton-Brown. Improved features for runtime prediction of domain-independent planners. In Proc. of ICAPS'14, pages 355-359, 2014.","title":"Improved features for runtime prediction of domain-independent planners","authors":"C Fawcett, M Vallati, F Hutter, J Hoffmann, H Hoos, K Leyton-Brown","year":"2014","venue":"Proc. of ICAPS'14","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fink, 1998] E. Fink. How to solve it automatically: Selection among problem-solving methods. In Proc. of ICANN'08, pages 128-136, 1998.","title":"How to solve it automatically: Selection among problem-solving methods","authors":"E Fink","year":"1998","venue":"Proc. of ICANN'08","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Frost et al., 1997] D. Frost, I. Rish, and L. Vila. Summarizing CSP hardness with continuous probability distributions. In Proc. of AAAI'97, pages 327-333, 1997.","title":"Summarizing CSP hardness with continuous probability distributions","authors":"D Frost, I Rish, L Vila","year":"1997","venue":"Proc. of AAAI'97","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gagliolo and Schmidhuber, 2005] M. Gagliolo and J. Schmidhuber. A neural network model for inter-problem adaptive online time allocation. In Proc. of ICANN'05, pages 752-752, 2005.","title":"A neural network model for inter-problem adaptive online time allocation","authors":"M Gagliolo, J Schmidhuber","year":"2005","venue":"Proc. of ICANN'05","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gagliolo and Schmidhuber, 2006a] M. Gagliolo and J. Schmidhu- ber. Impact of censored sampling on the performance of restart strategies. In Proc. of CP'06, pages 167-181, 2006.","title":"Impact of censored sampling on the performance of restart strategies","authors":"Schmidhuber Gagliolo, M Gagliolo, J Schmidhuber","year":"2006","venue":"Proc. of CP'06","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gagliolo and Schmidhuber, 2006b] M. Gagliolo and J. Schmidhu- ber. Learning dynamic algorithm portfolios. AMAI, 47(3-4):295- 328, 2006.","title":"Learning dynamic algorithm portfolios","authors":"Schmidhuber Gagliolo, M Gagliolo, J","year":"2006","venue":"AMAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gebser et al., 2012] M. Gebser, B. Kaufmann, and T. Schaub. Conflict-driven answer set solving: From theory to practice. AI, 187-188:52-89, 2012.","title":"Conflict-driven answer set solving: From theory to practice","authors":"M Gebser, B Kaufmann, T Schaub","year":"2012","venue":"AI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gent et al., 1999] I. Gent, H. Hoos, P. Prosser, and T. Walsh. Mor- phing: Combining structure and randomness. In Proc. of AAAI'99, pages 654-660, 1999.","title":"Morphing: Combining structure and randomness","authors":"Gent","year":"1999","venue":"Proc. of AAAI'99","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Gerevini and Serina, 2002] A. Gerevini and I. Serina. LPG: A plan- ner based on local search for planning graphs with action costs. In Proc. of AIPS'02, pages 13-22, 2002.","title":"LPG: A planner based on local search for planning graphs with action costs","authors":"A Gerevini, I Serina","year":"2002","venue":"Proc. of AIPS'02","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Gomes and Selman, 1997] C. Gomes and B. Selman. Problem structure in the presence of perturbations. In Proc. of AAAI'97, pages 221-226, 1997.","title":"Problem structure in the presence of perturbations","authors":"C Gomes, B Selman","year":"1997","venue":"Proc. of AAAI'97","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Gomes and Selman, 2001] C. Gomes and B. Selman. Algorithm portfolios. AIJ, 126(1-2):43-62, 2001.","title":"Algorithm portfolios","authors":"C Gomes, B Selman","year":"2001","venue":"AIJ","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Gomes et al., 2000] C. Gomes, B. Selman, N. Crato, and H. Kautz. Heavy-tailed phenomena in satisfiability and constraint satisfac- tion problems. JAR, 24:67-100, 2000.","title":"Heavy-tailed phenomena in satisfiability and constraint satisfaction problems","authors":"C Gomes, B Selman, N Crato, H Kautz","year":"2000","venue":"JAR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Goodfellow et al., 2016] I.","title":null,"authors":"Goodfellow","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"Goodfellow, Y. Bengio, and A. Courville. Deep Learning. MIT Press, 2016.","title":"Deep Learning","authors":"Y Goodfellow, A Bengio, Courville","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Haim and Walsh, 2009] S. Haim and T. Walsh. Restart strategy selection using machine learning techniques. In Proc. of SAT'09, pages 312-325, 2009.","title":"Restart strategy selection using machine learning techniques","authors":"S Haim, T Walsh","year":"2009","venue":"Proc. of SAT'09","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Hoos and Stützle, 2004] H. Hoos and T. Stützle. Stochastic Local Search: Foundations & Applications. Morgan Kaufmann Publish- ers Inc., 2004.","title":"Stochastic Local Search: Foundations & Applications","authors":"H Hoos, T Stützle","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Hurley and O'Sullivan, 2015] B. Hurley and B. O'Sullivan. Statis- tical regimes and runtime prediction. In Proc. of IJCAI'15, pages 318-324, 2015.","title":"Statistical regimes and runtime prediction","authors":"B Hurley, B O'sullivan","year":"2015","venue":"Proc. of IJCAI'15","doi":null,"url":null,"internal_url":"/proceedings/2015/51"},{"position":25,"raw":"[Hutter et al., 2002] F. Hutter, D. Tompkins, and H. Hoos. Scaling and probabilistic smoothing: Efficient dynamic local search for SAT. In Proc.of CP'02, pages 233-248, 2002.","title":"Scaling and probabilistic smoothing: Efficient dynamic local search for SAT","authors":"F Hutter, D Tompkins, H Hoos","year":"2002","venue":"Proc.of CP'02","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Hutter et al., 2006] F. Hutter, Y. Hamadi, H. Hoos, and K. Leyton- Brown. Performance prediction and automated tuning of random- ized and parametric algorithms. In Proc. of CP'06, pages 213-228, 2006.","title":"Performance prediction and automated tuning of randomized and parametric algorithms","authors":"F Hutter, Y Hamadi, H Hoos, K Leyton-Brown","year":"2006","venue":"Proc. of CP'06","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Hutter et al., 2011] F. Hutter, H. Hoos, and K. Leyton-Brown. Se- quential model-based optimization for general algorithm configu- ration. In Proc. of LION'11, pages 507-523, 2011.","title":"Sequential model-based optimization for general algorithm configuration","authors":"F Hutter, H Hoos, K Leyton-Brown","year":"2011","venue":"Proc. of LION'11","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Hutter et al., 2013] F. Hutter, H. Hoos, and K. Leyton-Brown. Iden- tifying key algorithm parameters and instance features using for- ward selection. In Proc. of LION'13, pages 364-381, 2013.","title":"Identifying key algorithm parameters and instance features using forward selection","authors":"F Hutter, H Hoos, K Leyton-Brown","year":"2013","venue":"Proc. of LION'13","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Hutter et al., 2014] F. Hutter, L. Xu, H. Hoos, and K. Leyton- Brown. Algorithm runtime prediction: Methods and evaluation. AIJ, 206:79-111, 2014.","title":"Algorithm runtime prediction: Methods and evaluation","authors":"F Hutter, L Xu, H Hoos, K Leyton-Brown","year":"2014","venue":"AIJ","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Jacobs et al., 1991] R. Jacobs, M. Jordan, S. Nowlan, and G. Hin- ton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-87, 1991.","title":"Adaptive mixtures of local experts","authors":"R Jacobs, M Jordan, S Nowlan, G Hinton","year":"1991","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Jones et al., 2001] E. Jones, T. Oliphant, and P. Peterson et al. SciPy: Open source scientific tools for Python. http://www.scipy. org/, 2001.","title":"SciPy: Open source scientific tools for Python","authors":"E Jones, T Oliphant, P Peterson","year":null,"venue":null,"doi":null,"url":"http://www.scipy.org/,2001","internal_url":null},{"position":32,"raw":"[Leyton-Brown et al., 2009] K. Leyton-Brown, E. Nudelman, and Y. Shoham. Empirical hardness models: Methodology and a case study on combinatorial auctions. Journal of ACM, 56(4):1-52, 2009.","title":"Empirical hardness models: Methodology and a case study on combinatorial auctions","authors":"K Leyton-Brown, E Nudelman, Y Shoham","year":"2009","venue":"Journal of ACM","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Loreggia et al., 2016] A. Loreggia, Y. Malitsky, H. Samulowitz, and V. Saraswat. Deep learning for algorithm portfolios. In Proc. of AAAI'16, pages 1280-1286, 2016.","title":"Deep learning for algorithm portfolios","authors":"A Loreggia, Y Malitsky, H Samulowitz, V Saraswat","year":"2016","venue":"Proc. of AAAI'16","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Luby et al., 1993] M. Luby, A. Sinclair, and D. Zuckerman. Opti- mal speedup of las vegas algorithms. Inf. Process. Lett., pages 173-180, 1993.","title":"Optimal speedup of las vegas algorithms","authors":"M Luby, A Sinclair, D Zuckerman","year":"1993","venue":"Inf. Process. Lett","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Nudelman et al., 2004] E. Nudelman, K. Leyton-Brown, A. De- vkar, Y. Shoham, and H. Hoos. Understanding random SAT: Beyond the clauses-to-variables ratio. In Proc. of CP'04, pages 438-452, 2004.","title":"Understanding random SAT: Beyond the clauses-to-variables ratio","authors":"E Nudelman, K Leyton-Brown, A Devkar, Y Shoham, H Hoos","year":"2004","venue":"Proc. of CP'04","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Pascanu et al., 2014] R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. In Proc. of ICML'13, pages 1310-1318, 2014.","title":"On the difficulty of training recurrent neural networks","authors":"R Pascanu, T Mikolov, Y Bengio","year":"2014","venue":"Proc. of ICML'13","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Pedregosa et al., 2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. JMLR, 12:2825-2830, 2011.","title":"Scikit-learn: Machine learning in Python","authors":"F Pedregosa, G Varoquaux, A Gramfort, V Michel, B Thirion, O Grisel, M Blondel, P Prettenhofer, R Weiss, V Dubourg, J Vanderplas, A Passos, D Cournapeau, M Brucher, M Perrot, E Duchesnay","year":"2011","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Penberthy and Weld, 1994] J. Penberthy and D. Weld. Temporal planning with continuous change. In Proc. of AAAI'94, pages 1010-1015, 1994.","title":"Temporal planning with continuous change","authors":"J Penberthy, D Weld","year":"1994","venue":"Proc. of AAAI'94","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Roberts and Howe, 2007] Mark Roberts and Adele Howe. Learned models of performance for many planners. In ICAPS 2007 Work- shop AI Planning and Learning, 2007.","title":"Learned models of performance for many planners","authors":"Mark Roberts, Adele Howe","year":"2007","venue":"ICAPS 2007 Workshop AI Planning and Learning","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Smith-Miles and Lopes, 2012] K. Smith-Miles and L. Lopes. Mea- suring instance difficulty for combinatorial optimization problems. Computers and Operations Research, 39(5):875-889, 2012.","title":"Measuring instance difficulty for combinatorial optimization problems","authors":"K Smith-Miles, L Lopes","year":"2012","venue":"Computers and Operations Research","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Xu et al., 2008] L. Xu, F. Hutter, H. Hoos, and K. Leyton-Brown. SATzilla: Portfolio-based algorithm selection for SAT. JAIR, 32:565-606, 2008.","title":"SATzilla: Portfolio-based algorithm selection for SAT","authors":"L Xu, F Hutter, H Hoos, K Leyton-Brown","year":"2008","venue":"JAIR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}