{"type":"ScholarlyArticle","title":"A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection","authors":[{"name":"Ron Kohavi","slug":"ron-kohavi","given":"Ron","family":"Kohavi","affiliation":""}],"abstract":"We review accuracy estimation methods and compare the two most common methods: crossvalidation and bootstrap. Recent experimenta l results on artificial data and theoretical results in restricted settings have shown that for selecting a good classifier from a set of classifi ers (model selection), ten-fold cross-validation may be better than the more expensive leaveone-out cross-validation. We report on a largescale experiment-ove r half a million runs of C4.5 and a Naive-Bayes alg orithm-to estimate the effects of different parameters on these a lgorithms on real-world datasets . For crossvalidation, we vary the number of folds and whether the folds are stratified or not; for bootstrap, we vary the number of bootstrap samples. Our results indicate that for real-word datasets simil ar to ours, the best method to use for model selection is ten-fold stratified cross validation, even if computation power allows using more folds.","doi":null,"url":"https://ijcai.org/proceedings/1995/147","pdf":"https://ijcai.org/proceedings/1995/0147.pdf","markdown":"https://ijcai.org/proceedings/1995/147.md","bibtex":"https://ijcai.org/proceedings/1995/bibtex/147","volume":{"slug":"1995","title":"Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 1995","url":"https://ijcai.org/proceedings/1995/","year":1995,"location":"Montréal, Québec, Canada","dates":"","editors":"Chris S. Mellish","isbn":"","publisher":"International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":"Learning","track":"Learning","number":147,"pages":{"first":1137,"last":1144},"keywords":[],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}