A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection

Abstract

We review accuracy estimation methods and compare the two most common methods: crossvalidation and bootstrap. Recent experimenta l results on artificial data and theoretical results in restricted settings have shown that for selecting a good classifier from a set of classifi ers (model selection), ten-fold cross-validation may be better than the more expensive leaveone-out cross-validation. We report on a largescale experiment-ove r half a million runs of C4.5 and a Naive-Bayes alg orithm-to estimate the effects of different parameters on these a lgorithms on real-world datasets . For crossvalidation, we vary the number of folds and whether the folds are stratified or not; for bootstrap, we vary the number of bootstrap samples. Our results indicate that for real-word datasets simil ar to ours, the best method to use for model selection is ten-fold stratified cross validation, even if computation power allows using more folds.