---
title: "A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection"
authors:
  - "Ron Kohavi"
doi: ""
url: "https://ijcai.org/proceedings/1995/147"
pdf: "https://ijcai.org/proceedings/1995/0147.pdf"
venue: "Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence"
year: 1995
pages: "1137-1144"
track: "Learning"
keywords:
---

# A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection

Ron Kohavi

*Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence*, pages 1137-1144.

## Abstract

We review accuracy estimation methods and compare the two most common methods: crossvalidation and bootstrap. Recent experimenta l results on artificial data and theoretical results in restricted settings have shown that for selecting a good classifier from a set of classifi ers (model selection), ten-fold cross-validation may be better than the more expensive leaveone-out cross-validation. We report on a largescale experiment-ove r half a million runs of C4.5 and a Naive-Bayes alg orithm-to estimate the effects of different parameters on these a lgorithms on real-world datasets . For crossvalidation, we vary the number of folds and whether the folds are stratified or not; for bootstrap, we vary the number of bootstrap samples. Our results indicate that for real-word datasets simil ar to ours, the best method to use for model selection is ten-fold stratified cross validation, even if computation power allows using more folds.