---
title: "Estimating the Accuracy of Learned Concepts"
authors:
  - "Timothy L. Bailey"
  - "Charles Elkan"
doi: ""
url: "https://ijcai.org/proceedings/1993/123"
pdf: "https://ijcai.org/proceedings/1993/0123.pdf"
venue: "Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence"
year: 1993
pages: "895-900"
track: "Machine Learning"
keywords:
---

# Estimating the Accuracy of Learned Concepts

Timothy L. Bailey, Charles Elkan

*Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence*, pages 895-900.

## Abstract

This paper investigates alternative estimators of the accuracy of concepts learned from examples. In particular, the cross-validation and 632 bootstrap estimators are studied, using synthetic training data and the FOIL learning algorithm . Our experimental results contradict previous papers in statistics, which advocate the 632 bootstrap method as superior to crossvalidation. Nevertheless, our results also suggest that conclusions based on cross-validation in previous machine learning papers are unreliable. Specifically, our observations are that (i) the true error of the concept learned by FOIL from independently drawn sets of examples of the same concept varies widely, (ii) the estimate of true error provided by cross-validation has high variability but is approximately unbiased, and (iii) the 632 bootstrap estimator has lower variability than cross-validation, but is systematically biased.