Evaluating Classifiers by Means of Test Data with Noisy Labels
Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence. Pages 513-517.
Abstract
Often the most expensive and time-consuming task in building a pattern recognition system is collecting and accurately labeling training and testing data. In this paper, we explore the use of inexpensive noisy testing data for evaluating a classifier's performance. We assume 1) the (human) labeler provides category labels with a known mislabeling rate and 2) the trained classifier and the labeler are statistically independent. We then derive the number of "noisy" test samples that are, on average, equivalent to a single perfectly labeled test sample for the task of evaluating the classifier's performance. For practical and realistic error and mislabeling rates, this number of equivalent test patterns can be surprisingly low. We also derive an upper and lower bound for the true error rate when the labeler and the classifier are not independent.
@inproceedings{ijcai2003p76,
title = {Evaluating Classifiers by Means of Test Data with Noisy Labels},
author = {Lam, Chuck P. and Stork, David G.},
booktitle = {Proceedings of the Eighteenth International Joint Conference on
Artificial Intelligence, {IJCAI-03}},
publisher = {International Joint Conferences on Artificial Intelligence},
editor = {Georg Gottlob},
pages = {513--517},
year = {2003},
url = {https://ijcai.org/proceedings/2003/76},
}
TY - CPAPER TI - Evaluating Classifiers by Means of Test Data with Noisy Labels AU - Lam, Chuck P. AU - Stork, David G. T2 - Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence, IJCAI-03 PB - International Joint Conferences on Artificial Intelligence PY - 2003 SP - 513 EP - 517 UR - https://ijcai.org/proceedings/2003/76 ER -Copied
Machine-readable versions of this paper, for tools and AI agents:
