Unbiased Assesment of Learning Algorithms
Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence. Learning. Pages 798-802.
Abstract
In order to rank the performance of machine learning a lg orithms, many researchers conduct experiments on be nchmark data sets. Since most lea rning algorithms have domain-specific parameters, it is a popular c ustom to adapt these parameters to obtain a minimal error rate on the test set. The same rat e is then used to rank the algorithm, which ca us es an optimistic bias. We quantify this bias, showing, in particular, that an algorithm with more parameters wi ll probab ly be ranked higher than an equally good algorithm with fewer parameters. We demonstrat e this res ult , showing the number of param eters and trials required in order to pretend to outperform C4.5 or FOIL , respectively, for various benc hm ar k problems. We then describe out how unbiased ranking experim ents should be co nduc ted.
@inproceedings{ijcai1997p115,
title = {Unbiased Assesment of Learning Algorithms},
author = {Scheffer, Tobias and Herbrich, Ralf},
booktitle = {Proceedings of the Fifteenth International Joint Conference on
Artificial Intelligence, {IJCAI-97}},
publisher = {International Joint Conferences on Artificial Intelligence},
editor = {Martha E. Pollack},
pages = {798--802},
year = {1997},
note = {Learning},
url = {https://ijcai.org/proceedings/1997/115},
}
TY - CPAPER TI - Unbiased Assesment of Learning Algorithms AU - Scheffer, Tobias AU - Herbrich, Ralf T2 - Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence, IJCAI-97 PB - International Joint Conferences on Artificial Intelligence PY - 1997 SP - 798 EP - 802 UR - https://ijcai.org/proceedings/1997/115 ER -Copied
Machine-readable versions of this paper, for tools and AI agents:
