{"type":"ScholarlyArticle","title":"Unbiased Assesment of Learning Algorithms","authors":[{"name":"Tobias Scheffer","slug":"tobias-scheffer","given":"Tobias","family":"Scheffer","affiliation":""},{"name":"Ralf Herbrich","slug":"ralf-herbrich","given":"Ralf","family":"Herbrich","affiliation":""}],"abstract":"In order to rank the performance of machine learning a lg orithms, many researchers conduct experiments on be nchmark data sets. Since most lea rning algorithms have domain-specific parameters, it is a popular c ustom to adapt these parameters to obtain a minimal error rate on the test set. The same rat e is then used to rank the algorithm, which ca us es an optimistic bias. We quantify this bias, showing, in particular, that an algorithm with more parameters wi ll probab ly be ranked higher than an equally good algorithm with fewer parameters. We demonstrat e this res ult , showing the number of param eters and trials required in order to pretend to outperform C4.5 or FOIL , respectively, for various benc hm ar k problems. We then describe out how unbiased ranking experim ents should be co nduc ted.","doi":null,"url":"https://ijcai.org/proceedings/1997/115","pdf":"https://ijcai.org/proceedings/1997/0115.pdf","markdown":"https://ijcai.org/proceedings/1997/115.md","bibtex":"https://ijcai.org/proceedings/1997/bibtex/115","volume":{"slug":"1997","title":"Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 1997","url":"https://ijcai.org/proceedings/1997/","year":1997,"location":"Nagoya, Japan","dates":"","editors":"Martha E. Pollack","isbn":"","publisher":"International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":"Learning","track":"Learning","number":115,"pages":{"first":798,"last":802},"keywords":[],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}