{"type":"ScholarlyArticle","title":"Process-Oriented Estimation of Generalization Error","authors":[{"name":"Pedro M. Domingos","slug":"pedro-m-domingos","given":"Pedro M.","family":"Domingos","affiliation":"Artificial Intelligence Group Instituto Superior Tecnico Lisbon 1049-001 , Portugal"}],"abstract":"Methods to avoid overfitting fall into two broad categories: data-oriented (using separate data for validation) and representation-oriented (penalizing complexity in the model) . Both have limitations that are hard to overcome. We argue that fully adequate model evaluation is only possible if the search process by which models are obtained is also taken into account. To this end, we recently proposed a method for process-oriented evaluation (POE), and successfully applied it to rule induction [Domingos, 1998b]. However, for the sake of simplicity this treatment made a number of rather artificial assumptions. In this paper the assumptions are removed, and a simple formula for error estimation is obtained. Empirical trials show the new, better-founded form of POE to be as accurate as the previous one, while further reducing theory sizes.","doi":null,"url":"https://ijcai.org/proceedings/1999/102","pdf":"https://ijcai.org/proceedings/1999/0102.pdf","markdown":"https://ijcai.org/proceedings/1999/102.md","bibtex":"https://ijcai.org/proceedings/1999/bibtex/102","volume":{"slug":"1999","title":"Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 1999","url":"https://ijcai.org/proceedings/1999/","year":1999,"location":"Stockholm, Sweden","dates":"","editors":"Thomas Dean","isbn":"","publisher":"International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":"Machine Learning","track":"Machine Learning","number":102,"pages":{"first":714,"last":720},"keywords":[],"videos":[],"references":[{"position":1,"raw":"[Bernardo and Smith, 1994] J . M. Bernardo and A. F . M. Smith. Bayesian Theory. Wiley, New York, NY, 1994.","title":"Bayesian Theory","authors":"Smith ; ] J M Bernardo, A F M Bernardo, Smith","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blake et al., 1998] C. Blake, E. Keogh, and C. J. Merz. UCI repository of machine learning databases. Machine-readable data repository, Department of Information and Computer Science, University of California at Irvine, Irvine, CA, 1998. http://- www .ics. uci.edu/ '\"\"mlearn/MLRepository.html.","title":"UCI repository of machine learning databases. Machine-readable data repository","authors":"C Blake, E Keogh, C J Merz","year":"1998","venue":null,"doi":null,"url":"http://-www.ics.uci.edu/'\"\"mlearn/MLRepository.html","internal_url":null},{"position":3,"raw":"[Breiman et al., 1984] L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone. Classification andRe- gression Trees. Wadsworth, Belmont, CA, 1984.","title":"Classification andRegression Trees","authors":"L Breiman, J H Friedman, R A Olshen, C J Stone","year":"1984","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brunk and Pazzani, 1991] C. Brunk and M. J. Pazzani. An investigation of noise-tolerant relational concept learning algorithms. In Proceedings of the Eighth International Workshop on Machine Learning, pages 389-393, Evanston , IL, 1991.","title":"An investigation of noise-tolerant relational concept learning algorithms","authors":"C Brunk, M J Pazzani","year":"1991","venue":"Proceedings of the Eighth International Workshop on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"Morgan Kaufmann. [Cestnik, 1990]","title":"Cestnik","authors":"Morgan Kaufmann","year":"1990","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"B. Cestnik. Estimating probabilities: A crucial task in machine learning. In Proceedings of the Ninth European Conference on Artificial Intelligence, pages 147-149, Stockholm, Sweden, 1990. Pitman.","title":"Estimating probabilities: A crucial task in machine learning","authors":"B Cestnik","year":"1990","venue":"Proceedings of the Ninth European Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chatfield, 1995] C. Chatfield. Model uncertainty, data mining and statistical inference. Journal of the Royal Statistical Society A, 158, 1995.","title":"Model uncertainty, data mining and statistical inference","authors":"C Chatfield","year":"1995","venue":"Journal of the Royal Statistical Society A","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chickering and Heckerman, 1997] D. M. Chickering and D. Heckerman. Efficent approximations for the marginal likelihood of Bayesian networks with hidden variables. Machine Learning, 29:181-212, 1997.","title":"Efficent approximations for the marginal likelihood of Bayesian networks with hidden variables","authors":"D M Chickering, D Heckerman","year":"1997","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Clark and Boswell, 1991] P. Clark and R. Boswell. Rule induction with CN2: Some recent improvements. In Proceedings of the Sixth European Working Session on Learning, pages 151-163, Porto, Portugal, 1991. Springer-Verlag.","title":"Rule induction with CN2: Some recent improvements","authors":"P Clark, R Boswell","year":"1991","venue":"Proceedings of the Sixth European Working Session on Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Clark and Niblett, 1989] P. Clark and T. Niblett . The CN2 induction algorithm. Machin e Learning, 3:261- 283, 1989.","title":"The CN2 induction algorithm","authors":"P Clark, T Niblett","year":"1989","venue":"Machin e Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Domingos , 1998a] P. Domingos. Occam's two razors: The sharp and the blunt. In Proceedings of the Fourth International Conference on Knowledge Dis- covery and Data Mining, pages 37-43, New York, NY, 1998. AAAI Press.","title":"Occam's two razors: The sharp and the blunt","authors":"; P Domingos, Domingos","year":"1998","venue":"Proceedings of the Fourth International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Domingos, 1998b] P. Domingos. A process-oriented heuristic for model selection. In Proceedings of the Fifteenth International Conference on Machine Learn- ing, pages 127-135, Madison, WI, 1998.","title":"A process-oriented heuristic for model selection","authors":"; P Domingos, Domingos","year":"1998","venue":"Proceedings of the Fifteenth International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"Morgan Kauf- mann.","title":null,"authors":"Morgan Kaufmann","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Efron and Tibshirani , 1993] B. Efron and R. J . Tibshi- rani. An Introdu ction to the Bootstrap. Chapman and Hall, New York, NY, 1993.","title":"An Introdu ction to the Bootstrap","authors":"B Efron, R J Tibshirani","year":"1993","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Freund, 1998] Y. Freund. Self bounding learning algo- rithms. In Proceedings of the Eleventh Annual Con- f erence on Computational Learning Th eory, Madison, WI, 1998.","title":"Self bounding learning algorithms","authors":"Y Freund","year":"1998","venue":"Proceedings of the Eleventh Annual Conf erence on Computational Learning Th eory","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"Morgan Kaufmann.","title":null,"authors":"Morgan Kaufmann","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Gaines, 1989] B. R. Gaines. An ounce of knowledge is worth a ton of data: Quantitative studies of the trade- off between expertise and data based on statistically well-founded empirical induction. In Proceedings of the Sixth International Workshop on Machin e Learn- ing, pages 156-159, Ithaca, NY, 1989.","title":"An ounce of knowledge is worth a ton of data: Quantitative studies of the tradeoff between expertise and data based on statistically well-founded empirical induction","authors":"B R Gaines","year":"1989","venue":"Proceedings of the Sixth International Workshop on Machin e Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Morgan Kauf- mann. [Holte et al., 1989]","title":null,"authors":"Morgan Kaufmann, ; Holte","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"R. C. Holte, L. E. Acker, and B. W. Porter. Concept learning and the problem of small disjuncts. In Proceedings of the Eleventh International Joint Conference on Artificial Intelligence, pages 813- 818, Detroit, MI, 1989.","title":"Concept learning and the problem of small disjuncts","authors":"R C Holte, L E Acker, B W Porter","year":"1989","venue":"Proceedings of the Eleventh International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/1989/130"},{"position":20,"raw":"Morgan Kaufmann.","title":null,"authors":"Morgan Kaufmann","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Jensen and Cohen, 1998] D. Jensen and P. R. Cohen. Multiple comparisons in induction algorithms. Ma- chine Learning, 1998. To appear.","title":"Multiple comparisons in induction algorithms","authors":"D Jensen, P R Cohen","year":"1998","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Jensen and Schmill, 1997] D. Jensen and M. Schmill. Adjusting for multiple comparisons in decision tree pruning. In Proceedings of the Third International Conference on Know ledge Discovery and Data Min- ing, pages 195-198, Newport Beach, CA , 1997. AAAI Press.","title":"Adjusting for multiple comparisons in decision tree pruning","authors":"D Jensen, M Schmill","year":"1997","venue":"Proceedings of the Third International Conference on Know ledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Kearns and Vazirani, 1994] M. J . Kearns and U. V. Vazirani. An Introduction to Computational Learning Theory. MIT Press, Cambridge, MA, 1994.","title":"An Introduction to Computational Learning Theory","authors":"M J Kearns, U V Vazirani","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Kearns , 1993] M. Kearns. Efficient noise-tolerant learn- ing from statistical queries. In Proceedings of the Twenty-Fifth ACM Symposium on the Theory of Computing, pages 392-401, New York, NY, 1993. ACM Press.","title":"Efficient noise-tolerant learning from statistical queries","authors":"M Kearns","year":"1993","venue":"Proceedings of the Twenty-Fifth ACM Symposium on the Theory of Computing","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"Klockars and Sax, 1986) A. J. Klockars and G. Sax. Multiple Comparisons. Sage, Beverly Hills, CA, 1986.","title":"Multiple Comparisons","authors":"Sax ; A J Klockars, G Klockars, Sax","year":"1986","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Michalski, 1983] R. S. Michalski. A theory and method- ology of inductive learning. Artificial Intelligence, 20:111-161, 1983.","title":"A theory and methodology of inductive learning","authors":"R S Michalski","year":"1983","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ng, 1997] A. Y. Ng. Preventing \"overfitting\" of cross- validation data. In Proceedings of the Fourteenth In- ternational Conference on Machine Learning, pages 245-253 , Nashville, TN, 1997.","title":"Preventing \"overfitting\" of crossvalidation data","authors":"A Y Ng","year":"1997","venue":"Proceedings of the Fourteenth International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"Morgan Kaufmann. [Pagallo and Haussler, 1990]","title":"Pagallo and Haussler","authors":"Morgan Kaufmann","year":"1990","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"G. Pagallo and D. Haus- sler. Boolean feature discovery in empirical learning. Machine Learning, 3:71-99, 1990.","title":"Boolean feature discovery in empirical learning","authors":"G Pagallo, D Haussler","year":"1990","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"Quinlan and Cameron-Jones, 1995) J. R. Quinlan and R. M. Cameron-Jones. Oversearching and layered search in empirical learning. In Proceedings of the Fourteenth International Joint Conference on Artifi- cial Intelligence, pages 1019-1024, Montreal, Canada, 1995.","title":"Oversearching and layered search in empirical learning","authors":"Quinlan, Cameron-Jones ; ) J R Quinlan, R M Cameron-Jones","year":"1995","venue":"Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/1995/132"},{"position":31,"raw":"Morgan Kaufmann. [Quinlan , 1993)","title":null,"authors":"Morgan Kaufmann","year":"1993","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"J. R. Quinlan. C4 .5: Programs forMa- chine Learning.","title":"C4 .5: Programs forMachine Learning","authors":"J R Quinlan","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"Morgan Kaufmann, San Mateo, CA, 1993.","title":null,"authors":"Morgan Kaufmann, San Mateo, Ca","year":"1993","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"Rissanen, 1978) J. Rissanen. Modeling by shortest data description. Automatica, 14:465-471, 1978.","title":"Modeling by shortest data description","authors":"; J Rissanen, Rissanen","year":"1978","venue":"Automatica","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"Shawe-Taylor et al. , 1996) J . Shawe-Taylor, P. L. Bartlett, R. C. Williamson, and M. Anthony. Struc- tural risk minimization over data-dependent hierar- chies. Technical Report NC-TR-96-053, Department of Computer Science, Royal Holloway, University of London, Egham, UK, 1996.","title":"Structural risk minimization over data-dependent hierarchies","authors":"Shawe-Taylor","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"Stone, 1974)","title":null,"authors":"Stone","year":"1974","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"M. Stone. Cross-validatory choice and as- sessment of statistical predictions. Journal of the Royal Statistical Society B, 36:111-147, 1974.","title":"Cross-validatory choice and assessment of statistical predictions","authors":"M Stone","year":"1974","venue":"Journal of the Royal Statistical Society B","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Vapnik, 1995] V. N. Vapnik. The Nature of Statisti- cal Learning Theory. Springer-Verlag, New York, NY, 1995.","title":"The Nature of Statistical Learning Theory","authors":"V N Vapnik","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}