{"type":"ScholarlyArticle","title":"Bagging is an Optimal PAC Learner (Extended Abstract)","authors":[{"name":"Kasper Green Larsen","slug":"kasper-green-larsen","given":"Kasper","family":"Green Larsen","affiliation":"Aarhus University"}],"abstract":"Determining the optimal sample complexity of PAC learning in the realizable setting was a central open problem in learning theory for decades. Finally,  seminal work by Hanneke gave an algorithm with a provably optimal sample complexity. His algorithm is based on a careful and structured sub-sampling of the training data and then returning a majority vote among hypotheses trained on each of the sub-samples. While being a very exciting theoretical result, it has not had much impact in practice, in part due to inefficiency, since it constructs a polynomial number of sub-samples of the training data, each of linear size.\r\n\r\nIn this work, we prove the surprising result that the practical and classic heuristic bagging (a.k.a. bootstrap aggregation), due to Breiman, is in fact also an optimal PAC learner. Bagging pre-dates Hanneke's algorithm by twenty years and is taught in most undergraduate machine learning courses. Moreover, we show that it only requires a logarithmic number of sub-samples to reach optimality.","doi":"10.24963/ijcai.2024/932","url":"https://ijcai.org/proceedings/2024/932","pdf":"https://ijcai.org/proceedings/2024/0932.pdf","markdown":"https://ijcai.org/proceedings/2024/932.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/932","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":932,"pages":{"first":8411,"last":8415},"keywords":["Machine Learning: ML: Supervised Learning","Machine Learning: ML: Learning theory"],"videos":[],"references":[{"position":1,"raw":"[Auer and Ortner, 2004] Peter Auer and Ronald Ortner. A new pac bound for intersection-closed concept classes. In Learning Theory, pages 408-414. Springer Berlin Heidel- berg, 2004.","title":"A new pac bound for intersection-closed concept classes","authors":"Ortner Auer, Peter Auer, Ronald Ortner","year":"2004","venue":"Learning Theory","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bartlett et al., 1998] Peter Bartlett, Yoav Freund, Wee Sun Lee, and Robert E. Schapire. Boosting the margin: a new explanation for the effectiveness of voting methods. The Annals of Statistics, 26(5):1651 -1686, 1998.","title":"Boosting the margin: a new explanation for the effectiveness of voting methods","authors":"Peter Bartlett, Yoav Freund, Wee Sun Lee, Robert E Schapire","year":"1998","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blumer et al., 1989] Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K Warmuth. Learnability and the vapnik-chervonenkis dimension. Journal of the ACM (JACM), 36(4):929-965, 1989.","title":"Learnability and the vapnik-chervonenkis dimension","authors":"Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, Manfred K Warmuth","year":"1989","venue":"Journal of the ACM (JACM)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Breiman, 1996] Leo Breiman. Bagging predictors. Machine Learning, 24(2):123-140, 1996.","title":"Bagging predictors","authors":"Breiman","year":"1996","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Breiman, 2001] Leo Breiman. Random forests. Machine Learning, 45(1):5-32, 2001.","title":"Leo Breiman. Random forests","authors":"Breiman","year":"2001","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ehrenfeucht et al., 1989] Andrzej Ehrenfeucht, David Haussler, Michael Kearns, and Leslie Valiant. A general lower bound on the number of examples needed for learning. Information and Computation, 82(3):247-261, 1989.","title":"A general lower bound on the number of examples needed for learning","authors":"Andrzej Ehrenfeucht, David Haussler, Michael Kearns, Leslie Valiant","year":"1989","venue":"Information and Computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Freund and Schapire, 1997] Yoav Freund and Robert E Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Com- puter and System Sciences, 55(1):119-139, 1997.","title":"A decision-theoretic generalization of on-line learning and an application to boosting","authors":"Schapire Freund, Yoav Freund, Robert E Schapire","year":"1997","venue":"Journal of Computer and System Sciences","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gao and Zhou, 2013] Wei Gao and Zhi-Hua Zhou. On the doubt about margin explanation of boosting. Artificial In- telligence, 203:1-18, 2013.","title":"On the doubt about margin explanation of boosting","authors":"Wei Gao, Zhi-Hua Zhou","year":"2013","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hanneke, 2016] Steve Hanneke. The optimal sample com- plexity of pac learning. The Journal of Machine Learning Research, 17(1):1319-1333, 2016.","title":"The optimal sample complexity of pac learning","authors":"Steve Hanneke","year":"2016","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Larsen and Ritzert, 2022] Kasper Green Larsen and Martin Ritzert. Optimal weak to strong learning. Advances in Neural Information Processing Systems, 2022. To appear.","title":"Optimal weak to strong learning","authors":"Ritzert ; Larsen, Green Kasper, Martin Larsen, Ritzert","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Larsen, 2023] Kasper Green Larsen. Bagging is an optimal PAC learner. In COLT, volume 195 of Proceedings of Ma- chine Learning Research, pages 450-468. PMLR, 2023.","title":"Bagging is an optimal PAC learner","authors":"Kasper Green Larsen","year":"2023","venue":"Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Simon, 2015] Hans U Simon. An almost optimal pac algo- rithm. In Conference on Learning Theory, pages 1552- 1563. PMLR, 2015.","title":"An almost optimal pac algorithm","authors":"Hans U Simon","year":"2015","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Valiant, 1984] Leslie G Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134-1142, 1984.","title":"A theory of the learnable","authors":"G Leslie, Valiant","year":"1984","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Vapnik and Chervonenkis, 1971] V. N. Vapnik and A. Ya. Chervonenkis. On the uniform convergence of relative fre- quencies of events to their probabilities. Theory of Proba- bility and its Applications, 16(2):264-280, 1971.","title":"On the uniform convergence of relative frequencies of events to their probabilities","authors":"V N Vapnik, A Ya, Chervonenkis","year":"1971","venue":"Theory of Probability and its Applications","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Vapnik, 1982] Vladimir Vapnik. Estimation of Depen- dences Based on Empirical Data: Springer Series in Statistics (Springer Series in Statistics). Springer-Verlag, Berlin, Heidelberg, 1982.","title":"Estimation of Dependences Based on Empirical Data: Springer Series in Statistics","authors":"Vladimir Vapnik","year":"1982","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}