{"type":"ScholarlyArticle","title":"Greedy Convex Ensemble","authors":[{"name":"Thanh Tan Nguyen","slug":"thanh-tan-nguyen","given":"Thanh Tan","family":"Nguyen","affiliation":"Queensland University of Technology"},{"name":"Nan Ye","slug":"nan-ye","given":"Nan","family":"Ye","affiliation":"The University of Queensland"},{"name":"Peter Bartlett","slug":"peter-bartlett","given":"Peter","family":"Bartlett","affiliation":"UC Berkeley"}],"abstract":"We consider learning a convex combination of basis models, and present some new theoretical and empirical results that demonstrate the effectiveness of a greedy approach. Theoretically, we first consider whether we can use linear, instead of convex, combinations, and obtain generalization results similar to existing ones for learning from a convex hull. We obtain a negative result that even the linear hull of very simple basis functions can have unbounded capacity, and is thus prone to overfitting; on the other hand, convex hulls are still rich but have bounded capacities. Secondly, we obtain a generalization bound for a general class of Lipschitz loss functions. Empirically, we first discuss how a convex combination can be greedily learned with early stopping, and how a convex combination can be non-greedily learned when the number of basis models is known a priori. Our experiments suggest that the greedy scheme is competitive with or better than several baselines, including boosting and random forests. The greedy algorithm requires little effort in hyper-parameter tuning, and also seems able to adapt to the underlying complexity of the problem. Our code is available at https://github.com/tan1889/gce.","doi":"10.24963/ijcai.2020/429","url":"https://ijcai.org/proceedings/2020/429","pdf":"https://ijcai.org/proceedings/2020/0429.pdf","markdown":"https://ijcai.org/proceedings/2020/429.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/429","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":429,"pages":{"first":3101,"last":3107},"keywords":["Machine Learning: Ensemble Methods","Machine Learning: Learning Theory"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25375/file_25375.m3u8","length_sec":297,"provider":"hls"}],"references":[{"position":1,"raw":"[Anthony and Bartlett, 2009] Martin Anthony and Peter L Bartlett. Neural network learning: Theoretical founda- tions. Cambridge University Press, 2009.","title":"Neural network learning: Theoretical foundations","authors":"Martin Anthony, Peter L Bartlett","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bach, 2017] Francis Bach. Breaking the curse of dimension- ality with convex neural networks. The Journal of Machine Learning Research, 18(1):629-681, 2017.","title":"Breaking the curse of dimensionality with convex neural networks","authors":"Francis Bach","year":"2017","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Barron, 1993] Andrew R Barron. Universal approximation bounds for superpositions of a sigmoidal function. Informa- tion Theory, IEEE Transactions on, 39(3):930-945, 1993.","title":"Universal approximation bounds for superpositions of a sigmoidal function","authors":"Andrew R Barron","year":"1993","venue":"Information Theory, IEEE Transactions on","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bartlett and Traskin, 2007] Peter L Bartlett and Mikhail Traskin. Adaboost is consistent. Journal of Machine Learn- ing Research, 8(Oct):2347-2368, 2007.","title":"Adaboost is consistent","authors":"L Peter, Mikhail Bartlett, Traskin","year":"2007","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Breiman, 2001] Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.","title":"Leo Breiman. Random forests","authors":"Breiman","year":"2001","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen and Guestrin, 2016] Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In KDD, pages 785-794. ACM, 2016.","title":"XGBoost: A scalable tree boosting system","authors":"Tianqi Chen, Carlos Guestrin","year":"2016","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dheeru and Karra Taniskidou, 2017] Dua Dheeru and Efi Karra Taniskidou. UCI machine learning repository, 2017.","title":"Dua Dheeru and Efi Karra Taniskidou. UCI machine learning repository","authors":"Karra Dheeru, Taniskidou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Donahue et al., 1997] Michael J Donahue, C Darken, Leonid Gurvits, and Eduardo Sontag. Rates of convex approximation in non-Hilbert spaces. Constructive Approx- imation, 13(2):187-220, 1997.","title":"Rates of convex approximation in non-Hilbert spaces","authors":"C Michael J Donahue, Leonid Darken, Eduardo Gurvits, Sontag","year":"1997","venue":"Constructive Approximation","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Frank and Wolfe, 1956] Marguerite Frank and Philip Wolfe. An algorithm for quadratic programming. Naval research logistics quarterly, 3(1-2):95-110, 1956.","title":"An algorithm for quadratic programming","authors":"Wolfe Frank, Marguerite Frank, Philip Wolfe","year":"1956","venue":"Naval research logistics quarterly","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Freund and Schapire, 1995] Yoav Freund and Robert E Schapire. A desicion-theoretic generalization of on-line learning and an application to boosting. In European con- ference on computational learning theory, pages 23-37. Springer, 1995.","title":"A desicion-theoretic generalization of on-line learning and an application to boosting","authors":"Schapire Freund, Yoav Freund, Robert E Schapire","year":"1995","venue":"European conference on computational learning theory","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gao and Zhou, 2013] Wei Gao and Zhi-Hua Zhou. On the doubt about margin explanation of boosting. Artificial Intelligence, 203:1-18, 2013.","title":"On the doubt about margin explanation of boosting","authors":"Wei Gao, Zhi-Hua Zhou","year":"2013","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Grover and Ermon, 2018] Aditya Grover and Stefano Er- mon. Boosted generative models. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.","title":"Boosted generative models","authors":"Aditya Grover, Stefano Ermon","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Guélat and Marcotte, 1986] Jacques Guélat and Patrice Mar- cotte. Some comments on Wolfe's 'away step'. Mathemati- cal Programming, 35(1):110-119, 1986.","title":"Jacques Guélat and Patrice Marcotte. Some comments on Wolfe's 'away step","authors":"Marcotte Guélat","year":"1986","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jaggi, 2013] Martin Jaggi. Revisiting Frank-Wolfe: Projection-free sparse convex optimization. In ICML, 2013.","title":"Revisiting Frank-Wolfe: Projection-free sparse convex optimization","authors":"Martin Jaggi","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jones, 1992] Lee K. Jones. A simple lemma on greedy ap- proximation in Hilbert space and convergence rates for projection pursuit regression and neural network training. The annals of Statistics, pages 608-613, 1992.","title":"A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training","authors":"K Lee, Jones","year":"1992","venue":"The annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Koltchinskii and Panchenko, 2005] Vladimir Koltchinskii and Dmitry Panchenko. Complexities of convex combina- tions and bounding the generalization error in classification. The Annals of Statistics, 33(4):1455-1496, 2005.","title":"Vladimir Koltchinskii and Dmitry Panchenko. Complexities of convex combinations and bounding the generalization error in classification","authors":"Panchenko Koltchinskii","year":"2005","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lacoste-Julien and Jaggi, 2015] Simon Lacoste-Julien and Martin Jaggi. On the global linear convergence of Frank- Wolfe optimization variants. In NIPS, 2015.","title":"On the global linear convergence of Frank-Wolfe optimization variants","authors":"Jaggi Lacoste-Julien, Simon Lacoste-Julien, Martin Jaggi","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lee et al., 1996] Wee Sun Lee, Peter L Bartlett, and Robert C Williamson. Efficient agnostic learning of neural networks with bounded fan-in. Information Theory, IEEE Transactions on, 42(6):2118-2132, 1996.","title":"Efficient agnostic learning of neural networks with bounded fan-in","authors":"Wee Sun, Lee, Peter L Bartlett, Robert C Williamson","year":"1996","venue":"IEEE Transactions on","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Locatello et al., 2018] Francesco Locatello, Gideon Dres- dner, Rajiv Khanna, Isabel Valera, and Gunnar Rätsch. Boosting black box variational inference. In Advances in Neural Information Processing Systems, 2018.","title":"Boosting black box variational inference","authors":"Francesco Locatello, Gideon Dresdner, Rajiv Khanna, Isabel Valera, Gunnar Rätsch","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Makovoz, 1996] Yuly Makovoz. Random approximants and neural networks. Journal of Approximation Theory, 85(1):98-109, 1996.","title":"Random approximants and neural networks","authors":"Yuly Makovoz, Makovoz","year":"1996","venue":"Journal of Approximation Theory","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mannor et al., 2003] Shie Mannor, Ron Meir, and Tong Zhang. Greedy algorithms for classification-consistency, convergence rates, and adaptivity. Journal of Machine Learning Research, 4(Oct):713-742, 2003.","title":"Greedy algorithms for classification-consistency, convergence rates, and adaptivity","authors":"Shie Mannor, Ron Meir, Tong Zhang","year":"2003","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mason et al., 2000a] Llew Mason, Jonathan Baxter, Peter L Bartlett, Marcus Frean, et al. Functional gradient tech- niques for combining hypotheses. In Advances in Large Margin Classifiers, pages 221-246. MIT, 2000.","title":"Functional gradient techniques for combining hypotheses","authors":"Mason","year":"2000","venue":"Advances in Large Margin Classifiers","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mason et al., 2000b] Llew Mason, Jonathan Baxter, Peter L Bartlett, and Marcus R Frean. Boosting algorithms as gradient descent. In NIPS, pages 512-518, 2000.","title":"Boosting algorithms as gradient descent","authors":"Mason","year":"2000","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Mendelson, 2003] Shahar Mendelson. A few notes on sta- tistical learning theory. In Advanced lectures on machine learning, pages 1-40. Springer, 2003.","title":"A few notes on statistical learning theory","authors":"Shahar Mendelson","year":"2003","venue":"Advanced lectures on machine learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Oglic and Gärtner, 2016] Dino Oglic and Thomas Gärtner. Greedy feature construction. In NeurIPS, 2016.","title":"Dino Oglic and Thomas Gärtner. Greedy feature construction","authors":"Gärtner Oglic","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Pollard, 1984] David Pollard. Convergence of stochastic processes. Springer, 1984.","title":"Convergence of stochastic processes","authors":"David Pollard","year":"1984","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Schapire, 2013] Robert E Schapire. Explaining adaboost. In Empirical inference, pages 37-52. Springer, 2013.","title":"Explaining adaboost","authors":"Robert E Schapire","year":"2013","venue":"Empirical inference","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tolstikhin et al., 2017] Ilya O Tolstikhin, Sylvain Gelly, Olivier Bousquet, Carl-Johann Simon-Gabriel, and Bern- hard Schölkopf. Adagan: Boosting generative models. In NeurIPS, 2017.","title":"Adagan: Boosting generative models","authors":"Sylvain Ilya O Tolstikhin, Olivier Gelly, Carl-Johann Bousquet, Bernhard Simon-Gabriel, Schölkopf","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Viola and Jones, 2004] Paul Viola and Michael J Jones. Ro- bust real-time face detection. IJCV, 57(2):137-154, 2004.","title":"Robust real-time face detection","authors":"Paul Viola, J Michael, Jones","year":"2004","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wager and Walther, 2015] Stefan Wager and Guenther Walther. Adaptive concentration of regression trees, with application to random forests. arXiv preprint arXiv:1503.06388, 2015.","title":"Adaptive concentration of regression trees, with application to random forests","authors":"Walther ; Wager, Stefan Wager, Guenther Walther","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wellner and Song, 2002] Jon A. Wellner and Shuguang Song. An upper bound for uniform entropy numbers, 2002.","title":"An upper bound for uniform entropy numbers","authors":"Jon A Wellner, Shuguang Song","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wyner et al., 2017] Abraham J Wyner, Matthew Olson, Justin Bleich, and David Mease. Explaining the success of adaboost and random forests as interpolating classifiers. The Journal of Machine Learning Research, 18(1):1558- 1590, 2017.","title":"Explaining the success of adaboost and random forests as interpolating classifiers","authors":"Matthew Abraham J Wyner, Justin Olson, David Bleich, Mease","year":"2017","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang, 2003] Tong Zhang. Sequential greedy approximation for certain convex optimization problems. IEEE Transac- tions on Information Theory, 49(3):682-691, 2003.","title":"Sequential greedy approximation for certain convex optimization problems","authors":"Tong Zhang","year":"2003","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}