{"type":"ScholarlyArticle","title":"Differentiable Model Selection for Ensemble Learning","authors":[{"name":"James Kotary","slug":"james-kotary","given":"James","family":"Kotary","affiliation":"University of Virginia"},{"name":"Vincenzo Di Vito","slug":"vincenzo-di-vito","given":"Vincenzo","family":"Di Vito","affiliation":"University of Virginia"},{"name":"Ferdinando Fioretto","slug":"ferdinando-fioretto","given":"Ferdinando","family":"Fioretto","affiliation":"University of Virginia"}],"abstract":"Model selection is a strategy aimed at creating accurate and robust models by identifying the optimal model for classifying any particular input sample. This paper proposes a novel framework for differentiable selection of groups of models by integrating machine learning and combinatorial optimization.\r\nThe framework is tailored for ensemble learning with a strategy that learns to combine the predictions of appropriately selected pre-trained ensemble models. It does so by modeling the ensemble learning task as a differentiable selection program trained end-to-end over a pretrained ensemble to optimize task performance. The proposed framework demonstrates its versatility and effectiveness, outperforming conventional and advanced consensus rules across a variety of classification tasks.","doi":"10.24963/ijcai.2023/217","url":"https://ijcai.org/proceedings/2023/217","pdf":"https://ijcai.org/proceedings/2023/0217.pdf","markdown":"https://ijcai.org/proceedings/2023/217.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/217","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Constraint Satisfaction and Optimization","track":"Main Track","number":217,"pages":{"first":1954,"last":1962},"keywords":["Machine Learning: ML: Applications","Constraint Satisfaction and Optimization: CSO: Constraint optimization"],"videos":[],"references":[{"position":1,"raw":"[Abernethy et al., 2016] Jacob Abernethy, Chansoo Lee, and Ambuj Tewari. Perturbation techniques in online learning and optimization. Perturbations, Optimization, and Statis- tics, 233, 2016.","title":"Perturbation techniques in online learning and optimization","authors":"Abernethy","year":"2016","venue":"Perturbations, Optimization, and Statistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bazaraa et al., 2008] Mokhtar S Bazaraa, John J Jarvis, and Hanis D Sherali. Linear programming and network flows. John Wiley & Sons, 2008.","title":"Linear programming and network flows","authors":"John J Mokhtar S Bazaraa, Hanis D Jarvis, Sherali","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Berthet et al., 2020] Quentin Berthet, Mathieu Blondel, Olivier Teboul, Marco Cuturi, Jean-Philippe Vert, and Francis Bach. Learning with differentiable pertubed op- timizers. Advances in neural information processing sys- tems, 33:9508-9519, 2020.","title":"Learning with differentiable pertubed optimizers","authors":"Quentin Berthet, Mathieu Blondel, Olivier Teboul, Marco Cuturi, Jean-Philippe Vert, Francis Bach","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cormen et al., 2022] Thomas H Cormen, Charles E Leiser- son, Ronald L Rivest, and Clifford Stein. Introduction to algorithms. MIT press, 2022.","title":"Introduction to algorithms","authors":"Charles E Thomas H Cormen, Ronald L Leiserson, Clifford Rivest, Stein","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deng, 2012] Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29(6):141-142, 2012.","title":"The mnist database of handwritten digit images for machine learning research [best of the web","authors":"Li Deng","year":"2012","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Diederik and Jimmy, 2014] PK Diederik and B Jimmy. Adam: A method for stochastic optimization. iclr. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Jimmy ; ] Pk Diederik, B Diederik, Jimmy","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dona and Gallinari, 2021] Jérémie Dona and Patrick Gal- linari. Differentiable feature selection, a reparameteri- zation approach. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Con- ference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part III 21, pages 414-429. Springer, 2021.","title":"Jérémie Dona and Patrick Gallinari. Differentiable feature selection, a reparameterization approach","authors":"Dona, Gallinari","year":"2021","venue":"Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Folland, 1999] Gerald B Folland. Real analysis: modern techniques and their applications, volume 40. John Wiley & Sons, 1999.","title":"Real analysis: modern techniques and their applications","authors":"Gerald B Folland","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fu et al., 2016] Jie Fu, Hongyin Luo, Jiashi Feng, Kian Hsiang Low, and Tat-Seng Chua. Drmad: Distilling reverse-mode automatic differentiation for optimizing hyperparameters of deep neural networks. arXiv preprint arXiv:1601.00917, 2016.","title":"Drmad: Distilling reverse-mode automatic differentiation for optimizing hyperparameters of deep neural networks","authors":"Jie Fu, Hongyin Luo, Jiashi Feng, Kian Hsiang Low, Tat-Seng Chua","year":"2016","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2016/211"},{"position":10,"raw":"[Gumbel, 1954] Emil Julius Gumbel. Statistical theory of extreme values and some practical applications: a series of lectures, volume 33. US Government Printing Office, 1954.","title":"Statistical theory of extreme values and some practical applications: a series of lectures","authors":"Emil Julius Gumbel, Gumbel","year":"1954","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ju et al., 2018] Cheng Ju, Aurélien Bibaut, and Mark van der Laan. The relative performance of ensemble meth- ods with deep convolutional neural networks for image classification. Journal of Applied Statistics, 45(15):2800- 2818, 2018.","title":"The relative performance of ensemble methods with deep convolutional neural networks for image classification","authors":"Cheng Ju, Aurélien Bibaut, Mark Van Der Laan","year":"2018","venue":"Journal of Applied Statistics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kairouz et al., 2021] Peter Kairouz, H. Brendan McMa- han, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Oliveira, Hubert Eichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konecný, Alek- sandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Hang Qi, Daniel Ramage, Ramesh Raskar, Mariana Raykova, Dawn Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. Foundations and Trends R in Machine Learning, 14(1-2):1-210, 2021.","title":"Advances and open problems in federated learning","authors":"H Brendan Peter Kairouz, Brendan Mcmahan, Aurélien Avent, Mehdi Bellet, Bennis, Nitin Arjun, Kallista Bhagoji, Zachary Bonawitz, Graham Charles, Rachel Cormode, Cummings, G L Rafael, Hubert D'oliveira, Salim El Eichner, David Rouayheb, Josh Evans, Zachary Gardner, Adrià Garrett, Badih Gascón, Phillip B Ghazi, Marco Gibbons, Zaid Gruteser, Chaoyang Harchaoui, Lie He, Zhouyuan He, Ben Huo, Justin Hutchinson, Martin Hsu, Tara Jaggi, Gauri Javidi, Mikhail Joshi, Jakub Khodak, Aleksandra Konecný, Farinaz Korolova, Sanmi Koushanfar, Tancrède Koyejo, Yang Lepoint, Prateek Liu, Mehryar Mittal, Richard Mohri, Ayfer Nock, Rasmus Özgür, Hang Pagh, Daniel Qi, Ramesh Ramage, Mariana Raskar, Dawn Raykova, Weikang Song, Sebastian U Song, Ziteng Stich, Ananda Sun, Florian Theertha Suresh, Praneeth Tramèr, Jianyu Vepakomma, Li Wang, Zheng Xiong, Qiang Xu, Felix X Yang, Han Yu, Sen Yu, Zhao","year":"2021","venue":"Foundations and Trends R in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kotary et al., 2021] James Kotary, Ferdinando Fioretto, Pascal Van Hentenryck, and Bryan Wilder. End-to-end constrained optimization learning: A survey. In Proceed- ings of the Thirtieth International Joint Conference on Ar- tificial Intelligence, IJCAI-21, pages 4475-4482, 2021.","title":"End-to-end constrained optimization learning: A survey","authors":"James Kotary, Ferdinando Fioretto, Pascal Van Hentenryck, Bryan Wilder","year":"2021","venue":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21","doi":null,"url":null,"internal_url":"/proceedings/2021/610"},{"position":14,"raw":"[Kotary et al., 2022] James Kotary, Vincenzo Di Vito, and Ferdinando Fioretto. Differentiable model selection for ensemble learning. arXiv preprint arXiv:2211.00251, 2022.","title":"Differentiable model selection for ensemble learning","authors":"James Kotary, Vincenzo Di Vito, Ferdinando Fioretto","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny im- ages. 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2016] Kuang Liu, Mingmin Zhang, and Zhigeng Pan. Facial expression recognition with cnn ensemble. In 2016 International Conference on Cyberworlds (CW), pages 163-166, 2016.","title":"Facial expression recognition with cnn ensemble","authors":"Kuang Liu, Mingmin Zhang, Zhigeng Pan","year":"2016","venue":"2016 International Conference on Cyberworlds (CW)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mienye and Sun, 2022] Ibomoiye Domor Mienye and Yanxia Sun. A survey of ensemble learning: Concepts, algorithms, applications, and prospects. IEEE Access, 10:99129-99149, 2022.","title":"Ibomoiye Domor Mienye and Yanxia Sun. A survey of ensemble learning: Concepts, algorithms, applications, and prospects","authors":"Sun Mienye","year":"2022","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Paszke et al., 2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neu- ral Information Processing Systems 32, pages 8024-8035.","title":"Pytorch: An imperative style, highperformance deep learning library","authors":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Devito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, Soumith Chintala","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"Curran Associates, Inc., 2019.","title":null,"authors":"Curran Associates, Inc","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ribeiro et al., 2015] Mauro Ribeiro, Katarina Grolinger, and Miriam A.M. Capretz. Mlaas: Machine learning as a service. In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), pages 896- 902, 2015.","title":"Mlaas: Machine learning as a service","authors":"Mauro Ribeiro, Katarina Grolinger, Miriam A M Capretz","year":"2015","venue":"IEEE 14th International Conference on Machine Learning and Applications (ICMLA)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ruder, 2016] Sebastian Ruder. An overview of gradi- ent descent optimization algorithms. arXiv preprint arXiv:1609.04747, 2016.","title":"An overview of gradient descent optimization algorithms","authors":"Sebastian Ruder","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sheth and Fusi, 2020] Rishit Sheth and Nicoló Fusi. Differ- entiable feature selection by discrete relaxation. In Inter- national Conference on Artificial Intelligence and Statis- tics, pages 1564-1572. PMLR, 2020.","title":"Rishit Sheth and Nicoló Fusi. Differentiable feature selection by discrete relaxation","authors":"Fusi Sheth","year":"2020","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wilder et al., 2019] Bryan Wilder, Bistra Dilkina, and Milind Tambe. Melding the data-decisions pipeline: Decision-focused learning for combinatorial optimization. In AAAI, volume 33, pages 1658-1665, 2019.","title":"Melding the data-decisions pipeline: Decision-focused learning for combinatorial optimization","authors":"Bryan Wilder, Bistra Dilkina, Milind Tambe","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Witten et al., 2005] Ian H Witten, Eibe Frank, Mark A Hall, Christopher J Pal, and MINING DATA. Practical machine learning tools and techniques. In Data Mining, volume 2, 2005.","title":"Practical machine learning tools and techniques","authors":"Eibe Ian H Witten, Mark A Frank, Christopher J Hall, Mining Pal, Data","year":"2005","venue":"Data Mining","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zeiler, 2012] Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.","title":"Adadelta: an adaptive learning rate method","authors":"D Matthew, Zeiler","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhifei Zhang, 2017] Hairong Qi Zhifei Zhang, Yang Song. Age progression/regression by conditional adversarial au- toencoder. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017.","title":"Age progression/regression by conditional adversarial autoencoder","authors":"Hairong Qi, Zhifei Zhang, Yang Song","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}