{"type":"ScholarlyArticle","title":"Calibrating a Deep Neural Network with Its Predecessors","authors":[{"name":"Linwei Tao","slug":"linwei-tao","given":"Linwei","family":"Tao","affiliation":"School of Computer Science , Faculty of Engineering , University of Sydney , Australia"},{"name":"Minjing Dong","slug":"minjing-dong","given":"Minjing","family":"Dong","affiliation":"School of Computer Science , Faculty of Engineering , University of Sydney , Australia"},{"name":"Daochang Liu","slug":"daochang-liu","given":"Daochang","family":"Liu","affiliation":"School of Computer Science , Faculty of Engineering , University of Sydney , Australia"},{"name":"Changming Sun","slug":"changming-sun","given":"Changming","family":"Sun","affiliation":"CSIRO's Data61 , Australia"},{"name":"Chang Xu","slug":"chang-xu","given":"Chang","family":"Xu","affiliation":"School of Computer Science , Faculty of Engineering , University of Sydney , Australia"}],"abstract":"Confidence calibration - the process to calibrate the output probability distribution of neural networks - is essential for safety-critical applications of such networks. Recent works verify the link between mis-calibration and overfitting. However, early stopping, as a well-known technique to mitigate overfitting, fails to calibrate networks. In this work, we study the limitions of early stopping and comprehensively analyze the overfitting problem of a network considering each individual block. We then propose a novel regularization method, predecessor combination search (PCS), to improve calibration by searching a combination of best-fitting block predecessors, where block predecessors are the corresponding network blocks with weight parameters from earlier training stages. PCS achieves the state-of-the-art calibration performance on multiple datasets and architectures. In addition, PCS improves model robustness under dataset distribution shift. Supplementary material and code are available at https://github.com/Linwei94/PCS","doi":"10.24963/ijcai.2023/475","url":"https://ijcai.org/proceedings/2023/475","pdf":"https://ijcai.org/proceedings/2023/0475.pdf","markdown":"https://ijcai.org/proceedings/2023/475.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/475","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":475,"pages":{"first":4271,"last":4279},"keywords":["Machine Learning: ML: Classification"],"videos":[],"references":[{"position":1,"raw":"[Bohdal et al., 2021] Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Meta-calibration: Meta-learning of model calibration using differentiable expected calibration error. arXiv preprint arXiv:2106.09613, 2021.","title":"Meta-calibration: Meta-learning of model calibration using differentiable expected calibration error","authors":"Bohdal","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bojarski et al., 2016] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Pra- soon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self- driving cars. arXiv preprint arXiv:1604.07316, 2016.","title":"End to end learning for selfdriving cars","authors":"Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brier et al., 1950] Glenn W Brier, Simon Kornblith, and Geoffrey E Hinton. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.","title":"Verification of forecasts expressed in terms of probability","authors":"Glenn W Brier, Simon Kornblith, Geoffrey E Hinton","year":"1950","venue":"Monthly weather review","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Caruana et al., 2015] Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 1721-1730, 2015.","title":"Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission","authors":"Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, Noemie Elhadad","year":"2015","venue":"Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deng and Zhang, 2021] Xiang Deng and Zhongfei Zhang. Learning with retrospection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7201-7209, 2021.","title":"Learning with retrospection","authors":"Xiang Deng, Zhongfei Zhang","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ding et al., 2020] Yukun Ding, Jinglan Liu, Jinjun Xiong, and Yiyu Shi. Revisiting the evaluation of uncertainty es- timation and its application to explore model complexity- uncertainty trade-off. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 4-5, 2020.","title":"Revisiting the evaluation of uncertainty estimation and its application to explore model complexityuncertainty trade-off","authors":"Yukun Ding, Jinglan Liu, Jinjun Xiong, Yiyu Shi","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goodfellow et al., 2013] Ian J Goodfellow, Yaroslav Bula- tov, Julian Ibarz, Sacha Arnoud, and Vinay Shet. Multi- digit number recognition from street view imagery us- ing deep convolutional neural networks. arXiv preprint arXiv:1312.6082, 2013.","title":"Multidigit number recognition from street view imagery using deep convolutional neural networks","authors":"Ian J Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnoud, Vinay Shet","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guo et al., 2017] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural net- works. In International conference on machine learning, pages 1321-1330. PMLR, 2017.","title":"On calibration of modern neural networks","authors":"Guo","year":"2017","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gupta et al., 2020] Kartik Gupta, Amir Rahimi, Tha- laiyasingam Ajanthan, Thomas Mensink, Cristian Smin- chisescu, and Richard Hartley. Calibration of neural net- works using splines. arXiv preprint arXiv:2006.12800, 2020.","title":"Calibration of neural networks using splines","authors":"Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, Richard Hartley","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hendrycks and Dietterich, 2018] Dan Hendrycks and Thomas G Dietterich. Benchmarking neural network robustness to common corruptions and surface variations. arXiv preprint arXiv:1807.01697, 2018.","title":"Benchmarking neural network robustness to common corruptions and surface variations","authors":"Dan Hendrycks, Thomas G Dietterich","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hendrycks et al., 2019] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781, 2019.","title":"Augmix: A simple data processing method to improve robustness and uncertainty","authors":"Dan Hendrycks, Norman Mu, D Ekin, Barret Cubuk, Justin Zoph, Balaji Gilmer, Lakshminarayanan","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 4700-4708, 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hui and Belkin, 2020] Like Hui and Mikhail Belkin. Eval- uation of neural architectures trained with square loss vs cross-entropy in classification tasks. arXiv preprint arXiv:2006.07322, 2020.","title":"Evaluation of neural architectures trained with square loss vs cross-entropy in classification tasks","authors":"Belkin ; Hui, Like Hui, Mikhail Belkin","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jang et al., 2016] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016.","title":"Categorical reparameterization with gumbel-softmax","authors":"Eric Jang, Shixiang Gu, Ben Poole","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Karandikar et al., 2021] Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Michael C Mozer, and Becca Roelofs. Soft calibration objectives for neural networks. Advances in Neural Infor- mation Processing Systems, 34:29768-29779, 2021.","title":"Soft calibration objectives for neural networks","authors":"Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Becca Michael C Mozer, Roelofs","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Krishnan and Tickoo, 2020] Ranganath Krishnan and Omesh Tickoo. Improving model calibration with accu- racy versus uncertainty optimization. Advances in Neural Information Processing Systems, 33:18237-18248, 2020.","title":"Improving model calibration with accuracy versus uncertainty optimization","authors":"Ranganath Krishnan, Omesh Tickoo","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Krizhevsky, 2012] Alex Krizhevsky. Learning multiple lay- ers of features from tiny images. University of Toronto, 05 2012.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kull et al., 2017] Meelis Kull, Telmo Silva Filho, and Peter Flach. Beta calibration: a well-founded and easily im- plemented improvement on logistic calibration for binary classifiers. In Artificial Intelligence and Statistics, pages 623-631. PMLR, 2017.","title":"Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers","authors":"Meelis Kull, Telmo Silva Filho, Peter Flach","year":"2017","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Kull et al., 2019] Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. Advances in neural information processing systems, 32, 2019.","title":"Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration","authors":"Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, Peter Flach","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Kumar et al., 2018] Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural net- works from kernel mean embeddings. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th Inter- national Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2805- 2814. PMLR, 10-15 Jul 2018.","title":"Trainable calibration measures for neural networks from kernel mean embeddings","authors":"Aviral Kumar, Sunita Sarawagi, Ujjwal Jain","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Kumar et al., 2019] Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified uncertainty calibration. Advances in Neural Information Processing Systems, 32, 2019.","title":"Verified uncertainty calibration","authors":"Ananya Kumar, Percy S Liang, Tengyu Ma","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Mukhoti et al., 2020] Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 33:15288-15299, 2020.","title":"Calibrating deep neural networks using focal loss","authors":"Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, Puneet Dokania","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Müller et al., 2019] Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.","title":"When does label smoothing help?","authors":"Müller","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Naeini et al., 2015] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.","title":"Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning","authors":"Naeini","year":"2015","venue":"Twenty-Ninth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Nixon et al., 2019] Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Mea- suring calibration in deep learning. In CVPR Workshops, volume 2, 2019.","title":"Measuring calibration in deep learning","authors":"Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, Dustin Tran","year":"2019","venue":"CVPR Workshops","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Ovadia et al., 2019] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural infor- mation processing systems, 32, 2019.","title":"Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift","authors":"Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, Jasper Snoek","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Pham et al., 2021] Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11557-11568, 2021.","title":"Meta pseudo labels","authors":"Hieu Pham, Zihang Dai, Qizhe Xie, Quoc V Le","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Platt et al., 1999] John Platt, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Michael C Mozer, and Becca\" Roelofs. Probabilistic outputs for sup- port vector machines and comparisons to regularized like- lihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.","title":"Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods","authors":"John Platt, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Becca \" Michael C Mozer, Roelofs","year":"1999","venue":"Advances in large margin classifiers","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Prechelt, 1998] Lutz Prechelt. Early stopping-but when? In Neural Networks: Tricks of the trade, pages 55-69. Springer, 1998.","title":"Lutz Prechelt. Early stopping-but when?","authors":"Prechelt","year":"1998","venue":"Neural Networks: Tricks of the trade","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Qiao et al., 2021] Siyuan Qiao, Liang-Chieh Chen, and Alan Yuille. Detectors: Detecting objects with recursive feature pyramid and switchable atrous convolution. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10213-10224, 2021.","title":"Detectors: Detecting objects with recursive feature pyramid and switchable atrous convolution","authors":"Siyuan Qiao, Liang-Chieh Chen, Alan Yuille","year":"2021","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Roelofs et al., 2022] Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, and Michael C Mozer. Mitigating bias in calibration error estimation. In International Conference on Artificial Intelligence and Statistics, pages 4036-4054.","title":"Mitigating bias in calibration error estimation","authors":"Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, Michael C Mozer","year":null,"venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"PMLR, 2022.","title":null,"authors":"Pmlr","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Szegedy et al., 2016] Christian Szegedy, Vincent Van- houcke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.","title":"Rethinking the inception architecture for computer vision","authors":"Szegedy","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Thulasidasan et al., 2019] Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibra- tion and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.","title":"On mixup training: Improved calibration and predictive uncertainty for deep neural networks","authors":"Gopinath Sunil Thulasidasan, Jeff A Chennupati, Tanmoy Bilmes, Sarah Bhattacharya, Michalak","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Tolstikhin et al., 2021] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural infor- mation processing systems, 34:24261-24272, 2021.","title":"Mlp-mixer: An all-mlp architecture for vision","authors":"Neil Ilya O Tolstikhin, Alexander Houlsby, Lucas Kolesnikov, Xiaohua Beyer, Thomas Zhai, Jessica Unterthiner, Andreas Yung, Daniel Steiner, Jakob Keysers, Uszkoreit","year":"2021","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Wang et al., 2022] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. Yolov7: Trainable bag-of- freebies sets new state-of-the-art for real-time object de- tectors. arXiv preprint arXiv:2207.02696, 2022.","title":"Yolov7: Trainable bag-offreebies sets new state-of-the-art for real-time object detectors","authors":"Chien-Yao Wang, Alexey Bochkovskiy, Hong-Yuan Mark Liao","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Wenger et al., 2020] Jonathan Wenger, Hedvig Kjellström, and Rudolph Triebel. Non-parametric calibration for clas- sification. In International Conference on Artificial Intel- ligence and Statistics, pages 178-190. PMLR, 2020.","title":"Non-parametric calibration for classification","authors":"Jonathan Wenger, Hedvig Kjellström, Rudolph Triebel","year":"2020","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Wortsman et al., 2022] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: aver- aging weights of multiple fine-tuned models improves ac- curacy without increasing inference time. In International Conference on Machine Learning, pages 23965-23998.","title":"Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time","authors":"Mitchell Wortsman, Gabriel Ilharco, Ya Samir, Rebecca Gadre, Raphael Roelofs, Ari S Gontijo-Lopes, Hongseok Morcos, Ali Namkoong, Yair Farhadi, Simon Carmon, Kornblith","year":null,"venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"PMLR, 2022.","title":null,"authors":"Pmlr","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Zadrozny and Elkan, 2001] Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from de- cision trees and naive bayesian classifiers. In Icml, vol- ume 1, pages 609-616. Citeseer, 2001.","title":"Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers","authors":"Elkan Zadrozny, Bianca Zadrozny, Charles Elkan","year":"2001","venue":"Icml","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Zadrozny and Elkan, 2002] Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multi- class probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694-699, 2002.","title":"Transforming classifier scores into accurate multiclass probability estimates","authors":"Elkan Zadrozny, Bianca Zadrozny, Charles Elkan","year":"2002","venue":"Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Zagoruyko and Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.","title":"Wide residual networks","authors":"Sergey Zagoruyko, Nikos Komodakis","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Zhou et al., 2021] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Probabilistic two-stage detection. arXiv preprint arXiv:2103.07461, 2021.","title":"Probabilistic two-stage detection","authors":"Zhou","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}