{"type":"ScholarlyArticle","title":"Multivariate Probability Calibration with Isotonic Bernstein Polynomials","authors":[{"name":"Yongqiao Wang","slug":"yongqiao-wang","given":"Yongqiao","family":"Wang","affiliation":"College of Finance , Zhejiang Gongshang University , China"},{"name":"Xudong Liu","slug":"xudong-liu","given":"Xudong","family":"Liu","affiliation":"School of Computing , University of North Florida , USA"}],"abstract":"Multivariate probability calibration is the problem of predicting class membership probabilities from classification scores of multiple classifiers. To achieve better performance, the calibrating function is often required to be coordinate-wise non-decreasing; that is, for every classifier, the higher the score,  the higher the probability of the class labeling being positive. To this end, we propose a multivariate regression method based on shape-restricted Bernstein polynomials. This method is universally flexible: it can approximate any continuous calibrating function with any specified error, as the polynomial degree increases to infinite. Moreover, it is universally consistent: the estimated calibrating function converges to any continuous calibrating function, as the training size increases to infinity. Our empirical study shows that the proposed method achieves better calibrating performance than benchmark methods.","doi":"10.24963/ijcai.2020/353","url":"https://ijcai.org/proceedings/2020/353","pdf":"https://ijcai.org/proceedings/2020/0353.pdf","markdown":"https://ijcai.org/proceedings/2020/353.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/353","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":353,"pages":{"first":2547,"last":2553},"keywords":["Machine Learning: Ensemble Methods"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24722/file_24722.m3u8","length_sec":282,"provider":"hls"}],"references":[{"position":1,"raw":"[Ben-Tal, 2019] Aharon Ben-Tal. Lectures on modern con- vex optimization. Technical report, School of Industrial & Systems Engineering, Georgia Institute of Technology, 2019.","title":"Lectures on modern convex optimization","authors":"Ben-Tal","year":"2019","venue":"Aharon Ben-Tal","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[DeVore and Lorentz, 1993] Ronald A. DeVore and George G. Lorentz. Constructive Approximation. Springer-Verlag, 1993.","title":"Constructive Approximation","authors":"Ronald A Devore, George G Lorentz","year":"1993","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dua and Graff, 2017] Dheeru Dua and Casey Graff. UCI Machine Learning Repository, 2017.","title":"UCI Machine Learning Repository","authors":"Graff Dua","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Grant and Boyd, 2014] Michael Grant and Stephen Boyd. CVX: Matlab software for disciplined convex program- ming, version 2.1. http://cvxr.com/cvx, March 2014.","title":"CVX: Matlab software for disciplined convex programming","authors":"Boyd ; Grant, Michael Grant, Stephen Boyd","year":"2014","venue":null,"doi":null,"url":"http://cvxr.com/cvx","internal_url":null},{"position":5,"raw":"[Guo et al., 2017] Chuan Guo, Geoff Pleiss, Yu Sun, and K- ilian Q. Weinberger. On calibration of modern neural net- works. In Proc. of ICML'17, pages 1321-1330, 2017.","title":"On calibration of modern neural networks","authors":"Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q Weinberger","year":"2017","venue":"Proc. of ICML'17","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Györfi et al., 2006] László Györfi, Michael Kohler, Adam Krzyzak, and Harro Walk. A Distribution-free Theory of Nonparametric Regression. Springer Science & Business Media, 2006.","title":"A Distribution-free Theory of Nonparametric Regression","authors":"Györfi","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2015] Jiazhen He, James Bailey, Benjamin I. P. Rubinstein, and Rui Zhang. Identifying at-risk students in massive open online courses. In Proc. of AAAI'15, pages 1749-1755, 2015.","title":"Identifying at-risk students in massive open online courses","authors":"He","year":"2015","venue":"Proc. of AAAI'15","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jiang et al., 2011] Xiaoqian Jiang, Melanie Osl, Jihoon Kim, and Lucila Ohno-Machado. Smooth isotonic regres- sion: A new method to calibrate predictive models. AMIA Summits on Translational Science Proceedings, 2011:16, 2011.","title":"Smooth isotonic regression: A new method to calibrate predictive models","authors":"Xiaoqian Jiang, Melanie Osl, Jihoon Kim, Lucila Ohno-Machado","year":"2011","venue":"AMIA Summits on Translational Science Proceedings","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kuleshov and Ermon, 2017] Volodymyr Kuleshov and Ste- fano Ermon. Estimating uncertainty online against an ad- versary. In Proc. of AAAI'17, pages 2110-2116, 2017.","title":"Volodymyr Kuleshov and Stefano Ermon. Estimating uncertainty online against an adversary","authors":"Ermon Kuleshov","year":"2017","venue":"Proc. of AAAI'17","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kuleshov and Liang, 2015] Volodymyr Kuleshov and Per- cy S Liang. Calibrated structured prediction. In Ad- vances in Neural Information Processing Systems 28, pages 3474-3482. MIT Press, 2015.","title":"Volodymyr Kuleshov and Percy S Liang. Calibrated structured prediction","authors":"Liang Kuleshov","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kuleshov et al., 2018] Volodymyr Kuleshov, Nathan Fen- ner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In Proc. of ICML'18, pages 2801-2809, 2018.","title":"Accurate uncertainties for deep learning using calibrated regression","authors":"Volodymyr Kuleshov, Nathan Fenner, Stefano Ermon","year":"2018","venue":"Proc. of ICML'18","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Leathart et al., 2019] Tim Leathart, Eibe Frank, Bernhard Pfahringer, and Geoffrey Holmes. On calibration of nested dichotomies. In Proc. of PAKDD'19, pages 69-80, 2019.","title":"On calibration of nested dichotomies","authors":"Leathart","year":"2019","venue":"Proc. of PAKDD'19","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Naeini and Cooper, 2016] Mahdi Pakdaman Naeini and Gregory F Cooper. Binary classifier calibration using an ensemble of near isotonic regression models. In Proc. of ICDM'16, pages 360-369, 2016.","title":"Mahdi Pakdaman Naeini and Gregory F Cooper. Binary classifier calibration using an ensemble of near isotonic regression models","authors":"Cooper Naeini","year":"2016","venue":"Proc. of ICDM'16","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ott et al., 2018] Myle Ott, Michael Auli, David Grangier, and Marc'Aurelio Ranzato. Analyzing uncertainty in neu- ral machine translation. In Proc. of ICML'18, pages 3956- 3965, 2018.","title":"Analyzing uncertainty in neural machine translation","authors":"Myle Ott, Michael Auli, David Grangier, Marc'aurelio Ranzato","year":"2018","venue":"Proc. of ICML'18","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ozdemir et al., 2017] Onur Ozdemir, Thomas G Allen, So- ra Choi, Thakshila Wimalajeewa, and Pramod K Varsh- ney. Copula based classifier fusion under statistical depen- dence. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 40(11):2740-2748, 2017.","title":"Copula based classifier fusion under statistical depen-dence","authors":"Onur Ozdemir, Sora Thomas G Allen, Thakshila Choi, Pramod K Wimalajeewa, Varshney","year":"2017","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Reemtsen and Rückmann, 1998] Rembert Reemtsen and Jan-J Rückmann. Semi-infinite Programming, volume 25. Springer Science & Business Media, 1998.","title":"Semi-infinite Programming","authors":"Rembert Reemtsen, Jan-J Rückmann","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Schwarz and Heider, 2019] Johanna Schwarz and Dominik Heider. GUESS: projecting machine learning scores to well-calibrated probability estimates for clinical decision- making. Bioinformatics, 35(14):2458-2465, 2019.","title":"GUESS: projecting machine learning scores to well-calibrated probability estimates for clinical decisionmaking","authors":"Heider Schwarz, Johanna Schwarz, Dominik Heider","year":"2019","venue":"Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ustun and Rudin, 2019] Berk Ustun and Cynthia Rudin. Learning optimized risk scores. Journal of Machine Learning Research, 20:1-75, 2019.","title":"Learning optimized risk scores","authors":"Rudin Ustun, Berk Ustun, Cynthia Rudin","year":"2019","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vellanki et al., 2019] Pratibha Vellanki, Santu Rana, Sunil Gupta, David Rubin de Celis Leal, Alessandra Sutti, Mur- ray Height, and Svetha Venkatesh. Bayesian functional optimisation with shape prior. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1617-1624, 2019.","title":"Bayesian functional optimisation with shape prior","authors":"Pratibha Vellanki, Santu Rana, Sunil Gupta, David Rubin De, Celis Leal, Alessandra Sutti, Murray Height, Svetha Venkatesh","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang and Ghosh, 2012] J. Wang and S.K. Ghosh. Shape restricted nonparametric regression with bernstein poly- nomials. Computational Statistics & Data Analysis, 56(9):2729 -2741, 2012.","title":"Shape restricted nonparametric regression with bernstein polynomials","authors":"Wang, Ghosh, ] J Wang, S K Ghosh","year":"2012","venue":"Computational Statistics & Data Analysis","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2019] Y. Wang, L. Li, and C. Dang. Calibrat- ing classification probabilities with shape-restricted poly- nomial regression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(8):1813-1827, 2019.","title":"Calibrating classification probabilities with shape-restricted polynomial regression","authors":"Y Wang, L Li, C Dang","year":"2019","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zadrozny and Elkan, 2002] Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate mul- ticlass probability estimates. In Proc. of KDD'02, pages 694-699, 2002.","title":"Transforming classifier scores into accurate multiclass probability estimates","authors":"Elkan Zadrozny, Bianca Zadrozny, Charles Elkan","year":"2002","venue":"Proc. of KDD'02","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhong and Kwok, 2013] Leon Wenliang Zhong and James T Kwok. Accurate probability calibration for mul- tiple classifiers. In Proc. of IJCAI'13, pages 1939-1945, 2013.","title":"Accurate probability calibration for multiple classifiers","authors":"Kwok ; ] Zhong, Wenliang Leon, James T Zhong, Kwok","year":"2013","venue":"Proc. of IJCAI'13","doi":null,"url":null,"internal_url":"/proceedings/2013/286"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}