Proceedings

Accurate Probability Calibration for Multiple Classifiers

Download BibTeX Cite

Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence. Pages 1939-1944.

© 2013 International Joint Conferences on Artificial Intelligence

Abstract

In classification problems,isotonic regression has been commonly used to map the prediction scores to posterior class probabilities. However, isotonic regression may suffer from overfitting, and the learned mapping is often discontinuous. Besides, current efforts mainly focus on the calibration of a single classifier. As different classifiers have different strengths, a combination of themcan lead to better performance. In this paper, we propose a novel probability calibration approach for such an ensemble of classifiers. We first construct isotonic constraints on the desired probabilities based on soft voting of the classifiers. Manifold information is also incorporated to combat overfitting and ensurefunction smoothness. Computationally,the extended isotonic regression model can be learned efficiently by a novel optimization algorithm based on the alternating direction method of multipliers (ADMM). Experimentson a number of real-world data sets demonstrate that the proposed approach consistently outperforms independent classifiers and other combinations of the classifiers' probabilities in terms of the Brier score and AUC.

  1. [Barlow et al., 1972] R.E. Barlow, D.J. Bartholomew, J.M. Bremner, and H.D. Brunk. Statistical Inference Under Or- der Restrictions. Wiley, New York, 1972.
  2. [Belkin et al., 2006] M. Belkin, P. Niyogi, and V. Sind- hwani. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of Machine Learning Research, 7:2399-2434, 2006.
  3. [Boyd, 2010] S. Boyd. Distributed optimization and statisti- cal learning via the alternating direction method of mul- tipliers. Foundations and Trends in Machine Learning, 3(1):1-122, 2010.
  4. [Brier, 1950] G.W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78:1-3, 1950.
  5. [Caruana and Niculescu-Mizil, 2006] R. Caruana and A. Niculescu-Mizil. An empirical evaluation of super- vised learning algorithms. In Proceedings of the 23rd International Conference on Machine Learning, pages 161-168, Pittsburgh, PA, USA, June 2006.
  6. [Caruana et al., 2008] R. Caruana, N. Karampatziakis, and A. Yessenalina. An empirical evaluation of supervised learning in high dimensions. In Proceedings of the 25th International Conference on Machine Learning, pages 96- 103, Helsinki, Finland, July 2008.
  7. [Cormen et al., 2009] T.H. Cormen, C.E. Leiserson, R.L. Rivest, and C. Stein. Introduction to Algorithms. MIT Press, 3rd edition, 2009.
  8. [Deng and Yin, 2012] W. Deng and W. Yin. On the global and linear convergence of the generalized alternating di- rection method of multipliers. Technical Report TR12-14, Rice University, 2012.
  9. [Fan et al., 2008] R.E. Fan, K.W. Chang, C.J. Hsieh, X.R. Wang, and C.J. Lin. LIBLINEAR: A library for large lin- ear classification. Journal of Machine Learning Research, 9:1871-1874, 2008.
  10. [Fawcett and Niculescu-Mizil, 2007] T. Fawcett and A. Niculescu-Mizil. PAV and the ROC convex hull. Machine Learning, 68(1):97-106, July 2007.
  11. T. Fawcett. An introduction to ROC analy- sis. Pattern Recognition Letters, 27:861-874, 2006.
  12. [Friedman and Tibshirani, 1984] J. Friedman and R. Tibshi- rani. The monotone smoothing of scatterplots. Technomet- rics, 26(3):243-250, 1984.
  13. [Gail et al., 1989] M.H. Gail, L.A. Brinton, D.P. Byar, D.K. Corle, S.B. Green, C. Schairer, and J.J. Mulvihill. Project- ing individualized probabilities of developing breast can- cer for white females who are being examined annually. Journal of the National Cancer Institute, 81(24):1879- 1886, 1989.
  14. [Glowinski and Marrocco, 1975] R. Glowinski and A. Mar- rocco. Sur l'approximation, par elements finis d'ordre un, et la resolution, par penalisation-dualite, d'une classe de problems de dirichlet non lineares. Revue Francaise d'Automatique, Informatique, et Recherche Opérationelle, 9:41-76, 1975.
  15. [Hall and Huang, 2001] P. Hall and L.S. Huang. Nonpara- metric kernel regression subject to monotonicity con- straints. Annals of Statistics, 29(3):624-647, 2001.
  16. [Jiang et al., 2011] X. Jiang, M. Osl, J. Kim, and L. Ohno- Machado. Smooth isotonic regression: A new method to calibrate predictive models. In Proceedings of the AMIA Summits on Translational Science, pages 16-20, San Fran- cisco, CA, USA, 2011.
  17. [Menon et al., 2012] A. Menon, X. Jiang, S. Vembu, C. Elkan, and L. Ohno-Machado. Predicting accurate probabilities with a ranking loss. In Proceedings of the 29th International Conference on Machine Learning, pages 703-710, Edinburgh, Scotland, UK, June 2012.
  18. [Niculescu-Mizil and Caruana, 2005] A. Niculescu-Mizil and R. Caruana. Predicting good probabilities with super- vised learning. In Proceedings of the 22nd International Conference on Machine Learning, pages 625-632, Bonn, Germany, August 2005.
  19. [Pardalos and Xue, 1999] P.M. Pardalos and G. Xue. Algo- rithms for a class of isotonic regression problems. Algo- rithmica, 23(3):211-222, 1999.
  20. [Platt, 1999] J. Platt. Probabilistic outputs for support vec- tor machines and comparisons to regularized likelihood methods. In A.J. Smola, P. Bartlett, B. Schölkopf, and D. Schuurmans, editors, Advances in Large Margin Clas- sifiers. MIT, 1999.
  21. [Richardson et al., 2007] M. Richardson, E. Dominowska, and R. Ragno. Predicting clicks: Estimating the click- through rate for new ads. In Proceedings of the 16th Inter- national Conference on World Wide Web, pages 521-529, New York, NY, USA, 2007.
  22. [Stout, 2013] Q.F. Stout. Isotonic regression via partitioning. Algorithmica, 66(1):93-112, May 2013.
  23. [Tumer and Ghosh, 1996] K. Tumer and J. Ghosh. Analysis of decision boundaries in linearly combined neural classi- fiers. Pattern Recognition, 29(2):341-348, 1996.
  24. [Wang and Li, 2008] X. Wang and F. Li. Isotonic smoothing spline regression. Journal of Computational and Graphi- cal Statistics, 17(1):21-37, 2008.
  25. [Zadrozny and Elkan, 2001] B. Zadrozny and C. Elkan. Learning and making decisions when costs and probabil- ities are both unknown. In Proceedings of the 7th Inter- national Conference on Knowledge Discovery and Data Mining, pages 204-213, New York, NY, USA, 2001.
  26. [Zadrozny and Elkan, 2002] B. Zadrozny and C. Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the International Conference on Knowledge Discovery and Data Mining, pages 694-699, Edmonton, Alberta, Canada, 2002.
  27. [Zhou, 2012] Z.-H. Zhou. Ensemble Methods: Foundations and Algorithms. Chapman & Hall, 2012.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards