{"type":"ScholarlyArticle","title":"Reliable Multi-class Classification based on Pairwise Epistemic and Aleatoric Uncertainty","authors":[{"name":"Vu-Linh Nguyen","slug":"vu-linh-nguyen","given":"Vu-Linh","family":"Nguyen","affiliation":"UMR CNRS 7253 Heudiasyc , Sorbonne universités ,"},{"name":"Sébastien Destercke","slug":"sebastien-destercke","given":"Sébastien","family":"Destercke","affiliation":"UMR CNRS 7253 Heudiasyc , Sorbonne universités ,"},{"name":"Marie-Hélène Masson","slug":"marie-helene-masson","given":"Marie-Hélène","family":"Masson","affiliation":"UMR CNRS 7253 Heudiasyc , Sorbonne universités ,; Université de Picardie Jules Verne , France; Université de technologie de Compiègne CS 60319 -60203 Compiègne cedex , France"},{"name":"Eyke Hüllermeier","slug":"eyke-hullermeier","given":"Eyke","family":"Hüllermeier","affiliation":"Department of Computer Science , Paderborn University , 33098 Paderborn , Germany"}],"abstract":"We propose a method for reliable prediction in multi-class classification, where reliability refers to the possibility of partial abstention in cases of uncertainty. More specifically, we allow for predictions in the form of preorder relations on the set of classes, thereby generalizing the idea of set-valued predictions. Our approach relies on combining learning by pairwise comparison with a recent proposal for modeling uncertainty in classification, in which a distinction is made between reducible (a.k.a. epistemic) uncertainty caused by a lack of information and irreducible (a.k.a. aleatoric) uncertainty due to intrinsic randomness. The problem of combining uncertain pairwise predictions into a most plausible preorder is then formalized as an integer programming problem. Experimentally, we show that our method is able to appropriately balance reliability and precision of predictions.","doi":"10.24963/ijcai.2018/706","url":"https://ijcai.org/proceedings/2018/706","pdf":"https://ijcai.org/proceedings/2018/0706.pdf","markdown":"https://ijcai.org/proceedings/2018/706.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/706","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Uncertainty in AI","track":"Main track","number":706,"pages":{"first":5089,"last":5095},"keywords":["Uncertainty in AI: Uncertainty in AI","Machine Learning: Classification","Machine Learning: Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Balasubramanian et al., 2014] Vineeth Balasubramanian, Shen-Shyang Ho, and Vladimir Vovk. Conformal predic- tion for reliable machine learning: Theory, Adaptations and applications. Newnes, 2014.","title":"Conformal prediction for reliable machine learning: Theory, Adaptations and applications","authors":"Shen-Shyang Vineeth Balasubramanian, Vladimir Ho, Vovk","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Birnbaum, 1962] Allan Birnbaum. On the foundations of statistical inference. Journal of the American Statistical Association, 57(298):269-306, 1962.","title":"On the foundations of statistical inference","authors":"Allan Birnbaum","year":"1962","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bishop, 2006] Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.","title":"Pattern recognition and machine learning","authors":"M Christopher, Bishop","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cheng and Hüllermeier, 2012] Weiwei Cheng and Eyke Hüllermeier. Probability estimation for multi-class classi- fication based on label ranking. In Proceedings of the 2012 European conference on Machine Learning and Knowl- edge Discovery in Databases, pages 83-98. Springer- Verlag, 2012.","title":"Probability estimation for multi-class classification based on label ranking","authors":"Hüllermeier Cheng, Weiwei Cheng, Eyke Hüllermeier","year":"2012","venue":"Proceedings of the 2012 European conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Corani et al., 2014] Giorgio Corani, Joaquín Abellán, Andrés Masegosa, Serafin Moral, and Marco Zaffalon. Classification, pages 230-257.","title":"Classification","authors":"Giorgio Corani, Joaquín Abellán, Andrés Masegosa, Serafin Moral, Marco Zaffalon","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Coz et al., 2009] Juan José del Coz, Jorge Díez, and An- tonio Bahamonde. Learning nondeterministic classifiers. Journal of Machine Learning Research, 10(Oct):2273- 2293, 2009.","title":"Juan José del Coz, Jorge Díez, and Antonio Bahamonde. Learning nondeterministic classifiers","authors":"Coz","year":"2009","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Czepiel, 2002] Scott A Czepiel. Maximum likelihood es- timation of logistic regression models: theory and imple- mentation. http://czep.net, 2002.","title":"Maximum likelihood estimation of logistic regression models: theory and implementation","authors":"A Scott, Czepiel","year":"2002","venue":null,"doi":null,"url":"http://czep.net","internal_url":null},{"position":8,"raw":"[Demšar, 2006] Janez Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7(Jan):1-30, 2006.","title":"Janez Demšar. Statistical comparisons of classifiers over multiple data sets","authors":"Demšar","year":"2006","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Denoeux, 2014] Thierry Denoeux. Likelihood-based belief function: justification and some extensions to low-quality data. International Journal of Approximate Reasoning, 55(7):1535-1547, 2014.","title":"Likelihood-based belief function: justification and some extensions to low-quality data","authors":"Thierry Denoeux","year":"2014","venue":"International Journal of Approximate Reasoning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fürnkranz, 2002] Johannes Fürnkranz. Round robin clas- sification. Journal of Machine Learning Research, 2(Mar):721-747, 2002.","title":"Johannes Fürnkranz. Round robin classification","authors":"Fürnkranz","year":"2002","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hüllermeier and Brinker, 2007] Eyke Hüllermeier and Klaus Brinker. Fuzzy-relational classification: Combining pairwise decomposition techniques with fuzzy preference modeling. In Proceedings of the 5th Conference of the European Society for Fuzzy Logic and Technology, pages 353-360, 2007.","title":"Fuzzy-relational classification: Combining pairwise decomposition techniques with fuzzy preference modeling","authors":"Brinker Hüllermeier, Eyke Hüllermeier, Klaus Brinker","year":"2007","venue":"Proceedings of the 5th Conference of the European Society for Fuzzy Logic and Technology","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hüllermeier and Vanderlooy, 2010] Eyke Hüllermeier and Stijn Vanderlooy. Combining predictions in pairwise clas- sification: An optimal adaptive voting strategy and its rela- tion to weighted voting. Pattern Recognition, 43(1):128- 142, 2010.","title":"Eyke Hüllermeier and Stijn Vanderlooy. Combining predictions in pairwise classification: An optimal adaptive voting strategy and its relation to weighted voting","authors":"Vanderlooy Hüllermeier","year":"2010","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kull and Flach, 2014] Meelis Kull and Peter Flach. Re- liability maps: a tool to enhance probability estimates and improve classification accuracy. In Proceedings of the 2014 European Conference on Machine Learning and Knowledge Discovery in Databases, pages 18-33. Springer-Verlag, 2014.","title":"Reliability maps: a tool to enhance probability estimates and improve classification accuracy","authors":"Flach Kull, Meelis Kull, Peter Flach","year":"2014","venue":"Proceedings of the 2014 European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Masson et al., 2016] Marie-Hélène Masson, Sébastien Destercke, and Thierry Denoeux. Modelling and predict- ing partial orders from pairwise belief functions. Soft Computing, 20(3):939-950, 2016.","title":"Modelling and predicting partial orders from pairwise belief functions","authors":"Marie-Hélène Masson, Sébastien Destercke, Thierry Denoeux","year":"2016","venue":"Soft Computing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nocedal and Wright, 2006] Jorge Nocedal and S Wright. Numerical Optimization. Springer New York, 2006.","title":"Numerical Optimization","authors":"Jorge Nocedal, Wright","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Philip and Elizabeth, 2010] E Philip and WONG Elizabeth. Sequential quadratic programming methods. UCSD De- partment of Mathematics Technical Report NA-10-03, 2010.","title":"Sequential quadratic programming methods","authors":"E Philip, Elizabeth","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rennie, 2005] Jason DM Rennie. Regularized logistic re- gression is strictly convex. Technical report, MIT, 2005.","title":"Regularized logistic regression is strictly convex","authors":"Jason Dm Rennie","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Senge et al., 2014] Robin Senge, Stefan Bösner, Krzysztof Dembczyński, Jörg Haasenritter, Oliver Hirsch, Norbert Donner-Banzhoff, and Eyke Hüllermeier. Reliable classi- fication: Learning classifiers that distinguish aleatoric and epistemic uncertainty. Information Sciences, 255:16-29, 2014.","title":"Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty","authors":"Senge","year":"2014","venue":"Information Sciences","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shafer and Vovk, 2008] Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(Mar):371-421, 2008.","title":"A tutorial on conformal prediction","authors":"Glenn Shafer, Vladimir Vovk","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Walley and Moral, 1999] Peter Walley and Serafin Moral. Upper probabilities based only on the likelihood function. Journal of the Royal Statistical Society: Series B (Statisti- cal Methodology), 61(4):831-847, 1999.","title":"Upper probabilities based only on the likelihood function","authors":"Moral Walley","year":"1999","venue":"Journal of the Royal Statistical Society: Series B (Statistical Methodology","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wu et al., 2004] Ting-Fan Wu, Chih-Jen Lin, and Ruby C Weng. Probability estimates for multi-class classification by pairwise coupling. Journal of Machine Learning Re- search, 5(Aug):975-1005, 2004.","title":"Probability estimates for multi-class classification by pairwise coupling","authors":"Ting-Fan, Chih-Jen Wu, Ruby C Lin, Weng","year":"2004","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu et al., 2016] Philippe Xu, Franck Davoine, Hongbin Zha, and Thierry Denoeux. Evidential calibration of bi- nary svm classifiers. International Journal of Approximate Reasoning, 72:55-70, 2016.","title":"Evidential calibration of binary svm classifiers","authors":"Philippe Xu, Franck Davoine, Hongbin Zha, Thierry Denoeux","year":"2016","venue":"International Journal of Approximate Reasoning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2014] Gen Yang, Sébastien Destercke, and Marie-Hélène Masson. Nested dichotomies with proba- bility sets for multi-class classification. In Proceedings of the Twenty-first European Conference on Artificial Intelli- gence, pages 363-368. IOS Press, 2014.","title":"Nested dichotomies with probability sets for multi-class classification","authors":"Yang","year":"2014","venue":"Proceedings of the Twenty-first European Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang et al., 2017] Gen Yang, Sébastien Destercke, and Marie-Hélène Masson. The costs of indeterminacy: How to determine them? IEEE Transactions on Cybernetics, 47(12):4316-4327, 2017.","title":"The costs of indeterminacy: How to determine them","authors":"Yang","year":"2017","venue":"IEEE Transactions on Cybernetics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zaffalon et al., 2012] Marco Zaffalon, Giorgio Corani, and Denis Mauá. Evaluating credal classifiers by utility- discounted predictive accuracy. International Journal of Approximate Reasoning, 53(8):1282-1301, 2012.","title":"Evaluating credal classifiers by utilitydiscounted predictive accuracy","authors":"Marco Zaffalon, Giorgio Corani, Denis Mauá","year":"2012","venue":"International Journal of Approximate Reasoning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}