{"type":"ScholarlyArticle","title":"On the Utility of Prediction Sets in Human-AI Teams","authors":[{"name":"Varun Babbar","slug":"varun-babbar","given":"Varun","family":"Babbar","affiliation":"University of Cambridge"},{"name":"Umang Bhatt","slug":"umang-bhatt","given":"Umang","family":"Bhatt","affiliation":"University of Cambridge; The Alan Turing Institute"},{"name":"Adrian Weller","slug":"adrian-weller","given":"Adrian","family":"Weller","affiliation":"University of Cambridge; The Alan Turing Institute"}],"abstract":"Research on human-AI teams usually provides experts with a single label, which ignores the uncertainty in a model's recommendation. Conformal prediction (CP) is a well established line of research that focuses on building a theoretically grounded, calibrated prediction set, which may contain multiple labels. We explore how such prediction sets impact expert decision-making in human-AI teams.  Our evaluation on human subjects finds that set valued predictions positively impact experts. However, we notice that the predictive sets provided by CP can be very large, which leads to unhelpful AI assistants. To mitigate this, we introduce D-CP, a method to perform CP on some examples and defer to experts. We prove that D-CP can reduce the prediction set size of non-deferred examples. We show how D-CP performs in quantitative and in human subject experiments (n=120). Our results suggest that CP prediction sets improve human-AI team performance over showing the top-1 prediction alone, and that experts find D-CP prediction sets are more useful than CP prediction sets.","doi":"10.24963/ijcai.2022/341","url":"https://ijcai.org/proceedings/2022/341","pdf":"https://ijcai.org/proceedings/2022/0341.pdf","markdown":"https://ijcai.org/proceedings/2022/341.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/341","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Humans and AI","track":"Main Track","number":341,"pages":{"first":2457,"last":2463},"keywords":["Machine Learning: Probabilistic Machine Learning","AI Ethics, Trust, Fairness: Trustworthy AI","Uncertainty in AI: Uncertainty Representations","Humans and AI: Human-AI Collaboration"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987002","length_sec":92,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985145","length_sec":731,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Angelopoulos and Bates, 2021] Anastasios N Angelopoulos and Stephen Bates. A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantifica- tion. arXiv preprint arXiv:2107.07511, 2021.","title":"A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification","authors":"N Anastasios, Stephen Angelopoulos, Bates","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Angelopoulos et al., 2020] Anastasios Nikolas Angelopou- los, Stephen Bates, Michael Jordan, and Jitendra Malik. Uncertainty Sets for Image Classifiers Using Conformal Prediction. In International Conference on Learning Rep- resentations, 2020.","title":"Uncertainty Sets for Image Classifiers Using Conformal Prediction","authors":"Anastasios Nikolas, Angelopoulos, Stephen Bates, Michael Jordan, Jitendra Malik","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bansal et al., 2019] Gagan Bansal, Besmira Nushi, Ece Ka- mar, Daniel S Weld, Walter S Lasecki, and Eric Horvitz. Up- dates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2429-2437, 2019.","title":"Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff","authors":"Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S Daniel S Weld, Eric Lasecki, Horvitz","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bartlett and Wegkamp, 2008] Peter L Bartlett and Marten H Wegkamp. Classification with a reject option using a hinge loss. Journal of Machine Learning Research, 9(8), 2008.","title":"Classification with a reject option using a hinge loss","authors":"L Peter, Marten H Bartlett, Wegkamp","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bellotti, 2021] Anthony Bellotti. Optimized conformal clas- sification using gradient descent approximation. CoRR, abs/2105.11255, 2021.","title":"Optimized conformal classification using gradient descent approximation","authors":"Anthony Bellotti","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bondi et al., 2022] Elizabeth Bondi, Raphael Koster, Han- nah Sheahan, Martin Chadwick, Yoram Bachrach, Tay- lan Cemgil, Ulrich Paquet, and Krishnamurthy Dvijotham. Role of Human-AI Interaction in Selective Prediction. Pro- ceedings of the AAAI Conference on Artificial Intelligence, 2022.","title":"Role of Human-AI Interaction in Selective Prediction","authors":"Elizabeth Bondi, Raphael Koster, Hannah Sheahan, Martin Chadwick, Yoram Bachrach, Taylan Cemgil, Ulrich Paquet, Krishnamurthy Dvijotham","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cortes et al., 2016] Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with Rejection. In Ronald Ortner, Hans Ulrich Simon, and Sandra Zilles, editors, Algorith- mic Learning Theory, pages 67-82, Cham, 2016. Springer International Publishing.","title":"Learning with Rejection","authors":"Corinna Cortes, Giulia Desalvo, Mehryar Mohri","year":"2016","venue":"Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lei et al., 2016] Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. Distribution- Free Predictive Inference For Regression. Journal of the American Statistical Association, 113:1094-1111, 4 2016.","title":"Distribution-Free Predictive Inference For Regression","authors":"Jing Lei, G' Max, Alessandro Sell, Ryan J Rinaldo, Larry Tibshirani, Wasserman","year":null,"venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lundberg et al., 2018] Scott M. Lundberg, Bala Nair, Mon- ica S. Vavilala, Mayumi Horibe, Michael J. Eisses, Trevor Adams, David E. Liston, Daniel King-Wai Low, Shu- Fang Newman, Jerry Kim, and Su-In Lee. Explainable Machine-Learning Predictions for the Prevention of Hy- poxaemia During Surgery. Nature Biomedical Engineering, 2(10):749-760, October 2018.","title":"Explainable Machine-Learning Predictions for the Prevention of Hypoxaemia During Surgery","authors":"M Scott, Bala Lundberg, Monica S Nair, Mayumi Vavilala, Michael J Horibe, Trevor Eisses, David E Adams, Daniel Liston, King-Wai Low, Shu-Fang Newman, Jerry Kim, Su-In Lee","year":"2018","venue":"Nature Biomedical Engineering","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Madras et al., 2018] David Madras, Toniann Pitassi, and Richard Zemel. Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 6150-6160, 2018.","title":"Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer","authors":"David Madras, Toniann Pitassi, Richard Zemel","year":"2018","venue":"Proceedings of the 32nd International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mozannar and Sontag, 2020] Hussein Mozannar and David Sontag. Consistent Estimators for Learning to Defer to an Expert. In International Conference on Machine Learning, pages 7076-7087. PMLR, 2020.","title":"Consistent Estimators for Learning to Defer to an Expert","authors":"Sontag Mozannar, Hussein Mozannar, David Sontag","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Okati et al., 2021] Nastaran Okati, Abir De, and Manuel Gomez-Rodriguez. Differentiable Learning Under Triage. In Advances in Neural Information Processing Systems, 2021.","title":"Differentiable Learning Under Triage","authors":"Nastaran Okati, Abir De, Manuel Gomez-Rodriguez","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Papadopoulos, 2008] Harris Papadopoulos. Inductive Con- formal Prediction: Theory and Application to Neural Net- works. In Paula Fritzsche, editor, Tools in Artificial Intelli- gence, chapter 18. IntechOpen, Rijeka, 2008.","title":"Inductive Conformal Prediction: Theory and Application to Neural Networks","authors":"Harris Papadopoulos","year":"2008","venue":"Tools in Artificial Intelligence, chapter 18","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Peterson et al., 2019] Joshua Peterson, Ruairidh Battleday, Thomas Griffiths, and Olga Russakovsky. Human Un- certainty Makes Classification More Robust. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9616-9625, 2019.","title":"Human Uncertainty Makes Classification More Robust","authors":"Peterson","year":"2019","venue":"IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Romano et al., 2020] Yaniv Romano, Matteo Sesia, and Em- manuel Candes. Classification with Valid and Adaptive Coverage. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Informa- tion Processing Systems, volume 33, pages 3581-3591.","title":"Classification with Valid and Adaptive Coverage","authors":"Yaniv Romano, Matteo Sesia, Emmanuel Candes","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"Curran Associates, Inc., 2020.","title":null,"authors":"Curran Associates, Inc","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sadinle et al., 2016] Mauricio Sadinle, Jing Lei, and Larry Wasserman. Least Ambiguous Set-Valued Classifiers with Bounded Error Levels. Journal of the American Statistical Association, 114:223-234, 9 2016.","title":"Least Ambiguous Set-Valued Classifiers with Bounded Error Levels","authors":"Mauricio Sadinle, Jing Lei, Larry Wasserman","year":"2016","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Straitouri et al., 2022] Eleni Straitouri, Lequn Wang, Nas- taran Okati, and Manuel Gomez Rodriguez. Provably improving expert predictions with conformal prediction. CoRR, abs/2201.12006, 2022.","title":"Provably improving expert predictions with conformal prediction","authors":"Eleni Straitouri, Lequn Wang, Nastaran Okati, Manuel Gomez Rodriguez","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Stutz et al., 2022] David Stutz, Krishnamurthy Dj Dvi- jotham, Ali Taylan Cemgil, and Arnaud Doucet. Learning optimal conformal classifiers. In ICLR, 2022.","title":"Learning optimal conformal classifiers","authors":"David Stutz, Dj Krishnamurthy, Ali Dvijotham, Arnaud Taylan Cemgil, Doucet","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tibshirani et al., 2019] Ryan J. Tibshirani, Rina Foygel Bar- ber, Emmanuel J. Candes, and Aaditya Ramdas. Conformal Prediction Under Covariate Shift. Advances in Neural In- formation Processing Systems, 32, 4 2019.","title":"Conformal Prediction Under Covariate Shift","authors":"Ryan J Tibshirani, Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vovk et al., 2005] Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, 01 2005.","title":"Algorithmic Learning in a Random World","authors":"Vladimir Vovk, Alex Gammerman, Glenn Shafer","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wilder et al., 2021] Bryan Wilder, Eric Horvitz, and Ece Ka- mar. Learning to Complement Humans. In Proceedings of the Twenty-Ninth International Joint Conference on Artifi- cial Intelligence, IJCAI'20, 2021.","title":"Learning to Complement Humans","authors":"Bryan Wilder, Eric Horvitz, Ece Kamar","year":"2021","venue":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI'20","doi":null,"url":null,"internal_url":"/proceedings/2020/212"},{"position":23,"raw":"[Zagoruyko and Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith, editors, Proceedings of the British Machine Vision Confer- ence (BMVC), pages 87.1-87.12. BMVA Press, September 2016.","title":"Wide Residual Networks","authors":"Sergey Zagoruyko, Nikos Komodakis","year":"2016","venue":"Proceedings of the British Machine Vision Conference (BMVC)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}