{"type":"ScholarlyArticle","title":"Quantifying Health Inequalities Induced by Data and AI Models","authors":[{"name":"Honghan Wu","slug":"honghan-wu","given":"Honghan","family":"Wu","affiliation":"Institute of Health Informatics , University College London , London , United Kingdom"},{"name":"Aneeta Sylolypavan","slug":"aneeta-sylolypavan","given":"Aneeta","family":"Sylolypavan","affiliation":"Institute of Health Informatics , University College London , London , United Kingdom"},{"name":"Minhong Wang","slug":"minhong-wang","given":"Minhong","family":"Wang","affiliation":"Institute of Health Informatics , University College London , London , United Kingdom"},{"name":"Sarah Wild","slug":"sarah-wild","given":"Sarah","family":"Wild","affiliation":"Usher Institute , University of Edinburgh , Edinburgh , United Kingdom"}],"abstract":"AI technologies are being increasingly tested and applied in critical environments including healthcare. Without an effective way to detect and mitigate AI induced inequalities, AI might do more harm than good, potentially leading to the widening of underlying inequalities. This paper proposes a generic allocation-deterioration framework for detecting and quantifying AI induced inequality. Specifically, AI induced inequalities are quantified as the area between two allocation-deterioration curves. To assess the framework’s performance, experiments were conducted on ten synthetic datasets (N>33,000) generated from HiRID - a real-world Intensive Care Unit (ICU) dataset, showing its ability to accurately detect and quantify inequality proportionally to controlled inequalities. Extensive analyses were carried out to quantify health inequalities (a) embedded in two real-world ICU datasets; (b) induced by AI models trained for two resource allocation scenarios. Results showed that compared to men, women had up to 33% poorer deterioration in markers of prognosis when admitted to HiRID ICUs. All four AI models assessed were shown to induce significant inequalities (2.45% to 43.2%) for non-White compared to White patients. The models exacerbated data embedded inequalities significantly in 3 out of 8 assessments, one of which was >9 times worse.","doi":"10.24963/ijcai.2022/721","url":"https://ijcai.org/proceedings/2022/721","pdf":"https://ijcai.org/proceedings/2022/0721.pdf","markdown":"https://ijcai.org/proceedings/2022/721.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/721","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Research Papers","track":"AI for Good","number":721,"pages":{"first":5192,"last":5198},"keywords":["AI Ethics, Trust, Fairness: Fairness & Diversity","AI Ethics, Trust, Fairness: Societal Impact of AI","Multidisciplinary Topics and Applications: Health and Medicine","AI Ethics, Trust, Fairness: Bias"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987135","length_sec":82,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985278","length_sec":717,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bailey et al., 2017] Zinzi D Bailey, Nancy Krieger, Madina Agénor, Jasmine Graves, Natalia Linos, and Mary T Bas- sett. Structural racism and health inequities in the USA: evidence and interventions. Lancet, 389(10077):1453- 1463, April 2017.","title":"Structural racism and health inequities in the USA: evidence and interventions","authors":"D Zinzi, Nancy Bailey, Madina Krieger, Jasmine Agénor, Natalia Graves, Mary T Linos, Bassett","year":"2017","venue":"Lancet","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Colbrook et al., 2020] Matthew J Colbrook, Zdravko I Botev, Karsten Kuritz, and Shev MacNamara. Kernel den- sity estimation with linked boundary conditions. Studies in Applied Mathematics, 145(3):357-396, 2020.","title":"Kernel density estimation with linked boundary conditions","authors":"Matthew J Colbrook, I Zdravko, Karsten Botev, Shev Kuritz, Macnamara","year":"2020","venue":"Studies in Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[de Groot et al., 2003] Vincent de Groot, Heleen Becker- man, Gustaaf J Lankhorst, and Lex M Bouter. How to measure comorbidity. a critical review of available meth- ods. J. Clin. Epidemiol., 56(3):221-229, March 2003.","title":"How to measure comorbidity. a critical review of available methods","authors":"De Groot","year":"2003","venue":"J. Clin. Epidemiol","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dinh et al., 2019] An Dinh, Stacey Miertschin, Amber Young, and Somya D Mohanty. A data-driven approach to predicting diabetes and cardiovascular disease with ma- chine learning. BMC Med. Inform. Decis. Mak., 19(1):211, November 2019.","title":"Amber Young, and Somya D Mohanty. A data-driven approach to predicting diabetes and cardiovascular disease with machine learning","authors":"Dinh","year":"2019","venue":"BMC Med. Inform. Decis. Mak","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Faltys et al., 2021] Martin Faltys, Marc Zimmermann, Xin- rui Lyu, Matthias Hüser, Stephanie Hyland, Gunnar Rätsch, and Tobias Merz. Hirid, a high time-resolution icu dataset (version 1.1.1). https://physionet.org/content/ hirid/1.1.1/, 2021. Accessed: 2022-06-07.","title":"Hirid, a high time-resolution icu dataset","authors":"Martin Faltys, Marc Zimmermann, Xinrui Lyu, Matthias Hüser, Stephanie Hyland, Gunnar Rätsch, Tobias Merz","year":"2022","venue":null,"doi":null,"url":"https://physionet.org/content/hirid/1.1.1/,2021","internal_url":null},{"position":6,"raw":"[Geenens, 2014] Gery Geenens. Probit transformation for kernel density estimation on the unit interval. Journal of the American Statistical Association, 109(505):346-358, 2014.","title":"Probit transformation for kernel density estimation on the unit interval","authors":"Gery Geenens","year":"2014","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Johnson et al., 2016] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo An- thony Celi, and Roger G Mark. Mimic-iii, a freely ac- cessible critical care database. Scientific data, 3(1):1-9, 2016.","title":"Mimic-iii, a freely accessible critical care database","authors":"E W Alistair, Tom J Johnson, Lu Pollard, Li-Wei H Shen, Mengling Lehman, Mohammad Feng, Benjamin Ghassemi, Peter Moody, Leo Szolovits, Roger G Anthony Celi, Mark","year":"2016","venue":"Scientific data","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kunde et al., 2005] Sachin S Kunde, Audrey J Lazenby, Ronald H Clements, and Gary A Abrams. Spectrum of nafld and diagnostic implications of the proposed new nor- mal range for serum alt in obese women. Hepatology, 42(3):650-656, 2005.","title":"Spectrum of nafld and diagnostic implications of the proposed new normal range for serum alt in obese women","authors":"Audrey J Sachin S Kunde, Ronald H Lazenby, Gary A Clements, Abrams","year":"2005","venue":"Hepatology","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Leslie et al., 2021] David Leslie, Anjali Mazumder, Aidan Peppin, Maria K Wolters, and Alexa Hagerty. Does \"ai\" stand for augmenting inequality in the era of covid-19 healthcare? bmj, 372, 2021.","title":"Does \"ai\" stand for augmenting inequality in the era of covid-19 healthcare?","authors":"David Leslie, Anjali Mazumder, Aidan Peppin, Maria K Wolters, Alexa Hagerty","year":"2021","venue":"bmj","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Muschelli, 2020] John Muschelli. Roc and auc with a bi- nary predictor: a potentially misleading metric. Journal of classification, 37(3):696-708, 2020.","title":"Roc and auc with a binary predictor: a potentially misleading metric","authors":"John Muschelli","year":"2020","venue":"Journal of classification","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Nelson, 2002] Alan Nelson. Unequal treatment: con- fronting racial and ethnic disparities in health care. J. Natl. Med. Assoc., 94(8):666-668, August 2002.","title":"Unequal treatment: confronting racial and ethnic disparities in health care","authors":"Alan Nelson","year":"2002","venue":"J. Natl. Med. Assoc","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Obermeyer et al., 2019] Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464):447-453, October 2019.","title":"Dissecting racial bias in an algorithm used to manage the health of populations","authors":"Ziad Obermeyer, Brian Powers, Christine Vogeli, Sendhil Mullainathan","year":"2019","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Passi and Barocas, 2019] Samir Passi and Solon Barocas. Problem formulation and fairness. In Proceedings of the Conference on Fairness, Accountability, and Trans- parency, FAT* '19, pages 39-48, New York, NY, USA, January 2019. Association for Computing Machinery.","title":"Problem formulation and fairness","authors":"Barocas Passi, Samir Passi, Solon Barocas","year":"2019","venue":"Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* '19","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rajkomar et al., 2018] Alvin Rajkomar, Michaela Hardt, Michael D Howell, Greg Corrado, and Marshall H Chin. Ensuring fairness in machine learning to advance health equity. Ann. Intern. Med., 169(12):866-872, December 2018.","title":"Ensuring fairness in machine learning to advance health equity","authors":"Alvin Rajkomar, Michaela Hardt, D Michael, Greg Howell, Marshall H Corrado, Chin","year":"2018","venue":"Ann. Intern. Med","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[St Sauver et al., 2021] Jennifer L St Sauver, Alanna M Chamberlain, William V Bobo, Cynthia M Boyd, Lila J Finney Rutten, Debra J Jacobson, Michaela E McGree, Brandon R Grossardt, and Walter A Rocca. Implementing the us department of health and human services definition of multimorbidity: a comparison between billing codes and medical record review in a population-based sample of persons 40-84 years old. BMJ open, 11(4):e042870, 2021.","title":"Implementing the us department of health and human services definition of multimorbidity: a comparison between billing codes and medical record review in a population-based sample of persons 40-84 years old","authors":"Jennifer L St Sauver, Alanna M Chamberlain, William V Bobo, Cynthia M Boyd, Lila J Finney Rutten, Debra J Jacobson, Michaela E Mcgree, Brandon R Grossardt, Walter A Rocca","year":"2021","venue":"BMJ open","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Topol, 2019] Eric J Topol. High-performance medicine: the convergence of human and artificial intelligence. Nat. Med., 25(1):44-56, January 2019.","title":"High-performance medicine: the convergence of human and artificial intelligence","authors":"Eric J Topol","year":"2019","venue":"Nat. Med","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[van Ryn and Burke, 2000] Michelle van Ryn and Jane Burke. The effect of patient race and socio-economic sta- tus on physicians' perceptions of patients. Soc. Sci. Med., 50(6):813-828, March 2000.","title":"The effect of patient race and socio-economic status on physicians' perceptions of patients","authors":"Burke ; Michelle Van Ryn, Jane Van Ryn, Burke","year":"2000","venue":"Soc. Sci. Med","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vasan, 2006] Ramachandran S Vasan. Biomarkers of car- diovascular disease: molecular basis and practical consid- erations. Circulation, 113(19):2335-2362, May 2006.","title":"Biomarkers of cardiovascular disease: molecular basis and practical considerations","authors":"Ramachandran S Vasan","year":"2006","venue":"Circulation","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vyas et al., 2020] Darshali A Vyas, Leo G Eisenstein, and David S Jones. Hidden in plain sight -reconsidering the use of race correction in clinical algorithms. New England Journal of Medicine, 383(9):874-882, 2020.","title":"Hidden in plain sight -reconsidering the use of race correction in clinical algorithms","authors":"Vyas","year":"2020","venue":"New England Journal of Medicine","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}