{"type":"ScholarlyArticle","title":"Learning When to Advise Human Decision Makers","authors":[{"name":"Gali Noti","slug":"gali-noti","given":"Gali","family":"Noti","affiliation":"Harvard John A. Paulson School of Engineering and Applied Sciences , Harvard University; The School of Computer Science and Engineering , The Hebrew University of Jerusalem"},{"name":"Yiling Chen","slug":"yiling-chen","given":"Yiling","family":"Chen","affiliation":"Harvard John A. Paulson School of Engineering and Applied Sciences , Harvard University"}],"abstract":"Artificial intelligence (AI) systems are increasingly used for providing advice to facilitate human decision making in a wide range of domains, such as healthcare, criminal justice, and finance. Motivated by limitations of the current practice where algorithmic advice is provided to human users as a constant element in the decision-making pipeline, in this paper we raise the question of when should algorithms provide advice? We propose a novel design of AI systems in which the algorithm interacts with the human user in a two-sided manner and aims to provide advice only when it is likely to be beneficial for the user in making their decision. The results of a large-scale experiment show that our advising approach manages to provide advice at times of need and to significantly improve human decision making compared to fixed, non-interactive, advising approaches. This approach has additional advantages in facilitating human learning, preserving complementary strengths of human decision makers, and leading to more positive responsiveness to the advice.","doi":"10.24963/ijcai.2023/339","url":"https://ijcai.org/proceedings/2023/339","pdf":"https://ijcai.org/proceedings/2023/0339.pdf","markdown":"https://ijcai.org/proceedings/2023/339.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/339","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Humans and AI","track":"Main Track","number":339,"pages":{"first":3038,"last":3048},"keywords":["Humans and AI: HAI: Applications","Humans and AI: HAI: Human-AI collaboration"],"videos":[],"references":[{"position":1,"raw":"[Albright, 2019] Alex Albright. If you give a judge a risk score: evidence from kentucky bail decisions. Harvard John M. Olin Fellow's Discussion Paper, 85:16, 2019.","title":"If you give a judge a risk score: evidence from kentucky bail decisions","authors":"Alex Albright","year":"2019","venue":"Harvard John M. Olin Fellow's Discussion Paper","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anderson et al., 2014] Bonnie Anderson, Anthony Vance, Brock Kirwan, David Eargle, and Seth Howard. Users aren't (necessarily) lazy: Using neurois to explain habit- uation to security warnings. 2014.","title":"Users aren't (necessarily) lazy: Using neurois to explain habituation to security warnings","authors":"Bonnie Anderson, Anthony Vance, Brock Kirwan, David Eargle, Seth Howard","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Angwin et al., 2016] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. ProPublica, 2016.","title":"Machine bias","authors":"Julia Angwin, Jeff Larson, Surya Mattu, Lauren Kirchner","year":"2016","venue":"ProPublica","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bansal et al., 2019] Gagan Bansal, Besmira Nushi, Ece Ka- mar, Walter S Lasecki, Daniel S Weld, and Eric Horvitz. Beyond accuracy: The role of mental models in human-ai team performance. In HCOMP 2019, 2019.","title":"Beyond accuracy: The role of mental models in human-ai team performance","authors":"Gagan Bansal, Besmira Nushi, Ece Kamar, Daniel S Walter S Lasecki, Eric Weld, Horvitz","year":"2019","venue":"HCOMP 2019","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bansal et al., 2021] Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. Does the whole exceed its parts? the effect of ai explanations on comple- mentary team performance. In CHI 2021, 2021.","title":"Does the whole exceed its parts? the effect of ai explanations on complementary team performance","authors":"Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, Daniel Weld","year":"2021","venue":"CHI 2021","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Barberis, 2013] Nicholas C Barberis. Thirty years of prospect theory in economics: A review and assessment. Journal of Economic Perspectives, 2013.","title":"Thirty years of prospect theory in economics: A review and assessment","authors":"C Nicholas, Barberis","year":"2013","venue":"Journal of Economic Perspectives","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bonaccio and Dalal, 2006] Silvia Bonaccio and Reeshad S Dalal. Advice taking and decision-making: An integrative literature review, and implications for the organizational sciences. Organizational behavior and human decision processes, 2006.","title":"Advice taking and decision-making: An integrative literature review, and implications for the organizational sciences","authors":"Dalal Bonaccio, Silvia Bonaccio, Reeshad S Dalal","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Buc ¸inca et al., 2021] Zana Buc ¸inca, Maja Barbara Malaya, and Krzysztof Z Gajos. To trust or to think: cognitive forc- ing functions can reduce overreliance on ai in ai-assisted decision-making. Proceedings of the ACM on Human- Computer Interaction, 2021.","title":"To trust or to think: cognitive forcing functions can reduce overreliance on ai in ai-assisted decision-making","authors":"Zana Buc ¸inca, Maja Barbara Malaya, Krzysztof Z Gajos","year":"2021","venue":"Proceedings of the ACM on Human-Computer Interaction","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chouldechova et al., 2018] Alexandra Chouldechova, Di- ana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In FAT* 2018, 2018.","title":"A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions","authors":"Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, Rhema Vaithianathan","year":"2018","venue":"FAT* 2018","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chouldechova, 2017] Alexandra Chouldechova. Fair pre- diction with disparate impact: A study of bias in recidi- vism prediction instruments. Big data, 2017.","title":"Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments","authors":"Chouldechova","year":"2017","venue":"Big data","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Cohen et al., 2018] Thomas H Cohen, Christopher T Lowenkamp, and William E Hicks. Revalidating the fed- eral pretrial risk assessment instrument (ptra): A research summary. Fed. Probation, 2018.","title":"Revalidating the federal pretrial risk assessment instrument (ptra): A research summary","authors":"Christopher T Thomas H Cohen, William E Lowenkamp, Hicks","year":"2018","venue":"Fed. Probation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Dawes et al., 1989] Robyn M Dawes, David Faust, and Paul E Meehl. Clinical versus actuarial judgment. Sci- ence, 1989.","title":"Clinical versus actuarial judgment","authors":"Robyn M Dawes, David Faust, Paul E Meehl","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Desmarais et al., 2016] Sarah L Desmarais, Kiersten L Johnson, and Jay P Singh. Performance of recidivism risk assessment instruments in us correctional settings. Psy- chological services, 13(3):206, 2016.","title":"Performance of recidivism risk assessment instruments in us correctional settings","authors":"Sarah L Desmarais, Kiersten L Johnson, Jay P Singh","year":"2016","venue":"Psychological services","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[DOJ., 2014] US DOJ. State court processing statistics, 1990-2009: Felony defendants in large urban counties. Inter-university Consortium for Political and Social Re- search, 2014.","title":"US DOJ. State court processing statistics, 1990-2009: Felony defendants in large urban counties. Inter-university Consortium for Political and Social Research","authors":"Doj","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Garcia-Vidal et al., 2019] Carolina Garcia-Vidal, Gemma Sanjuan, Pedro Puerta-Alcalde, Estela Moreno-García, and Alex Soriano. Artificial intelligence to support clin- ical decision-making processes. EBioMedicine, 2019.","title":"Carolina Garcia-Vidal","authors":"Garcia-Vidal","year":"2019","venue":"Artificial intelligence to support clinical decision-making processes","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Gneiting and Raftery, 2007] Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, pre- diction, and estimation. Journal of the American statistical Association, 2007.","title":"Strictly proper scoring rules, prediction, and estimation","authors":"Tilmann Gneiting, Adrian E Raftery","year":"2007","venue":"Journal of the American statistical Association","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Green and Chen, 2019a] Ben Green and Yiling Chen. Dis- parate interactions: An algorithm-in-the-loop analysis of fairness in risk assessments. In FAT* 2019, 2019.","title":"Disparate interactions: An algorithm-in-the-loop analysis of fairness in risk assessments","authors":"Chen ; Green, Ben Green, Yiling Chen","year":"2019","venue":"FAT* 2019","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Green and Chen, 2019b] Ben Green and Yiling Chen. The principles and limits of algorithm-in-the-loop decision making. Proceedings of the ACM on Human-Computer Interaction, 2019.","title":"The principles and limits of algorithm-in-the-loop decision making","authors":"Chen ; Green, Ben Green, Yiling Chen","year":"2019","venue":"Proceedings of the ACM on Human-Computer Interaction","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Green, 2022] Ben Green. The flaws of policies requiring hu- man oversight of government algorithms. Computer Law & Security Review, 45:105681, 2022.","title":"The flaws of policies requiring human oversight of government algorithms","authors":"Ben Green","year":"2022","venue":"Computer Law & Security Review","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Grgić-Hlača et al., 2019] Nina Grgić-Hlača, Christoph En- gel, and Krishna P Gummadi. Human decision making with machine assistance: An experiment on bailing and jailing. Proceedings of the ACM on Human-Computer In- teraction, 2019.","title":"Human decision making with machine assistance: An experiment on bailing and jailing","authors":"Nina Grgić-Hlača, Christoph Engel, Krishna P Gummadi","year":"2019","venue":"Proceedings of the ACM on Human-Computer Interaction","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Groh et al., 2022] Matthew Groh, Ziv Epstein, Chaz Fire- stone, and Rosalind Picard. Deepfake detection by human crowds, machines, and machine-informed crowds. Pro- ceedings of the National Academy of Sciences, 2022.","title":"Deepfake detection by human crowds, machines, and machine-informed crowds","authors":"Matthew Groh, Ziv Epstein, Chaz Firestone, Rosalind Picard","year":"2022","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Grove et al., 2000] William M Grove, David H Zald, Boyd S Lebow, Beth E Snitz, and Chad Nelson. Clinical versus mechanical prediction: a meta-analysis. Psycho- logical assessment, 2000.","title":"Clinical versus mechanical prediction: a meta-analysis","authors":"David H William M Grove, Boyd S Zald, Beth E Lebow, Chad Snitz, Nelson","year":"2000","venue":"Psychological assessment","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Guthrie et al., 2000] Chris Guthrie, Jeffrey J Rachlinski, and Andrew J Wistrich. Inside the judicial mind. Cornell L. Rev., 2000.","title":"Inside the judicial mind","authors":"Chris Guthrie, Jeffrey J Rachlinski, Andrew J Wistrich","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Haenssle et al., 2018] Holger A Haenssle, Christine Fink, R Schneiderbauer, Ferdinand Toberer, Timo Buhl, A Blum, A Kalloo, A Ben Hadj Hassen, Luc Thomas, A Enk, et al. Man against machine: diagnostic perfor- mance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists. Annals of Oncology, 2018.","title":"Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists","authors":"Christine Holger A Haenssle, R Fink, Ferdinand Schneiderbauer, Timo Toberer, Buhl, Blum, A Kalloo, Luc Ben Hadj Hassen, Thomas, Enk","year":"2018","venue":"Annals of Oncology","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Jacobs et al., 2021] Maia Jacobs, Melanie F Pradier, Thomas H McCoy, Roy H Perlis, Finale Doshi-Velez, and Krzysztof Z Gajos. How machine-learning recommenda- tions influence clinician treatment selections: the example of antidepressant selection. Translational psychiatry, 2021.","title":"How machine-learning recommendations influence clinician treatment selections: the example of antidepressant selection","authors":"Maia Jacobs, Melanie F Pradier, Thomas H Mccoy, Roy H Perlis, Finale Doshi-Velez, Krzysztof Z Gajos","year":"2021","venue":"Translational psychiatry","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Kalsher and Williams, 2006] MJ Kalsher and KJ Williams. Behavioral compliance: Theory, methodology, and re- sults.(chap. 23) in ms wogalter (ed.) handbook of warnings (pp. 313-329), 2006.","title":"Behavioral compliance: Theory, methodology, and results","authors":"Williams ; Mj Kalsher, Kalsher, Williams","year":"2006","venue":"handbook of warnings","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Kamar et al., 2012] Ece Kamar, Severin Hacker, and Eric Horvitz. Combining human and machine intelligence in large-scale crowdsourcing. In AAMAS, 2012.","title":"Combining human and machine intelligence in large-scale crowdsourcing","authors":"Ece Kamar, Severin Hacker, Eric Horvitz","year":"2012","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Kleinberg et al., 2015] Jon Kleinberg, Jens Ludwig, Send- hil Mullainathan, and Ziad Obermeyer. Prediction policy problems. American Economic Review, 2015.","title":"Prediction policy problems","authors":"Kleinberg","year":"2015","venue":"American Economic Review","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Kleinberg et al., 2017] Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. Human Decisions and Machine Predic- tions. The Quarterly Journal of Economics, 2017.","title":"Human Decisions and Machine Predictions","authors":"Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, Sendhil Mullainathan","year":"2017","venue":"The Quarterly Journal of Economics","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Kononenko, 2001] Igor Kononenko. Machine learning for medical diagnosis: history, state of the art and perspective. Artificial Intelligence in medicine, 2001.","title":"Machine learning for medical diagnosis: history, state of the art and perspective","authors":"Igor Kononenko, Kononenko","year":"2001","venue":"Artificial Intelligence in medicine","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Kullback, 1997] Solomon Kullback. Information theory and statistics. Courier Corporation, 1997.","title":"Information theory and statistics","authors":"Kullback","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Lai and Tan, 2019] Vivian Lai and Chenhao Tan. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In FAT* 2019, 2019.","title":"On human predictions with explanations and predictions of machine learning models: A case study on deception detection","authors":"Tan ; ] Lai, Lai Vivian, Chenhao Tan","year":"2019","venue":"FAT* 2019","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Lai et al., 2020] Vivian Lai, Han Liu, and Chenhao Tan. \" why is' chicago'deceptive?\" towards building model- driven tutorials for humans. In CHI 2020, 2020.","title":"why is' chicago'deceptive?\" towards building modeldriven tutorials for humans","authors":"Vivian Lai, Han Liu, Chenhao Tan","year":"2020","venue":"CHI 2020","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Lowenkamp, 2009] Christopher T Lowenkamp. The devel- opment of an actuarial risk assessment instrument for us pretrial services. Fed. Probation, 2009.","title":"The development of an actuarial risk assessment instrument for us pretrial services","authors":"Christopher T Lowenkamp","year":"2009","venue":"Fed. Probation","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Madras et al., 2018] David Madras, Toni Pitassi, and Richard Zemel. Predict responsibly: improving fairness and accuracy by learning to defer. NeurIPS 2018, 2018.","title":"Predict responsibly: improving fairness and accuracy by learning to defer","authors":"David Madras, Toni Pitassi, Richard Zemel","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Meehl, 1954] Paul E Meehl. Clinical versus statistical pre- diction: A theoretical analysis and a review of the evi- dence. Minneapolis: University of Minnesota Press, 1954.","title":"Clinical versus statistical prediction: A theoretical analysis and a review of the evidence","authors":"Paul E Meehl","year":"1954","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Mullainathan and Spiess, 2017] Sendhil Mullainathan and Jann Spiess. Machine learning: an applied econometric approach. Journal of Economic Perspectives, 2017.","title":"Machine learning: an applied econometric approach","authors":"Spiess Mullainathan, Sendhil Mullainathan, Jann Spiess","year":"2017","venue":"Journal of Economic Perspectives","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Musen et al., 2021] Mark A Musen, Blackford Middleton, and Robert A Greenes. Clinical decision-support systems. In Biomedical informatics. 2021.","title":"Clinical decision-support systems","authors":"Blackford Mark A Musen, Robert A Middleton, Greenes","year":"2021","venue":"Biomedical informatics","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[NJ-Courts, 2020] NJ-Courts. Annual report to the governor and the legislature. New Jersey Courts, 2020.","title":"NJ-Courts","authors":null,"year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"Inc. Northpointe. Practitioner's guide to compas core. https://www.equivant.com/wp-content/ uploads/Practitioners-Guide-to-COMPAS-Core-040419. pdf, 2019. Accessed: 2023-06-12.","title":"Practitioner's guide to compas core","authors":null,"year":"2019","venue":null,"doi":null,"url":"https://www.equivant.com/wp-content/uploads/Practitioners-Guide-to-COMPAS-Core-040419.pdf","internal_url":null},{"position":41,"raw":"[Noti and Chen, 2023] Gali Noti and Yiling Chen. Learning when to advise human decision makers. https://arxiv.org/abs/2209.13578, 2023.","title":"Learning when to advise human decision makers","authors":"Chen ; Noti, Gali Noti, Yiling Chen","year":"2023","venue":null,"doi":null,"url":"https://arxiv.org/abs/2209.13578","internal_url":null},{"position":42,"raw":"[Obermeyer and Emanuel, 2016] Ziad Obermeyer and Ezekiel J Emanuel. Predicting the future-big data, ma- chine learning, and clinical medicine. The New England journal of medicine, 2016.","title":"Predicting the future-big data, machine learning, and clinical medicine","authors":"Emanuel ; Obermeyer, Ziad Obermeyer, Ezekiel J Emanuel","year":"2016","venue":"The New England journal of medicine","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[PJI, 2019] Pretrial Justice Institute PJI. Scan of pretrial practices. Pretrial Justice Institute, 2019.","title":"Pretrial Justice Institute PJI. Scan of pretrial practices","authors":"Pji","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Rankin et al., 2009] Catharine H Rankin, Thomas Abrams, Robert J Barry, Seema Bhatnagar, David F Clayton, John Colombo, Gianluca Coppola, Mark A Geyer, David L Glanzman, Stephen Marsland, et al. Habituation revisited: an updated and revised description of the behavioral char- acteristics of habituation. Neurobiology of learning and memory, 2009.","title":"Habituation revisited: an updated and revised description of the behavioral characteristics of habituation","authors":"Thomas Catharine H Rankin, Robert J Abrams, Seema Barry, Bhatnagar, John David F Clayton, Gianluca Colombo, Mark A Coppola, David L Geyer, Stephen Glanzman, Marsland","year":"2009","venue":"Neurobiology of learning and memory","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Sniezek and Buckley, 1995] Janet A Sniezek and Timothy Buckley. Cueing and cognitive conflict in judge-advisor decision making. Organizational behavior and human de- cision processes, 1995.","title":"Cueing and cognitive conflict in judge-advisor decision making","authors":"Buckley Sniezek, Janet A Sniezek, Timothy Buckley","year":"1995","venue":"Organizational behavior and human decision processes","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Steyvers et al., 2022] Mark Steyvers, Heliodoro Tejeda, Gavin Kerrigan, and Padhraic Smyth. Bayesian modeling of human-ai complementarity. PNAS, 2022.","title":"Bayesian modeling of human-ai complementarity","authors":"Mark Steyvers, Heliodoro Tejeda, Gavin Kerrigan, Padhraic Smyth","year":"2022","venue":"PNAS","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Tomašev et al., 2019] Nenad Tomašev, Xavier Glorot, Jack W Rae, Michal Zielinski, Harry Askham, Andre Saraiva, Anne Mottram, Clemens Meyer, Suman Ravuri, Ivan Protsyuk, et al. A clinically applicable approach to continuous prediction of future acute kidney injury. Nature, 2019.","title":"A clinically applicable approach to continuous prediction of future acute kidney injury","authors":"Nenad Tomašev, Xavier Glorot, Michal Jack W Rae, Harry Zielinski, Andre Askham, Anne Saraiva, Clemens Mottram, Suman Meyer, Ivan Ravuri, Protsyuk","year":"2019","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Tschandl et al., 2020] Philipp Tschandl, Christoph Rinner, Zoe Apalla, Giuseppe Argenziano, Noel Codella, Allan Halpern, Monika Janda, Aimilios Lallas, Caterina Longo, Josep Malvehy, et al. Human-computer collaboration for skin cancer recognition. Nature Medicine, 2020.","title":"Human-computer collaboration for skin cancer recognition","authors":"Philipp Tschandl, Christoph Rinner, Zoe Apalla, Giuseppe Argenziano, Noel Codella, Allan Halpern, Monika Janda, Aimilios Lallas, Caterina Longo, Josep Malvehy","year":"2020","venue":"Nature Medicine","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Vaithianathan et al., 2017] Rhema Vaithianathan, Emily Putnam-Hornstein, Nan Jiang, Parma Nand, and Tim Mal- oney. Developing predictive models to support child mal- treatment hotline screening decisions: Allegheny county methodology and implementation. Center for Social data Analytics, 2017.","title":"Developing predictive models to support child maltreatment hotline screening decisions: Allegheny county methodology and implementation","authors":"Vaithianathan","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Van Swol and Sniezek, 2005] Lyn M Van Swol and Janet A Sniezek. Factors affecting the acceptance of expert advice. British journal of social psychology, 2005.","title":"Factors affecting the acceptance of expert advice","authors":"Van Swol, Lyn M Sniezek, Janet A Van Swol, Sniezek","year":"2005","venue":"British journal of social psychology","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Wilder et al., 2020] Bryan Wilder, Eric Horvitz, and Ece Kamar. Learning to complement humans. In IJCAI-20, 2020.","title":"Learning to complement humans","authors":"Bryan Wilder, Eric Horvitz, Ece Kamar","year":"2020","venue":"IJCAI-20","doi":null,"url":null,"internal_url":"/proceedings/2020/212"},{"position":52,"raw":"[Yaniv, 2004] Ilan Yaniv. Receiving other people's advice: Influence and benefit. Organizational Behavior and Hu- man Decision Processes, 2004.","title":"Receiving other people's advice: Influence and benefit","authors":"Ilan Yaniv, Yaniv","year":"2004","venue":"Organizational Behavior and Human Decision Processes","doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Yeomans et al., 2019] Michael Yeomans, Anuj Shah, Send- hil Mullainathan, and Jon Kleinberg. Making sense of rec- ommendations. Journal of Behavioral Decision Making, 2019.","title":"Making sense of recommendations","authors":"Michael Yeomans, Anuj Shah, Sendhil Mullainathan, Jon Kleinberg","year":"2019","venue":"Journal of Behavioral Decision Making","doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Yin et al., 2019] Ming Yin, Jennifer Wortman Vaughan, and Hanna Wallach. Understanding the effect of accuracy on trust in machine learning models. In CHI '19, 2019.","title":"Understanding the effect of accuracy on trust in machine learning models","authors":"Ming Yin, Jennifer Wortman Vaughan, Hanna Wallach","year":"2019","venue":"CHI '19","doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Zhang et al., 2020] Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. Effect of confidence and explana- tion on accuracy and trust calibration in ai-assisted deci- sion making. In FAT* '20, 2020.","title":"Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making","authors":"Zhang","year":"2020","venue":"FAT* '20","doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Zou et al., 2016] Quan Zou, Sifa Xie, Ziyu Lin, Meihong Wu, and Ying Ju. Finding the best classification threshold in imbalanced classification. Big Data Research, 5:2-8, 2016.","title":"Finding the best classification threshold in imbalanced classification","authors":"Quan Zou, Sifa Xie, Ziyu Lin, Meihong Wu, Ying Ju","year":"2016","venue":"Big Data Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}