{"type":"ScholarlyArticle","title":"Learning to Complement Humans","authors":[{"name":"Bryan Wilder","slug":"bryan-wilder","given":"Bryan","family":"Wilder","affiliation":"School of Engineering and Applied Sciences , Harvard University"},{"name":"Eric Horvitz","slug":"eric-horvitz","given":"Eric","family":"Horvitz","affiliation":"Microsoft Research"},{"name":"Ece Kamar","slug":"ece-kamar","given":"Ece","family":"Kamar","affiliation":"Microsoft Research"}],"abstract":"A rising vision for AI in the open world centers on the development of systems that can complement humans for perceptual, diagnostic, and reasoning tasks. To date, systems aimed at complementing the skills of people have employed models trained to be as accurate as possible in isolation. We demonstrate how an end-to-end learning strategy can be harnessed to optimize the combined performance of human-machine teams by considering the distinct abilities of people and machines. The goal is to focus machine learning on problem instances that are difficult for humans, while recognizing instances that are difficult for the machine and seeking human input on them. We demonstrate in two real-world domains (scientific discovery and medical diagnosis) that human-machine teams built via these methods outperform the individual performance of machines and people. We then analyze conditions under which this complementarity is strongest, and which training methods amplify it. Taken together, our work provides the first systematic investigation of how machine learning systems can be trained to complement human reasoning.","doi":"10.24963/ijcai.2020/212","url":"https://ijcai.org/proceedings/2020/212","pdf":"https://ijcai.org/proceedings/2020/0212.pdf","markdown":"https://ijcai.org/proceedings/2020/212.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/212","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Humans and AI","track":"Main track","number":212,"pages":{"first":1526,"last":1533},"keywords":["Humans and AI: Human-AI Collaboration"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25252/file_25252.m3u8","length_sec":275,"provider":"hls"}],"references":[{"position":1,"raw":"[Bansal et al., 2019a] Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S Lasecki, Daniel S Weld, and Eric Horvitz. Beyond accuracy: The role of mental models in human-AI team performance. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 7, pages 2-11, 2019.","title":"Beyond accuracy: The role of mental models in human-AI team performance","authors":"Bansal","year":"2019","venue":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bansal et al., 2019b] Gagan Bansal, Besmira Nushi, Ece Kamar, Daniel S Weld, Walter S Lasecki, and Eric Horvitz. Updates in human-AI teams: Understanding and address- ing the performance/compatibility tradeoff. In AAAI, vol- ume 33, pages 2429-2437, 2019.","title":"Updates in human-AI teams: Understanding and addressing the performance/compatibility tradeoff","authors":"Bansal","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bejnordi et al., 2017] Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes Van Diest, Bram Van Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen AWM Van Der Laak, Meyke Hermsen, Quirine F Manson, Maschenka Balken- hol, et al. Diagnostic assessment of deep learning algo- rithms for detection of lymph node metastases in women with breast cancer. JAMA, 318(22):2199-2210, 2017.","title":"Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer","authors":"Ehteshami Babak, Mitko Bejnordi, Paul Johannes Veta, Bram Van Diest, Nico Van Ginneken, Geert Karssemeijer, Litjens, Awm Jeroen, Meyke Van Der Laak, Hermsen, Maschenka Quirine F Manson, Balkenhol","year":"2017","venue":"JAMA","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2018] Hongming Chen, Ola Engkvist, Yinhai Wang, Marcus Olivecrona, and Thomas Blaschke. The rise of deep learning in drug discovery. Drug discovery today, 23(6):1241-1250, 2018.","title":"The rise of deep learning in drug discovery","authors":"Hongming Chen, Ola Engkvist, Yinhai Wang, Marcus Olivecrona, Thomas Blaschke","year":"2018","venue":"Drug discovery today","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[De et al., 2020] Abir De, Paramita Koley, Niloy Ganguly, and Manuel Gomez-Rodriguez. Regression under human assistance. In AAAI, 2020.","title":"Regression under human assistance","authors":"Abir De, Paramita Koley, Niloy Ganguly, Manuel Gomez-Rodriguez","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Esteva et al., 2017] Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542(7639):115, 2017.","title":"Dermatologist-level classification of skin cancer with deep neural networks","authors":"Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, Sebastian Thrun","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Green and Chen, 2019] Ben Green and Yiling Chen. The principles and limits of algorithm-in-the-loop decision making. CSCW, 2019.","title":"The principles and limits of algorithm-in-the-loop decision making","authors":"Chen ; Green, Ben Green, Yiling Chen","year":"2019","venue":"CSCW","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Grgić-Hlača et al., 2019] Nina Grgić-Hlača, Christoph En- gel, and Krishna P Gummadi. Human decision making with machine assistance: An experiment on bailing and jailing. CSCW, 2019.","title":"Human decision making with machine assistance: An experiment on bailing and jailing","authors":"Nina Grgić-Hlača, Christoph Engel, Krishna P Gummadi","year":"2019","venue":"CSCW","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hilgard et al., 2019] Sophie Hilgard, Nir Rosenfeld, Mahzarin R Banaji, Jack Cao, and David C Parkes. Learning representations by humans, for humans. arXiv preprint arXiv:1905.12686, 2019.","title":"Learning representations by humans, for humans","authors":"Sophie Hilgard, Nir Rosenfeld, R Mahzarin, Jack Banaji, David C Cao, Parkes","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Horvitz and Paek, 2007] Eric Horvitz and Tim Paek. Com- plementary computing: policies for transferring callers from dialog systems to human receptionists. User Mod- eling and User-Adapted Interaction, 17(1-2):159-182, 2007.","title":"Complementary computing: policies for transferring callers from dialog systems to human receptionists","authors":"Paek Horvitz, Eric Horvitz, Tim Paek","year":"2007","venue":"User Modeling and User-Adapted Interaction","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kamar et al., 2012] Ece Kamar, Severin Hacker, and Eric Horvitz. Combining human and machine intelligence in large-scale crowdsourcing. In AAMAS, pages 467-474, 2012.","title":"Combining human and machine intelligence in large-scale crowdsourcing","authors":"Ece Kamar, Severin Hacker, Eric Horvitz","year":"2012","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lage et al., 2018] Isaac Lage, Andrew Ross, Samuel J Ger- shman, Been Kim, and Finale Doshi-Velez. Human-in- the-loop interpretability prior. In Advances in Neural In- formation Processing Systems, 2018.","title":"Human-inthe-loop interpretability prior","authors":"Isaac Lage, Andrew Ross, Been Samuel J Gershman, Finale Kim, Doshi-Velez","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lintott et al., 2008] Chris J Lintott, Kevin Schawinski, Anže Slosar, Kate Land, Steven Bamford, Daniel Thomas, M Jordan Raddick, Robert C Nichol, Alex Szalay, Dan Andreescu, et al. Galaxy zoo: morphologies derived from visual inspection of galaxies from the sloan digital sky sur- vey. Monthly Notices of the Royal Astronomical Society, 389(3):1179-1189, 2008.","title":"Galaxy zoo: morphologies derived from visual inspection of galaxies from the sloan digital sky survey","authors":"Chris J Lintott, Kevin Schawinski, Anže Slosar, Kate Land, Steven Bamford, Daniel Thomas, M Jordan Raddick, Robert C Nichol, Alex Szalay, Dan Andreescu","year":"2008","venue":"Monthly Notices of the Royal Astronomical Society","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[McGinnis and Pearce, 2019] John O McGinnis and Rus- sell G Pearce. The great disruption: How machine intelli- gence will transform the role of lawyers in the delivery of legal services. Actual Probs. Econ. & L., page 1230, 2019.","title":"The great disruption: How machine intelligence will transform the role of lawyers in the delivery of legal services","authors":"O John, Russell G Mcginnis, Pearce","year":"2019","venue":"Actual Probs. Econ. & L","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Niculescu-Mizil and Caruana, 2005] Alexandru Niculescu- Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In ICML, pages 625-632, 2005.","title":"Predicting good probabilities with supervised learning","authors":"Alexandru Niculescu, -Mizil, Rich Caruana","year":"2005","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Platt, 1999] John C Platt. Using analytic qp and sparseness to speed training of support vector machines. In Advances in neural information processing systems, pages 557-563, 1999.","title":"Using analytic qp and sparseness to speed training of support vector machines","authors":"C John, Platt","year":"1999","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Raghu et al., 2019] Maithra Raghu, Katy Blumer, Greg Corrado, Jon Kleinberg, Ziad Obermeyer, and Send- hil Mullainathan. The algorithmic automation problem: Prediction, triage, and human effort. arXiv preprint arXiv:1903.12220, 2019.","title":"The algorithmic automation problem: Prediction, triage, and human effort","authors":"Raghu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Szegedy et al., 2016] Christian Szegedy, Vincent Van- houcke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, pages 2818-2826, 2016.","title":"Rethinking the inception architecture for computer vision","authors":"Szegedy","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tan et al., 2018] Sarah Tan, Julius Adebayo, Kori Inkpen, and Ece Kamar. Investigating human+ machine com- plementarity for recidivism predictions. arXiv preprint arXiv:1808.09123, 2018.","title":"Investigating human+ machine complementarity for recidivism predictions","authors":"Sarah Tan, Julius Adebayo, Kori Inkpen, Ece Kamar","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tversky and Kahneman, 1974] Amos Tversky and Daniel Kahneman. Judgment under uncertainty: Heuristics and biases. Science, 185(4157):1124-1131, 1974.","title":"Judgment under uncertainty: Heuristics and biases","authors":"Kahneman ; Tversky, Daniel Tversky, Kahneman","year":"1974","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vekariya, 2016] Arjun Vekariya. Implementation of camelyon'16 grand challenge. https://github.com/ arjunvekariyagithub/camelyon16-grand-challenge, 2016.","title":"Arjun Vekariya. Implementation of camelyon'16 grand challenge","authors":"Vekariya","year":"2016","venue":null,"doi":null,"url":"https://github.com/arjunvekariyagithub/camelyon16-grand-challenge","internal_url":null},{"position":22,"raw":"[Wang et al., 2016] Dayong Wang, Aditya Khosla, Rishab Gargeya, Humayun Irshad, and Andrew H Beck. Deep learning for identifying metastatic breast cancer. arXiv preprint arXiv:1606.05718, 2016.","title":"Deep learning for identifying metastatic breast cancer","authors":"Dayong Wang, Aditya Khosla, Rishab Gargeya, Humayun Irshad, Andrew H Beck","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2020] Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. Effect of confidence and explana- tion on accuracy and trust calibration in ai-assisted deci- sion making. In FAT*, 2020.","title":"Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making","authors":"Zhang","year":"2020","venue":"FAT*","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}