{"type":"ScholarlyArticle","title":"Designing Behavior-Aware AI to Improve the Human-AI Team Performance in AI-Assisted Decision Making","authors":[{"name":"Syed Hasan Amin Mahmood","slug":"syed-hasan-amin-mahmood","given":"Syed Hasan Amin","family":"Mahmood","affiliation":"Purdue University"},{"name":"Zhuoran Lu","slug":"zhuoran-lu","given":"Zhuoran","family":"Lu","affiliation":"Purdue University"},{"name":"Ming Yin","slug":"ming-yin","given":"Ming","family":"Yin","affiliation":"Purdue University"}],"abstract":"With the rapid development of decision aids that are driven by AI models, the practice of AI-assisted decision making has become increasingly prevalent. To improve the human-AI team performance in decision making, earlier studies mostly focus on enhancing humans' capability in better utilizing a given AI-driven decision aid. In this paper, we tackle this challenge through a complementary approach—we aim to train \"behavior-aware AI\" by adjusting the AI model underlying the decision aid to account for humans'  behavior in adopting AI advice. In particular, as humans are observed to accept AI advice more when their confidence in their own judgement is low, we propose to train AI models with a human-confidence-based instance weighting strategy, instead of solving the standard empirical risk minimization problem. Under an assumed, threshold-based model characterizing when humans will adopt the AI advice, we first derive the optimal instance weighting strategy for training AI models. We then validate the efficacy and robustness of our proposed method in improving the human-AI joint decision making performance through systematic experimentation on synthetic datasets. Finally, via randomized experiments with real human subjects along with their actual behavior in adopting the AI advice, we demonstrate that our method can significantly improve the decision making performance of the human-AI team in practice.","doi":"10.24963/ijcai.2024/344","url":"https://ijcai.org/proceedings/2024/344","pdf":"https://ijcai.org/proceedings/2024/0344.pdf","markdown":"https://ijcai.org/proceedings/2024/344.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/344","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Humans and AI","track":"Main Track","number":344,"pages":{"first":3106,"last":3114},"keywords":["Humans and AI: HAI: Personalization and user modeling","Humans and AI: HAI: Human computation and crowdsourcing","Humans and AI: HAI: Human-AI collaboration","Humans and AI: HAI: Human-computer interaction"],"videos":[],"references":[{"position":1,"raw":"[Bansal et al., 2019] Gagan Bansal, Besmira Nushi, Ece Ka- mar, Walter S Lasecki, Daniel S Weld, and Eric Horvitz. Beyond accuracy: The role of mental models in human-ai team performance. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 7, pages 2-11, 2019.","title":"Beyond accuracy: The role of mental models in human-ai team performance","authors":"Gagan Bansal, Besmira Nushi, Ece Kamar, Daniel S Walter S Lasecki, Eric Weld, Horvitz","year":"2019","venue":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bansal et al., 2021a] Gagan Bansal, Besmira Nushi, Ece Ka- mar, Eric Horvitz, and Daniel S Weld. Is the most accurate ai the best teammate? optimizing ai for teamwork. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, pages 11405-11414, 2021.","title":"Is the most accurate ai the best teammate? optimizing ai for teamwork","authors":"Bansal","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bansal et al., 2021b] Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. Does the whole exceed its parts? the effect of ai explanations on comple- mentary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-16, 2021.","title":"Does the whole exceed its parts? the effect of ai explanations on complementary team performance","authors":"Bansal","year":"2021","venue":"Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bertrand et al., 2022] Astrid Bertrand, Rafik Belloum, James R Eagan, and Winston Maxwell. How cognitive biases affect xai-assisted decision-making: A systematic review. In Proceedings of the 2022 AAAI/ACM conference on AI, ethics, and society, pages 78-91, 2022.","title":"How cognitive biases affect xai-assisted decision-making: A systematic review","authors":"Bertrand","year":"2022","venue":"Proceedings of the 2022 AAAI/ACM conference on AI, ethics, and society","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bondi et al., 2022] Elizabeth Bondi, Raphael Koster, Han- nah Sheahan, Martin Chadwick, Yoram Bachrach, Tay- lan Cemgil, Ulrich Paquet, and Krishnamurthy Dvijotham. Role of human-ai interaction in selective prediction. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 5286-5294, 2022.","title":"Role of human-ai interaction in selective prediction","authors":"Elizabeth Bondi, Raphael Koster, Hannah Sheahan, Martin Chadwick, Yoram Bachrach, Taylan Cemgil, Ulrich Paquet, Krishnamurthy Dvijotham","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Buc ¸inca et al., 2021] Zana Buc ¸inca, Maja Barbara Malaya, and Krzysztof Z Gajos. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI- assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1):1-21, 2021.","title":"To trust or to think: cognitive forcing functions can reduce overreliance on AI in AIassisted decision-making","authors":"Zana Buc ¸inca, Maja Barbara Malaya, Krzysztof Z Gajos","year":"2021","venue":"Proceedings of the ACM on Human-Computer Interaction","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Carroll et al., 2019] Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dra- gan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019.","title":"On the utility of learning about humans for human-ai coordination","authors":"Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, Anca Dragan","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chong et al., 2022] Leah Chong, Guanglu Zhang, Kosa Goucher-Lambert, Kenneth Kotovsky, and Jonathan Ca- gan. Human confidence in artificial intelligence and in themselves: The evolution and impact of confidence on adoption of ai advice. Computers in Human Behavior, 127:107018, 2022.","title":"Human confidence in artificial intelligence and in themselves: The evolution and impact of confidence on adoption of ai advice","authors":"Chong","year":"2022","venue":"Computers in Human Behavior","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dunning, 2011] David Dunning. The dunning-kruger effect: On being ignorant of one's own ignorance. In Advances in experimental social psychology, volume 44, pages 247-296. Elsevier, 2011.","title":"The dunning-kruger effect: On being ignorant of one's own ignorance","authors":"David Dunning, Dunning","year":"2011","venue":"Advances in experimental social psychology","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dvijotham et al., 2023] Krishnamurthy Dvijotham, Jim Winkens, Melih Barsbey, Sumedh Ghaisas, Robert Stan- forth, Nick Pawlowski, Patricia Strachan, Zahra Ahmed, Shekoofeh Azizi, Yoram Bachrach, et al. Enhancing the reliability and accuracy of ai-enabled diagnosis via complementarity-driven deferral to clinicians. Nature Medicine, 29(7):1814-1820, 2023.","title":"Enhancing the reliability and accuracy of ai-enabled diagnosis via complementarity-driven deferral to clinicians","authors":"Krishnamurthy Dvijotham, Jim Winkens, Melih Barsbey, Sumedh Ghaisas, Robert Stanforth, Nick Pawlowski, Patricia Strachan, Zahra Ahmed, Shekoofeh Azizi, Yoram Bachrach","year":"2023","venue":"Nature Medicine","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Haider et al., 2022] Chowdhury Mohammad Rakin Haider, Chris Clifton, and Yan Zhou. Unfair ai: It isn't just biased data. In 2022 IEEE International Conference on Data Mining (ICDM), pages 957-962. IEEE, 2022.","title":"Unfair ai: It isn't just biased data","authors":"Chowdhury Mohammad, Rakin Haider, Chris Clifton, Yan Zhou","year":"2022","venue":"2022 IEEE International Conference on Data Mining (ICDM)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kwon et al., 2020] Minae Kwon, Erdem Biyik, Aditi Talati, Karan Bhasin, Dylan P Losey, and Dorsa Sadigh. When humans aren't optimal: Robots that collaborate with risk- aware humans. In Proceedings of the 2020 ACM/IEEE international conference on human-robot interaction, pages 43-52, 2020.","title":"When humans aren't optimal: Robots that collaborate with riskaware humans","authors":"Minae Kwon, Erdem Biyik, Aditi Talati, Karan Bhasin, Dylan P Losey, Dorsa Sadigh","year":"2020","venue":"Proceedings of the 2020 ACM/IEEE international conference on human-robot interaction","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Langford and Clance, 1993] Joe Langford and Pauline Rose Clance. The imposter phenomenon: Recent research find- ings regarding dynamics, personality and family patterns and their implications for treatment. Psychotherapy: theory, research, practice, training, 30(3):495, 1993.","title":"Joe Langford and Pauline Rose Clance. The imposter phenomenon: Recent research findings regarding dynamics, personality and family patterns and their implications for treatment","authors":"Clance Langford","year":"1993","venue":"Psychotherapy: theory, research, practice","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2023] Zhuoyan Li, Zhuoran Lu, and Ming Yin. Modeling human trust and reliance in ai-assisted decision making: A markovian approach. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6056-6064, 2023.","title":"Modeling human trust and reliance in ai-assisted decision making: A markovian approach","authors":"Li","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2024] Zhuoyan Li, Zhuoran Lu, and Ming Yin. De- coding ai's nudge: A unified framework to predict human behavior in ai-assisted decision making. arXiv preprint arXiv:2401.05840, 2024.","title":"Decoding ai's nudge: A unified framework to predict human behavior in ai-assisted decision making","authors":"Li","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lu and Yin, 2021] Zhuoran Lu and Ming Yin. Human re- liance on machine learning models when performance feed- back is limited: Heuristics and risks. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-16, 2021.","title":"Human reliance on machine learning models when performance feedback is limited: Heuristics and risks","authors":"Yin ; ] Zhuoran Lu, Ming Lu, Yin","year":"2021","venue":"Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Madras et al., 2018] David Madras, Toni Pitassi, and Richard Zemel. Predict responsibly: improving fairness and accuracy by learning to defer. Advances in Neural Information Processing Systems, 31, 2018.","title":"Predict responsibly: improving fairness and accuracy by learning to defer","authors":"David Madras, Toni Pitassi, Richard Zemel","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mozannar et al., 2022] Hussein Mozannar, Arvind Satya- narayan, and David Sontag. Teaching humans when to defer to a classifier via exemplars. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 5323-5331, 2022.","title":"Teaching humans when to defer to a classifier via exemplars","authors":"Mozannar","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rastogi et al., 2022] Charvi Rastogi, Yunfeng Zhang, Den- nis Wei, Kush R Varshney, Amit Dhurandhar, and Richard Tomsett. Deciding fast and slow: The role of cognitive biases in ai-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW1):1-22, 2022.","title":"Deciding fast and slow: The role of cognitive biases in ai-assisted decision-making","authors":"Rastogi","year":"2022","venue":"Proceedings of the ACM on Human-Computer Interaction","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schemmer et al., 2023] Max Schemmer, Niklas Kuehl, Ca- rina Benz, Andrea Bartos, and Gerhard Satzger. Appropri- ate reliance on ai advice: Conceptualization and the effect of explanations. In Proceedings of the 28th International Conference on Intelligent User Interfaces, pages 410-422, 2023.","title":"Appropriate reliance on ai advice: Conceptualization and the effect of explanations","authors":"Schemmer","year":"2023","venue":"Proceedings of the 28th International Conference on Intelligent User Interfaces","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang and Du, 2018] Xiuxin Wang and Xiufang Du. Why does advice discounting occur? the combined roles of confidence and trust. Frontiers in psychology, 9:2381, 2018.","title":"Xiuxin Wang and Xiufang Du. Why does advice discounting occur? the combined roles of confidence and trust","authors":"Du Wang","year":"2018","venue":"Frontiers in psychology","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2022] Xinru Wang, Zhuoran Lu, and Ming Yin. Will you accept the ai recommendation? predicting human behavior in ai-assisted decision making. In Proceedings of the ACM Web Conference 2022, pages 1697-1708, 2022.","title":"Will you accept the ai recommendation? predicting human behavior in ai-assisted decision making","authors":"Wang","year":"2022","venue":"Proceedings of the ACM Web Conference 2022","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wilder et al., 2020] Bryan Wilder, Eric Horvitz, and Ece Ka- mar. Learning to complement humans. In Proceedings of the Twenty-Ninth International Conference on Interna- tional Joint Conferences on Artificial Intelligence, pages 1526-1533, 2020.","title":"Learning to complement humans","authors":"Bryan Wilder, Eric Horvitz, Ece Kamar","year":"2020","venue":"Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2020/212"},{"position":25,"raw":"[Yang et al., 2020] Fumeng Yang, Zhuanyi Huang, Jean Scholtz, and Dustin L Arendt. How do visual explanations foster end users' appropriate trust in machine learning? In Proceedings of the 25th International Conference on Intelligent User Interfaces, pages 189-201, 2020.","title":"How do visual explanations foster end users' appropriate trust in machine learning?","authors":"Yang","year":"2020","venue":"Proceedings of the 25th International Conference on Intelligent User Interfaces","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2020] Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. Effect of confidence and explana- tion on accuracy and trust calibration in ai-assisted decision making. In Proceedings of the 2020 Conference on Fair- ness, Accountability, and Transparency, pages 295-305, 2020.","title":"Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making","authors":"Zhang","year":"2020","venue":"Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}