{"type":"ScholarlyArticle","title":"Strategic Adversarial Attacks in AI-assisted Decision Making to Reduce Human Trust and Reliance","authors":[{"name":"Zhuoran Lu","slug":"zhuoran-lu","given":"Zhuoran","family":"Lu","affiliation":"Purdue University"},{"name":"Zhuoyan Li","slug":"zhuoyan-li","given":"Zhuoyan","family":"Li","affiliation":"Purdue University"},{"name":"Chun-Wei Chiang","slug":"chun-wei-chiang","given":"Chun-Wei","family":"Chiang","affiliation":"Purdue University"},{"name":"Ming Yin","slug":"ming-yin","given":"Ming","family":"Yin","affiliation":"Purdue University"}],"abstract":"With the increased integration of AI technologies in human decision making processes, adversarial attacks on AI models become a greater concern than ever before as they may significantly hurt humans’ trust in AI models and decrease the effectiveness of human-AI collaboration. While many adversarial attack methods have been proposed to decrease the performance of an AI model, limited attention has been paid on understanding how these attacks will impact the human decision makers interacting with the model, and accordingly, how to strategically deploy adversarial attacks to maximize the reduction of human trust and reliance. In this paper, through a human-subject experiment, we first show that in AI-assisted decision making, the timing of the attacks largely influences how much humans decrease their trust in and reliance on AI—the decrease is particularly salient when attacks occur on decision making tasks that humans are highly confident themselves. Based on these insights, we next propose an algorithmic framework to infer the human decision maker’s hidden trust in the AI model and dynamically decide when the attacker should launch an attack to the model. Our evaluations show that following the proposed approach, attackers deploy more efficient attacks and achieve higher utility than adopting other baseline strategies.","doi":"10.24963/ijcai.2023/337","url":"https://ijcai.org/proceedings/2023/337","pdf":"https://ijcai.org/proceedings/2023/0337.pdf","markdown":"https://ijcai.org/proceedings/2023/337.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/337","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Humans and AI","track":"Main Track","number":337,"pages":{"first":3020,"last":3028},"keywords":["Humans and AI: HAI: Human-computer interaction","Humans and AI: HAI: Applications","Humans and AI: HAI: Human-AI collaboration"],"videos":[],"references":[{"position":1,"raw":"[Akhtar and Mian, 2018] Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access, 6:14410-14430, 2018.","title":"Threat of adversarial attacks on deep learning in computer vision: A survey","authors":"Naveed Akhtar, Ajmal Mian","year":"2018","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bansal et al., 2021a] Gagan Bansal, Besmira Nushi, Ece Kamar, Eric Horvitz, and Daniel S Weld. Is the most ac- curate ai the best teammate? optimizing ai for teamwork. In Proceedings of the AAAI Conference on Artificial Intel- ligence, volume 35, pages 11405-11414, 2021.","title":"Is the most accurate ai the best teammate? optimizing ai for teamwork","authors":"Bansal","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bansal et al., 2021b] Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. Does the whole exceed its parts? the effect of ai explanations on comple- mentary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Sys- tems, pages 1-16, 2021.","title":"Does the whole exceed its parts? the effect of ai explanations on complementary team performance","authors":"Bansal","year":"2021","venue":"Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Buc ¸inca et al., 2021] Zana Buc ¸inca, Maja Barbara Malaya, and Krzysztof Z Gajos. To trust or to think: cognitive forc- ing functions can reduce overreliance on ai in ai-assisted decision-making. Proceedings of the ACM on Human- Computer Interaction, 5(CSCW1):1-21, 2021.","title":"To trust or to think: cognitive forcing functions can reduce overreliance on ai in ai-assisted decision-making","authors":"Zana Buc ¸inca, Maja Barbara Malaya, Krzysztof Z Gajos","year":"2021","venue":"Proceedings of the ACM on Human-Computer Interaction","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chiang and Yin, 2022] Chun-Wei Chiang and Ming Yin. Exploring the effects of machine learning literacy inter- ventions on laypeople's reliance on machine learning mod- els. In 27th International Conference on Intelligent User Interfaces, pages 148-161, 2022.","title":"Exploring the effects of machine learning literacy interventions on laypeople's reliance on machine learning models","authors":"Chun-Wei Chiang, Ming Yin","year":"2022","venue":"27th International Conference on Intelligent User Interfaces","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chong et al., 2022] Leah Chong, Guanglu Zhang, Kosa Goucher-Lambert, Kenneth Kotovsky, and Jonathan Ca- gan. Human confidence in artificial intelligence and in themselves: The evolution and impact of confidence on adoption of ai advice. Computers in Human Behavior, 127:107018, 2022.","title":"Human confidence in artificial intelligence and in themselves: The evolution and impact of confidence on adoption of ai advice","authors":"Leah Chong, Guanglu Zhang, Kosa Goucher-Lambert, Kenneth Kotovsky, Jonathan Cagan","year":"2022","venue":"Computers in Human Behavior","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dietvorst et al., 2015] Berkeley J Dietvorst, Joseph P Sim- mons, and Cade Massey. Algorithm aversion: people er- roneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1):114, 2015.","title":"Algorithm aversion: people erroneously avoid algorithms after seeing them err","authors":"Dietvorst","year":"2015","venue":"Journal of Experimental Psychology: General","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Evtimov et al., 2017] Ivan Evtimov, Kevin Eykholt, Ear- lence Fernandes, Tadayoshi Kohno, Bo Li, Atul Prakash, Amir Rahmati, and Dawn Song. Robust physical-world attacks on machine learning models. arXiv preprint arXiv:1707.08945, 2(3):4, 2017.","title":"Robust physical-world attacks on machine learning models","authors":"Ivan Evtimov, Kevin Eykholt, Earlence Fernandes, Tadayoshi Kohno, Bo Li, Atul Prakash, Amir Rahmati, Dawn Song","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Eykholt et al., 2018] Kevin Eykholt, Ivan Evtimov, Ear- lence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-world attacks on deep learning visual classifica- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1625-1634, 2018.","title":"Robust physical-world attacks on deep learning visual classification","authors":"Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, Dawn Song","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ghaffari Laleh et al., 2022] Narmin Ghaffari Laleh, Daniel Truhn, Gregory Patrick Veldhuizen, Tianyu Han, Marko van Treeck, Roman D Buelow, Rupert Langer, Bastian Dislich, Peter Boor, Volkmar Schulz, et al. Adversarial at- tacks and adversarial robustness in computational pathol- ogy. Nature Communications, 13(1):1-10, 2022.","title":"Adversarial attacks and adversarial robustness in computational pathology","authors":"Narmin Ghaffari, Laleh, Daniel Truhn, Gregory Patrick Veldhuizen, Tianyu Han, Marko Van Treeck, Roman D Buelow, Rupert Langer, Bastian Dislich, Peter Boor, Volkmar Schulz","year":"2022","venue":"Nature Communications","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jia et al., 2020] Yunhan Jia Jia, Yantao Lu, Junjie Shen, Qi Alfred Chen, Hao Chen, Zhenyu Zhong, and Tao Wei Wei. Fooling detection alone is not enough: Ad- versarial attack against multiple object tracking. In International Conference on Learning Representations (ICLR'20), 2020.","title":"Fooling detection alone is not enough: Adversarial attack against multiple object tracking","authors":"Yunhan Jia, Jia, Yantao Lu, Junjie Shen, Qi Alfred Chen, Hao Chen, Zhenyu Zhong, Tao Wei Wei","year":"2020","venue":"International Conference on Learning Representations (ICLR'20)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kim et al., 2022] Sunnie SY Kim, Nicole Meister, Vikram V Ramaswamy, Ruth Fong, and Olga Rus- sakovsky. Hive: evaluating the human interpretability of visual explanations. In European Conference on Computer Vision, pages 280-298. Springer, 2022.","title":"Hive: evaluating the human interpretability of visual explanations","authors":"Nicole Sunnie Sy Kim, Meister, V Vikram, Ruth Ramaswamy, Olga Fong, Russakovsky","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lai and Tan, 2019] Vivian Lai and Chenhao Tan. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the conference on fairness, accountability, and transparency, pages 29-38, 2019.","title":"On human predictions with explanations and predictions of machine learning models: A case study on deception detection","authors":"Tan ; ] Lai, Lai Vivian, Chenhao Tan","year":"2019","venue":"Proceedings of the conference on fairness, accountability, and transparency","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lai et al., 2021] Vivian Lai, Chacha Chen, Q Vera Liao, Al- ison Smith-Renner, and Chenhao Tan. Towards a science of human-ai decision making: a survey of empirical stud- ies. arXiv preprint arXiv:2112.11471, 2021.","title":"Towards a science of human-ai decision making: a survey of empirical studies","authors":"Vivian Lai, Chacha Chen, Q Vera Liao, Alison Smith-Renner, Chenhao Tan","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lee et al., 2021] Jieun Lee, Genya Abe, Kenji Sato, and Makoto Itoh. Developing human-machine trust: Impacts of prior instruction and automation failure on driver trust in partially automated vehicles. Transportation research part F: traffic psychology and behaviour, 81:384-395, 2021.","title":"Developing human-machine trust: Impacts of prior instruction and automation failure on driver trust in partially automated vehicles","authors":"Jieun Lee, Genya Abe, Kenji Sato, Makoto Itoh","year":"2021","venue":"Transportation research part F: traffic psychology and behaviour","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2023] Zhuoyan Li, Zhuoran Lu, and Ming Yin. Modeling human trust and reliance in ai-assisted decision making: A markovian approach. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.","title":"Modeling human trust and reliance in ai-assisted decision making: A markovian approach","authors":"Li","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lu and Yin, 2021] Zhuoran Lu and Ming Yin. Human re- liance on machine learning models when performance feedback is limited: Heuristics and risks. In Proceedings of the 2021 CHI Conference on Human Factors in Com- puting Systems, pages 1-16, 2021.","title":"Human reliance on machine learning models when performance feedback is limited: Heuristics and risks","authors":"Yin ; ] Zhuoran Lu, Ming Lu, Yin","year":"2021","venue":"Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ma et al., 2023] Shuai Ma, Ying Lei, Xinru Wang, Chengbo Zheng, Chuhan Shi, Ming Yin, and Xiaojuan Ma. Who should i trust: Ai or myself? leveraging human and ai correctness likelihood to promote appropriate trust in ai- assisted decision-making. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Sys- tems, 2023.","title":"Who should i trust: Ai or myself? leveraging human and ai correctness likelihood to promote appropriate trust in aiassisted decision-making","authors":"Shuai Ma, Ying Lei, Xinru Wang, Chengbo Zheng, Chuhan Shi, Ming Yin, Xiaojuan Ma","year":"2023","venue":"Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Madry et al., 2017] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.","title":"Towards deep learning models resistant to adversarial attacks","authors":"Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, Adrian Vladu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[McLachlan and Krishnan, 2007] Geoffrey J McLachlan and Thriyambakam Krishnan. The EM algorithm and exten- sions. John Wiley & Sons, 2007.","title":"The EM algorithm and extensions","authors":"J Geoffrey, Thriyambakam Mclachlan, Krishnan","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Meissner and Keding, 2021] Philip Meissner and Christoph Keding. The human factor in ai-based decision-making. MIT Sloan Management Review, 63(1):1-5, 2021.","title":"The human factor in ai-based decision-making","authors":"Keding Meissner, Philip Meissner, Christoph Keding","year":"2021","venue":"MIT Sloan Management Review","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Nourani et al., 2020] Mahsan Nourani, Joanie King, and Eric Ragan. The role of domain expertise in user trust and the impact of first impressions with intelligent systems. In Proceedings of the AAAI Conference on Human Computa- tion and Crowdsourcing, volume 8, pages 112-121, 2020.","title":"The role of domain expertise in user trust and the impact of first impressions with intelligent systems","authors":"Nourani","year":"2020","venue":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Povolny and Trivedi, 2020] Steve Povolny and Shivangee Trivedi. Model hacking adas to pave safer roads for au- tonomous vehicles. McAfee Advanced Threat Research, 2020.","title":"Steve Povolny and Shivangee Trivedi. Model hacking adas to pave safer roads for autonomous vehicles","authors":"Trivedi Povolny","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Qiu et al., 2020] Haonan Qiu, Chaowei Xiao, Lei Yang, Xinchen Yan, Honglak Lee, and Bo Li. Semanticadv: Generating adversarial examples via attribute-conditioned image editing. In Computer Vision-ECCV 2020: 16th Eu- ropean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 19-37. Springer, 2020.","title":"Semanticadv: Generating adversarial examples via attribute-conditioned image editing","authors":"Qiu","year":"2020","venue":"Computer Vision-ECCV 2020: 16th European Conference","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Rechkemmer and Yin, 2022] Amy Rechkemmer and Ming Yin. When confidence meets accuracy: Exploring the ef- fects of multiple performance indicators on trust in ma- chine learning models. In CHI Conference on Human Fac- tors in Computing Systems, pages 1-14, 2022.","title":"When confidence meets accuracy: Exploring the effects of multiple performance indicators on trust in machine learning models","authors":"Yin Rechkemmer, Amy Rechkemmer, Ming Yin","year":"2022","venue":"CHI Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Samangouei et al., 2018] Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gan: Protecting classifiers against adversarial attacks using generative models. arXiv preprint arXiv:1805.06605, 2018.","title":"Defense-gan: Protecting classifiers against adversarial attacks using generative models","authors":"Pouya Samangouei, Maya Kabkab, Rama Chellappa","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Szegedy et al., 2013] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.","title":"Intriguing properties of neural networks","authors":"Szegedy","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tejeda et al., 2022] Heliodoro Tejeda, Aakriti Kumar, Padhraic Smyth, and Mark Steyvers. Ai-assisted decision- making: A cognitive modeling approach to infer latent reliance strategies. Computational Brain & Behavior, 5(4):491-508, 2022.","title":"Ai-assisted decisionmaking: A cognitive modeling approach to infer latent reliance strategies","authors":"Heliodoro Tejeda, Aakriti Kumar, Padhraic Smyth, Mark Steyvers","year":"2022","venue":"Computational Brain & Behavior","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wah et al., 2011a] C Wah, S. Branson, P. Welinder, P. Per- ona, and S. Belongie. Caltech-ucsd birds-200-2011. Tech- nical Report CNS-TR-2011-001, 2011.","title":null,"authors":"Wah","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wah et al., 2011b] Catherine Wah, Steve Branson, Pietro Perona, and Serge Belongie. Multiclass recognition and part localization with humans in the loop. In 2011 Interna- tional Conference on Computer Vision, pages 2524-2531. IEEE, 2011.","title":"Multiclass recognition and part localization with humans in the loop","authors":"Wah","year":"2011","venue":"2011 International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2022] Xinru Wang, Zhuoran Lu, and Ming Yin. Will you accept the ai recommendation? predicting human behavior in ai-assisted decision making. In Proceedings of the ACM Web Conference 2022, pages 1697-1708, 2022.","title":"Will you accept the ai recommendation? predicting human behavior in ai-assisted decision making","authors":"Wang","year":"2022","venue":"Proceedings of the ACM Web Conference 2022","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Xiao et al., 2018] Chaowei Xiao, Bo Li, Jun-Yan Zhu, War- ren He, Mingyan Liu, and Dawn Song. Generating adver- sarial examples with adversarial networks. arXiv preprint arXiv:1801.02610, 2018.","title":"Generating adversarial examples with adversarial networks","authors":"Chaowei Xiao, Bo Li, Jun-Yan Zhu, Warren He, Mingyan Liu, Dawn Song","year":"2018","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2018/543"},{"position":34,"raw":"[Yuan et al., 2019] Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. Adversarial examples: Attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems, 30(9):2805-2824, 2019.","title":"Adversarial examples: Attacks and defenses for deep learning","authors":"Xiaoyong Yuan, Pan He, Qile Zhu, Xiaolin Li","year":"2019","venue":"IEEE transactions on neural networks and learning systems","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang et al., 2020] Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. Effect of confidence and explana- tion on accuracy and trust calibration in ai-assisted deci- sion making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 295- 305, 2020.","title":"Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making","authors":"Zhang","year":"2020","venue":"Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}