{"type":"ScholarlyArticle","title":"Detecting and Understanding Vulnerabilities in Language Models via Mechanistic Interpretability","authors":[{"name":"Jorge García-Carrasco","slug":"jorge-garcia-carrasco","given":"Jorge","family":"García-Carrasco","affiliation":"Lucentia Research , Department of Software and Computing Systems , University of Alicante"},{"name":"Alejandro Maté","slug":"alejandro-mate","given":"Alejandro","family":"Maté","affiliation":"Lucentia Research , Department of Software and Computing Systems , University of Alicante"},{"name":"Juan Trujillo","slug":"juan-trujillo","given":"Juan","family":"Trujillo","affiliation":"Lucentia Research , Department of Software and Computing Systems , University of Alicante"}],"abstract":"Large Language Models (LLMs), characterized by being trained on broad amounts of data in a self-supervised manner, have shown impressive performance across a wide range of tasks. Indeed, their generative abilities have aroused interest on the application of LLMs across a wide range of contexts. However, neural networks in general, and LLMs in particular, are known to be vulnerable to adversarial attacks, where an imperceptible change to the input can mislead the output of the model. This is a serious concern that impedes the use of LLMs on high-stakes applications, such as healthcare, where a wrong prediction can imply serious consequences. Even though there are many efforts on making LLMs more robust to adversarial attacks, there are almost no works that study how and where these vulnerabilities that make LLMs prone to adversarial attacks happen. Motivated by these facts, we explore how to localize and understand vulnerabilities, and propose a method, based on Mechanistic Interpretability (MI) techniques, to guide this process. Specifically, this method enables us to detect vulnerabilities related to a concrete task by (i) obtaining the subset of the model that is responsible for that task, (ii) generating adversarial samples for that task, and (iii) using MI techniques together with the previous samples to discover and understand the possible vulnerabilities. We showcase our method on a pretrained GPT-2 Small model carrying out the task of predicting 3-letter acronyms to demonstrate its effectiveness on locating and understanding concrete vulnerabilities of the model.","doi":"10.24963/ijcai.2024/43","url":"https://ijcai.org/proceedings/2024/43","pdf":"https://ijcai.org/proceedings/2024/0043.pdf","markdown":"https://ijcai.org/proceedings/2024/43.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/43","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"AI Ethics, Trust, Fairness","track":"Main Track","number":43,"pages":{"first":385,"last":393},"keywords":["AI Ethics, Trust, Fairness: ETF: Trustworthy AI","Machine Learning: ML: Explainable/Interpretable machine learning","AI Ethics, Trust, Fairness: ETF: Explainability and interpretability"],"videos":[],"references":[{"position":1,"raw":"[Ba et al., 2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Ge- offrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.","title":"Layer normalization","authors":"Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E Hinton","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Sys- tems, NeurIPS 2020, 33:1877-1901, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cascella et al., 2023] Marco Cascella, Jonathan Monto- moli, Valentina Bellini, and Elena Bignami. Evaluating the feasibility of chatgpt in healthcare: an analysis of mul- tiple clinical and research scenarios. Journal of Medical Systems, 47(1):33, 2023.","title":"Evaluating the feasibility of chatgpt in healthcare: an analysis of multiple clinical and research scenarios","authors":"Marco Cascella, Jonathan Montomoli, Valentina Bellini, Elena Bignami","year":"2023","venue":"Journal of Medical Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chan et al., 2022] Lawrence Chan, Adrià Garriga-Alonso, Nicholas Goldwosky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. Causal scrubbing, a method for rigorously testing interpretability hypotheses. https: //www.alignmentforum.org/posts/JvZhhzycHu2Yd57RN/ causal-scrubbing-a-method-for-rigorously-testing, 2022. Accessed: 2024-01-14.","title":"Causal scrubbing, a method for rigorously testing interpretability hypotheses","authors":"Lawrence Chan, Adrià Garriga-Alonso, Nicholas Goldwosky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, Nate Thomas","year":"2022","venue":null,"doi":null,"url":"https://www.alignmentforum.org/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-rigorously-testing","internal_url":null},{"position":5,"raw":"[Conmy et al., 2023] Arthur Conmy, Augustine N. Mavor- Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In Thirty-seventh Conference on Neural Information Processing Systems, NeurIPS 2023, 2023.","title":"Towards automated circuit discovery for mechanistic interpretability","authors":"Arthur Conmy, Augustine N Mavor-Parker, Aengus Lynch, Stefan Heimersheim, Adrià Garriga-Alonso","year":"2023","venue":"Thirty-seventh Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Elhage et al., 2021] Nelson Elhage, Neel Nanda, Cather- ine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1, 2021. https://transformer- circuits.pub/2021/framework/index.html Accessed: 2024- 01-14.","title":"A mathematical framework for transformer circuits","authors":"Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly","year":"2021","venue":"Transformer Circuits Thread","doi":null,"url":"https://transformer-circuits.pub/2021/framework/index.html","internal_url":null},{"position":7,"raw":"[Elhage et al., 2022] Nelson Elhage, Tristan Hume, Cather- ine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022.","title":"Toy models of superposition","authors":"Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[García-Carrasco et al., 2024] Jorge García-Carrasco, Ale- jandro Maté, and Juan Trujillo. How does GPT-2 pre- dict acronyms? Extracting and understanding a circuit via mechanistic interpretability. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learn- ing Research, pages 3322-3330. PMLR, 02-04 May 2024.","title":"Jorge García-Carrasco, Alejandro Maté, and Juan Trujillo. How does GPT-2 predict acronyms? Extracting and understanding a circuit via mechanistic interpretability","authors":"García-Carrasco","year":"2024","venue":"Proceedings of The 27th International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goodfellow et al., 2015] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.","title":"Explaining and harnessing adversarial examples","authors":"Ian J Goodfellow, Jonathon Shlens, Christian Szegedy","year":"2015","venue":"3rd International Conference on Learning Representations, ICLR 2015","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guo et al., 2021] Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient-based adver- sarial attacks against text transformers. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Em- pirical Methods in Natural Language Processing, pages 5747-5757, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics.","title":"Gradient-based adversarial attacks against text transformers","authors":"Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, Douwe Kiela","year":"2021","venue":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hanna et al., 2023] Michael Hanna, Ollie Liu, and Alexan- dre Variengien. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained lan- guage model. In Thirty-seventh Conference on Neural In- formation Processing Systems, NeurIPS 2023, 2023.","title":"How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model","authors":"Michael Hanna, Ollie Liu, Alexandre Variengien","year":"2023","venue":"Thirty-seventh Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Heimersheim and Janiak, 2023] Stefan Heimersheim and Jett Janiak. A circuit for Python docstrings in a 4-layer attention-only transformer. https://www. alignmentforum.org/posts/u6KXXmKFbXfWzoAXn/ a-circuit-for-python-docstrings-in-a-4-layer-attention-only, 2023. Accessed: 2024-01-14.","title":"A circuit for Python docstrings in a 4-layer attention-only transformer","authors":"Janiak Heimersheim, Stefan Heimersheim, Jett Janiak","year":"2023","venue":null,"doi":null,"url":"https://www.alignmentforum.org/posts/u6KXXmKFbXfWzoAXn/a-circuit-for-python-docstrings-in-a-4-layer-attention-only","internal_url":null},{"position":13,"raw":"[Huang et al., 2020] Xiaowei Huang, Daniel Kroening, Wenjie Ruan, James Sharp, Youcheng Sun, Emese Thamo, Min Wu, and Xinping Yi. A survey of safety and trust- worthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability. Com- puter Science Review, 37:100270, 2020.","title":"A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability","authors":"Xiaowei Huang, Daniel Kroening, Wenjie Ruan, James Sharp, Youcheng Sun, Emese Thamo, Min Wu, Xinping Yi","year":"2020","venue":"Computer Science Review","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kaplan et al., 2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.","title":"Scaling laws for neural language models","authors":"Jared Kaplan, Sam Mccandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2020] Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao. Adversarial training for large neural language mod- els. arXiv preprint arXiv:2004.08994, 2020.","title":"Adversarial training for large neural language models","authors":"Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, Jianfeng Gao","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Madry et al., 2018] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.","title":"Towards deep learning models resistant to adversarial attacks","authors":"Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, Adrian Vladu","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[McGrath et al., 2023] Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model com- putations. arXiv preprint arXiv:2307.15771, 2023.","title":"The hydra effect: Emergent self-repair in language model computations","authors":"Thomas Mcgrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, Shane Legg","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Meng et al., 2022] Kevin Meng, David Bau, Alex Ando- nian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Pro- cessing Systems, NeurIPS 2022, 35:17359-17372, 2022.","title":"Locating and editing factual associations in gpt","authors":"Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nanda and Bloom, 2022] Neel Nanda and Joseph Bloom. Transformerlens. https://github.com/neelnanda-io/ TransformerLens, 2022. Accessed: 2024-01-14.","title":"Transformerlens","authors":"Neel Nanda, Joseph Bloom","year":"2022","venue":null,"doi":null,"url":"https://github.com/neelnanda-io/TransformerLens","internal_url":null},{"position":20,"raw":"[Olah et al., 2020] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3):e00024- 001, 2020.","title":"Zoom in: An introduction to circuits","authors":"Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, Shan Carter","year":"2020","venue":"Distill","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Olsson et al., 2022] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield- Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. https://transformer-circuits.pub/2022/in-context- learning-and-induction-heads/index.html Accessed: 2024- 01-14.","title":"In-context learning and induction heads","authors":"Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova Dassarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam Mccandlish, Chris Olah","year":"2022","venue":"Transformer Circuits Thread","doi":null,"url":"https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html","internal_url":null},{"position":22,"raw":"[Paszke et al., 2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in Neural Information Processing Systems, NeurIPS 2019, 32, 2019.","title":"Pytorch: An imperative style, highperformance deep learning library","authors":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Piscitelli, 2016] Jordan Piscitelli. Simple wordlists. https:// github.com/taikuukaits/SimpleWordlists, 2016. Accessed: 2023-11-18.","title":"Simple wordlists","authors":"Jordan Piscitelli","year":"2016","venue":null,"doi":null,"url":"https://github.com/taikuukaits/SimpleWordlists","internal_url":null},{"position":24,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":"OpenAI blog","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shayegani et al., 2023] Erfan Shayegani, Md Abdullah Al Mamun, Yu Fu, Pedram Zaree, Yue Dong, and Nael Abu- Ghazaleh. Survey of vulnerabilities in large language models revealed by adversarial attacks. arXiv preprint arXiv:2310.10844, 2023.","title":"Survey of vulnerabilities in large language models revealed by adversarial attacks","authors":"Erfan Shayegani, Md Abdullah Al Mamun, Yu Fu, Pedram Zaree, Yue Dong, Nael Abu-Ghazaleh","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Sys- tems, NeurIPS 2017, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2019] Dilin Wang, Chengyue Gong, and Qiang Liu. Improving neural language modeling via adversarial training. In International Conference on Machine Learn- ing, ICML 2019, pages 6555-6565. PMLR, 2019.","title":"Improving neural language modeling via adversarial training","authors":"Wang","year":"2019","venue":"International Conference on Machine Learning, ICML 2019","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2023] Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. In- terpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh Inter- national Conference on Learning Representations, ICLR 2023, 2023.","title":"Interpretability in the wild: a circuit for indirect object identification in GPT-2 small","authors":"Kevin Ro, Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, Jacob Steinhardt","year":"2023","venue":"The Eleventh International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wen et al., 2023] Yuxin Wen, Neel Jain, John Kirchen- bauer, Micah Goldblum, Jonas Geiping, and Tom Gold- stein. Hard prompts made easy: Gradient-based dis- crete optimization for prompt tuning and discovery. In Thirty-seventh Conference on Neural Information Pro- cessing Systems, NeurIPS 2023, 2023.","title":"Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery","authors":"Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, Tom Goldstein","year":"2023","venue":"Thirty-seventh Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}