{"type":"ScholarlyArticle","title":"Down the Toxicity Rabbit Hole: A Framework to Bias Audit Large Language Models with Key Emphasis on Racism, Antisemitism, and Misogyny","authors":[{"name":"Arka Dutta","slug":"arka-dutta","given":"Arka","family":"Dutta","affiliation":"Rochester Institute of Technology"},{"name":"Adel Khorramrouz","slug":"adel-khorramrouz","given":"Adel","family":"Khorramrouz","affiliation":"Rochester Institute of Technology"},{"name":"Sujan Dutta","slug":"sujan-dutta","given":"Sujan","family":"Dutta","affiliation":"Rochester Institute of Technology"},{"name":"Ashiqur R. KhudaBukhsh","slug":"ashiqur-r-khudabukhsh","given":"Ashiqur R.","family":"KhudaBukhsh","affiliation":"Rochester Institute of Technology"}],"abstract":"This paper makes three contributions. First, it presents a generalizable, novel framework dubbed toxicity rabbit hole that iteratively elicits toxic content from a wide suite of large language models. Spanning a set of 1,266 identity groups, we first conduct a bias audit of PaLM 2 guardrails presenting key insights. Next, we report generalizability across several other models. Through the elicited toxic content, we present a broad analysis with a key emphasis on racism, antisemitism, misogyny, Islamophobia, homophobia, and transphobia. We release a massive dataset of machine-generated toxic content with a view toward safety for all. Finally, driven by concrete examples, we discuss potential ramifications.","doi":"10.24963/ijcai.2024/801","url":"https://ijcai.org/proceedings/2024/801","pdf":"https://ijcai.org/proceedings/2024/0801.pdf","markdown":"https://ijcai.org/proceedings/2024/801.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/801","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"AI for Good","track":"AI for Good","number":801,"pages":{"first":7242,"last":7250},"keywords":["Natural Language Processing: General","AI Ethics, Trust, Fairness: General"],"videos":[],"references":[{"position":1,"raw":"[Abid et al., 2021] Abubakar Abid, Maheen Farooqi, and James Zou. Persistent anti-muslim bias in large language models. In AIES, pages 298-306, 2021.","title":"Persistent anti-muslim bias in large language models","authors":"Abubakar Abid, Maheen Farooqi, James Zou","year":"2021","venue":"AIES","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anil et al., 2023] Rohan Anil, Andrew M Dai, Orhan Fi- rat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.","title":"Palm 2 technical report","authors":"Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Azevedo, 1998] Mario Joaquim Azevedo. Roots of vio- lence: a history of war in Chad, volume 4. Psychology Press, 1998.","title":"Roots of violence: a history of war in Chad","authors":"Mario Joaquim, Azevedo","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bender et al., 2021] Emily M. Bender, Timnit Gebru, An- gelina McMillan-Major, and Shmargaret Shmitchell. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In ACM FaccT, pages 610-623, 2021.","title":"Angelina McMillan-Major, and Shmargaret Shmitchell. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?","authors":"Emily M Bender, Timnit Gebru","year":"2021","venue":"ACM FaccT","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bommasani et al., 2022] Rishi Bommasani, Kathleen A Creel, Ananya Kumar, Dan Jurafsky, and Percy S Liang. Picking on the same person: Does algorithmic monocul- ture lead to outcome homogenization? Advances in Neural Information Processing Systems, 35:3663-3678, 2022.","title":"Picking on the same person: Does algorithmic monoculture lead to outcome homogenization?","authors":"Rishi Bommasani, Kathleen A Creel, Ananya Kumar, Dan Jurafsky, Percy S Liang","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bommasani et al., 2023] Rishi Bommasani, Sayash Kapoor, Kevin Klyman, Shayne Longpre, Ashwin Ra- maswami, Daniel Zhang, Marietje Schaake, Daniel E. Ho, Arvind Narayanan, and Percy Liang. Considerations for governing open foundation models, Dec 2023. [Castelvecchi, 2024] Davide Castelvecchi. Deepmind ai out- does human mathematicians on unsolved problem. Nature, 625(7993):12-13, 2024.","title":"Davide Castelvecchi. Deepmind ai outdoes human mathematicians on unsolved problem","authors":"Rishi Bommasani, Sayash Kapoor, Kevin Klyman, Shayne Longpre, Ashwin Ramaswami, Daniel Zhang, Marietje Schaake, Daniel E Ho, Arvind Narayanan, Percy Liang","year":"2024","venue":"Castelvecchi","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2023] Lingjiao Chen, Matei Zaharia, and James Zou. How is chatgpt's behavior changing over time? arXiv preprint arXiv:2307.09009, 2023.","title":"How is chatgpt's behavior changing over time","authors":"Chen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chiang et al., 2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing GPT- 4 with 90%* ChatGPT quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2023.","title":"Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality","authors":"Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez","year":"2023","venue":null,"doi":null,"url":"https://vicuna.lmsys.org","internal_url":null},{"position":9,"raw":"[Conover et al., 2023] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction- tuned llm, 2023.","title":"Free dolly: Introducing the world's first truly open instructiontuned llm","authors":"Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, Reynold Xin","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Demszky et al., 2019] Dorottya Demszky, Nikhil Garg, Rob Voigt, James Zou, Jesse Shapiro, Matthew Gentzkow, and Dan Jurafsky. Analyzing polarization in social media: Method and application to tweets on 21 mass shootings. In Proceedings of NAACL-HLT. ACL, 2019.","title":"Analyzing polarization in social media: Method and application to tweets on 21 mass shootings","authors":"Dorottya Demszky, Nikhil Garg, Rob Voigt, James Zou, Jesse Shapiro, Matthew Gentzkow, Dan Jurafsky","year":"2019","venue":"Proceedings of NAACL-HLT. ACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Deshpande et al., 2023] Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. Toxicity in chatgpt: Analyzing persona-assigned language models. arXiv preprint arXiv:2304.05335, 2023.","title":"Toxicity in chatgpt: Analyzing persona-assigned language models","authors":"Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Dettmers et al., 2023] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetun- ing of quantized llms. arXiv preprint arXiv:2305.14314, 2023.","title":"Qlora: Efficient finetuning of quantized llms","authors":"Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Fang et al., 2023] Chuyu Fang, Chuan Qin, Qi Zhang, Kaichun Yao, Jingshuai Zhang, Hengshu Zhu, Fuzhen Zhuang, and Hui Xiong. Recruitpro: A pretrained lan- guage model with skill-aware prompt learning for intelli- gent recruitment. In ACM SIGKDD, 2023.","title":"Recruitpro: A pretrained language model with skill-aware prompt learning for intelligent recruitment","authors":"Chuyu Fang, Chuan Qin, Qi Zhang, Kaichun Yao, Jingshuai Zhang, Hengshu Zhu, Fuzhen Zhuang, Hui Xiong","year":"2023","venue":"ACM SIGKDD","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Fearon, 2003] James D Fearon. Ethnic and cultural diversity by country. Journal of economic growth, 8:195-222, 2003.","title":"Ethnic and cultural diversity by country","authors":"James D Fearon","year":"2003","venue":"Journal of economic growth","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Glaese et al., 2022] Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Mari- beth Rauh, Laura Weidinger, Martin J. Chadwick, et al. Improving alignment of dialogue agents via targeted hu- man judgements. CoRR, abs/2209.14375, 2022.","title":"Improving alignment of dialogue agents via targeted human judgements","authors":"Glaese","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Harris and Fiske, 2015] Lasana T Harris and Susan T Fiske. Dehumanized perception. Zeitschrift für Psychologie, 2015.","title":"Dehumanized perception","authors":"T Lasana, Susan T Harris, Fiske","year":"2015","venue":"Zeitschrift für Psychologie","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hartvigsen et al., 2022] Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. arXiv preprint arXiv:2203.09509, 2022.","title":"Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection","authors":"Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, Ece Kamar","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Hilberg, 2003] Raul Hilberg. The destruction of the Euro- pean Jews. Yale University Press, 2003.","title":"The destruction of the European Jews","authors":"Raul Hilberg","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Holzmeyer, 2021] Cheryl Holzmeyer. Beyond 'ai for social good'(ai4sg): social transformations-not tech-fixes-for health equity. Interdisciplinary Science Reviews, 46(1- 2):94-125, 2021.","title":"Beyond 'ai for social good'(ai4sg): social transformations-not tech-fixes-for health equity","authors":"Cheryl Holzmeyer","year":"2021","venue":"Interdisciplinary Science Reviews","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Iyengar and Westwood, 2015] Shanto Iyengar and Sean J Westwood. Fear and loathing across party lines: New evi- dence on group polarization. American Journal of Political Science, 59(3):690-707, 2015.","title":"Fear and loathing across party lines: New evidence on group polarization","authors":"Shanto Iyengar, Sean J Westwood","year":"2015","venue":"American Journal of Political Science","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Jeshion, 2013] Robin Jeshion. Slurs and stereotypes. Ana- lytic Philosophy, 54(3):314-329, 2013.","title":"Robin Jeshion. Slurs and stereotypes","authors":"Jeshion","year":"2013","venue":"Analytic Philosophy","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Jiang et al., 2023] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.","title":"Mistral 7b","authors":"Alexandre Albert Q Jiang, Arthur Sablayrolles, Chris Mensch, Devendra Bamford, Diego Singh Chaplot, Florian De Las Casas, Gianna Bressand, Guillaume Lengyel, Lucile Lample, Saulnier","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[KhudaBukhsh et al., 2021] Ashiqur R. KhudaBukhsh, Ru- pak Sarkar, Mark S. Kamlet, and Tom M. Mitchell. We don't speak the same language: Interpreting polarization through machine translation. In AAAI 2021, pages 14893- 14901. AAAI Press, 2021.","title":"We don't speak the same language: Interpreting polarization through machine translation","authors":"R Ashiqur, Rupak Khudabukhsh, Mark S Sarkar, Tom M Kamlet, Mitchell","year":"2021","venue":"AAAI 2021","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Korbak et al., 2023] Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Ja- son Phang, Samuel R Bowman, and Ethan Perez. Pretrain- ing language models with human preferences. In ICML, pages 17506-17533. PMLR, 2023.","title":"Pretraining language models with human preferences","authors":"Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Ethan Samuel R Bowman, Perez","year":"2023","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Li et al., 2023] Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yangqiu Song. Multi-step jailbreaking privacy attacks on chatgpt. arXiv preprint arXiv:2304.05197, 2023.","title":"Multi-step jailbreaking privacy attacks on chatgpt","authors":"Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Yangqiu Song","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Liu et al., 2023] Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tian- wei Zhang, and Yang Liu. Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study. arXiv preprint arXiv:2305.13860, 2023.","title":"Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study","authors":"Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, Yang Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Mendelsohn et al., 2020] Julia Mendelsohn, Yulia Tsvetkov, and Dan Jurafsky. A framework for the computational linguistic analysis of dehumanization. Frontiers in artificial intelligence, 3:55, 2020.","title":"A framework for the computational linguistic analysis of dehumanization","authors":"Julia Mendelsohn, Yulia Tsvetkov, Dan Jurafsky","year":"2020","venue":"Frontiers in artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ouyang et al., 2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instruc- tions with human feedback. Advances in Neural Informa- tion Processing Systems, 35:27730-27744, 2022.","title":"Training language models to follow instructions with human feedback","authors":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Pierson et al., 2023] Emma Pierson, Divya Shanmugam, Rajiv Movva, Jon Kleinberg, Monica Agrawal, Mark Dredze, Kadija Ferryman, Judy Wawira Gichoya, Dan Ju- rafsky, Pang Wei Koh, et al. Use large language models to promote equity. arXiv preprint arXiv:2312.14804, 2023.","title":"Use large language models to promote equity","authors":"Emma Pierson, Divya Shanmugam, Rajiv Movva, Jon Kleinberg, Monica Agrawal, Mark Dredze, Kadija Ferryman, Judy Wawira Gichoya, Dan Jurafsky, Pang Wei Koh","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Reimers and Gurevych, 2019] Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings us- ing Siamese BERT-networks. In EMNLP 2019, pages 3982-3992, 2019.","title":"Sentence-BERT: Sentence embeddings using Siamese BERT-networks","authors":"Gurevych Reimers, Nils Reimers, Iryna Gurevych","year":"2019","venue":"EMNLP 2019","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sedgwick, 1882] William Thompson Sedgwick. On varia- tions of reflex-excitability in the frog, induced by changes of temperature. Stud. Biol. Lab., Johns Hopkins Univer- sity, 385, 1882.","title":"On variations of reflex-excitability in the frog, induced by changes of temperature","authors":"William Thompson, Sedgwick","year":null,"venue":"Stud. Biol. Lab","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Steuter and Wills, 2010] Erin Steuter and Deborah Wills. 'the vermin have struck again': Dehumanizing the enemy in post 9/11 media representations. Media, War & Conflict, 3(2):152-167, 2010.","title":"the vermin have struck again': Dehumanizing the enemy in post 9/11 media representations","authors":"Wills Steuter, Erin Steuter, Deborah Wills","year":"2010","venue":"Media, War & Conflict","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Taori et al., 2023] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/ tatsu-lab/stanford alpaca, 2023.","title":"Stanford alpaca: An instruction-following llama model","authors":"Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, Tatsunori B Hashimoto","year":"2023","venue":null,"doi":null,"url":"https://github.com/tatsu-lab/stanfordalpaca","internal_url":null},{"position":34,"raw":"[Team, 2023] MosaicML NLP Team. Introducing mpt-30b: Raising the bar for open-source foundation models, 2023. Accessed: 2023-06-22.","title":"MosaicML NLP Team. Introducing mpt-30b: Raising the bar for open-source foundation models","authors":null,"year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Thirunavukarasu et al., 2023] Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8):1930-1940, 2023.","title":"Large language models in medicine","authors":"Arun James Thirunavukarasu, Darren Shu, Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, Daniel Shu, Wei Ting","year":"2023","venue":"Nature medicine","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Tunstall et al., 2023] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944, 2023.","title":"Zephyr: Direct distillation of lm alignment","authors":"Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Clémentine Fourrier, Nathan Habib","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wang et al., 2023a] Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khy- athi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. How far can camels go? exploring the state of instruction tuning on open resources, 2023.","title":"How far can camels go? exploring the state of instruction tuning on open resources","authors":"Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wang et al., 2023b] Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. Do-not-answer: A dataset for evaluating safeguards in llms. arXiv preprint arXiv:2308.13387, 2023.","title":"Do-not-answer: A dataset for evaluating safeguards in llms","authors":"Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Wei et al., 2023] Alexander Wei, Nika Haghtalab, and Ja- cob Steinhardt. Jailbroken: How Does LLM Safety Train- ing Fail? arXiv preprint arXiv:2307.02483, 2023.","title":"Jailbroken: How Does LLM Safety Training Fail","authors":"Alexander Wei, Nika Haghtalab, Jacob Steinhardt","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Xu et al., 2023] Can Xu, Qingfeng Sun, Kai Zheng, Xi- ubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023.","title":"Wizardlm: Empowering large language models to follow complex instructions","authors":"Xu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zhuo et al., 2023] Terry Yue Zhuo, Yujin Huang, Chun- yang Chen, and Zhenchang Xing. Exploring AI ethics of ChatGPT: A diagnostic analysis. arXiv preprint arXiv:2301.12867, 2023.","title":"Exploring AI ethics of ChatGPT: A diagnostic analysis","authors":"Terry Yue Zhuo, Yujin Huang, Chunyang Chen, Zhenchang Xing","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Ziems et al., 2023] Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. Can large language models transform computational social sci- ence?, 2023.","title":"Can large language models transform computational social science?","authors":"Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, Diyi Yang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Zou et al., 2023] Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversar- ial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023.","title":"Universal and transferable adversarial attacks on aligned language models","authors":"Andy Zou, Zifan Wang, J Zico Kolter, Matt Fredrikson","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}