{"type":"ScholarlyArticle","title":"Memorizing Documents with Guidance in Large Language Models","authors":[{"name":"Bumjin Park","slug":"bumjin-park","given":"Bumjin","family":"Park","affiliation":"INEEJI"},{"name":"Jaesik Choi","slug":"jaesik-choi","given":"Jaesik","family":"Choi","affiliation":"INEEJI"}],"abstract":"Training data plays a pivotal role in AI models. Large language models (LLMs) are trained with massive amounts of documents, and their parameters hold document-related contents. Recently, several studies identified content-specific locations in LLMs by examining the parameters. Instead of the post hoc interpretation, we propose another approach. We propose document-wise memory architecture to track document memories in training. The proposed architecture maps document representations to memory entries, which softly mask memories in the forward process of LLMs. Additionally, we propose document guidance loss, which increases the likelihood of text with document memories and reduces the likelihood of the text with the memories of other documents. Experimental results on Wikitext-103-v1 with Pythia-1B show that the proposed methods provide different memory entries for documents and high recall of document-related content in generation with trained document-wise memories.","doi":"10.24963/ijcai.2024/714","url":"https://ijcai.org/proceedings/2024/714","pdf":"https://ijcai.org/proceedings/2024/0714.pdf","markdown":"https://ijcai.org/proceedings/2024/714.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/714","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Natural Language Processing","track":"Main Track","number":714,"pages":{"first":6460,"last":6468},"keywords":["Natural Language Processing: NLP: Interpretability and analysis of models for NLP","Natural Language Processing: NLP: Language generation","Natural Language Processing: NLP: Language models","Natural Language Processing: NLP: Embeddings"],"videos":[],"references":[{"position":1,"raw":"[Biderman et al., 2023] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Puro- hit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across train- ing and scaling. In International Conference on Machine Learning. PMLR, 2023.","title":"Pythia: A suite for analyzing large language models across training and scaling","authors":"Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, O' Kyle, Eric Brien, Mohammad Hallahan, Shivanshu Aflah Khan, Purohit, Edward Usvsn Sai Prashanth, Raff","year":"2023","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bills et al., 2023] Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saun- ders. Language models can explain neurons in lan- guage models. https://openaipublic.blob.core.windows. net/neuron-explainer/paper/index.html, 2023.","title":"Language models can explain neurons in language models","authors":"Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, William Saunders","year":"2023","venue":null,"doi":null,"url":"https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html","internal_url":null},{"position":3,"raw":"[Bricken et al., 2023] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023.","title":"Towards monosemanticity: Decomposing language models with dictionary learning","authors":"Bricken","year":"2023","venue":"Transformer Circuits Thread","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learn- ers. In Advances in Neural Information Processing Sys- tems, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mccandlish, Alec Radford, Ilya Sutskever, Dario Amodei","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dai et al., 2022] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2022.","title":"Knowledge neurons in pretrained transformers","authors":"Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, Furu Wei","year":"2022","venue":"Proceedings of the Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dave et al., 2023] Tirth Dave, Sai Anirudh Athaluri, and Satyam Singh. Chatgpt in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Frontiers in Artificial Intelligence, 2023.","title":"Chatgpt in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations","authors":"Tirth Dave, Sai Anirudh Athaluri, Satyam Singh","year":"2023","venue":"Frontiers in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dhariwal and Nichol, 2021] Prafulla Dhariwal and Alexan- der Nichol. Diffusion models beat gans on image synthe- sis. In Advances in Neural Information Processing Sys- tems, 2021.","title":"Diffusion models beat gans on image synthesis","authors":"Nichol ; Dhariwal, Nichol","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Diao et al., 2023] Shizhe Diao, Tianyang Xu, Ruijia Xu, Jiawei Wang, and Tong Zhang. Mixture-of-domain- adapters: Decoupling and injecting domain knowledge to pre-trained language models' memories. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2023.","title":"Mixture-of-domainadapters: Decoupling and injecting domain knowledge to pre-trained language models' memories","authors":"Diao","year":"2023","venue":"Proceedings of the Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Du and Mordatch, 2019] Yilun Du and Igor Mordatch. Im- plicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems, 2019.","title":"Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models","authors":"Mordatch Du","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Elhage et al., 2022] Nelson Elhage, Tristan Hume, Cather- ine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022.","title":"Toy models of superposition","authors":"Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Geva et al., 2021] Mor Geva, Roei Schuster, Jonathan Be- rant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2021.","title":"Transformer feed-forward layers are key-value memories","authors":"Mor Geva, Roei Schuster, Jonathan Berant, Omer Levy","year":"2021","venue":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gong et al., 2019] Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel. Memorizing nor- mality to detect anomaly: Memory-augmented deep au- toencoder for unsupervised anomaly detection. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, 2019.","title":"Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection","authors":"Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, Anton Van Den, Hengel","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hacker et al., 2023] Philipp Hacker, Andreas Engel, and Marco Mauer. Regulating chatgpt and other large genera- tive ai models. In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency, 2023.","title":"Regulating chatgpt and other large generative ai models","authors":"Philipp Hacker, Andreas Engel, Marco Mauer","year":"2023","venue":"Proceedings of the ACM Conference on Fairness, Accountability, and Transparency","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ho and Salimans, 2021] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS Workshop on Deep Generative Models and Downstream Applica- tions, 2021.","title":"Classifier-free diffusion guidance","authors":"Jonathan Ho, Tim Salimans","year":"2021","venue":"NeurIPS Workshop on Deep Generative Models and Downstream Applications","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hu et al., 2022] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learn- ing Representations, 2022.","title":"LoRA: Low-rank adaptation of large language models","authors":"J Edward, Phillip Hu, Zeyuan Wallis, Yuanzhi Allen-Zhu, Shean Li, Lu Wang, Weizhu Wang, Chen","year":"2022","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jones et al., 1993] Donald R Jones, Cary D Perttunen, and Bruce E Stuckman. Lipschitzian optimization without the lipschitz constant. Journal of optimization Theory and Ap- plications, 1993.","title":"Lipschitzian optimization without the lipschitz constant","authors":"Cary D Donald R Jones, Bruce E Perttunen, Stuckman","year":"1993","venue":"Journal of optimization Theory and Applications","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Khandelwal et al., 2020] Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Gen- eralization through memorization: Nearest neighbor lan- guage models. In International Conference on Learning Representations, 2020.","title":"Generalization through memorization: Nearest neighbor language models","authors":"Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, Mike Lewis","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lee et al., 2024] Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, et al. Gecko: Versa- tile text embeddings distilled from large language models. arXiv preprint arXiv:2403.20327, 2024.","title":"Gecko: Versatile text embeddings distilled from large language models","authors":"Lee","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2021] Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu. Selfdoc: Self-supervised document rep- resentation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.","title":"Selfdoc: Self-supervised document representation learning","authors":"Peizhao Li, Jiuxiang Gu, Jason Kuen, Handong Vlad I Morariu, Rajiv Zhao, Varun Jain, Hongfu Manjunatha, Liu","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lin, 2004] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 2004.","title":"Rouge: A package for automatic evaluation of summaries","authors":"Chin-Yew Lin","year":"2004","venue":"Text summarization branches out","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Longo et al., 2024] Luca Longo, Mario Brcic, Federico Cabitza, Jaesik Choi, Roberto Confalonieri, Javier Del Ser, Riccardo Guidotti, Yoichi Hayashi, Francisco Herrera, Andreas Holzinger, et al. Explainable artificial intelligence (xai) 2.0: A manifesto of open challenges and interdisci- plinary research directions. Information Fusion, 2024.","title":"Explainable artificial intelligence (xai) 2.0: A manifesto of open challenges and interdisciplinary research directions","authors":"Luca Longo, Mario Brcic, Federico Cabitza, Jaesik Choi, Roberto Confalonieri, Javier Del Ser, Riccardo Guidotti, Yoichi Hayashi, Francisco Herrera, Andreas Holzinger","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Manakul et al., 2023] Potsawee Manakul, Adian Liusie, and Mark Gales. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language mod- els. In Proceedings of the Conference on Empirical Meth- ods in Natural Language Processing, 2023.","title":"SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models","authors":"Manakul","year":"2023","venue":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Meng et al., 2022a] Kevin Meng, David Bau, Alex Ando- nian, and Yonatan Belinkov. Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems, 2022.","title":"Locating and editing factual associations in gpt","authors":"Meng","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Meng et al., 2022b] Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In International Conference on Learning Representations, 2022.","title":"Mass-editing memory in a transformer","authors":"Meng","year":"2022","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Merity et al., 2017] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Repre- sentations, 2017.","title":"Pointer sentinel mixture models","authors":"Stephen Merity, Caiming Xiong, James Bradbury, Richard Socher","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Mitchell et al., 2023] Eric Mitchell, Yoonho Lee, Alexan- der Khazatsky, Christopher D Manning, and Chelsea Finn. Detectgpt: Zero-shot machine-generated text detection us- ing probability curvature. In International Conference on Machine Learning. PMLR, 2023.","title":"Detectgpt: Zero-shot machine-generated text detection using probability curvature","authors":"Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, Chelsea Finn","year":"2023","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Nguyen et al., 2023] Dung Nguyen, Phuoc Nguyen, Hung Le, Kien Do, Svetha Venkatesh, and Truyen Tran. Memory-augmented theory of mind network. In Proceed- ings of the AAAI Conference on Artificial Intelligence, 2023.","title":"Memory-augmented theory of mind network","authors":"Dung Nguyen, Phuoc Nguyen, Hung Le, Kien Do, Svetha Venkatesh, Truyen Tran","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Nichol et al., 2022] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Proceedings of the International Conference on Machine Learning. PMLR, 2022.","title":"GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models","authors":"Alexander Quinn, Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, Mark Chen","year":"2022","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Pan et al., 2020] Xudong Pan, Mi Zhang, Shouling Ji, and Min Yang. Privacy risks of general-purpose language models. In Proceedings of the IEEE Symposium on Se- curity and Privacy, 2020.","title":"Privacy risks of general-purpose language models","authors":"Xudong Pan, Mi Zhang, Shouling Ji, Min Yang","year":"2020","venue":"Proceedings of the IEEE Symposium on Security and Privacy","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Reimers and Gurevych, 2019] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the Conference on Empirical Methods in Natural Language Process- ing and the International Joint Conference on Natural Language Processing, 2019.","title":"Sentence-bert: Sentence embeddings using siamese bert-networks","authors":"Gurevych Reimers, Nils Reimers, Iryna Gurevych","year":"2019","venue":"Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sallam et al., 2023] Malik Sallam, Nesreen Salim, Muna Barakat, and Alaa Al-Tammemi. Chatgpt applications in medical, dental, pharmacy, and public health education: A descriptive study highlighting the advantages and limita- tions.","title":"Chatgpt applications in medical, dental, pharmacy, and public health education: A descriptive study highlighting the advantages and limitations","authors":"Malik Sallam, Nesreen Salim, Muna Barakat, Alaa Al-Tammemi","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"Narra J, 2023. [Sukhbaatar et al., 2015] Sainbayar Sukhbaatar, arthur szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In Advances in Neural Information Processing Systems, 2015.","title":"End-to-end memory networks","authors":"J ; Narra, Sukhbaatar","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Touvron et al., 2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.","title":"Llama 2: Open foundation and fine-tuned chat models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wang et al., 2022] Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. Finding skill neurons in pre-trained transformer-based language mod- els. In Proceedings of the Conference on Empirical Meth- ods in Natural Language Processing, 2022.","title":"Finding skill neurons in pre-trained transformer-based language models","authors":"Wang","year":"2022","venue":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wu et al., 2022] Yuhuai Wu, Markus Norman Rabe, DeLes- ley Hutchins, and Christian Szegedy. Memorizing trans- formers. In International Conference on Learning Repre- sentations, 2022.","title":"Memorizing transformers","authors":"Wu","year":"2022","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Xie et al., 2023] Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. Defending chatgpt against jailbreak attack via self- reminders. Nature Machine Intelligence, 2023.","title":"Defending chatgpt against jailbreak attack via selfreminders","authors":"Xie","year":"2023","venue":"Nature Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Xu et al., 2023] Frank F. Xu, Uri Alon, and Graham Neu- big. Why do nearest neighbor language models work? In Proceedings of the International Conference on Machine Learning. PMLR, 2023.","title":"Why do nearest neighbor language models work","authors":"F Frank, Uri Xu, Graham Alon, Neubig","year":"2023","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Yao et al., 2023] Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Prob- lems, methods, and opportunities. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2023.","title":"Editing large language models: Problems, methods, and opportunities","authors":"Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, Ningyu Zhang","year":"2023","venue":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Yu et al., 2023] Zhiyuan Yu, Yuhao Wu, Ning Zhang, Chen- guang Wang, Yevgeniy Vorobeychik, and Chaowei Xiao. Codeipprompt: intellectual property infringement assess- ment of code language models. In Proceedings of the International Conference on Machine Learning. PMLR, 2023.","title":"Codeipprompt: intellectual property infringement assessment of code language models","authors":"Zhiyuan Yu, Yuhao Wu, Ning Zhang, Chenguang Wang, Yevgeniy Vorobeychik, Chaowei Xiao","year":"2023","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}