{"type":"ScholarlyArticle","title":"Generate Synthetic Text Approximating the Private Distribution with Differential Privacy","authors":[{"name":"Wenhao Zhao","slug":"wenhao-zhao","given":"Wenhao","family":"Zhao","affiliation":"Computer Science Dept , Renmin University of China , Beijing , CHINA"},{"name":"Shaoyang Song","slug":"shaoyang-song","given":"Shaoyang","family":"Song","affiliation":"Computer Science Dept , Renmin University of China , Beijing , CHINA"},{"name":"Chunlai Zhou","slug":"chunlai-zhou","given":"Chunlai","family":"Zhou","affiliation":"Computer Science Dept , Renmin University of China , Beijing , CHINA"}],"abstract":"Due to the potential leakage of sensitive information in text, there is a societal call for feeding privacy-preserving text to model training. Recently, a lot of work showed that using synthetic text with differential privacy, rather than private text, can provide a strong privacy protection. However, achieving higher semantic similarity between synthetic and private text has not been thoroughly investigated. In this paper, we propose an approach that combines the iteratively optimized mindset from genetic algorithms to align the distribution of synthetic text with that of private text. Furthermore, not only does the final synthetic text meet the requirements of privacy protection, but also has a high level of quality. Through comparisons with various baselines on different datasets, we demonstrate that our synthetic text can closely match the utility of private text, while providing privacy protection standards robust enough to resist membership inference attacks from malicious users.","doi":"10.24963/ijcai.2024/735","url":"https://ijcai.org/proceedings/2024/735","pdf":"https://ijcai.org/proceedings/2024/0735.pdf","markdown":"https://ijcai.org/proceedings/2024/735.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/735","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Natural Language Processing","track":"Main Track","number":735,"pages":{"first":6651,"last":6659},"keywords":["Natural Language Processing: NLP: Language models","Multidisciplinary Topics and Applications: MTA: Security and privacy"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martin Abadi, Andy Chu, Ian Goodfel- low, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on com- puter and communications security, pages 308-318, 2016.","title":"Deep learning with differential privacy","authors":"Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan Mcmahan, Ilya Mironov, Kunal Talwar, Li Zhang","year":"2016","venue":"Proceedings of the 2016 ACM SIGSAC conference on computer and communications security","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anil et al., 2021] Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. Large-scale differen- tially private bert. arXiv preprint arXiv:2108.01624, 2021.","title":"Large-scale differentially private bert","authors":"Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, Pasin Manurangsi","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bansal et al., 2019] Trapit Bansal, Rishikesh Jha, and An- drew McCallum. Learning to few-shot learn across di- verse natural language classification tasks. arXiv preprint arXiv:1911.03863, 2019.","title":"Learning to few-shot learn across diverse natural language classification tasks","authors":"Trapit Bansal, Rishikesh Jha, Andrew Mccallum","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing sys- tems, 33:1877-1901, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Carlini et al., 2021] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ul- far Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650, 2021.","title":"Extracting training data from large language models","authors":"Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson","year":"2021","venue":"30th USENIX Security Symposium (USENIX Security 21)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2023] Sai Chen, Fengran Mo, Yanhao Wang, Cen Chen, Jian-Yun Nie, Chengyu Wang, and Jamie Cui. A customized text sanitization mechanism with differen- tial privacy. In Findings of the Association for Computa- tional Linguistics: ACL 2023, pages 5747-5758, 2023.","title":"A customized text sanitization mechanism with differential privacy","authors":"Sai Chen, Fengran Mo, Yanhao Wang, Cen Chen, Jian-Yun Nie, Chengyu Wang, Jamie Cui","year":"2023","venue":"Findings of the Association for Computational Linguistics: ACL 2023","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Du et al., 2023a] Minxin Du, Xiang Yue, Sherman SM Chow, and Huan Sun. Sanitizing sentence embeddings (and labels) for local differential privacy. In Proceedings of the ACM Web Conference 2023, pages 2349-2359, 2023.","title":"Sanitizing sentence embeddings (and labels) for local differential privacy","authors":"Du","year":"2023","venue":"Proceedings of the ACM Web Conference 2023","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Du et al., 2023b] Minxin Du, Xiang Yue, Sherman SM Chow, Tianhao Wang, Chenyu Huang, and Huan Sun. Dp- forward: Fine-tuning and inference on language models with differential privacy in forward pass. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 2665-2679, 2023.","title":"Dpforward: Fine-tuning and inference on language models with differential privacy in forward pass","authors":"Du","year":"2023","venue":"Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Duan et al., 2023] Haonan Duan, Adam Dziedzic, Nicolas Papernot, and Franziska Boenisch. Flocks of stochastic parrots: Differentially private prompt learning for large language models. arXiv preprint arXiv:2305.15594, 2023.","title":"Flocks of stochastic parrots: Differentially private prompt learning for large language models","authors":"Duan","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Durfee and Rogers, 2019] David Durfee and Ryan M Rogers. Practical differentially private top-k selection with pay-what-you-get composition. Advances in Neural Infor- mation Processing Systems, 32, 2019.","title":"Practical differentially private top-k selection with pay-what-you-get composition","authors":"David Durfee, Ryan M Rogers","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Dwork et al., 2006] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptog- raphy: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pages 265-284. Springer, 2006.","title":"Calibrating noise to sensitivity in private data analysis","authors":"Cynthia Dwork, Frank Mcsherry, Kobbi Nissim, Adam Smith","year":"2006","venue":"Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Dwork et al., 2010] Cynthia Dwork, Guy N Rothblum, and Salil Vadhan. Boosting and differential privacy. In 2010 IEEE 51st Annual Symposium on Foundations of Com- puter Science, pages 51-60. IEEE, 2010.","title":"Boosting and differential privacy","authors":"Cynthia Dwork, Guy N Rothblum, Salil Vadhan","year":"2010","venue":"IEEE 51st Annual Symposium on Foundations of Computer Science","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gonen et al., 2022] Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke Zettlemoyer. Demystifying prompts in language models via perplexity estimation. arXiv preprint arXiv:2212.04037, 2022.","title":"Demystifying prompts in language models via perplexity estimation","authors":"Srini Hila Gonen, Terra Iyer, Noah A Blevins, Luke Smith, Zettlemoyer","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2023] Haoran Li, Mingshi Xu, and Yangqiu Song. Sentence embedding leaks more information than you ex- pect: Generative embedding inversion attack to recover the whole sentence. arXiv preprint arXiv:2305.03010, 2023.","title":"Sentence embedding leaks more information than you expect: Generative embedding inversion attack to recover the whole sentence","authors":"Haoran Li, Mingshi Xu, Yangqiu Song","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2023] Zinan Lin, Sivakanth Gopi, Janardhan Kulkarni, Harsha Nori, and Sergey Yekhanin. Differen- tially private synthetic data via foundation model apis 1: Images. arXiv preprint arXiv:2305.15560, 2023.","title":"Differentially private synthetic data via foundation model apis 1: Images","authors":"Lin","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[McSherry and Talwar, 2007] Frank McSherry and Kunal Talwar. Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Sci- ence (FOCS'07), pages 94-103. IEEE, 2007.","title":"Mechanism design via differential privacy","authors":"Talwar Mcsherry, Frank Mcsherry, Kunal Talwar","year":"2007","venue":"48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Meehan et al., 2022] Casey Meehan, Khalil Mrini, and Ka- malika Chaudhuri. Sentence-level privacy for document embeddings. arXiv preprint arXiv:2205.04605, 2022.","title":"Sentence-level privacy for document embeddings","authors":"Casey Meehan, Khalil Mrini, Kamalika Chaudhuri","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Min et al., 2022] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837, 2022.","title":"Rethinking the role of demonstrations: What makes in-context learning work","authors":"Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mitchell and Taylor, 1999] Melanie Mitchell and Charles E Taylor. Evolutionary computation: an overview. Annual Review of Ecology and Systematics, 30(1):593-616, 1999.","title":"Evolutionary computation: an overview","authors":"Melanie Mitchell, Charles E Taylor","year":"1999","venue":"Annual Review of Ecology and Systematics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Morris et al., 2023] John X Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander M Rush. Text embed- dings reveal (almost) as much as text. arXiv preprint arXiv:2310.06816, 2023.","title":"Text embeddings reveal (almost) as much as text","authors":"Volodymyr John X Morris, Vitaly Kuleshov, Alexander M Shmatikov, Rush","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Narayanan and Shmatikov, 2008] Arvind Narayanan and Vitaly Shmatikov. Robust de-anonymization of large sparse datasets. In 2008 IEEE Symposium on Security and Privacy (sp 2008), pages 111-125. IEEE, 2008.","title":"Robust de-anonymization of large sparse datasets","authors":"Arvind Narayanan, Vitaly Shmatikov","year":"2008","venue":"IEEE Symposium on Security and Privacy","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ni et al., 2021] Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming-Wei Chang, et al. Large dual encoders are generalizable retrievers. arXiv preprint arXiv:2112.07899, 2021.","title":"Large dual encoders are generalizable retrievers","authors":"Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Y Vincent, Yi Zhao, Keith B Luan, Ming-Wei Hall, Chang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pan et al., 2020] Xudong Pan, Mi Zhang, Shouling Ji, and Min Yang. Privacy risks of general-purpose language models. In 2020 IEEE Symposium on Security and Pri- vacy (SP), pages 1314-1331. IEEE, 2020.","title":"Privacy risks of general-purpose language models","authors":"Xudong Pan, Mi Zhang, Shouling Ji, Min Yang","year":"2020","venue":"IEEE Symposium on Security and Privacy (SP)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Papernot et al., 2016] Nicolas Papernot, Martín Abadi, Ul- far Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi- supervised knowledge transfer for deep learning from pri- vate training data. arXiv preprint arXiv:1610.05755, 2016.","title":"Semisupervised knowledge transfer for deep learning from private training data","authors":"Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, Kunal Talwar","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Pilán et al., 2022] Ildikó Pilán, Pierre Lison, Lilja Øvre- lid, Anthi Papadopoulou, David Sánchez, and Montser- rat Batet. The text anonymization benchmark (tab): A dedicated corpus and evaluation framework for text anonymization. Computational Linguistics, 48(4):1053- 1101, 2022.","title":"The text anonymization benchmark (tab): A dedicated corpus and evaluation framework for text anonymization","authors":"Ildikó Pilán, Pierre Lison, Lilja Øvrelid, Anthi Papadopoulou, David Sánchez, Montserrat Batet","year":"2022","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sampson, 1976] Jeffrey R Sampson. Adaptation in natural and artificial systems (john h. holland), 1976.","title":"Adaptation in natural and artificial systems (john h. holland","authors":"Jeffrey R Sampson","year":"1976","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Santambrogio, 2015] Filippo Santambrogio. Optimal trans- port for applied mathematicians. Birkäuser, NY, 55(58- 63):94, 2015.","title":"Optimal transport for applied mathematicians","authors":"Filippo Santambrogio","year":"2015","venue":"Birkäuser","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Shin et al., 2022] Seongjin Shin, Sang-Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung- Woo Ha, et al. On the effect of pretraining corpora on in- context learning by a large-scale language model. arXiv preprint arXiv:2204.13509, 2022.","title":"On the effect of pretraining corpora on incontext learning by a large-scale language model","authors":"Shin","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Shokri et al., 2017] Reza Shokri, Marco Stronati, Con- gzheng Song, and Vitaly Shmatikov. Membership infer- ence attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3- 18. IEEE, 2017.","title":"Membership inference attacks against machine learning models","authors":"Reza Shokri, Marco Stronati, Congzheng Song, Vitaly Shmatikov","year":"2017","venue":"IEEE symposium on security and privacy (SP)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Song and Raghunathan, 2020] Congzheng Song and Ananth Raghunathan. Information leakage in embedding models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pages 377-390, 2020.","title":"Congzheng Song and Ananth Raghunathan. Information leakage in embedding models","authors":"Raghunathan Song","year":"2020","venue":"Proceedings of the 2020 ACM SIGSAC conference on computer and communications security","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Tang et al., 2023] Xinyu Tang, Richard Shin, Huseyin A Inan, Andre Manoel, Fatemehsadat Mireshghallah, Zi- nan Lin, Sivakanth Gopi, Janardhan Kulkarni, and Robert Sim. Privacy-preserving in-context learning with dif- ferentially private few-shot generation. arXiv preprint arXiv:2309.11765, 2023.","title":"Privacy-preserving in-context learning with differentially private few-shot generation","authors":"Xinyu Tang, Richard Shin, Andre Huseyin A Inan, Fatemehsadat Manoel, Zinan Mireshghallah, Sivakanth Lin, Janardhan Gopi, Robert Kulkarni, Sim","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Tukey, 1975] John W Tukey. Mathematics and the picturing of data. In Proceedings of the International Congress of Mathematicians, Vancouver, 1975, volume 2, pages 523- 531, 1975.","title":"Mathematics and the picturing of data","authors":"John W Tukey","year":"1975","venue":"Proceedings of the International Congress of Mathematicians","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Voorhees and Tice, 2000] Ellen M Voorhees and Dawn M Tice. Building a question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 200-207, 2000.","title":"Building a question answering test collection","authors":"M Ellen, Dawn M Voorhees, Tice","year":"2000","venue":"Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wang et al., 2020] Xiaofei Wang, Tong Han, and Hui Zhao. An estimation of distribution algorithm with multi-leader search. IEEE Access, 8:37383-37405, 2020.","title":"An estimation of distribution algorithm with multi-leader search","authors":"Xiaofei Wang, Tong Han, Hui Zhao","year":"2020","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Welleck et al., 2022] Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self- correct. arXiv preprint arXiv:2211.00053, 2022.","title":"Generating sequences by learning to selfcorrect","authors":"Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, Yejin Choi","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wu et al., 2023] Tong Wu, Ashwinee Panda, Jiachen T Wang, and Prateek Mittal. Privacy-preserving in-context learning for large language models. arXiv e-prints, pages arXiv-2305, 2023.","title":"Privacy-preserving in-context learning for large language models","authors":"Tong Wu, Ashwinee Panda, Jiachen T Wang, Prateek Mittal","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Yue et al., 2021] Xiang Yue, Minxin Du, Tianhao Wang, Yaliang Li, Huan Sun, and Sherman SM Chow. Differen- tial privacy for text analytics via natural text sanitization. arXiv preprint arXiv:2106.01221, 2021.","title":"Differential privacy for text analytics via natural text sanitization","authors":"Xiang Yue, Minxin Du, Tianhao Wang, Yaliang Li, Huan Sun, Sherman Sm Chow","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Yue et al., 2022] Xiang Yue, Huseyin A Inan, Xuechen Li, Girish Kumar, Julia McAnallen, Hoda Shajari, Huan Sun, David Levitan, and Robert Sim. Synthetic text generation with differential privacy: A simple and practical recipe. arXiv preprint arXiv:2210.14348, 2022.","title":"Synthetic text generation with differential privacy: A simple and practical recipe","authors":"Xiang Yue, Xuechen Huseyin A Inan, Girish Li, Julia Kumar, Hoda Mcanallen, Huan Shajari, David Sun, Robert Levitan, Sim","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhang et al., 2015] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.","title":"Character-level convolutional networks for text classification","authors":"Xiang Zhang, Junbo Zhao, Yann Lecun","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}