{"type":"ScholarlyArticle","title":"On Privacy Protection of Latent Dirichlet Allocation Model Training","authors":[{"name":"Fangyuan Zhao","slug":"fangyuan-zhao","given":"Fangyuan","family":"Zhao","affiliation":"School of Computer Science and Technology , Xi'an Jiaotong University , China; National Engineering Laboratory for Big Data Analytics , Xi'an Jiaotong University , China; Ministry of Education Key Lab For Intelligent Networks and Network Security , Xi'an Jiaotong University , China"},{"name":"Xuebin Ren","slug":"xuebin-ren","given":"Xuebin","family":"Ren","affiliation":"School of Computer Science and Technology , Xi'an Jiaotong University , China; National Engineering Laboratory for Big Data Analytics , Xi'an Jiaotong University , China"},{"name":"Shusen Yang","slug":"shusen-yang","given":"Shusen","family":"Yang","affiliation":"National Engineering Laboratory for Big Data Analytics , Xi'an Jiaotong University , China; Ministry of Education Key Lab For Intelligent Networks and Network Security , Xi'an Jiaotong University , China"},{"name":"Xinyu Yang","slug":"xinyu-yang","given":"Xinyu","family":"Yang","affiliation":"School of Computer Science and Technology , Xi'an Jiaotong University , China; National Engineering Laboratory for Big Data Analytics , Xi'an Jiaotong University , China"}],"abstract":"Latent Dirichlet Allocation (LDA) is a popular topic modeling technique for discovery of hidden semantic architecture of text datasets, and plays a fundamental role in many machine learning applications. However, like many other machine learning algorithms, the process of training a LDA model may leak the sensitive information of the training datasets and bring significant privacy risks. To mitigate the privacy issues in LDA, we focus on studying privacy-preserving algorithms of LDA model training in this paper. In particular, we first develop a privacy monitoring algorithm to investigate the privacy guarantee obtained from the inherent randomness of the Collapsed Gibbs Sampling (CGS) process in a typical LDA training algorithm on centralized curated datasets. Then, we further propose a locally private LDA training algorithm on crowdsourced data to provide local differential privacy for individual data contributors. The experimental results on real-world datasets demonstrate the effectiveness of our proposed algorithms.","doi":"10.24963/ijcai.2019/675","url":"https://ijcai.org/proceedings/2019/675","pdf":"https://ijcai.org/proceedings/2019/0675.pdf","markdown":"https://ijcai.org/proceedings/2019/675.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/675","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Multidisciplinary Topics and Applications","track":"Main track","number":675,"pages":{"first":4860,"last":4866},"keywords":["Machine Learning: Learning Generative Models","Machine Learning: Probabilistic Machine Learning","Machine Learning: Recommender Systems","Multidisciplinary Topics and Applications: Security and Privacy"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martin Abadi, Andy Chu, Ian Goodfel- low, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 308-318. ACM, 2016.","title":"Deep learning with differential privacy","authors":"Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan Mcmahan, Ilya Mironov, Kunal Talwar, Li Zhang","year":"2016","venue":"Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blei et al., 2003] David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993-1022, 2003.","title":"Latent dirichlet allocation","authors":"Andrew Y David M Blei, Michael I Ng, Jordan","year":"2003","venue":"Journal of machine Learning research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chaudhuri et al., 2011] Kamalika Chaudhuri, Claire Mon- teleoni, and Anand D Sarwate. Differentially private em- pirical risk minimization. Journal of Machine Learning Research, 12(Mar):1069-1109, 2011.","title":"Differentially private empirical risk minimization","authors":"Kamalika Chaudhuri, Claire Monteleoni, Anand D Sarwate","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dwork et al., 2006] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sen- sitivity in private data analysis. In Theory of cryptography conference, pages 265-284. Springer, 2006.","title":"Calibrating noise to sensitivity in private data analysis","authors":"Cynthia Dwork, Frank Mcsherry, Kobbi Nissim, Adam Smith","year":"2006","venue":"Theory of cryptography conference","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dwork et al., 2014] Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Founda- tions and Trends R in Theoretical Computer Science, 9(3- 4):211-407, 2014.","title":"The algorithmic foundations of differential privacy","authors":"Cynthia Dwork, Aaron Roth","year":"2014","venue":"Foundations and Trends R in Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Foulds et al., 2016] James Foulds, Joseph Geumlek, Max Welling, and Kamalika Chaudhuri. On the theory and practice of privacy-preserving bayesian data analysis. arXiv preprint arXiv:1603.07294, 2016.","title":"On the theory and practice of privacy-preserving bayesian data analysis","authors":"James Foulds, Joseph Geumlek, Max Welling, Kamalika Chaudhuri","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fredrikson et al., 2014] Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon Lin, David Page, and Thomas Ris- tenpart. Privacy in pharmacogenetics: An end-to-end case study of personalized warfarin dosing. In 23rd {USENIX} Security Symposium ({USENIX} Security 14), pages 17- 32, 2014.","title":"Privacy in pharmacogenetics: An end-to-end case study of personalized warfarin dosing","authors":"Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon Lin, David Page, Thomas Ristenpart","year":"2014","venue":"23rd {USENIX} Security Symposium ({USENIX} Security 14)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Heinrich, 2005] Gregor Heinrich. Parameter estimation for text analysis. Technical report, Technical report, 2005.","title":"Parameter estimation for text analysis","authors":"Gregor Heinrich","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2016] Ninghui Li, Min Lyu, Dong Su, and Wein- ing Yang. Differential privacy: From theory to practice. Synthesis Lectures on Information Security, Privacy, & Trust, 8(4):1-138, 2016.","title":"Differential privacy: From theory to practice","authors":"Li","year":"2016","venue":"Synthesis Lectures on Information Security, Privacy, & Trust","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2019] Yanan Li, Xuebin Ren, Shusen Yang, and Xinyu Yang. Impact of prior knowledge and data correla- tion on privacy leakage: A unified analysis. IEEE Trans- actions on Information Forensics and Security, 2019.","title":"Impact of prior knowledge and data correlation on privacy leakage: A unified analysis","authors":"Yanan Li, Xuebin Ren, Shusen Yang, Xinyu Yang","year":"2019","venue":"IEEE Transactions on Information Forensics and Security","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Park et al., 2016] Mijung Park, James Foulds, Kamalika Chaudhuri, and Max Welling. Variational bayes in private settings (vips). arXiv preprint arXiv:1611.00340, 2016.","title":"Variational bayes in private settings","authors":"Park","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ren et al., 2018] Xuebin Ren, Chia-Mu Yu, Weiren Yu, Shusen Yang, Xinyu Yang, Julie A McCann, and S Yu Philip. Lopub: High-dimensional crowdsourced data pub- lication with local differential privacy. IEEE Transactions on Information Forensics and Security, 13(9):2151-2166, 2018.","title":"Lopub: High-dimensional crowdsourced data publication with local differential privacy","authors":"Ren","year":"2018","venue":"IEEE Transactions on Information Forensics and Security","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Shokri et al., 2017] Reza Shokri, Marco Stronati, Con- gzheng Song, and Vitaly Shmatikov. Membership infer- ence attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pages 3- 18. IEEE, 2017.","title":"Membership inference attacks against machine learning models","authors":"Reza Shokri, Marco Stronati, Congzheng Song, Vitaly Shmatikov","year":"2017","venue":"IEEE Symposium on Security and Privacy (SP)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Smola and Narayanamurthy, 2010] Alexander Smola and Shravan Narayanamurthy. An architecture for parallel topic models. Proceedings of the VLDB Endowment, 3(1- 2):703-710, 2010.","title":"An architecture for parallel topic models","authors":"Narayanamurthy Smola, Alexander Smola, Shravan Narayanamurthy","year":"2010","venue":"Proceedings of the VLDB Endowment","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Wang et al., 2014] Yi Wang, Xuemin Zhao, Zhenlong Sun, Hao Yan, Lifeng Wang, Zhihui Jin, Liubin Wang, Yang Gao, Jia Zeng, Qiang Yang, et al. Towards topic modeling for big data. arXiv preprint arXiv:1405.4402, 2014.","title":"Towards topic modeling for big data","authors":"Yi Wang, Xuemin Zhao, Zhenlong Sun, Hao Yan, Lifeng Wang, Zhihui Jin, Liubin Wang, Yang Gao, Jia Zeng, Qiang Yang","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wang et al., 2015] Yu-Xiang Wang, Stephen E Fienberg, and Alexander J Smola. Privacy for free: Posterior sam- pling and stochastic gradient monte carlo. In ICML, vol- ume 15, pages 2493-2502, 2015.","title":"Privacy for free: Posterior sampling and stochastic gradient monte carlo","authors":"Yu-Xiang Wang, Stephen E Fienberg, Alexander J Smola","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Yuan et al., 2015] Jinhui Yuan, Fei Gao, Qirong Ho, Wei Dai, Jinliang Wei, Xun Zheng, Eric Po Xing, Tie-Yan Liu, and Wei-Ying Ma. Lightlda: Big topic models on modest computer clusters. In Proceedings of the 24th In- ternational Conference on World Wide Web, pages 1351- 1361. International World Wide Web Conferences Steer- ing Committee, 2015.","title":"Lightlda: Big topic models on modest computer clusters","authors":"Jinhui Yuan, Fei Gao, Qirong Ho, Wei Dai, Jinliang Wei, Xun Zheng, Eric Po Xing, Tie-Yan Liu, Wei-Ying Ma","year":"2015","venue":"Proceedings of the 24th International Conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Yut et al., 2017] Lele Yut, Ce Zhang, Yingxia Shao, and Bin Cui. Lda*: a robust and large-scale topic modeling system. Proceedings of the VLDB Endowment, 10(11):1406-1417, 2017.","title":"Lda*: a robust and large-scale topic modeling system","authors":"Lele Yut, Ce Zhang, Yingxia Shao, Bin Cui","year":"2017","venue":"Proceedings of the VLDB Endowment","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zhao et al., 2019] Fangyuan Zhao, Xuebin Ren, Shusen Yang, and Xinyu Yang. On privacy pro- tection of latent dirichlet allocation model train- ing. full version of this paper, available online at https://arxiv.org/pdf/1906.01178v2.pdf, 2019.","title":"On privacy protection of latent dirichlet allocation model training","authors":"Fangyuan Zhao, Xuebin Ren, Shusen Yang, Xinyu Yang","year":"2019","venue":null,"doi":null,"url":"https://arxiv.org/pdf/1906.01178v2.pdf","internal_url":null},{"position":20,"raw":"[Zhu et al., 2016] Tianqing Zhu, Gang Li, Wanlei Zhou, Ping Xiong, and Cao Yuan. Privacy-preserving topic model for tagging recommender systems. Knowledge and information systems, 46(1):33-58, 2016.","title":"Privacy-preserving topic model for tagging recommender systems","authors":"Tianqing Zhu, Gang Li, Wanlei Zhou, Ping Xiong, Cao Yuan","year":"2016","venue":"Knowledge and information systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}