{"type":"ScholarlyArticle","title":"Variational Bayes in Private Settings (VIPS) (Extended Abstract)","authors":[{"name":"James R. Foulds","slug":"james-r-foulds","given":"James R.","family":"Foulds","affiliation":"Department of Information Systems , University of Maryland , Baltimore County"},{"name":"Mijung Park","slug":"mijung-park","given":"Mijung","family":"Park","affiliation":"Max Planck Institute for Intelligent Systems; Department of Computer Science , University of Tübingen"},{"name":"Kamalika Chaudhuri","slug":"kamalika-chaudhuri","given":"Kamalika","family":"Chaudhuri","affiliation":"Department of Computer Science , University of California , San Diego"},{"name":"Max Welling","slug":"max-welling","given":"Max","family":"Welling","affiliation":"Amsterdam Machine Learning LAB (AMLAB) , Informatics Institute , University of Amsterdam"}],"abstract":"Many applications of Bayesian data analysis involve sensitive information such as personal documents or medical records, motivating methods which ensure that privacy is protected. We introduce a general privacy-preserving framework for Variational Bayes (VB), a widely used optimization-based Bayesian inference method. Our framework respects differential privacy, the gold-standard privacy criterion. The iterative nature of variational Bayes presents a challenge since iterations increase the amount of noise needed to ensure privacy. We overcome this by combining: (1) an improved composition method, called the moments accountant, and (2) the privacy amplification effect of subsampling mini-batches from large-scale data in stochastic learning.   We empirically demonstrate the effectiveness of our method on LDA topic models, evaluated on Wikipedia.  In the full paper we extend our method to a broad class of models, including Bayesian logistic regression and sigmoid belief networks.","doi":"10.24963/ijcai.2020/705","url":"https://ijcai.org/proceedings/2020/705","pdf":"https://ijcai.org/proceedings/2020/0705.pdf","markdown":"https://ijcai.org/proceedings/2020/705.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/705","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Journal track","track":"Journal track","number":705,"pages":{"first":5050,"last":5054},"keywords":["Multidisciplinary Topics and Applications: Security and Privacy","Uncertainty in AI: Approximate Probabilistic Inference","Machine Learning: Probabilistic Machine Learning","Natural Language Processing: Natural Language Processing"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25566/file_25566.m3u8","length_sec":295,"provider":"hls"}],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martin Abadi, Andy Chu, Ian Goodfel- low, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 308-318. ACM, 2016.","title":"Deep learning with differential privacy","authors":"Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan Mcmahan, Ilya Mironov, Kunal Talwar, Li Zhang","year":"2016","venue":"Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Beal, 2003] Matthew J. Beal. Variational Algorithms for Approximate Bayesian Inference. PhD thesis, Gatsby Unit, University College London, 2003.","title":"Variational Algorithms for Approximate Bayesian Inference","authors":"J Matthew, Beal","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blei et al., 2003] David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 3(Jan):993-1022, 2003.","title":"Latent Dirichlet allocation","authors":"David M Blei, Andrew Y Ng, Michael I Jordan","year":"2003","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Daries et al., 2014] Jon P. Daries, Justin Reich, Jim Waldo, Elise M. Young, Jonathan Whittinghill, Andrew Dean Ho, Daniel Thomas Seaton, and Isaac Chuang. Privacy, anonymity, and big data in the social sciences. Communi- cations of the ACM, 57(9):56-63, 2014.","title":"Privacy, anonymity, and big data in the social sciences","authors":"Jon P Daries, Justin Reich, Jim Waldo, Elise M Young, Jonathan Whittinghill, Andrew Dean Ho, Daniel Thomas Seaton, Isaac Chuang","year":"2014","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dwork and Roth, 2014] Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 9:211-407, August 2014.","title":"The algorithmic foundations of differential privacy","authors":"Roth ; Dwork, Cynthia Dwork, Aaron Roth","year":"2014","venue":"Found. Trends Theor. Comput. Sci","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dwork et al., 2006a] Cynthia Dwork, Krishnaram Kentha- padi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Annual International Conference on the Theory and Ap- plications of Cryptographic Techniques, pages 486-503. Springer, 2006.","title":"Our data, ourselves: Privacy via distributed noise generation","authors":"Dwork","year":"2006","venue":"Annual International Conference on the Theory and Applications of Cryptographic Techniques","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dwork et al., 2006b] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sen- sitivity in private data analysis. In Theory of Cryptography Conference, pages 265-284. Springer, 2006.","title":"Calibrating noise to sensitivity in private data analysis","authors":"Dwork","year":"2006","venue":"Theory of Cryptography Conference","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hoffman et al., 2010] Matthew Hoffman, Francis R. Bach, and David M. Blei. Online learning for latent Dirichlet allocation. In Advances in Neural Information Processing Systems 23, pages 856-864.","title":"Online learning for latent Dirichlet allocation","authors":"Matthew Hoffman, Francis R Bach, David M Blei","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Curran Associates, Inc., 2010.","title":null,"authors":"Curran Associates, Inc","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Husmeier et al., 2006] Dirk Husmeier, Richard Dybowski, and Stephen Roberts. Probabilistic modeling in bioinfor- matics and medical informatics. Springer Science & Busi- ness Media, 2006.","title":"Probabilistic modeling in bioinformatics and medical informatics","authors":"Husmeier","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Park et al., 2020] Mijung Park, James R. Foulds, Kamalika Chaudhuri, and Max Welling. Variational Bayes in private settings (VIPS). Journal of Artificial Intelligence Research (JAIR), 68:109-157, 2020.","title":"Variational Bayes in private settings (VIPS)","authors":"Park","year":"2020","venue":"Journal of Artificial Intelligence Research (JAIR)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Piech et al., 2013] Chris Piech, Jonathan Huang, Zhenghao Chen, Chuong Do, Andrew Ng, and Daphne Koller. Tuned models of peer assessment in MOOCs. In Proceedings of the 6th International Conference on Educational Data Mining, pages 153-160, 2013.","title":"Tuned models of peer assessment in MOOCs","authors":"Chris Piech, Jonathan Huang, Zhenghao Chen, Chuong Do, Andrew Ng, Daphne Koller","year":"2013","venue":"Proceedings of the 6th International Conference on Educational Data Mining","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Sarwate and Chaudhuri, 2013] Anand D. Sarwate and Ka- malika Chaudhuri. Signal processing and machine learn- ing with differential privacy: Algorithms and challenges for continuous data. IEEE Signal Process. Mag., 30(5):86- 94, 2013.","title":"Signal processing and machine learning with differential privacy: Algorithms and challenges for continuous data","authors":"D Anand, Kamalika Sarwate, Chaudhuri","year":"2013","venue":"IEEE Signal Process. Mag","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Wang et al., 2019] Yu-Xiang Wang, Borja Balle, and Shiva Prasad Kasiviswanathan. Subsampled Renyi dif- ferential privacy and analytical moments accountant. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Pro- ceedings of the 22nd International Conference on Ar- tificial Intelligence and Statistics (AISTATS) 2019, vol- ume 89 of Proceedings of Machine Learning Research, pages 1226-1235, 2019.","title":"Subsampled Renyi differential privacy and analytical moments accountant","authors":"Yu-Xiang Wang, Borja Balle, Shiva Prasad Kasiviswanathan","year":"2019","venue":"Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS) 2019","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}