{"type":"ScholarlyArticle","title":"Data Poisoning against Differentially-Private Learners: Attacks and Defenses","authors":[{"name":"Yuzhe Ma","slug":"yuzhe-ma","given":"Yuzhe","family":"Ma","affiliation":"University of Wisconsin-Madison"},{"name":"Xiaojin Zhu","slug":"xiaojin-zhu","given":"Xiaojin","family":"Zhu","affiliation":"University of Wisconsin-Madison"},{"name":"Justin Hsu","slug":"justin-hsu","given":"Justin","family":"Hsu","affiliation":"University of Wisconsin-Madison"}],"abstract":"Data poisoning attacks aim to manipulate the model produced by a learning algorithm by adversarially modifying the training set. We consider differential privacy as a defensive measure against this type of attack. We show that private learners are resistant to data poisoning attacks when the adversary is only able to poison a small number of items. However, this protection degrades as the adversary is allowed to poison more data. We emprically evaluate this protection by designing attack algorithms targeting objective and output perturbation learners, two standard approaches to differentially-private machine learning. Experiments show that our methods are effective when the attacker is allowed to poison sufficiently many training items.","doi":"10.24963/ijcai.2019/657","url":"https://ijcai.org/proceedings/2019/657","pdf":"https://ijcai.org/proceedings/2019/0657.pdf","markdown":"https://ijcai.org/proceedings/2019/657.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/657","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Multidisciplinary Topics and Applications","track":"Main track","number":657,"pages":{"first":4732,"last":4738},"keywords":["Multidisciplinary Topics and Applications: Security and Privacy","Machine Learning: Adversarial Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Alfeld et al., 2016] Scott Alfeld, Xiaojin Zhu, and Paul Bar- ford. Data poisoning attacks against autoregressive models. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.","title":"Data poisoning attacks against autoregressive models","authors":"Scott Alfeld, Xiaojin Zhu, Paul Barford","year":"2016","venue":"Thirtieth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Biggio et al., 2012] Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. arXiv preprint arXiv:1206.6389, 2012.","title":"Poisoning attacks against support vector machines","authors":"Battista Biggio, Blaine Nelson, Pavel Laskov","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chaudhuri et al., 2011] Kamalika Chaudhuri, Claire Mon- teleoni, and Anand D Sarwate. Differentially private em- pirical risk minimization. Journal of Machine Learning Research, 12(Mar):1069-1109, 2011.","title":"Differentially private empirical risk minimization","authors":"Kamalika Chaudhuri, Claire Monteleoni, Anand D Sarwate","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dua and Karra Taniskidou, 2017] Dheeru Dua and Efi Karra Taniskidou. UCI machine learning repository, 2017.","title":"Dheeru Dua and Efi Karra Taniskidou. UCI machine learning repository","authors":"Karra Dua, Taniskidou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dwork et al., 2006] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pages 265-284. Springer, 2006.","title":"Calibrating noise to sensitivity in private data analysis","authors":"Cynthia Dwork, Frank Mcsherry, Kobbi Nissim, Adam Smith","year":"2006","venue":"Theory of cryptography conference","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dwork, 2011] Cynthia Dwork. Differential privacy. Ency- clopedia of Cryptography and Security, pages 338-340, 2011.","title":"Differential privacy","authors":"Cynthia Dwork","year":"2011","venue":"Encyclopedia of Cryptography and Security","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fredrikson et al., 2015] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Pro- ceedings of the 22nd ACM SIGSAC Conference on Com- puter and Communications Security, pages 1322-1333. ACM, 2015.","title":"Model inversion attacks that exploit confidence information and basic countermeasures","authors":"Matt Fredrikson, Somesh Jha, Thomas Ristenpart","year":"2015","venue":"Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jun et al., 2018] Kwang-Sung Jun, Lihong Li, Yuzhe Ma, and Jerry Zhu. Adversarial attacks on stochastic bandits. In Advances in Neural Information Processing Systems, pages 3640-3649, 2018.","title":"Adversarial attacks on stochastic bandits","authors":"Kwang-Sung Jun, Lihong Li, Yuzhe Ma, Jerry Zhu","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kifer et al., 2012] Daniel Kifer, Adam Smith, and Abhradeep Thakurta. Private convex empirical risk mini- mization and high-dimensional regression. In Conference on Learning Theory, pages 25-1, 2012.","title":"Private convex empirical risk minimization and high-dimensional regression","authors":"Daniel Kifer, Adam Smith, Abhradeep Thakurta","year":"2012","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Koh et al., 2018] Pang Wei Koh, Jacob Steinhardt, and Percy Liang. Stronger data poisoning attacks break data sanitiza- tion defenses. arXiv preprint arXiv:1811.00741, 2018.","title":"Stronger data poisoning attacks break data sanitization defenses","authors":"Pang Wei Koh, Jacob Steinhardt, Percy Liang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lecuyer et al., 2018] Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certi- fied robustness to adversarial examples with differential privacy. In Certified Robustness to Adversarial Examples with Differential Privacy. IEEE, 2018.","title":"Certified robustness to adversarial examples with differential privacy","authors":"Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, Suman Jana","year":"2018","venue":"Certified Robustness to Adversarial Examples with Differential Privacy","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2016] Bo Li, Yining Wang, Aarti Singh, and Yev- geniy Vorobeychik. Data poisoning attacks on factorization- based collaborative filtering. In Advances in Neural In- formation Processing Systems (NIPS), pages 1885-1893, 2016.","title":"Data poisoning attacks on factorizationbased collaborative filtering","authors":"Bo Li, Yining Wang, Aarti Singh, Yevgeniy Vorobeychik","year":"2016","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mei and Zhu, 2015] Shike Mei and Xiaojin Zhu. Using ma- chine teaching to identify optimal training-set attacks on machine learners. In The 29th AAAI Conference on Artifi- cial Intelligence, 2015.","title":"Using machine teaching to identify optimal training-set attacks on machine learners","authors":"Zhu ; ] Shike Mei, Xiaojin Mei, Zhu","year":"2015","venue":"The 29th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Muñoz-González et al., 2017] Luis Muñoz-González, Bat- tista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back-gradient optimization. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 27-38. ACM, 2017.","title":"Towards poisoning of deep learning algorithms with back-gradient optimization","authors":"Luis Muñoz-González, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C Lupu, Fabio Roli","year":"2017","venue":"Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nozari et al., 2016] Erfan Nozari, Pavankumar Tallapragada, and Jorge Cortés. Differentially private distributed convex optimization via objective perturbation. In 2016 American Control Conference (ACC), pages 2061-2066. IEEE, 2016.","title":"Differentially private distributed convex optimization via objective perturbation","authors":"Erfan Nozari, Pavankumar Tallapragada, Jorge Cortés","year":"2016","venue":"2016 American Control Conference (ACC)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Raghunathan et al., 2018] Aditi Raghunathan, Jacob Stein- hardt, and Percy Liang. Certified defenses against adversar- ial examples. arXiv preprint arXiv:1801.09344, 2018.","title":"Certified defenses against adversarial examples","authors":"Aditi Raghunathan, Jacob Steinhardt, Percy Liang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Steinhardt et al., 2017] Jacob Steinhardt, Pang Wei W Koh, and Percy S Liang. Certified defenses for data poisoning attacks. In Advances in Neural Information Processing Systems (NIPS), pages 3517-3529, 2017.","title":"Certified defenses for data poisoning attacks","authors":"Jacob Steinhardt, Pang Wei W Koh, Percy S Liang","year":"2017","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Szegedy et al., 2013] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.","title":"Intriguing properties of neural networks","authors":"Szegedy","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tramèr et al., 2016] Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing ma- chine learning models via prediction apis. In 25th USENIX Security Symposium, pages 601-618, 2016.","title":"Stealing machine learning models via prediction apis","authors":"Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, Thomas Ristenpart","year":"2016","venue":"25th USENIX Security Symposium","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xiao et al., 2015] Huang Xiao, Battista Biggio, Gavin Brown, Giorgio Fumera, Claudia Eckert, and Fabio Roli. Is feature selection secure against training data poisoning? In International Conference on Machine Learning, pages 1689-1698, 2015.","title":"Is feature selection secure against training data poisoning?","authors":"Battista Huang Xiao, Gavin Biggio, Giorgio Brown, Claudia Fumera, Fabio Eckert, Roli","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang and Zhu, 2019] Xuezhou Zhang and Xiaojin Zhu. Online data poisoning attack. arXiv preprint arXiv:1903.01666, 2019.","title":"Online data poisoning attack","authors":"Zhu Zhang, Xuezhou Zhang, Xiaojin Zhu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhao et al., 2017] Mengchen Zhao, Bo An, Wei Gao, and Teng Zhang. Efficient label contamination attacks against black-box learning models. In IJCAI, pages 3945-3951, 2017.","title":"Efficient label contamination attacks against black-box learning models","authors":"Zhao","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/551"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}