{"type":"ScholarlyArticle","title":"Differentially Private Iterative Gradient Hard Thresholding for Sparse Learning","authors":[{"name":"Lingxiao Wang","slug":"lingxiao-wang","given":"Lingxiao","family":"Wang","affiliation":"Department of Computer Science , University of California , Los Angeles"},{"name":"Quanquan Gu","slug":"quanquan-gu","given":"Quanquan","family":"Gu","affiliation":"Department of Computer Science , University of California , Los Angeles"}],"abstract":"We consider the differentially private sparse learning problem, where the goal is to estimate the underlying sparse parameter vector of a statistical model in the high-dimensional regime while preserving the privacy of each training example. We propose a generic differentially private iterative gradient hard threshoding algorithm with a linear convergence rate and strong utility guarantee. We demonstrate the superiority of our algorithm through two specific applications: sparse linear regression and sparse logistic regression. Specifically, for sparse linear regression, our algorithm can achieve the best known utility guarantee without any extra support selection procedure used in previous work \\cite{kifer2012private}. For sparse logistic regression, our algorithm can obtain the utility guarantee with a logarithmic dependence on the problem dimension.  Experiments on both synthetic data and real world datasets verify the effectiveness of our proposed algorithm.","doi":"10.24963/ijcai.2019/519","url":"https://ijcai.org/proceedings/2019/519","pdf":"https://ijcai.org/proceedings/2019/0519.pdf","markdown":"https://ijcai.org/proceedings/2019/519.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/519","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":519,"pages":{"first":3740,"last":3747},"keywords":["Multidisciplinary Topics and Applications: Security and Privacy","Machine Learning: Feature Selection ; Learning Sparse Models"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martin Abadi, Andy Chu, Ian Goodfel- low, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 308-318. ACM, 2016.","title":"Deep learning with differential privacy","authors":"Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan Mcmahan, Ilya Mironov, Kunal Talwar, Li Zhang","year":"2016","venue":"Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bassily et al., 2014] Raef Bassily, Adam Smith, and Abhradeep Thakurta. Differentially private empirical risk minimization: Efficient algorithms and tight error bounds. arXiv preprint arXiv:1405.7085, 2014.","title":"Differentially private empirical risk minimization: Efficient algorithms and tight error bounds","authors":"Raef Bassily, Adam Smith, Abhradeep Thakurta","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bickel et al., 2009] Peter J Bickel, Ya'acov Ritov, Alexan- dre B Tsybakov, et al. Simultaneous analysis of lasso and dantzig selector. The Annals of Statistics, 37(4):1705- 1732, 2009.","title":"Simultaneous analysis of lasso and dantzig selector","authors":"J Peter, ; B Bickel, Tsybakov","year":"2009","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bun and Steinke, 2016] Mark Bun and Thomas Steinke. Concentrated differential privacy: Simplifications, exten- sions, and lower bounds. In Theory of Cryptography Con- ference, pages 635-658. Springer, 2016. Chaudhuri and Monteleoni, 2009]","title":"Concentrated differential privacy: Simplifications, extensions, and lower bounds","authors":"Steinke Bun, Mark Bun, Thomas Steinke","year":"2009","venue":"Theory of Cryptography Conference","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"Kamalika Chaudhuri and Claire Monteleoni. Privacy-preserving logistic regression. In Advances in Neural Information Processing Systems, pages 289-296, 2009.","title":"Privacy-preserving logistic regression","authors":"Kamalika Chaudhuri, Claire Monteleoni","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chaudhuri et al., 2011] Kamalika Chaudhuri, Claire Mon- teleoni, and Anand D Sarwate. Differentially private em- pirical risk minimization. Journal of Machine Learning Research, 12(Mar):1069-1109, 2011.","title":"Differentially private empirical risk minimization","authors":"Kamalika Chaudhuri, Claire Monteleoni, Anand D Sarwate","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen and Gu, 2016] Jinghui Chen and Quanquan Gu. Ac- celerated stochastic block coordinate gradient descent for sparsity constrained nonconvex optimization. In UAI, 2016.","title":"Accelerated stochastic block coordinate gradient descent for sparsity constrained nonconvex optimization","authors":"Jinghui Chen, Quanquan Gu","year":"2016","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ding et al., 2017] Bolin Ding, Janardhan Kulkarni, and Sergey Yekhanin. Collecting telemetry data privately. In Advances in Neural Information Processing Systems, pages 3571-3580, 2017.","title":"Collecting telemetry data privately","authors":"Janardhan Bolin Ding, Sergey Kulkarni, Yekhanin","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dwork et al., 2006] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sen- sitivity in private data analysis. In Theory of Cryptography Conference, pages 265-284. Springer, 2006.","title":"Calibrating noise to sensitivity in private data analysis","authors":"Cynthia Dwork, Frank Mcsherry, Kobbi Nissim, Adam Smith","year":"2006","venue":"Theory of Cryptography Conference","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Erlingsson et al., 2014] Úlfar Erlingsson, Vasyl Pihur, and Aleksandra Korolova. Rappor: Randomized aggregatable privacy-preserving ordinal response. In Proceedings of the 2014 ACM SIGSAC conference on computer and commu- nications security, pages 1054-1067. ACM, 2014.","title":"Aleksandra Korolova. Rappor: Randomized aggregatable privacy-preserving ordinal response","authors":"Erlingsson","year":"2014","venue":"Proceedings of the 2014 ACM SIGSAC conference on computer and communications security","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jain and Thakurta, 2014] Prateek Jain and Abhradeep Guha Thakurta. (near) dimension independent risk bounds for differentially private learning. In International Conference on Machine Learning, pages 476-484, 2014.","title":"near) dimension independent risk bounds for differentially private learning","authors":"Prateek Jain, Abhradeep Guha, Thakurta","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jain et al., 2014] Prateek Jain, Ambuj Tewari, and Pu- rushottam Kar. On iterative hard thresholding methods for high-dimensional m-estimation. In Advances in Neural In- formation Processing Systems, pages 685-693, 2014.","title":"On iterative hard thresholding methods for high-dimensional m-estimation","authors":"Prateek Jain, Ambuj Tewari, Purushottam Kar","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jayaraman et al., 2018] Bargav Jayaraman, Lingxiao Wang, David Evans, and Quanquan Gu. Distributed learning without distress: Privacy-preserving empirical risk mini- mization. In Advances in Neural Information Processing Systems, pages 6346-6357, 2018.","title":"Distributed learning without distress: Privacy-preserving empirical risk minimization","authors":"Jayaraman","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kifer et al., 2012] Daniel Kifer, Adam Smith, and Abhradeep Thakurta. Private convex empirical risk minimization and high-dimensional regression. In Conference on Learning Theory, pages 25-1, 2012.","title":"Private convex empirical risk minimization and high-dimensional regression","authors":"Daniel Kifer, Adam Smith, Abhradeep Thakurta","year":"2012","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kogan et al., 2009] Shimon Kogan, Dimitry Levin, Bryan R Routledge, Jacob S Sagi, and Noah A Smith. Predicting risk from financial reports with regression. In Proceedings of Human Language Technologies: The 2009 Annual Con- ference of the North American Chapter of the Association for Computational Linguistics, pages 272-280. Associa- tion for Computational Linguistics, 2009.","title":"Predicting risk from financial reports with regression","authors":"Shimon Kogan, Dimitry Levin, Jacob S Bryan R Routledge, Noah A Sagi, Smith","year":"2009","venue":"Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lewis et al., 2004] David D Lewis, Yiming Yang, Tony G Rose, and Fan Li. Rcv1: A new benchmark collection for text categorization research. Journal of machine learning research, 5(Apr):361-397, 2004.","title":"Rcv1: A new benchmark collection for text categorization research","authors":"Yiming David D Lewis, Tony G Yang, Fan Rose, Li","year":"2004","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2016] Xingguo Li, Raman Arora, Han Liu, Jarvis Haupt, and Tuo Zhao. Nonconvex sparse learning via stochastic optimization with progressive variance reduc- tion. arXiv preprint arXiv:1605.02711, 2016.","title":"Nonconvex sparse learning via stochastic optimization with progressive variance reduction","authors":"Xingguo Li, Raman Arora, Han Liu, Jarvis Haupt, Tuo Zhao","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Loh and Wainwright, 2013] Po-Ling Loh and Martin J Wainwright. Regularized m-estimators with nonconvexity: Statistical and algorithmic theory for local optima. In Ad- vances in Neural Information Processing Systems, pages 476-484, 2013.","title":"Regularized m-estimators with nonconvexity: Statistical and algorithmic theory for local optima","authors":"Wainwright ; Po-Ling Loh, Martin J Loh, Wainwright","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Negahban et al., 2009] Sahand Negahban, Bin Yu, Martin J Wainwright, and Pradeep K Ravikumar. A unified frame- work for high-dimensional analysis of m-estimators with decomposable regularizers. In Advances in Neural Infor- mation Processing Systems, pages 1348-1356, 2009.","title":"A unified framework for high-dimensional analysis of m-estimators with decomposable regularizers","authors":"Bin Sahand Negahban, Martin J Yu, Pradeep K Wainwright, Ravikumar","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Raskutti et al., 2011] Garvesh Raskutti, Martin J Wain- wright, and Bin Yu. Minimax rates of estimation for high- dimensional linear regression over ℓ q -balls. IEEE transac- tions on information theory, 57(10):6976-6994, 2011.","title":"Minimax rates of estimation for highdimensional linear regression over ℓ q -balls","authors":"Raskutti","year":"2011","venue":"IEEE transactions on information theory","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rényi, 1961] Alfréd Rényi. On measures of entropy and in- formation. Technical report, HUNGARIAN ACADEMY OF SCIENCES Budapest Hungary, 1961.","title":"Alfréd Rényi. On measures of entropy and information","authors":"Rényi","year":"1961","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Talwar et al., 2015] Kunal Talwar, Abhradeep Guha Thakurta, and Li Zhang. Nearly optimal private lasso. In Advances in Neural Information Processing Systems, pages 3025-3033, 2015.","title":"Nearly optimal private lasso","authors":"Kunal Talwar, Abhradeep Guha Thakurta, Li Zhang","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Thakurta and Smith, 2013] Abhradeep Guha Thakurta and Adam Smith. Differentially private feature selection via stability arguments, and the robustness of the lasso. In Conference on Learning Theory, pages 819-850, 2013.","title":"Differentially private feature selection via stability arguments, and the robustness of the lasso","authors":"Smith ; Thakurta, Guha Abhradeep, Adam Thakurta, Smith","year":"2013","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tibshirani, 1996] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), pages 267-288, 1996.","title":"Regression shrinkage and selection via the lasso","authors":"Robert Tibshirani","year":"1996","venue":"Journal of the Royal Statistical Society. Series B (Methodological)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Van de Geer and others, 2008] Sara A Van de Geer et al. High-dimensional generalized linear models and the lasso. The Annals of Statistics, 36(2):614-645, 2008.","title":"High-dimensional generalized linear models and the lasso","authors":"A Sara, Van De Geer","year":"2008","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2017] Di Wang, Minwei Ye, and Jinhui Xu. Differentially private empirical risk minimization revis- ited: Faster and more general. In Advances in Neural In- formation Processing Systems, pages 2719-2728, 2017.","title":"Differentially private empirical risk minimization revisited: Faster and more general","authors":"Di Wang, Minwei Ye, Jinhui Xu","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2018] Di Wang, Marco Gaboardi, and Jinhui Xu. Empirical risk minimization in non-interactive local differential privacy revisited. In Advances in Neural Infor- mation Processing Systems, pages 973-982, 2018.","title":"Empirical risk minimization in non-interactive local differential privacy revisited","authors":"Di Wang, Marco Gaboardi, Jinhui Xu","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xu et al., 2012] Huan Xu, Constantine Caramanis, and Shie Mannor. Sparse algorithms are not stable: A no-free-lunch theorem. IEEE transactions on pattern analysis and ma- chine intelligence, 34(1):187-193, 2012.","title":"Sparse algorithms are not stable: A no-free-lunch theorem","authors":"Huan Xu, Constantine Caramanis, Shie Mannor","year":"2012","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yuan et al., 2014] Xiaotong Yuan, Ping Li, and Tong Zhang. Gradient hard thresholding pursuit for sparsity- constrained optimization. In International Conference on Machine Learning, pages 127-135, 2014.","title":"Gradient hard thresholding pursuit for sparsityconstrained optimization","authors":"Xiaotong Yuan, Ping Li, Tong Zhang","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2017] Jiaqi Zhang, Kai Zheng, Wenlong Mou, and Liwei Wang. Efficient private erm for smooth objec- tives. arXiv preprint arXiv:1703.09947, 2017.","title":"Efficient private erm for smooth objectives","authors":"Jiaqi Zhang, Kai Zheng, Wenlong Mou, Liwei Wang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2017/548"},{"position":31,"raw":"[Zhang, 2011] Tong Zhang. Adaptive forward-backward greedy algorithm for learning sparse representations. IEEE transactions on information theory, 57(7):4689-4708, 2011.","title":"Adaptive forward-backward greedy algorithm for learning sparse representations","authors":"Tong Zhang","year":"2011","venue":"IEEE transactions on information theory","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}