{"type":"ScholarlyArticle","title":"Private Stochastic Convex Optimization and Sparse Learning with Heavy-tailed Data Revisited","authors":[{"name":"Youming Tao","slug":"youming-tao","given":"Youming","family":"Tao","affiliation":"School of Computer Science , Shandong University"},{"name":"Yulian Wu","slug":"yulian-wu","given":"Yulian","family":"Wu","affiliation":"CEMSE , KAUST"},{"name":"Xiuzhen Cheng","slug":"xiuzhen-cheng","given":"Xiuzhen","family":"Cheng","affiliation":"School of Computer Science , Shandong University"},{"name":"Di Wang","slug":"di-wang","given":"Di","family":"Wang","affiliation":"CEMSE , KAUST"}],"abstract":"In this paper, we revisit the problem of Differentially Private Stochastic Convex Optimization (DP-SCO) with heavy-tailed data, where the gradient of the loss function has bounded moments. Instead of the case where the loss function is Lipschitz or each coordinate of the gradient has bounded second moment studied previously, we consider a relaxed scenario where  each coordinate of the gradient only has bounded (1+v)-th moment with some v∈(0, 1]. Firstly, we start from the one dimensional private mean estimation for heavy-tailed distributions. We propose a novel robust and private mean estimator which is optimal. Based on its idea, we then extend to the general d-dimensional space and study DP-SCO with general convex and strongly convex loss functions. We also provide  lower bounds for these two classes of loss under our setting and show that our upper bounds are optimal up to a factor of  O(Poly(d)). To address the high dimensionality issue, we also study DP-SCO with heavy-tailed gradient under some sparsity constraint (DP sparse learning). We propose a new method and show it is also optimal up to a factor of O(s*), where s* is the underlying sparsity of the constraint.","doi":"10.24963/ijcai.2022/548","url":"https://ijcai.org/proceedings/2022/548","pdf":"https://ijcai.org/proceedings/2022/0548.pdf","markdown":"https://ijcai.org/proceedings/2022/548.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/548","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":548,"pages":{"first":3947,"last":3953},"keywords":["Multidisciplinary Topics and Applications: Security and Privacy"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986851","length_sec":81,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984994","length_sec":336,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Acharya et al., 2021] Jayadev Acharya, Ziteng Sun, and Huanyu Zhang. Differentially private assouad, fano, and le cam. In ALT, volume 132 of Proceedings of Machine Learning Research, pages 48-78. PMLR, 2021.","title":"Differentially private assouad, fano, and le cam","authors":"Jayadev Acharya, Ziteng Sun, Huanyu Zhang","year":"2021","venue":"Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bahmani et al., 2013] Sohail Bahmani, Bhiksha Raj, and Petros T Boufounos. Greedy sparsity-constrained op- timization. Journal of Machine Learning Research, 14(Mar):807-841, 2013.","title":"Greedy sparsity-constrained optimization","authors":"Sohail Bahmani, Bhiksha Raj, Petros T Boufounos","year":"2013","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Barber and Duchi, 2014] Rina Foygel Barber and John C. Duchi. Privacy and statistical risk: Formalisms and mini- max bounds, 2014.","title":"Privacy and statistical risk: Formalisms and minimax bounds","authors":"Rina Foygel Barber, John C Duchi","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bassily et al., 2014] Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In Foun- dations of Computer Science (FOCS), 2014 IEEE 55th Annual Symposium on, pages 464-473. IEEE, 2014.","title":"Private empirical risk minimization: Efficient algorithms and tight error bounds","authors":"Raef Bassily, Adam Smith, Abhradeep Thakurta","year":"2014","venue":"Foundations of Computer Science (FOCS)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bassily et al., 2019] Raef Bassily, Vitaly Feldman, Kunal Talwar, and Abhradee Thakurta. Private stochastic convex optimization with optimal rates. In NeurIPS, 2019.","title":"Private stochastic convex optimization with optimal rates","authors":"Raef Bassily, Vitaly Feldman, Kunal Talwar, Abhradee Thakurta","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Blumensath and Davies, 2009] Thomas Blumensath and Mike E Davies. Iterative hard thresholding for compressed sensing. Applied and computational harmonic analysis, 27(3):265-274, 2009.","title":"Iterative hard thresholding for compressed sensing","authors":"Thomas Blumensath, Mike E Davies","year":"2009","venue":"Applied and computational harmonic analysis","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cai et al., 2019] T Tony Cai, Yichen Wang, and Linjun Zhang. The cost of privacy: Optimal rates of convergence for parameter estimation with differential privacy. arXiv preprint arXiv:1902.04495, 2019.","title":"The cost of privacy: Optimal rates of convergence for parameter estimation with differential privacy","authors":"Tony Cai, Yichen Wang, Linjun Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cai et al., 2020] T Tony Cai, Yichen Wang, and Linjun Zhang. The cost of privacy in generalized linear models: Algorithms and minimax lower bounds. arXiv preprint arXiv:2011.03900, 2020.","title":"The cost of privacy in generalized linear models: Algorithms and minimax lower bounds","authors":"Tony Cai, Yichen Wang, Linjun Zhang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Duchi et al., 2018] John C Duchi, Michael I Jordan, and Martin J Wainwright. Minimax optimal procedures for locally private estimation. Journal of the American Sta- tistical Association, 113(521):182-201, 2018.","title":"Minimax optimal procedures for locally private estimation","authors":"Michael I John C Duchi, Martin J Jordan, Wainwright","year":"2018","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dwork et al., 2006] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sen- sitivity in private data analysis. In Theory of cryptography conference, pages 265-284. Springer, 2006.","title":"Calibrating noise to sensitivity in private data analysis","authors":"Cynthia Dwork, Frank Mcsherry, Kobbi Nissim, Adam Smith","year":"2006","venue":"Theory of cryptography conference","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Feldman et al., 2020] Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimization: op- timal rates in linear time. In STOC, pages 439-449, 2020.","title":"Private stochastic convex optimization: optimal rates in linear time","authors":"Tomer Vitaly Feldman, Kunal Koren, Talwar","year":"2020","venue":"STOC","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hu et al., 2021] Lijie Hu, Shuo Ni, Hanshen Xiao, and Di Wang. High dimensional differentially private stochas- tic optimization with heavy-tailed data. arXiv preprint arXiv:2107.11136, 2021.","title":"High dimensional differentially private stochastic optimization with heavy-tailed data","authors":"Hu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kamath et al., 2020] Gautam Kamath, Vikrant Singhal, and Jonathan Ullman. Private mean estimation of heavy-tailed distributions. In Conference on Learning Theory, pages 2204-2235. PMLR, 2020.","title":"Private mean estimation of heavy-tailed distributions","authors":"Gautam Kamath, Vikrant Singhal, Jonathan Ullman","year":"2020","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kamath et al., 2021] Gautam Kamath, Xingtu Liu, and Huanyu Zhang. Improved rates for differentially pri- vate stochastic convex optimization with heavy-tailed data. arXiv preprint arXiv:2106.01336, 2021.","title":"Improved rates for differentially private stochastic convex optimization with heavy-tailed data","authors":"Gautam Kamath, Xingtu Liu, Huanyu Zhang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2021] Xiyang Liu, Weihao Kong, Sham Kakade, and Sewoong Oh. Robust and differentially private mean estimation. arXiv preprint arXiv:2102.09159, 2021.","title":"Robust and differentially private mean estimation","authors":"Xiyang Liu, Weihao Kong, Sham Kakade, Sewoong Oh","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tao et al., 2021] Youming Tao, Yulian Wu, Peng Zhao, and Di Wang. Optimal rates of (locally) differentially pri- vate heavy-tailed multi-armed bandits. arXiv preprint arXiv:2106.02575, 2021.","title":"Optimal rates of (locally) differentially private heavy-tailed multi-armed bandits","authors":"Youming Tao, Yulian Wu, Peng Zhao, Di Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Vapnik, 1999] Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.","title":"The nature of statistical learning theory","authors":"Vladimir Vapnik","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang and Gu, 2019] Lingxiao Wang and Quanquan Gu. Differentially private iterative gradient hard thresholding for sparse learning. In 28th International Joint Conference on Artificial Intelligence, 2019.","title":"Differentially private iterative gradient hard thresholding for sparse learning","authors":"Wang, Gu, Lingxiao Wang, Quanquan Gu","year":"2019","venue":"28th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2019/519"},{"position":19,"raw":"[Wang and Xu, 2019] Di Wang and Jinhui Xu. On sparse linear regression in the local differential privacy model. In ICML, volume 97 of Proceedings of Machine Learning Research, pages 6628-6637. PMLR, 2019.","title":"On sparse linear regression in the local differential privacy model","authors":"Di Wang, Jinhui Xu","year":"2019","venue":"Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang and Xu, 2021] Di Wang and Jinhui Xu. On sparse lin- ear regression in the local differential privacy model. IEEE Trans. Inf. Theory, 67(2):1182-1200, 2021.","title":"On sparse linear regression in the local differential privacy model","authors":"Di Wang, Jinhui Xu","year":"2021","venue":"IEEE Trans. Inf. Theory","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2020a] Di Wang, Jiahao Ding, Lijie Hu, Zejun Xie, Miao Pan, and Jinhui Xu. Differentially private (gra- dient) expectation maximization algorithm with statistical guarantees. arXiv preprint arXiv:2010.13520, 2020.","title":"Differentially private (gradient) expectation maximization algorithm with statistical guarantees","authors":"Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2020b] Di Wang, Hanshen Xiao, Srinivas De- vadas, and Jinhui Xu. On differentially private stochas- tic convex optimization with heavy-tailed data. In Inter- national Conference on Machine Learning, pages 10081- 10091. PMLR, 2020.","title":"On differentially private stochastic convex optimization with heavy-tailed data","authors":"Wang","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Woolson and Clarke, 2011] Robert F Woolson and William R Clarke. Statistical methods for the analy- sis of biomedical data, volume 371.","title":"Statistical methods for the analysis of biomedical data","authors":"F Robert, William R Woolson, Clarke","year":null,"venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}