{"type":"ScholarlyArticle","title":"Efficient Private ERM for Smooth Objectives","authors":[{"name":"Jiaqi Zhang","slug":"jiaqi-zhang","given":"Jiaqi","family":"Zhang","affiliation":"† Key Laboratory of Machine Perception , MOE ,"},{"name":"Kai Zheng","slug":"kai-zheng","given":"Kai","family":"Zheng","affiliation":"† Key Laboratory of Machine Perception , MOE ,"},{"name":"Wenlong Mou","slug":"wenlong-mou","given":"Wenlong","family":"Mou","affiliation":"† Key Laboratory of Machine Perception , MOE ,; School of Electronics Engineering and Computer Science , Peking University , Beijing 100871 , China"},{"name":"Liwei Wang","slug":"liwei-wang","given":"Liwei","family":"Wang","affiliation":"† Key Laboratory of Machine Perception , MOE ,"}],"abstract":"In this paper, we consider efficient differentially private empirical risk minimization from the viewpoint of optimization algorithms. For strongly convex and smooth objectives, we prove that gradient descent with output perturbation not only achieves nearly optimal utility, but also significantly improves the running time of previous state-of-the-art private optimization algorithms, for both $\\epsilon$-DP and $(\\epsilon, \\delta)$-DP. For non-convex but smooth objectives, we propose an RRPSGD (Random Round Private Stochastic Gradient Descent) algorithm, which provably converges to a stationary point with privacy guarantee. Besides the expected utility bounds, we also provide guarantees in high probability form. Experiments demonstrate that our algorithm consistently outperforms existing method in both utility and running time.","doi":"10.24963/ijcai.2017/548","url":"https://ijcai.org/proceedings/2017/548","pdf":"https://ijcai.org/proceedings/2017/0548.pdf","markdown":"https://ijcai.org/proceedings/2017/548.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/548","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Multidisciplinary Topics and Applications","track":"Main track","number":548,"pages":{"first":3922,"last":3928},"keywords":["Machine Learning: Machine Learning","Multidisciplinary Topics and Applications: AI&Security and Privacy"],"videos":[],"references":[{"position":1,"raw":"[Bassily et al., 2014] Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In Foundations of Computer Science (FOCS), 2014 IEEE 55th Annual Sym- posium on, pages 464-473, New York, USA, October 2014. IEEE.","title":"Private empirical risk minimization: Efficient algorithms and tight error bounds","authors":"Bassily","year":"2014","venue":"Foundations of Computer Science (FOCS)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chaudhuri et al., 2011] Kamalika Chaudhuri, Claire Mon- teleoni, and Anand D. Sarwate. Differentially private em- pirical risk minimization. The Journal of Machine Learning Research, 12:1069-1109, March 2011.","title":"Differentially private empirical risk minimization","authors":"Kamalika Chaudhuri, Claire Monteleoni, Anand D Sarwate","year":"2011","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chaudhuri et al., 2013] Kamalika Chaudhuri, Anand D. Sar- wate, and Kaushik Sinha. A near-optimal algorithm for differentially-private principal components. The Journal of Machine Learning Research, 14(1):2905-2943, September 2013.","title":"A near-optimal algorithm for differentially-private principal components","authors":"Kamalika Chaudhuri, Anand D Sarwate, Kaushik Sinha","year":"2013","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dwork and Roth, 2014] Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foun- dations and Trends in Theoretical Computer Science, 9(3- 4):211-407, August 2014.","title":"The algorithmic foundations of differential privacy","authors":"Roth ; Dwork, Cynthia Dwork, Aaron Roth","year":"2014","venue":"Foundations and Trends in Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dwork et al., 2006] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adame Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography, pages 265-284, Berlin, Germany, March 2006. Springer.","title":"Calibrating noise to sensitivity in private data analysis","authors":"Cynthia Dwork, Frank Mcsherry, Kobbi Nissim, Adame Smith","year":"2006","venue":"Theory of cryptography","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ghadimi and Lan, 2013] Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341-2368, December 2013.","title":"Stochastic first-and zeroth-order methods for nonconvex stochastic programming","authors":"Saeed Ghadimi, Guanghui Lan","year":"2013","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hardt et al., 2016] Moritz Hardt, Benjamin Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In Proceedings of The 33rd In- ternational Conference on Machine Learning, pages 1225- 1234, Princeton, New Jersey, June 2016. The International Machine Learning Society.","title":"Train faster, generalize better: Stability of stochastic gradient descent","authors":"Moritz Hardt, Benjamin Recht, Yoram Singer","year":"2016","venue":"Proceedings of The 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jing, 2011] Lei Jing. Differentially private m-estimators. In Advances in Neural Information Processing Systems, pages 361-369, California, USA, December 2011. Neural Information Processing Systems Foundation.","title":"Differentially private m-estimators","authors":"Lei Jing","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kifer et al., 2012] Daniel Kifer, Adam Smith, and Abhradeep Thakurta. Private convex empirical risk mini- mization and high-dimensional regression. In Conference on Learning Theory (COLT), 2012.","title":"Private convex empirical risk minimization and high-dimensional regression","authors":"Daniel Kifer, Adam Smith, Abhradeep Thakurta","year":"2012","venue":"Conference on Learning Theory (COLT)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lan et al., 2012] Guanghui Lan, Arkadi Nemirovski, and Alexander Shapiro. Validation analysis of mirror descent stochastic approximation method. Mathematical program- ming, 134(2):425-458, September 2012.","title":"Validation analysis of mirror descent stochastic approximation method","authors":"Lan","year":"2012","venue":"Mathematical programming","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lichman, 2013] Moshe Lichman. Uci machine learning repository, 2013.","title":"Uci machine learning repository","authors":"Moshe Lichman","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Nesterov, 2013] Yurii Nesterov. Introductory lectures on convex optimization: A basic course. Springer Science & Business Media, Berlin, Germany, 2013.","title":"Introductory lectures on convex optimization: A basic course","authors":"Yurii Nesterov","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Rubinstein et al., 2012] Benjamin IP Rubinstein, Peter L. Bartlett, Ling Huang, and Nina Taft. Learning in a large function space: Privacy-preserving mechanisms for svm learning. Journal of Privacy and Confidentiality, 4(1):65- 100, August 2012.","title":"Learning in a large function space: Privacy-preserving mechanisms for svm learning","authors":"Peter L Benjamin Ip Rubinstein, Ling Bartlett, Nina Huang, Taft","year":"2012","venue":"Journal of Privacy and Confidentiality","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Shalev-Shwartz et al., 2009] Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Stochastic convex optimization. In Conference on Learning Theory (COLT), 2009.","title":"Stochastic convex optimization","authors":"Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, Karthik Sridharan","year":"2009","venue":"Conference on Learning Theory (COLT)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Shalev-Shwartz et al., 2010] Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Learn- ability, stability and uniform convergence. The Journal of Machine Learning Research, 11:2635-2670, October 2010.","title":"Learnability, stability and uniform convergence","authors":"Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, Karthik Sridharan","year":"2010","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Song et al., 2013] Shuang Song, Kamalika Chaudhuri, and Anand D. Sarwate. Stochastic gradient descent with differ- entially private updates. In Global Conference on Signal and Information Processing (GlobalSIP), 2013 IEEE, pages 245-248, New York, USA, December 2013. IEEE.","title":"Stochastic gradient descent with differentially private updates","authors":"Shuang Song, Kamalika Chaudhuri, Anand D Sarwate","year":"2013","venue":"Global Conference on Signal and Information Processing (GlobalSIP)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Talwar et al., 2014] Kunal Talwar, Abhradeep Thakurta, and Li Zhang. Private empirical risk minimization beyond the worst case: The effect of the constraint set geometry. arXiv preprint arXiv:1411.5417, 2014.","title":"Private empirical risk minimization beyond the worst case: The effect of the constraint set geometry","authors":"Kunal Talwar, Abhradeep Thakurta, Li Zhang","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Talwar et al., 2015] Kunal Talwar, Abhradeep Thakurta, and Li Zhang. Nearly optimal private lasso. In Advances in Neural Information Processing Systems, pages 3007-3015, California, USA, December 2015. Neural Information Pro- cessing Systems Foundation.","title":"Nearly optimal private lasso","authors":"Kunal Talwar, Abhradeep Thakurta, Li Zhang","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wu et al., 2017] Xi Wu, Fengan Li, Arun Kumar, Kamalika Chaudhuri, Somesh Jha, and Jeffrey Naughton. Bolt-on differential privacy for scalable stochastic gradient descent- based analytics. In Proceedings of the 2017 ACM Interna- tional Conference on Management of Data, pages 1307- 1322. ACM, 2017.","title":"Bolt-on differential privacy for scalable stochastic gradient descentbased analytics","authors":"Xi Wu, Fengan Li, Arun Kumar, Kamalika Chaudhuri, Somesh Jha, Jeffrey Naughton","year":"2017","venue":"Proceedings of the 2017 ACM International Conference on Management of Data","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}