{"type":"ScholarlyArticle","title":"Privacy-Preserving Stacking with Application to Cross-organizational Diabetes Prediction","authors":[{"name":"Quanming Yao","slug":"quanming-yao","given":"Quanming","family":"Yao","affiliation":"Department of Computer Science and Engineering , HKUST"},{"name":"Xiawei Guo","slug":"xiawei-guo","given":"Xiawei","family":"Guo","affiliation":"Department of Computer Science and Engineering , HKUST"},{"name":"James Kwok","slug":"james-kwok","given":"James","family":"Kwok","affiliation":"Department of Computer Science and Engineering , HKUST"},{"name":"Weiwei Tu","slug":"weiwei-tu","given":"Weiwei","family":"Tu","affiliation":"Department of Computer Science and Engineering , HKUST"},{"name":"Yuqiang Chen","slug":"yuqiang-chen","given":"Yuqiang","family":"Chen","affiliation":"Department of Computer Science and Engineering , HKUST"},{"name":"Wenyuan Dai","slug":"wenyuan-dai","given":"Wenyuan","family":"Dai","affiliation":"Department of Computer Science and Engineering , HKUST"},{"name":"Qiang Yang","slug":"qiang-yang","given":"Qiang","family":"Yang","affiliation":"Department of Computer Science and Engineering , HKUST"}],"abstract":"To meet the standard of differential privacy, noise is usually added into the original data, which inevitably deteriorates the predicting performance of subsequent learning algorithms. In this paper, motivated by the success of improving predicting performance by ensemble learning, we propose to enhance privacy-preserving logistic regression by stacking. We show that this can be done either by sample-based or feature-based partitioning. However, we prove that when privacy-budgets are the same, feature-based partitioning requires fewer samples than sample-based one, and thus likely has better empirical performance. As transfer learning is difficult to be integrated with a differential privacy guarantee, we further combine the proposed method with hypothesis transfer learning to address the problem of learning across different organizations. Finally, we not only demonstrate the effectiveness of our method on two benchmark data sets, i.e., MNIST and NEWS20, but also apply it into a real application of cross-organizational diabetes prediction from RUIJIN data set, where privacy is of a significant concern.","doi":"10.24963/ijcai.2019/571","url":"https://ijcai.org/proceedings/2019/571","pdf":"https://ijcai.org/proceedings/2019/0571.pdf","markdown":"https://ijcai.org/proceedings/2019/571.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/571","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":571,"pages":{"first":4114,"last":4120},"keywords":["Machine Learning: Ensemble Methods","Machine Learning Applications: Bio;Medicine","Multidisciplinary Topics and Applications: Security and Privacy"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2016] M. Abadi, A. Chu, I. Goodfellow, B. M- cMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In SIGSAC, pages 308- 318. ACM, 2016.","title":"Deep learning with differential privacy","authors":"M Abadi, A Chu, I Goodfellow, B M-Cmahan, I Mironov, K Talwar, L Zhang","year":"2016","venue":"SIGSAC","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bassily et al., 2014] R. Bassily, A. Smith, and A. Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In FOCS, pages 464-473. IEEE, 2014.","title":"Private empirical risk minimization: Efficient algorithms and tight error bounds","authors":"R Bassily, A Smith, A Thakurta","year":"2014","venue":"FOCS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Boyd et al., 2011] S. Boyd, N. Parikh, and E. Chu. Dis- tributed optimization and statistical learning via the alter- nating direction method of multipliers. Foundations and Trends R in Machine learning, 3(1):1-122, 2011.","title":"Distributed optimization and statistical learning via the alternating direction method of multipliers","authors":"S Boyd, N Parikh, E Chu","year":"2011","venue":"Foundations and Trends R in Machine learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Breiman, 1996a] L. Breiman. Bagging predictors. ML, 24(2):123-140, 1996.","title":"Bagging predictors","authors":"; L Breiman, Breiman","year":"1996","venue":"ML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Breiman, 1996b] L. Breiman. Stacked regressions. ML, 24(1):49-64, 1996.","title":"Stacked regressions","authors":"; L Breiman, Breiman","year":"1996","venue":"ML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chaudhuri et al., 2011] K. Chaudhuri, C. Monteleoni, and A. Sarwate. Differentially private empirical risk minimiza- tion. JMLR, 12(Mar):1069-1109, 2011.","title":"Differentially private empirical risk minimization","authors":"K Chaudhuri, C Monteleoni, A Sarwate","year":"2011","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dwork and Roth, 2014] C. Dwork and A. Roth. The algo- rithmic foundations of differential privacy. Foundation- s and Trends R in Theoretical Computer Science, 9(3- 4):211-407, 2014.","title":"The algorithmic foundations of differential privacy","authors":"C Dwork, A Roth","year":"2014","venue":"Foundations and Trends R in Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dwork et al., 2006] C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating noise to sensitivity in private data analysis. In TCC, pages 265-284. Springer, 2006.","title":"Calibrating noise to sensitivity in private data analysis","authors":"C Dwork, F Mcsherry, K Nissim, A Smith","year":"2006","venue":"TCC","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Džeroski and Ženko, 2004] S. Džeroski and B. Ženko. Is combining classifiers with stacking better than selecting the best one? ML, 54(3):255-273, 2004.","title":"Is combining classifiers with stacking better than selecting the best one?","authors":"S Džeroski, B Ženko","year":"2004","venue":"ML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Emekc ¸i et al., 2007] F. Emekc ¸i, O. Sahin, D. Agrawal, and A. El Abbadi. Privacy preserving decision tree learning over multiple parties. TKDE, 63(2):348-361, 2007.","title":"Privacy preserving decision tree learning over multiple parties","authors":"F Emekc ¸i, O Sahin, D, A El Abbadi","year":"2007","venue":"TKDE","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fong and Weber-Jahnke, 2012] P. Fong and J. Weber- Jahnke. Privacy preserving decision tree learning using unrealized data sets. TKDE, 24(2):353-364, 2012.","title":"Privacy preserving decision tree learning using unrealized data sets","authors":"P Fong, J Weber-Jahnke","year":"2012","venue":"TKDE","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Friedman et al., 2000] J. Friedman, T. Hastie, and R. Tib- shirani. Additive logistic regression: a statistical view of boosting. Annals of Statistics, 28(2):337-407, 2000.","title":"Additive logistic regression: a statistical view of boosting","authors":"J Friedman, T Hastie, R Tibshirani","year":"2000","venue":"Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Friedman et al., 2012] J. Friedman, T. Hastie, and R. Tib- shirani. The elements of statistical learning. Springer, 2012.","title":"The elements of statistical learning","authors":"J Friedman, T Hastie, R Tibshirani","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hamm et al., 2016] J. Hamm, Y. Cao, and M. Belkin. Learning privately from multiparty data. In ICML, pages 555-563, 2016.","title":"Learning privately from multiparty data","authors":"J Hamm, Y Cao, M Belkin","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hanley and McNeil, 1983] J. Hanley and B. McNeil. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiol- ogy, 148(3):839-843, 1983.","title":"A method of comparing the areas under receiver operating characteristic curves derived from the same cases","authors":"J Hanley, B Mcneil","year":"1983","venue":"Radiology","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kasiviswanathan and Jin, 2016] P. Kasiviswanathan and H. Jin. Efficient private empirical risk minimization for high-dimensional learning. In ICML, pages 488-497, 2016.","title":"Efficient private empirical risk minimization for high-dimensional learning","authors":"P Kasiviswanathan, H Jin","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kifer et al., 2012] D. Kifer, A. Smith, and A. Thakurta. Private convex empirical risk minimization and high- dimensional regression. JMLR, 23, 2012.","title":"Private convex empirical risk minimization and highdimensional regression","authors":"D Kifer, A Smith, A Thakurta","year":"2012","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kuzborskij and Orabona, 2013] I. Kuzborskij and F. Orabona. Stability and hypothesis transfer learning. In ICML, pages 942-950, 2013.","title":"Stability and hypothesis transfer learning","authors":"Orabona Kuzborskij, I Kuzborskij, F Orabona","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lang, 1995] Ken Lang. Newsweeder: Learning to filter netnews. In ICML. Citeseer, 1995.","title":"Newsweeder: Learning to filter netnews","authors":"Ken Lang","year":"1995","venue":"ICML. Citeseer","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[LeCun et al., 1998] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Y Lecun, L Bottou, Y Bengio, P Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ozay and Vural, 2012] M. Ozay and F. Vural. A new fuzzy stacked generalization technique and analysis of its performance. Technical report, arXiv:1204.0171, 2012.","title":"A new fuzzy stacked generalization technique and analysis of its performance","authors":"M Ozay, F Vural","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pan and Yang, 2010] J. Pan and Q. Yang. A survey on transfer learning. TKDE, 22(10):1345-1359, 2010.","title":"A survey on transfer learning","authors":"J Pan, Q Yang","year":"2010","venue":"TKDE","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Papernot et al., 2017] N. Papernot, M. Abadi, U. Erlings- son, I. Goodfellow, and K. Talwar. Semi-supervised knowledge transfer for deep learning from private training data. In ICLR, 2017.","title":"Semi-supervised knowledge transfer for deep learning from private training data","authors":"N Papernot, M Abadi, U Erlingsson, I Goodfellow, K Talwar","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pathak et al., 2010] M. Pathak, S. Rane, and B. Raj. Multi- party differential privacy via aggregation of locally trained classifiers. In NIPS, pages 1876-1884, 2010.","title":"Multiparty differential privacy via aggregation of locally trained classifiers","authors":"M Pathak, S Rane, B Raj","year":"2010","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rajkumar and Agarwal, 2012] A. Rajkumar and S. Agarw- al. A differentially private stochastic gradient descent algorithm for multiparty classification. In AISTAT, pages 933-941, 2012.","title":"A differentially private stochastic gradient descent algorithm for multiparty classification","authors":"Agarwal Rajkumar, A Rajkumar, S Agarwal","year":"2012","venue":"AISTAT","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shalev-Shwartz and Srebro, 2008] S. Shalev-Shwartz and N. Srebro. SVM optimization: inverse dependence on training set size. In ICML, pages 928-935. ACM, 2008.","title":"SVM optimization: inverse dependence on training set size","authors":"Shalev-Shwartz Srebro, ; ] S Shalev-Shwartz, N Srebro","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Shokri and Shmatikov, 2015] R. Shokri and V. Shmatikov. Privacy-preserving deep learning. In SIGSAC, pages 1310-1321, 2015.","title":"Privacy-preserving deep learning","authors":"R Shokri, V Shmatikov","year":"2015","venue":"SIGSAC","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Smyth and Wolpert, 1999] P. Smyth and D. Wolpert. Lin- early combining density estimators via stacking. ML, 36(1-2):59-83, 1999.","title":"Linearly combining density estimators via stacking","authors":"P Smyth, D Wolpert","year":"1999","venue":"ML","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Ting and Witten, 1999] K. Ting and I. Witten. Issues in stacked generalization. JAIR, 10:271-289, 1999.","title":"Issues in stacked generalization","authors":"K Ting, I Witten","year":"1999","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2018] Y. Wang, Q. Gu, and D. Brown. Dif- ferentially private hypothesis transfer learning. In ECML, 2018.","title":"Differentially private hypothesis transfer learning","authors":"Y Wang, Q Gu, D Brown","year":"2018","venue":"ECML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wolpert, 1992] D. Wolpert. Stacked generalization. Neural Networks, 5(2):241-259, 1992.","title":"Stacked generalization","authors":"D Wolpert","year":"1992","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Xie et al., 2017] L. Xie, I. Baytas, K. Lin, and J. Zhou. Privacy-preserving distributed multi-task learning with asynchronous updates. In SIGKDD, pages 1195-1204, 2017.","title":"Privacy-preserving distributed multi-task learning with asynchronous updates","authors":"L Xie, I Baytas, K Lin, J Zhou","year":"2017","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhou, 2012] Z.-H. Zhou. Ensemble methods: foundations and algorithms. Chapman and Hall/CRC, 2012.","title":"Ensemble methods: foundations and algorithms","authors":"Z.-H Zhou","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}