{"type":"ScholarlyArticle","title":"Improving Heterogeneous Model Reuse by Density Estimation","authors":[{"name":"Anke Tang","slug":"anke-tang","given":"Anke","family":"Tang","affiliation":"School of Computer Science , National Engineering Research Center for Multimedia Software , Institute of Artificial Intelligence and Hubei Key Laboratory of Multimedia and Network Communication Engineering , Wuhan University , China; Hubei Luojia Laboratory , Wuhan , China"},{"name":"Yong Luo","slug":"yong-luo","given":"Yong","family":"Luo","affiliation":"School of Computer Science , National Engineering Research Center for Multimedia Software , Institute of Artificial Intelligence and Hubei Key Laboratory of Multimedia and Network Communication Engineering , Wuhan University , China; Hubei Luojia Laboratory , Wuhan , China"},{"name":"Han Hu","slug":"han-hu","given":"Han","family":"Hu","affiliation":"School of Information and Electronics , Beijing Institute of Technology , China"},{"name":"Fengxiang He","slug":"fengxiang-he","given":"Fengxiang","family":"He","affiliation":"JD Explore Academy , JD.com , Inc. , China"},{"name":"Kehua Su","slug":"kehua-su","given":"Kehua","family":"Su","affiliation":"School of Computer Science , National Engineering Research Center for Multimedia Software , Institute of Artificial Intelligence and Hubei Key Laboratory of Multimedia and Network Communication Engineering , Wuhan University , China"},{"name":"Bo Du","slug":"bo-du","given":"Bo","family":"Du","affiliation":"School of Computer Science , National Engineering Research Center for Multimedia Software , Institute of Artificial Intelligence and Hubei Key Laboratory of Multimedia and Network Communication Engineering , Wuhan University , China; Hubei Luojia Laboratory , Wuhan , China"},{"name":"Yixin Chen","slug":"yixin-chen","given":"Yixin","family":"Chen","affiliation":"Department of CSE , Washington University in St. Louis , USA"},{"name":"Dacheng Tao","slug":"dacheng-tao","given":"Dacheng","family":"Tao","affiliation":"The University of Sydney , Australia"}],"abstract":"This paper studies multiparty learning, aiming to learn a model using the private data of different participants. Model reuse is a promising solution for multiparty learning, assuming that a local model has been trained for each party. Considering the potential sample selection bias among different parties, some heterogeneous model reuse approaches have been developed. However, although pre-trained local classifiers are utilized in these approaches, the characteristics of the local data are not well exploited. This motivates us to estimate the density of local data and design an auxiliary model together with the local classifiers for reuse. To address the scenarios where some local models are not well pre-trained, we further design a multiparty cross-entropy loss for calibration. Upon existing works, we address a challenging problem of heterogeneous model reuse from a decision theory perspective and take advantage of recent advances in density estimation. Experimental results on both synthetic and benchmark data demonstrate the superiority of the proposed method.","doi":"10.24963/ijcai.2023/472","url":"https://ijcai.org/proceedings/2023/472","pdf":"https://ijcai.org/proceedings/2023/0472.pdf","markdown":"https://ijcai.org/proceedings/2023/472.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/472","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":472,"pages":{"first":4244,"last":4252},"keywords":["Machine Learning: ML: Classification","Computer Vision: CV: Transfer, low-shot, semi- and un- supervised learning","Machine Learning: ML: Multi-task and transfer learning"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martin Abadi, Andy Chu, Ian Goodfel- low, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on com- puter and communications security, pages 308-318, 2016.","title":"Deep learning with differential privacy","authors":"Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan Mcmahan, Ilya Mironov, Kunal Talwar, Li Zhang","year":"2016","venue":"Proceedings of the 2016 ACM SIGSAC conference on computer and communications security","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bishop and Nasrabadi, 2006] Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.","title":"Pattern recognition and machine learning","authors":"M Christopher, Bishop, M Nasser, Nasrabadi","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ding et al., 2020] Yao-Xiang Ding, Zhi-Hua Zhou, Sinno Jialin Pan, and Masashi Sugiyama. Boosting-Based Reliable Model Reuse. Proceedings of Machine Learning Research, 129(1):145-160, 2020.","title":"Boosting-Based Reliable Model Reuse","authors":"Yao-Xiang Ding, Zhi-Hua Zhou, Sinno Jialin Pan, Masashi Sugiyama","year":"2020","venue":"Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dinh et al., 2016] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. 5th Inter- national Conference on Learning Representations, ICLR 2017 -Conference Track Proceedings, pages 1-32, may 2016.","title":"Density estimation using real nvp. 5th International Conference on Learning Representations","authors":"Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio","year":"2016","venue":"ICLR 2017 -Conference Track Proceedings","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dwork et al., 2006] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sen- sitivity in private data analysis. In Theory of cryptography conference, pages 265-284. Springer, 2006.","title":"Calibrating noise to sensitivity in private data analysis","authors":"Cynthia Dwork, Frank Mcsherry, Kobbi Nissim, Adam Smith","year":"2006","venue":"Theory of cryptography conference","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dwork et al., 2014] Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Founda- tions and Trends® in Theoretical Computer Science, 9(3- 4):211-407, 2014.","title":"The algorithmic foundations of differential privacy","authors":"Cynthia Dwork, Aaron Roth","year":"2014","venue":"Foundations and Trends® in Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dwork, 2008] Cynthia Dwork. Differential privacy: A sur- vey of results. In International conference on theory and applications of models of computation, pages 1-19. Springer, 2008.","title":"Differential privacy: A survey of results","authors":"Cynthia Dwork","year":"2008","venue":"International conference on theory and applications of models of computation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dwork, 2011] Cynthia Dwork. A firm foundation for pri- vate data analysis. Communications of the ACM, 54(1):86- 95, 2011.","title":"A firm foundation for private data analysis","authors":"Cynthia Dwork","year":"2011","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Juvekar et al., 2018] Chiraag Juvekar, Vinod Vaikun- tanathan, and Anantha Chandrakasan. {GAZELLE}: A low latency framework for secure neural network inference. In 27th USENIX Security Symposium (USENIX Security 18), pages 1651-1669, 2018.","title":"{GAZELLE}: A low latency framework for secure neural network inference","authors":"Juvekar","year":"2018","venue":"27th USENIX Security Symposium (USENIX Security 18)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Knott et al., 2021] Brian Knott, Shobha Venkataraman, Awni Hannun, Shubho Sengupta, Mark Ibrahim, and Lau- rens van der Maaten. Crypten: Secure multi-party compu- tation meets machine learning. Advances in Neural Infor- mation Processing Systems, 34:1-13, 2021.","title":"Crypten: Secure multi-party computation meets machine learning","authors":"Brian Knott, Shobha Venkataraman, Awni Hannun, Shubho Sengupta, Mark Ibrahim, Laurens Van Der Maaten","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kumar et al., 2020] Nishant Kumar, Mayank Rathee, Nis- hanth Chandran, Divya Gupta, Aseem Rastogi, and Rahul Sharma. Cryptflow: Secure tensorflow inference. In 2020 IEEE Symposium on Security and Privacy (SP), pages 336-353. IEEE, 2020.","title":"Cryptflow: Secure tensorflow inference","authors":"Nishant Kumar, Mayank Rathee, Nishanth Chandran, Divya Gupta, Aseem Rastogi, Rahul Sharma","year":"2020","venue":"IEEE Symposium on Security and Privacy (SP)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2012] Nan Li, Ivor W Tsang, and Zhi-Hua Zhou. Efficient optimization of performance measures by classi- fier adaptation. IEEE transactions on pattern analysis and machine intelligence, 35(6):1370-1382, 2012.","title":"Efficient optimization of performance measures by classifier adaptation","authors":"Nan Li, Ivor W Tsang, Zhi-Hua Zhou","year":"2012","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2019] Liping Li, Wei Xu, Tianyi Chen, Georgios B Giannakis, and Qing Ling. Rsa: Byzantine-robust stochas- tic aggregation methods for distributed learning from het- erogeneous datasets. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 33, pages 1544- 1551, 2019.","title":"Rsa: Byzantine-robust stochastic aggregation methods for distributed learning from heterogeneous datasets","authors":"Liping Li, Wei Xu, Tianyi Chen, Georgios B Giannakis, Qing Ling","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lindell, 2005] Yehida Lindell. Secure multiparty computa- tion for privacy preserving data mining. In Encyclopedia of Data Warehousing and Mining, pages 1005-1009. IGI global, 2005.","title":"Secure multiparty computation for privacy preserving data mining","authors":"Yehida Lindell, Lindell","year":"2005","venue":"Encyclopedia of Data Warehousing and Mining","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[McMahan et al., 2016] H Brendan McMahan, Eider Moore, Daniel Ramage, and Blaise Agüera y Arcas. Federated learning of deep networks using model averaging. arXiv preprint arXiv:1602.05629, pages 1-11, 2016.","title":"Federated learning of deep networks using model averaging","authors":"Eider H Brendan Mcmahan, Daniel Moore, Blaise Ramage, Agüera Y Arcas","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.","title":"Communication-efficient learning of deep networks from decentralized data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"Artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mohassel and Rindal, 2018] Payman Mohassel and Peter Rindal. Aby3: A mixed protocol framework for machine learning. In Proceedings of the 2018 ACM SIGSAC con- ference on computer and communications security, pages 35-52, 2018.","title":"Aby3: A mixed protocol framework for machine learning","authors":"Rindal Mohassel, Payman Mohassel, Peter Rindal","year":"2018","venue":"Proceedings of the 2018 ACM SIGSAC conference on computer and communications security","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pathak et al., 2010] Manas Pathak, Shantanu Rane, and Bhiksha Raj. Multiparty differential privacy via aggre- gation of locally trained classifiers. In Advances in Neu- ral Information Processing Systems, volume 23, page 1876-1884.","title":"Multiparty differential privacy via aggregation of locally trained classifiers","authors":"Manas Pathak, Shantanu Rane, Bhiksha Raj","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"Curran Associates, Inc., 2010.","title":null,"authors":"Curran Associates, Inc","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pedregosa et al., 2011] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Andreas Müller, Joel Nothman, Gilles Louppe, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. Scikit-learn: Machine learn- ing in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.","title":"Scikit-learn: Machine learning in Python","authors":"Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Andreas Müller, Joel Nothman, Gilles Louppe, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, Édouard Duchesnay","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rajkomar et al., 2019] Alvin Rajkomar, Jeffrey Dean, and Isaac Kohane. Machine learning in medicine. New Eng- land Journal of Medicine, 380(14):1347-1358, 2019.","title":"Machine learning in medicine","authors":"Rajkomar","year":"2019","venue":"New England Journal of Medicine","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rajkumar and Agarwal, 2012] Arun Rajkumar and Shivani Agarwal. A differentially private stochastic gradient de- scent algorithm for multiparty classification. In Artificial Intelligence and Statistics, pages 933-941. PMLR, 2012.","title":"A differentially private stochastic gradient descent algorithm for multiparty classification","authors":"Arun Rajkumar, Shivani Agarwal","year":"2012","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Savage, 2017] Neil Savage. Calculating disease. Nature, 550(7676):S115-S117, 2017.","title":"Calculating disease","authors":"Neil Savage","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shokri and Shmatikov, 2015] Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep learning. In Proceed- ings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1310-1321, 2015.","title":"Privacy-preserving deep learning","authors":"Reza Shokri, Vitaly Shmatikov","year":"2015","venue":"Proceedings of the 22nd ACM SIGSAC conference on computer and communications security","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sweeney, 2002] Latanya Sweeney. k-anonymity: A model for protecting privacy. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 10(05):557-570, 2002.","title":"k-anonymity: A model for protecting privacy","authors":"Latanya Sweeney","year":"2002","venue":"International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Waites and Cummings, 2021] Chris Waites and Rachel Cummings. Differentially private normalizing flows for privacy-preserving density estimation. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Soci- ety, pages 1000-1009, 2021.","title":"Differentially private normalizing flows for privacy-preserving density estimation","authors":"Chris Waites, Rachel Cummings","year":"2021","venue":"Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Warnat-Herresthal et al., 2021] Stefanie Warnat-Herresthal, Hartmut Schultze, Krishnaprasad Lingadahalli Shastry, Sathyanarayanan Manamohan, Saikat Mukherjee, Vishesh Garg, Ravi Sarveswara, Kristian Händler, Peter Pickkers, N Ahmad Aziz, et al. Swarm learning for decentral- ized and confidential clinical machine learning. Nature, 594(7862):265-270, 2021.","title":"Swarm learning for decentralized and confidential clinical machine learning","authors":"Stefanie Warnat-Herresthal, Hartmut Schultze, Krishnaprasad Lingadahalli Shastry, Sathyanarayanan Manamohan, Saikat Mukherjee, Vishesh Garg, Ravi Sarveswara, Kristian Händler, Peter Pickkers, Ahmad Aziz","year":"2021","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wu et al., 2019] Xi Zhu Wu, Song Liu, and Zhi Hua Zhou. Heterogeneous model reuse via optimizing multiparty multiclass margin. 36th International Conference on Machine Learning, ICML 2019, 2019-June:11862-11871, 2019.","title":"Heterogeneous model reuse via optimizing multiparty multiclass margin","authors":"Zhu Xi, Song Wu, Zhi Hua Liu, Zhou","year":"2019","venue":"36th International Conference on Machine Learning, ICML 2019","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wu et al., 2023] Xi-Zhu Wu, Wenkai Xu, Song Liu, and Zhi-Hua Zhou. Model reuse with reduced kernel mean embedding specification. IEEE Transactions on Knowl- edge and Data Engineering, 35(01):699-710, jan 2023.","title":"Model reuse with reduced kernel mean embedding specification","authors":"Xi-Zhu Wu, Wenkai Xu, Song Liu, Zhi-Hua Zhou","year":"2023","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xiao et al., 2017] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. (arXiv:1708.07747):1-6, September 2017.","title":"Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms","authors":"Han Xiao, Kashif Rasul, Roland Vollgraf","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang et al., 2017] Yang Yang, De-Chuan Zhan, Ying Fan, Yuan Jiang, and Zhi-Hua Zhou. Deep Learning for Fixed Model Reuse. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1):2831-2837, February 2017.","title":"Deep Learning for Fixed Model Reuse","authors":"Yang Yang, Ying De-Chuan Zhan, Yuan Fan, Zhi-Hua Jiang, Zhou","year":"2017","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2019] Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-19, 2019.","title":"Federated machine learning: Concept and applications","authors":"Qiang Yang, Yang Liu, Tianjian Chen, Yongxin Tong","year":"2019","venue":"ACM Transactions on Intelligent Systems and Technology (TIST)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yao, 1986] Andrew Chi-Chih Yao. How to generate and ex- change secrets. In 27th Annual Symposium on Foundations of Computer Science (sfcs 1986), pages 162-167. IEEE, 1986.","title":"How to generate and exchange secrets","authors":"Chi-Chih Andrew, Yao","year":"1986","venue":"27th Annual Symposium on Foundations of Computer Science (sfcs 1986)","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhou, 2012] Zhi-Hua Zhou. Ensemble methods: founda- tions and algorithms. CRC press, 2012.","title":"Ensemble methods: foundations and algorithms","authors":"Zhi-Hua Zhou","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhou, 2016] Zhi Hua Zhou. Learnware: on the future of ma- chine learning. Frontiers of Computer Science, 10(4):589- 590, 2016.","title":"Learnware: on the future of machine learning","authors":"Hua Zhi, Zhou","year":"2016","venue":"Frontiers of Computer Science","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}