{"type":"ScholarlyArticle","title":"On the Fairness Impacts of Private Ensembles Models","authors":[{"name":"Cuong Tran","slug":"cuong-tran","given":"Cuong","family":"Tran","affiliation":"Syracuse University"},{"name":"Ferdinando Fioretto","slug":"ferdinando-fioretto","given":"Ferdinando","family":"Fioretto","affiliation":"University of Virginia"}],"abstract":"The Private Aggregation of Teacher Ensembles (PATE) is a machine learning framework that enables the creation of private models through the combination of multiple \"teacher\" models and a \"student\" model. The student model learns to predict an output based on the voting of the teachers, and the resulting model satisfies differential privacy. PATE has been shown to be effective in creating private models in semi-supervised settings or when protecting data labels is a priority. \r\nThis paper explores whether the use of PATE can result in unfairness, and demonstrates that it can lead to accuracy disparities among groups of individuals. The paper also analyzes the algorithmic and data properties that contribute to these disproportionate impacts, why these aspects are affecting different groups disproportionately, and offers recommendations for mitigating these effects.","doi":"10.24963/ijcai.2023/57","url":"https://ijcai.org/proceedings/2023/57","pdf":"https://ijcai.org/proceedings/2023/0057.pdf","markdown":"https://ijcai.org/proceedings/2023/57.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/57","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"AI Ethics, Trust, Fairness","track":"Main Track","number":57,"pages":{"first":510,"last":518},"keywords":["Computer Vision: CV: Bias, fairness and privacy","Multidisciplinary Topics and Applications: MDA: Security and privacy","AI Ethics, Trust, Fairness: ETF: Trustworthy AI"],"videos":[],"references":[{"position":1,"raw":"[Abadi and et al., 2016] Abadi and et al. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, 2016.","title":"Deep learning with differential privacy","authors":"Abadi","year":"2016","venue":"Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bagdasaryan et al., 2019] Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential privacy has disparate impact on model accuracy. In Advances in Neu- ral Information Processing Systems, pages 15479-15488, 2019.","title":"Differential privacy has disparate impact on model accuracy","authors":"Eugene Bagdasaryan, Omid Poursaeed, Vitaly Shmatikov","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cummings et al., 2019] Rachel Cummings, Varun Gupta, Dhamma Kimpara, and Jamie Morgenstern. On the com- patibility of privacy and fairness. In Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization, pages 309-315, 2019.","title":"On the compatibility of privacy and fairness","authors":"Rachel Cummings, Varun Gupta, Dhamma Kimpara, Jamie Morgenstern","year":"2019","venue":"Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dwork et al., 2006] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sen- sitivity in private data analysis. In Theory of cryptography conference, pages 265-284. Springer, 2006.","title":"Calibrating noise to sensitivity in private data analysis","authors":"Cynthia Dwork, Frank Mcsherry, Kobbi Nissim, Adam Smith","year":"2006","venue":"Theory of cryptography conference","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dwork et al., 2014] Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Founda- tions and Trends R in Theoretical Computer Science, 9(3- 4):211-407, 2014.","title":"The algorithmic foundations of differential privacy","authors":"Cynthia Dwork, Aaron Roth","year":"2014","venue":"Foundations and Trends R in Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ekstrand et al., 2018] Michael D Ekstrand, Rezvan Joshaghani, and Hoda Mehrpouyan. Privacy for all: Ensuring fair and equitable privacy protections. In Con- ference on Fairness, Accountability and Transparency, pages 35-47, 2018.","title":"Privacy for all: Ensuring fair and equitable privacy protections","authors":"Rezvan Michael D Ekstrand, Hoda Joshaghani, Mehrpouyan","year":"2018","venue":"Conference on Fairness, Accountability and Transparency","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Farrand et al., 2020] Tom Farrand, Fatemehsadat Mireshghallah, Sahib Singh, and Andrew Trask. Neither private nor fair: Impact of data imbalance on utility and fairness in differential privacy. In Proceedings of the 2020 Workshop on Privacy-Preserving Machine Learning in Practice, pages 15-19, 2020.","title":"Neither private nor fair: Impact of data imbalance on utility and fairness in differential privacy","authors":"Farrand","year":"2020","venue":"Proceedings of the 2020 Workshop on Privacy-Preserving Machine Learning in Practice","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fioretto et al., 2020] Ferdinando Fioretto, Pascal Van Hen- tenryck, Terrence W. K. Mak, Cuong Tran, Federico Baldo, and Michele Lombardi. Lagrangian duality for constrained deep learning. In Machine Learning and Knowledge Discovery in Databases. European Confer- ence, ECML PKDD 2020, volume 12461 of Lecture Notes in Computer Science, pages 118-135, 2020.","title":"Lagrangian duality for constrained deep learning","authors":"Ferdinando Fioretto, Pascal Van Hentenryck, Terrence W K Mak, Cuong Tran, Federico Baldo, Michele Lombardi","year":"2020","venue":"Machine Learning and Knowledge Discovery in Databases. European Conference, ECML PKDD 2020","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fioretto et al., 2021] Ferdinando Fioretto, Cuong Tran, and Pascal Van Hentenryck. Decision making with differential privacy under a fairness lens. arXiv preprint arXiv: Arxiv- 2105.07513, 2021.","title":"Decision making with differential privacy under a fairness lens","authors":"Ferdinando Fioretto, Cuong Tran, Pascal Van Hentenryck","year":"2021","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2021/78"},{"position":10,"raw":"[Fioretto et al., 2022] Ferdinando Fioretto, Cuong Tran, Pas- cal Van Hentenryck, and Keyu Zhu. Differential privacy and fairness in decisions and learning tasks: A survey. In In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pages 5470-5477, 2022.","title":"Differential privacy and fairness in decisions and learning tasks: A survey","authors":"Ferdinando Fioretto, Cuong Tran, Pascal Van Hentenryck, Keyu Zhu","year":"2022","venue":"Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2022/766"},{"position":11,"raw":"[Ghazi et al., 2021] Badih Ghazi, Noah Golowich, Ravi Ku- mar, Pasin Manurangsi, and Chiyuan Zhang. Deep learn- ing with label differential privacy. In Marc'Aurelio Ran- zato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neu- ral Information Processing Systems 34, 2021.","title":"Deep learning with label differential privacy","authors":"Ghazi","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jagielski et al., 2018] Matthew Jagielski, Michael Kearns, Jieming Mao, Alina Oprea, Aaron Roth, Saeed Sharifi- Malvajerdi, and Jonathan Ullman. Differentially private fair learning. arXiv preprint arXiv:1812.02696, 2018.","title":"Differentially private fair learning","authors":"Matthew Jagielski, Michael Kearns, Jieming Mao, Alina Oprea, Aaron Roth, Saeed Sharifi-Malvajerdi, Jonathan Ullman","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lahoti et al., 2020] Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed Chi. Fairness without demographics through ad- versarially reweighted learning. Advances in neural infor- mation processing systems, 33:728-740, 2020.","title":"Fairness without demographics through adversarially reweighted learning","authors":"Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, Ed Chi","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mahjoubfar et al., 2017] Ata Mahjoubfar, Claire Lifan Chen, and Bahram Jalali. Deep learning and classifica- tion. In Artificial Intelligence in Label-free Microscopy, pages 73-85. Springer, 2017.","title":"Deep learning and classification","authors":"Ata Mahjoubfar, Claire Lifan Chen, Bahram Jalali","year":"2017","venue":"Artificial Intelligence in Label-free Microscopy","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mozannar et al., 2020] Hussein Mozannar, Mesrob I. Ohan- nessian, and Nathan Srebro. Fair learning with private de- mographic data. In Proceedings of the 37th International Conference on Machine Learning, 2020.","title":"Hussein Mozannar, Mesrob I. Ohannessian, and Nathan Srebro. Fair learning with private demographic data","authors":"Mozannar","year":"2020","venue":"Proceedings of the 37th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Papernot and Steinke, 2021] Nicolas Papernot and Thomas Steinke. Hyperparameter tuning with renyi differential pri- vacy. arXiv preprint arXiv:2110.03620, 2021.","title":"Nicolas Papernot and Thomas Steinke. Hyperparameter tuning with renyi differential privacy","authors":"Steinke Papernot","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Papernot et al., 2016] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neu- ral networks. In 2016 IEEE symposium on security and privacy (SP), pages 582-597. IEEE, 2016.","title":"Distillation as a defense to adversarial perturbations against deep neural networks","authors":"Nicolas Papernot, Patrick Mcdaniel, Xi Wu, Somesh Jha, Ananthram Swami","year":"2016","venue":"IEEE symposium on security and privacy (SP)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Papernot et al., 2018] Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Ulfar Erlingsson. Scalable private learning with pate. In Inter- national Conference on Learning Representations, 2018.","title":"Scalable private learning with pate","authors":"Nicolas Papernot, Shuang Song, Ilya Mironov, Kunal Ananth Raghunathan, Ulfar Talwar, Erlingsson","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Patrini et al., 2014] Giorgio Patrini, Richard Nock, Paul Rivera, and Tiberio Caetano. (almost) no label no cry. Advances in Neural Information Processing Systems, 27:190-198, 2014.","title":"almost) no label no cry","authors":"Giorgio Patrini, Richard Nock, Paul Rivera, Tiberio Caetano","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pujol et al., 2020] David Pujol, Ryan McKenna, Satya Kup- pam, Michael Hay, Ashwin Machanavajjhala, and Gerome Miklau. Fair decision making using privacy-protected data. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 189-199, 2020.","title":"Fair decision making using privacy-protected data","authors":"David Pujol, Ryan Mckenna, Satya Kuppam, Michael Hay, Ashwin Machanavajjhala, Gerome Miklau","year":"2020","venue":"Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sanyal et al., 2022] Amartya Sanyal, Yaxi Hu, and Fanny Yang. How unfair is private learning? In Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, pages 1738-1748, 2022.","title":"How unfair is private learning?","authors":"Amartya Sanyal, Yaxi Hu, Fanny Yang","year":"2022","venue":"Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shi et al., 2023] Zheng Shi, Abdurakhmon Sadiev, Nicolas Loizou, Peter Richtárik, and Martin Takáč. AI-SARAH: Adaptive and implicit stochastic recursive gradient meth- ods. Transactions on Machine Learning Research, 2023.","title":"AI-SARAH: Adaptive and implicit stochastic recursive gradient methods","authors":"Zheng Shi, Abdurakhmon Sadiev, Nicolas Loizou, Peter Richtárik, Martin Takáč","year":"2023","venue":"Transactions on Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tran and Fioretto, 2023] Cuong Tran and Ferdinando Fioretto. On the fairness impacts of private ensembles models. arXiv preprint arXiv:2305.11807, 2023.","title":"On the fairness impacts of private ensembles models","authors":"Tran, Fioretto ;, Cuong Tran, Ferdinando Fioretto","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tran et al., 2021a] Cuong Tran, My Dinh, and Ferdinando Fioretto. Differentially private empirical risk minimization under the fairness lens. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 27555- 27565.","title":"Differentially private empirical risk minimization under the fairness lens","authors":"Tran","year":"2021","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"Curran Associates, Inc., 2021.","title":null,"authors":"Curran Associates, Inc","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tran et al., 2021b] Cuong Tran, My H. Dinh, and Ferdi- nando Fioretto. Differentially private deep learning under the fairness lens. arXiv preprint arXiv: Arxiv-2106.02674, 2021.","title":"Differentially private deep learning under the fairness lens","authors":"Tran","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tran et al., 2021c] Cuong Tran, Ferdinando Fioretto, and Pascal Van Hentenryck. Differentially private and fair deep learning: A lagrangian dual approach. In Thirty-Fifth AAAI Conference on Artificial Intelligence, pages 9932- 9939. AAAI Press, 2021.","title":"Differentially private and fair deep learning: A lagrangian dual approach","authors":"Tran","year":"2021","venue":"Thirty-Fifth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tran et al., 2021d] Cuong Tran, Ferdinando Fioretto, Pas- cal Van Hentenryck, and Zhiyan Yao. Decision making with differential privacy under a fairness lens. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 560-566, 2021.","title":"Decision making with differential privacy under a fairness lens","authors":"Tran","year":"2021","venue":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event","doi":null,"url":null,"internal_url":"/proceedings/2021/78"},{"position":29,"raw":"[Uniyal et al., 2021] Archit Uniyal, Rakshit Naidu, Sasikanth Kotti, Sahib Singh, Patrik Joslin Kenfack, Fatemehsadat Mireshghallah, and Andrew Trask. Dp-sgd vs pate: Which has less disparate impact on model accuracy? arXiv, 2106.12576, 2021.","title":"Dp-sgd vs pate: Which has less disparate impact on model accuracy?","authors":"Archit Uniyal, Rakshit Naidu, Sasikanth Kotti, Sahib Singh, Patrik Joslin Kenfack, Fatemehsadat Mireshghallah, Andrew Trask","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2017] Jiaqi Zhang, Kai Zheng, Wenlong Mou, and Liwei Wang. Efficient private erm for smooth ob- jectives. In Proceedings of the Twenty-Sixth Interna- tional Joint Conference on Artificial Intelligence, IJCAI- 17, pages 3922-3928, 2017.","title":"Efficient private erm for smooth objectives","authors":"Jiaqi Zhang, Kai Zheng, Wenlong Mou, Liwei Wang","year":"2017","venue":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17","doi":null,"url":null,"internal_url":"/proceedings/2017/548"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}