{"type":"ScholarlyArticle","title":"Adapt to Adaptation: Learning Personalization for Cross-Silo Federated Learning","authors":[{"name":"Jun Luo","slug":"jun-luo","given":"Jun","family":"Luo","affiliation":"Intelligent Systems Program , University of Pittsburgh"},{"name":"Shandong Wu","slug":"shandong-wu","given":"Shandong","family":"Wu","affiliation":"Intelligent Systems Program , University of Pittsburgh; Department of Radiology , University of Pittsburgh; Department of Biomedical Informatics , University of Pittsburgh; Department of Bioengineering , University of Pittsburgh"}],"abstract":"Conventional federated learning (FL) trains one global model for a federation of clients with decentralized data, reducing the privacy risk of centralized training. However, the distribution shift across non-IID datasets, often poses a challenge to this one-model-fits-all solution. Personalized FL aims to mitigate this issue systematically. In this work, we propose APPLE, a personalized cross-silo FL framework that adaptively learns how much each client can benefit from other clients’ models. We also introduce a method to flexibly control the focus of training APPLE between global and local objectives. We empirically evaluate our method's convergence and generalization behaviors, and perform extensive experiments on two benchmark datasets and two medical imaging datasets under two non-IID settings. The results show that the proposed personalized FL framework, APPLE, achieves state-of-the-art performance compared to several other personalized FL approaches in the literature. The code is publicly available at https://github.com/ljaiverson/pFL-APPLE.","doi":"10.24963/ijcai.2022/301","url":"https://ijcai.org/proceedings/2022/301","pdf":"https://ijcai.org/proceedings/2022/0301.pdf","markdown":"https://ijcai.org/proceedings/2022/301.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/301","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Data Mining","track":"Main Track","number":301,"pages":{"first":2166,"last":2173},"keywords":["Knowledge Representation and Reasoning: Reasong about actions","Computer Vision: Biomedical Image Analysis","Data Mining: Federated Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986770","length_sec":84,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984913","length_sec":340,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Acar et al., 2020] Durmus Alp Emre Acar, Yue Zhao, Ra- mon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dy- namic regularization. In International Conference on Learning Representations, 2020.","title":"Federated learning based on dynamic regularization","authors":"Durmus Alp, Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, Venkatesh Saligrama","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Deng et al., 2020] Yuyang Deng, Mohammad Mahdi Ka- mani, and Mehrdad Mahdavi. Adaptive personalized fed- erated learning. arXiv preprint arXiv:2003.13461, 2020.","title":"Adaptive personalized federated learning","authors":"Yuyang Deng, Mohammad Mahdi Kamani, Mehrdad Mahdavi","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fallah et al., 2020] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learn- ing: A meta-learning approach. arXiv preprint arXiv:2002.07948, 2020.","title":"Personalized federated learning: A meta-learning approach","authors":"Alireza Fallah, Aryan Mokhtari, Asuman Ozdaglar","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1126-1135. PMLR, 2017.","title":"Model-agnostic meta-learning for fast adaptation of deep networks","authors":"Finn","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hard et al., 2018] Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Franc ¸oise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604, 2018.","title":"Federated learning for mobile keyboard prediction","authors":"Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Sean Franc ¸oise Beaufays, Hubert Augenstein, Chloé Eichner, Daniel Kiddon, Ramage","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hsieh et al., 2020] Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. In International Con- ference on Machine Learning, pages 4387-4398. PMLR, 2020.","title":"The non-iid data quagmire of decentralized machine learning","authors":"Kevin Hsieh, Amar Phanishayee, Onur Mutlu, Phillip Gibbons","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2021] Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, and Yong Zhang. Personalized cross-silo federated learning on non- iid data. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 35, pages 7865-7873, 2021.","title":"Personalized cross-silo federated learning on noniid data","authors":"Huang","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiang et al., 2019] Yihan Jiang, Jakub Konečnỳ, Keith Rush, and Sreeram Kannan. Improving federated learning personalization via model agnostic meta learning. arXiv preprint arXiv:1909.12488, 2019.","title":"Improving federated learning personalization via model agnostic meta learning","authors":"Yihan Jiang, Jakub Konečnỳ, Keith Rush, Sreeram Kannan","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kairouz et al., 2019] Peter Kairouz, H Brendan McMa- han, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977, 2019.","title":"Advances and open problems in federated learning","authors":"Peter Kairouz, Brendan Mcmahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Nitin Arjun, Kallista Bhagoji, Zachary Bonawitz, Graham Charles, Rachel Cormode, Cummings","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Karimireddy et al., 2020] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebas- tian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132-5143. PMLR, 2020.","title":"Scaffold: Stochastic controlled averaging for federated learning","authors":"Praneeth Sai, Satyen Karimireddy, Mehryar Kale, Sashank Mohri, Sebastian Reddi, Ananda Stich, Suresh Theertha","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Khodak et al., 2019] Mikhail Khodak, Maria-Florina Bal- can, and Ameet Talwalkar. Adaptive gradient-based meta- learning methods. arXiv preprint arXiv:1906.02717, 2019.","title":"Adaptive gradient-based metalearning methods","authors":"Mikhail Khodak, Maria-Florina Balcan, Ameet Talwalkar","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kulkarni et al., 2020] Viraj Kulkarni, Milind Kulkarni, and Aniruddha Pant. Survey of personalization techniques for federated learning. In 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4), pages 794-797. IEEE, 2020.","title":"Survey of personalization techniques for federated learning","authors":"Viraj Kulkarni, Milind Kulkarni, Aniruddha Pant","year":"2020","venue":"Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2019] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189, 2019.","title":"On the convergence of fedavg on non-iid data","authors":"Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, Zhihua Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2020] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Pro- ceedings of Machine Learning and Systems, 2:429-450, 2020.","title":"Federated optimization in heterogeneous networks","authors":"Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, Virginia Smith","year":"2020","venue":"Proceedings of Machine Learning and Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.","title":"Communication-efficient learning of deep networks from decentralized data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"Artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rieke et al., 2020] Nicola Rieke, Jonny Hancox, Wenqi Li, Fausto Milletari, Holger R Roth, Shadi Albarqouni, Spyri- don Bakas, Mathieu N Galtier, Bennett A Landman, Klaus Maier-Hein, et al. The future of digital health with feder- ated learning. NPJ digital medicine, 3(1):1-7, 2020.","title":"The future of digital health with federated learning","authors":"Nicola Rieke, Jonny Hancox, Wenqi Li, Fausto Milletari, Shadi Holger R Roth, Spyridon Albarqouni, Bakas, N Mathieu, Galtier, Klaus Bennett A Landman, Maier-Hein","year":"2020","venue":"NPJ digital medicine","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rockafellar, 1976] R Tyrrell Rockafellar. Monotone opera- tors and the proximal point algorithm. SIAM journal on control and optimization, 14(5):877-898, 1976.","title":"Monotone operators and the proximal point algorithm","authors":"Rockafellar Tyrrell","year":"1976","venue":"SIAM journal on control and optimization","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sahu et al., 2018] Anit Kumar Sahu, Tian Li, Maziar San- jabi, Manzil Zaheer, Ameet Talwalkar, and Virginia Smith. On the convergence of federated optimization in heteroge- neous networks. arXiv preprint arXiv:1812.06127, 3:3, 2018.","title":"On the convergence of federated optimization in heterogeneous networks","authors":"Anit Kumar Sahu, Tian Li, Maziar Sanjabi, Manzil Zaheer, Ameet Talwalkar, Virginia Smith","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sattler et al., 2020] Felix Sattler, Klaus-Robert Müller, and Wojciech Samek. Clustered federated learning: Model- agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural networks and learning systems, 2020.","title":"Clustered federated learning: Modelagnostic distributed multitask optimization under privacy constraints","authors":"Felix Sattler, Klaus-Robert Müller, Wojciech Samek","year":"2020","venue":"IEEE transactions on neural networks and learning systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Smith et al., 2017] Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning. arXiv preprint arXiv:1705.10467, 2017.","title":"Federated multi-task learning","authors":"Smith","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2019a] Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Eichner, Franc ¸oise Beaufays, and Daniel Ramage. Federated evaluation of on-device personalization. arXiv preprint arXiv:1910.10252, 2019.","title":"Federated evaluation of on-device personalization","authors":"Wang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2019b] Yiru Wang, Weihao Gan, Jie Yang, Wei Wu, and Junjie Yan. Dynamic curriculum learning for imbalanced data classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5017-5026, 2019.","title":"Dynamic curriculum learning for imbalanced data classification","authors":"Wang","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2018] Yuhui Xu, Yongzhuang Wang, Aojun Zhou, Weiyao Lin, and Hongkai Xiong. Deep neural net- work compression with single and multiple level quantiza- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.","title":"Deep neural network compression with single and multiple level quantization","authors":"Yuhui Xu, Yongzhuang Wang, Aojun Zhou, Weiyao Lin, Hongkai Xiong","year":"2018","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2018] Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Franc ¸oise Beaufays. Applied federated learning: Improving google keyboard query suggestions. arXiv preprint arXiv:1812.02903, 2018.","title":"Applied federated learning: Improving google keyboard query suggestions","authors":"Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, Franc, Beaufays","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2021] Jiancheng Yang, Rui Shi, and Bingbing Ni. Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis. In 2021 IEEE 18th International Symposium on Biomedical Imag- ing (ISBI), pages 191-195. IEEE, 2021.","title":"Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis","authors":"Jiancheng Yang, Rui Shi, Bingbing Ni","year":"2021","venue":"IEEE 18th International Symposium on Biomedical Imaging (ISBI)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2020] Michael Zhang, Karan Sapra, Sanja Fi- dler, Serena Yeung, and Jose M Alvarez. Personalized fed- erated learning with first order model optimization. arXiv preprint arXiv:2012.08565, 2020.","title":"Personalized federated learning with first order model optimization","authors":"Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, Jose M Alvarez","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhao et al., 2018] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Fed- erated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018.","title":"Federated learning with non-iid data","authors":"Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, Vikas Chandra","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}