{"type":"ScholarlyArticle","title":"Personalized Federated Learning with Contextualized Generalization","authors":[{"name":"Xueyang Tang","slug":"xueyang-tang","given":"Xueyang","family":"Tang","affiliation":"Department of Computing , The Hong Kong Polytechnic University , Hong Kong , China"},{"name":"Song Guo","slug":"song-guo","given":"Song","family":"Guo","affiliation":"Department of Computing , The Hong Kong Polytechnic University , Hong Kong , China; The Hong Kong Polytechnic University Shenzhen Research Institute , Shenzhen , China"},{"name":"Jingcai Guo","slug":"jingcai-guo","given":"Jingcai","family":"Guo","affiliation":"Department of Computing , The Hong Kong Polytechnic University , Hong Kong , China; The Hong Kong Polytechnic University Shenzhen Research Institute , Shenzhen , China"}],"abstract":"The prevalent personalized federated learning (PFL) usually pursues a trade-off between personalization and generalization by maintaining a shared global model to guide the training process of local models. However, the sole global model may easily transfer deviated context knowledge to some local models when multiple latent contexts exist across the local datasets. In this paper, we propose a novel concept called contextualized generalization (CG) to provide each client with fine-grained context knowledge that can better fit the local data distributions and facilitate faster model convergence, based on which we properly design a framework of PFL, dubbed CGPFL. We conduct detailed theoretical analysis, in which the convergence guarantee is presented and a speedup of order 1/2 w.r.t the number of contexts over most existing methods is granted. To quantitatively study the generalization-personalization trade-off, we introduce the generalization error measure and prove that the proposed CGPFL can achieve a better trade-off than existing solutions. Moreover, our theoretical analysis further inspires a heuristic algorithm to find a near-optimal trade-off in CGPFL. Experimental results on multiple real-world datasets show that our approach surpasses the state-of-the-art methods on test accuracy by a significant margin.","doi":"10.24963/ijcai.2022/311","url":"https://ijcai.org/proceedings/2022/311","pdf":"https://ijcai.org/proceedings/2022/0311.pdf","markdown":"https://ijcai.org/proceedings/2022/311.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/311","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Data Mining","track":"Main Track","number":311,"pages":{"first":2241,"last":2247},"keywords":["Data Mining: Federated Learning","Data Mining: Mining Heterogenous Data","Humans and AI: Personalization and User Modeling","Data Mining: Parallel, Distributed and Cloud-based High Performance Mining","Machine Learning: Optimisation"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986937","length_sec":91,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985080","length_sec":349,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Arivazhagan et al., 2019] Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. arXiv preprint arXiv:1912.00818, 2019.","title":"Federated learning with personalization layers","authors":"Vinay Manoj Ghuhan Arivazhagan, Aaditya Kumar Aggarwal, Sunav Singh, Choudhary","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Briggs et al., 2020] Christopher Briggs, Zhong Fan, and Pe- ter Andras. Federated learning with hierarchical clustering of local updates to improve training on non-iid data. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-9. IEEE, 2020.","title":"Federated learning with hierarchical clustering of local updates to improve training on non-iid data","authors":"Christopher Briggs, Zhong Fan, Peter Andras","year":"2020","venue":"2020 International Joint Conference on Neural Networks (IJCNN)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2020] Yuyang Deng, Mohammad Mahdi Ka- mani, and Mehrdad Mahdavi. Adaptive personalized fed- erated learning. arXiv preprint arXiv:2003.13461, 2020.","title":"Adaptive personalized federated learning","authors":"Yuyang Deng, Mohammad Mahdi Kamani, Mehrdad Mahdavi","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fallah et al., 2020] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33, 2020.","title":"Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach","authors":"Alireza Fallah, Aryan Mokhtari, Asuman Ozdaglar","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ghosh et al., 2020] Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural In- formation Processing Systems, 33, 2020.","title":"An efficient framework for clustered federated learning","authors":"Avishek Ghosh, Jichan Chung, Dong Yin, Kannan Ramchandran","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hanzely and Richtárik, 2020] Filip Hanzely and Peter Richtárik. Federated learning of a mixture of global and local models. arXiv preprint arXiv:2002.05516, 2020.","title":"Filip Hanzely and Peter Richtárik. Federated learning of a mixture of global and local models","authors":"Richtárik Hanzely","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hanzely et al., 2020] Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtarik. Lower bounds and optimal algorithms for personalized federated learning. Advances in Neural Information Processing Systems, 33, 2020.","title":"Lower bounds and optimal algorithms for personalized federated learning","authors":"Hanzely","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hsieh et al., 2020] Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. In International Con- ference on Machine Learning, pages 4387-4398. PMLR, 2020.","title":"The non-iid data quagmire of decentralized machine learning","authors":"Kevin Hsieh, Amar Phanishayee, Onur Mutlu, Phillip Gibbons","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kairouz et al., 2019] Peter Kairouz, H Brendan McMa- han, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Gra- ham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977, 2019.","title":"Advances and open problems in federated learning","authors":"Peter Kairouz, Brendan Mcmahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Nitin Arjun, Keith Bhagoji, Zachary Bonawitz, Graham Charles, Rachel Cormode, Cummings","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Karimireddy et al., 2020] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebas- tian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132-5143. PMLR, 2020.","title":"Scaffold: Stochastic controlled averaging for federated learning","authors":"Praneeth Sai, Satyen Karimireddy, Mehryar Kale, Sashank Mohri, Sebastian Reddi, Ananda Stich, Suresh Theertha","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krizhevsky, 2009] Alex Krizhevsky. Learning multiple lay- ers of features from tiny images. Technical report, Univer- sity of Toronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2021] Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learn- ing through personalization. In International Conference on Machine Learning, pages 6357-6368. PMLR, 2021.","title":"Ditto: Fair and robust federated learning through personalization","authors":"Tian Li, Shengyuan Hu, Ahmad Beirami, Virginia Smith","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lloyd, 1982] Stuart Lloyd. Least squares quantization in pcm. IEEE transactions on information theory, 28(2):129-137, 1982.","title":"Least squares quantization in pcm","authors":"Stuart Lloyd","year":"1982","venue":"IEEE transactions on information theory","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mansour et al., 2020] Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002.10619, 2020.","title":"Three approaches for personalization with applications to federated learning","authors":"Yishay Mansour, Mehryar Mohri, Jae Ro, Ananda Theertha, Suresh","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.","title":"Communication-efficient learning of deep networks from decentralized data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"Artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Reddi et al., 2020] Sashank J Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečnỳ, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Confer- ence on Learning Representations, 2020.","title":"Adaptive federated optimization","authors":"J Sashank, Zachary Reddi, Manzil Charles, Zachary Zaheer, Keith Garrett, Jakub Rush, Sanjiv Konečnỳ, Hugh Brendan Kumar, Mcmahan","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sattler et al., 2020] Felix Sattler, Klaus-Robert Müller, and Wojciech Samek. Clustered federated learning: Model- agnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems, 2020.","title":"Clustered federated learning: Modelagnostic distributed multitask optimization under privacy constraints","authors":"Felix Sattler, Klaus-Robert Müller, Wojciech Samek","year":"2020","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Smith et al., 2017] Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. Advances in neural information processing systems, 30, 2017.","title":"Federated multi-task learning","authors":"Smith","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[T Dinh et al., 2020] Canh T Dinh, Nguyen Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes. Advances in Neural Information Pro- cessing Systems, 33, 2020.","title":"Personalized federated learning with moreau envelopes","authors":"Nguyen Canh T Dinh, Tuan Dung Tran, Nguyen","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tang et al., 2021] Xueyang Tang, Song Guo, and Jingcai Guo. Personalized federated learning with clustered gen- eralization. arXiv preprint arXiv:2106.13044, 2021.","title":"Personalized federated learning with clustered generalization","authors":"Tang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xiao et al., 2017] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for bench- marking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.","title":"Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms","authors":"Han Xiao, Kashif Rasul, Roland Vollgraf","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2021] Jie Zhang, Song Guo, Xiaosong Ma, Haozhao Wang, Wenchao Xu, and Feijie Wu. Parameter- ized knowledge transfer for personalized federated learn- ing. Advances in Neural Information Processing Systems, 34, 2021.","title":"Parameterized knowledge transfer for personalized federated learning","authors":"Jie Zhang, Song Guo, Xiaosong Ma, Haozhao Wang, Wenchao Xu, Feijie Wu","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}