{"type":"ScholarlyArticle","title":"FedDUAP: Federated Learning with Dynamic Update and Adaptive Pruning Using Shared Data on the Server","authors":[{"name":"Hong Zhang","slug":"hong-zhang","given":"Hong","family":"Zhang","affiliation":"School of Computer Science and Technology , Soochow University , China,"},{"name":"Ji Liu","slug":"ji-liu","given":"Ji","family":"Liu","affiliation":"Baidu Inc. , China,"},{"name":"Juncheng Jia","slug":"juncheng-jia","given":"Juncheng","family":"Jia","affiliation":"School of Computer Science and Technology , Soochow University , China,; Collaborative Innovation Center of Novel Software Technology and Industrialization , China,"},{"name":"Yang Zhou","slug":"yang-zhou","given":"Yang","family":"Zhou","affiliation":"Auburn University , United States,"},{"name":"Huaiyu Dai","slug":"huaiyu-dai","given":"Huaiyu","family":"Dai","affiliation":"North Carolina State University , United States"},{"name":"Dejing Dou","slug":"dejing-dou","given":"Dejing","family":"Dou","affiliation":"Baidu Inc. , China,"}],"abstract":"Despite achieving remarkable performance, Federated Learning (FL) suffers from two critical challenges, i.e., limited computational resources and low training efficiency. In this paper, we propose a novel FL framework, i.e., FedDUAP, with two original contributions, to exploit the insensitive data on the server and the decentralized data in edge devices to further improve the training efficiency. First, a dynamic server update algorithm is designed to exploit the insensitive data on the server, in order to dynamically determine the optimal steps of the server update for improving the convergence and accuracy of the global model. Second, a layer-adaptive model pruning method is developed to perform unique pruning operations adapted to the different dimensions and importance of multiple layers, to achieve a good balance between efficiency and effectiveness. By integrating the two original techniques together, our proposed FL model, FedDUAP, significantly outperforms baseline approaches in terms of accuracy (up to 4.8% higher), efficiency (up to 2.8 times faster), and computational cost (up to 61.9% smaller).","doi":"10.24963/ijcai.2022/385","url":"https://ijcai.org/proceedings/2022/385","pdf":"https://ijcai.org/proceedings/2022/0385.pdf","markdown":"https://ijcai.org/proceedings/2022/385.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/385","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Knowledge Representation and Reasoning","track":"Main Track","number":385,"pages":{"first":2776,"last":2782},"keywords":["Data Mining: Federated Learning","Knowledge Representation and Reasoning: Reasong about actions"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986868","length_sec":84,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985011","length_sec":336,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Amazon, 2021] Amazon. Amazon cloud. https://aws. amazon.com, 2021. Online; accessed 30/12/2021.","title":"Amazon. Amazon cloud","authors":null,"year":"2021","venue":null,"doi":null,"url":"https://aws.amazon.com","internal_url":null},{"position":2,"raw":"[CCP, 2018] California consumer privacy act home page. https://www.caprivacy.org/, 2018. Online; accessed 14/02/2021.","title":"California consumer privacy act home page","authors":null,"year":"2018","venue":null,"doi":null,"url":"https://www.caprivacy.org/,2018","internal_url":null},{"position":3,"raw":"[Fuglede and Topsoe, 2004] Bent Fuglede and Flemming Topsoe. Jensen-shannon divergence and Hilbert space em- bedding. In Int. Symposium on Information Theory (ISIT), page 31, 2004.","title":"Bent Fuglede and Flemming Topsoe. Jensen-shannon divergence and Hilbert space embedding","authors":"Topsoe Fuglede","year":"2004","venue":"Int. Symposium on Information Theory (ISIT)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE/CVF Conf. on Computer Vision and Pat- tern Recognition (CVPR), pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2020] Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group Knowledge Transfer: Fed- erated Learning of Large CNNs at the Edge. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.","title":"Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge","authors":"He","year":"2020","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jeong et al., 2018] Eunjeong Jeong, Seungeun Oh, Hye- sung Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim. Communication-efficient on-device machine learn- ing: Federated distillation and augmentation under non-iid private data. arXiv preprint arXiv:1811.11479, 2018.","title":"Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data","authors":"Jeong","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jiang et al., 2019] Yuang Jiang, Shiqiang Wang, Victor Valls, Bong Jun Ko, Wei-Han Lee, Kin K Leung, and Leandros Tassiulas. Model pruning enables efficient federated learning on edge devices. arXiv preprint arXiv:1909.12326, 2019.","title":"Model pruning enables efficient federated learning on edge devices","authors":"Yuang Jiang, Shiqiang Wang, Victor Valls, Bong Jun Ko, Wei-Han Lee, Kin K Leung, Leandros Tassiulas","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kairouz et al., 2021] Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, and Mehdi Bennis et al. Advances and open problems in federated learning. Foun- dations and Trends® in Machine Learning, 14(1), 2021.","title":"Advances and open problems in federated learning","authors":"H Brendan Peter Kairouz, Brendan Mcmahan, Aurélien Avent, Mehdi Bellet, Bennis","year":"2021","venue":"Foundations and Trends® in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Konečnỳ et al., 2016] Jakub Konečnỳ, H Brendan McMa- han, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492, 2016.","title":"Federated learning: Strategies for improving communication efficiency","authors":"Jakub Konečnỳ, Brendan Mcmahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, Dave Bacon","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny im- ages. Technical report, University of Toronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kullback, 1997] Solomon Kullback. Information theory and statistics. Courier Corporation, 1997.","title":"Information theory and statistics","authors":"Kullback","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lai et al., 2021] Fan Lai, Xiangfeng Zhu, Harsha V Mad- hyastha, and Mosharaf Chowdhury. Oort: Efficient feder- ated learning via guided participant selection. In USENIX Symposium on Operating Systems Design and Implemen- tation (OSDI), pages 19-35, 2021.","title":"Oort: Efficient federated learning via guided participant selection","authors":"Fan Lai, Xiangfeng Zhu, Mosharaf Harsha V Madhyastha, Chowdhury","year":"2021","venue":"USENIX Symposium on Operating Systems Design and Implementation (OSDI)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2018] He Li, Kaoru Ota, and Mianxiong Dong. Learning iot in edge: Deep learning for the internet of things with edge computing. IEEE Network, 32(1):96- 101, 2018.","title":"Learning iot in edge: Deep learning for the internet of things with edge computing","authors":"Li","year":"2018","venue":"IEEE Network","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2021] Qinbin Li, Bingsheng He, and Dawn Song. Practical one-shot federated learning for cross-silo setting. In Int. Joint Conf. on Artificial Intelligence (IJCAI), 2021.","title":"He, and Dawn Song. Practical one-shot federated learning for cross-silo setting","authors":"Li","year":"2021","venue":"Int. Joint Conf. on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2020a] Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 1529-1538, 2020.","title":"Hrank: Filter pruning using high-rank feature map","authors":"Lin","year":"2020","venue":"IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2020b] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.","title":"Ensemble distillation for robust model fusion in federated learning","authors":"Lin","year":"2020","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2022] Ji Liu, Jizhou Huang, Yang Zhou, Xuhong Li, Shilei Ji, Haoyi Xiong, and Dejing Dou. From dis- tributed machine learning to federated learning: a sur- vey. Knowledge and Information Systems, 64(4):885-917, 2022.","title":"From distributed machine learning to federated learning: a survey","authors":"Ji Liu, Jizhou Huang, Yang Zhou, Xuhong Li, Shilei Ji, Haoyi Xiong, Dejing Dou","year":"2022","venue":"Knowledge and Information Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics (AISTATS), pages 1273-1282, 2017.","title":"Communication-efficient learning of deep networks from decentralized data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"Artificial Intelligence and Statistics (AISTATS)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nagalapatti and Narayanam, 2021] Lokesh Nagalapatti and Ramasuri Narayanam. Game of gradients: Mitigating ir- relevant clients in federated learning. AAAI Conf. on Arti- ficial Intelligence, 35(10):9046-9054, 2021.","title":"Lokesh Nagalapatti and Ramasuri Narayanam. Game of gradients: Mitigating irrelevant clients in federated learning","authors":"Narayanam Nagalapatti","year":"2021","venue":"AAAI Conf. on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In Int. Conf. on Learning Representations (ICLR), 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"Int. Conf. on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2020] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective in- consistency problem in heterogeneous federated optimiza- tion. Advances in Neural Information Processing Systems (NeurIPS), 2020.","title":"Tackling the objective inconsistency problem in heterogeneous federated optimization","authors":"Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, H Vincent Poor","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yoshida et al., 2020] Naoya Yoshida, Takayuki Nishio, Masahiro Morikura, Koji Yamamoto, and Ryo Yonetani. Hybrid-FL for wireless networks: Cooperative learning mechanism using non-iid data. In IEEE Int. Conf. on Com- munications (ICC), pages 1-7, 2020.","title":"Hybrid-FL for wireless networks: Cooperative learning mechanism using non-iid data","authors":"Naoya Yoshida, Takayuki Nishio, Masahiro Morikura, Koji Yamamoto, Ryo Yonetani","year":"2020","venue":"IEEE Int. Conf. on Communications (ICC)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2021] Zeru Zhang, Jiayin Jin, Zijie Zhang, Yang Zhou, Xin Zhao, Jiaxiang Ren, Ji Liu, Lingfei Wu, Ruoming Jin, and Dejing Dou. Validating the lottery ticket hypothesis with inertial manifold theory. Advances in Neu- ral Information Processing Systems (NeurIPS), 34, 2021.","title":"Validating the lottery ticket hypothesis with inertial manifold theory","authors":"Zeru Zhang, Jiayin Jin, Zijie Zhang, Yang Zhou, Xin Zhao, Jiaxiang Ren, Ji Liu, Lingfei Wu, Ruoming Jin, Dejing Dou","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhao et al., 2018] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Fed- erated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018.","title":"Federated learning with non-iid data","authors":"Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, Vikas Chandra","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou et al., 2021] Chendi Zhou, Ji Liu, Juncheng Jia, Jingbo Zhou, Yang Zhou, Huaiyu Dai, and Dejing Dou. Efficient device scheduling with multi-job federated learn- ing. arXiv preprint arXiv:2112.05928, 2021.","title":"Efficient device scheduling with multi-job federated learning","authors":"Chendi Zhou, Ji Liu, Juncheng Jia, Jingbo Zhou, Yang Zhou, Huaiyu Dai, Dejing Dou","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}