{"type":"ScholarlyArticle","title":"Breaking Barriers of System Heterogeneity: Straggler-Tolerant Multimodal Federated Learning via Knowledge Distillation","authors":[{"name":"Jinqian Chen","slug":"jinqian-chen","given":"Jinqian","family":"Chen","affiliation":"School of Software , Shandong University; School of Software Engineering , Xi'an Jiaotong University"},{"name":"Haoyu Tang","slug":"haoyu-tang","given":"Haoyu","family":"Tang","affiliation":"School of Software , Shandong University"},{"name":"Junhao Cheng","slug":"junhao-cheng","given":"Junhao","family":"Cheng","affiliation":"School of Software , Shandong University"},{"name":"Ming Yan","slug":"ming-yan","given":"Ming","family":"Yan","affiliation":"Alibaba Group"},{"name":"Ji Zhang","slug":"ji-zhang","given":"Ji","family":"Zhang","affiliation":"Alibaba Group"},{"name":"Mingzhu Xu","slug":"mingzhu-xu","given":"Mingzhu","family":"Xu","affiliation":"School of Software , Shandong University"},{"name":"Yupeng Hu","slug":"yupeng-hu","given":"Yupeng","family":"Hu","affiliation":"School of Software , Shandong University"},{"name":"Liqiang Nie","slug":"liqiang-nie","given":"Liqiang","family":"Nie","affiliation":"Harbin Institute of Technology (Shenzhen)"}],"abstract":"Internet of Things (IoT) devices possess valuable yet private multimodal data, calling for a decentralized machine learning scheme. Though several multimodal federated learning (MFL) methods have been proposed, most of them merely overlook the system heterogeneity across IoT devices, resulting in the inadaptability to real world applications. Aiming at this, we conduct theoretical analysis and exploration experiments on straggler impacts and uncover the fact that stragglers caused by system heterogeneity are fatal to MFL, resulting in catastrophic time overhead. Motivated by this, we propose a novel Multimodal Federated Learning with Accelerated Knowledge Distillation (MFL-AKD) framework, which is the first attempt to integrate knowledge distillation to combat stragglers in complex multimodal federated scenarios. Concretely, given the pretrained large-scale vision-language models deployed in the central server, we apply a fast knowledge transfer mechanism to conduct early training of local models with part of the local data. The early-trained model is then enhanced through the distillation of the pretrained large model and further trained on the remaining data. Extensive experiments on two datasets for video moment retrieval and two datasets for image-text retrieval demonstrate that our method achieves superior results with high straggler robustness.","doi":"10.24963/ijcai.2024/419","url":"https://ijcai.org/proceedings/2024/419","pdf":"https://ijcai.org/proceedings/2024/0419.pdf","markdown":"https://ijcai.org/proceedings/2024/419.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/419","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":419,"pages":{"first":3789,"last":3797},"keywords":["Machine Learning: ML: Multi-modal learning"],"videos":[],"references":[{"position":1,"raw":"[Ahn et al., 2020] Sungyong Ahn, Seunghyun Yoo, and Ki- jung Shin. Fedkd: Collaborative learning with weight- sharing and knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4598-4605, 2020.","title":"Fedkd: Collaborative learning with weightsharing and knowledge distillation","authors":"Ahn","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bonawitz et al., 2017] Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Alec Segal, Karn Seth, Vin- cent Vanhoucke, et al. Practical secure aggregation for privacy-preserving machine learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pages 1175-1191, 2017.","title":"Practical secure aggregation for privacy-preserving machine learning","authors":"Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, Brendan Mcmahan, Sarvar Patel, Daniel Ramage, Alec Segal, Karn Seth, Vincent Vanhoucke","year":"2017","venue":"Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cheng et al., 2021] Sijie Cheng, Jingwen Wu, Yanghua Xiao, and Yang Liu. Fedgems: Federated learning of larger server models via selective knowledge fusion. CoRR, abs/2110.11027, 2021.","title":"Fedgems: Federated learning of larger server models via selective knowledge fusion","authors":"Sijie Cheng, Jingwen Wu, Yanghua Xiao, Yang Liu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gao et al., 2017] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 5277-5285, 2017.","title":"Tall: Temporal activity localization via language query","authors":"Jiyang Gao, Chen Sun, Zhenheng Yang, Ram Nevatia","year":"2017","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Han et al., 2020] Pengchao Han, Shiqiang Wang, and Kin K Leung. Adaptive gradient sparsification for efficient fed- erated learning: An online learning approach. In 2020 IEEE 40th international conference on distributed com- puting systems (ICDCS), pages 300-310. IEEE, 2020.","title":"Adaptive gradient sparsification for efficient federated learning: An online learning approach","authors":"Han","year":"2020","venue":"IEEE 40th international conference on distributed computing systems (ICDCS)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2023] Yupeng Hu, Kun Wang, Meng Liu, Haoyu Tang, and Liqiang Nie. Semantic collaborative learning for cross-modal moment localization. ACM Transactions on Information Systems, 42(2):1-26, 2023.","title":"Semantic collaborative learning for cross-modal moment localization","authors":"Yupeng Hu, Kun Wang, Meng Liu, Haoyu Tang, Liqiang Nie","year":"2023","venue":"ACM Transactions on Information Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2022] Jie Huang, Zheng Liu, Dong Yang, and Yang Li. Multimodal federated learning via contrastive representation ensemble. In AAAI, 2022.","title":"Multimodal federated learning via contrastive representation ensemble","authors":"Jie Huang, Zheng Liu, Dong Yang, Yang Li","year":"2022","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kairouz et al., 2021] Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Gra- ham Cormode, Rachel Cummings, and et al. Advances and open problems in federated learning. Found. Trends Mach. Learn., 14(1-2):1-210, jun 2021.","title":"Advances and open problems in federated learning","authors":"H Brendan Peter Kairouz, Brendan Mcmahan, Aurélien Avent, Mehdi Bellet, Bennis, Nitin Arjun, Kallista Bhagoji, Zachary Bonawitz, Graham Charles, Rachel Cormode, Cummings","year":"2021","venue":"Found. Trends Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Komodakis and Pesquet, 2017] Nikos Komodakis and Jean- Christophe Pesquet. Paying more attention to attention: Improving the performance of convolutional neural net- works via attention transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5294-5303, 2017.","title":"Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer","authors":"Nikos Komodakis, Jean-Christophe Pesquet","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krishna et al., 2017] Ranjay Krishna, Kenji Hata, Fred- eric Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense- captioning events in videos. In 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, October 2017.","title":"Densecaptioning events in videos","authors":"Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, Juan Carlos Niebles","year":"2017","venue":"IEEE International Conference on Computer Vision (ICCV). IEEE","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2020a] Mingsheng Li, Haoqiang Fan, Haoyi Xiong, Wei Zhu, Jianfei Hu, and Ling Li. Federated learn- ing: Challenges, methods, and future directions. IEEE Sig- nal Processing Magazine, 37(3):50-60, 2020.","title":"Federated learning: Challenges, methods, and future directions","authors":"Li","year":"2020","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2020b] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Pro- ceedings of Machine learning and systems, 2:429-450, 2020.","title":"Federated optimization in heterogeneous networks","authors":"Li","year":"2020","venue":"Proceedings of Machine learning and systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2020c] Xiang Li, Yifeng Wang, and Xiaoming M. Wu. Fedmd: Heterogenous federated learning via model distillation. In Proceedings of the 26th ACM SIGKDD In- ternational Conference on Knowledge Discovery & Data Mining, pages 2423-2433, 2020.","title":"Fedmd: Heterogenous federated learning via model distillation","authors":"Li","year":"2020","venue":"Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, Septem- ber 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"Computer Vision-ECCV 2014: 13th European Conference","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2019] Ronghang Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Elnaz Frank, Alexander Sergeev, and Kellen Yocum. Federated learning for vision- and-language grounding problems. In EMNLP-IJCNLP 2019, 2019.","title":"Federated learning for visionand-language grounding problems","authors":"Liu","year":"2019","venue":"EMNLP-IJCNLP 2019","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2021] Jianchun Liu, Hongli Xu, Lun Wang, Yang Xu, Chen Qian, Jinyang Huang, and He Huang. Adaptive asynchronous federated learning in resource-constrained edge computing. IEEE Transactions on Mobile Comput- ing, 2021.","title":"Adaptive asynchronous federated learning in resource-constrained edge computing","authors":"Jianchun Liu, Hongli Xu, Lun Wang, Yang Xu, Chen Qian, Jinyang Huang, He Huang","year":"2021","venue":"IEEE Transactions on Mobile Computing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Pro- ceedings of Machine Learning Research, pages 1273- 1282. PMLR, 20-22 Apr 2017.","title":"Communication-Efficient Learning of Deep Networks from Decentralized Data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"Proceedings of the 20th International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Qu et al., 2021] Leigang Qu, Meng Liu, Jianlong Wu, Zan Gao, and Liqiang Nie. Dynamic modality interaction mod- eling for image-text retrieval. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1104-1113, 2021.","title":"Dynamic modality interaction modeling for image-text retrieval","authors":"Qu","year":"2021","venue":"Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar-Wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Reisizadeh et al., 2022] Amirhossein Reisizadeh, Isidoros Tziotis, Hamed Hassani, Aryan Mokhtari, and Ramtin Pedarsani. Straggler-resilient federated learning: Leverag- ing the interplay between statistical accuracy and system heterogeneity. IEEE Journal on Selected Areas in Infor- mation Theory, 3(2):197-205, 2022.","title":"Straggler-resilient federated learning: Leveraging the interplay between statistical accuracy and system heterogeneity","authors":"Reisizadeh","year":"2022","venue":"IEEE Journal on Selected Areas in Information Theory","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Romero et al., 2015] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets, 2015.","title":"Fitnets: Hints for thin deep nets","authors":"Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, Yoshua Bengio","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Smith et al., 2017] Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. Advances in neural information processing systems, 30, 2017.","title":"Federated multi-task learning","authors":"Smith","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tang et al., 2021a] Haoyu Tang, Jihua Zhu, Meng Liu, Zan Gao, and Zhiyong Cheng. Frame-wise cross-modal match- ing for video moment retrieval. IEEE Transactions on Multimedia, 24:1338-1349, 2021.","title":"Frame-wise cross-modal matching for video moment retrieval","authors":"Tang","year":"2021","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tang et al., 2021b] Haoyu Tang, Jihua Zhu, Lin Wang, Qinghai Zheng, and Tianwei Zhang. Multi-level query interaction for temporal language grounding. IEEE Transactions on Intelligent Transportation Systems, 23(12):25479-25488, 2021.","title":"Multi-level query interaction for temporal language grounding","authors":"Tang","year":"2021","venue":"IEEE Transactions on Intelligent Transportation Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2021] Yuzhou Wang, Wenjie Jiang, Jingkuan Song, and Xiangyang Xue. Fedvmr: A new federated learning method for video moment retrieval. IEEE Trans- actions on Multimedia, 2021.","title":"Fedvmr: A new federated learning method for video moment retrieval","authors":"Yuzhou Wang, Wenjie Jiang, Jingkuan Song, Xiangyang Xue","year":"2021","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wu et al., 2020] Wentai Wu, Ligang He, Weiwei Lin, Rui Mao, Carsten Maple, and Stephen Jarvis. Safa: A semi- asynchronous protocol for fast federated learning with low overhead. IEEE Transactions on Computers, 70(5):655- 668, 2020.","title":"Safa: A semiasynchronous protocol for fast federated learning with low overhead","authors":"Wentai Wu, Ligang He, Weiwei Lin, Rui Mao, Carsten Maple, Stephen Jarvis","year":"2020","venue":"IEEE Transactions on Computers","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wu et al., 2022] Yuezhou Wu, Yan Kang, Jiahuan Luo, Yuanqin He, Lixin Fan, Rong Pan, and Qiang Yang. Fedcg: Leverage conditional gan for protecting privacy and maintaining competitive performance in federated learning. In Lud De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 2334-2340. International Joint Conferences on Artificial Intelligence Organization, 7 2022. Main Track.","title":"Fedcg: Leverage conditional gan for protecting privacy and maintaining competitive performance in federated learning","authors":"Yuezhou Wu, Yan Kang, Jiahuan Luo, Yuanqin He, Lixin Fan, Rong Pan, Qiang Yang","year":"2022","venue":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22","doi":null,"url":null,"internal_url":"/proceedings/2022/324"},{"position":30,"raw":"[Yang et al., 2020] Qiang Yang, Yang Liu, Tianjian Chen, and Yuan Tong. Federated Learning, volume 14 of Synthe- sis Lectures on Artificial Intelligence and Machine Learn- ing. Morgan & Claypool Publishers, 2020.","title":"Federated Learning, volume 14 of Synthesis Lectures on Artificial Intelligence and Machine Learning","authors":"Qiang Yang, Yang Liu, Tianjian Chen, Yuan Tong","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Young et al., 2014] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.","title":"From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions","authors":"Young","year":"2014","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhao et al., 2021] Liang Zhao, Junlin Wang, Zhigang Chen, Qiang Wu, and Jie Chen. Multimodal federated learning on iot data. IEEE Internet of Things Journal, 2021.","title":"Multimodal federated learning on iot data","authors":"Liang Zhao, Junlin Wang, Zhigang Chen, Qiang Wu, Jie Chen","year":"2021","venue":"IEEE Internet of Things Journal","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}