{"type":"ScholarlyArticle","title":"Federated Multi-Task Attention for Cross-Individual Human Activity Recognition","authors":[{"name":"Qiang Shen","slug":"qiang-shen","given":"Qiang","family":"Shen","affiliation":"College of Computer Science and Technology , Jilin University"},{"name":"Haotian Feng","slug":"haotian-feng","given":"Haotian","family":"Feng","affiliation":"College of Computer Science and Technology , Jilin University"},{"name":"Rui Song","slug":"rui-song","given":"Rui","family":"Song","affiliation":"School of Artificial Intelligence , Jilin University"},{"name":"Stefano Teso","slug":"stefano-teso","given":"Stefano","family":"Teso","affiliation":"University of Trento"},{"name":"Fausto Giunchiglia","slug":"fausto-giunchiglia","given":"Fausto","family":"Giunchiglia","affiliation":"College of Computer Science and Technology , Jilin University; University of Trento"},{"name":"Hao Xu","slug":"hao-xu","given":"Hao","family":"Xu","affiliation":"College of Computer Science and Technology , Jilin University; School of Artificial Intelligence , Jilin University"}],"abstract":"Federated Learning (FL) is an emerging privacy-aware machine learning technique that applies successfully to the collaborative learning of global models for Human Activity Recognition (HAR). As of now, the applications of FL for HAR assume that the data associated with diverse individuals follow the same distribution. However, this assumption is impractical in real-world scenarios where the same activity is frequently performed differently by different individuals. To tackle this issue, we propose FedMAT, a Federated Multi-task ATtention framework for HAR, which extracts and fuses shared as well as individual-specific multi-modal sensor data features. Specifically, we treat the HAR problem associated with each individual as a different task and train a federated multi-task model, composed of a shared feature representation network in a central server plus multiple individual-specific networks with attention modules stored in decentralized nodes. In this architecture, the attention module operates as a mask that allows to learn individual-specific features from the global model, whilst simultaneously allowing for features to be shared among different individuals. We conduct extensive experiments based on publicly available HAR datasets, which are collected in both controlled environments and real-world scenarios. Numeric results verify that our proposed FedMAT significantly outperforms baselines not only in generalizing to existing individuals but also in adapting to new individuals.","doi":"10.24963/ijcai.2022/475","url":"https://ijcai.org/proceedings/2022/475","pdf":"https://ijcai.org/proceedings/2022/0475.pdf","markdown":"https://ijcai.org/proceedings/2022/475.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/475","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":475,"pages":{"first":3423,"last":3429},"keywords":["Humans and AI: Computational Sustainability and Human Well-Being","Machine Learning: Multi-modal learning","Machine Learning: Multi-task and Transfer Learning","Knowledge Representation and Reasoning: Reasong about actions","Machine Learning: Meta-Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986788","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984931","length_sec":351,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bettini et al., 2021] Claudio Bettini, Gabriele Civitarese, and Riccardo Presotto. Personalized semi-supervised fed- erated learning for human activity recognition. arXiv preprint arXiv:2104.08094, 2021.","title":"Personalized semi-supervised federated learning for human activity recognition","authors":"Claudio Bettini, Gabriele Civitarese, Riccardo Presotto","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bison et al., 2021] Ivano Bison, Fausto Giunchiglia, Mattia Zeni, Enrico Bignotti, Matteo Busso, and Ronald Chenu- Abente. Trento 2018 -an extended pilot on the daily rou- tines of university students, 2021. University of Trento Technical Report -DataScientia dataset descriptors.","title":"Trento 2018 -an extended pilot on the daily routines of university students","authors":"Ivano Bison, Fausto Giunchiglia, Mattia Zeni, Enrico Bignotti, Matteo Busso, Ronald Chenu-Abente","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2021] Kaixuan Chen, Dalin Zhang, Lina Yao, Bin Guo, Zhiwen Yu, and Yunhao Liu. Deep learning for sensor-based human activity recognition: Overview, challenges, and opportunities. ACM Computing Surveys, 54(4):1-40, 2021.","title":"Deep learning for sensor-based human activity recognition: Overview, challenges, and opportunities","authors":"Kaixuan Chen, Dalin Zhang, Lina Yao, Bin Guo, Zhiwen Yu, Yunhao Liu","year":"2021","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Francisco and Daniel, 2016] O. Francisco and R. Daniel. Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors, 16(1):115, 2016.","title":"Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition","authors":"O Francisco, R Daniel","year":"2016","venue":"Sensors","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hammerla et al., 2016] Nils Y Hammerla, Shane Halloran, and Thomas Plötz. Deep, convolutional, and recurrent models for human activity recognition using wearables. arXiv preprint arXiv:1604.08880, 2016.","title":"Deep, convolutional, and recurrent models for human activity recognition using wearables","authors":"Y Nils, Shane Hammerla, Thomas Halloran, Plötz","year":"2016","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2016/220"},{"position":6,"raw":"[Li et al., 2020] Tian Li, Anit Kumar Sahu, Ameet Tal- walkar, and Virginia Smith. Federated learning: Chal- lenges, methods, and future directions. IEEE Signal Pro- cessing Magazine, 37(3):50-60, 2020.","title":"Federated learning: Challenges, methods, and future directions","authors":"Tian Li, Anit Kumar Sahu, Ameet Talwalkar, Virginia Smith","year":"2020","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Li et al., 2021] Chenglin Li, Di Niu, Bei Jiang, Xiao Zuo, and Jianming Yang. Meta-har: Federated representation learning for human activity recognition. In Proceedings of the Web Conference 2021, pages 912-922, 2021.","title":"Meta-har: Federated representation learning for human activity recognition","authors":"Chenglin Li, Di Niu, Bei Jiang, Xiao Zuo, Jianming Yang","year":"2021","venue":"Proceedings of the Web Conference 2021","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ma et al., 2019] Haojie Ma, Wenzhong Li, Xiao Zhang, Songcheng Gao, and Sanglu Lu. Attnsense: Multi-level attention mechanism for multimodal human activity recog- nition. In IJCAI, pages 3109-3115, 2019.","title":"Attnsense: Multi-level attention mechanism for multimodal human activity recognition","authors":"Haojie Ma, Wenzhong Li, Xiao Zhang, Songcheng Gao, Sanglu Lu","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/431"},{"position":9,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.","title":"Communication-efficient learning of deep networks from decentralized data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"Artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ouyang et al., 2021] Xiaomin Ouyang, Zhiyuan Xie, Jiayu Zhou, Jianwei Huang, and Guoliang Xing. Clusterfl: a similarity-aware federated learning system for human ac- tivity recognition. In Proceedings of the 19th Annual In- ternational Conference on Mobile Systems, Applications, and Services, pages 54-66, 2021.","title":"Clusterfl: a similarity-aware federated learning system for human activity recognition","authors":"Xiaomin Ouyang, Zhiyuan Xie, Jiayu Zhou, Jianwei Huang, Guoliang Xing","year":"2021","venue":"Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Reiss and Stricker, 2012] Attila Reiss and Didier Stricker. Introducing a new benchmarked dataset for activity moni- toring. In 2012 16th international symposium on wearable computers, pages 108-109. IEEE, 2012.","title":"Introducing a new benchmarked dataset for activity monitoring","authors":"Stricker Reiss, Attila Reiss, Didier Stricker","year":"2012","venue":"16th international symposium on wearable computers","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Smith et al., 2017] Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning. arXiv preprint arXiv:1705.10467, 2017.","title":"Federated multi-task learning","authors":"Smith","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Stisen et al., 2015] Allan Stisen, Henrik Blunck, Sourav Bhattacharya, Thor Siiger Prentow, Mikkel Baun Kjær- gaard, Anind Dey, Tobias Sonne, and Mads Møller Jensen. Smart devices are different: Assessing and mitigatingmo- bile sensing heterogeneities for activity recognition. In Proceedings of the 13th ACM conference on embedded networked sensor systems, pages 127-140, 2015.","title":"Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition","authors":"Allan Stisen, Henrik Blunck, Sourav Bhattacharya, Thor Siiger Prentow, Mikkel Baun Kjærgaard, Anind Dey, Tobias Sonne, Mads Møller Jensen","year":"2015","venue":"Proceedings of the 13th ACM conference on embedded networked sensor systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Straczkiewicz et al., 2021] Marcin Straczkiewicz, Peter James, and Jukka-Pekka Onnela. A systematic review of smartphone-based human activity recognition methods for health research. NPJ Digital Medicine, 4(1):1-15, 2021.","title":"A systematic review of smartphone-based human activity recognition methods for health research","authors":"Marcin Straczkiewicz, Peter James, Jukka-Pekka Onnela","year":"2021","venue":"NPJ Digital Medicine","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Tu et al., 2021] Linlin Tu, Xiaomin Ouyang, Jiayu Zhou, Yuze He, and Guoliang Xing. Feddl: Federated learning via dynamic layer sharing for human activity recognition. In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, pages 15-28, 2021.","title":"Feddl: Federated learning via dynamic layer sharing for human activity recognition","authors":"Linlin Tu, Xiaomin Ouyang, Jiayu Zhou, Yuze He, Guoliang Xing","year":"2021","venue":"Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Vaizman et al., 2017] Yonatan Vaizman, Katherine Ellis, and Gert Lanckriet. Recognizing detailed human context in the wild from smartphones and smartwatches. IEEE pervasive computing, 16(4):62-74, 2017.","title":"Recognizing detailed human context in the wild from smartphones and smartwatches","authors":"Vaizman","year":"2017","venue":"IEEE pervasive computing","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2019] Jindong Wang, Yiqiang Chen, Shuji Hao, Xiaohui Peng, and Lisha Hu. Deep learning for sensor-based activity recognition: A survey. Pattern Recognition Letters, 119:3-11, 2019.","title":"Deep learning for sensor-based activity recognition: A survey","authors":"Jindong Wang, Yiqiang Chen, Shuji Hao, Xiaohui Peng, Lisha Hu","year":"2019","venue":"Pattern Recognition Letters","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Yang et al., 2019] Qiang Yang, Yang Liu, Yong Cheng, Yan Kang, Tianjian Chen, and Han Yu. Federated learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 13(3):1-207, 2019.","title":"Federated learning","authors":"Qiang Yang, Yang Liu, Yong Cheng, Yan Kang, Tianjian Chen, Han Yu","year":"2019","venue":"Synthesis Lectures on Artificial Intelligence and Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yao et al., 2017] Shuochao Yao, Shaohan Hu, Yiran Zhao, Aston Zhang, and Tarek Abdelzaher. Deepsense: A unified deep learning framework for time-series mobile sensing data processing. In Proceedings of the 26th International Conference on World Wide Web, pages 351-360, 2017.","title":"Deepsense: A unified deep learning framework for time-series mobile sensing data processing","authors":"Yao","year":"2017","venue":"Proceedings of the 26th International Conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zeng et al., 2014] Ming Zeng, Le T Nguyen, Bo Yu, Ole J Mengshoel, Jiang Zhu, Pang Wu, and Joy Zhang. Con- volutional neural networks for human activity recognition using mobile sensors. In 6th international conference on mobile computing, applications and services, pages 197- 205. IEEE, 2014.","title":"Convolutional neural networks for human activity recognition using mobile sensors","authors":"Ming Zeng, Le T Nguyen, Bo Yu, Ole J Mengshoel, Jiang Zhu, Pang Wu, Joy Zhang","year":"2014","venue":"6th international conference on mobile computing, applications and services","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang and Yang, 2021] Yu Zhang and Qiang Yang. A sur- vey on multi-task learning. IEEE Transactions on Knowl- edge and Data Engineering, 2021.","title":"A survey on multi-task learning","authors":"Yu Zhang, Qiang Yang","year":"2021","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2021a] Le Zhang, Wei Cui, Bing Li, Zhenghua Chen, Min Wu, and Teo Sin Gee. Privacy-preserving cross-environment human activity recognition. IEEE Transactions on Cybernetics, pages 1-11, 2021.","title":"Privacy-preserving cross-environment human activity recognition","authors":"Zhang","year":"2021","venue":"IEEE Transactions on Cybernetics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2021b] Wanyi Zhang, Qiang Shen, Stefano Teso, Bruno Lepri, Andrea Passerini, Ivano Bison, and Fausto Giunchiglia. Putting human behavior predictability in context. EPJ Data Science, 10(1):42, 2021.","title":"Putting human behavior predictability in context","authors":"Zhang","year":"2021","venue":"EPJ Data Science","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhao et al., 2018] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Fed- erated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018.","title":"Federated learning with non-iid data","authors":"Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, Vikas Chandra","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}