{"type":"ScholarlyArticle","title":"Multi-scale Context-Aware Networks Based on Fragment Association for Human Activity Recognition","authors":[{"name":"Zhiqiong Wang","slug":"zhiqiong-wang","given":"Zhiqiong","family":"Wang","affiliation":"Medicine and Biological Information Engineering , Northeastern University"},{"name":"Hanyu Liu","slug":"hanyu-liu","given":"Hanyu","family":"Liu","affiliation":"Medicine and Biological Information Engineering , Northeastern University"},{"name":"Boyang Zhao","slug":"boyang-zhao","given":"Boyang","family":"Zhao","affiliation":"Medicine and Biological Information Engineering , Northeastern University"},{"name":"Qi Shen","slug":"qi-shen","given":"Qi","family":"Shen","affiliation":"Medicine and Biological Information Engineering , Northeastern University"},{"name":"Mingzhe Li","slug":"mingzhe-li","given":"Mingzhe","family":"Li","affiliation":"Medicine and Biological Information Engineering , Northeastern University; Northeastern University"},{"name":"Ningfeng Que","slug":"ningfeng-que","given":"Ningfeng","family":"Que","affiliation":"Medicine and Biological Information Engineering , Northeastern University"},{"name":"Mingke Yan","slug":"mingke-yan","given":"Mingke","family":"Yan","affiliation":"Medicine and Biological Information Engineering , Northeastern University"},{"name":"Junchang Xin","slug":"junchang-xin","given":"Junchang","family":"Xin","affiliation":"Computer Science And Engineering"}],"abstract":"Sensor-based Human Activity Recognition (HAR) constitutes a key component of many artificial intelligence applications. Although deep feature extraction technology is constantly updated and iterated with excellent results, it is still a difficult task to find a balance between performance and computational efficiency. Through an in-depth exploration of the inherent characteristics of HAR data, we propose a lightweight feature perception model, which encompasses an internal feature extractor and a contextual feature perceiver. The model mainly consists of two stages. The first stage is a hierarchical multi-scale feature extraction module, which is composed of deep separable convolution and multi-head attention mechanism. This module serves to extract conventional features for Human Activity Recognition. After the feature goes through a fragment recombination operation, it is passed into the Context-Aware module of the second stage, which is based on Retentive Transformer and optimized by Dropkey method to efficiently extract the relationship between the feature fragments, so as to mine more valuable feature information. Importantly, this does not add too much complexity to the model, thereby preventing excessive resource consumption. We conducted extensive experimental validation on multiple publicly available HAR datasets.","doi":"10.24963/ijcai.2024/351","url":"https://ijcai.org/proceedings/2024/351","pdf":"https://ijcai.org/proceedings/2024/0351.pdf","markdown":"https://ijcai.org/proceedings/2024/351.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/351","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Humans and AI","track":"Main Track","number":351,"pages":{"first":3169,"last":3177},"keywords":["Data Mining: DM: Networks","Humans and AI: HAI: Applications"],"videos":[],"references":[{"position":1,"raw":"[Abdelrazik et al., 2023] Mostafa A Abdelrazik, Abdelhaliem Zekry, and Wael A Mohamed. Efficient hybrid algorithm for human action recognition. Journal of Image and Graphics, 11(1):72-81, 2023.","title":"Efficient hybrid algorithm for human action recognition","authors":"Abdelhaliem Mostafa A Abdelrazik, Zekry, Wael, Mohamed","year":"2023","venue":"Journal of Image and Graphics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chang et al., 2020] Youngjae Chang, Akhil Mathur, Anton Isopoussu, Junehwa Song, and Fahim Kawsar. A systematic study of unsupervised domain adaptation for robust human-activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4(1):1-30, 2020.","title":"A systematic study of unsupervised domain adaptation for robust human-activity recognition","authors":"Youngjae Chang, Akhil Mathur, Anton Isopoussu, Junehwa Song, Fahim Kawsar","year":"2020","venue":"Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chavarriaga et al., 2013] Ricardo Chavarriaga, Hesam Sagha, Alberto Calatroni, Sundara Tejaswi Digumarti, Gerhard Tröster, José del R Millán, and Daniel Roggen. The opportunity challenge: A benchmark database for on-body sensor-based activity recognition. Pattern Recognition Letters, 34(15):2033-2042, 2013.","title":"The opportunity challenge: A benchmark database for on-body sensor-based activity recognition","authors":"Ricardo Chavarriaga, Hesam Sagha, Alberto Calatroni, Tejaswi Sundara, Gerhard Digumarti, José Tröster, Daniel Del R Millán, Roggen","year":"2013","venue":"Pattern Recognition Letters","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2019] Kaixuan Chen, Lina Yao, Dalin Zhang, Xiaojun Chang, Guodong Long, and Sen Wang. Distributionally robust semi-supervised learning for people-centric sensing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3321-3328, 2019.","title":"Distributionally robust semi-supervised learning for people-centric sensing","authors":"Kaixuan Chen, Lina Yao, Dalin Zhang, Xiaojun Chang, Guodong Long, Sen Wang","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dang et al., 2020a] L Minh Dang, Kyungbok Min, Hanxiang Wang, Md Jalil Piran, Cheol Hee Lee, and Hyeonjoon Moon. Sensor-based and vision-based human activity recognition: A comprehensive survey. Pattern Recognition, 108:107561, 2020.","title":"Sensor-based and vision-based human activity recognition: A comprehensive survey","authors":"Dang","year":"2020","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dang et al., 2020b] L Minh Dang, Kyungbok Min, Hanxiang Wang, Md Jalil Piran, Cheol Hee Lee, and Hyeonjoon Moon. Sensor-based and vision-based human activity recognition: A comprehensive survey. Pattern Recognition, 108:107561, 2020.","title":"Sensor-based and vision-based human activity recognition: A comprehensive survey","authors":"Dang","year":"2020","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dua et al., 2021] Nidhi Dua, Shiva Nand Singh, and Vijay Bhaskar Semwal. Multi-input cnn-gru based human activity recognition using wearable sensors. Computing, 103:1461-1478, 2021.","title":"Multi-input cnn-gru based human activity recognition using wearable sensors","authors":"Nidhi Dua, Shiva Nand Singh, Vijay Bhaskar, Semwal","year":"2021","venue":"Computing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gao et al., 2019] Shang-Hua Gao, Ming-Ming Cheng, Kai Zhao, Xin-Yu Zhang, Ming-Hsuan Yang, and Philip Torr. Res2net: A new multi-scale backbone architecture. IEEE transactions on pattern analysis and machine intelligence, 43(2):652-662, 2019.","title":"Res2net: A new multi-scale backbone architecture","authors":"Shang-Hua Gao, Ming-Ming Cheng, Kai Zhao, Xin-Yu Zhang, Ming-Hsuan Yang, Philip Torr","year":"2019","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gholamiangonabadi and Grolinger, 2023] Davoud Gholamiangonabadi and Katarina Grolinger. Personalized models for human activity recognition with wearable sensors: deep neural networks and signal processing. Applied Intelligence, 53(5):6041-6061, 2023.","title":"Davoud Gholamiangonabadi and Katarina Grolinger. Personalized models for human activity recognition with wearable sensors: deep neural networks and signal processing","authors":"Grolinger Gholamiangonabadi","year":"2023","venue":"Applied Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ignatov, 2018] Andrey Ignatov. Real-time human activity recognition from accelerometer data using convolutional neural networks. Applied Soft Computing, 62:915-922, 2018.","title":"Real-time human activity recognition from accelerometer data using convolutional neural networks","authors":"Andrey Ignatov","year":"2018","venue":"Applied Soft Computing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Islam et al., 2022] Md. Milon Islam, Sheikh Nooruddin, Fakhri Karray, and Ghulam Muhammad. Human activity recognition using tools of convolutional neural networks: A state of the art review, data sets, challenges, and future prospects. Computers in Biology and Medicine, 149:106060, 2022.","title":"Human activity recognition using tools of convolutional neural networks: A state of the art review, data sets, challenges, and future prospects","authors":"Milon Md, Sheikh Islam, Fakhri Nooruddin, Ghulam Karray, Muhammad","year":"2022","venue":"Computers in Biology and Medicine","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ismail et al., 2023] Walaa N. Ismail, Hessah A. Alsalamah, Mohammad Mehedi Hassan, and Ebtesam Mohamed. Auto-har: An adaptive human activity recognition framework using an automated cnn architecture design. Heliyon, 9(2):e13636, 2023.","title":"Auto-har: An adaptive human activity recognition framework using an automated cnn architecture design","authors":"N Walaa, Hessah A Ismail, Mohammad Alsalamah, Ebtesam Mehedi Hassan, Mohamed","year":"2023","venue":"Heliyon","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jia et al., 2023] Qi Jia, Jing Guo, Po Yang, et al. A holistic multi-source transfer learning approach using wearable sensors for personalized daily activity recognition. Complex & Intelligent Systems, 2023, 2023.","title":"A holistic multi-source transfer learning approach using wearable sensors for personalized daily activity recognition","authors":"Qi Jia, Jing Guo, Po Yang","year":"2023","venue":"Complex & Intelligent Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kwapisz et al., 2011] Jennifer R Kwapisz, Gary M Weiss, and Samuel A Moore. Activity recognition using cell phone accelerometers. ACM SigKDD Explorations Newsletter, 12(2):74-82, 2011.","title":"Activity recognition using cell phone accelerometers","authors":"Gary M Jennifer R Kwapisz, Samuel A Weiss, Moore","year":"2011","venue":"ACM SigKDD Explorations Newsletter","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2023] Bonan Li, Yinhan Hu, Xuecheng Nie, Congying Han, Xiangjian Jiang, Tiande Guo, and Luoqi Liu. Dropkey for vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22700-22709, 2023.","title":"Dropkey for vision transformer","authors":"Bonan Li, Yinhan Hu, Xuecheng Nie, Congying Han, Xiangjian Jiang, Tiande Guo, Luoqi Liu","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2023] Hanyu Liu, Boyang Zhao, Chubo Dai, Boxin Sun, Ang Li, and Zhiqiong Wang. Mag-res2net: A novel deep learning network for human activity recognition. Physiological Measurement, 44(11):115007, 2023.","title":"Mag-res2net: A novel deep learning network for human activity recognition","authors":"Hanyu Liu, Boyang Zhao, Chubo Dai, Boxin Sun, Ang Li, Zhiqiong Wang","year":"2023","venue":"Physiological Measurement","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mekruksavanich et al., 2022] Sakorn Mekruksavanich, Ponnipa Jantawong, and Anuchit Jitpattanakul. A deep learning-based model for human activity recognition using biosensors embedded into a smart knee bandage. Procedia Computer Science, 214:621-627, 2022.","title":"A deep learning-based model for human activity recognition using biosensors embedded into a smart knee bandage","authors":"Sakorn Mekruksavanich, Ponnipa Jantawong, Anuchit Jitpattanakul","year":"2022","venue":"Procedia Computer Science","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Patiño-Saucedo et al., 2022] Janns Alvaro Patiño-Saucedo, Paola Patricia Ariza-Colpas, Shariq Butt-Aziz, Marlon Alberto Piñeres-Melo, José Luis López-Ruiz, Roberto Cesar Morales-Ortega, and Emiro De-la-hoz Franco. Predictive model for human activity recognition based on machine learning and feature selection techniques. International Journal of Environmental Research and Public Health, 19(19), 2022.","title":"Predictive model for human activity recognition based on machine learning and feature selection techniques","authors":"Patiño-Saucedo","year":"2022","venue":"International Journal of Environmental Research and Public Health","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Phyo et al., 2023] Jaeun Phyo, Wonjun Ko, Eunjin Jeon, and Heung-Il Suk. Transsleep: Transitioning-aware attention-based deep neural network for sleep staging. IEEE Transactions on Cybernetics, 53(7):4500-4510, 2023.","title":"Transsleep: Transitioning-aware attention-based deep neural network for sleep staging","authors":"Phyo","year":"2023","venue":"IEEE Transactions on Cybernetics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pramanik et al., 2023] Rishav Pramanik, Ritodeep Sikdar, and Ram Sarkar. Transformer-based deep reverse attention network for multi-sensory human activity recognition. Engineering Applications of Artificial Intelligence, 122:106150, 2023.","title":"Transformer-based deep reverse attention network for multi-sensory human activity recognition","authors":"Pramanik","year":"2023","venue":"Engineering Applications of Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Qian et al., 2019] Hangwei Qian, Sinno Jialin Pan, Bingshui Da, and Chunyan Miao. A novel distribution-embedded neural network for sensor-based activity recognition. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5614-5620. International Joint Conferences on Artificial Intelligence Organization, 7 2019.","title":"A novel distribution-embedded neural network for sensor-based activity recognition","authors":"Qian","year":"2019","venue":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19","doi":null,"url":null,"internal_url":"/proceedings/2019/779"},{"position":22,"raw":"[Ravi et al., 2016] Daniele Ravi, Charence Wong, Benny Lo, and Guang-Zhong Yang. Deep learning for human activity recognition: A resource efficient implementation on low-power devices. In 2016 IEEE 13th international conference on wearable and implantable body sensor networks (BSN), pages 71-76. IEEE, 2016.","title":"Deep learning for human activity recognition: A resource efficient implementation on low-power devices","authors":"Daniele Ravi, Charence Wong, Benny Lo, Guang-Zhong Yang","year":"2016","venue":"IEEE 13th international conference on wearable and implantable body sensor networks (BSN)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Reiss and Stricker, 2012] Attila Reiss and Didier Stricker. Introducing a new benchmarked dataset for activity monitoring. In 2012 16th international symposium on wearable computers, pages 108-109. IEEE, 2012.","title":"Introducing a new benchmarked dataset for activity monitoring","authors":"Stricker Reiss, Attila Reiss, Didier Stricker","year":"2012","venue":"16th international symposium on wearable computers","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rokni et al., 2018] Seyed Ali Rokni, Marjan Nourollahi, and Hassan Ghasemzadeh. Personalized human activity recognition using convolutional neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018.","title":"Personalized human activity recognition using convolutional neural networks","authors":"Seyed Ali Rokni, Marjan Nourollahi, Hassan Ghasemzadeh","year":"2018","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ruiz et al., 2021] Alejandro Pasos Ruiz, Michael Flynn, James Large, Matthew Middlehurst, and Anthony Bagnall. The great multivariate time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data Mining and Knowledge Discovery, 35(2):401-449, 2021.","title":"The great multivariate time series classification bake off: a review and experimental evaluation of recent algorithmic advances","authors":"Alejandro Pasos Ruiz, Michael Flynn, James Large, Matthew Middlehurst, Anthony Bagnall","year":"2021","venue":"Data Mining and Knowledge Discovery","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sun et al., 2023] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023.","title":"Retentive network: A successor to transformer for large language models","authors":"Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2016] Zhelong Wang, Donghui Wu, Jianming Chen, Ahmed Ghoneim, and Mohammad Anwar Hossain. A triaxial accelerometer-based human activity recognition via eemd-based features and game-theory-based feature selection. IEEE Sensors Journal, 16(9):3198-3207, 2016.","title":"A triaxial accelerometer-based human activity recognition via eemd-based features and game-theory-based feature selection","authors":"Wang","year":"2016","venue":"IEEE Sensors Journal","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xia et al., 2020] Kun Xia, Jianguang Huang, and Hanyu Wang. Lstm-cnn architecture for human activity recognition. IEEE Access, 8:56855-56866, 2020.","title":"Lstm-cnn architecture for human activity recognition","authors":"Kun Xia, Jianguang Huang, Hanyu Wang","year":"2020","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2020] Chao Yang, Mingxing Jiang, Zhongwen Guo, and Yuan Liu. Gated res2net for multivariate time series analysis. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-7. IEEE, 2020.","title":"Gated res2net for multivariate time series analysis","authors":"Chao Yang, Mingxing Jiang, Zhongwen Guo, Yuan Liu","year":"2020","venue":"2020 International Joint Conference on Neural Networks (IJCNN)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zeng et al., 2014] Ming Zeng, Le T. Nguyen, Bo Yu, Ole J. Mengshoel, Jiang Zhu, Pang Wu, and Joy Zhang. Convolutional neural networks for human activity recognition using mobile sensors. In 6th International Conference on Mobile Computing, Applications and Services, pages 197-205, 2014.","title":"Convolutional neural networks for human activity recognition using mobile sensors","authors":"Ming Zeng, Le T Nguyen, Bo Yu, Ole J Mengshoel, Jiang Zhu, Pang Wu, Joy Zhang","year":"2014","venue":"6th International Conference on Mobile Computing, Applications and Services","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}