{"type":"ScholarlyArticle","title":"AttnSense: Multi-level Attention Mechanism For Multimodal Human Activity Recognition","authors":[{"name":"HaoJie Ma","slug":"haojie-ma","given":"HaoJie","family":"Ma","affiliation":"State Key Laboratory for Novel Software Technology , Nanjing University"},{"name":"Wenzhong Li","slug":"wenzhong-li","given":"Wenzhong","family":"Li","affiliation":"State Key Laboratory for Novel Software Technology , Nanjing University"},{"name":"Xiao Zhang","slug":"xiao-zhang","given":"Xiao","family":"Zhang","affiliation":"State Key Laboratory for Novel Software Technology , Nanjing University"},{"name":"Songcheng Gao","slug":"songcheng-gao","given":"Songcheng","family":"Gao","affiliation":"State Key Laboratory for Novel Software Technology , Nanjing University"},{"name":"Sanglu Lu","slug":"sanglu-lu","given":"Sanglu","family":"Lu","affiliation":"State Key Laboratory for Novel Software Technology , Nanjing University"}],"abstract":"Sensor-based human activity recognition is a fundamental research problem in ubiquitous computing, which uses the rich sensing data from multimodal embedded sensors such as accelerometer and gyroscope to infer human activities. The existing activity recognition approaches either rely on domain knowledge or fail to address the spatial-temporal dependencies of the sensing signals. In this paper, we propose a novel attention-based multimodal neural network model called AttnSense for multimodal human activity recognition. AttnSense introduce the framework of combining attention mechanism with a convolutional neural network (CNN) and a Gated Recurrent Units (GRU) network to capture the dependencies of sensing signals in both spatial and temporal domains, which shows advantages in prioritized sensor selection and improves the comprehensibility. Extensive experiments based on three public datasets show that AttnSense achieves a competitive performance in activity recognition compared with several state-of-the-art methods.","doi":"10.24963/ijcai.2019/431","url":"https://ijcai.org/proceedings/2019/431","pdf":"https://ijcai.org/proceedings/2019/0431.pdf","markdown":"https://ijcai.org/proceedings/2019/431.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/431","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":431,"pages":{"first":3109,"last":3115},"keywords":["Multidisciplinary Topics and Applications: Ubiquitous Computing Systems","Machine Learning Applications: Applications of Supervised Learning","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Alsheikh et al., 2016] Mohammad Abu Alsheikh, Ahmed Selim, Dusit Niyato, Linda Doyle, Shaowei Lin, and Hwee-Pink Tan. Deep activity recognition models with triaxial accelerometers. In Workshops at the Thirtieth AAAI Conference on Artificial Intelligence, 2016.","title":"Deep activity recognition models with triaxial accelerometers","authors":"Mohammad Abu, Alsheikh, Ahmed Selim, Dusit Niyato, Linda Doyle, Shaowei Lin, Hwee-Pink Tan","year":"2016","venue":"Workshops at the Thirtieth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio, 2013] Yoshua Bengio. Deep learning of representations: Looking forward. In International Conference on Statistical Language and Speech Processing(SLSP'13), pages 1-37. Springer, 2013.","title":"Deep learning of representations: Looking forward","authors":"Yoshua Bengio, Bengio","year":"2013","venue":"International Conference on Statistical Language and Speech Processing(SLSP'13)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bulling et al., 2011] Andreas Bulling, Jamie A Ward, Hans Gellersen, and Gerhard Troster. Eye movement analysis for activity recognition using electrooculography. IEEE transactions on pattern analysis and machine intelligence, 33(4):741-753, 2011.","title":"Eye movement analysis for activity recognition using electrooculography","authors":"Andreas Bulling, Jamie A Ward, Hans Gellersen, Gerhard Troster","year":"2011","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[chu, 2014] Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3555, 2014.","title":"Empirical evaluation of gated recurrent neural networks on sequence modeling","authors":"Chu","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Figo et al., 2010] Davide Figo, Pedro C. Diniz, Diogo R. Ferreira, and João M. Cardoso. Preprocessing techniques for context recognition from accelerometer data. Personal Ubiquitous Comput., 14(7):645-662, October 2010.","title":"Preprocessing techniques for context recognition from accelerometer data","authors":"Davide Figo, Pedro C Diniz, R Diogo, João M Ferreira, Cardoso","year":"2010","venue":"Personal Ubiquitous Comput","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Greff et al., 2015] Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, and Jürgen Schmidhuber. LSTM: A search space odyssey. CoRR, abs/1503.04069, 2015.","title":"LSTM: A search space odyssey","authors":"Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, R Bas, Jürgen Steunebrink, Schmidhuber","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Guan and Plötz, 2017] Yu Guan and Thomas Plötz. Ensembles of deep lstm learners for activity recognition using wearables. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 1(2):11:1-11:28, June 2017.","title":"Ensembles of deep lstm learners for activity recognition using wearables","authors":"Yu Guan, Thomas Plötz","year":"2017","venue":"Proc. ACM Interact. Mob. Wearable Ubiquitous Technol","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hammerla et al., 2016] Nils Y. Hammerla, Shane Halloran, and Thomas Plötz. Deep, convolutional, and recurrent models for human activity recognition using wearables. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI'16, pages 1533-1540. AAAI Press, 2016.","title":"Deep, convolutional, and recurrent models for human activity recognition using wearables","authors":"Nils Y Hammerla, Shane Halloran, Thomas Plötz","year":"2016","venue":"Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI'16","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Marti A. Hearst. Support vector machines. IEEE Intelligent Systems, 13(4):18-28, July 1998.","title":"Support vector machines","authors":"Marti A Hearst","year":"1998","venue":"IEEE Intelligent Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(8):1735-1780, November 1997.","title":"Long short-term memory","authors":"Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Comput","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Khan et al., 2013] Wazir Zada Khan, Yang Xiang, Mohammed Y Aalsalem, and Quratulain Arshad. Mobile phone sensing systems: A survey. IEEE Communications Surveys & Tutorials, 15(1):402-427, 2013.","title":"Mobile phone sensing systems: A survey","authors":"Zada Wazir, Yang Khan, Mohammed Y Xiang, Quratulain Aalsalem, Arshad","year":"2013","venue":"IEEE Communications Surveys & Tutorials","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim and Lane, 2017] Suyoun Kim and Ian Lane. End-to-end speech recognition with auditory attention for multi-microphone distance speech recognition. In INTERSPEECH, 2017.","title":"End-to-end speech recognition with auditory attention for multi-microphone distance speech recognition","authors":"Lane ; Kim, Suyoun Kim, Ian Lane","year":"2017","venue":"INTERSPEECH","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436-444, 5 2015.","title":"Deep learning","authors":"Yann Lecun, Yoshua Bengio, Geoffrey Hinton","year":null,"venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liaw and Wiener, 2002] Andy Liaw and Matthew Wiener. Classification and Regression by randomForest. R News, 2(3):18-22, 2002.","title":"Classification and Regression by randomForest","authors":"Wiener Liaw, Andy Liaw, Matthew Wiener","year":"2002","venue":"R News","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ordóñez and Roggen, 2016] Francisco Javier Ordóñez and Daniel Roggen. Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors, 16(1), 2016.","title":"Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition","authors":"Roggen ; Ordóñez, Javier Francisco, Daniel Ordóñez, Roggen","year":"2016","venue":"Sensors","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Plötz et al., 2011] Thomas Plötz, Nils Y. Hammerla, and Patrick Olivier. Feature learning for activity recognition in ubiquitous computing. In Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, IJCAI'11, pages 1729-1734. AAAI Press, 2011.","title":"Feature learning for activity recognition in ubiquitous computing","authors":"Thomas Plötz, Nils Y Hammerla, Patrick Olivier","year":"2011","venue":"Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, IJCAI'11","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Radu et al., 2017] Valentin Radu, Catherine Tong, Sourav Bhattacharya, Nicholas D. Lane, Cecilia Mascolo, Mahesh K. Marina, and Fahim Kawsar. Multimodal deep learning for activity and context recognition. IMWUT, 1:157:1-157:27, 2017.","title":"Multimodal deep learning for activity and context recognition","authors":"Valentin Radu, Catherine Tong, Sourav Bhattacharya, Nicholas D Lane, Cecilia Mascolo, Mahesh K Marina, Fahim Kawsar","year":"2017","venue":"IMWUT","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Reiss and Stricker, 2012] Attila Reiss and Didier Stricker. Introducing a new benchmarked dataset for activity monitoring. In Proceedings of the 2012 16th Annual International Symposium on Wearable Computers, ISWC '12, pages 108-109, Washington, DC, USA, 2012. IEEE Computer Society.","title":"Introducing a new benchmarked dataset for activity monitoring","authors":"Stricker Reiss, Attila Reiss, Didier Stricker","year":"2012","venue":"Proceedings of the 2012 16th Annual International Symposium on Wearable Computers, ISWC '12","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Stiefmeier et al., 2008] Thomas Stiefmeier, Daniel Roggen, Georg Ogris, Paul Lukowicz, and Gerhard Tröster. Wearable activity tracking in car manufacturing. IEEE Pervasive Computing, 7, 2008.","title":"Wearable activity tracking in car manufacturing","authors":"Thomas Stiefmeier, Daniel Roggen, Georg Ogris, Paul Lukowicz, Gerhard Tröster","year":"2008","venue":"IEEE Pervasive Computing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"Stisen et al., ] Allan Stisen, Henrik Blunck, Sourav Bhattacharya, Mikkel Baun Prentow, Anind Dey, Tobias Sonne, and Mads Møller Jensen. Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition. In Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems (SenSys'15), pages 127-140. ACM.","title":"Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition","authors":"Stisen","year":null,"venue":"Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems (SenSys'15)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[van Kasteren et al., 2008] Tim van Kasteren, Athanasios Noulas, Gwenn Englebienne, and Ben Kröse. Accurate activity recognition in a home setting. In Proceedings of the 10th International Conference on Ubiquitous Computing, UbiComp '08, pages 1-9, New York, NY, USA, 2008. ACM.","title":"Accurate activity recognition in a home setting","authors":"Van Kasteren","year":"2008","venue":"Proceedings of the 10th International Conference on Ubiquitous Computing, UbiComp '08","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2018] Jindong Wang, Yiqiang Chen, Shuji Hao, Xiaohui Peng, and Lisha Hu. Deep learning for sensor-based activity recognition: A survey. Pattern Recognition Letters, 2018.","title":"Deep learning for sensor-based activity recognition: A survey","authors":"Jindong Wang, Yiqiang Chen, Shuji Hao, Xiaohui Peng, Lisha Hu","year":"2018","venue":"Pattern Recognition Letters","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2015] Jianbo Yang, Minh Nhut Nguyen, Phyo Phyo San, Xiaoli Li, and Shonali Krishnaswamy. Deep convolutional neural networks on multichannel time series for human activity recognition. In Proceedings of the 24th International Conference on Artificial Intelligence (Ijcai'15), volume 15 of IJCAI'15, pages 3995-4001. AAAI Press, 2015.","title":"Deep convolutional neural networks on multichannel time series for human activity recognition","authors":"Jianbo Yang, Minh Nhut Nguyen, Phyo Phyo San, Xiaoli Li, Shonali Krishnaswamy","year":"2015","venue":"Proceedings of the 24th International Conference on Artificial Intelligence (Ijcai'15)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yao et al., 2017] Shuochao Yao, Shaohan Hu, Yiran Zhao, Aston Zhang, and Tarek Abdelzaher. Deepsense: A unified deep learning framework for time-series mobile sensing data processing. In Proceedings of the 26th International Conference on World Wide Web (WWW'17), pages 351-360. International World Wide Web Conferences Steering Committee, 2017.","title":"Deepsense: A unified deep learning framework for time-series mobile sensing data processing","authors":"Yao","year":"2017","venue":"Proceedings of the 26th International Conference on World Wide Web (WWW'17)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}