{"type":"ScholarlyArticle","title":"Multi-modality Sensor Data Classification with Selective Attention","authors":[{"name":"Xiang Zhang","slug":"xiang-zhang","given":"Xiang","family":"Zhang","affiliation":"† School of Computer Science and Engineering , University of New South Wales"},{"name":"Lina Yao","slug":"lina-yao","given":"Lina","family":"Yao","affiliation":"† School of Computer Science and Engineering , University of New South Wales"},{"name":"Chaoran Huang","slug":"chaoran-huang","given":"Chaoran","family":"Huang","affiliation":"† School of Computer Science and Engineering , University of New South Wales"},{"name":"Sen Wang","slug":"sen-wang","given":"Sen","family":"Wang","affiliation":"§ School of Information and Communication Technology , Griffith University"},{"name":"Mingkui Tan","slug":"mingkui-tan","given":"Mingkui","family":"Tan","affiliation":""},{"name":"Guodong Long","slug":"guodong-long","given":"Guodong","family":"Long","affiliation":"‡ Center for Quantum Computation and Intelligent Systems , University of Technology Sydney; ¶ School of Software Engineering , South China University of Technology"},{"name":"Can Wang","slug":"can-wang","given":"Can","family":"Wang","affiliation":"§ School of Information and Communication Technology , Griffith University"}],"abstract":"Multimodel wearable sensor data classificationplays an important role in ubiquitous computingand has a wide range of applications in variousscenarios from healthcare to entertainment. How-ever, most of the existing work in this field em-ploys domain-specific approaches and is thus inef-fective in complex situations where multi-modalitysensor data is collected. Moreover, the wearablesensor data is less informative than the conven-tional data such as texts or images. In this paper,to improve the adaptability of such classificationmethods across different application contexts, weturn this classification task into a game and applya deep reinforcement learning scheme to dynami-cally deal with complex situations. We also intro-duce a selective attention mechanism into the rein-forcement learning scheme to focus on the crucialdimensions of the data. This mechanism helps tocapture extra information from the signal, and canthus significantly improve the discriminative powerof the classifier. We carry out several experimentson three wearable sensor datasets, and demonstratecompetitive performance of the proposed approachcompared to several state-of-the-art baselines.","doi":"10.24963/ijcai.2018/432","url":"https://ijcai.org/proceedings/2018/432","pdf":"https://ijcai.org/proceedings/2018/0432.pdf","markdown":"https://ijcai.org/proceedings/2018/432.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/432","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":432,"pages":{"first":3111,"last":3117},"keywords":["Machine Learning: Classification","Machine Learning: Reinforcement Learning","Machine Learning: Data Mining","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Akaike, 1969] Hirotugu Akaike. Fitting autoregressive models for prediction. Annals of the institute of Statisti- cal Mathematics, 21(1):243-247, 1969.","title":"Fitting autoregressive models for prediction","authors":"Akaike","year":"1969","venue":"Annals of the institute of Statistical Mathematics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bigdely-Shamlo et al., 2015] Nima Bigdely-Shamlo, Tim Mullen, Christian Kothe, Kyung-Min Su, and Kay A Rob- bins. The prep pipeline: standardized preprocessing for large-scale eeg analysis. Frontiers in neuroinformatics, 9:16, 2015.","title":"The prep pipeline: standardized preprocessing for large-scale eeg analysis","authors":"Nima Bigdely-Shamlo, Tim Mullen, Christian Kothe, Kyung-Min Su, Kay A Robbins","year":"2015","venue":"Frontiers in neuroinformatics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cavanagh and others, 1992] Patrick Cavanagh et al. Attention-based motion perception. Science, 257(5076):1563-1565, 1992.","title":"Attention-based motion perception","authors":"Patrick Cavanagh","year":"1992","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chowdhury et al., 2017] Alok Chowdhury, Dian Tjon- dronegoro, Vinod Chandran, and Stewart Trost. Physical activity recognition using posterior-adapted class-based fusion of multi-accelerometers data. IEEE Journal of Biomedical and Health Informatics, 2017.","title":"Physical activity recognition using posterior-adapted class-based fusion of multi-accelerometers data","authors":"Alok Chowdhury, Dian Tjondronegoro, Vinod Chandran, Stewart Trost","year":"2017","venue":"IEEE Journal of Biomedical and Health Informatics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chung et al., 2014] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evalua- tion of gated recurrent neural networks on sequence mod- eling. arXiv preprint arXiv:1412.3555, 2014.","title":"Empirical evaluation of gated recurrent neural networks on sequence modeling","authors":"Chung","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Erfani et al., 2017] Sarah M Erfani, Mahsa Baktashmot- lagh, Masud Moshtaghi, Vinh Nguyen, Christopher Leckie, James Bailey, and Kotagiri Ramamohanarao. From shared subspaces to shared landmarks: A robust multi-source classification approach. In AAAI, pages 1854-1860, 2017.","title":"From shared subspaces to shared landmarks: A robust multi-source classification approach","authors":"Sarah M Erfani, Mahsa Baktashmotlagh, Masud Moshtaghi, Vinh Nguyen, Christopher Leckie, James Bailey, Kotagiri Ramamohanarao","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fida et al., 2015] Benish Fida, Daniele Bibbo, Ivan Bernabucci, Antonino Proto, Silvia Conforto, and Mau- rizio Schmid. Real time event-based segmentation to classify locomotion activities through a single inertial sensor. In MOBIHEALTH, pages 104-107, 2015.","title":"Real time event-based segmentation to classify locomotion activities through a single inertial sensor","authors":"Benish Fida, Daniele Bibbo, Ivan Bernabucci, Antonino Proto, Silvia Conforto, Maurizio Schmid","year":"2015","venue":"MOBIHEALTH","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gers and Schmidhuber, 2001] Felix A Gers and E Schmid- huber. Lstm recurrent networks learn simple context-free and context-sensitive languages. IEEE Transactions on Neural Networks, 12(6):1333-1340, 2001.","title":"Felix A Gers and E Schmidhuber. Lstm recurrent networks learn simple context-free and context-sensitive languages","authors":"Schmidhuber Gers","year":"2001","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gers et al., 1999] Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with lstm. 1999. [Kamburugamuve et al., 2015]","title":"Learning to forget: Continual prediction with lstm","authors":"Jürgen Felix A Gers, Fred Schmidhuber, Cummins","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"Supun Kamburugamuve, Leif Christiansen, and Geoffrey Fox. A framework for real time processing of sensor data in the cloud. Journal of Sensors, 2015, 2015.","title":"A framework for real time processing of sensor data in the cloud","authors":"Supun Kamburugamuve, Leif Christiansen, Geoffrey Fox","year":"2015","venue":"Journal of Sensors","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Laurentini, 1994] Aldo Laurentini. The visual hull concept for silhouette-based image understanding. IEEE Trans- actions on pattern analysis and machine intelligence, 16(2):150-162, 1994.","title":"Aldo Laurentini. The visual hull concept for silhouette-based image understanding","authors":"Laurentini","year":"1994","venue":"IEEE Transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lipton et al., 2015] Zachary C Lipton, David C Kale, Charles Elkan, and Randall Wetzel. Learning to diag- nose with lstm recurrent neural networks. arXiv preprint arXiv:1511.03677, 2015.","title":"Learning to diagnose with lstm recurrent neural networks","authors":"David C Zachary C Lipton, Charles Kale, Randall Elkan, Wetzel","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lovmar et al., 2005] Lovisa Lovmar, Annika Ahlford, Mats Jonsson, and Ann-Christine Syvänen. Silhouette scores for assessment of snp genotype clusters. BMC genomics, 6(1):35, 2005.","title":"Silhouette scores for assessment of snp genotype clusters","authors":"Lovmar","year":"2005","venue":"BMC genomics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Markham and Townshend, 1981] BL Markham and JRG Townshend. Land cover classification accuracy as a func- tion of sensor spatial resolution. 1981.","title":"Land cover classification accuracy as a function of sensor spatial resolution","authors":"Townshend ; Bl Markham, Jrg Markham, Townshend","year":"1981","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sundermeyer et al., 2012] Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. Lstm neural networks for language modeling. In Thirteenth Annual Conference of the International Speech Communication Association, 2012.","title":"Lstm neural networks for language modeling","authors":"Martin Sundermeyer, Ralf Schlüter, Hermann Ney","year":"2012","venue":"Thirteenth Annual Conference of the International Speech Communication Association","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tan et al., 2015] Ming Tan, Cicero dos Santos, Bing Xi- ang, and Bowen Zhou. Lstm-based deep learning mod- els for non-factoid answer selection. arXiv preprint arXiv:1511.04108, 2015.","title":"Lstm-based deep learning models for non-factoid answer selection","authors":"Ming Tan, Bing Cicero Dos Santos, Bowen Xiang, Zhou","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tokic, 2010] Michel Tokic. Adaptive ε-greedy exploration in reinforcement learning based on value differences. In Annual Conference on Artificial Intelligence, pages 203- 210. Springer, 2010.","title":"Adaptive ε-greedy exploration in reinforcement learning based on value differences","authors":"Michel Tokic, Tokic","year":"2010","venue":"Annual Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2015] Ziyu Wang, Tom Schaul, Matteo Hes- sel, Hado Van Hasselt, Marc Lanctot, and Nando De Fre- itas. Dueling network architectures for deep reinforcement learning. arXiv preprint arXiv:1511.06581, 2015.","title":"Dueling network architectures for deep reinforcement learning","authors":"Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot, Nando De Freitas","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yao et al., 2018] Lina Yao, Quan Z Sheng, Xue Li, Tao Gu, Mingkui Tan, Xianzhi Wang, Sen Wang, and Wenjie Ruan. Compressive representation for device-free activity recog- nition with passive rfid signal strength. IEEE Transactions on Mobile Computing, 17(2):293-306, 2018.","title":"Compressive representation for device-free activity recognition with passive rfid signal strength","authors":"Lina Yao, Z Quan, Xue Sheng, Tao Li, Mingkui Gu, Xianzhi Tan, Sen Wang, Wenjie Wang, Ruan","year":"2018","venue":"IEEE Transactions on Mobile Computing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"Zhang et al., ] Xiang Zhang, Lina Yao, Quan Z Sheng, Salil S Kanhere, Tao Gu, and Dalin Zhang. Converting your thoughts to texts: Enabling brain typing via deep fea- ture learning of eeg signals. In PerCom 2018.","title":"Converting your thoughts to texts: Enabling brain typing via deep feature learning of eeg signals","authors":"Zhang","year":"2018","venue":"PerCom","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2017] Xiang Zhang, Lina Yao, Chaoran Huang, Quan Z Sheng, and Xianzhi Wang. Intent recog- nition in smart living through deep recurrent neural net- works. In International Conference on Neural Information Processing, pages 748-758. Springer, 2017.","title":"Intent recognition in smart living through deep recurrent neural networks","authors":"Xiang Zhang, Lina Yao, Chaoran Huang, Xianzhi Quan Z Sheng, Wang","year":"2017","venue":"International Conference on Neural Information Processing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2018] Dalin Zhang, Lina Yao, Xiang Zhang, Sen Wang, Weitong Chen, and Robert Boots. Eeg-based intention recognition from spatio-temporal representations via cascade and parallel convolutional recurrent neural net- works. In AAAI, 2018.","title":"Eeg-based intention recognition from spatio-temporal representations via cascade and parallel convolutional recurrent neural networks","authors":"Dalin Zhang, Lina Yao, Xiang Zhang, Sen Wang, Weitong Chen, Robert Boots","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zheng et al., 2014] Yi Zheng, Qi Liu, Enhong Chen, Yong Ge, and J Leon Zhao. Time series classification using multi-channels deep convolutional neural networks. In In- ternational Conference on Web-Age Information Manage- ment, pages 298-310. Springer, 2014.","title":"Time series classification using multi-channels deep convolutional neural networks","authors":"Yi Zheng, Qi Liu, Enhong Chen, Yong Ge, Leon Zhao","year":"2014","venue":"International Conference on Web-Age Information Management","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}