{"type":"ScholarlyArticle","title":"Multi-agent Attentional Activity Recognition","authors":[{"name":"Kaixuan Chen","slug":"kaixuan-chen","given":"Kaixuan","family":"Chen","affiliation":"University of New South Wales"},{"name":"Lina Yao","slug":"lina-yao","given":"Lina","family":"Yao","affiliation":"University of New South Wales"},{"name":"Dalin Zhang","slug":"dalin-zhang","given":"Dalin","family":"Zhang","affiliation":"University of New South Wales"},{"name":"Bin Guo","slug":"bin-guo","given":"Bin","family":"Guo","affiliation":"Northwestern Polytechnical University"},{"name":"Zhiwen Yu","slug":"zhiwen-yu","given":"Zhiwen","family":"Yu","affiliation":"Northwestern Polytechnical University"}],"abstract":"Multi-modality is an important feature of sensor based activity recognition. In this work, we consider two inherent characteristics of human activities, the spatially-temporally varying salience of features and the relations between activities and corresponding body part motions. Based on these, we propose a multi-agent spatial-temporal attention model. The spatial-temporal attention mechanism helps intelligently select informative modalities and their active periods. And the multiple agents in the proposed model represent activities with collective motions across body parts by independently selecting modalities associated with single motions. With a joint recognition goal, the agents share gained information and coordinate their selection policies to learn the optimal recognition model. The experimental results on four real-world datasets demonstrate that the proposed model outperforms the state-of-the-art methods.","doi":"10.24963/ijcai.2019/186","url":"https://ijcai.org/proceedings/2019/186","pdf":"https://ijcai.org/proceedings/2019/0186.pdf","markdown":"https://ijcai.org/proceedings/2019/186.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/186","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Humans and AI","track":"Main track","number":186,"pages":{"first":1344,"last":1350},"keywords":["Machine Learning: Deep Learning","Humans and AI: Human-Computer Interaction"],"videos":[],"references":[{"position":1,"raw":"[Anguita et al., 2013] Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge Luis Reyes-Ortiz. A public domain dataset for human activity recognition using smartphones. In ESANN, 2013.","title":"A public domain dataset for human activity recognition using smartphones","authors":"Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, Jorge Luis, Reyes-Ortiz","year":"2013","venue":"ESANN","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Banos et al., 2014] Oresti Banos, Rafael Garcia, Juan A Holgado-Terriza, Miguel Damas, Hector Pomares, Ignacio Rojas, Alejandro Saez, and Claudia Villalonga. mhealth- droid: a novel framework for agile development of mobile health applications. In International Workshop on Ambient Assisted Living, pages 91-98. Springer, 2014.","title":"mhealthdroid: a novel framework for agile development of mobile health applications","authors":"Banos","year":"2014","venue":"International Workshop on Ambient Assisted Living","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cai et al., 2009] Chenghui Cai, Xuejun Liao, and Lawrence Carin. Learning to explore and exploit in pomdps. In Ad- vances in Neural Information Processing Systems (NIPS), pages 198-206, 2009.","title":"Learning to explore and exploit in pomdps","authors":"Cai","year":"2009","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2018] Kaixuan Chen, Lina Yao, Xianzhi Wang, Dalin Zhang, Tao Gu, Zhiwen Yu, and Zheng Yang. Inter- pretable parallel recurrent neural networks with convolu- tional attentions for multi-modality activity modeling. In Neural Networks (IJCNN), 2018 International Joint Con- ference on, pages 3016-3021. IEEE, 2018.","title":"Interpretable parallel recurrent neural networks with convolutional attentions for multi-modality activity modeling","authors":"Kaixuan Chen, Lina Yao, Xianzhi Wang, Dalin Zhang, Tao Gu, Zhiwen Yu, Zheng Yang","year":"2018","venue":"Neural Networks (IJCNN), 2018 International Joint Conference on","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Denil et al., 2012] Misha Denil, Loris Bazzani, Hugo Larochelle, and Nando de Freitas. Learning where to at- tend with deep architectures for image tracking. Neural computation, 24(8):2151-2184, 2012.","title":"Learning where to attend with deep architectures for image tracking","authors":"Misha Denil, Loris Bazzani, Hugo Larochelle, Nando De Freitas","year":"2012","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Freedman and Zilberstein, 2018] Richard Gabriel Freed- man and Shlomo Zilberstein. Roles that plan, activity, and intent recognition with planning can play in games. In Workshops at the Thirty-Second AAAI Conference on Ar- tificial Intelligence, 2018.","title":"Richard Gabriel Freedman and Shlomo Zilberstein. Roles that plan, activity, and intent recognition with planning can play in games","authors":"Zilberstein Freedman","year":"2018","venue":"Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Guan and Plötz, 2017] Yu Guan and Thomas Plötz. En- sembles of deep lstm learners for activity recognition us- ing wearables. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, IMWUT, 1(2):11, 2017.","title":"Ensembles of deep lstm learners for activity recognition using wearables","authors":"Yu Guan, Thomas Plötz","year":"2017","venue":"Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Guo et al., 2016] Haodong Guo, Ling Chen, Liangying Peng, and Gencai Chen. Wearable sensor based multi- modal human activity recognition exploiting the diversity of classifier ensemble. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiqui- tous Computing, UbiComp 2016, Heidelberg, Germany, September 12-16, 2016, pages 1112-1123, 2016.","title":"Wearable sensor based multimodal human activity recognition exploiting the diversity of classifier ensemble","authors":"Guo","year":"2016","venue":"Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp 2016","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guo et al., 2017] Xiaonan Guo, Jian Liu, and Yingying Chen. Fitcoach: Virtual fitness coach empowered by wear- able mobile devices. In 2017 IEEE Conference on Com- puter Communications, INFOCOM 2017, Atlanta, GA, USA, May 1-4, 2017, pages 1-9, 2017.","title":"Fitcoach: Virtual fitness coach empowered by wearable mobile devices","authors":"Xiaonan Guo, Jian Liu, Yingying Chen","year":"2017","venue":"IEEE Conference on Computer Communications, INFOCOM 2017","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hammerla et al., 2016] Nils Y. Hammerla, Shane Halloran, and Thomas Plötz. Deep, convolutional, and recurrent models for human activity recognition using wearables. In Proceedings of the Twenty-Fifth International Joint Con- ference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016, pages 1533-1540, 2016.","title":"Deep, convolutional, and recurrent models for human activity recognition using wearables","authors":"Nils Y Hammerla, Shane Halloran, Thomas Plötz","year":"2016","venue":"Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016","doi":null,"url":null,"internal_url":"/proceedings/2016/220"},{"position":11,"raw":"[Lara and Labrador, 2013] Oscar D Lara and Miguel A Labrador. A survey on human activity recognition using wearable sensors. IEEE Communications Surveys & Tuto- rials, 15(3):1192-1209, 2013.","title":"A survey on human activity recognition using wearable sensors","authors":"D Oscar, Miguel A Lara, Labrador","year":"2013","venue":"IEEE Communications Surveys & Tutorials","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2014] Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In Advances in neural information processing systems, pages 2204-2212, 2014.","title":"Recurrent models of visual attention","authors":"Volodymyr Mnih, Nicolas Heess, Alex Graves","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pouyanfar et al., 2018] Samira Pouyanfar, Saad Sadiq, Yilin Yan, Haiman Tian, Yudong Tao, Maria Presa Reyes, Mei-Ling Shyu, Shu-Ching Chen, and SS Iyengar. A sur- vey on deep learning: Algorithms, techniques, and ap- plications. ACM Computing Surveys (CSUR), 51(5):92, 2018.","title":"A survey on deep learning: Algorithms, techniques, and applications","authors":"Samira Pouyanfar, Saad Sadiq, Yilin Yan, Haiman Tian, Yudong Tao, Maria Presa Reyes, Mei-Ling Shyu, Shu-Ching Chen, Iyengar","year":"2018","venue":"ACM Computing Surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Radu et al., 2018] Valentin Radu, Catherine Tong, Sourav Bhattacharya, Nicholas D Lane, Cecilia Mascolo, Ma- hesh K Marina, and Fahim Kawsar. Multimodal deep learning for activity and context recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiqui- tous Technologies, IMWUT, 1(4):157, 2018.","title":"Multimodal deep learning for activity and context recognition","authors":"Valentin Radu, Catherine Tong, Sourav Bhattacharya, Nicholas D Lane, Cecilia Mascolo, Mahesh K Marina, Fahim Kawsar","year":"2018","venue":"Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Reiss and Stricker, 2012] Attila Reiss and Didier Stricker. Introducing a new benchmarked dataset for activity moni- toring. In Wearable Computers (ISWC), 2012 16th Inter- national Symposium on, pages 108-109. IEEE, 2012.","title":"Introducing a new benchmarked dataset for activity monitoring","authors":"Stricker Reiss, Attila Reiss, Didier Stricker","year":"2012","venue":"Wearable Computers (ISWC), 2012 16th International Symposium on","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wang and Wang, 2017] Hongsong Wang and Liang Wang. Modeling temporal dynamics and spatial configurations of actions using two-stream recurrent neural networks. The Conference on Computer Vision and Pattern Recognition (CVPR), 2017.","title":"Modeling temporal dynamics and spatial configurations of actions using two-stream recurrent neural networks","authors":"Wang, Wang ;, Hongsong Wang, Liang Wang","year":"2017","venue":"The Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4):229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Yang et al., 2015] Jianbo Yang, Minh Nhut Nguyen, Phyo Phyo San, Xiaoli Li, and Shonali Krishnaswamy. Deep convolutional neural networks on multichannel time series for human activity recognition. In IJCAI, pages 3995-4001, 2015.","title":"Deep convolutional neural networks on multichannel time series for human activity recognition","authors":"Jianbo Yang, Minh Nhut Nguyen, Phyo Phyo San, Xiaoli Li, Shonali Krishnaswamy","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/561"},{"position":19,"raw":"[Zhang et al., 2018] Xiang Zhang, Lina Yao, Chaoran Huang, Sen Wang, Mingkui Tan, Guodong Long, and Can Wang. Multi-modality sensor data classification with se- lective attention. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden., pages 3111-3117, 2018.","title":"Multi-modality sensor data classification with selective attention","authors":"Xiang Zhang, Lina Yao, Chaoran Huang, Sen Wang, Mingkui Tan, Guodong Long, Can Wang","year":"2018","venue":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018","doi":null,"url":null,"internal_url":"/proceedings/2018/432"},{"position":20,"raw":"[Zontak et al., 2013] Maria Zontak, Inbar Mosseri, and Michal Irani. Separating signal from noise using patch recurrence across scales. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1195-1202, 2013.","title":"Separating signal from noise using patch recurrence across scales","authors":"Maria Zontak, Inbar Mosseri, Michal Irani","year":"2013","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}