{"type":"ScholarlyArticle","title":"Predicting Complex Activities from Ongoing Multivariate Time Series","authors":[{"name":"Weihao Cheng","slug":"weihao-cheng","given":"Weihao","family":"Cheng","affiliation":"School of Computing Information and System , The University of Melbourne"},{"name":"Sarah Erfani","slug":"sarah-erfani","given":"Sarah","family":"Erfani","affiliation":"School of Computing Information and System , The University of Melbourne"},{"name":"Rui Zhang","slug":"rui-zhang","given":"Rui","family":"Zhang","affiliation":"School of Computing Information and System , The University of Melbourne"},{"name":"Ramamohanarao Kotagiri","slug":"ramamohanarao-kotagiri","given":"Ramamohanarao","family":"Kotagiri","affiliation":"School of Computing Information and System , The University of Melbourne"}],"abstract":"The rapid development of sensor networks enables recognition of complex activities (CAs) using multivariate time series. However, CAs are usually performed over long periods of time, which causes slow recognition by models based on fully observed data. Therefore, predicting CAs at early stages becomes an important problem. In this paper, we propose Simultaneous Complex Activities Recognition and Action Sequence Discovering (SimRAD), an algorithm which predicts a CA over time by mining a sequence of multivariate actions from sensor data using a Deep Neural Network. SimRAD simultaneously learns two probabilistic models for inferring CAs and action sequences, where the estimations of the two models are conditionally dependent on each other. SimRAD continuously predicts the CA and the action sequence, thus the predictions are mutually updated until the end of the CA. We conduct evaluations on a real-world CA dataset consisting of a rich amount of sensor data, and the results show that SimRAD outperforms state-of-the-art methods by average 7.2% in prediction accuracy with high confidence.","doi":"10.24963/ijcai.2018/461","url":"https://ijcai.org/proceedings/2018/461","pdf":"https://ijcai.org/proceedings/2018/0461.pdf","markdown":"https://ijcai.org/proceedings/2018/461.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/461","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning Applications","track":"Main track","number":461,"pages":{"first":3322,"last":3328},"keywords":["Machine Learning Applications: Other Applications","Multidisciplinary Topics and Applications: Ubiquitous Computing Systems"],"videos":[],"references":[{"position":1,"raw":"[Anderson et al., 2012] H. S Anderson, N. Parrish, K. Tsukida, and M. R Gupta. Reliable early classi- fication of time series. In Proceedings of ICASSP, pages 2073-2076, 2012.","title":"Reliable early classification of time series","authors":"H Anderson, N Parrish, K Tsukida, M R Gupta","year":"2012","venue":"Proceedings of ICASSP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chollet and others, 2015] Franc ¸ois Chollet et al. Keras. https://github.com/keras-team/keras, 2015.","title":"Franc ¸ois Chollet et al. Keras","authors":"Others Chollet","year":"2015","venue":null,"doi":null,"url":"https://github.com/keras-team/keras,2015","internal_url":null},{"position":3,"raw":"[Dachraoui et al., 2015] A. Dachraoui, A. Bondu, and A. Cornuéjols. Early classification of time series as a non myopic sequential decision making problem. In Proceed- ings of ECML-PKDD, pages 433-447, 2015.","title":"Early classification of time series as a non myopic sequential decision making problem","authors":"A Dachraoui, A Bondu, A Cornuéjols","year":"2015","venue":"Proceedings of ECML-PKDD","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fan et al., 2016] X. Fan, H. Zhang, C. Leung, and C. Miao. Comparative study of machine learning algorithms for ac- tivity recognition with data sequence in home-like envi- ronment. In Proceedings of MFI, pages 168-173, 2016.","title":"Comparative study of machine learning algorithms for activity recognition with data sequence in home-like environment","authors":"X Fan, H Zhang, C Leung, C Miao","year":"2016","venue":"Proceedings of MFI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ghalwash and Obradovic, 2012] M. F Ghalwash and Z. Obradovic. Early classification of multivariate tempo- ral observations by extraction of interpretable shapelets. BMC bioinformatics, 13(1):195, 2012.","title":"Early classification of multivariate temporal observations by extraction of interpretable shapelets","authors":"M Ghalwash, Z Obradovic","year":"2012","venue":"BMC bioinformatics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hammerla et al., 2016] N. Y. Hammerla, S. Halloran, and T. Plötz. Deep, convolutional, and recurrent models for human activity recognition using wearables. In Proceed- ings of IJCAI, pages 1533-1540, 2016.","title":"Deep, convolutional, and recurrent models for human activity recognition using wearables","authors":"N Y Hammerla, S Halloran, T Plötz","year":"2016","venue":"Proceedings of IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/220"},{"position":7,"raw":"[Höppner, 2001] F. Höppner. Discovery of temporal pat- terns. In Proceedings of PKDD, pages 192-203, 2001.","title":"Discovery of temporal patterns","authors":"F Höppner","year":"2001","venue":"Proceedings of PKDD","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Krishnan and Cook, 2014] N. C Krishnan and D. J Cook. Activity recognition on streaming sensor data. Pervasive and mobile computing, 10:138-154, 2014.","title":"Activity recognition on streaming sensor data","authors":"N Krishnan, D J Cook","year":"2014","venue":"Pervasive and mobile computing","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kwapisz et al., 2011] J. R Kwapisz, G. M Weiss, and S. A Moore. Activity recognition using cell phone accelerome- ters. ACM SigKDD Explorations Newsletter, 12(2):74-82, 2011.","title":"Activity recognition using cell phone accelerometers","authors":"J Kwapisz, G M Weiss, S A Moore","year":"2011","venue":"ACM SigKDD Explorations Newsletter","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li and Fu, 2014] K. Li and Y. Fu. Prediction of human ac- tivity by discovering temporal sequence patterns. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 36(8):1644-1657, 2014.","title":"Prediction of human activity by discovering temporal sequence patterns","authors":"K Li, Y Fu","year":"2014","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2014] K. Li, S. Li, and Y. Fu. Early classification of ongoing observation. In Proceedings of ICDM, pages 310-319, 2014.","title":"Early classification of ongoing observation","authors":"K Li, S Li, Y Fu","year":"2014","venue":"Proceedings of ICDM","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin et al., 2015] Y. Lin, H. Chen, V. S Tseng, and J. Pei. Re- liable early classification on multivariate time series with numerical and categorical attributes. In Proceedings of PAKDD, pages 199-211, 2015.","title":"Reliable early classification on multivariate time series with numerical and categorical attributes","authors":"Y Lin, H Chen, V S Tseng, J Pei","year":"2015","venue":"Proceedings of PAKDD","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2015] Y. Liu, L. Nie, L. Han, L. Zhang, and D. S Rosenblum. Action2activity: Recognizing complex ac- tivities from sensor data. In Proceedings of IJCAI, pages 1617-1623, 2015.","title":"Action2activity: Recognizing complex activities from sensor data","authors":"Y Liu, L Nie, L Han, L Zhang, D S Rosenblum","year":"2015","venue":"Proceedings of IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/231"},{"position":14,"raw":"[Liu et al., 2016] L. Liu, L. Cheng, Y. Liu, Y. Jia, and D. S Rosenblum. Recognizing complex activities by a prob- abilistic interval-based model. In Proceedings of AAAI, pages 1266-1272, 2016.","title":"Recognizing complex activities by a probabilistic interval-based model","authors":"Liu","year":"2016","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ma et al., 2016] S. Ma, L. Sigal, and S. Sclaroff. Learn- ing activity progression in lstms for activity detection and early detection. In Proceedings of CVPR, pages 1942- 1950, 2016.","title":"Learning activity progression in lstms for activity detection and early detection","authors":"S Ma, L Sigal, S Sclaroff","year":"2016","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pedregosa et al., 2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blon- del, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.","title":"Scikit-learn: Machine learning in Python","authors":"F Pedregosa, G Varoquaux, A Gramfort, V Michel, B Thirion, O Grisel, M Blondel, P Prettenhofer, R Weiss, V Dubourg, J Vanderplas, A Passos, D Cournapeau, M Brucher, M Perrot, E Duchesnay","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Roggen et al., 2010] D. Roggen, A. Calatroni, M. Rossi, T. Holleczek, K. Förster, G. Tröster, P. Lukowicz, D. Ban- nach, G. Pirkl, A. Ferscha, J. Doppler, C. Holzmann, M. Kurz, G. Holl, R. Chavarriaga, H. Sagha, H. Bayati, M. Creatura, and J. d. R. Millàn. Collecting complex activ- ity datasets in highly rich networked sensor environments. In Proceedings of INSS, pages 233-240, 2010.","title":"Collecting complex activity datasets in highly rich networked sensor environments","authors":"D Roggen, A Calatroni, M Rossi, T Holleczek, K Förster, G Tröster, P Lukowicz, D Bannach, G Pirkl, A Ferscha, J Doppler, C Holzmann, M Kurz, G Holl, R Chavarriaga, H Sagha, H Bayati, M Creatura, J D R Millàn","year":"2010","venue":"Proceedings of INSS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ryoo, 2011] M. S Ryoo. Human activity prediction: Early recognition of ongoing activities from streaming videos. In Proceedings of ICCV, pages 1036-1043, 2011.","title":"Human activity prediction: Early recognition of ongoing activities from streaming videos","authors":"M S Ryoo","year":"2011","venue":"Proceedings of ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Xing et al., 2009] Z. Xing, J. Pei, and P. Y. Yu. Early pre- diction on time series: A nearest neighbor approach. In Proceedings of IJCAI, pages 1297-1302, 2009.","title":"Early prediction on time series: A nearest neighbor approach","authors":"Z Xing, J Pei, P Y Yu","year":"2009","venue":"Proceedings of IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2009/218"},{"position":20,"raw":"[Xing et al., 2011] Z. Xing, J. Pei, P. S Yu, and K. Wang. Extracting interpretable features for early classification on time series. In Proceedings of SDM, pages 247-258, 2011.","title":"Extracting interpretable features for early classification on time series","authors":"Z Xing, J Pei, P S Yu, K Wang","year":"2011","venue":"Proceedings of SDM","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2015] J. B. Yang, M. N. Nguyen, P. P. San, X. L. Li, and S. Krishnaswamy. Deep convolutional neural networks on multichannel time series for human activity recognition. In Proceedings of IJCAI, pages 3995-4001, 2015.","title":"Deep convolutional neural networks on multichannel time series for human activity recognition","authors":"J B Yang, M N Nguyen, P P San, X L Li, S Krishnaswamy","year":"2015","venue":"Proceedings of IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/561"},{"position":22,"raw":"[Yang, 2009] Q. Yang. Activity recognition: Linking low- level sensors to high-level intelligence. In Proceedings of IJCAI, pages 20-25, 2009.","title":"Activity recognition: Linking lowlevel sensors to high-level intelligence","authors":"Q Yang","year":"2009","venue":"Proceedings of IJCAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2013] Y. Zhang, Y. Zhang, E. Swears, N. Lar- ios, Z. Wang, and Q. Ji. Modeling temporal interactions with interval temporal bayesian networks for complex ac- tivity recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(10):2468-2483, 2013.","title":"Modeling temporal interactions with interval temporal bayesian networks for complex activity recognition","authors":"Y Zhang, Y Zhang, E Swears, N Larios, Z Wang, Q Ji","year":"2013","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}