{"type":"ScholarlyArticle","title":"A Reinforcement Learning-Informed Pattern Mining Framework for Multivariate Time Series Classification","authors":[{"name":"Ge Gao","slug":"ge-gao","given":"Ge","family":"Gao","affiliation":"Department of Computer Science , North Carolina State University , USA"},{"name":"Qitong Gao","slug":"qitong-gao","given":"Qitong","family":"Gao","affiliation":"Department of Electrical and Computer Engineering , Duke University , USA"},{"name":"Xi Yang","slug":"xi-yang","given":"Xi","family":"Yang","affiliation":"Department of Computer Science , North Carolina State University , USA"},{"name":"Miroslav Pajic","slug":"miroslav-pajic","given":"Miroslav","family":"Pajic","affiliation":"Department of Electrical and Computer Engineering , Duke University , USA"},{"name":"Min Chi","slug":"min-chi","given":"Min","family":"Chi","affiliation":"Department of Computer Science , North Carolina State University , USA"}],"abstract":"Multivariate time series (MTS) classification is a challenging and important task in various domains and real-world applications. Much of prior work on MTS can be roughly divided into neural network (NN)- and pattern-based methods. The former can lead to robust classification performance, but many of the generated patterns are challenging to interpret; while the latter often produce interpretable patterns that may not be helpful for the classification task. In this work, we propose a reinforcement learning (RL) informed PAttern Mining framework (RLPAM) to identify interpretable yet important patterns for MTS classification. Our framework has been validated by 30 benchmark datasets as well as real-world large-scale electronic health records (EHRs) for an extremely challenging task: sepsis shock early prediction. We show that RLPAM outperforms the state-of-the-art NN-based methods on 14 out of 30 datasets as well as on the EHRs. Finally, we show how RL informed patterns can be interpretable and can improve our understanding of septic shock progression.","doi":"10.24963/ijcai.2022/415","url":"https://ijcai.org/proceedings/2022/415","pdf":"https://ijcai.org/proceedings/2022/0415.pdf","markdown":"https://ijcai.org/proceedings/2022/415.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/415","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":415,"pages":{"first":2994,"last":3000},"keywords":["Multidisciplinary Topics and Applications: Health and Medicine","Machine Learning: Classification","Machine Learning: Deep Reinforcement Learning","Machine Learning: Time-series; Data Streams"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986802","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984945","length_sec":356,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Anzai, 2012] Yuichiro Anzai. Pattern recognition and machine learning. Elsevier, 2012.","title":"Pattern recognition and machine learning","authors":"Yuichiro Anzai, Anzai","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bagnall et al., 2018] Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive. arXiv preprint arXiv:1811.00075, 2018.","title":"The uea multivariate time series classification archive","authors":"Anthony Bagnall, Anh Hoang, Jason Dau, Michael Lines, James Flynn, Aaron Large, Paul Bostrom, Eamonn Southam, Keogh","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dempster et al., 2020] Angus Dempster, Franc ¸ois Petitjean, and Geoffrey I Webb. Rocket: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 34(5):1454-1495, 2020.","title":"Rocket: exceptionally fast and accurate time series classification using random convolutional kernels","authors":"Dempster","year":"2020","venue":"Data Mining and Knowledge Discovery","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dempster et al., 2021] Angus Dempster, Daniel F Schmidt, and Geoffrey I Webb. Minirocket: A very fast (almost) determin- istic transform for time series classification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 248-257, 2021.","title":"Minirocket: A very fast (almost) deterministic transform for time series classification","authors":"Dempster","year":"2021","venue":"Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Demšar, 2006] Janez Demšar. Statistical comparisons of classifiers over multiple data sets. JMLR, 7:1-30, 2006.","title":"Janez Demšar. Statistical comparisons of classifiers over multiple data sets","authors":"Demšar","year":"2006","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dulac-Arnold et al., 2015] Gabriel Dulac-Arnold, Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. Deep reinforcement learning in large discrete action spaces. arXiv preprint arXiv:1512.07679, 2015.","title":"Deep reinforcement learning in large discrete action spaces","authors":"Gabriel Dulac-Arnold, Richard Evans, Hado Van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, Ben Coppin","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gao et al., 2021] Ge Gao, Samiha Marwan, and Thomas W Price. Early performance prediction using interpretable patterns in pro- gramming process data. In Proceedings of the 52nd ACM Techni- cal Symposium on Computer Science Education, pages 342-348, 2021.","title":"Early performance prediction using interpretable patterns in programming process data","authors":"Ge Gao, Samiha Marwan, Thomas W Price","year":"2021","venue":"Proceedings of the 52nd ACM Technical Symposium on Computer Science Education","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gao et al., 2022a] Ge Gao, Farzaneh Khoshnevisan, and Min Chi. Reconstructing missing ehrs using time-aware within-and cross- visit information for septic shock early prediction. ICHI, 2022.","title":"Reconstructing missing ehrs using time-aware within-and crossvisit information for septic shock early prediction","authors":"Gao","year":"2022","venue":"ICHI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gao et al., 2022b] Qitong Gao, Dong Wang, Joshua D Amason, Siyang Yuan, Chenyang Tao, Ricardo Henao, Majda Hadziahme- tovic, Lawrence Carin, and Miroslav Pajic. Gradient importance learning for incomplete observations. ICLR, 2022.","title":"Gradient importance learning for incomplete observations","authors":"Gao","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hallac et al., 2017] David Hallac, Sagar Vare, Stephen Boyd, and Jure Leskovec. Toeplitz inverse covariance-based clustering of multivariate time series data. In ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, pages 215-223, 2017.","title":"Toeplitz inverse covariance-based clustering of multivariate time series data","authors":"David Hallac, Sagar Vare, Stephen Boyd, Jure Leskovec","year":"2017","venue":"ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":"/proceedings/2018/732"},{"position":11,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Holm, 1979] Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pages 65-70, 1979.","title":"A simple sequentially rejective multiple test procedure","authors":"Sture Holm","year":"1979","venue":"Scandinavian journal of statistics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Johnson et al., 2016] Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghas- semi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.","title":"Mimic-iii, a freely accessible critical care database","authors":"E W Alistair, Tom J Johnson, Lu Pollard, H Shen, Mengling Lehman Li-Wei, Mohammad Feng, Benjamin Ghassemi, Peter Moody, Leo Szolovits, Roger G Anthony Celi, Mark","year":"2016","venue":"Scientific data","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kang and Choi, 2014] Hyohyeong Kang and Seungjin Choi. Bayesian common spatial patterns for multi-subject eeg classi- fication. Neural Networks, 57:39-50, 2014.","title":"Bayesian common spatial patterns for multi-subject eeg classification","authors":"Choi Kang, Hyohyeong Kang, Seungjin Choi","year":"2014","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Karim et al., 2019] Fazle Karim, Somshubra Majumdar, Houshang Darabi, and Samuel Harford. Multivariate lstm-fcns for time series classification. Neural Networks, 116:237-245, 2019.","title":"Multivariate lstm-fcns for time series classification","authors":"Fazle Karim, Somshubra Majumdar, Houshang Darabi, Samuel Harford","year":"2019","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Keogh and Lin, 2005] Eamonn Keogh and Jessica Lin. Clustering of time-series subsequences is meaningless: implications for pre- vious and future research. Knowledge and information systems, 8(2):154-177, 2005.","title":"Clustering of time-series subsequences is meaningless: implications for previous and future research","authors":"Lin ; ] Eamonn Keogh, Jessica Keogh, Lin","year":"2005","venue":"Knowledge and information systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Le Nguyen et al., 2019] Thach Le Nguyen, Severin Gsponer, Iulia Ilie, Martin O'Reilly, and Georgiana Ifrim. Interpretable time se- ries classification using linear models and multi-resolution multi- domain symbolic representations. Data mining and knowledge discovery, 33(4):1183-1222, 2019.","title":"Interpretable time series classification using linear models and multi-resolution multidomain symbolic representations","authors":"Le Thach, Severin Nguyen, Iulia Gsponer, Ilie, O' Martin, Georgiana Reilly, Ifrim","year":"2019","venue":"Data mining and knowledge discovery","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2021] Guozhong Li, Byron Choi, Jianliang Xu, et al. Shapenet: A shapelet-neural network approach for multivariate time series classification. In AAAI, volume 35, pages 8375-8383, 2021.","title":"Shapenet: A shapelet-neural network approach for multivariate time series classification","authors":"Guozhong Li, Byron Choi, Jianliang Xu","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lillicrap et al., 2016] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, et al. Continuous control with deep reinforce- ment learning. ICLR, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Pritzel","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lloyd, 1982] Stuart Lloyd. Least squares quantization in pcm. IEEE Tran. on Info. Theory, 28(2):129-137, 1982.","title":"Least squares quantization in pcm","authors":"Stuart Lloyd","year":"1982","venue":"IEEE Tran. on Info. Theory","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Minnen et al., 2006] David Minnen, Thad Starner, et al. Discover- ing characteristic actions from on-body sensor data. In 2006 10th IEEE international symposium on wearable computers, pages 11-18. IEEE, 2006.","title":"Discovering characteristic actions from on-body sensor data","authors":"David Minnen, Thad Starner","year":"2006","venue":"10th IEEE international symposium on wearable computers","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rakthanmanon and Keogh, 2013] Thanawin Rakthanmanon and Eamonn Keogh. Fast shapelets: A scalable algorithm for dis- covering time series shapelets. In SDM, pages 668-676. SIAM, 2013.","title":"Thanawin Rakthanmanon and Eamonn Keogh. Fast shapelets: A scalable algorithm for discovering time series shapelets","authors":"Keogh Rakthanmanon","year":"2013","venue":"SDM","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ruder, 2016] Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747, 2016.","title":"An overview of gradient descent optimization algorithms","authors":"Sebastian Ruder","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ruiz et al., 2021] Alejandro Pasos Ruiz, Michael Flynn, James Large, Matthew Middlehurst, and Anthony Bagnall. The great multivariate time series classification bake off: a review and ex- perimental evaluation of recent algorithmic advances. Data Min- ing and Knowledge Discovery, 35(2):401-449, 2021.","title":"The great multivariate time series classification bake off: a review and experimental evaluation of recent algorithmic advances","authors":"Alejandro Pasos Ruiz, Michael Flynn, James Large, Matthew Middlehurst, Anthony Bagnall","year":"2021","venue":"Data Mining and Knowledge Discovery","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Schäfer and Leser, 2018] Patrick Schäfer and Ulf Leser. Multivari- ate time series classification with weasel+ muse. arXiv, 2018.","title":"Multivariate time series classification with weasel+ muse","authors":"Leser Schäfer, Patrick Schäfer, Ulf Leser","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Senin and Malinchik, 2013] Pavel Senin and Sergey Malinchik. Sax-vsm: Interpretable time series classification using sax and vector space model. In ICDM, pages 1175-1180. IEEE, 2013.","title":"Sax-vsm: Interpretable time series classification using sax and vector space model","authors":"Malinchik Senin, Pavel Senin, Sergey Malinchik","year":"2013","venue":"ICDM","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Shokoohi-Yekta et al., 2015] Mohammad Shokoohi-Yekta, Jun Wang, and Eamonn Keogh. On the non-trivial generalization of dynamic time warping to the multi-dimensional case. In SDM, pages 289-297. SIAM, 2015.","title":"On the non-trivial generalization of dynamic time warping to the multi-dimensional case","authors":"Mohammad Shokoohi-Yekta, Jun Wang, Eamonn Keogh","year":"2015","venue":"SDM","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tan et al., 2016] Pang-Ning Tan, Michael Steinbach, and Vipin Kumar. Introduction to data mining. Pearson Education India, 2016.","title":"Introduction to data mining","authors":"Pang-Ning Tan, Michael Steinbach, Vipin Kumar","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2021] Xi Yang, Yuan Zhang, and Min Chi. Multi- series time-aware sequence partitioning for disease progression modeling. In IJCAI, 2021.","title":"Multiseries time-aware sequence partitioning for disease progression modeling","authors":"Xi Yang, Yuan Zhang, Min Chi","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Ye and Keogh, 2009] Lexiang Ye and Eamonn Keogh. Time series shapelets: a new primitive for data mining. In ACM SIGKDD Int. Conf. on Knowledge discovery and data mining, pages 947-956, 2009.","title":"Time series shapelets: a new primitive for data mining","authors":"Ye, Keogh, Lexiang Ye, Eamonn Keogh","year":"2009","venue":"ACM SIGKDD Int. Conf. on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2020] Xuchao Zhang, Yifeng Gao, Jessica Lin, and Chang-Tien Lu. Tapnet: Multivariate time series classification with attentional prototypical network. In AAAI, volume 34, pages 6845-6852, 2020.","title":"Tapnet: Multivariate time series classification with attentional prototypical network","authors":"Xuchao Zhang, Yifeng Gao, Jessica Lin, Chang-Tien Lu","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zoph and Le, 2017] Barret Zoph and Quoc V Le. Neural architec- ture search with reinforcement learning. ICLR, 2017.","title":"Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning","authors":"Le Zoph","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}