{"type":"ScholarlyArticle","title":"A New Attention Mechanism to Classify Multivariate Time Series","authors":[{"name":"Yifan Hao","slug":"yifan-hao","given":"Yifan","family":"Hao","affiliation":"New Mexico State University"},{"name":"Huiping Cao","slug":"huiping-cao","given":"Huiping","family":"Cao","affiliation":"New Mexico State University"}],"abstract":"Classifying multivariate time series (MTS), which record the values of multiple variables over a continuous period of time, has gained a lot of attention. However, existing techniques suffer from two major issues. First, the long-range dependencies of the time-series sequences are not well captured. Second, the interactions of multiple variables are generally not represented in features. To address these aforementioned issues, we propose a novel Cross Attention Stabilized Fully Convolutional Neural Network (CA-SFCN) to classify MTS data. First, we introduce a temporal attention mechanism to extract long- and short-term memories across all time steps. Second, variable attention is designed to select relevant variables at each time step. CA-SFCN is compared with 16 approaches using 14 different MTS datasets. The extensive experimental results show that the CA-SFCN outperforms state-of-the-art classification methods, and the cross attention mechanism achieves better performance than other attention mechanisms.","doi":"10.24963/ijcai.2020/277","url":"https://ijcai.org/proceedings/2020/277","pdf":"https://ijcai.org/proceedings/2020/0277.pdf","markdown":"https://ijcai.org/proceedings/2020/277.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/277","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":277,"pages":{"first":1999,"last":2005},"keywords":["Machine Learning: Deep Learning: Convolutional networks","Machine Learning: Classification","Data Mining: Classification, Semi-Supervised Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25530/file_25530.m3u8","length_sec":249,"provider":"hls"}],"references":[{"position":1,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"Conference Track Proceedings","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baydogan and Runger, 2015] Mustafa Gokce Baydogan and George C. Runger. Learning a symbolic representa- tion for multivariate time series classification. Data Min. Knowl. Discov., 29(2):400-422, 2015.","title":"Learning a symbolic representation for multivariate time series classification","authors":"Gokce Mustafa, George C Baydogan, Runger","year":"2015","venue":"Data Min. Knowl. Discov","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baydogan and Runger, 2016] Mustafa Gokce Baydogan and George C. Runger. Time series representation and similarity based on local autopatterns. Data Min. Knowl. Discov., 30(2):476-509, 2016.","title":"Time series representation and similarity based on local autopatterns","authors":"Gokce Mustafa, George C Baydogan, Runger","year":"2016","venue":"Data Min. Knowl. Discov","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Boser et al., 1992] Bernhard E. Boser, Isabelle M. Guyon, and Vladimir N. Vapnik. A training algorithm for opti- mal margin classifiers. In Proceedings of the Fifth Annual Workshop on Computational Learning Theory, COLT, pages 144-152. ACM, 1992.","title":"A training algorithm for optimal margin classifiers","authors":"Bernhard E Boser, Isabelle M Guyon, Vladimir N Vapnik","year":"1992","venue":"Proceedings of the Fifth Annual Workshop on Computational Learning Theory","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dua and Graff, 2017] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.","title":"Dheeru Dua and Casey Graff. UCI machine learning repository","authors":"Graff Dua","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hao et al., 2019] Yifan Hao, Huiping Cao, Abdullah Mueen, and Sukumar Brahma. Identify significant phenomenon-specific variables for multivariate time se- ries. IEEE Trans. Knowl. Data Eng., 31(1):1-13, 2019.","title":"Identify significant phenomenon-specific variables for multivariate time series","authors":"Yifan Hao, Huiping Cao, Abdullah Mueen, Sukumar Brahma","year":"2019","venue":"IEEE Trans. Knowl. Data Eng","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ho, 1995] Tin Kam Ho. Random decision forests. In Pro- ceedings of the 3rd ICDAR Third International Conference on Document Analysis and Recognition (Volume 1) -Vol- ume 1, ICDAR '95, pages 278-. IEEE Computer Society, 1995.","title":"Random decision forests","authors":"Kam Tin, Ho","year":"1995","venue":"Proceedings of the 3rd ICDAR Third International Conference on Document Analysis and Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pages 448-456, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning, ICML 2015","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Karim et al., 2018] Fazle Karim, Somshubra Majumdar, Houshang Darabi, and Shun Chen. LSTM fully convolu- tional networks for time series classification. IEEE Access, 6:1662-1669, 2018.","title":"LSTM fully convolutional networks for time series classification","authors":"Fazle Karim, Somshubra Majumdar, Houshang Darabi, Shun Chen","year":"2018","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Karim et al., 2019] Fazle Karim, Somshubra Majumdar, Houshang Darabi, and Samuel Harford. Multivariate lstm-fcns for time series classification. Neural Networks, 116:237-245, 2019.","title":"Multivariate lstm-fcns for time series classification","authors":"Fazle Karim, Somshubra Majumdar, Houshang Darabi, Samuel Harford","year":"2019","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 3431-3440, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Jonathan Long, Evan Shelhamer, Trevor Darrell","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Miyato et al., 2018] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral nor- malization for generative adversarial networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 -May 3, 2018, Conference Track Proceedings, 2018.","title":"Spectral normalization for generative adversarial networks","authors":"Miyato","year":"2018","venue":"6th International Conference on Learning Representations, ICLR 2018","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pascanu et al., 2014] Razvan Pascanu, C ¸aglar Gülc ¸ehre, Kyunghyun Cho, and Yoshua Bengio. How to construct deep recurrent neural networks. In 2nd International Con- ference on Learning Representations, 2014.","title":"How to construct deep recurrent neural networks","authors":"Razvan Pascanu, C ¸aglar Gülc ¸ehre, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":"2nd International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pei et al., 2018] Wenjie Pei, Hamdi Dibeklioglu, David M. J. Tax, and Laurens van der Maaten. Multivariate time- series classification using the hidden-unit logistic model. IEEE Trans. Neural Netw. Learning Syst., 29(4):920-931, 2018.","title":"Multivariate timeseries classification using the hidden-unit logistic model","authors":"Wenjie Pei, Hamdi Dibeklioglu, M J David, Laurens Tax, Van Der Maaten","year":"2018","venue":"IEEE Trans. Neural Netw. Learning Syst","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Qin et al., 2017] Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison W. Cottrell. A dual-stage attention-based recurrent neural network for time series prediction. In Proceedings of the 26th IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 2627-2633, 2017.","title":"A dual-stage attention-based recurrent neural network for time series prediction","authors":"Dongjin Yao Qin, Haifeng Song, Wei Chen, Guofei Cheng, Garrison W Jiang, Cottrell","year":"2017","venue":"Proceedings of the 26th IJCAI 2017","doi":null,"url":null,"internal_url":"/proceedings/2017/366"},{"position":16,"raw":"[Schäfer and Leser, 2017] Patrick Schäfer and Ulf Leser. Multivariate time series classification with WEASEL+MUSE. CoRR, abs/1711.11343, 2017.","title":"Multivariate time series classification with WEASEL+MUSE","authors":"Leser Schäfer, Patrick Schäfer, Ulf Leser","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Seto et al., 2015] Skyler Seto, Wenyu Zhang, and Yichen Zhou. Multivariate time series classification using dy- namic time warping template selection for human activity recognition. In IEEE Symposium Series on Computational Intelligence, SSCI, pages 1399-1406, 2015.","title":"Multivariate time series classification using dynamic time warping template selection for human activity recognition","authors":"Skyler Seto, Wenyu Zhang, Yichen Zhou","year":"2015","venue":"IEEE Symposium Series on Computational Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tuncel and Baydogan, 2018] Kerem Sinan Tuncel and Mustafa Gokce Baydogan. Autoregressive forests for multivariate time series modeling. Pattern Recognition, 73:202-215, 2018.","title":"Autoregressive forests for multivariate time series modeling","authors":"Baydogan ; Tuncel, Sinan Kerem, Mustafa Tuncel, Gokce Baydogan","year":"2018","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems 30: Annual Conference on Neural Information Pro- cessing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2017] Zhiguang Wang, Weizhong Yan, and Tim Oates. Time series classification from scratch with deep neural networks: A strong baseline. In 2017 Interna- tional Joint Conference on Neural Networks, IJCNN 2017, Anchorage, AK, USA, May 14-19, 2017, pages 1578-1585, 2017.","title":"Time series classification from scratch with deep neural networks: A strong baseline","authors":"Zhiguang Wang, Weizhong Yan, Tim Oates","year":"2017","venue":"2017 International Joint Conference on Neural Networks, IJCNN 2017","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2015] Jianbo Yang, Minh Nhut Nguyen, Phyo Phyo San, Xiaoli Li, and Shonali Krishnaswamy. Deep convolutional neural networks on multichannel time series for human activity recognition. In Intl. Joint Conference on Artificial Intelligence, IJCAI, pages 3995-4001, 2015.","title":"Deep convolutional neural networks on multichannel time series for human activity recognition","authors":"Jianbo Yang, Minh Nhut Nguyen, Phyo Phyo San, Xiaoli Li, Shonali Krishnaswamy","year":"2015","venue":"Intl. Joint Conference on Artificial Intelligence, IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/561"},{"position":22,"raw":"[Yoon et al., 2005] Hyunjin Yoon, Kiyoung Yang, and Cyrus Shahabi. Feature subset selection and feature ranking for multivariate time series. IEEE Trans. Knowl. Data Eng., 17(9):1186-1198, 2005.","title":"Feature subset selection and feature ranking for multivariate time series","authors":"Hyunjin Yoon, Kiyoung Yang, Cyrus Shahabi","year":"2005","venue":"IEEE Trans. Knowl. Data Eng","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2019] Han Zhang, Ian J. Goodfellow, Dim- itris N. Metaxas, and Augustus Odena. Self-attention gen- erative adversarial networks. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pages 7354-7363, 2019.","title":"Self-attention generative adversarial networks","authors":"Han Zhang, Ian J Goodfellow, Dimitris N Metaxas, Augustus Odena","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning, ICML 2019","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}