{"type":"ScholarlyArticle","title":"DyAt Nets: Dynamic Attention Networks for State Forecasting in Cyber-Physical Systems","authors":[{"name":"Nikhil Muralidhar","slug":"nikhil-muralidhar","given":"Nikhil","family":"Muralidhar","affiliation":"Computer Science Department , Virginia Tech , USA"},{"name":"Sathappan Muthiah","slug":"sathappan-muthiah","given":"Sathappan","family":"Muthiah","affiliation":"Computer Science Department , Virginia Tech , USA"},{"name":"Naren Ramakrishnan","slug":"naren-ramakrishnan","given":"Naren","family":"Ramakrishnan","affiliation":"Computer Science Department , Virginia Tech , USA"}],"abstract":"Multivariate time series forecasting is an important task in state forecasting for cyber-physical systems  (CPS). State forecasting in CPS is imperative for optimal planning of system energy utility and understanding normal operational characteristics of the system thus enabling anomaly detection. Forecasting models can also be used to identify sub-optimal or worn out components and are thereby useful for overall system monitoring. Most existing work only performs single step forecasting but in CPS it is imperative to forecast the next sequence of system states (i.e  curve  forecasting). In this paper, we propose DyAt (Dynamic Attention) networks, a novel deep learning sequence to sequence (Seq2Seq) model with a novel hierarchical attention mechanism for long-term time series state forecasting. We evaluate our method on several CPS state forecasting and electric load forecasting tasks and find that our proposed DyAt models yield a performance improvement of at least 13.69% for the CPS state forecasting task and  a performance improvement of at least 18.83% for the electric load forecasting task over other state-of-the-art forecasting baselines. We perform rigorous experimentation with several variants of the DyAt model and demonstrate that the DyAt models indeed learn better representations over the entire course of the long term forecast as compared to their counterparts with or without traditional attention mechanisms. All data and source code has been made available online.","doi":"10.24963/ijcai.2019/441","url":"https://ijcai.org/proceedings/2019/441","pdf":"https://ijcai.org/proceedings/2019/0441.pdf","markdown":"https://ijcai.org/proceedings/2019/441.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/441","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":441,"pages":{"first":3180,"last":3186},"keywords":["Machine Learning: Deep Learning","Machine Learning: Data Mining","Machine Learning: Time-series;Data Streams","Machine Learning Applications: Big data ; Scalability"],"videos":[],"references":[{"position":1,"raw":"[Carvallo et al., 2017] Juan Pablo Carvallo, Peter H Larsen, Alan H Sanstad, and Charles A Goldman. Load forecast- ing in electric utility integrated resource planning. Tech- nical report, Lawrence Berkeley National Lab.(LBNL), Berkeley, CA (United States), 2017.","title":"Load forecasting in electric utility integrated resource planning","authors":"Carvallo","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cho et al., 2014a] Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the proper- ties of neural machine translation: Encoder-decoder ap- proaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103-111, 2014.","title":"On the properties of neural machine translation: Encoder-decoder approaches","authors":"Cho","year":"2014","venue":"Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cho et al., 2014b] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Process- ing (EMNLP), pages 1724-1734, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[De Gooijer and Hyndman, 2006] Jan G De Gooijer and Rob J Hyndman. 25 years of time series forecasting. In- ternational journal of forecasting, 2006.","title":"25 years of time series forecasting","authors":"Jan G, De Gooijer, Rob J Hyndman","year":"2006","venue":"ternational journal of forecasting","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Filonov et al., 2016] Pavel Filonov, Andrey Lavrentyev, and Artem Vorontsov. Multivariate industrial time se- ries with cyber-attack simulation: Fault detection us- ing an lstm-based predictive data model. arXiv preprint arXiv:1612.06676, 2016.","title":"Multivariate industrial time series with cyber-attack simulation: Fault detection using an lstm-based predictive data model","authors":"Filonov","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Filonov et al., 2017] Pavel Filonov, Fedor Kitashov, and Andrey Lavrentyev. Rnn-based early cyber-attack detec- tion for the tennessee eastman process. arXiv preprint arXiv:1709.02232, 2017.","title":"Rnn-based early cyber-attack detection for the tennessee eastman process","authors":"Filonov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gehring et al., 2017] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolu- tional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1243-1252. JMLR. org, 2017.","title":"Convolutional sequence to sequence learning","authors":"Gehring","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Juricek et al., 2001] Ben C Juricek, Dale E Seborg, and Wallace E Larimore. Identification of the tennessee east- man challenge process with subspace methods. Control Engineering Practice, 9(12):1337-1351, 2001.","title":"Identification of the tennessee eastman challenge process with subspace methods","authors":"Dale E Ben C Juricek, Wallace E Seborg, Larimore","year":"2001","venue":"Control Engineering Practice","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lai et al., 2018] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research & De- velopment in Information Retrieval, pages 95-104. ACM, 2018.","title":"Modeling long-and short-term temporal patterns with deep neural networks","authors":"Lai","year":"2018","venue":"The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin et al., 2018] Junyang Lin, SUN Xu, Shuming Ma, and Qi Su. Global encoding for abstractive summarization. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 163-169, 2018.","title":"Global encoding for abstractive summarization","authors":"Lin","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Luong et al., 2015] Thang Luong, Hieu Pham, and Christo- pher D Manning. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421, 2015.","title":"Effective approaches to attention-based neural machine translation","authors":"Thang Luong, Hieu Pham, Christopher D Manning","year":"2015","venue":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Malhotra et al., 2016] Pankaj Malhotra, Anusha Ramakr- ishnan, Gaurangi Anand, Lovekesh Vig, Puneet Agar- wal, and Gautam Shroff. Lstm-based encoder-decoder for multi-sensor anomaly detection. arXiv preprint arXiv:1607.00148, 2016.","title":"Lstm-based encoder-decoder for multi-sensor anomaly detection","authors":"Malhotra","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Nallapati et al., 2016] Ramesh Nallapati, Bowen Zhou, Ci- cero dos Santos, Caglar Gulcehre, and Bing Xiang. Ab- stractive text summarization using sequence-to-sequence rnns and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learn- ing, pages 280-290, 2016.","title":"Abstractive text summarization using sequence-to-sequence rnns and beyond","authors":"Ramesh Nallapati, Bowen Zhou, Caglar Cicero Dos Santos, Bing Gulcehre, Xiang","year":"2016","venue":"Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Qiu et al., 2014] Xueheng Qiu, Le Zhang, Ye Ren, Pon- nuthurai N Suganthan, and Gehan Amaratunga. Ensemble deep learning for regression and time series forecasting. In 2014 IEEE symposium on computational intelligence in ensemble learning (CIEL), pages 1-6. IEEE, 2014.","title":"Ensemble deep learning for regression and time series forecasting","authors":"Qiu","year":"2014","venue":"IEEE symposium on computational intelligence in ensemble learning (CIEL)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Roberts et al., 2013] Stephen Roberts, Michael Osborne, Mark Ebden, Steven Reece, Neale Gibson, and Suzanne Aigrain. Gaussian processes for time-series mod- elling. Philosophical Transactions of the Royal Soci- ety A: Mathematical, Physical and Engineering Sciences, 371(1984):20110550, 2013.","title":"Gaussian processes for time-series modelling","authors":"Stephen Roberts, Michael Osborne, Mark Ebden, Steven Reece, Neale Gibson, Suzanne Aigrain","year":"1984","venue":"Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Romeu et al., 2013] Pablo Romeu, Francisco Zamora- Martínez, Paloma Botella-Rocamora, and Juan Pardo. Time-series forecasting of indoor temperature using pre- trained deep neural networks. In International Conference on Artificial Neural Networks, pages 451-458. Springer, 2013.","title":"Time-series forecasting of indoor temperature using pretrained deep neural networks","authors":"Romeu","year":"2013","venue":"International Conference on Artificial Neural Networks","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Scherer et al., 2010] Dominik Scherer, Andreas Müller, and Sven Behnke. Evaluation of pooling operations in con- volutional architectures for object recognition. In Inter- national Conference on Artificial Neural Networks, pages 92-101. Springer, 2010.","title":"Evaluation of pooling operations in convolutional architectures for object recognition","authors":"Dominik Scherer, Andreas Müller, Sven Behnke","year":"2010","venue":"International Conference on Artificial Neural Networks","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sennrich et al., 2016] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1715-1725, 2016.","title":"Neural machine translation of rare words with subword units","authors":"Sennrich","year":"2016","venue":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104-3112, 2014.","title":"Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vapnik et al., 1997] Vladimir Vapnik, Steven E Golowich, and Alex J Smola. Support vector method for function approximation, regression estimation and signal process- ing. In Advances in Neural Information Processing Sys- tems, pages 281-287, 1997.","title":"Support vector method for function approximation, regression estimation and signal processing","authors":"Vladimir Vapnik, Steven E Golowich, Alex J Smola","year":"1997","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}