{"type":"ScholarlyArticle","title":"A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction","authors":[{"name":"Yao Qin","slug":"yao-qin","given":"Yao","family":"Qin","affiliation":"University of California , San Diego"},{"name":"Dongjin Song","slug":"dongjin-song","given":"Dongjin","family":"Song","affiliation":"NEC Laboratories America, Inc."},{"name":"Haifeng Chen","slug":"haifeng-chen","given":"Haifeng","family":"Chen","affiliation":"NEC Laboratories America, Inc."},{"name":"Wei Cheng","slug":"wei-cheng","given":"Wei","family":"Cheng","affiliation":"NEC Laboratories America, Inc."},{"name":"Guofei Jiang","slug":"guofei-jiang","given":"Guofei","family":"Jiang","affiliation":"NEC Laboratories America, Inc."},{"name":"Garrison W. Cottrell","slug":"garrison-w-cottrell","given":"Garrison W.","family":"Cottrell","affiliation":"University of California , San Diego"}],"abstract":"The Nonlinear autoregressive exogenous (NARX) model, which predicts the current value of a time series based upon its previous values as well as the current and past values of multiple driving (exogenous) series, has been studied for decades. Despite the fact that various NARX models have been developed, few of them can capture the long-term temporal dependencies appropriately and select the relevant driving series to make predictions. In this paper, we propose a dual-stage attention-based recurrent neural network (DA-RNN) to address these two issues. In the first stage, we introduce an input attention mechanism to adaptively extract relevant driving series (a.k.a., input features) at each time step by referring to the previous encoder hidden state. In the second stage, we use a temporal attention mechanism to select relevant encoder hidden states across all time steps. With this dual-stage attention scheme, our model can not only make predictions effectively, but can also be easily interpreted. Thorough empirical studies based upon the SML 2010 dataset and the NASDAQ 100 Stock dataset demonstrate that the DA-RNN can outperform state-of-the-art methods for time series prediction.","doi":"10.24963/ijcai.2017/366","url":"https://ijcai.org/proceedings/2017/366","pdf":"https://ijcai.org/proceedings/2017/0366.pdf","markdown":"https://ijcai.org/proceedings/2017/366.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/366","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":366,"pages":{"first":2627,"last":2633},"keywords":["Machine Learning: Time-series/Data Streams","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2015] Martın Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous systems. 2015.","title":"Tensorflow: Large-scale machine learning on heterogeneous systems","authors":"Abadi","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Asteriou and Hall, 2011] Dimitros Asteriou and Stephen G Hall. Arima models and the box-jenkins methodology. Applied Econo- metrics, 2(2):265-286, 2011.","title":"Dimitros Asteriou and Stephen G Hall. Arima models and the box-jenkins methodology","authors":"Hall Asteriou","year":"2011","venue":"Applied Econometrics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bengio et al., 1994] Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157- 166, 1994.","title":"Learning long-term dependencies with gradient descent is difficult","authors":"Yoshua Bengio, Patrice Simard, Paolo Frasconi","year":"1994","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bouchachia and Bouchachia, 2008] Abdelhamid Bouchachia and Saliha Bouchachia. Ensemble learning for time series prediction. In Proceedings of the 1st International Workshop on Nonlinear Dynamics and Synchronization, 2008.","title":"Ensemble learning for time series prediction","authors":"Abdelhamid Bouchachia, Saliha Bouchachia","year":"2008","venue":"Proceedings of the 1st International Workshop on Nonlinear Dynamics and Synchronization","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brockwell and Davis, 2009] Peter J. Brockwell and Richard A Davis. Time Series: Theory and Methods (2nd ed.). Springer, 2009.","title":"Time Series: Theory and Methods","authors":"Peter J Brockwell, Richard A Davis","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chakraborty et al., 2012] Prithwish Chakraborty, Manish Mar- wah, Martin F Arlitt, and Naren Ramakrishnan. Fine-grained photovoltaic output prediction using a bayesian ensemble. In AAAI, 2012.","title":"Fine-grained photovoltaic output prediction using a bayesian ensemble","authors":"Chakraborty","year":"2012","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2008] S. Chen, X. X. Wang, and C. J. Harris. Narx- based nonlinear system identification using orthogonal least squares basis hunting. IEEE Transactions on Control Systems Technology, 16(1):78-84, 2008.","title":"Narxbased nonlinear system identification using orthogonal least squares basis hunting","authors":"S Chen, X X Wang, C J Harris","year":"2008","venue":"IEEE Transactions on Control Systems Technology","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cho et al., 2014a] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv:1409.1259, 2014.","title":"On the properties of neural machine translation: Encoder-decoder approaches","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Cho et al., 2014b] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations us- ing rnn encoder-decoder for statistical machine translation. arXiv:1406.1078, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Diaconescu, 2008] Eugen Diaconescu. The use of NARX neural networks to predict chaotic time series. WSEA Transactions on Computer Research, 3(3), 2008.","title":"The use of NARX neural networks to predict chaotic time series","authors":"Eugen Diaconescu","year":"2008","venue":"WSEA Transactions on Computer Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Elman, 1991] Jeffrey L Elman. Distributed representations, simple recurrent networks, and grammatical structure. Machine learn- ing, 7(2-3):195-225, 1991.","title":"Distributed representations, simple recurrent networks, and grammatical structure","authors":"Jeffrey L Elman","year":"1991","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Frigola and Rasmussen, 2014] R. Frigola and C. E. Rasmussen. Integrated pre-processing for bayesian nonlinear system identifi- cation with gaussian processes. In IEEE Conference on Decision and Control, pages 552-560, 2014.","title":"Integrated pre-processing for bayesian nonlinear system identification with gaussian processes","authors":"R Frigola, C E Rasmussen","year":"2014","venue":"IEEE Conference on Decision and Control","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gao and Er, 2005] Yang Gao and Meng Joo Er. Narmax time se- ries model prediction: feedforward and recurrent fuzzy neural network approaches. Fuzzy Sets and Systems, 150(2):331-350, 2005.","title":"Narmax time series model prediction: feedforward and recurrent fuzzy neural network approaches","authors":"Yang Gao, Meng Joo, Er","year":"2005","venue":"Fuzzy Sets and Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Graves et al., 2013] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neu- ral networks. In ICASSP, pages 6645-6649, 2013.","title":"Speech recognition with deep recurrent neural networks","authors":"Alex Graves, Abdel-Rahman Mohamed, Geoffrey Hinton","year":"2013","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hübner et al., 2010] Ronald Hübner, Marco Steinhauser, and Car- ola Lehle. A dual-stage two-phase model of selective attention. Psychological Review, 117(3):759-784, 2010.","title":"A dual-stage two-phase model of selective attention","authors":"Hübner","year":"2010","venue":"Psychological Review","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kalchbrenner and Blunsom, 2013] Nal Kalchbrenner and Phil Blunsom. Recurrent continuous translation models. In EMNLP, volume 3, pages 413-422, 2013.","title":"Recurrent continuous translation models","authors":"Nal Kalchbrenner, Phil Blunsom","year":"2013","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Karpathy and Li, 2015] Andrej Karpathy and Fei-Fei Li. Deep visual-semantic alignments for generating image descriptions. In CVPR, pages 3128-3137, 2015.","title":"Deep visual-semantic alignments for generating image descriptions","authors":"Li ; Karpathy, Andrej Karpathy, Fei-Fei Li","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lin et al., 1996] Tsungnan Lin, Bill G. Horne, Peter Tino, and C. Lee Giles. Learning long-term dependencies in NARX recur- rent neural networks. IEEE Transactions on Neural Networks, 7(6):1329-1338, 1996.","title":"Learning long-term dependencies in NARX recurrent neural networks","authors":"Tsungnan Lin, Bill G Horne, Peter Tino, C Lee Giles","year":"1996","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu and Hauskrecht, 2015] Zitao Liu and Milos Hauskrecht. A regularized linear dynamical system framework for multivariate time series analysis. In AAAI, pages 1798-1805, 2015.","title":"A regularized linear dynamical system framework for multivariate time series analysis","authors":"Hauskrecht Liu, Zitao Liu, Milos Hauskrecht","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Plutowski et al., 1996] Mark Plutowski, Garrison Cottrell, and Halbert White. Experience with selecting exemplars from clean data. Neural Networks, 9(2):273-294, 1996.","title":"Experience with selecting exemplars from clean data","authors":"Mark Plutowski, Garrison Cottrell, Halbert White","year":"1996","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pu et al., 2016] Yunchen Pu, Martin Renqiang Min, Zhe Gan, and Lawrence Carin. Adaptive feature abstraction for translating video to language. arXiv preprint arXiv:1611.07837, 2016.","title":"Adaptive feature abstraction for translating video to language","authors":"Pu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Qin et al., 2015] Yao Qin, Huchuan Lu, Yiqun Xu, and He Wang. Saliency detection via cellular automata. In CVPR, pages 110- 119, 2015.","title":"Saliency detection via cellular automata","authors":"Huchuan Yao Qin, Yiqun Lu, He Xu, Wang","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Rumelhart et al., 1986] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back- propagating errors. Nature, 323(9):533-536, 1986.","title":"Learning representations by backpropagating errors","authors":"Geoffrey E David E Rumelhart, Ronald J Hinton, Williams","year":"1986","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Song et al., 2015] Dongjin Song, Wei Liu, Rongrong Ji, David A Meyer, and John R Smith. Top rank supervised binary coding for visual search. In ICCV, pages 1922-1930, 2015.","title":"Top rank supervised binary coding for visual search","authors":"Dongjin Song, Wei Liu, Rongrong Ji, David A Meyer, John R Smith","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Song et al., 2016] Dongjin Song, Wei Liu, and David A Meyer. Fast structural binary coding. In IJCAI, pages 2018-2024, 2016.","title":"Fast structural binary coding","authors":"Dongjin Song, Wei Liu, David A Meyer","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/288"},{"position":29,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NIPS, pages 3104-3112, 2014.","title":"Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Werbos, 1990] Paul J Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78(10):1550-1560, 1990.","title":"Backpropagation through time: what it does and how to do it","authors":"Werbos Paul","year":"1990","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Whittle, 1951] P. Whittle. Hypothesis Testing in Time Series Anal- ysis. PhD thesis, 1951.","title":"Hypothesis Testing in Time Series Analysis","authors":"P Whittle","year":"1951","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wu et al., 2013] Yue Wu, José Miguel Hernández-Lobato, and Zoubin Ghahramani. Dynamic covariance models for multivari- ate financial time series. In ICML, pages 558-566, 2013.","title":"Dynamic covariance models for multivariate financial time series","authors":"Yue Wu, José Miguel Hernández-Lobato, Zoubin Ghahramani","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdinov, Richard S Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, volume 14, pages 77-81, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdinov, Richard S Zemel, Yoshua Bengio","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yan et al., 2013] Linjun Yan, Ahmed Elgamal, and Garrison W. Cottrell. Substructure vibration NARX neural network approach for statistical damage inference. Journal of Engineering Mechan- ics, 139:737-747, 2013.","title":"Substructure vibration NARX neural network approach for statistical damage inference","authors":"Yan","year":"2013","venue":"Journal of Engineering Mechanics","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yang et al., 2016] Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. Hierarchical attention net- works for document classification. In NAACL, 2016.","title":"Hierarchical attention networks for document classification","authors":"Yang","year":"2016","venue":"NAACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}