{"type":"ScholarlyArticle","title":"Memory Augmented State Space Model for Time Series Forecasting","authors":[{"name":"Yinbo Sun","slug":"yinbo-sun","given":"Yinbo","family":"Sun","affiliation":"Ant Group"},{"name":"Lintao Ma","slug":"lintao-ma","given":"Lintao","family":"Ma","affiliation":"Ant Group"},{"name":"Yu Liu","slug":"yu-liu","given":"Yu","family":"Liu","affiliation":"Ant Group"},{"name":"Shijun Wang","slug":"shijun-wang","given":"Shijun","family":"Wang","affiliation":"Ant Group"},{"name":"James Zhang","slug":"james-zhang","given":"James","family":"Zhang","affiliation":"Ant Group"},{"name":"YangFei Zheng","slug":"yangfei-zheng","given":"YangFei","family":"Zheng","affiliation":"Ant Group"},{"name":"Hu Yun","slug":"hu-yun","given":"Hu","family":"Yun","affiliation":"Ant Group"},{"name":"Lei Lei","slug":"lei-lei","given":"Lei","family":"Lei","affiliation":"Ant Group"},{"name":"Yulin Kang","slug":"yulin-kang","given":"Yulin","family":"Kang","affiliation":"Ant Group"},{"name":"Llinbao Ye","slug":"llinbao-ye","given":"Llinbao","family":"Ye","affiliation":"Ant Group"}],"abstract":"State space model (SSM) provides a general and flexible forecasting framework for time series. Conventional SSM with fixed-order Markovian assumption often falls short in handling the long-range temporal dependencies and/or highly non-linear correlation in time-series data, which is crucial for accurate forecasting. To this extend, we present External Memory Augmented State Space Model (EMSSM) within the sequential Monte Carlo (SMC) framework. Unlike the common fixed-order Markovian SSM, our model features an external memory system, in which we store informative latent state experience, whereby to create ``memoryful\" latent dynamics modeling complex long-term dependencies. Moreover, conditional normalizing flows are incorporated in our emission model, enabling the adaptation to a broad class of underlying data distributions. We further propose a Monte Carlo Objective that employs an efficient variational proposal distribution, which fuses the filtering and the dynamic prior information, to approximate the posterior state with proper particles. Our results demonstrate the competitiveness of forecasting performance of our proposed model comparing with other state-of-the-art SSMs.","doi":"10.24963/ijcai.2022/479","url":"https://ijcai.org/proceedings/2022/479","pdf":"https://ijcai.org/proceedings/2022/0479.pdf","markdown":"https://ijcai.org/proceedings/2022/479.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/479","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":479,"pages":{"first":3451,"last":3457},"keywords":["Machine Learning: Time-series; Data Streams","Machine Learning: Probabilistic Machine Learning","Uncertainty in AI: Inference","Machine Learning: Bayesian Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986733","length_sec":88,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984876","length_sec":355,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Alaa and van der Schaar, 2019] Ahmed M Alaa and Mi- haela van der Schaar. Attentive state-space modeling of disease progression. Advances in Neural Information Pro- cessing Systems, 32:11338-11348, 2019.","title":"Attentive state-space modeling of disease progression","authors":"Alaa, Van Der Schaar ; Mihaela Van Der Schaar","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arasaratnam and Haykin, 2009] Ienkaran Arasaratnam and Simon Haykin. Cubature kalman filters. IEEE Transac- tions on automatic control, 54(6):1254-1269, 2009.","title":"Ienkaran Arasaratnam and Simon Haykin. Cubature kalman filters","authors":"Haykin Arasaratnam","year":"2009","venue":"IEEE Transactions on automatic control","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bauwens et al., 2006] Luc Bauwens, Sébastien Laurent, and Jeroen VK Rombouts. Multivariate garch models: a survey. Journal of applied econometrics, 21(1):79-109, 2006.","title":"Multivariate garch models: a survey","authors":"Bauwens","year":"2006","venue":"Journal of applied econometrics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bialek et al., 2001] William Bialek, Ilya Nemenman, and Naftali Tishby. Predictability, complexity, and learning. Neural computation, 13(11):2409-2463, 2001.","title":"Predictability, complexity, and learning","authors":"William Bialek, Ilya Nemenman, Naftali Tishby","year":"2001","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Blei et al., 2017] David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisti- cians. Journal of the American statistical Association, 112(518):859-877, 2017.","title":"Variational inference: A review for statisticians","authors":"Alp David M Blei, Jon D Kucukelbir, Mcauliffe","year":"2017","venue":"Journal of the American statistical Association","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bošnjak et al., 2017] Matko Bošnjak, Tim Rocktäschel, Ja- son Naradowsky, and Sebastian Riedel. Programming with a differentiable forth interpreter. In International confer- ence on machine learning, pages 547-556. PMLR, 2017.","title":"Programming with a differentiable forth interpreter","authors":"Matko Bošnjak, Tim Rocktäschel, Jason Naradowsky, Sebastian Riedel","year":"2017","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chung et al., 2015] Junyoung Chung, Kyle Kastner, Lau- rent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. Advances in neural information processing systems, 28:2980-2988, 2015.","title":"A recurrent latent variable model for sequential data","authors":"Chung","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dinh et al., 2016] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803, 2016.","title":"Density estimation using real nvp","authors":"Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fraccaro et al., 2016] Marco Fraccaro, Søren Kaae Sønderby, Ulrich Paquet, and Ole Winther. Sequential neural models with stochastic layers. arXiv preprint arXiv:1605.07571, 2016.","title":"Sequential neural models with stochastic layers","authors":"Marco Fraccaro, Kaae Søren, Ulrich Sønderby, Ole Paquet, Winther","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fraccaro et al., 2017] Marco Fraccaro, Simon Kamronn, Ulrich Paquet, and Ole Winther. A disentangled recogni- tion and nonlinear dynamics model for unsupervised learn- ing. Advances in Neural Information Processing Systems, pages 3604--3613, 2017.","title":"A disentangled recognition and nonlinear dynamics model for unsupervised learning","authors":"Marco Fraccaro, Simon Kamronn, Ulrich Paquet, Ole Winther","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gemici et al., 2017] Mevlana Gemici, Chia-Chun Hung, Adam Santoro, Greg Wayne, Shakir Mohamed, Danilo J Rezende, David Amos, and Timothy Lillicrap. Gen- erative temporal models with memory. arXiv preprint arXiv:1702.04649, 2017.","title":"Generative temporal models with memory","authors":"Gemici","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gordon et al., 1993] Neil J Gordon, David J Salmond, and Adrian FM Smith. Novel approach to nonlinear/non- gaussian bayesian state estimation. In IEE Proceedings F-radar and signal processing, volume 140, pages 107- 113. IET, 1993.","title":"Novel approach to nonlinear/nongaussian bayesian state estimation","authors":"David J Neil J Gordon, Adrian Fm Salmond, Smith","year":"1993","venue":"IEE Proceedings F-radar and signal processing","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Karl et al., 2016] Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick Van der Smagt. Deep variational bayes filters: Unsupervised learning of state space models from raw data. arXiv preprint arXiv:1605.06432, 2016.","title":"Deep variational bayes filters: Unsupervised learning of state space models from raw data","authors":"Maximilian Karl, Maximilian Soelch, Justin Bayer, Patrick Van Der Smagt","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kitagawa, 1996] Genshiro Kitagawa. Monte carlo filter and smoother for non-gaussian nonlinear state space models. Journal of computational and graphical statistics, 5(1):1- 25, 1996.","title":"Genshiro Kitagawa. Monte carlo filter and smoother for non-gaussian nonlinear state space models","authors":"Kitagawa","year":"1996","venue":"Journal of computational and graphical statistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krishnan et al., 2015] Rahul G Krishnan, Uri Shalit, and David Sontag. Deep kalman filters. arXiv preprint arXiv:1511.05121, 2015.","title":"Deep kalman filters","authors":"Uri Rahul G Krishnan, David Shalit, Sontag","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Krishnan et al., 2017] Rahul Krishnan, Uri Shalit, and David Sontag. Structured inference networks for nonlin- ear state space models. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 31, 2017.","title":"Structured inference networks for nonlinear state space models","authors":"Rahul Krishnan, Uri Shalit, David Sontag","year":"2017","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kumar et al., 2016] Ankit Kumar, Ozan Irsoy, Peter On- druska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. Ask me anything: Dynamic memory networks for natural lan- guage processing. In International conference on machine learning, pages 1378-1387. PMLR, 2016.","title":"Ask me anything: Dynamic memory networks for natural language processing","authors":"Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, Richard Socher","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Le et al., 2017] Tuan Anh Le, Maximilian Igl, Tom Rain- forth, Tom Jin, and Frank Wood. Auto-encoding sequen- tial monte carlo. arXiv preprint arXiv:1705.10306, 2017.","title":"Auto-encoding sequential monte carlo","authors":"Anh Tuan, Maximilian Le, Tom Igl, Tom Rainforth, Frank Jin, Wood","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lütkepohl, 2005] Helmut Lütkepohl. New introduction to multiple time series analysis. Springer Science & Business Media, 2005.","title":"New introduction to multiple time series analysis","authors":"Helmut Lütkepohl","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Maddison et al., 2017] Chris J Maddison, John Lawson, George Tucker, Nicolas Heess, Mohammad Norouzi, An- driy Mnih, Arnaud Doucet, and Yee Teh. Filtering vari- ational objectives. Advances in Neural Information Pro- cessing Systems, 30, 2017.","title":"Filtering variational objectives","authors":"Chris J Maddison, John Lawson, George Tucker, Nicolas Heess, Mohammad Norouzi, Andriy Mnih, Arnaud Doucet, Yee Teh","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Naesseth et al., 2018] Christian Naesseth, Scott Linderman, Rajesh Ranganath, and David Blei. Variational sequential monte carlo. In International conference on artificial in- telligence and statistics, pages 968-977. PMLR, 2018.","title":"Variational sequential monte carlo","authors":"Naesseth","year":"2018","venue":"International conference on artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rasul et al., 2020] Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs Bergmann, and Roland Voll- graf. Multivariate probabilistic time series forecast- ing via conditioned normalizing flows. arXiv preprint arXiv:2002.06103, 2020.","title":"Multivariate probabilistic time series forecasting via conditioned normalizing flows","authors":"Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs Bergmann, Roland Vollgraf","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rasul et al., 2021] Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. arXiv preprint arXiv:2101.12072, 2021.","title":"Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting","authors":"Kashif Rasul, Calvin Seward, Ingmar Schuster, Roland Vollgraf","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Salinas et al., 2019] David Salinas, Michael Bohlke- Schneider, Laurent Callot, Roberto Medico, and Jan Gasthaus. High-dimensional multivariate forecasting with low-rank gaussian copula processes. arXiv preprint arXiv:1910.03002, 2019.","title":"High-dimensional multivariate forecasting with low-rank gaussian copula processes","authors":"David Salinas, Michael Bohlke-Schneider, Laurent Callot, Roberto Medico, Jan Gasthaus","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sukhbaatar et al., 2015] Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. arXiv preprint arXiv:1503.08895, 2015.","title":"End-to-end memory networks","authors":"Sukhbaatar","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wen et al., 2022] Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: A survey. arXiv preprint arXiv:2202.07125, 2022.","title":"Transformers in time series: A survey","authors":"Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, Liang Sun","year":"2022","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2023/759"},{"position":27,"raw":"[Weston et al., 2015] Jason Weston, Sumit Chopra, and An- toine Bordes. Memory networks. International Confer- ence on Learning Representations (ICLR), 2015.","title":"Antoine Bordes","authors":"Jason Weston, Sumit Chopra","year":"2015","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}