{"type":"ScholarlyArticle","title":"A Functional Dynamic Boltzmann Machine","authors":[{"name":"Hiroshi Kajino","slug":"hiroshi-kajino","given":"Hiroshi","family":"Kajino","affiliation":"IBM Research -Tokyo"}],"abstract":"Dynamic Boltzmann machines (DyBMs) are recently developed generative models of a time series. They are designed to learn a time series by efficient online learning algorithms, whilst taking long-term dependencies into account with help of eligibility traces, recursively updatable memory units storing descriptive statistics of all the past data. The current DyBMs assume a finite-dimensional time series and cannot be applied to a functional time series, in which the dimension goes to infinity (e.g., spatiotemporal data on a continuous space). In this paper, we present a functional dynamic Boltzmann machine (F-DyBM) as a generative model of a functional time series. A technical challenge is to devise an online learning algorithm with which F-DyBM, consisting of functions and integrals, can learn a functional time series using only finite observations of it. We rise to the above challenge by combining a kernel-based function approximation method along with a statistical interpolation method and finally derive closed-form update rules. We design numerical experiments to empirically confirm the effectiveness of our solutions. The experimental results demonstrate consistent error reductions as compared to baseline methods, from which we conclude the effectiveness of F-DyBM for functional time series prediction.","doi":"10.24963/ijcai.2017/276","url":"https://ijcai.org/proceedings/2017/276","pdf":"https://ijcai.org/proceedings/2017/0276.pdf","markdown":"https://ijcai.org/proceedings/2017/276.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/276","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":276,"pages":{"first":1987,"last":1993},"keywords":["Machine Learning: Online Learning","Machine Learning: Time-series/Data Streams"],"videos":[],"references":[{"position":1,"raw":"[Anava et al., 2015] Oren Anava, Elad Hazan, and Assaf Zeevi. Online time series prediction with missing data. In Proceedings of the 32nd International Conference on Machine Learning, pages 2191-2199, 2015.","title":"Online time series prediction with missing data","authors":"Oren Anava, Elad Hazan, Assaf Zeevi","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Besse et al., 2000] Philippe C. Besse, Hervé Cardot, and David B. Stephenson. Autoregressive forecasting of some functional climatic variations. Scandinavian Journal of Statistics, 27:673-687, 2000.","title":"Autoregressive forecasting of some functional climatic variations","authors":"Philippe C Besse, Hervé Cardot, David B Stephenson","year":"2000","venue":"Scandinavian Journal of Statistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bosq, 2000] Denis Bosq. Linear Processes in Function Spaces: Theory and Applications, chapter 5. Springer New York, 2000.","title":"Linear Processes in Function Spaces: Theory and Applications","authors":"Denis Bosq","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Damon and Guillas, 2002] Julien Damon and Serge Guillas. The inclusion of exogenous variables in functional autore- gressive ozone forecasting. Environmetrics, 13:759-774, 2002.","title":"The inclusion of exogenous variables in functional autoregressive ozone forecasting","authors":"Damon, Guillas ;, Julien Damon, Serge Guillas","year":"2002","venue":"Environmetrics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dasgupta and Osogami, 2017] Sakyasingha Dasgupta and Takayuki Osogami. Nonlinear dynamic Boltzmann ma- chines for time-series prediction. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 2017.","title":"Sakyasingha Dasgupta and Takayuki Osogami. Nonlinear dynamic Boltzmann machines for time-series prediction","authors":"Osogami Dasgupta","year":"2017","venue":"Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Didericksen et al., 2012] Devin Didericksen, Piotr Kokoszka, and Xi Zhang. Empirical properties of forecasts with the functional autoregressive model. Computational Statistics, 27(2):285-298, 2012.","title":"Empirical properties of forecasts with the functional autoregressive model","authors":"Devin Didericksen, Piotr Kokoszka, Xi Zhang","year":"2012","venue":"Computational Statistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Elman, 1990] Jeffrey L. Elman. Finding structure in time. Cognitive Science, 14:179-211, 1990.","title":"Finding structure in time","authors":"Jeffrey L Elman","year":"1990","venue":"Cognitive Science","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hyndman and Ullah, 2007] Rob J. Hyndman and Md Shahid Ullah. Robust forecasting of mortality and fertility rates: a functional data approach. Compu- tational Statistics & Data Analysis, 51(10):4942-4956, 2007.","title":"Robust forecasting of mortality and fertility rates: a functional data approach","authors":"Rob J Hyndman, Md Shahid, Ullah","year":"2007","venue":"Computational Statistics & Data Analysis","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jaeger and Haas, 2004] Herbert Jaeger and Harald Haas. Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science, 304(5667):78-80, 2004.","title":"Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication","authors":"Haas Jaeger, Herbert Jaeger, Harald Haas","year":"2004","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kargin and Onatski, 2008] Vladislav Kargin and Alexei Onatski. Curve forecasting by functional autoregres- sion. Journal of Multivariate Analysis, 99(10):2508-2526, 2008.","title":"Curve forecasting by functional autoregression","authors":"Vladislav Kargin, Alexei Onatski","year":"2008","venue":"Journal of Multivariate Analysis","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lütkepohl, 2005] Helmut Lütkepohl. New Introduction to Multiple Time Series Analysis. Part I. Springer Berlin Hei- delberg, 2005.","title":"New Introduction to Multiple Time Series Analysis. Part I","authors":"Helmut Lütkepohl","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Osogami and Otsuka, 2015] Takayuki Osogami and Makoto Otsuka. Seven neurons memorizing sequences of alphabetical images via spike-timing dependent plasticity. Scientific Reports, 5:14149 EP -, 09 2015.","title":"Seven neurons memorizing sequences of alphabetical images via spike-timing dependent plasticity","authors":"Takayuki Osogami, Makoto Otsuka","year":"2015","venue":"Scientific Reports","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Osogami, 2016] Takayuki Osogami. Learning binary or real-valued time-series via spike-timing dependent plastic- ity. In The First NIPS Workshop Computing with Spikes, 2016.","title":"Learning binary or real-valued time-series via spike-timing dependent plasticity","authors":"Takayuki Osogami, Osogami","year":"2016","venue":"The First NIPS Workshop Computing with Spikes","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Quiñonero-Candela and Rasmussen, 2005] Joaquin Quiñonero-Candela and Carl Edward Rasmussen. A unifying view of sparse approximate Gaussian process regression. Journal of Machine Learning Research, 6:1939-1959, 2005.","title":"Joaquin Quiñonero-Candela and Carl Edward Rasmussen. A unifying view of sparse approximate Gaussian process regression","authors":"Quiñonero-Candela, Rasmussen","year":"2005","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ramsay and Silverman, 2005] James Ramsay and Bernard W. Silverman. Functional Data Analysis, chapter 8. Springer-Verlag New York, 2005.","title":"Functional Data Analysis","authors":"Ramsay, Silverman, James Ramsay, Bernard W Silverman","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rasmussen and Williams, 2006] Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning, chapter 8. MIT Press, 2006.","title":"Gaussian Processes for Machine Learning","authors":"Carl Edward Rasmussen, Christopher K I Williams","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rumelhart et al., 1986] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning internal rep- resentations by error propagation, chapter 8, pages 318- 362. MIT Press, 1986.","title":"Learning internal representations by error propagation","authors":"David E Rumelhart, Geoffrey E Hinton, Ronald J Williams","year":"1986","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shumway and Stoffer, 1982] Robert H. Shumway and David S. Stoffer. An approach to time series smoothing and forecasting using the EM algorithm. Journal of Time Series Analysis, 3(4):253-264, 1982.","title":"An approach to time series smoothing and forecasting using the EM algorithm","authors":"Robert H Shumway, David S Stoffer","year":"1982","venue":"Journal of Time Series Analysis","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sundermeyer et al., 2012] Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. LSTM neural networks for language modeling. In Interspeech, pages 194-197, 2012.","title":"LSTM neural networks for language modeling","authors":"Martin Sundermeyer, Ralf Schlüter, Hermann Ney","year":"2012","venue":"Interspeech","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tieleman and Hinton, 2012] Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a run- ning average of its recent magnitude, coursera: Neural net- works for machine learning, 2012.","title":"Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude, coursera: Neural networks for machine learning","authors":"Hinton Tieleman","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wilson et al., 2015] Andrew Gordon Wilson, Christopher Dann, and Hannes Nickisch. Thoughts on massively scalable Gaussian processes. Technical report, eprint arXiv:1511.01870, 2015.","title":"Thoughts on massively scalable Gaussian processes","authors":"Andrew Gordon, Wilson, Christopher Dann, Hannes Nickisch","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}