{"type":"ScholarlyArticle","title":"Symplectic Neural Gaussian Processes for Meta-learning Hamiltonian Dynamics","authors":[{"name":"Tomoharu Iwata","slug":"tomoharu-iwata","given":"Tomoharu","family":"Iwata","affiliation":"NTT Corporation"},{"name":"Yusuke Tanaka","slug":"yusuke-tanaka","given":"Yusuke","family":"Tanaka","affiliation":"NTT Corporation"}],"abstract":"We propose a meta-learning method for modeling Hamiltonian dynamics from a limited number of data. Although Hamiltonian neural networks have been successfully used for modeling dynamics that obey the energy conservation law, they require many data to achieve high performance. The proposed method meta-learns our neural network-based model using datasets in various dynamical systems, such that our model can predict vector fields of unseen systems. In our model, a system representation is inferred from given small data using an encoder network. Then, the system-specific vector field is predicted by modeling the Hamiltonian using a Gaussian process (GP) with neural network-based mean and kernel functions that depend on the inferred system representation. This GP-based Hamiltonian allows us to analytically obtain predictions that are adapted to small data while imposing the constraint of the conservation law. The neural networks are shared across systems, which enables us to learn knowledge from multiple systems, and use it for unseen systems. In our experiments, we demonstrate that the proposed method outperforms existing methods for predicting dynamics from a small number of observations in target systems.","doi":"10.24963/ijcai.2024/465","url":"https://ijcai.org/proceedings/2024/465","pdf":"https://ijcai.org/proceedings/2024/0465.pdf","markdown":"https://ijcai.org/proceedings/2024/465.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/465","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":465,"pages":{"first":4210,"last":4218},"keywords":["Machine Learning: ML: Meta-learning","Multidisciplinary Topics and Applications: MTA: Physical sciences"],"videos":[],"references":[{"position":1,"raw":"[Baxter, 2000] Jonathan Baxter. A model of inductive bias learning. Journal of Artificial Intelligence Research, 12:149-198, 2000.","title":"A model of inductive bias learning","authors":"Jonathan Baxter","year":"2000","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bertinetto et al., 2018] Luca Bertinetto, Joao F Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with dif- ferentiable closed-form solvers. In International Confer- ence on Learning Representations, 2018.","title":"Meta-learning with differentiable closed-form solvers","authors":"Luca Bertinetto, Joao F Henriques, Philip Torr, Andrea Vedaldi","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Braun and Kivshar, 1998] Oleg M Braun and Yuri S Kivshar. Nonlinear dynamics of the Frenkel-Kontorova model. Physics Reports, 306(1-2):1-108, 1998.","title":"Nonlinear dynamics of the Frenkel-Kontorova model","authors":"M Oleg, Yuri S Braun, Kivshar","year":"1998","venue":"Physics Reports","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chang and Zeng, 2023] Cheng Chang and Tieyong Zeng. A hybrid data-driven-physics-constrained Gaussian pro- cess regression framework with deep kernel for uncer- tainty quantification. Journal of Computational Physics, 486:112129, 2023.","title":"A hybrid data-driven-physics-constrained Gaussian process regression framework with deep kernel for uncertainty quantification","authors":"Cheng Chang, Tieyong Zeng","year":"2023","venue":"Journal of Computational Physics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chang et al., 2023] Cheng Chang, Tieyong Zeng, and Zhouping Xin. A conservative hybrid physics-informed neural network method for ampère-Nernst-Planck equa- tions. SSRN 4435273, 2023.","title":"A conservative hybrid physics-informed neural network method for ampère-Nernst-Planck equations","authors":"Cheng Chang, Tieyong Zeng, Zhouping Xin","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Che et al., 2018] Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific Reports, 8(1):6085, 2018.","title":"Recurrent neural networks for multivariate time series with missing values","authors":"Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, Yan Liu","year":"2018","venue":"Scientific Reports","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2018] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary dif- ferential equations. Advances in Neural Information Pro- cessing Systems, 31, 2018.","title":"Neural ordinary differential equations","authors":"Yulia Ricky Tq Chen, Jesse Rubanova, David K Bettencourt, Duvenaud","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2020] Zhengdao Chen, Jianyu Zhang, Martin Arjovsky, and Léon Bottou. Symplectic recurrent neural networks. In International Conference on Learning Rep- resentations, 2020.","title":"Symplectic recurrent neural networks","authors":"Chen","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2021] Yuhan Chen, Takashi Matsubara, and Takaharu Yaguchi. Neural symplectic form: learn- ing Hamiltonian equations on general coordinate sys- tems. Advances in Neural Information Processing Sys- tems, 34:16659-16670, 2021.","title":"Neural symplectic form: learning Hamiltonian equations on general coordinate systems","authors":"Yuhan Chen, Takashi Matsubara, Takaharu Yaguchi","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen et al., 2022a] Yuyan Chen, Bin Dong, and Jinchao Xu. Meta-MgNet: Meta multigrid networks for solving parameterized partial differential equations. Journal of computational physics, 455:110996, 2022.","title":"Meta-MgNet: Meta multigrid networks for solving parameterized partial differential equations","authors":"Chen","year":"2022","venue":"Journal of computational physics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Chen et al., 2022b] Zhijie Chen, Mingquan Feng, Junchi Yan, and Hongyuan Zha. Learning neural Hamiltonian dynamics: a methodological overview. arXiv preprint arXiv:2203.00128, 2022.","title":"Learning neural Hamiltonian dynamics: a methodological overview","authors":"Chen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Daniels and Nemenman, 2015] Bryan C Daniels and Ilya Nemenman. Efficient inference of parsimonious phe- nomenological models of cellular dynamics using S- systems and alternating regression. PLOS ONE, 10(3):e0119821, 2015.","title":"Efficient inference of parsimonious phenomenological models of cellular dynamics using Ssystems and alternating regression","authors":"C Bryan, Ilya Daniels, Nemenman","year":"2015","venue":"PLOS ONE","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Dormand and Prince, 1980] John R Dormand and Peter J Prince. A family of embedded runge-kutta formulae. Jour- nal of Computational and Applied Mathematics, 6(1):19- 26, 1980.","title":"A family of embedded runge-kutta formulae","authors":"R John, Peter J Dormand, Prince","year":"1980","venue":"Journal of Computational and Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In Proceedings of the 34th Inter- national Conference on Machine Learning, pages 1126- 1135, 2017.","title":"Model-agnostic meta-learning for fast adaptation of deep networks","authors":"Finn","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Finzi et al., 2020] Marc Finzi, Ke Alexander Wang, and Andrew G Wilson. Simplifying Hamiltonian and Lagrangian neural networks via explicit constraints. Advances in Neural Information Processing Systems, 33:13880-13889, 2020.","title":"Simplifying Hamiltonian and Lagrangian neural networks via explicit constraints","authors":"Marc Finzi, Ke, Alexander Wang, Andrew G Wilson","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Fortuin et al., 2019] Vincent Fortuin, Heiko Strathmann, and Gunnar Rätsch. Meta-learning mean functions for Gaussian processes. arXiv preprint arXiv:1901.08098, 2019.","title":"Meta-learning mean functions for Gaussian processes","authors":"Heiko Vincent Fortuin, Gunnar Strathmann, Rätsch","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Garnelo et al., 2018] Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In Inter- national Conference on Machine Learning, pages 1690- 1699, 2018.","title":"Conditional neural processes","authors":"Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, Eslami Ali","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Graves and Graves, 2012] Alex Graves and Alex Graves. Long short-term memory. Supervised sequence labelling with recurrent neural networks, pages 37-45, 2012.","title":"Long short-term memory. Supervised sequence labelling with recurrent neural networks","authors":"Alex Graves, Alex Graves","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Greydanus et al., 2019] Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advances in Neural Information Processing Systems, 32, 2019.","title":"Hamiltonian neural networks","authors":"Greydanus","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Haussler, 1988] David Haussler. Quantifying inductive bias: AI learning algorithms and valiant's learning frame- work. Artificial Intelligence, 36(2):177-221, 1988.","title":"Quantifying inductive bias: AI learning algorithms and valiant's learning framework","authors":"David Haussler","year":"1988","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[He and Zou, 2021] Horace He and Richard Zou. functorch: JAX-like composable function transforms for PyTorch. https://github.com/pytorch/functorch, 2021.","title":"Horace He and Richard Zou. functorch: JAX-like composable function transforms for PyTorch","authors":"Zou He","year":"2021","venue":null,"doi":null,"url":"https://github.com/pytorch/functorch,2021","internal_url":null},{"position":22,"raw":"[Iwata and Tanaka, 2022] Tomoharu Iwata and Yusuke Tanaka. Few-shot learning for spatial regression via neural embedding-based Gaussian processes. Machine Learning, pages 1-19, 2022.","title":"Few-shot learning for spatial regression via neural embedding-based Gaussian processes","authors":"Tanaka ; ] Tomoharu Iwata, Yusuke Iwata, Tanaka","year":"2022","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Lee et al., 2021] Seungjun Lee, Haesang Yang, and Woojae Seong. Identifying physical law of Hamiltonian systems via meta-learning. In International Conference on Learn- ing Representations, 2021.","title":"Identifying physical law of Hamiltonian systems via meta-learning","authors":"Lee","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Leonessa et al., 2001] Alexander Leonessa, Wassim M Haddad, and VijaySekhar Chellaboina. Nonlinear sys- tem stabilization via hierarchical switching control. IEEE Transactions on Automatic Control, 46(1):17-28, 2001.","title":"Nonlinear system stabilization via hierarchical switching control","authors":"Alexander Leonessa, M Wassim, Vijaysekhar Haddad, Chellaboina","year":"2001","venue":"IEEE Transactions on Automatic Control","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Mattheakis et al., 2022] Marios Mattheakis, David Sondak, Akshunna S Dogra, and Pavlos Protopapas. Hamiltonian neural networks for solving equations of motion. Physical Review E, 105(6):065305, 2022.","title":"Hamiltonian neural networks for solving equations of motion","authors":"Marios Mattheakis, David Sondak, Akshunna S Dogra, Pavlos Protopapas","year":"2022","venue":"Physical Review E","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Paszke et al., 2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.","title":"Pytorch: An imperative style, highperformance deep learning library","authors":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Patacchiola et al., 2020] Massimiliano Patacchiola, Jack Turner, Elliot J Crowley, Michael O'Boyle, and Amos J Storkey. Bayesian meta-learning for the few-shot setting via deep kernels. Advances in Neural Information Pro- cessing Systems, 33:16108-16118, 2020.","title":"Bayesian meta-learning for the few-shot setting via deep kernels","authors":"Massimiliano Patacchiola, Jack Turner, Elliot J Crowley, O' Michael, Amos J Boyle, Storkey","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Psaros et al., 2022] Apostolos F Psaros, Kenji Kawaguchi, and George Em Karniadakis. Meta-learning PINN loss functions. Journal of Computational Physics, 458:111121, 2022.","title":"Meta-learning PINN loss functions","authors":"Apostolos F Psaros, Kenji Kawaguchi, George Em Karniadakis","year":"2022","venue":"Journal of Computational Physics","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Raghu et al., 2020] Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or fea- ture reuse? Towards understanding the effectiveness of MAML. In International Conference on Learning Rep- resentations, 2020.","title":"Rapid learning or feature reuse? Towards understanding the effectiveness of MAML","authors":"Aniruddh Raghu, Maithra Raghu, Samy Bengio, Oriol Vinyals","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Rath et al., 2021] Katharina Rath, Christopher G Albert, Bernd Bischl, and Udo von Toussaint. Symplectic Gaus- sian process regression of maps in Hamiltonian systems. Chaos: An Interdisciplinary Journal of Nonlinear Science, 31(5):053121, 2021.","title":"Symplectic Gaussian process regression of maps in Hamiltonian systems","authors":"Katharina Rath, G Christopher, Bernd Albert, Bischl, Udo Von Toussaint","year":"2021","venue":"Chaos: An Interdisciplinary Journal of Nonlinear Science","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Ravi and Larochelle, 2017] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International Conference on Learning Representations, 2017.","title":"Optimization as a model for few-shot learning","authors":"Larochelle ; Ravi, Sachin Ravi, Hugo Larochelle","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Rothfuss et al., 2021] Jonas Rothfuss, Vincent Fortuin, Martin Josifoski, and Andreas Krause. PACOH: Bayes- optimal meta-learning with PAC-guarantees. In Inter- national Conference on Machine Learning, pages 9116- 9126, 2021.","title":"PACOH: Bayesoptimal meta-learning with PAC-guarantees","authors":"Rothfuss","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Snell et al., 2017] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Ad- vances in Neural Information Processing Systems, 30, 2017.","title":"Prototypical networks for few-shot learning","authors":"Jake Snell, Kevin Swersky, Richard Zemel","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Tanaka et al., 2022] Yusuke Tanaka, Tomoharu Iwata, and Naonori Ueda. Symplectic spectrum Gaussian processes: Learning Hamiltonians from noisy and sparse data. In Ad- vances in Neural Information Processing Systems, 2022.","title":"Symplectic spectrum Gaussian processes: Learning Hamiltonians from noisy and sparse data","authors":"Tanaka","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Thangamuthu et al., 2022] Abishek Thangamuthu, Gunjan Kumar, Suresh Bishnoi, Ravinder Bhattoo, NM Krish- nan, and Sayan Ranu. Unravelling the performance of physics-informed graph neural networks for dynamical systems. Advances in Neural Information Processing Sys- tems, 35:3691-3702, 2022.","title":"Unravelling the performance of physics-informed graph neural networks for dynamical systems","authors":"Abishek Thangamuthu, Gunjan Kumar, Suresh Bishnoi, Ravinder Bhattoo, Sayan Krishnan, Ranu","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Tong et al., 2021] Yunjin Tong, Shiying Xiong, Xingzhe He, Guanghan Pan, and Bo Zhu. Symplectic neural net- works in Taylor series form for Hamiltonian systems. Journal of Computational Physics, 437:110325, 2021.","title":"Symplectic neural networks in Taylor series form for Hamiltonian systems","authors":"Yunjin Tong, Shiying Xiong, Xingzhe He, Guanghan Pan, Bo Zhu","year":"2021","venue":"Journal of Computational Physics","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Toth et al., 2020] Peter Toth, Danilo J Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, and Irina Higgins. Hamiltonian generative networks. In Interna- tional Conference on Learning Representations, 2020.","title":"Hamiltonian generative networks","authors":"Peter Toth, Danilo J Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, Irina Higgins","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Vinyals et al., 2016] Oriol Vinyals, Charles Blundell, Tim- othy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, pages 3630-3638, 2016.","title":"Matching networks for one shot learning","authors":"Vinyals","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Vovk, 2013] Vladimir Vovk. Kernel ridge regression. Em- pirical Inference: Festschrift in Honor of Vladimir N. Vap- nik, pages 105-116, 2013.","title":"Kernel ridge regression","authors":"Vladimir Vovk","year":"2013","venue":"Empirical Inference: Festschrift in Honor of Vladimir N. Vapnik","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Wilson et al., 2016] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learn- ing. In Artificial intelligence and statistics, pages 370- 378, 2016.","title":"Deep kernel learning","authors":"Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, Eric P Xing","year":"2016","venue":"Artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Zhong et al., 2020] Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ODE-net: Learn- ing Hamiltonian dynamics with control. In International Conference on Learning Representations, 2020.","title":"Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ODE-net: Learning Hamiltonian dynamics with control","authors":"Zhong","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}