{"type":"ScholarlyArticle","title":"Taylor-Lagrange Neural Ordinary Differential Equations: Toward Fast Training and Evaluation of Neural ODEs","authors":[{"name":"Franck Djeumou","slug":"franck-djeumou","given":"Franck","family":"Djeumou","affiliation":"The University of Texas at Austin , United States"},{"name":"Cyrus Neary","slug":"cyrus-neary","given":"Cyrus","family":"Neary","affiliation":"The University of Texas at Austin , United States"},{"name":"Eric Goubault","slug":"eric-goubault","given":"Eric","family":"Goubault","affiliation":"LIX , CNRS ,; École Polytechnique , Institut Polytechnique de Paris , France"},{"name":"Sylvie Putot","slug":"sylvie-putot","given":"Sylvie","family":"Putot","affiliation":"LIX , CNRS ,"},{"name":"Ufuk Topcu","slug":"ufuk-topcu","given":"Ufuk","family":"Topcu","affiliation":"The University of Texas at Austin , United States"}],"abstract":"Neural ordinary differential equations (NODEs) -- parametrizations of differential equations using neural networks -- have shown tremendous promise in learning models of unknown continuous-time dynamical systems from data. However, every forward evaluation of a NODE requires numerical integration of the neural network used to capture the system dynamics, making their training prohibitively expensive. Existing works rely on off-the-shelf adaptive step-size numerical integration schemes, which often require an excessive number of evaluations of the underlying dynamics network to obtain sufficient accuracy for training. By contrast, we accelerate the evaluation and the training of NODEs by proposing a data-driven approach to their numerical integration. The proposed Taylor-Lagrange NODEs (TL-NODEs) use a fixed-order Taylor expansion for numerical integration, while also learning to estimate the expansion's approximation error. As a result, the proposed approach achieves the same accuracy as adaptive step-size schemes while employing only low-order Taylor expansions, thus greatly reducing the computational cost necessary to integrate the NODE. A suite of numerical experiments, including modeling dynamical systems, image classification, and density estimation, demonstrate that TL-NODEs can be trained more than an order of magnitude faster than state-of-the-art approaches, without any loss in performance.","doi":"10.24963/ijcai.2022/405","url":"https://ijcai.org/proceedings/2022/405","pdf":"https://ijcai.org/proceedings/2022/0405.pdf","markdown":"https://ijcai.org/proceedings/2022/405.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/405","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":405,"pages":{"first":2923,"last":2929},"keywords":["Machine Learning: Recurrent Networks","Machine Learning: Time-series; Data Streams","Machine Learning: Other"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986759","length_sec":92,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984902","length_sec":340,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bettencourt et al., 2019] J. Bettencourt, Matthew J. John- son, and D. Duvenaud. Taylor-mode automatic differen- tiation for higher-order derivatives in jax. In Workshop on Program Transformations for ML, NeurIPS, 2019.","title":"Taylor-mode automatic differentiation for higher-order derivatives in jax","authors":"J Bettencourt, Matthew J Johnson, D Duvenaud","year":"2019","venue":"Workshop on Program Transformations for ML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2018] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary dif- ferential equations. Advances in Neural Information Pro- cessing Systems, 2018.","title":"Neural ordinary differential equations","authors":"Yulia Ricky Tq Chen, Jesse Rubanova, David K Bettencourt, Duvenaud","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cranmer et al., 2020] M. Cranmer, Sam Greydanus, Stephan Hoyer, Peter W. Battaglia, David N. Spergel, and Shirley Ho. Lagrangian neural networks. ArXiv, abs/2003.04630, 2020.","title":"Lagrangian neural networks","authors":"M Cranmer, Sam Greydanus, Stephan Hoyer, Peter W Battaglia, David N Spergel, Shirley Ho","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deng, 2012] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 2012.","title":"The mnist database of handwritten digit images for machine learning research","authors":"Li Deng","year":"2012","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Djeumou et al., 2021] Franck Djeumou, Abraham P Vinod, Eric Goubault, Sylvie Putot, and Ufuk Topcu. On-the-fly control of unknown smooth systems from limited data. In 2021 American Control Conference. IEEE, 2021.","title":"On-the-fly control of unknown smooth systems from limited data","authors":"Franck Djeumou, Abraham P Vinod, Eric Goubault, Sylvie Putot, Ufuk Topcu","year":"2021","venue":"2021 American Control Conference","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Djeumou et al., 2022a] Franck Djeumou, Cyrus Neary, Eric Goubault, Sylvie Putot, and Ufuk Topcu. Neural net- works with physics-informed architectures and constraints for dynamical systems modeling. In Learning for Dynam- ics and Control. PMLR, 2022.","title":"Neural networks with physics-informed architectures and constraints for dynamical systems modeling","authors":"Djeumou","year":"2022","venue":"Learning for Dynamics and Control","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Djeumou et al., 2022b] Franck Djeumou, Cyrus Neary, Eric Goubault, Sylvie Putot, and Ufuk Topcu. Taylor-lagrange neural ordinary differential equations: Toward fast train- ing and evaluation of neural odes. ArXiv, abs/2201.05715, 2022.","title":"Taylor-lagrange neural ordinary differential equations: Toward fast training and evaluation of neural odes","authors":"Djeumou","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Finlay et al., 2020] Chris Finlay, J. Jacobsen, L. Nurbekyan, and Adam M. Oberman. How to train your neural ode: the world of jacobian and kinetic regularization. In International Conference on Machine Learning. PMLR, 2020.","title":"How to train your neural ode: the world of jacobian and kinetic regularization","authors":"Chris Finlay, J Jacobsen, L Nurbekyan, Adam M Oberman","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Finzi et al., 2020] Marc Finzi, Ke Alexander Wang, and Andrew G Wilson. Simplifying hamiltonian and la- grangian neural networks via explicit constraints. Ad- vances in Neural Information Processing Systems, 2020.","title":"Simplifying hamiltonian and lagrangian neural networks via explicit constraints","authors":"Marc Finzi, Ke, Alexander Wang, Andrew G Wilson","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ghosh et al., 2020] Arna Ghosh, Harkirat Singh Behl, Em- ilien Dupont, Philip H. S. Torr, and Vinay Namboodiri. Steer : Simple temporal regularization for neural odes. ArXiv, abs/2006.10711, 2020.","title":"Steer : Simple temporal regularization for neural odes","authors":"Arna Ghosh, Harkirat Singh Behl, Emilien Dupont, H S Philip, Vinay Torr, Namboodiri","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Grathwohl et al., 2019] Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Kristjan- son Duvenaud. Ffjord: Free-form continuous dynam- ics for scalable reversible generative models. ArXiv, abs/1810.01367, 2019.","title":"Ffjord: Free-form continuous dynamics for scalable reversible generative models","authors":"Grathwohl","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Greydanus et al., 2019] Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advances in Neural Information Processing Systems, 2019.","title":"Hamiltonian neural networks","authors":"Greydanus","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Griewank and Walther, 2008] Andreas Griewank and An- drea Walther. Evaluating derivatives: principles and tech- niques of algorithmic differentiation. SIAM, 2008.","title":"Andreas Griewank and Andrea Walther. Evaluating derivatives: principles and techniques of algorithmic differentiation","authors":"Walther Griewank","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gupta et al., 2020] Jayesh K Gupta, Kunal Menda, Zachary Manchester, and Mykel Kochenderfer. Structured mechan- ical models for robot learning and control. In Learning for Dynamics and Control. PMLR, 2020.","title":"Structured mechanical models for robot learning and control","authors":"Kunal Jayesh K Gupta, Zachary Menda, Mykel Manchester, Kochenderfer","year":"2020","venue":"Learning for Dynamics and Control","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kelly et al., 2020] Jacob Kelly, Jesse Bettencourt, Matthew J Johnson, and David K Duvenaud. Learn- ing differential equations that are easy to solve. Advances in Neural Information Processing Systems, 2020.","title":"Learning differential equations that are easy to solve","authors":"Jacob Kelly, Jesse Bettencourt, Matthew J Johnson, David K Duvenaud","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mathieu and Nickel, 2020] Emile Mathieu and Maximilian Nickel. Riemannian continuous normalizing flows. ArXiv, abs/2006.10605, 2020.","title":"Emile Mathieu and Maximilian Nickel. Riemannian continuous normalizing flows","authors":"Nickel Mathieu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Menda et al., 2019] Kunal Menda, Jayesh K Gupta, Zachary Manchester, and Mykel J Kochenderfer. Struc- tured mechanical models for efficient reinforcement learning. In Workshop on Structure and Priors in Reinforcement Learning, International Conference on Learning Representations, 2019.","title":"Structured mechanical models for efficient reinforcement learning","authors":"Kunal Menda, K Jayesh, Zachary Gupta, Manchester, Mykel, Kochenderfer","year":"2019","venue":"Workshop on Structure and Priors in Reinforcement Learning, International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pal et al., 2021] Avik Pal, Yingbo Ma, Viral Shah, and Christopher V Rackauckas. Opening the blackbox: Ac- celerating neural differential equations by regularizing in- ternal solver heuristics. In International Conference on Machine Learning. PMLR, 2021.","title":"Opening the blackbox: Accelerating neural differential equations by regularizing internal solver heuristics","authors":"Avik Pal, Yingbo Ma, Viral Shah, Christopher V Rackauckas","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Papamakarios et al., 2017] George Papamakarios, Iain Murray, and Theo Pavlakou. Masked autoregressive flow for density estimation. ArXiv, abs/1705.07057, 2017.","title":"Masked autoregressive flow for density estimation","authors":"George Papamakarios, Iain Murray, Theo Pavlakou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Poli et al., 2020] Michael Poli, Stefano Massaroli, Atsushi Yamashita, Hajime Asama, and Jinkyoo Park. Hyper- solvers: Toward fast continuous-depth models. ArXiv, abs/2007.09601, 2020.","title":"Hypersolvers: Toward fast continuous-depth models","authors":"Michael Poli, Stefano Massaroli, Atsushi Yamashita, Hajime Asama, Jinkyoo Park","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Roe et al., 2005] Byron P Roe, Hai-Jun Yang, Ji Zhu, Yong Liu, Ion Stancu, and Gordon McGregor. Boosted decision trees as an alternative to artificial neural networks for par- ticle identification. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 2005.","title":"Boosted decision trees as an alternative to artificial neural networks for particle identification","authors":"Hai-Jun Byron P Roe, Ji Yang, Yong Zhu, Ion Liu, Gordon Stancu, Mcgregor","year":"2005","venue":"Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Salman et al., 2018] Hadi Salman, Payman Yadollahpour, Tom Fletcher, and Nematollah Batmanghelich. Deep dif- feomorphic normalizing flows. ArXiv, abs/1810.03256, 2018.","title":"Deep diffeomorphic normalizing flows","authors":"Payman Hadi Salman, Tom Yadollahpour, Nematollah Fletcher, Batmanghelich","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhong et al., 2021] Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Benchmarking energy- conserving neural networks for learning dynamics from data. In Learning for Dynamics and Control. PMLR, 2021.","title":"Benchmarking energyconserving neural networks for learning dynamics from data","authors":"Zhong","year":"2021","venue":"Learning for Dynamics and Control","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}