{"type":"ScholarlyArticle","title":"ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs","authors":[{"name":"Amir Gholaminejad","slug":"amir-gholaminejad","given":"Amir","family":"Gholaminejad","affiliation":"Berkeley Artificial Intelligence Research Lab , EECS , UC Berkeley"},{"name":"Kurt Keutzer","slug":"kurt-keutzer","given":"Kurt","family":"Keutzer","affiliation":"Berkeley Artificial Intelligence Research Lab , EECS , UC Berkeley"},{"name":"George Biros","slug":"george-biros","given":"George","family":"Biros","affiliation":"Oden Institute for Computational Engineering and Sciences , UT Austin"}],"abstract":"Residual neural networks can be viewed as the forward Euler discretization of an Ordinary Differential Equation (ODE) with a unit time step.  This has recently motivated researchers to explore other discretization approaches and train ODE based networks. However, an important challenge of neural ODEs is their prohibitive memory cost during gradient backpropogation. Recently a method proposed in arXiv:1806.07366, claimed that this memory overhead can be reduced from LNt, where Nt is the number of time steps, down to O(L) by solving forward ODE backwards in time, where L is the depth of the network. However, we will show that this approach may lead to several problems: (i) it may be numerically unstable for ReLU/non-ReLU activations and general convolution operators, and (ii) the proposed optimize-then-discretize approach may lead to divergent training due to inconsistent gradients for small time step sizes. We discuss the underlying problems, and to address them we propose ANODE, a neural ODE framework which avoids the numerical instability related problems noted above. ANODE has a memory footprint of O(L) + O(Nt), with the same computational cost as reversing ODE solve. We furthermore, discuss a memory efficient algorithm which can further reduce this footprint with a tradeoff of additional computational cost. We show results on Cifar-10/100 datasets using ResNet and SqueezeNext neural networks.","doi":"10.24963/ijcai.2019/103","url":"https://ijcai.org/proceedings/2019/103","pdf":"https://ijcai.org/proceedings/2019/0103.pdf","markdown":"https://ijcai.org/proceedings/2019/103.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/103","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":103,"pages":{"first":730,"last":736},"keywords":["Computer Vision: Statistical Methods and Machine Learning","Computer Vision: Computer Vision","Machine Learning: Explainable Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Abraham et al., 2007] Ralph Abraham, Jerrold E Marsden, and Tudor Ratiu. Manifolds, tensor analysis, and appli- cations, volume 75. Springer Science & Business Media, 2007.","title":"Manifolds, tensor analysis, and applications","authors":"Abraham","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Biros and Ghattas, 2005a] George Biros and Omar Ghattas. Parallel lagrange-newton-krylov-schur methods for pde- constrained optimization. part i: The krylov-schur solver. SIAM Journal on Scientific Computing, 27(2):687-713, 2005.","title":"Parallel lagrange-newton-krylov-schur methods for pdeconstrained optimization. part i: The krylov-schur solver","authors":"Ghattas Biros, George Biros, Omar Ghattas","year":"2005","venue":"SIAM Journal on Scientific Computing","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Biros and Ghattas, 2005b] George Biros and Omar Ghattas. Parallel lagrange-newton-krylov-schur methods for pde- constrained optimization. part ii: The lagrange-newton solver and its application to optimal control of steady viscous flows. SIAM Journal on Scientific Computing, 27(2):714-739, 2005.","title":"Parallel lagrange-newton-krylov-schur methods for pdeconstrained optimization. part ii: The lagrange-newton solver and its application to optimal control of steady viscous flows","authors":"Ghattas Biros, George Biros, Omar Ghattas","year":"2005","venue":"SIAM Journal on Scientific Computing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Calcaterra and Boldt, 2008] Craig Calcaterra and Axel Boldt. Journal of Mathematical Analysis and Applica- tions, 338(2):1108 -1115, 2008.","title":"Journal of Mathematical Analysis and Applications","authors":"Craig Calcaterra, Axel Boldt","year":"2008","venue":"Journal of Mathematical Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chang et al., 2018] Bo Chang, Lili Meng, Eldad Haber, Lars Ruthotto, David Begert, and Elliot Holtham. Re- versible architectures for arbitrarily deep residual neural networks. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.","title":"Reversible architectures for arbitrarily deep residual neural networks","authors":"Bo Chang, Lili Meng, Eldad Haber, Lars Ruthotto, David Begert, Elliot Holtham","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Charpentier and Ghemires, 2000] Isabelle Charpentier and Mohammed Ghemires. Efficient adjoint derivatives: appli- cation to the meteorological model meso-nh. Optimization Methods and Software, 13(1):35-63, 2000.","title":"Isabelle Charpentier and Mohammed Ghemires. Efficient adjoint derivatives: application to the meteorological model meso-nh","authors":"Ghemires Charpentier","year":"2000","venue":"Optimization Methods and Software","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen and Hsieh, 2018] Patrick H Chen and Cho-jui Hsieh. A comparison of second-order methods for deep convo- lutional neural networks. openreview under ICLR 2018, 2018.","title":"A comparison of second-order methods for deep convolutional neural networks","authors":"H Patrick, Cho-Jui Chen, Hsieh","year":"2018","venue":"openreview under ICLR 2018","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2016] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016.","title":"Training deep nets with sublinear memory cost","authors":"Tianqi Chen, Bing Xu, Chiyuan Zhang, Carlos Guestrin","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2018] Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differ- ential equations. arXiv preprint arXiv:1806.07366, 2018.","title":"Neural ordinary differential equations","authors":"Tian Qi, Chen, Yulia Rubanova, Jesse Bettencourt, David Duvenaud","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ciccone et al., 2018] Marco Ciccone, Marco Gallieri, Jonathan Masci, Christian Osendorfer, and Faustino Gomez. Nais-net: Stable deep networks from non- autonomous differential equations. arXiv preprint arXiv:1804.07209, 2018.","title":"Nais-net: Stable deep networks from nonautonomous differential equations","authors":"Marco Ciccone, Marco Gallieri, Jonathan Masci, Christian Osendorfer, Faustino Gomez","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dinh et al., 2014] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent compo- nents estimation. arXiv preprint arXiv:1410.8516, 2014.","title":"Nice: Non-linear independent components estimation","authors":"Laurent Dinh, David Krueger, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Dinh et al., 2016] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803, 2016.","title":"Density estimation using real nvp","authors":"Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Fu et al., 2007] Chu-Li Fu, Xiang-Tuan Xiong, and Zhi Qian. Fourier regularization for a backward heat equa- tion. Journal of Mathematical Analysis and Applications, 331(1):472-480, 2007.","title":"Fourier regularization for a backward heat equation","authors":"Chu-Li Fu, Xiang-Tuan Xiong, Zhi Qian","year":"2007","venue":"Journal of Mathematical Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gholami, 2017] A. Gholami. Fast Algorithms for Biophysically-Constrained Inverse Problems in Med- ical imaging. PhD thesis, The University of Texas at Austin, 2017.","title":"Fast Algorithms for Biophysically-Constrained Inverse Problems in Medical imaging","authors":"A Gholami","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gomez et al., 2017] Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual net- work: Backpropagation without storing activations. In Ad- vances in Neural Information Processing Systems, pages 2214-2224, 2017.","title":"The reversible residual network: Backpropagation without storing activations","authors":"Aidan N Gomez, Mengye Ren, Raquel Urtasun, Roger B Grosse","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Griewank and Walther, 2000] Andreas Griewank and An- drea Walther. Algorithm 799: revolve: an implementation of checkpointing for the reverse or adjoint mode of com- putational differentiation. ACM Transactions on Mathe- matical Software (TOMS), 26(1):19-45, 2000.","title":"Andreas Griewank and Andrea Walther. Algorithm 799: revolve: an implementation of checkpointing for the reverse or adjoint mode of computational differentiation","authors":"Walther Griewank","year":"2000","venue":"ACM Transactions on Mathematical Software (TOMS)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Griewank, 1992] Andreas Griewank. Achieving logarith- mic growth of temporal and spatial complexity in reverse automatic differentiation. Optimization Methods and soft- ware, 1(1):35-54, 1992.","title":"Achieving logarithmic growth of temporal and spatial complexity in reverse automatic differentiation","authors":"Andreas Griewank","year":"1992","venue":"Optimization Methods and software","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Gruslys et al., 2016] Audrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory- efficient backpropagation through time. In Advances in Neural Information Processing Systems, pages 4125- 4133, 2016.","title":"Memoryefficient backpropagation through time","authors":"Gruslys","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Haber and Ruthotto, 2017] Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34(1):014004, 2017.","title":"Stable architectures for deep neural networks","authors":"Eldad Haber, Lars Ruthotto","year":"2017","venue":"Inverse Problems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lafore et al., 1997] Jean Philippe Lafore, Joël Stein, Nicole Asencio, Philippe Bougeault, Véronique Ducrocq, Jacque- line Duron, Claude Fischer, Philippe Héreil, Patrick Mas- cart, Valéry Masson, et al. The meso-nh atmospheric sim- ulation system. part i: Adiabatic formulation and control simulations. In Annales Geophysicae, volume 16, pages 90-109. Springer, 1997.","title":"The meso-nh atmospheric simulation system. part i: Adiabatic formulation and control simulations","authors":"Jean Philippe Lafore, Joël Stein, Nicole Asencio, Philippe Bougeault, Véronique Ducrocq, Jacqueline Duron, Claude Fischer, Philippe Héreil, Patrick Mascart, Valéry Masson","year":"1997","venue":"Annales Geophysicae","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lions, 1971] Jacques Louis Lions. Optimal control of systems governed by partial differential equations (Grundlehren der Mathematischen Wissenschaften), vol- ume 170. Springer Berlin, 1971.","title":"Optimal control of systems governed by partial differential equations","authors":"Jacques Louis, Lions","year":"1971","venue":"Grundlehren der Mathematischen Wissenschaften","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lu et al., 2017] Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridg- ing deep architectures and numerical differential equa- tions. arXiv preprint arXiv:1710.10121, 2017.","title":"Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations","authors":"Yiping Lu, Aoxiao Zhong, Quanzheng Li, Bin Dong","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Martens and Sutskever, 2012] James Martens and Ilya Sutskever. Training deep and recurrent networks with hessian-free optimization. In Neural networks: Tricks of the trade, pages 479-535. Springer, 2012.","title":"Training deep and recurrent networks with hessian-free optimization","authors":"Sutskever Martens, James Martens, Ilya Sutskever","year":"2012","venue":"Neural networks: Tricks of the trade","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ruthotto and Haber, 2018] Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. arXiv preprint arXiv:1804.04272, 2018.","title":"Deep neural networks motivated by partial differential equations","authors":"Haber Ruthotto, Lars Ruthotto, Eldad Haber","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shen, 2001] Jianhong Shen. On the singular values of gaus- sian random matrices. Linear Algebra and its Applica- tions, 326(1-3):1-14, 2001.","title":"On the singular values of gaussian random matrices","authors":"Jianhong Shen","year":"2001","venue":"Linear Algebra and its Applications","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Weinan, 2017] E Weinan. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1):1-11, 2017.","title":"A proposal on machine learning via dynamical systems","authors":"Weinan","year":"2017","venue":"Communications in Mathematics and Statistics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}