{"type":"ScholarlyArticle","title":"Can Stationary Distributions of Scale-Invariant Neural Networks Be Described by the Thermodynamics of an Ideal Gas?","authors":[{"name":"Ildus Sadrtdinov","slug":"ildus-sadrtdinov","given":"Ildus","family":"Sadrtdinov","affiliation":"Constructor University"},{"name":"Ekaterina Lobacheva","slug":"ekaterina-lobacheva","given":"Ekaterina","family":"Lobacheva","affiliation":"Mila -Quebec AI Institute; Université de Montréal"},{"name":"Ivan Klimov","slug":"ivan-klimov","given":"Ivan","family":"Klimov","affiliation":"Constructor University"},{"name":"Mikhail Burtsev","slug":"mikhail-burtsev","given":"Mikhail","family":"Burtsev","affiliation":"London Institute for Mathematical Sciences"},{"name":"Mikhail I. Katsnelson","slug":"mikhail-i-katsnelson","given":"Mikhail I.","family":"Katsnelson","affiliation":"Constructor University; Institute for Molecules and Materials , Radboud University"},{"name":"Dmitry Vetrov","slug":"dmitry-vetrov","given":"Dmitry","family":"Vetrov","affiliation":"Constructor University"}],"abstract":"Understanding the training dynamics of deep neural networks remains a major open problem, with physics-inspired approaches offering promising insights. Building on this perspective, we develop a thermodynamic framework to describe the stationary distributions of stochastic gradient descent (SGD) with weight decay for scale-invariant neural networks, a setting that both reflects practical architectures with normalization layers and permits theoretical analysis. We establish analogies between training hyperparameters (e.g., learning rate, weight decay) and thermodynamic variables such as temperature, pressure, and volume. Starting with a simplified isotropic noise model, we uncover a close correspondence between SGD dynamics and ideal gas behavior, validated through theory and simulation. Extending to training of neural networks, we show that key predictions of the framework, including the behavior of stationary entropy, align closely with experimental observations. This framework provides a principled foundation for interpreting training dynamics and may guide future work on hyperparameter tuning and the design of learning rate schedulers.","doi":"10.24963/ijcai.2026/530","url":"https://ijcai.org/proceedings/2026/530","pdf":"https://ijcai.org/proceedings/2026/0530.pdf","markdown":"https://ijcai.org/proceedings/2026/530.md","bibtex":"https://ijcai.org/proceedings/2026/bibtex/530","volume":{"slug":"2026","title":"Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2026","url":"https://ijcai.org/proceedings/2026/","year":2026,"location":"Bremen, Germany","dates":"15-21 August 2026","editors":"Diego Calvanese","isbn":"978-1-956792-09-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2026"},"section":"Machine Learning","track":"Main Track","number":530,"pages":{"first":4759,"last":4767},"keywords":["Multidisciplinary Topics and Applications: Physical sciences","Machine Learning: Supervised Learning","Machine Learning: Theory of deep learning"],"videos":[],"references":[{"position":1,"raw":"[Ali Mehmeti-Göpel and Wand, 2024] Christian Ali Mehmeti-Göpel and Michael Wand. On the weight dynamics of deep normalized networks. In Proceedings of the 41st International Conference on Machine Learn- ing, volume 235 of Proceedings of Machine Learning Research, pages 992-1007. PMLR, 21-27 Jul 2024.","title":"Christian Ali Mehmeti-Göpel and Michael Wand. On the weight dynamics of deep normalized networks","authors":"Ali Mehmeti, - Göpel, Wand","year":"2024","venue":"Proceedings of the 41st International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ba et al., 2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Ge- offrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.","title":"Layer normalization","authors":"Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E Hinton","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chaudhari and Soatto, 2018] Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. In International Conference on Learning Representations, 2018.","title":"Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks","authors":"Pratik Chaudhari, Stefano Soatto","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2024] Xiaoli Chen, Beatrice W Soh, Zi-En Ooi, Eleonore Vissol-Gaudin, Haijun Yu, Kostya S Novoselov, Kedar Hippalgaonkar, and Qianxiao Li. Constructing cus- tom thermodynamics using deep learning. Nature Compu- tational Science, 4(1):66-85, 2024.","title":"Constructing custom thermodynamics using deep learning","authors":"Xiaoli Chen, Beatrice W Soh, Zi-En Ooi, Eleonore Vissol-Gaudin, Haijun Yu, S Kostya, Kedar Novoselov, Qianxiao Hippalgaonkar, Li","year":"2024","venue":"Nature Computational Science","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cho and Lee, 2017] Minhyung Cho and Jaehyung Lee. Rie- mannian approach to batch normalization. In Proceedings of the 31st International Conference on Neural Informa- tion Processing Systems, NIPS'17, page 5231-5241.","title":"Minhyung Cho and Jaehyung Lee. Riemannian approach to batch normalization","authors":"Lee Cho","year":"2017","venue":"Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Delattre and Fournier, 2017] Sylvain Delattre and Nicolas Fournier. On the kozachenko-leonenko entropy estimator. Journal of Statistical Planning and Inference, 185:69-93, 2017.","title":"On the kozachenko-leonenko entropy estimator","authors":"Fournier Delattre, Sylvain Delattre, Nicolas Fournier","year":"2017","venue":"Journal of Statistical Planning and Inference","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Izmailov et al., 2018] Pavel Izmailov, Andrew Gordon Wil- son, Dmitrii Podoprikhin, Dmitry Vetrov, and Timur Garipov. Averaging weights leads to wider optima and better generalization. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, pages 876-885, 2018.","title":"Averaging weights leads to wider optima and better generalization","authors":"Izmailov","year":"2018","venue":"34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jastrzębski et al., 2017] Stanisław Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influ- encing minima in sgd. arXiv preprint arXiv:1711.04623, 2017.","title":"Three factors influencing minima in sgd","authors":"Stanisław Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, Amos Storkey","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kodryan et al., 2022] Maxim Kodryan, Ekaterina Lobacheva, Maksim Nakhodnov, and Dmitry Vetrov. Training scale-invariant neural networks on the sphere can happen in three regimes. In Advances in Neural Information Processing Systems, 2022.","title":"Training scale-invariant neural networks on the sphere can happen in three regimes","authors":"Kodryan","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kozachenko and Leonenko, 1987] Leonid Fedorovich Kozachenko and Nikolai Nikolaevich Leonenko. Sample estimate of the entropy of a random vector. Problems Inform. Transmission, 23:95-101, 1987.","title":"Sample estimate of the entropy of a random vector","authors":"Leonid Fedorovich, Kozachenko, Nikolai Nikolaevich, Leonenko","year":"1987","venue":"Problems Inform. Transmission","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kozyrev, 2025] Sergei Kozyrev. How to explain grokking, 2025.","title":"How to explain grokking","authors":"Sergei Kozyrev","year":"2025","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krizhevsky et al., 2009a] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-10 (canadian institute for ad- vanced research). 2009.","title":"CIFAR-10","authors":"Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Krizhevsky et al., 2009b] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-100 (canadian institute for ad- vanced research). 2009.","title":"CIFAR-100 (canadian institute for advanced research","authors":"Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kunin et al., 2023] Daniel Kunin, Javier Sagastuy-Brena, Lauren Gillespie, Eshed Margalit, Hidenori Tanaka, Surya Ganguli, and Daniel L. K. Yamins. The limiting dynam- ics of SGD: Modified loss, phase-space oscillations, and anomalous diffusion. Neural Computation, 36(1):151- 174, 2023.","title":"The limiting dynamics of SGD: Modified loss, phase-space oscillations, and anomalous diffusion","authors":"Javier Daniel Kunin, Lauren Sagastuy-Brena, Eshed Gillespie, Hidenori Margalit, Surya Tanaka, Ganguli, L K Daniel, Yamins","year":"2023","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li and Arora, 2020] Zhiyuan Li and Sanjeev Arora. An ex- ponential learning rate schedule for deep learning. In International Conference on Learning Representations, 2020.","title":"An exponential learning rate schedule for deep learning","authors":"Li, Arora, Zhiyuan Li, Sanjeev Arora","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2020] Zhiyuan Li, Kaifeng Lyu, and Sanjeev Arora. Reconciling modern deep learning with tradi- tional optimization analyses: The intrinsic learning rate. Advances in Neural Information Processing Systems, 33, 2020.","title":"Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate","authors":"Zhiyuan Li, Kaifeng Lyu, Sanjeev Arora","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2025] Ziming Liu, Yizhou Liu, Jeff Gore, and Max Tegmark. Neural thermodynamic laws for large lan- guage model training, 2025.","title":"Neural thermodynamic laws for large language model training","authors":"Ziming Liu, Yizhou Liu, Jeff Gore, Max Tegmark","year":"2025","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Loshchilov et al., 2025] Ilya Loshchilov, Cheng-Ping Hsieh, Simeng Sun, and Boris Ginsburg. nGPT: Nor- malized transformer with representation learning on the hypersphere. In The Thirteenth International Conference on Learning Representations, 2025.","title":"nGPT: Normalized transformer with representation learning on the hypersphere","authors":"Loshchilov","year":"2025","venue":"The Thirteenth International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lyle et al., 2024] Clare Lyle, Zeyu Zheng, Khimya Khetarpal, James Martens, Hado van Hasselt, Razvan Pascanu, and Will Dabney. Normalization and effective learning rates in reinforcement learning. In Advances in Neural Information Processing Systems, volume 37, pages 106440-106473.","title":"Normalization and effective learning rates in reinforcement learning","authors":"Clare Lyle, Zeyu Zheng, Khimya Khetarpal, James Martens, Hado Van Hasselt, Razvan Pascanu, Will Dabney","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"Curran Associates, Inc., 2024.","title":null,"authors":"Curran Associates, Inc","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mandt et al., 2017] Stephan Mandt, Matthew D. Hoffman, and David M. Blei. Stochastic gradient descent as ap- proximate bayesian inference. J. Mach. Learn. Res., 18(1):4873-4907, January 2017.","title":"Stochastic gradient descent as approximate bayesian inference","authors":"Stephan Mandt, Matthew D Hoffman, David M Blei","year":"2017","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sadrtdinov et al., 2024] Ildus Sadrtdinov, Maxim Kodryan, Eduard Pokonechny, Ekaterina Lobacheva, and Dmitry Vetrov. Where do large learning rates lead us? In Advances in Neural Information Processing Systems, volume 37, pages 58445-58479.","title":"Ildus Sadrtdinov, Maxim Kodryan, Eduard Pokonechny, Ekaterina Lobacheva, and Dmitry Vetrov. Where do large learning rates lead us","authors":"Sadrtdinov","year":"2024","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"Curran Associates, Inc., 2024.","title":null,"authors":"Curran Associates, Inc","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wan et al., 2021] Ruosi Wan, Zhanxing Zhu, Xiangyu Zhang, and Jian Sun. Spherical motion dynamics: Learn- ing dynamics of normalized neural network using sgd and weight decay. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 6380-6391.","title":"Spherical motion dynamics: Learning dynamics of normalized neural network using sgd and weight decay","authors":"Wan","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"Curran Associates, Inc., 2021.","title":null,"authors":"Curran Associates, Inc","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang and Wang, 2022] Yi Wang and Zhiren Wang. Three- stage evolution and fast equilibrium for SGD with non- degerate critical points. In Proceedings of the 39th Inter- national Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 23092-23113. PMLR, 17-23 Jul 2022.","title":"Threestage evolution and fast equilibrium for SGD with nondegerate critical points","authors":"Wang, Wang ; ] Yi, Wang, Zhiren Wang","year":"2022","venue":"Proceedings of the 39th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang, 2024] Dong Zhang. On the temperature of machine learning systems, 2024.","title":"On the temperature of machine learning systems","authors":"Dong Zhang","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}