{"type":"ScholarlyArticle","title":"Physics-Informed Neural Networks: Minimizing Residual Loss with Wide Networks and Effective Activations","authors":[{"name":"Nima Hosseini Dashtbayaz","slug":"nima-hosseini-dashtbayaz","given":"Nima","family":"Hosseini Dashtbayaz","affiliation":"Department of Computer Science , University of Western Ontario"},{"name":"Ghazal Farhani","slug":"ghazal-farhani","given":"Ghazal","family":"Farhani","affiliation":"National Research Council of Canada"},{"name":"Boyu Wang","slug":"boyu-wang","given":"Boyu","family":"Wang","affiliation":"Department of Computer Science , University of Western Ontario; Vector Institute"},{"name":"Charles X. Ling","slug":"charles-x-ling","given":"Charles X.","family":"Ling","affiliation":"Department of Computer Science , University of Western Ontario"}],"abstract":"The residual loss in Physics-Informed Neural Networks (PINNs) alters the simple recursive relation of layers in a feed-forward neural network by applying a differential operator, resulting in a loss landscape that is inherently different from those of common supervised problems. Therefore, relying on the existing theory leads to unjustified design choices and suboptimal performance. In this work, we analyze the residual loss by studying its characteristics at critical points to find the conditions that result in effective training of PINNs. Specifically, we first show that under certain conditions, the residual loss of PINNs can be globally minimized by a wide neural network. Furthermore, our analysis also reveals that an activation function with well-behaved high-order derivatives plays a crucial role in minimizing the residual loss. In particular, to solve a k-th order PDE, the k-th derivative of the activation function should be bijective. The established theory paves the way for designing and choosing effective activation functions for PINNs and explains why periodic activations have shown promising performance in certain cases. Finally, we verify our findings by conducting a set of experiments on several PDEs. Our code is publicly available at https://github.com/nimahsn/pinns_tf2.","doi":"10.24963/ijcai.2024/647","url":"https://ijcai.org/proceedings/2024/647","pdf":"https://ijcai.org/proceedings/2024/0647.pdf","markdown":"https://ijcai.org/proceedings/2024/647.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/647","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":647,"pages":{"first":5853,"last":5861},"keywords":["Machine Learning: ML: Applications","Multidisciplinary Topics and Applications: MTA: Physical sciences"],"videos":[],"references":[{"position":1,"raw":"[Allen-Zhu et al., 2019] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In International conference on ma- chine learning, pages 242-252. PMLR, 2019.","title":"A convergence theory for deep learning via over-parameterization","authors":"Allen-Zhu","year":"2019","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Belbute-Peres and Kolter, 2022] Filipe de Avila Belbute- Peres and J Zico Kolter. Simple initialization and parametrization of sinusoidal networks via their kernel bandwidth. arXiv preprint arXiv:2211.14503, 2022.","title":"Simple initialization and parametrization of sinusoidal networks via their kernel bandwidth","authors":"Filipe De, Avila Belbute-Peres, J Zico, Kolter","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cai et al., 2021] Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Karniadakis. Physics-informed neural networks (pinns) for fluid me- chanics: A review. Acta Mechanica Sinica, 37(12):1727- 1738, 2021.","title":"Physics-informed neural networks (pinns) for fluid mechanics: A review","authors":"Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, George Em Karniadakis","year":"2021","venue":"Acta Mechanica Sinica","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2020a] Yuyao Chen, Lu Lu, George Em Kar- niadakis, and Luca Dal Negro. Physics-informed neural networks for inverse problems in nano-optics and meta- materials. Optics express, 28(8):11618-11633, 2020.","title":"Physics-informed neural networks for inverse problems in nano-optics and metamaterials","authors":"Chen","year":"2020","venue":"Optics express","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2020b] Zixiang Chen, Yuan Cao, Quanquan Gu, and Tong Zhang. A generalized neural tangent ker- nel analysis for two-layer neural networks. Advances in Neural Information Processing Systems, 33:13363-13373, 2020.","title":"A generalized neural tangent kernel analysis for two-layer neural networks","authors":"Chen","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dong and Ni, 2021] Suchuan Dong and Naxian Ni. A method for representing periodic functions and enforcing exactly periodic boundary conditions with deep neural net- works. Journal of Computational Physics, 435:110242, 2021.","title":"Suchuan Dong and Naxian Ni. A method for representing periodic functions and enforcing exactly periodic boundary conditions with deep neural networks","authors":"Ni Dong","year":"2021","venue":"Journal of Computational Physics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Du and Hu, 2019] Simon Du and Wei Hu. Width provably matters in optimization for deep linear neural networks. In International Conference on Machine Learning, pages 1655-1664. PMLR, 2019.","title":"Width provably matters in optimization for deep linear neural networks","authors":"Simon Du, Wei Hu","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Du et al., 2019] Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global min- ima of deep neural networks. In International conference on machine learning, pages 1675-1685. PMLR, 2019.","title":"Gradient descent finds global minima of deep neural networks","authors":"Simon Du, Jason Lee, Haochuan Li, Liwei Wang, Xiyu Zhai","year":"2019","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Farhani et al., 2022] G. Farhani, Alexander Kazachek, and Boyu Wang. Momentum diminishes the effect of spectral bias in physics-informed neural networks. arXiv preprint arXiv:2206.14862, 2022.","title":"Momentum diminishes the effect of spectral bias in physics-informed neural networks","authors":"G Farhani, Alexander Kazachek, Boyu Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jacot et al., 2018] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.","title":"Neural tangent kernel: Convergence and generalization in neural networks","authors":"Arthur Jacot, Franck Gabriel, Clément Hongler","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krishnapriyan et al., 2021] Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems, 34:26548-26560, 2021.","title":"Characterizing possible failure modes in physics-informed neural networks","authors":"Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, Michael W Mahoney","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lee et al., 2019] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient de- scent. Advances in neural information processing systems, 32, 2019.","title":"Wide neural networks of any depth evolve as linear models under gradient descent","authors":"Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, Jeffrey Pennington","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2020] Zongyi Li, Nikola Kovachki, Kamyar Az- izzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural op- erator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020.","title":"Fourier neural operator for parametric partial differential equations","authors":"Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2020] Chaoyue Liu, Libin Zhu, and Misha Belkin. On the linearity of large non-linear models: when and why the tangent kernel is constant. Advances in Neural Information Processing Systems, 33:15954-15964, 2020.","title":"On the linearity of large non-linear models: when and why the tangent kernel is constant","authors":"Liu","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[McClenny and Braga-Neto, 2020] Levi McClenny and Ulisses Braga-Neto. Self-adaptive physics-informed neural networks using a soft attention mechanism. arXiv preprint arXiv:2009.04544, 2020.","title":"Self-adaptive physics-informed neural networks using a soft attention mechanism","authors":"Braga-Neto ; Mcclenny, Levi Mcclenny, Ulisses Braga-Neto","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Meronen et al., 2021] Lassi Meronen, Martin Trapp, and Arno Solin. Periodic activation functions induce stationar- ity. Advances in Neural Information Processing Systems, 34:1673-1685, 2021.","title":"Periodic activation functions induce stationarity","authors":"Lassi Meronen, Martin Trapp, Arno Solin","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Milnor et al., 1965] John Milnor, L. Siebenmann, and J. Sondow. Lectures on the H-Cobordism Theorem. Princeton University Press, 1965.","title":"Lectures on the H-Cobordism Theorem","authors":"L John Milnor, J Siebenmann, Sondow","year":"1965","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nguyen and Hein, 2017] Quynh Nguyen and Matthias Hein. The loss surface of deep and wide neural networks. In International conference on machine learning, pages 2603-2612. PMLR, 2017.","title":"The loss surface of deep and wide neural networks","authors":"Quynh Nguyen, Matthias Hein","year":"2017","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nguyen and Mondelli, 2020] Quynh N Nguyen and Marco Mondelli. Global convergence of deep networks with one wide layer followed by pyramidal topology. Advances in Neural Information Processing Systems, 33:11961-11972, 2020.","title":"Global convergence of deep networks with one wide layer followed by pyramidal topology","authors":"N Quynh, Marco Nguyen, Mondelli","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Oymak and Soltanolkotabi, 2020] Samet Oymak and Mahdi Soltanolkotabi. Toward moderate overparame- terization: Global convergence guarantees for training shallow neural networks. IEEE Journal on Selected Areas in Information Theory, 1(1):84-105, 2020.","title":"Toward moderate overparameterization: Global convergence guarantees for training shallow neural networks","authors":"Samet Oymak, Mahdi Soltanolkotabi","year":"2020","venue":"IEEE Journal on Selected Areas in Information Theory","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rahaman et al., 2018] Nasim Rahaman, A. Baratin, De- vansh Arpit, Felix Dräxler, Min Lin, F. Hamprecht, Yoshua Bengio, and Aaron C. Courville. On the spectral bias of neural networks. International Conference On Ma- chine Learning, 2018.","title":"On the spectral bias of neural networks","authors":"A Nasim Rahaman, Devansh Baratin, Felix Arpit, Min Dräxler, F Lin, Yoshua Hamprecht, Aaron C Bengio, Courville","year":"2018","venue":"International Conference On Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Raissi et al., 2017] Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differ- ential equations. arXiv preprint arXiv:1711.10561, 2017.","title":"Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations","authors":"Maziar Raissi, Paris Perdikaris, George Em Karniadakis","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Reiser et al., 2022] Patrick Reiser, Marlen Neubert, André Eberhard, Luca Torresi, Chen Zhou, Chen Shao, Houssam Metni, Clint van Hoesel, Henrik Schopmans, Timo Som- mer, et al. Graph neural networks for materials science and chemistry. Communications Materials, 3(1):93, 2022.","title":"Graph neural networks for materials science and chemistry","authors":"Patrick Reiser, Marlen Neubert, André Eberhard, Luca Torresi, Chen Zhou, Chen Shao, Houssam Metni, Clint Van Hoesel, Henrik Schopmans, Timo Sommer","year":"2022","venue":"Communications Materials","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Safran and Shamir, 2016] Itay Safran and Ohad Shamir. On the quality of the initial basin in overspecified neural net- works. In International Conference on Machine Learning, pages 774-782. PMLR, 2016.","title":"On the quality of the initial basin in overspecified neural networks","authors":"Itay Safran, Ohad Shamir","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sirignano and Spiliopoulos, 2018] Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algo- rithm for solving partial differential equations. Journal of computational physics, 375:1339-1364, 2018.","title":"Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algo-rithm for solving partial differential equations","authors":"Spiliopoulos Sirignano","year":"2018","venue":"Journal of computational physics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sitzmann et al., 2020] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wet- zstein. Implicit neural representations with periodic activa- tion functions. Advances in neural information processing systems, 33:7462-7473, 2020.","title":"Implicit neural representations with periodic activation functions","authors":"Julien Vincent Sitzmann, Alexander Martel, David Bergman, Gordon Lindell, Wetzstein","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2020a] Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient pathologies in physics-informed neural networks. arXiv preprint arXiv:2001.04536, 2020.","title":"Understanding and mitigating gradient pathologies in physics-informed neural networks","authors":"Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2020b] Sifan Wang, Xinling Yu, and P. Perdikaris. When and why pinns fail to train: A neural tangent kernel perspective. Journal Of Computa- tional Physics, 2020.","title":"When and why pinns fail to train: A neural tangent kernel perspective","authors":"Wang","year":"2020","venue":"Journal Of Computational Physics","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2021] Sifan Wang, Hanwen Wang, and Paris Perdikaris. On the eigenvector bias of fourier feature net- works: From regression to solving multi-scale pdes with physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 384:113938, 2021.","title":"On the eigenvector bias of fourier feature networks: From regression to solving multi-scale pdes with physics-informed neural networks","authors":"Wang","year":"2021","venue":"Computer Methods in Applied Mechanics and Engineering","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2022] Sifan Wang, Shyam Sankaran, and Paris Perdikaris. Respecting causality is all you need for train- ing physics-informed neural networks. arXiv preprint arXiv:2203.07404, 2022.","title":"Respecting causality is all you need for training physics-informed neural networks","authors":"Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wight and Zhao, 2020] Colby L Wight and Jia Zhao. Solv- ing allen-cahn and cahn-hilliard equations using the adap- tive physics informed neural networks. arXiv preprint arXiv:2007.04542, 2020.","title":"Solving allen-cahn and cahn-hilliard equations using the adaptive physics informed neural networks","authors":"L Colby, Jia Wight, Zhao","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wong et al., 2022] Jian Cheng Wong, Chinchun Ooi, Ab- hishek Gupta, and Yew-Soon Ong. Learning in sinusoidal spaces with physics-informed neural networks. IEEE Transactions on Artificial Intelligence, 2022.","title":"Learning in sinusoidal spaces with physics-informed neural networks","authors":"Jian Cheng Wong, Chinchun Ooi, Abhishek Gupta, Yew-Soon Ong","year":"2022","venue":"IEEE Transactions on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2022] Jie Zhang, Yihui Zhao, Fergus Shone, Zhenhong Li, Alejandro F Frangi, Sheng Quan Xie, and Zhi-Qiang Zhang. Physics-informed deep learning for musculoskeletal modeling: Predicting muscle forces and joint kinematics from surface emg. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31:484- 493, 2022.","title":"Physics-informed deep learning for musculoskeletal modeling: Predicting muscle forces and joint kinematics from surface emg","authors":"Jie Zhang, Yihui Zhao, Fergus Shone, Zhenhong Li, Alejandro F Frangi, Sheng Quan Xie, Zhi-Qiang Zhang","year":"2022","venue":"IEEE Transactions on Neural Systems and Rehabilitation Engineering","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}