{"type":"ScholarlyArticle","title":"Distilling Governing Laws and Source Input for Dynamical Systems from Videos","authors":[{"name":"Lele Luan","slug":"lele-luan","given":"Lele","family":"Luan","affiliation":"Department of Civil and Environmental Engineering , Northeastern University , Boston , MA , USA"},{"name":"Yang Liu","slug":"yang-liu","given":"Yang","family":"Liu","affiliation":"School of Engineering Sciences , University of the Chinese Academy of Sciences , Beijing , China"},{"name":"Hao Sun","slug":"hao-sun","given":"Hao","family":"Sun","affiliation":"Gaoling School of Artificial Intelligence , Renmin University of China , Beijing , 100872 , China"}],"abstract":"Distilling interpretable physical laws from videos has led to expanded interest in the computer vision community recently thanks to the advances in deep learning, but still remains a great challenge. This paper introduces an end-to-end unsupervised deep learning framework to uncover the explicit governing equations of dynamics presented by moving object(s), based on recorded videos. Instead in the pixel (spatial) coordinate system of image space, the physical law is modeled in a regressed underlying physical coordinate system where the physical states follow potential explicit governing equations. A numerical integrator-based sparse regression module is designed and serves as a physical constraint to the autoencoder and coordinate system regression, and, in the meanwhile, uncover the parsimonious closed-form governing equations from the learned physical states. Experiments on simulated dynamical scenes show that the proposed method is able to distill closed-form governing equations and simultaneously identify unknown excitation input for several dynamical systems recorded by videos, which fills in the gap in literature where no existing methods are available and applicable for solving this type of problem.","doi":"10.24963/ijcai.2022/541","url":"https://ijcai.org/proceedings/2022/541","pdf":"https://ijcai.org/proceedings/2022/0541.pdf","markdown":"https://ijcai.org/proceedings/2022/541.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/541","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":541,"pages":{"first":3898,"last":3904},"keywords":["Machine Learning: Explainable/Interpretable Machine Learning","Machine Learning: Autoencoders","Computer Vision: Video analysis and understanding","Multidisciplinary Topics and Applications: Physical Science","Computer Vision: Interpretability and Transparency"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986747","length_sec":79,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984890","length_sec":356,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Belbute-Peres et al., 2018] Filipe de A Belbute-Peres, Kevin A Smith, Kelsey R Allen, Joshua B Tenenbaum, and J Zico Kolter. End-to-end differentiable physics for learning and control. Advances in Neural Information Processing Systems, 31:7178-7189, 2018.","title":"End-to-end differentiable physics for learning and control","authors":"Filipe De A Belbute-Peres, Kevin A Smith, Joshua B Kelsey R Allen, J Zico Tenenbaum, Kolter","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brunton et al., 2016] Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical sys- tems. Proceedings of the National Academy of Sciences, 113(15):3932-3937, 2016.","title":"Discovering governing equations from data by sparse identification of nonlinear dynamical systems","authors":"Joshua L Steven L Brunton, Nathan Proctor, Kutz","year":"2016","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Champion et al., 2019] Kathleen Champion, Bethany Lusch, J Nathan Kutz, and Steven L Brunton. Data- driven discovery of coordinates and governing equations. Proceedings of the National Academy of Sciences, 116(45):22445-22451, 2019.","title":"Datadriven discovery of coordinates and governing equations","authors":"Kathleen Champion, Bethany Lusch, Nathan Kutz, Steven L Brunton","year":"2019","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2021] Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan-Yee Kenneth Wong, Joshua B Tenenbaum, and Chuang Gan. Grounding physical concepts of objects and events through dynamic visual reasoning. International Conference on Learning Representations, 2021.","title":"Grounding physical concepts of objects and events through dynamic visual reasoning","authors":"Chen","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ehrhardt et al., 2017] Sebastien Ehrhardt, Aron Monszpart, Niloy J Mitra, and Andrea Vedaldi. Learning a physical long-term predictor. arXiv:1703.00247, 2017.","title":"Learning a physical long-term predictor","authors":"Sebastien Ehrhardt, Aron Monszpart, J Niloy, Andrea Mitra, Vedaldi","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ehrhardt et al., 2018] Sebastien Ehrhardt, Aron Monszpart, Niloy Mitra, and Andrea Vedaldi. Unsupervised intuitive physics from visual observations. In Asian Conference on Computer Vision, pages 700-716. Springer, 2018.","title":"Unsupervised intuitive physics from visual observations","authors":"Sebastien Ehrhardt, Aron Monszpart, Niloy Mitra, Andrea Vedaldi","year":"2018","venue":"Asian Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fragkiadaki et al., 2016] Katerina Fragkiadaki, Pulkit Agrawal, Sergey Levine, and Jitendra Malik. Learning visual predictive models of physics for playing billiards. In International Conference on Learning Representations, 2016.","title":"Learning visual predictive models of physics for playing billiards","authors":"Katerina Fragkiadaki, Pulkit Agrawal, Sergey Levine, Jitendra Malik","year":"2016","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hsieh et al., 2018] Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video pre- diction. In Advances in Neural Information Processing Systems, pages 515-524, 2018.","title":"Learning to decompose and disentangle representations for video prediction","authors":"Hsieh","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jaques et al., 2020] Miguel Jaques, Michael Burke, and Timothy Hospedales. Physics-as-inverse-graphics: Un- supervised physical parameter estimation from video. In International Conference on Learning Representations, 2020.","title":"Physics-as-inverse-graphics: Unsupervised physical parameter estimation from video","authors":"Jaques","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jonschkowski et al., 2017] Rico Jonschkowski, Roland Hafner, Jonathan Scholz, and Martin Riedmiller. Pves: Position-velocity encoders for unsupervised learning of structured state representations. arXiv:1705.09805, 2017.","title":"Pves: Position-velocity encoders for unsupervised learning of structured state representations","authors":"Jonschkowski","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kosiorek et al., 2018] Adam R Kosiorek, Hyunjik Kim, In- gmar Posner, and Yee Whye Teh. Sequential attend, infer, repeat: generative modelling of moving objects. In Ad- vances in Neural Information Processing Systems, pages 8615-8625, 2018.","title":"Sequential attend, infer, repeat: generative modelling of moving objects","authors":"Kosiorek","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kossen et al., 2020] Jannik Kossen, Karl Stelzner, Marcel Hussing, Claas Voelcker, and Kristian Kersting. Struc- tured object-aware physics prediction for video modeling and planning. International Conference on Learning Rep- resentations, 2020.","title":"Structured object-aware physics prediction for video modeling and planning","authors":"Jannik Kossen, Karl Stelzner, Marcel Hussing, Claas Voelcker, Kristian Kersting","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kutz et al., 2016] J Nathan Kutz, Steven L Brunton, Bingni W Brunton, and Joshua L Proctor. Dynamic mode decomposition: data-driven modeling of complex systems. SIAM, 2016.","title":"Dynamic mode decomposition: data-driven modeling of complex systems","authors":"Kutz","year":"2016","venue":"SIAM","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mottaghi et al., 2016] Roozbeh Mottaghi, Hessam Bagher- inezhad, Mohammad Rastegari, and Ali Farhadi. Newto- nian scene understanding: Unfolding the dynamics of ob- jects in static images. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 3521-3529, 2016.","title":"Newtonian scene understanding: Unfolding the dynamics of objects in static images","authors":"Roozbeh Mottaghi, Hessam Bagherinezhad, Mohammad Rastegari, Ali Farhadi","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Purushwalkam et al., 2019] Senthil Purushwalkam, Abhi- nav Gupta, Danny M Kaufman, and Bryan Russell. Bounce and learn: Modeling scene dynamics with real- world bounces. International Conference on Learning Representations, 2019.","title":"Bounce and learn: Modeling scene dynamics with realworld bounces","authors":"Senthil Purushwalkam, Abhinav Gupta, Danny M Kaufman, Bryan Russell","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rudy et al., 2019] Samuel Rudy, Alessandro Alla, Steven L Brunton, and J Nathan Kutz. Data-driven identification of parametric partial differential equations. SIAM Journal on Applied Dynamical Systems, 18(2):643-660, 2019.","title":"Data-driven identification of parametric partial differential equations","authors":"Samuel Rudy, Alessandro Alla, Steven L Brunton, Nathan Kutz","year":"2019","venue":"SIAM Journal on Applied Dynamical Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Udrescu and Tegmark, 2020] Silviu-Marian Udrescu and Max Tegmark. AI Feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16):2631, 2020.","title":"AI Feynman: A physics-inspired method for symbolic regression","authors":"Marian Silviu, Max Udrescu, Tegmark","year":"2020","venue":"Science Advances","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Udrescu and Tegmark, 2021] Silviu-Marian Udrescu and Max Tegmark. Symbolic pregression: discovering phys- ical laws from distorted video. Physical Review E, 103(4):043307, 2021.","title":"Symbolic pregression: discovering physical laws from distorted video","authors":"Marian Silviu, Max Udrescu, Tegmark","year":"2021","venue":"Physical Review E","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wu et al., 2015] Jiajun Wu, Ilker Yildirim, Joseph J Lim, Bill Freeman, and Josh Tenenbaum. Galileo: Perceiving physical object properties by integrating a physics engine with deep learning. Advances in Neural Information Pro- cessing Systems, 28:127-135, 2015.","title":"Galileo: Perceiving physical object properties by integrating a physics engine with deep learning","authors":"Jiajun Wu, Ilker Yildirim, Joseph J Lim, Bill Freeman, Josh Tenenbaum","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wu et al., 2016] Jiajun Wu, Joseph J Lim, Hongyi Zhang, Joshua B Tenenbaum, and William T Freeman. Physics 101: Learning physical object properties from unlabeled videos. In BMVC, volume 2, page 7, 2016.","title":"Physics 101: Learning physical object properties from unlabeled videos","authors":"Jiajun Wu, Joseph J Lim, Hongyi Zhang, Joshua B Tenenbaum, William T Freeman","year":"2016","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wu et al., 2017] Jiajun Wu, Erika Lu, Pushmeet Kohli, Bill Freeman, and Josh Tenenbaum. Learning to see physics via visual de-animation. In Advances in Neural Informa- tion Processing Systems, pages 153-164, 2017.","title":"Learning to see physics via visual de-animation","authors":"Jiajun Wu, Erika Lu, Pushmeet Kohli, Bill Freeman, Josh Tenenbaum","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ye et al., 2018] Tian Ye, Xiaolong Wang, James Davidson, and Abhinav Gupta. Interpretable intuitive physics model. In Proceedings of the European Conference on Computer Vision (ECCV), pages 87-102, 2018.","title":"Interpretable intuitive physics model","authors":"Tian Ye, Xiaolong Wang, James Davidson, Abhinav Gupta","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}