{"type":"ScholarlyArticle","title":"Vision-based Discovery of Nonlinear Dynamics for 3D Moving Target","authors":[{"name":"Zitong Zhang","slug":"zitong-zhang","given":"Zitong","family":"Zhang","affiliation":"Gaoling School of Artificial Intelligence , Renmin University of China , Beijing , China"},{"name":"Yang Liu","slug":"yang-liu","given":"Yang","family":"Liu","affiliation":"School of Engineering Science , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Hao Sun","slug":"hao-sun","given":"Hao","family":"Sun","affiliation":"Gaoling School of Artificial Intelligence , Renmin University of China , Beijing , China"}],"abstract":"Data-driven discovery of governing equations has kindled significant interests in many science and engineering areas. Existing studies primarily focus on uncovering equations that govern nonlinear dynamics based on direct measurement of the system states (e.g., trajectories). Limited efforts have been placed on distilling governing laws of dynamics directly from videos for moving targets in a 3D space. To this end, we propose a vision-based approach to automatically uncover governing equations of nonlinear dynamics for 3D moving targets via raw videos recorded by a set of cameras. The approach is composed of three key blocks: (1) a target tracking module that extracts plane pixel motions of the moving target in each video, (2) a Rodrigues' rotation formula-based coordinate transformation learning module that reconstructs the 3D coordinates with respect to a predefined reference point, and (3) a spline-enhanced library-based sparse regressor that uncovers the underlying governing law of dynamics. This framework is capable of effectively handling the challenges associated with measurement data, e.g., noise in the video, imprecise tracking of the target that causes data missing, etc. The efficacy of our method has been demonstrated through multiple sets of synthetic videos considering different nonlinear dynamics.","doi":"10.24963/ijcai.2024/682","url":"https://ijcai.org/proceedings/2024/682","pdf":"https://ijcai.org/proceedings/2024/0682.pdf","markdown":"https://ijcai.org/proceedings/2024/682.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/682","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":682,"pages":{"first":6170,"last":6178},"keywords":["Computer Vision: CV: Motion and tracking","Computer Vision: CV: Applications","Machine Learning: ML: Regression","Multidisciplinary Topics and Applications: MTA: Physical sciences"],"videos":[],"references":[{"position":1,"raw":"[Bongard and Lipson, 2007] Josh Bongard and Hod Lipson. Automated reverse engineering of nonlinear dynamical systems. Proceedings of the National Academy of Sci- ences, 104(24):9943-9948, 2007.","title":"Automated reverse engineering of nonlinear dynamical systems","authors":"Lipson Bongard, Josh Bongard, Hod Lipson","year":"2007","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brunton et al., 2016] Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical sys- tems. Proceedings of the National Academy of Sciences, 113(15):3932-3937, 2016.","title":"Discovering governing equations from data by sparse identification of nonlinear dynamical systems","authors":"Joshua L Steven L Brunton, Nathan Proctor, Kutz","year":"2016","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Champion et al., 2019] Kathleen Champion, Bethany Lusch, J Nathan Kutz, and Steven L Brunton. Data- driven discovery of coordinates and governing equations. Proceedings of the National Academy of Sciences, 116(45):22445-22451, 2019.","title":"Datadriven discovery of coordinates and governing equations","authors":"Kathleen Champion, Bethany Lusch, Nathan Kutz, Steven L Brunton","year":"2019","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2021a] Z Chen, J Mao, J Wu, KKY Wong, JB Tenenbaum, and C Gan. Grounding physical concepts of objects and events through dynamic visual reasoning. In International Conference on Learning Representations, 2021.","title":"Grounding physical concepts of objects and events through dynamic visual reasoning","authors":"Chen","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2021b] Zhao Chen, Yang Liu, and Hao Sun. Physics-informed learning of governing equations from scarce data. Nature Communications, 12(1):6136, 2021.","title":"Physics-informed learning of governing equations from scarce data","authors":"Chen","year":"2021","venue":"Nature Communications","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2022] Boyuan Chen, Kuang Huang, Sunand Raghupathi, Ishaan Chandratreya, Qiang Du, and Hod Lipson. Automated discovery of fundamental variables hidden in experimental data. Nature Computational Sci- ence, 2(7):433-442, 2022.","title":"Automated discovery of fundamental variables hidden in experimental data","authors":"Boyuan Chen, Kuang Huang, Sunand Raghupathi, Ishaan Chandratreya, Qiang Du, Hod Lipson","year":"2022","venue":"Nature Computational Science","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ciaparrone et al., 2020] Gioele Ciaparrone, Fran- cisco Luque Sánchez, Siham Tabik, Luigi Troiano, Roberto Tagliaferri, and Francisco Herrera. Deep learning in video multi-object tracking: A survey. Neurocomputing, 381:61-88, 2020.","title":"Deep learning in video multi-object tracking: A survey","authors":"Ciaparrone","year":"2020","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Floryan and Graham, 2022] Daniel Floryan and Michael D Graham. Data-driven discovery of intrinsic dynamics. Na- ture Machine Intelligence, 4(12):1113-1120, 2022.","title":"Data-driven discovery of intrinsic dynamics","authors":"Graham ; ] Floryan, Michael D Daniel Floryan, Graham","year":"2022","venue":"Nature Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Flynn et al., 2016] John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. Deepstereo: Learning to pre- dict new views from the world's imagery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5515-5524, 2016.","title":"Deepstereo: Learning to predict new views from the world's imagery","authors":"John Flynn, Ivan Neulander, James Philbin, Noah Snavely","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Galliani et al., 2016] Silvano Galliani, Katrin Lasinger, and Konrad Schindler. Gipuma: Massively parallel multi- view stereo reconstruction. Publikationen der Deutschen Gesellschaft für Photogrammetrie, Fernerkundung und Geoinformation e. V, 25(361-369):2, 2016.","title":"Gipuma: Massively parallel multiview stereo reconstruction","authors":"Silvano Galliani, Katrin Lasinger, Konrad Schindler","year":"2016","venue":"Publikationen der Deutschen Gesellschaft für Photogrammetrie, Fernerkundung und Geoinformation e. V","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gilpin, 2021] William Gilpin. Chaos as an interpretable benchmark for forecasting and data-driven modelling. arXiv preprint arXiv:2110.05266, 2021.","title":"Chaos as an interpretable benchmark for forecasting and data-driven modelling","authors":"William Gilpin, Gilpin","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huang et al., 2018] Po-Han Huang, Kevin Matzen, Jo- hannes Kopf, Narendra Ahuja, and Jia-Bin Huang. Deep- mvs: Learning multi-view stereopsis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2821-2830, 2018.","title":"Deepmvs: Learning multi-view stereopsis","authors":"Huang","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jaques et al., 2020] Miguel Jaques, Michael Burke, and Timothy Hospedales. Physics-as-inverse-graphics: Un- supervised physical parameter estimation from video. In International Conference on Learning Representations, 2020.","title":"Physics-as-inverse-graphics: Unsupervised physical parameter estimation from video","authors":"Jaques","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Koza, 1994] John R Koza. Genetic programming as a means for programming computers by natural selection. Statistics and computing, 4:87-112, 1994.","title":"Genetic programming as a means for programming computers by natural selection","authors":"John R Koza","year":"1994","venue":"Statistics and computing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2021] Mingyang Li, Zhijiang Du, Xiaoxing Ma, Wei Dong, and Yongzhuo Gao. A robot hand-eye calibra- tion method of line laser sensor based on 3d reconstruc- tion. Robotics and Computer-Integrated Manufacturing, 71:102136, 2021.","title":"A robot hand-eye calibration method of line laser sensor based on 3d reconstruction","authors":"Mingyang Li, Zhijiang Du, Xiaoxing Ma, Wei Dong, Yongzhuo Gao","year":"2021","venue":"Robotics and Computer-Integrated Manufacturing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lu et al., 2021] Qiang Lu, Fan Tao, Shuo Zhou, and Zhiguang Wang. Incorporating actor-critic in monte carlo tree search for symbolic regression. Neural Computing and Applications, pages 1-17, 2021.","title":"Incorporating actor-critic in monte carlo tree search for symbolic regression","authors":"Qiang Lu, Fan Tao, Shuo Zhou, Zhiguang Wang","year":"2021","venue":"Neural Computing and Applications","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Luan et al., 2022] Lele Luan, Yang Liu, and Hao Sun. Dis- tilling governing laws and source input for dynamical sys- tems from videos. Proceedings of the 31st International Joint Conference on Artificial Intelligence, pages 3873- 3879, 2022.","title":"Distilling governing laws and source input for dynamical systems from videos","authors":"Lele Luan, Yang Liu, Hao Sun","year":"2022","venue":"Proceedings of the 31st International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2022/541"},{"position":18,"raw":"[Ma et al., 2019] Xinzhu Ma, Zhihui Wang, Haojie Li, Pengbo Zhang, Wanli Ouyang, and Xin Fan. Accurate monocular 3d object detection via color-embedded 3d re- construction for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 6851-6860, 2019.","title":"Accurate monocular 3d object detection via color-embedded 3d reconstruction for autonomous driving","authors":"Ma","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Marvasti-Zadeh et al., 2021] Seyed Mojtaba Marvasti- Zadeh, Li Cheng, Hossein Ghanei-Yakhdan, and Shohreh Kasaei. Deep learning for visual tracking: A com- prehensive survey. IEEE Transactions on Intelligent Transportation Systems, 23(5):3943-3968, 2021.","title":"Deep learning for visual tracking: A comprehensive survey","authors":"Seyed Mojtaba Marvasti-Zadeh, Li Cheng, Hossein Ghanei-Yakhdan, Shohreh Kasaei","year":"2021","venue":"IEEE Transactions on Intelligent Transportation Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mueller et al., 2017] Matthias Mueller, Neil Smith, and Bernard Ghanem. Context-aware correlation filter track- ing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1396-1404, 2017.","title":"Context-aware correlation filter tracking","authors":"Matthias Mueller, Neil Smith, Bernard Ghanem","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mundhenk et al., 2021] T Nathan Mundhenk, Mikel Landa- juela, Ruben Glatt, Claudio P Santiago, Daniel M Faissol, and Brenden K Petersen. Symbolic regression via neural- guided genetic programming population seeding. arXiv preprint arXiv:2111.00053, 2021.","title":"Symbolic regression via neuralguided genetic programming population seeding","authors":"Mikel Nathan Mundhenk, Ruben Landajuela, Claudio P Glatt, Daniel M Santiago, Brenden K Faissol, Petersen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nalpantidis and Gasteratos, 2011] Lazaros Nalpantidis and Antonios Gasteratos. Stereovision-based fuzzy obstacle avoidance method. International Journal of Humanoid Robotics, 8(01):169-183, 2011.","title":"Lazaros Nalpantidis and Antonios Gasteratos. Stereovision-based fuzzy obstacle avoidance method","authors":"Gasteratos Nalpantidis","year":"2011","venue":"International Journal of Humanoid Robotics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Peng et al., 2020] Wanli Peng, Hao Pan, He Liu, and Yi Sun. Ida-3d: Instance-depth-aware 3d object detection from stereo vision for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 13015-13024, 2020.","title":"Ida-3d: Instance-depth-aware 3d object detection from stereo vision for autonomous driving","authors":"Wanli Peng, He Hao Pan, Yi Liu, Sun","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Petersen et al., 2021] Brenden K Petersen, Mikel Landa- juela Larma, Terrell N Mundhenk, Claudio Prata Santi- ago, Soo Kyung Kim, and Joanne Taery Kim. Deep sym- bolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. In Interna- tional Conference on Learning Representations, 2021.","title":"Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients","authors":"Mikel Landajuela Brenden K Petersen, Terrell N Larma, Claudio Prata Mundhenk, Soo Santiago, Kyung Kim, Joanne Taery Kim","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Raissi et al., 2019] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and in- verse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686- 707, 2019.","title":"Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations","authors":"Maziar Raissi, Paris Perdikaris, George E Karniadakis","year":"2019","venue":"Journal of Computational Physics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Rao et al., 2022] Chengping Rao, Pu Ren, Yang Liu, and Hao Sun. Discovering nonlinear pdes from scarce data with physics-encoded learning. In The Tenth International Conference on Learning Representations, 2022.","title":"Discovering nonlinear pdes from scarce data with physics-encoded learning","authors":"Chengping Rao, Pu Ren, Yang Liu, Hao Sun","year":"2022","venue":"The Tenth International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Rao et al., 2023] Chengping Rao, Pu Ren, Qi Wang, Oral Buyukozturk, Hao Sun, and Yang Liu. Encoding physics to learn reaction-diffusion processes. Nature Machine In- telligence, 5:765-779, 2023.","title":"Encoding physics to learn reaction-diffusion processes","authors":"Rao","year":"2023","venue":"Nature Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Redmon et al., 2016] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 779-788, 2016.","title":"You only look once: Unified, real-time object detection","authors":"Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Rudy et al., 2017] Samuel H Rudy, Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Data-driven dis- covery of partial differential equations. Science Advances, 3(4):e1602614, 2017.","title":"Data-driven discovery of partial differential equations","authors":"Steven L Samuel H Rudy, Joshua L Brunton, Nathan Proctor, Kutz","year":"2017","venue":"Science Advances","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sahoo et al., 2018] Subham Sahoo, Christoph Lampert, and Georg Martius. Learning equations for extrapolation and control. In International Conference on Machine Learn- ing, pages 4442-4450, 2018.","title":"Learning equations for extrapolation and control","authors":"Subham Sahoo, Christoph Lampert, Georg Martius","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Schmidt and Lipson, 2009] Michael Schmidt and Hod Lip- son. Distilling free-form natural laws from experimental data. Science, 324(5923):81-85, 2009.","title":"Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data","authors":"Lipson Schmidt","year":"2009","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Schönberger et al., 2016] Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. Pixel- wise view selection for unstructured multi-view stereo. In Computer Vision-ECCV 2016: 14th European Confer- ence, pages 501-518. Springer, 2016.","title":"Pixelwise view selection for unstructured multi-view stereo","authors":"Enliang Johannes L Schönberger, Jan-Michael Zheng, Marc Frahm, Pollefeys","year":"2016","venue":"Computer Vision-ECCV 2016: 14th European Conference","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Sun et al., 2021] Fangzheng Sun, Yang Liu, and Hao Sun. Physics-informed spline learning for nonlinear dynamics discovery. Proceedings of the 30th International Joint Conference on Artificial Intelligence, pages 2054-2061, 2021.","title":"Physics-informed spline learning for nonlinear dynamics discovery","authors":"Fangzheng Sun, Yang Liu, Hao Sun","year":"2021","venue":"Proceedings of the 30th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2021/283"},{"position":34,"raw":"[Sun et al., 2022] Luning Sun, Daniel Zhengyu Huang, Hao Sun, and Jian-Xun Wang. Bayesian spline learning for equation discovery of nonlinear dynamics with quantified uncertainty. In Advances in Neural Information Process- ing Systems, 2022.","title":"Bayesian spline learning for equation discovery of nonlinear dynamics with quantified uncertainty","authors":"Luning Sun, Daniel Zhengyu Huang, Hao Sun, Jian-Xun Wang","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Sun et al., 2023] Fangzheng Sun, Yang Liu, Jian-Xun Wang, and Hao Sun. Symbolic physics learner: Discov- ering governing equations via monte carlo tree search. In The Eleventh International Conference on Learning Rep- resentations, 2023.","title":"Symbolic physics learner: Discovering governing equations via monte carlo tree search","authors":"Fangzheng Sun, Yang Liu, Jian-Xun Wang, Hao Sun","year":"2023","venue":"The Eleventh International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Udrescu and Tegmark, 2021] Silviu-Marian Udrescu and Max Tegmark. Symbolic pregression: Discovering phys- ical laws from distorted video. Physical Review E, 103(4):043307, 2021.","title":"Symbolic pregression: Discovering physical laws from distorted video","authors":"Marian Silviu, Max Udrescu, Tegmark","year":"2021","venue":"Physical Review E","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wang et al., 2021] Hengli Wang, Rui Fan, and Ming Liu. Cot-amflow: Adaptive modulation network with co- teaching strategy for unsupervised optical flow estimation. In Conference on Robot Learning, pages 143-155, 2021.","title":"Cot-amflow: Adaptive modulation network with coteaching strategy for unsupervised optical flow estimation","authors":"Hengli Wang, Rui Fan, Ming Liu","year":"2021","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wojke et al., 2017] Nicolai Wojke, Alex Bewley, and Diet- rich Paulus. Simple online and realtime tracking with a deep association metric. In 2017 IEEE International Con- ference on Image Processing (ICIP), pages 3645-3649, 2017.","title":"Simple online and realtime tracking with a deep association metric","authors":"Nicolai Wojke, Alex Bewley, Dietrich Paulus","year":"2017","venue":"IEEE International Conference on Image Processing (ICIP)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}