{"type":"ScholarlyArticle","title":"Fast Model Identification via Physics Engines for Data-Efficient Policy Search","authors":[{"name":"Shaojun Zhu","slug":"shaojun-zhu","given":"Shaojun","family":"Zhu","affiliation":"Department of Computer Science , Rutgers University , New Jersey , USA"},{"name":"Andrew Kimmel","slug":"andrew-kimmel","given":"Andrew","family":"Kimmel","affiliation":"Department of Computer Science , Rutgers University , New Jersey , USA"},{"name":"Kostas E. Bekris","slug":"kostas-e-bekris","given":"Kostas E.","family":"Bekris","affiliation":"Department of Computer Science , Rutgers University , New Jersey , USA"},{"name":"Abdeslam Boularias","slug":"abdeslam-boularias","given":"Abdeslam","family":"Boularias","affiliation":"Department of Computer Science , Rutgers University , New Jersey , USA"}],"abstract":"This paper presents a method for identifying mechanical parameters of robots or objects, such as their mass and friction coefficients. Key features are the use of off-the-shelf physics engines and the adaptation of a Bayesian optimization technique towards minimizing the number of real-world experiments needed for model-based reinforcement learning. The proposed framework reproduces in a physics engine experiments performed on a real robot and optimizes the model's mechanical parameters so as to match real-world trajectories. The optimized model is then used for learning a policy in simulation, before real-world deployment. It is well understood, however, that it is hard to exactly reproduce real trajectories in simulation. Moreover, a near-optimal policy can be frequently found with an imperfect model. Therefore, this work proposes a strategy for identifying a model that is just good enough to approximate the value of a locally optimal policy with a certain confidence, instead of wasting effort on identifying the most accurate model. Evaluations, performed both in simulation and on a real robotic manipulation task, indicate that the proposed strategy results in an overall time-efficient, integrated model identification and learning solution, which significantly improves the data-efficiency of existing policy search algorithms.","doi":"10.24963/ijcai.2018/451","url":"https://ijcai.org/proceedings/2018/451","pdf":"https://ijcai.org/proceedings/2018/0451.pdf","markdown":"https://ijcai.org/proceedings/2018/451.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/451","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":451,"pages":{"first":3249,"last":3256},"keywords":["Machine Learning: Active Learning","Machine Learning: Online Learning","Machine Learning: Reinforcement Learning","Robotics: Learning in Robotics"],"videos":[],"references":[{"position":1,"raw":"[Abbeel et al., 2006] Pieter Abbeel, Morgan Quigley, and Andrew Y Ng. Using inaccurate models in reinforcement learning. In Proc. of ICML. ACM, 2006.","title":"Using inaccurate models in reinforcement learning","authors":"Pieter Abbeel, Morgan Quigley, Andrew Y Ng","year":"2006","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agrawal et al., 2016] Pulkit Agrawal, Ashvin Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. NIPS, 2016.","title":"Learning to poke by poking: Experiential learning of intuitive physics","authors":"Pulkit Agrawal, Ashvin Nair, Pieter Abbeel, Jitendra Malik, Sergey Levine","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bansal et al., 2017] Somil Bansal, Roberto Calandra, Ted Xiao, Sergey Levine, and Claire J Tomlin. Goal-driven dy- namics learning via bayesian optimization. In IEEE CDC, 2017.","title":"Goal-driven dynamics learning via bayesian optimization","authors":"Bansal","year":"2017","venue":"IEEE CDC","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bekris and Kavraki, 2008] K. E. Bekris and L. E. Kavraki. Informed and Probabilistically Complete Search for Mo- tion Planning under Differential Constraints. In STAIR, 2008.","title":"Informed and Probabilistically Complete Search for Motion Planning under Differential Constraints","authors":"K E Bekris, L E Kavraki","year":"2008","venue":"STAIR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bongard et al., 2006] Josh Bongard, Victor Zykov, and Hod Lipson. Resilient machines through continuous self- modeling. Science, 314(5802):1118-1121, 2006.","title":"Resilient machines through continuous selfmodeling","authors":"Josh Bongard, Victor Zykov, Hod Lipson","year":"2006","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv:1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Byravan and Fox, 2017] Arunkumar Byravan and Dieter Fox. Se3-nets: Learning rigid body motion using deep neural networks. In ICRA. IEEE, 2017.","title":"Se3-nets: Learning rigid body motion using deep neural networks","authors":"Arunkumar Byravan, Dieter Fox","year":"2017","venue":"ICRA. IEEE","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chatzilygeroudis et al., 2017] Konstantinos Chatzilyger- oudis, Roberto Rama, Rituraj Kaushik, Dorian Goepp, Vassilis Vassiliades, and Jean-Baptiste Mouret. Black- Box Data-efficient Policy Search for Robotics. In IROS, September 2017.","title":"Black-Box Data-efficient Policy Search for Robotics","authors":"Konstantinos Chatzilygeroudis, Roberto Rama, Rituraj Kaushik, Dorian Goepp, Vassilis Vassiliades, Jean-Baptiste Mouret","year":"2017","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cutler and How, 2015] Mark Cutler and Jonathan P How. Efficient reinforcement learning for robots using informa- tive simulated priors. In ICRA. IEEE, 2015.","title":"Efficient reinforcement learning for robots using informative simulated priors","authors":"Mark Cutler, Jonathan P How","year":"2015","venue":"ICRA. IEEE","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Deisenroth et al., 2011] M. Deisenroth, C. Rasmussen, and D. Fox. Learning to Control a Low-Cost Manipulator us- ing Data-Efficient Reinforcement Learning. In R:SS, 2011.","title":"Learning to Control a Low-Cost Manipulator using Data-Efficient Reinforcement Learning","authors":"M Deisenroth, C Rasmussen, D Fox","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dogar et al., 2012] Mehmet Dogar, Kaijen Hsiao, Matei Ciocarlie, and Siddhartha Srinivasa. Physics-Based Grasp Planning Through Clutter. In R:SS, July 2012.","title":"Physics-Based Grasp Planning Through Clutter","authors":"Mehmet Dogar, Kaijen Hsiao, Matei Ciocarlie, Siddhartha Srinivasa","year":"2012","venue":"R:SS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Duan et al., 2016] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In ICML, 2016.","title":"Benchmarking deep reinforcement learning for continuous control","authors":"Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Erez et al., 2015] Tom Erez, Yuval Tassa, and Emanuel Todorov. Simulation tools for model-based robotics: Com- parison of bullet, havok, mujoco, ODE and physx. In IEEE ICRA, 2015.","title":"Simulation tools for model-based robotics: Comparison of bullet, havok, mujoco, ODE and physx","authors":"Tom Erez, Yuval Tassa, Emanuel Todorov","year":"2015","venue":"IEEE ICRA","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"Finn and Levine, ] Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. ICRA 2017.","title":"Deep visual foresight for planning robot motion","authors":"Levine Finn, Chelsea Finn, Sergey Levine","year":"2017","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hansen, 2006] Nikolaus Hansen. The cma evolution strat- egy: a comparing review. Towards a new evolutionary computation, pages 75-102, 2006.","title":"The cma evolution strategy: a comparing review","authors":"Nikolaus Hansen, Hansen","year":"2006","venue":"Towards a new evolutionary computation","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hennig and Schuler, 2012] Philipp Hennig and Christian J. Schuler. Entropy Search for Information-Efficient Global Optimization. JMLR, 13:1809-1837, 2012.","title":"Entropy Search for Information-Efficient Global Optimization","authors":"Philipp Hennig, Christian J Schuler","year":"2012","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kober and Peters, 2009] Jens Kober and Jan R Peters. Pol- icy search for motor primitives in robotics. In NIPS, 2009.","title":"Policy search for motor primitives in robotics","authors":"Peters Kober, Jens Kober, Jan R Peters","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kober et al., 2013] J. Kober, J. Andrew (Drew) Bagnell, and J. Peters. Reinforcement learning in robotics: A survey. IJRR, 2013.","title":"Reinforcement learning in robotics: A survey","authors":"J Kober, J Andrew (drew) Bagnell, J Peters","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lee et al., 2017] Gilwoo Lee, Siddhartha S Srinivasa, and Matthew T Mason. Gp-ilqg: Data-driven robust optimal control for uncertain nonlinear dynamical systems. arXiv preprint arXiv:1705.05344, 2017.","title":"Gp-ilqg: Data-driven robust optimal control for uncertain nonlinear dynamical systems","authors":"Lee","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Littlefield et al., 2017] Z. Littlefield, D. Surovik, W. Wang, and K. E. Bekris. From Quasi-Static To Kinodynamic Planning For Spherical Tensegrity Locomotion. In ISRR, 2017.","title":"From Quasi-Static To Kinodynamic Planning For Spherical Tensegrity Locomotion","authors":"Z Littlefield, D Surovik, W Wang, K E Bekris","year":"2017","venue":"ISRR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lynch and Mason, 1996] K. M. Lynch and M. T. Mason. Stable pushing: Mechanics, control-lability, and planning. International Journal of Robotics Research, 18, 1996.","title":"Stable pushing: Mechanics, control-lability, and planning","authors":"K M Lynch, M T Mason","year":"1996","venue":"International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Marco et al., 2017] Alonso Marco, Felix Berkenkamp, Philipp Hennig, Angela P. Schoellig, Andreas Krause, Ste- fan Schaal, and Sebastian Trimpe. Virtual vs. real: Trading off simulations and physical experiments in reinforcement learning with bayesian optimization. In ICRA, 2017.","title":"Virtual vs. real: Trading off simulations and physical experiments in reinforcement learning with bayesian optimization","authors":"Alonso Marco, Felix Berkenkamp, Philipp Hennig, Angela P Schoellig, Andreas Krause, Stefan Schaal, Sebastian Trimpe","year":"2017","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Nguyen-Tuong and Peters, 2010] Duy Nguyen-Tuong and Jan Peters. Using model knowledge for learning inverse dynamics. In ICRA, pages 2677-2682, 2010.","title":"Using model knowledge for learning inverse dynamics","authors":"Nguyen-Tuong, Duy Peters, Jan Nguyen-Tuong, Peters","year":"2010","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pautrat et al., 2017] Rémi Pautrat, Konstantinos Chatzi- lygeroudis, and Jean-Baptiste Mouret. Bayesian optimiza- tion with automatic prior selection for data-efficient direct policy search. arXiv preprint arXiv:1709.06919, 2017.","title":"Bayesian optimization with automatic prior selection for data-efficient direct policy search","authors":"Rémi Pautrat, Konstantinos Chatzilygeroudis, Jean-Baptiste Mouret","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Peters et al., 2010] Jan Peters, Katharina Mülling, and Yasemin Altün. Relative entropy policy search. In AAAI, 2010.","title":"Relative entropy policy search","authors":"Peters","year":"2010","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Rasmussen and Williams, 2005] Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2005.","title":"Gaussian Processes for Machine Learning","authors":"Carl Edward Rasmussen, Christopher K I Williams","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Saveriano et al., 2017] Matteo Saveriano, Yuchao Yin, Pietro Falco, and Dongheui Lee. Data-efficient control policy search using residual dynamics learning. In Proc. of IROS, 2017.","title":"Data-efficient control policy search using residual dynamics learning","authors":"Matteo Saveriano, Yuchao Yin, Pietro Falco, Dongheui Lee","year":"2017","venue":"Proc. of IROS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Scholz et al., 2014] Jonathan Scholz, Martin Levihn, Charles L. Isbell, and David Wingate. A Physics-Based Model Prior for Object-Oriented MDPs. In ICML, 2014.","title":"A Physics-Based Model Prior for Object-Oriented MDPs","authors":"Jonathan Scholz, Martin Levihn, Charles L Isbell, David Wingate","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998.","title":"Introduction to Reinforcement Learning","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Swevers et al., 1997] Jan Swevers, Chris Ganseman, D Bil- gin Tukel, Joris De Schutter, and Hendrik Van Brussel. Optimal robot excitation and identification. IEEE TRO-A, 13(5):730-740, 1997.","title":"Optimal robot excitation and identification","authors":"Jan Swevers, Chris Ganseman, Bilgin Tukel, Joris De Schutter, Hendrik Van Brussel","year":"1997","venue":"IEEE TRO-A","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Tan et al., 2016] Jie Tan, Zhaoming Xie, Byron Boots, and C Karen Liu. Simulation-based design of dynamic con- trollers for humanoid balancing. In IROS. IEEE, 2016.","title":"Simulation-based design of dynamic controllers for humanoid balancing","authors":"Jie Tan, Zhaoming Xie, Byron Boots, Karen Liu","year":"2016","venue":"IROS. IEEE","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wu et al., 2015] Jiajun Wu, Ilker Yildirim, Joseph J Lim, Bill Freeman, and Josh Tenenbaum. Galileo: Perceiving physical object properties by integrating a physics engine with deep learning. In NIPS, pages 127-135, 2015.","title":"Galileo: Perceiving physical object properties by integrating a physics engine with deep learning","authors":"Jiajun Wu, Ilker Yildirim, Joseph J Lim, Bill Freeman, Josh Tenenbaum","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yu et al., 2017] Wenhao Yu, Jie Tan, C. Karen Liu, and Greg Turk. Preparing for the unknown: Learning a uni- versal policy with online system identification. In Pro- ceedings of R:SS, July 2017.","title":"Preparing for the unknown: Learning a universal policy with online system identification","authors":"Wenhao Yu, Jie Tan, C Karen Liu, Greg Turk","year":"2017","venue":"Proceedings of R:SS","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhou et al., 2016] Jiaji Zhou, Robert Paolini, J. Andrew Bagnell, and Matthew T. Mason. A convex polynomial force-motion model for planar sliding: Identification and application. In ICRA, pages 372-377, 2016.","title":"A convex polynomial force-motion model for planar sliding: Identification and application","authors":"Jiaji Zhou, Robert Paolini, J Andrew Bagnell, Matthew T Mason","year":"2016","venue":"ICRA","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}