{"type":"ScholarlyArticle","title":"Energy-Efficient Slithering Gait Exploration for a Snake-Like Robot Based on Reinforcement Learning","authors":[{"name":"Zhenshan Bing","slug":"zhenshan-bing","given":"Zhenshan","family":"Bing","affiliation":"Department of Computer Science , Technical University of Munich , Germany"},{"name":"Christian Lemke","slug":"christian-lemke","given":"Christian","family":"Lemke","affiliation":"Department of Computer Science , Ludwig Maximilian University of Munich , Germany"},{"name":"Zhuangyi Jiang","slug":"zhuangyi-jiang","given":"Zhuangyi","family":"Jiang","affiliation":"Department of Computer Science , Technical University of Munich , Germany"},{"name":"Kai Huang","slug":"kai-huang","given":"Kai","family":"Huang","affiliation":"School of Data and Computer Science , Sun Yat-sen University , China"},{"name":"Alois Knoll","slug":"alois-knoll","given":"Alois","family":"Knoll","affiliation":"Department of Computer Science , Technical University of Munich , Germany"}],"abstract":"Similar to their counterparts in nature, the flexible bodies of snake-like robots enhance their movement capability and adaptability in diverse environments. However, this flexibility corresponds to a complex control task involving highly redundant degrees of freedom, where traditional model-based methods usually fail to propel the robots energy-efficiently. In this work, we present a novel approach for designing an energy-efficient slithering gait for a snake-like robot using a model-free reinforcement learning (RL) algorithm. Specifically, we present an RL-based controller for generating locomotion gaits at a wide range of velocities, which is trained using the proximal policy optimization (PPO) algorithm. Meanwhile, a traditional parameterized gait controller is presented and the parameter sets are optimized using the grid search and Bayesian optimization algorithms for the purposes of reasonable comparisons. Based on the analysis of the simulation results, we demonstrate that this RL-based controller exhibits very natural and adaptive movements, which are also substantially more energy-efficient than the gaits generated by the parameterized controller. Videos are shown at https://videoviewsite.wixsite.com/rlsnake .","doi":"10.24963/ijcai.2019/785","url":"https://ijcai.org/proceedings/2019/785","pdf":"https://ijcai.org/proceedings/2019/0785.pdf","markdown":"https://ijcai.org/proceedings/2019/785.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/785","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Robotics","track":"Main track","number":785,"pages":{"first":5663,"last":5669},"keywords":["Robotics: Motion and Path Planning","Robotics: Learning in Robotics","Machine Learning: Reinforcement Learning","Robotics: Behavior and Control"],"videos":[],"references":[{"position":1,"raw":"[Bing et al., 2017] Zhenshan Bing, Long Cheng, Kai Huang, Zhuang Jiang, Guang Chen, Florian Röhrbein, and Alois Knoll. Towards autonomous locomotion: Slithering gait design of a snake-like robot for target observation and tracking. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2698-2703, Sep. 2017.","title":"Towards autonomous locomotion: Slithering gait design of a snake-like robot for target observation and tracking","authors":"Bing","year":"2017","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Calandra et al., 2014] Roberto Calandra, Nakul Gopalan, André Seyfarth, Jan Peters, and Marc Peter Deisenroth. Bayesian gait optimization for bipedal locomotion. In In- ternational Conference on Learning and Intelligent Opti- mization, pages 274-290. Springer, 2014.","title":"Bayesian gait optimization for bipedal locomotion","authors":"Roberto Calandra, Nakul Gopalan, André Seyfarth, Jan Peters, Marc Peter, Deisenroth","year":"2014","venue":"International Conference on Learning and Intelligent Optimization","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chernova and Veloso, 2004] Sonia Chernova and Manuela Veloso. An evolutionary approach to gait learning for four- legged robots. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2004., volume 3, pages 2562-2567. IEEE, 2004.","title":"An evolutionary approach to gait learning for fourlegged robots","authors":"Sonia Chernova, Manuela Veloso","year":"2004","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Crespi and Ijspeert, 2008] Alessandro Crespi and Auke J. Ijspeert. Online optimization of swimming and crawl- ing in an amphibious snake robot. IEEE Transactions on Robotics, 24(1):75-87, Feb 2008.","title":"Online optimization of swimming and crawling in an amphibious snake robot","authors":"Alessandro Crespi, Auke J Ijspeert","year":"2008","venue":"IEEE Transactions on Robotics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cully et al., 2015] Antoine Cully, Jeff Clune, Danesh Tara- pore, and Jean-Baptiste Mouret. Robots that can adapt like animals. Nature, 521(7553):503, 2015.","title":"Robots that can adapt like animals","authors":"Antoine Cully, Jeff Clune, Danesh Tarapore, Jean-Baptiste Mouret","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dowling, 1996] Kevin J Dowling. Limbless locomotion: learning to crawl with a snake robot. Unpublished Ph. D Thesis, The Robotics Institute, Carnegie Mellon Univer- sity, 5000, 1996.","title":"Limbless locomotion: learning to crawl with a snake robot","authors":"Kevin J Dowling","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dowling, 1997] Kevin Dowling. Power sources for small robots. Carnegie Mellon University, 1997.","title":"Power sources for small robots","authors":"Kevin Dowling","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gong et al., 2016] Chaohui Gong, Daniel I Goldman, and Howie Choset. Simplifying gait design via shape basis optimization. In Robotics: Science and Systems, 2016.","title":"Simplifying gait design via shape basis optimization","authors":"Gong","year":"2016","venue":"Robotics: Science and Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hirose, 1993] Shigeo Hirose. Biologically inspired robots: snake-like locomotors and manipulators, volume 1093. Oxford University Press Oxford, 1993.","title":"Biologically inspired robots: snake-like locomotors and manipulators","authors":"Shigeo Hirose","year":"1093","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hu et al., 2009] David L Hu, Jasmine Nirody, Terri Scott, and Michael J Shelley. The mechanics of slithering loco- motion. Proceedings of the National Academy of Sciences, 106(25):10081-10085, 2009.","title":"The mechanics of slithering locomotion","authors":"Jasmine David L Hu, Terri Nirody, Michael J Scott, Shelley","year":"2009","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim and Uther, 2003] Min Sub Kim and William Uther. Automatic gait optimisation for quadruped robots. In Aus- tralasian Conference on Robotics and Automation, pages 1-3. Citeseer, 2003.","title":"Min Sub Kim and William Uther. Automatic gait optimisation for quadruped robots","authors":"Uther Kim","year":"2003","venue":"Australasian Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kober et al., 2013] Jens Kober, J. Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238- 1274, 2013.","title":"Reinforcement learning in robotics: A survey","authors":"Jens Kober, J Andrew Bagnell, Jan Peters","year":"2013","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kohl and Stone, 2004] Nate Kohl and Peter Stone. Machine learning for fast quadrupedal locomotion. In AAAI, vol- ume 4, pages 611-616, 2004.","title":"Machine learning for fast quadrupedal locomotion","authors":"Stone Kohl, Nate Kohl, Peter Stone","year":"2004","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kormushev et al., 2018] Petar Kormushev, Barkan Ugurlu, Darwin Caldwell, and Nikos Tsagarakis. Learning to ex- ploit passive compliance for energy-efficient gait genera- tion on a compliant humanoid. Autonomous Robots, Feb 2018.","title":"Learning to exploit passive compliance for energy-efficient gait generation on a compliant humanoid","authors":"Petar Kormushev, Barkan Ugurlu, Darwin Caldwell, Nikos Tsagarakis","year":"2018","venue":"Autonomous Robots","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liljebäck et al., 2012] Pål Liljebäck, Kristin Ytterstad Pet- tersen, Øyvind Stavdahl, and Jan Tommy Gravdahl. Snake robots: modelling, mechatronics, and control. Springer Science & Business Media, 2012.","title":"Pål Liljebäck, Kristin Ytterstad Pettersen, Øyvind Stavdahl, and Jan Tommy Gravdahl. Snake robots: modelling, mechatronics, and control","authors":"Liljebäck","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lizotte et al., 2007] Daniel J Lizotte, Tao Wang, Michael H Bowling, and Dale Schuurmans. Automatic gait optimiza- tion with gaussian process regression. In IJCAI, volume 7, pages 944-949, 2007.","title":"Automatic gait optimization with gaussian process regression","authors":"Tao Daniel J Lizotte, Michael H Wang, Dale Bowling, Schuurmans","year":"2007","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2007/152"},{"position":17,"raw":"[Long et al., 2018] Pinxin Long, Tingxiang Fanl, Xinyi Liao, Wenxi Liu, Hao Zhang, and Jia Pan. Towards op- timally decentralized multi-robot collision avoidance via deep reinforcement learning. In IEEE International Con- ference on Robotics and Automation (ICRA), pages 6252- 6259, May 2018.","title":"Towards optimally decentralized multi-robot collision avoidance via deep reinforcement learning","authors":"Long","year":"2018","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ma, 1999] Shugen Ma. Analysis of snake movement forms for realization of snake-like robots. In Proceedings 1999 IEEE International Conference on Robotics and Automa- tion (Cat. No.99CH36288C), volume 4, pages 3007-3013 vol.4, May 1999.","title":"Analysis of snake movement forms for realization of snake-like robots","authors":"Ma ;, Shugen Ma","year":"1999","venue":"Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No.99CH36288C)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Melo and Paez, 2014] Kamilo Melo and Laura Paez. Exper- imental determination of control parameter intervals for repeatable gaits in modular snake robots. In 2014 IEEE International Symposium on Safety, Security, and Rescue Robotics (2014), pages 1-7, Oct 2014.","title":"Experimental determination of control parameter intervals for repeatable gaits in modular snake robots","authors":"Kamilo Melo, Laura Paez","year":"2014","venue":"IEEE International Symposium on Safety, Security, and Rescue Robotics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Peng et al., 2017] Xue Bin Peng, Glen Berseth, KangKang Yin, and Michiel Van De Panne. Deeploco: Dynamic loco- motion skills using hierarchical deep reinforcement learn- ing. ACM Transactions on Graphics, 36(4):41, 2017.","title":"Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning","authors":"Xue Bin Peng, Glen Berseth, Kangkang Yin, Michiel Van De Panne","year":"2017","venue":"ACM Transactions on Graphics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rajeswaran et al., 2017] Aravind Rajeswaran, Vikash Ku- mar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint 1709.10087, 2017.","title":"Learning complex dexterous manipulation with deep reinforcement learning and demonstrations","authors":"Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, Sergey Levine","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Saito et al., 2002] Masashi Saito, Masakazu Fukaya, and Tetsuya Iwasaki. Modeling, analysis, and synthesis of ser- pentine locomotion with a multilink robotic snake. IEEE control systems magazine, 22(1):64-81, 2002.","title":"Modeling, analysis, and synthesis of serpentine locomotion with a multilink robotic snake","authors":"Saito","year":"2002","venue":"IEEE control systems magazine","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tesch et al., 2009] Matthew Tesch, Kevin Lipkin, Isaac Brown, Ross Hatton, Aaron Peck, Justine Rembisz, and Howie Choset. Parameterized and scripted gaits for mod- ular snake robots. Advanced Robotics, 23(9):1131-1158, 2009.","title":"Parameterized and scripted gaits for modular snake robots","authors":"Matthew Tesch, Kevin Lipkin, Isaac Brown, Ross Hatton, Aaron Peck, Justine Rembisz, Howie Choset","year":"2009","venue":"Advanced Robotics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tesch et al., 2011] Matthew Tesch, Jeff Schneider, and Howie Choset. Using response surfaces and expected im- provement to optimize snake robot gait parameters. In 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1069-1074, Sep. 2011.","title":"Using response surfaces and expected improvement to optimize snake robot gait parameters","authors":"Matthew Tesch, Jeff Schneider, Howie Choset","year":"2011","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. Mujoco: A physics engine for model-based con- trol. In 2012 IEEE/RSJ International Conference on Intel- ligent Robots and Systems, pages 5026-5033, Oct 2012.","title":"Mujoco: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tucker, 1975] Vance A. Tucker. The energetic cost of moving about: Walking and running are extremely in- efficient forms of locomotion. much greater efficiency is achieved by birds, fish-and bicyclists. American Scien- tist, 63(4):413-419, 1975.","title":"The energetic cost of moving about: Walking and running are extremely inefficient forms of locomotion. much greater efficiency is achieved by birds, fish-and bicyclists","authors":"Vance A Tucker","year":"1975","venue":"American Scientist","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yu et al., 2018] Wenhao Yu, Greg Turk, and C Karen Liu. Learning symmetric and low-energy locomotion. ACM Transactions on Graphics (TOG), 37(4):144, 2018.","title":"Learning symmetric and low-energy locomotion","authors":"Wenhao Yu, Greg Turk, Karen Liu","year":"2018","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}