{"type":"ScholarlyArticle","title":"Reinforcement Learning for Athletic Intelligence: Lessons from the 1st “AI Olympics with RealAIGym” Competition","authors":[{"name":"Felix Wiebe","slug":"felix-wiebe","given":"Felix","family":"Wiebe","affiliation":"Robotics Innovation Center , German Research Center for Artificial Intelligence (DFKI) , Germany"},{"name":"Niccolò Turcato","slug":"niccolo-turcato","given":"Niccolò","family":"Turcato","affiliation":"Department of Information Engineering , University of Padova , Italy"},{"name":"Alberto Dalla Libera","slug":"alberto-dalla-libera","given":"Alberto","family":"Dalla Libera","affiliation":"Department of Information Engineering , University of Padova , Italy"},{"name":"Chi Zhang","slug":"chi-zhang","given":"Chi","family":"Zhang","affiliation":"Technical University of Munich , Germany"},{"name":"Theo Vincent","slug":"theo-vincent","given":"Theo","family":"Vincent","affiliation":"Systems AI for Robot Learning , German Research Center for Artificial Intelligence (DFKI) , Germany"},{"name":"Shubham Vyas","slug":"shubham-vyas","given":"Shubham","family":"Vyas","affiliation":"Robotics Innovation Center , German Research Center for Artificial Intelligence (DFKI) , Germany"},{"name":"Giulio Giacomuzzo","slug":"giulio-giacomuzzo","given":"Giulio","family":"Giacomuzzo","affiliation":"Department of Information Engineering , University of Padova , Italy"},{"name":"Ruggero Carli","slug":"ruggero-carli","given":"Ruggero","family":"Carli","affiliation":"Department of Information Engineering , University of Padova , Italy"},{"name":"Diego Romeres","slug":"diego-romeres","given":"Diego","family":"Romeres","affiliation":"Mitsubishi Electric Research Lab (MERL) , USA"},{"name":"Akhil Sathuluri","slug":"akhil-sathuluri","given":"Akhil","family":"Sathuluri","affiliation":"Technical University of Munich , Germany"},{"name":"Markus Zimmermann","slug":"markus-zimmermann","given":"Markus","family":"Zimmermann","affiliation":"Technical University of Munich , Germany"},{"name":"Boris Belousov","slug":"boris-belousov","given":"Boris","family":"Belousov","affiliation":"Systems AI for Robot Learning , German Research Center for Artificial Intelligence (DFKI) , Germany"},{"name":"Jan Peters","slug":"jan-peters","given":"Jan","family":"Peters","affiliation":"Systems AI for Robot Learning , German Research Center for Artificial Intelligence (DFKI) , Germany; Technical University of Darmstadt , Germany; Centre for Cognitive Science , Germany; Hessian.AI , Germany"},{"name":"Frank Kirchner","slug":"frank-kirchner","given":"Frank","family":"Kirchner","affiliation":"Robotics Innovation Center , German Research Center for Artificial Intelligence (DFKI) , Germany; University of Bremen , Germany"},{"name":"Shivesh Kumar","slug":"shivesh-kumar","given":"Shivesh","family":"Kumar","affiliation":"Robotics Innovation Center , German Research Center for Artificial Intelligence (DFKI) , Germany; Chalmers University of Technology , Sweden"}],"abstract":"As artificial intelligence gains new capabilities, it\r\nbecomes important to evaluate it on real-world\r\ntasks. In particular, the fields of robotics and reinforcement learning (RL) are lacking in standardized benchmarking tasks on real hardware. To\r\nfacilitate reproducibility and stimulate algorithmic\r\nadvancements, we held an AI Olympics competition at IJCAI 2023 conference based on the double\r\npendulum system in the RealAIGym project where\r\nthe participants were asked to develop a controller\r\nfor the swing up and stabilization task. This paper\r\npresents the methods and results from the top participating teams and provides insights into the real-world performance of RL algorithms with respect\r\nto a baseline time-varying LQR controller.","doi":"10.24963/ijcai.2024/1043","url":"https://ijcai.org/proceedings/2024/1043","pdf":"https://ijcai.org/proceedings/2024/1043.pdf","markdown":"https://ijcai.org/proceedings/2024/1043.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/1043","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Demo Track","track":"Demo Track","number":1043,"pages":{"first":8833,"last":8837},"keywords":["Robotics: ROB: Learning in robotics","Machine Learning: ML: Deep reinforcement learning","Robotics: ROB: Motion and path planning","Robotics: ROB: Behavior and control"],"videos":[],"references":[{"position":1,"raw":"[Al-Hafez et al., 2023] F. Al-Hafez, G. Zhao, J. Peters, and D. Tateo. Locomujoco: A comprehensive imitation learn- ing benchmark for locomotion. In Robot Learning Work- shop, Conference on Neural Information Processing Sys- tems (NeurIPS), 2023.","title":"Locomujoco: A comprehensive imitation learning benchmark for locomotion","authors":"F Al-Hafez, G Zhao, J Peters, D Tateo","year":"2023","venue":"Robot Learning Workshop, Conference on Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amadio et al., 2022] Fabio Amadio, Alberto Dalla Libera, Riccardo Antonello, Daniel Nikovski, Ruggero Carli, and Diego Romeres. Model-based policy search using monte carlo gradient estimation with real systems application. IEEE Transactions on Robotics, 38(6):3879-3898, 2022.","title":"Model-based policy search using monte carlo gradient estimation with real systems application","authors":"Fabio Amadio, Alberto Dalla Libera, Riccardo Antonello, Daniel Nikovski, Ruggero Carli, Diego Romeres","year":"2022","venue":"IEEE Transactions on Robotics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Amadio et al., 2023] Fabio Amadio, Alberto Dalla Libera, Daniel Nikovski, Ruggero Carli, and Diego Romeres. Learning control from raw position measurements. In 2023 American Control Conference (ACC), pages 2171- 2178. IEEE, 2023.","title":"Learning control from raw position measurements","authors":"Fabio Amadio, Alberto Dalla Libera, Daniel Nikovski, Ruggero Carli, Diego Romeres","year":"2023","venue":"2023 American Control Conference (ACC)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016. Deng et al., 2009]","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Feng et al., 2023] Gilbert Feng, Hongbo Zhang, Zhongyu Li, Xue Bin Peng, Bhuvan Basireddy, Linzhu Yue, Zhi- tao Song, Lizhi Yang, Yunhui Liu, Koushil Sreenath, et al. Genloco: Generalized locomotion controllers for quadrupedal robots. In Conference on Robot Learning, pages 1893-1903. PMLR, 2023.","title":"Genloco: Generalized locomotion controllers for quadrupedal robots","authors":"Gilbert Feng, Hongbo Zhang, Zhongyu Li, Xue Bin Peng, Bhuvan Basireddy, Linzhu Yue, Zhitao Song, Lizhi Yang, Yunhui Liu, Koushil Sreenath","year":"2023","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Funk et al., 2021] Niklas Funk, Charles Schaff, Rishabh Madan, Takuma Yoneda, Julen Urain De Jesus, Joe Wat- son, Ethan K Gordon, Felix Widmaier, Stefan Bauer, Sid- dhartha S Srinivasa, et al. Benchmarking structured poli- cies and policy optimization for real-world dexterous ob- ject manipulation. IEEE Robotics and Automation Letters, 7(1):478-485, 2021.","title":"Benchmarking structured policies and policy optimization for real-world dexterous object manipulation","authors":"Niklas Funk, Charles Schaff, Rishabh Madan, Takuma Yoneda, Julen Urain De, Joe Jesus, Ethan K Watson, Felix Gordon, Stefan Widmaier, Siddhartha S Bauer, Srinivasa","year":"2021","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Grimminger et al., 2020] Felix Grimminger, Avadesh Meduri, Majid Khadiv, Julian Viereck, Manuel Wüthrich, Maximilien Naveau, Vincent Berenz, Steve Heim, Felix Widmaier, Thomas Flayols, et al. An open torque-controlled modular robot architecture for legged locomotion research. IEEE Robotics and Automation Letters, 5(2):3650-3657, 2020.","title":"An open torque-controlled modular robot architecture for legged locomotion research","authors":"Felix Grimminger, Avadesh Meduri, Majid Khadiv, Julian Viereck, Manuel Wüthrich, Maximilien Naveau, Vincent Berenz, Steve Heim, Felix Widmaier, Thomas Flayols","year":"2020","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gürtler et al., 2023] Nico Gürtler, Sebastian Blaes, Pavel Kolev, Felix Widmaier, Manuel Wuthrich, Stefan Bauer, Bernhard Schölkopf, and Georg Martius. Benchmarking offline reinforcement learning on real-robot hardware. In The Eleventh International Conference on Learning Rep- resentations, 2023.","title":"Benchmarking offline reinforcement learning on real-robot hardware","authors":"Nico Gürtler, Sebastian Blaes, Pavel Kolev, Felix Widmaier, Manuel Wuthrich, Stefan Bauer, Bernhard Schölkopf, Georg Martius","year":"2023","venue":"The Eleventh International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on ma- chine learning, pages 1861-1870. PMLR, 2018.","title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","authors":"Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[James et al., 2020] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019-3026, 2020.","title":"Rlbench: The robot learning benchmark & learning environment","authors":"Stephen James, Zicong Ma, David Rovick Arrojo, Andrew J Davison","year":"2020","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Javadi et al., 2023] Mahdi Javadi, Daniel Harnack, Paula Stocco, Shivesh Kumar, Shubham Vyas, Daniel Pizzutilo, and Frank Kirchner. Acromonk: A minimalist underac- tuated brachiating robot. IEEE Robotics and Automation Letters, 8(6):3637-3644, 2023.","title":"Acromonk: A minimalist underactuated brachiating robot","authors":"Javadi","year":"2023","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li. and Todorov., 2004] Weiwei Li. and Emanuel Todorov. Iterative linear quadratic regulator design for nonlinear bi- ological movement systems. In Proceedings of the First International Conference on Informatics in Control, Au- tomation and Robotics -Volume 1: ICINCO, pages 222- 229. INSTICC, SciTePress, 2004.","title":"Iterative linear quadratic regulator design for nonlinear biological movement systems","authors":"Li, Todorov, Li Weiwei, Emanuel Todorov","year":"2004","venue":"Proceedings of the First International Conference on Informatics in Control, Automation and Robotics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Maywald et al., 2022] Lasse Jenning Maywald, Felix Wiebe, Shivesh Kumar, Mahdi Javadi, and Frank Kirch- ner. Co-optimization of acrobot design and controller for increased certifiable stability. In 2022 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), pages 2636-2641. IEEE, 2022.","title":"Co-optimization of acrobot design and controller for increased certifiable stability","authors":"Maywald","year":"2022","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mittal et al., 2023] Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, et al. Orbit: A unified simulation framework for interactive robot learn- ing environments. IEEE Robotics and Automation Letters, 2023.","title":"Orbit: A unified simulation framework for interactive robot learning environments","authors":"Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar","year":"2023","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al. Human-level control through deep re- inforcement learning. nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver","year":"2015","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Soni et al., 2023] Raghav Soni, Daniel Harnack, Hauke Is- ermann, Sotaro Fushimi, Shivesh Kumar, and Frank Kirchner. End-to-end reinforcement learning for torque based variable height hopping. In 2023 IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems (IROS), pages 7531-7538, 2023.","title":"End-to-end reinforcement learning for torque based variable height hopping","authors":"Raghav Soni, Daniel Harnack, Hauke Isermann, Sotaro Fushimi, Shivesh Kumar, Frank Kirchner","year":"2023","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tedrake, 2022] Russ Tedrake. Underactuated Robotics. 2022.","title":"Underactuated Robotics","authors":"Russ Tedrake","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Turcato et al., 2023] Niccolo' Turcato, Alberto Dalla Lib- era, Giulio Giacomuzzo, and Ruggero Carli. Teaching a robot to toss arbitrary objects with model-based reinforce- ment learning. In 2023 9th International Conference on Control, Decision and Information Technologies (CoDIT), pages 1126-1131, 2023.","title":"Niccolo' Turcato, Alberto Dalla Libera, Giulio Giacomuzzo, and Ruggero Carli. Teaching a robot to toss arbitrary objects with model-based reinforcement learning","authors":"Turcato","year":"2023","venue":"2023 9th International Conference on Control, Decision and Information Technologies (CoDIT)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wiebe et al., 2022a] Felix Wiebe, Jonathan Babel, Shivesh Kumar, Shubham Vyas, Daniel Harnack, Melya Boukhed- dimi, Mihaela Popescu, and Frank Kirchner. Torque- limited simple pendulum: A toolkit for getting familiar with control algorithms in underactuated robotics. Jour- nal of Open Source Software, 7(74):3884, 2022.","title":"Torquelimited simple pendulum: A toolkit for getting familiar with control algorithms in underactuated robotics","authors":"Wiebe","year":"2022","venue":"Journal of Open Source Software","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wiebe et al., 2022b] Felix Wiebe, Shubham Vyas, Lasse Jenning Maywald, Shivesh Kumar, and Frank Kirchner. RealAIGym: Education and Research Platform for Studying Athletic Intelligence. In Proceedings of Robotics Science and Systems Workshop Mind the Gap: Opportunities and Challenges in the Transition Between Research and Industry, New York, July 2022.","title":"RealAIGym: Education and Research Platform for Studying Athletic Intelligence","authors":"Wiebe","year":"2022","venue":"Proceedings of Robotics Science and Systems Workshop Mind the Gap: Opportunities and Challenges in the Transition Between Research and Industry","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wiebe et al., 2023] Felix Wiebe, Shivesh Kumar, Lasse J. Shala, Shubham Vyas, Mahdi Javadi, and Frank Kirch- ner. Open source dual-purpose acrobot and pendubot plat- form: Benchmarking control algorithms for underactuated robotics. IEEE Robotics Automation Magazine, pages 2- 13, 2023.","title":"Open source dual-purpose acrobot and pendubot platform: Benchmarking control algorithms for underactuated robotics","authors":"Felix Wiebe, Shivesh Kumar, Lasse J Shala, Shubham Vyas, Mahdi Javadi, Frank Kirchner","year":"2023","venue":"IEEE Robotics Automation Magazine","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}