---
title: "Reinforcement Learning for Athletic Intelligence: Lessons from the 1st “AI Olympics with RealAIGym” Competition"
authors:
  - "Felix Wiebe"
  - "Niccolò Turcato"
  - "Alberto Dalla Libera"
  - "Chi Zhang"
  - "Theo Vincent"
  - "Shubham Vyas"
  - "Giulio Giacomuzzo"
  - "Ruggero Carli"
  - "Diego Romeres"
  - "Akhil Sathuluri"
  - "Markus Zimmermann"
  - "Boris Belousov"
  - "Jan Peters"
  - "Frank Kirchner"
  - "Shivesh Kumar"
doi: "10.24963/ijcai.2024/1043"
url: "https://ijcai.org/proceedings/2024/1043"
pdf: "https://ijcai.org/proceedings/2024/1043.pdf"
venue: "Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence"
year: 2024
pages: "8833-8837"
track: "Demo Track"
keywords:
  - "Robotics: ROB: Learning in robotics"
  - "Machine Learning: ML: Deep reinforcement learning"
  - "Robotics: ROB: Motion and path planning"
  - "Robotics: ROB: Behavior and control"
---

# Reinforcement Learning for Athletic Intelligence: Lessons from the 1st “AI Olympics with RealAIGym” Competition

Felix Wiebe, Niccolò Turcato, Alberto Dalla Libera, Chi Zhang, Theo Vincent, Shubham Vyas, Giulio Giacomuzzo, Ruggero Carli, Diego Romeres, Akhil Sathuluri, Markus Zimmermann, Boris Belousov, Jan Peters, Frank Kirchner, Shivesh Kumar

*Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence*, pages 8833-8837. DOI: [10.24963/ijcai.2024/1043](https://doi.org/10.24963/ijcai.2024/1043)

## Abstract

As artificial intelligence gains new capabilities, it
becomes important to evaluate it on real-world
tasks. In particular, the fields of robotics and reinforcement learning (RL) are lacking in standardized benchmarking tasks on real hardware. To
facilitate reproducibility and stimulate algorithmic
advancements, we held an AI Olympics competition at IJCAI 2023 conference based on the double
pendulum system in the RealAIGym project where
the participants were asked to develop a controller
for the swing up and stabilization task. This paper
presents the methods and results from the top participating teams and provides insights into the real-world performance of RL algorithms with respect
to a baseline time-varying LQR controller.

## Full text

## 1 Introduction

Benchmarks and competitions have proven extremely successful in computer vision and machine learning for driving algorithmic innovation [Deng et al., 2009]. In robotics, a number of simulated benchmarking environments gained popularity for evaluating learning algorithms [Brockman et al., 2016; James et al., 2020; Mittal et al., 2023; Al-Hafez et al., 2023]. However, only a handful of real-robot standardized environments are reliable, easy to simulate, and cheap to reproduce. Importantly, different environments have different focus, e.g., locomotion [Grimminger et al., 2020; Feng et al., 2023], finger-based manipulation [Funk et al., 2021; G¨urtler et al., 2023], etc. Our RealAIGym project [Wiebe et al., 2022b] 1 offers a suite of canonical underactuated systems

[1 https://dfki-ric-underactuated-lab.github.io/real-ai-gym/](https://dfki-ric-underactuated-lab.github.io/real-ai-gym/)

Figure 1: Left: Double pendulum hardware, top right: Acrobot, bottom right: Pendubot with active motors colored in red.

![Image](https://ijcai.org/proceedings/2024/fulltext/1043/image_000000_df3f0e8b30a9cf151560d88695ba1cde9316067b50a21f52d6f5b7be9f2afb06.png)

![Image](https://ijcai.org/proceedings/2024/fulltext/1043/image_000001_aaefbe22ed7461020b808fe0dbf131abc0c9249a714e59f2655096f1faa624c5.png)

(simple pendulum [Wiebe et al., 2022a], AcroMonk [Javadi et al., 2023], hopper [Soni et al., 2023]) for benchmarking learning and control algorithms for athletic intelligence on real hardware with a user-friendly Python API. By opensourcing both the software and the hardware, we aim to establish a real-world equivalent to the well-known OpenAI Gym [Brockman et al., 2016].

The "AI Olympics with RealAIGym" competition 23 is based on a Double Pendulum system (see Fig. 1) in the RealAIGym project. The Double Pendulum [Wiebe et al., 2023] can operate in two modes: Pendubot when the actuator in the shoulder joint is active but the elbow is passive, and Acrobot with passive shoulder and active elbow. The challenge consists in performing a swing-up and stabilization from the free-hanging position to the upright position. For scoring, the Acrobot and Pendubot tasks are treated as two separate tracks.

[2 https://ijcai-23.dfki-bremen.de/competitions/ai olympics/](https://ijcai-23.dfki-bremen.de/competitions/ai_olympics/)

[3 https://youtu.be/eYDH1v1FqF8](https://youtu.be/eYDH1v1FqF8)

The competition was held in two phases. In the simulation phase, the participants were provided model parameters, a simulation environment, and the scoring metric to rank the behaviors. The submitted controllers were evaluated with a performance score and a robustness score. The performance score, calculated in the simulation and in the hardware phase, evaluates how well the swing-up is performed and is given by a weighted sum of swing-up time, energy cost, maximum torque, integrated torque, torque cost, torque smoothness and velocity cost and is multiplied with the swing-up success rate. The robustness score evaluates the sensitivity of the controllers to model inaccuracies, measurement noise, torque noise, torque response, and time delay.

The teams who achieved high simulation and robustness scores were advanced to the hardware phase, where they were given up to 20h remote-access to a real double pendulum system to train and test their controllers. The final controllers were evaluated on the same PC using the performance score over 10 consecutive executions to provide the final ranking.

## 2 Algorithms

In this section, three families of RL methods that participated in the competition are presented, together with a baseline optimal control approach for comparison.

## 2.1 Model-based RL: MCPILCO

MC-PILCO (Monte Carlo - Probabilistic Inference for Learning COntrol) [Amadio et al., 2022] is a model-based policy gradient algorithm that relies on Gaussian Processes (GPs) to learn the system dynamics from data. Let xt and ut be, respectively, the state and input of the system at step t. A cost function c(xt) encodes the task to be accomplished. A policy πθ : x → u that depends on the parameters θ selects the inputs applied to the system. The objective is to find policy parameters θ∗ that minimize the cumulative expected cost:

<!-- formula-not-decoded -->

MC-PILCO performs several successive attempts to solve the desired task, also called trials. Each trial consists of three main phases: (i) model learning, (ii) policy update, and (iii) policy execution.

In the model learning step, previous experience is used to derive a one-step-ahead stochastic model of the system dynamics using Gaussian Process Regression. The policy update step aims at minimizing the cost in eq. (1) w.r.t. the policy parameters θ. The expectation in eq. (1) is approximated based on the GP dynamics derived in (i) and Monte Carlo simulation. Finally, in the last step, the current optimized policy is applied to the system and the collected samples are stored to update the model in the next trials. Examples of MC-PILCO applications have been reported in [Amadio et al., 2023] and [Turcato et al., 2023].

## 2.2 Model-free RL, Actor-Critic Methods: SAC

A potential solution in the field of model-free reinforcement learning involves combining the Soft Actor Critic

(SAC) [Haarnoja et al., 2018] algorithm with the Linear Quadratic Regulator (LQR). The SAC algorithm is utilized to train a reinforcement learning (RL) agent for swing-up tasks, while the LQR controller is responsible for stabilizing the system at its highest position. The transition between the SAC and LQR occurs on entering the Region of Attraction (RoA) of the LQR controller, which is numerically approximated [Maywald et al., 2022]. A three-stage reward function is designed to steer the agent into the LQR controller's RoA, facilitating the switch. The initial stage involves a quadratic function that penalizes state errors and torque usage. Upon the end-effector reaching a specific threshold line, an additional reward is introduced. Furthermore, a substantial reward is granted when the state is within the LQR's RoA.

## 2.3 Model-free RL, Value-basedMethods: DQN

Deep Q-Network (DQN) [Mnih et al., 2015] is the seminal algorithm in the field of Deep RL, which has demonstrated that end-to-end controllers with high-dimensional observation spaces can be learned by trial and error. DQN is a valuebased method that learns the optimal action-value function Q(x, u) such that the optimal action can be obtained by optimization over u. To learn Q, DQN iteratively applies the Bellman operator, which is a contraction mapping and hence its successive application leads to a fixed point. The theory guarantees that the fixed point of the Bellman operator is the optimal action-value function corresponding to the optimal policy, yielding the maximal reward / minimal cost.

DQN is known to work well in discrete action spaces. Since the action space of Pendubot is 1-dimensional, we consider discretizing it into 9 bins. A logarithmic discretization centered around zero yields better performance in practice, because the agent requires more actions around zero to control the pendulum close to the upright position. The same rewards function as for the SAC algorithm is used for training DQN, and the LQR controller provided by RealAIGym is employed to stabilize the Pendubot when it enters the RoA.

## 2.4 Model-based Optimal Control: TVLQR

An optimal control (OC) controller serves as a baseline for the learning-based controllers in the competition. The controller utilizes three versions of the linear quadratic regulator (LQR). The basis is a trajectory (udt , xdt ) computed with the trajectory optimization technique iterative LQR (iLQR) [Li. and Todorov., 2004] and a time-varying LQR (TVLQR) [Tedrake, 2022] is used to stabilize the system towards that trajectory during the execution. This results in the control law at time t:

<!-- formula-not-decoded -->

with the linear feedback matrix Kt. The controller switches to a LQR control once its RoA is entered.

## 3 Results

The results of the competition are listed in Table 1 and visualized in Fig. 2. All data, figures and videos of the experiments can be found online 4 . Swing-up attempts are evaluated based on the performance score (simulation and hardware) and the robustness score (simulation) as introduced in section 1.

[4 https://dfki-ric-underactuated-lab.github.io/real ai gym leaderboard/](https://dfki-ric-underactuated-lab.github.io/real_ai_gym_leaderboard/)

Figure 2: Results for all controllers on Acrobot and Pendubot in the three competition categories. The vertical axis is scaled in the units of the corresponding criterion, e.g. seconds (s), Newton meters (Nm), successful percentage (%), etc. Note that in the 'Simulation' and 'Hardware' categories smaller values are better (except for successful attempts and the scores) while in the 'Robustness' category larger values are better. Bars for controllers which did not compete in a category are left blank.

![Image](https://ijcai.org/proceedings/2024/fulltext/1043/image_000002_e305d3fd0615e03869d964409f3d9229ce5b7e7a47447aaf96b02fdc6eeee631.png)

## 3.1 Acrobot

Two submitted controllers (MC-PILCO and SAC) passed the simulation stage for the Acrobot swingup problem. MCPILCO achieved a high performance score (0.869), beating the optimal control baseline (0.8) and a mediocre robustness score (0.595). SAC's performance score is a little lower (0.811) while the robustness score is significantly higher (0.82) and closer to the OC baseline (0.861). Despite the high performance and robustness scores, the SAC team did not succeed in transferring the controller to the real hardware. A successful swing up was not achieved. The MCPILCO team, on the other hand, retrained their policy on the real hardware and demonstrated a 100% success rate in the 10 evaluation trials resulting in a hardware score of 0.817 which is only slightly lower than the OC baseline (0.821). MCPILCO's mediocre robustness score indicates that a policy trained with MC-PILCO has limits when exposed to dynamics which have not been experienced during training. However, the notable performance of MC-PILCO on the real hardware shows its applicability even for real dynamics when directly trained under these conditions.

## 3.2 Pendubot

On the Pendubot, MC-PILCO, SAC and DQN passed the simulation stage of the competition. MC-PILCO scored the highest in the simulation (MC-PILCO: 0.891, SAC: 0.876, DQN: 0.815, TVLQR: 0.827) while SAC showed the highest robustness score only secondary to TVLQR (MC-PILCO: 0.852, SAC: 0.986, DQN: 0.226, TVLQR: 0.950). MCPILCO again was retrained on the real hardware and achieved a 100% success rate with a hardware score of 0.839. SAC achieved a 40% success rate and a hardware score of 0.298. Note that the SAC policy was not retrained on the hardware but only in a simulation with added noise. DQN could not be successfully transferred to the real system. The OC baseline has an 80% success rate and a 0.547 score.

Table 1: Final scores of the submitted controllers and the baseline for simulation, robustness and hardware tests.

|                  |   MC-PILCO | SAC   | DQN   |   TVLQR |
|------------------|------------|-------|-------|---------|
| Sim.             |      0.869 | 0.811 | -     |     0.8 |
| Acrobot Robust.  |      0.595 | 0.820 | -     |   0.861 |
| HW               |      0.817 | -     | -     |   0.821 |
| Sim.             |      0.891 | 0.876 | 0.815 |   0.827 |
| Pendubot Robust. |      0.852 | 0.896 | 0.226 |   0.950 |
| HW               |      0.839 | 0.298 | -     |   0.547 |

## 4 Conclusion

We introduced a competition to evaluate the performance and robustness of learning-based vs. optimal control approaches. From the first competition conduction, we conclude that learning-based approaches provide a strong alternative to model-based optimal control, being able to learn from a few samples on the real system. The model-based MC-PILCO outperformed the model-free methods SAC and DQN, showing a more reliable performance on the real Pendubot system than the baseline TVLQR method. We hope this challenge inspires further research into control algorithms for athletic intelligence; in particular, tackling generalization to different initial conditions, robustness to perturbations and handling of unmodelled dynamic effects.

## Ethical Statement

There are no ethical issues.

## Acknowledgments

The competition organizers at DFKI-RIC (FW, SV, FK, SK) acknowledge the support of M-RoCK project funded by the German Aerospace Center (DLR) with federal funds (Grant Number: FKZ 01IW21002) from the Federal Ministry of Education and Research (BMBF) and is additionally supported with project funds from the federal state of Bremen for setting up the Underactuated Robotics Lab (Grant Number: 201342-04-1/2023-4-1). TV, BB, and JP acknowledge the grant "Einrichtung eines Labors des Deutschen Forschungszentrum f¨ur K¨unstliche Intelligenz (DFKI) an der Technischen Universit¨at Darmstadt" of the Hessisches Ministerium f¨ur Wissenschaft und Kunst. This research was supported by Research Clusters "The Adaptive Mind" and "Third Wave of AI", funded by the Excellence Program of the Hessian Ministry of Higher Education, Science, Research and the Arts. Parts of the calculations for this research were conducted on the Lichtenberg high-performance computer of the TU Darmstadt. TV and BB also thank Daniel Palenicek and Tim Schneider for their support while performing the experiments on the IAS group's computing cluster at TU Darmstadt.

## Contribution Statement

NT, ADL, GG, RC and DR constitute the winning team and contributed to the MC-PILCO controller. CZ, AS and MZ formed the runner-up team which implemented the SAC+LQR controller. TV and BB provided the DQN implementation. FW, SV and SK implemented the baseline controller and benchmarking software. SK and BB conceptualized and executed the competition at the IJCAI venue. All authors contributed to the preparation of this manuscript.

## References

1. [Al-Hafez et al., 2023] F. Al-Hafez, G. Zhao, J. Peters, and D. Tateo. Locomujoco: A comprehensive imitation learn- ing benchmark for locomotion. In Robot Learning Work- shop, Conference on Neural Information Processing Sys- tems (NeurIPS), 2023.
2. [Amadio et al., 2022] Fabio Amadio, Alberto Dalla Libera, Riccardo Antonello, Daniel Nikovski, Ruggero Carli, and Diego Romeres. Model-based policy search using monte carlo gradient estimation with real systems application. IEEE Transactions on Robotics, 38(6):3879-3898, 2022.
3. [Amadio et al., 2023] Fabio Amadio, Alberto Dalla Libera, Daniel Nikovski, Ruggero Carli, and Diego Romeres. Learning control from raw position measurements. In 2023 American Control Conference (ACC), pages 2171- 2178. IEEE, 2023.
4. [Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016. Deng et al., 2009]
5. Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.
6. [Feng et al., 2023] Gilbert Feng, Hongbo Zhang, Zhongyu Li, Xue Bin Peng, Bhuvan Basireddy, Linzhu Yue, Zhi- tao Song, Lizhi Yang, Yunhui Liu, Koushil Sreenath, et al. Genloco: Generalized locomotion controllers for quadrupedal robots. In Conference on Robot Learning, pages 1893-1903. PMLR, 2023.
7. [Funk et al., 2021] Niklas Funk, Charles Schaff, Rishabh Madan, Takuma Yoneda, Julen Urain De Jesus, Joe Wat- son, Ethan K Gordon, Felix Widmaier, Stefan Bauer, Sid- dhartha S Srinivasa, et al. Benchmarking structured poli- cies and policy optimization for real-world dexterous ob- ject manipulation. IEEE Robotics and Automation Letters, 7(1):478-485, 2021.
8. [Grimminger et al., 2020] Felix Grimminger, Avadesh Meduri, Majid Khadiv, Julian Viereck, Manuel Wüthrich, Maximilien Naveau, Vincent Berenz, Steve Heim, Felix Widmaier, Thomas Flayols, et al. An open torque-controlled modular robot architecture for legged locomotion research. IEEE Robotics and Automation Letters, 5(2):3650-3657, 2020.
9. [Gürtler et al., 2023] Nico Gürtler, Sebastian Blaes, Pavel Kolev, Felix Widmaier, Manuel Wuthrich, Stefan Bauer, Bernhard Schölkopf, and Georg Martius. Benchmarking offline reinforcement learning on real-robot hardware. In The Eleventh International Conference on Learning Rep- resentations, 2023.
10. [Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on ma- chine learning, pages 1861-1870. PMLR, 2018.
11. [James et al., 2020] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019-3026, 2020.
12. [Javadi et al., 2023] Mahdi Javadi, Daniel Harnack, Paula Stocco, Shivesh Kumar, Shubham Vyas, Daniel Pizzutilo, and Frank Kirchner. Acromonk: A minimalist underac- tuated brachiating robot. IEEE Robotics and Automation Letters, 8(6):3637-3644, 2023.
13. [Li. and Todorov., 2004] Weiwei Li. and Emanuel Todorov. Iterative linear quadratic regulator design for nonlinear bi- ological movement systems. In Proceedings of the First International Conference on Informatics in Control, Au- tomation and Robotics -Volume 1: ICINCO, pages 222- 229. INSTICC, SciTePress, 2004.
14. [Maywald et al., 2022] Lasse Jenning Maywald, Felix Wiebe, Shivesh Kumar, Mahdi Javadi, and Frank Kirch- ner. Co-optimization of acrobot design and controller for increased certifiable stability. In 2022 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), pages 2636-2641. IEEE, 2022.
15. [Mittal et al., 2023] Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, et al. Orbit: A unified simulation framework for interactive robot learn- ing environments. IEEE Robotics and Automation Letters, 2023.
16. [Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al. Human-level control through deep re- inforcement learning. nature, 518(7540):529-533, 2015.
17. [Soni et al., 2023] Raghav Soni, Daniel Harnack, Hauke Is- ermann, Sotaro Fushimi, Shivesh Kumar, and Frank Kirchner. End-to-end reinforcement learning for torque based variable height hopping. In 2023 IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems (IROS), pages 7531-7538, 2023.
18. [Tedrake, 2022] Russ Tedrake. Underactuated Robotics. 2022.
19. [Turcato et al., 2023] Niccolo' Turcato, Alberto Dalla Lib- era, Giulio Giacomuzzo, and Ruggero Carli. Teaching a robot to toss arbitrary objects with model-based reinforce- ment learning. In 2023 9th International Conference on Control, Decision and Information Technologies (CoDIT), pages 1126-1131, 2023.
20. [Wiebe et al., 2022a] Felix Wiebe, Jonathan Babel, Shivesh Kumar, Shubham Vyas, Daniel Harnack, Melya Boukhed- dimi, Mihaela Popescu, and Frank Kirchner. Torque- limited simple pendulum: A toolkit for getting familiar with control algorithms in underactuated robotics. Jour- nal of Open Source Software, 7(74):3884, 2022.
21. [Wiebe et al., 2022b] Felix Wiebe, Shubham Vyas, Lasse Jenning Maywald, Shivesh Kumar, and Frank Kirchner. RealAIGym: Education and Research Platform for Studying Athletic Intelligence. In Proceedings of Robotics Science and Systems Workshop Mind the Gap: Opportunities and Challenges in the Transition Between Research and Industry, New York, July 2022.
22. [Wiebe et al., 2023] Felix Wiebe, Shivesh Kumar, Lasse J. Shala, Shubham Vyas, Mahdi Javadi, and Frank Kirch- ner. Open source dual-purpose acrobot and pendubot plat- form: Benchmarking control algorithms for underactuated robotics. IEEE Robotics Automation Magazine, pages 2- 13, 2023.