State Revisit and Re-explore: Bridging Sim-to-Real Gaps in Offline-and-Online Reinforcement Learning with An Imperfect Simulator

Chen, Xingyu; Xie, Jiayi; Xu, Zhijian; Liu, Ruixun; Yang, Shuai; Liu, Zeyang; Wan, Lipeng; Lan, Xuguang

doi:10.24963/ijcai.2025/970

State Revisit and Re-explore: Bridging Sim-to-Real Gaps in Offline-and-Online Reinforcement Learning with An Imperfect Simulator

Xingyu Chen, Jiayi Xie, Zhijian Xu, Ruixun Liu, Shuai Yang, Zeyang Liu, Lipeng Wan, Xuguang Lan

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 8724-8732.

https://doi.org/10.24963/ijcai.2025/970

Keywords

Machine Learning: ML: Reinforcement learning
Robotics: ROB: Learning in robotics

Abstract

In reinforcement learning (RL) based robot skill acquisition, a high-fidelity simulator is usually indispensable but unattainable since the real environment dynamics are difficult to model, which leads to severe sim-to-real gaps. Existing methods solve this problem by combining offline and online RL to jointly learn transferable policies from limited offline data and imperfect simulators. However, due to the unrestricted exploration in the imperfect simulator, the hybrid offline-and-online RL methods inevitably suffer from low sample efficiency and insufficient state-action space coverage during training. To solve this problem, we propose a State Revisit and Re-exploration (SR2) hybrid offline-and-online RL framework. In particular, the proposed algorithm employs a meta-policy and a sub-policy, where the meta-policy aims to find high-quality states in the offline trajectories for online exploration, and the sub-policy learns the robot skill using mixed offline and online data. By introducing the state revisit and explore mechanism, our approach efficiently improves performance on a set of sim-to-real robotic tasks. Through extensive simulation and real-world tasks, we demonstrate the superior performance of our approach against other state-of-the-art methods.

Xingyu Chen, Jiayi Xie, Zhijian Xu, Ruixun Liu, Shuai Yang, Zeyang Liu, Lipeng Wan, and Xuguang Lan. 2025. State Revisit and Re-explore: Bridging Sim-to-Real Gaps in Offline-and-Online Reinforcement Learning with An Imperfect Simulator. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. IJCAI, 8724–8732. https://doi.org/10.24963/ijcai.2025/970

@inproceedings{ijcai2025p970,
  title     = {State Revisit and Re-explore: Bridging Sim-to-Real Gaps in Offline-and-Online Reinforcement Learning with An Imperfect Simulator},
  author    = {Chen, Xingyu and Xie, Jiayi and Xu, Zhijian and Liu, Ruixun and Yang, Shuai and Liu, Zeyang and Wan, Lipeng and Lan, Xuguang},
  booktitle = {Proceedings of the Thirty-Fourth International Joint Conference on
               Artificial Intelligence, {IJCAI-25}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {James Kwok},
  pages     = {8724--8732},
  year      = {2025},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2025/970},
  url       = {https://doi.org/10.24963/ijcai.2025/970},
}

TY  - CPAPER
TI  - State Revisit and Re-explore: Bridging Sim-to-Real Gaps in Offline-and-Online Reinforcement Learning with An Imperfect Simulator
AU  - Chen, Xingyu
AU  - Xie, Jiayi
AU  - Xu, Zhijian
AU  - Liu, Ruixun
AU  - Yang, Shuai
AU  - Liu, Zeyang
AU  - Wan, Lipeng
AU  - Lan, Xuguang
T2  - Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2025
SP  - 8724
EP  - 8732
DO  - 10.24963/ijcai.2025/970
UR  - https://doi.org/10.24963/ijcai.2025/970
ER  -