Proceedings

Inversely Learning Transferable Rewards via Abstracted States

Download BibTeX Cite

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. Main Track. Pages 4322-4329.

https://doi.org/10.24963/ijcai.2026/481

Abstract

Inverse reinforcement learning (IRL) has made significant progress in recovering reward functions from expert demonstrations. However, a key challenge remains: how to extract reward functions that generalize across related but distinct tasks. In this paper, we address this by focusing on transferable IRL, learning intrinsic rewards that can drive effective behavior in unseen but structurally aligned environments. Our method leverages a variational autoencoder to learn an abstract representation of the state space shared across multiple source tasks. This abstracted space captures high-level features that are invariant across tasks, enabling the learning of a unified abstract reward function. The learned reward is then used to train policies in a separate, previously unseen target task without requiring new demonstrations in the target task. We evaluate our approach on multiple environments from Gymnasium and AssistiveGym, demonstrating that the learned abstract rewards consistently support successful policy learning in novel task settings.

  1. Target Run (rear disabled) Run (front disabled) Run (no disability) f -IRL 3,252.72 ± 67.9 3,523.01 ± 91.1 3,582.10 ± 94.3 AIRL-ME 4,014.52 ± 79.3 3,905.38 ± 73.1 3,725.63 ± 80.9 RIME 4,271.67 ± 36.1 4,129.19 ± 83.4 4,061.22 ± 58.6 I2L 4,396.43 ± 45.4 4,518.66 ± 52.2 4,512.71 ± 66.4 TraIRL 4,404.07 ± 57.6 4,359.35 ± 99.2 5,835.11 ± 74.0 Expert 5,052.25 ± 25.4 5,499.07 ± 156.1 6,420.38 ± 37.9
  2. [Allen et al., 2021] Cameron Allen, Neev Parikh, Omer Gottesman, and George Konidaris. Learning markov state abstractions for deep reinforcement learning. Advances in Neural Information Processing Systems, 34:8229-8241, 2021.
  3. [Arora and Doshi, 2021] Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500, 2021.
  4. [Buening et al., 2022] Thomas Kleine Buening, Victor Villin, and Christos Dimitrakakis. Environment design for inverse reinforcement learning. In Proceedings of the 41st International Conference on Machine Learning, 2022.
  5. [Chae et al., 2022] Jongseong Chae, Seungyul Han, Whiy- oung Jung, Myungsik Cho, Sungho Choi, and Youngchul Sung. Robust imitation learning against variations in envi- ronment dynamics. The 39th International Conference on Machine Learning, pages 2828-2852, 2022.
  6. [Erickson et al., 2020] Zackory Erickson, Vamsee Gan- garam, Ariel Kapusta, C. Karen Liu, and Charles C. Kemp. Assistive gym: A physics simulation framework for as- sistive robotics. In IEEE International Conference on Robotics and Automation (ICRA), 2020.
  7. [Fu et al., 2018] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse rein- forcement learning. In International Conference on Learn- ing Representations, 2018.
  8. [Garg et al., 2021] Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34:4028-4039, 2021.
  9. [Ghasemipour et al., 2020] Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. In The 3rd Conference on robot learning, 2020.
  10. [Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th Inter- national Conference on Machine Learning, 2018.
  11. [Jain et al., 2025] Arnav Kumar Jain, Harley Wiltzer, Jesse Farebrother, Irina Rish, Glen Berseth, and Sanjiban Choudhury. Non-adversarial inverse reinforcement learn- ing via successor feature matching. In The International Conference on Learning Representations, 2025.
  12. [Kim et al., 2021] Kuno Kim, Shivam Garg, Kirankumar Shiragur, and Stefano Ermon. Reward identification in in- verse reinforcement learning. Proceedings of the 38th In- ternational Conference on Machine Learning, 139:5496- 5505, 2021.
  13. [Lee et al., 2019] Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhut- dinov. Efficient exploration via state marginal matching. In arXiv, 2019.
  14. [Ni et al., 2021] Tianwei Ni, Harshit Sikchi, Yufei Wang, Tejus Gupta, Lisa Lee, and Ben Eysenbach. f-irl: Inverse reinforcement learning via state marginal matching. In The 4th Conference on Robot Learning, 2021.
  15. [Rolland et al., 2022] Paul Rolland, Luca Viano, Norman Schürhoff, Boris Nikolov, and Volkan Cevher. Identifia- bility and generalizability from multiple experts in inverse reinforcement learning. Advances in Neural Information Processing Systems, 35:550-564, 2022.
  16. [Schlaginhaufen and Kamgarpour, 2024] Andreas Schlagin- haufen and Maryam Kamgarpour. Towards the transfer- ability of rewards recovered via regularized inverse rein- forcement learning. In Advances in Neural Information Processing Systems, 2024.
  17. [Schulman et al., 2016] John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In 4th International Conference on Learning Representations, 2016.
  18. [Tanmay and Jian, 2020] Gangwani Tanmay and Peng Jian. State-only imitation with transition dynamics mismatch. In The 8th International Conference on Learning Repre- sentations, 2020.
  19. [Tanwani and Billard, 2013] Ajay Kumar Tanwani and Aude Billard. Transfer in inverse reinforcement learning for multiple strategies. 2013 IEEE/RSJ International Confer- ence on Intelligent Robots and Systems, pages 3244-3250, 2013.
  20. [Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. Mujoco: A physics engine for model-based con- trol. 2012 IEEE/RSJ International Conference on Intelli- gent Robots and Systems, pages 5026-5033, 2012.
  21. [Viano et al., 2024] Luca Viano, Yu-Ting Huang, Parameswaran Kamalaruban, Adrian Weller, and Volkan Cevher. Robust inverse reinforcement learning under transition dynamics mismatch. In Proceedings of the 35th International Conference on Neural Information Processing Systems, 2024.
  22. [Yoo and Seo, 2022] Se-Wook Yoo and Seung-Woo Seo. Learning multi-task transferable rewards via variational inverse reinforcement learning. 2022 International Con- ference on Robotics and Automation (ICRA), pages 434- 440, 2022.
  23. [Ziebart et al., 2010] Brian D. Ziebart, J. Andrew Bagnell, and Anind K. Dey. Modeling interaction via the principle of maximum causal entropy. In Proceedings of the 27th International Conference on Machine Learning, 2010.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards