The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning

Schoepp, Sheila; Jafaripour, Masoud; Cao, Yingyue; Yang, Tianpei; Abdollahi, Fatemeh; Golestan, Shadan; Sufiyan, Zahin; Zaiane, Osmar R.; Taylor, Matthew E.

doi:10.24963/ijcai.2025/1181

The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning

Sheila Schoepp, Masoud Jafaripour, Yingyue Cao, Tianpei Yang, Fatemeh Abdollahi, Shadan Golestan, Zahin Sufiyan, Osmar R. Zaiane, Matthew E. Taylor

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Survey Track. Pages 10641-10649.

https://doi.org/10.24963/ijcai.2025/1181

Keywords

Machine Learning: ML: Reinforcement learning
Natural Language Processing: NLP: Language models

Abstract

Reinforcement learning (RL) has shown impressive results in sequential decision-making tasks. Large Language Models (LLMs) and Vision-Language Models (VLMs) have recently emerged, exhibiting impressive capabilities in multimodal understanding and reasoning. These advances have led to a surge of research integrating LLMs and VLMs into RL. This survey reviews representative works in which LLMs and VLMs are used to overcome key challenges in RL, such as lack of prior knowledge, long-horizon planning, and reward design. We present a taxonomy that categorizes these LLM/VLM-assisted RL approaches into three roles: agent, planner, and reward. We conclude by exploring open problems, including grounding, bias mitigation, improved representations, and action advice. By consolidating existing research and identifying future directions, this survey establishes a framework for integrating LLMs and VLMs into RL, advancing approaches that unify natural language and visual understanding with sequential decision-making.

Sheila Schoepp, Masoud Jafaripour, Yingyue Cao, Tianpei Yang, Fatemeh Abdollahi, Shadan Golestan, Zahin Sufiyan, Osmar R. Zaiane, and Matthew E. Taylor. 2025. The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. IJCAI, 10641–10649. https://doi.org/10.24963/ijcai.2025/1181

@inproceedings{ijcai2025p1181,
  title     = {The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning},
  author    = {Schoepp, Sheila and Jafaripour, Masoud and Cao, Yingyue and Yang, Tianpei and Abdollahi, Fatemeh and Golestan, Shadan and Sufiyan, Zahin and Zaiane, Osmar R. and Taylor, Matthew E.},
  booktitle = {Proceedings of the Thirty-Fourth International Joint Conference on
               Artificial Intelligence, {IJCAI-25}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {James Kwok},
  pages     = {10641--10649},
  year      = {2025},
  month     = {8},
  note      = {Survey Track},
  doi       = {10.24963/ijcai.2025/1181},
  url       = {https://doi.org/10.24963/ijcai.2025/1181},
}

TY  - CPAPER
TI  - The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning
AU  - Schoepp, Sheila
AU  - Jafaripour, Masoud
AU  - Cao, Yingyue
AU  - Yang, Tianpei
AU  - Abdollahi, Fatemeh
AU  - Golestan, Shadan
AU  - Sufiyan, Zahin
AU  - Zaiane, Osmar R.
AU  - Taylor, Matthew E.
T2  - Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2025
SP  - 10641
EP  - 10649
DO  - 10.24963/ijcai.2025/1181
UR  - https://doi.org/10.24963/ijcai.2025/1181
ER  -