MapGo: Model-Assisted Policy Optimization for Goal-Oriented Tasks

Zhu, Menghui; Liu, Minghuan; Shen, Jian; Zhang, Zhicheng; Chen, Sheng; Zhang, Weinan; Ye, Deheng; Yu, Yong; Fu, Qiang; Yang, Wei

doi:10.24963/ijcai.2021/480

MapGo: Model-Assisted Policy Optimization for Goal-Oriented Tasks

Menghui Zhu, Minghuan Liu, Jian Shen, Zhicheng Zhang, Sheng Chen, Weinan Zhang, Deheng Ye, Yong Yu, Qiang Fu, Wei Yang

Download BibTeX Cite

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. Main Track. Pages 3484-3491.

https://doi.org/10.24963/ijcai.2021/480

Keywords

Machine Learning: Reinforcement Learning
Machine Learning: Deep Reinforcement Learning

Abstract

In Goal-oriented Reinforcement learning, relabeling the raw goals in past experience to provide agents with hindsight ability is a major solution to the reward sparsity problem. In this paper, to enhance the diversity of relabeled goals, we develop FGI (Foresight Goal Inference), a new relabeling strategy that relabels the goals by looking into the future with a learned dynamics model. Besides, to improve sample efficiency, we propose to use the dynamics model to generate simulated trajectories for policy training. By integrating these two improvements, we introduce the MapGo framework (Model-Assisted Policy optimization for Goal-oriented tasks). In our experiments, we first show the effectiveness of the FGI strategy compared with the hindsight one, and then show that the MapGo framework achieves higher sample efficiency when compared to model-free baselines on a set of complicated tasks.

Menghui Zhu, Minghuan Liu, Jian Shen, Zhicheng Zhang, Sheng Chen, Weinan Zhang, Deheng Ye, Yong Yu, Qiang Fu, and Wei Yang. 2021. MapGo: Model-Assisted Policy Optimization for Goal-Oriented Tasks. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. IJCAI, 3484–3491. https://doi.org/10.24963/ijcai.2021/480

@inproceedings{ijcai2021p480,
  title     = {MapGo: Model-Assisted Policy Optimization for Goal-Oriented Tasks},
  author    = {Zhu, Menghui and Liu, Minghuan and Shen, Jian and Zhang, Zhicheng and Chen, Sheng and Zhang, Weinan and Ye, Deheng and Yu, Yong and Fu, Qiang and Yang, Wei},
  booktitle = {Proceedings of the Thirtieth International Joint Conference on
               Artificial Intelligence, {IJCAI-21}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Prof. Zhi-Hua Zhou},
  pages     = {3484--3491},
  year      = {2021},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2021/480},
  url       = {https://doi.org/10.24963/ijcai.2021/480},
}

TY  - CPAPER
TI  - MapGo: Model-Assisted Policy Optimization for Goal-Oriented Tasks
AU  - Zhu, Menghui
AU  - Liu, Minghuan
AU  - Shen, Jian
AU  - Zhang, Zhicheng
AU  - Chen, Sheng
AU  - Zhang, Weinan
AU  - Ye, Deheng
AU  - Yu, Yong
AU  - Fu, Qiang
AU  - Yang, Wei
T2  - Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2021
SP  - 3484
EP  - 3491
DO  - 10.24963/ijcai.2021/480
UR  - https://doi.org/10.24963/ijcai.2021/480
ER  -