Potential Based Reward Shaping for Hierarchical Reinforcement Learning
Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence. Special Track on Machine Learning. Pages 3504-3509.
Abstract
Hierarchical Reinforcement Learning (HRL) outperforms many ‘flat’ Reinforcement Learning (RL) algorithms in some application domains. However, HRL may need longer time to obtain the optimal policy because of its large action space. Potential Based Reward Shaping (PBRS) has been widely used to incorporate heuristics into flat RL algorithms so as to reduce their exploration. In this paper, we investigate the integration of PBRS and HRL, and propose a new algorithm: PBRS-MAXQ-0. We prove that under certain conditions, PBRS-MAXQ-0 is guaranteed to converge. Empirical results show that PBRS-MAXQ-0 significantly outperforms MAXQ-0 given good heuristics, and can converge even when given misleading heuristics.
@inproceedings{ijcai2015p493,
title = {Potential Based Reward Shaping for Hierarchical Reinforcement Learning},
author = {Gao, Yang and Toni, Francesca},
booktitle = {Proceedings of the Twenty-Fourth International Joint Conference on
Artificial Intelligence, {IJCAI-15}},
publisher = {International Joint Conferences on Artificial Intelligence},
editor = {Qiang Yang and Michael Wooldridge},
pages = {3504--3509},
year = {2015},
month = {7},
note = {Special Track on Machine Learning},
url = {https://ijcai.org/proceedings/2015/493},
}
TY - CPAPER TI - Potential Based Reward Shaping for Hierarchical Reinforcement Learning AU - Gao, Yang AU - Toni, Francesca T2 - Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-15 PB - International Joint Conferences on Artificial Intelligence PY - 2015 SP - 3504 EP - 3509 UR - https://ijcai.org/proceedings/2015/493 ER -Copied
Machine-readable versions of this paper, for tools and AI agents:
