A Convergent Reinforcement Learning Algorithm in the Continuous Case Based on a Finite Difference Method

Abstract

In this paper, we propose a convergent Reinforcement Learning algorithm for solving optimal control problems for which the state space and the time are continuous variables. The problem of computing a good approximation of the value function, which is essential because this provides the optimal control, is a difficult task in the continuous case. Indeed, as it has been pointed out by several authors, the use of parameterized functions such as neura l networks for approximating the value function may produce very bad resul ts and even diverge. In fact, we show that classical algorithms, like Q-learning, used with a simp le look-up table built on a regular grid, may fai l to converge. The main reason is that the discretization of the state space implies a lost of the Markov property even for deterministic continuous processes. We propose to approximate the value function with a convergent numerical scheme based on a Finite Difference approximation of the Hamilton-Jacobi-Bellman equation. Then we present a model-free reinforcement learning algorithm. ca ll ed Finite Difference Reinforcement Learning, and prove its convergence to the value function of the continuous prob lem.