---
title: "A Convergent Reinforcement Learning Algorithm in the Continuous Case Based on a Finite Difference Method"
authors:
  - "Rémi Munos"
doi: ""
url: "https://ijcai.org/proceedings/1997/119"
pdf: "https://ijcai.org/proceedings/1997/0119.pdf"
venue: "Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence"
year: 1997
pages: "826-830"
track: "Learning"
keywords:
---

# A Convergent Reinforcement Learning Algorithm in the Continuous Case Based on a Finite Difference Method

Rémi Munos

*Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence*, pages 826-830.

## Abstract

In this paper, we propose a convergent Reinforcement Learning algorithm for solving optimal control problems for which the state space and the time are continuous variables. The problem of computing a good approximation of the value function, which is essential because this provides the optimal control, is a difficult task in the continuous case. Indeed, as it has been pointed out by several authors, the use of parameterized functions such as neura l networks for approximating the value function may produce very bad resul ts and even diverge. In fact, we show that classical algorithms, like Q-learning, used with a simp le look-up table built on a regular grid, may fai l to converge. The main reason is that the discretization of the state space implies a lost of the Markov property even for deterministic continuous processes. We propose to approximate the value function with a convergent numerical scheme based on a Finite Difference approximation of the Hamilton-Jacobi-Bellman equation. Then we present a model-free reinforcement learning algorithm. ca ll ed Finite Difference Reinforcement Learning, and prove its convergence to the value function of the continuous prob lem.