{"type":"ScholarlyArticle","title":"A Convergent Reinforcement Learning Algorithm in the Continuous Case Based on a Finite Difference Method","authors":[{"name":"Rémi Munos","slug":"remi-munos","given":"Rémi","family":"Munos","affiliation":""}],"abstract":"In this paper, we propose a convergent Reinforcement Learning algorithm for solving optimal control problems for which the state space and the time are continuous variables. The problem of computing a good approximation of the value function, which is essential because this provides the optimal control, is a difficult task in the continuous case. Indeed, as it has been pointed out by several authors, the use of parameterized functions such as neura l networks for approximating the value function may produce very bad resul ts and even diverge. In fact, we show that classical algorithms, like Q-learning, used with a simp le look-up table built on a regular grid, may fai l to converge. The main reason is that the discretization of the state space implies a lost of the Markov property even for deterministic continuous processes. We propose to approximate the value function with a convergent numerical scheme based on a Finite Difference approximation of the Hamilton-Jacobi-Bellman equation. Then we present a model-free reinforcement learning algorithm. ca ll ed Finite Difference Reinforcement Learning, and prove its convergence to the value function of the continuous prob lem.","doi":null,"url":"https://ijcai.org/proceedings/1997/119","pdf":"https://ijcai.org/proceedings/1997/0119.pdf","markdown":"https://ijcai.org/proceedings/1997/119.md","bibtex":"https://ijcai.org/proceedings/1997/bibtex/119","volume":{"slug":"1997","title":"Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 1997","url":"https://ijcai.org/proceedings/1997/","year":1997,"location":"Nagoya, Japan","dates":"","editors":"Martha E. Pollack","isbn":"","publisher":"International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":"Learning","track":"Learning","number":119,"pages":{"first":826,"last":830},"keywords":[],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}