{"type":"ScholarlyArticle","title":"Convergence of Reinforcement Learning with General Function Approximators","authors":[{"name":"Vassilis A. Papavassiliou","slug":"vassilis-a-papavassiliou","given":"Vassilis A.","family":"Papavassiliou","affiliation":""},{"name":"Stuart J. Russell","slug":"stuart-j-russell","given":"Stuart J.","family":"Russell","affiliation":""}],"abstract":"A key open problem in reinforcement learning is to assure convergence when using a compact hypothesis class to approximate the value function. Although the standard temporal-difference learning algorithm has been shown to converge when the hypothesis class is a linear combination of fixed basis functions, it may diverge with a general (nonlinear) hypothesis class. This paper describes the Bridge algorithm, a new method for reinforcement learning, and shows that it converges to an approximate global optimum for any agnostically learnable hypothesis class. Convergence is demonstrated on a simple example for which temporal-difference learning fails. Weak conditions are identified under which the Bridge algorithm converges for any hypothesis class. Finally, connections are made between the complexity of reinforcement learning and the PAC-learnability of the hypothesis class.","doi":null,"url":"https://ijcai.org/proceedings/1999/107","pdf":"https://ijcai.org/proceedings/1999/0107.pdf","markdown":"https://ijcai.org/proceedings/1999/107.md","bibtex":"https://ijcai.org/proceedings/1999/bibtex/107","volume":{"slug":"1999","title":"Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 1999","url":"https://ijcai.org/proceedings/1999/","year":1999,"location":"Stockholm, Sweden","dates":"","editors":"Thomas Dean","isbn":"","publisher":"International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":"Machine Learning","track":"Machine Learning","number":107,"pages":{"first":748,"last":756},"keywords":[],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}