{"type":"ScholarlyArticle","title":"From Q(lambda) to Average Q-learning: Efficient Implementation of an Asymptotic Approximation","authors":[{"name":"Frédérick Garçia","slug":"frederick-garcia","given":"Frédérick","family":"Garçia","affiliation":""},{"name":"Florent Serre","slug":"florent-serre","given":"Florent","family":"Serre","affiliation":""}],"abstract":"Q(\u0002 ) is a reinforcement learning algorithm that combines Q-learning and TD(\u0002 ). Online implementations of Q(\u0002 ) that use eligibility traces have been shown to speed basic Q-learning. In this paper we present an asymptotic analysis of Watkins’ Q(\u0002 ) with accumulative eligibility traces. We first introduce an asymptotic approximation of Q(\u0002 ) that appears to be a gain matrix variant of basic Qlearning. Using the ODE method, we then determine an optimal gain matrix for Q-learning that maximizes its rate of convergence toward the optimal value function \u0003 \u0005\u0004 . The similarity between this optimal gain and the asymptotic gain of Q(\u0002 ) explains the relative efficiency of the latter for \u0002 \u0007\u0006\b. Furthermore, by minimizing the difference between these two gains, optimal values for the \u0002 parameter and the decreasing learning rates can be determined. This optimal \u0002 strongly depends on the exploration policy during learning. A robust approximation of these learning parameters leads to the definition of a new efficient algorithm called AQ-learning (Average Q-learning), that shows a close resemblance to Schwartz’ R-learning. Our results have been demonstrated through numerical simulations.","doi":null,"url":"https://ijcai.org/proceedings/2001/131","pdf":"https://ijcai.org/proceedings/2001/0131.pdf","markdown":"https://ijcai.org/proceedings/2001/131.md","bibtex":"https://ijcai.org/proceedings/2001/bibtex/131","volume":{"slug":"2001","title":"Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2001","url":"https://ijcai.org/proceedings/2001/","year":2001,"location":"Seattle, Washington, USA","dates":"","editors":"","isbn":"","publisher":"International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":null,"track":"","number":131,"pages":{"first":959,"last":963},"keywords":[],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}