Finite Sample Analysis of LSTD with Random Projections and Eligibility Traces
Finite Sample Analysis of LSTD with Random Projections and Eligibility Traces
Haifang Li, Yingce Xia, Wensheng Zhang
Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence
Main track. Pages 2390-2396.
https://doi.org/10.24963/ijcai.2018/331
Policy evaluation with linear function approximation is an important problem in reinforcement learning. When facing high-dimensional feature spaces, such a problem becomes extremely hard considering the computation efficiency and quality of approximations. We propose a new algorithm, LSTD(lambda)-RP, which leverages random projection techniques and takes eligibility traces into consideration to tackle the above two challenges. We carry out theoretical analysis of LSTD(lambda)-RP, and provide meaningful upper bounds of the estimation error, approximation error and total generalization error. These results demonstrate that LSTD(lambda)-RP can benefit from random projection and eligibility traces strategies, and LSTD(lambda)-RP can achieve better performances than prior LSTD-RP and LSTD(lambda) algorithms.
Keywords:
Machine Learning: Learning Theory
Machine Learning: Reinforcement Learning