{"type":"ScholarlyArticle","title":"Finite Sample Analysis of LSTD with Random Projections and Eligibility Traces","authors":[{"name":"Haifang Li","slug":"haifang-li","given":"Haifang","family":"Li","affiliation":"Institute of Automation , Chinese Academy of Sciences , Beijing , China"},{"name":"Yingce Xia","slug":"yingce-xia","given":"Yingce","family":"Xia","affiliation":"University of Science and Technology of China , Hefei , Anhui , China"},{"name":"Wensheng Zhang","slug":"wensheng-zhang","given":"Wensheng","family":"Zhang","affiliation":"Institute of Automation , Chinese Academy of Sciences , Beijing , China"}],"abstract":"Policy evaluation with linear function approximation is an important problem in reinforcement learning. When facing high-dimensional feature spaces, such a problem becomes extremely hard considering the computation efficiency and quality of approximations. We propose a new algorithm, LSTD(lambda)-RP, which leverages random projection techniques and takes eligibility traces into consideration to tackle the above two challenges. We carry out theoretical analysis of LSTD(lambda)-RP, and provide meaningful upper bounds of the estimation error, approximation error and total generalization error. These results demonstrate that LSTD(lambda)-RP can benefit from random projection and eligibility traces strategies, and LSTD(lambda)-RP can achieve better performances than prior LSTD-RP and LSTD(lambda) algorithms.","doi":"10.24963/ijcai.2018/331","url":"https://ijcai.org/proceedings/2018/331","pdf":"https://ijcai.org/proceedings/2018/0331.pdf","markdown":"https://ijcai.org/proceedings/2018/331.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/331","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":331,"pages":{"first":2390,"last":2396},"keywords":["Machine Learning: Learning Theory","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Att et al., 2000] Michael Kearns Att, Michael Kearns, and Satin- der Singh. Bias-variance error bounds for temporal difference updates. In In Proceedings of the 13th Annual Conference on Computational Learning Theory. Citeseer, 2000.","title":"Bias-variance error bounds for temporal difference updates","authors":"Michael Kearns, Att, Michael Kearns, Satinder Singh","year":"2000","venue":"Proceedings of the 13th Annual Conference on Computational Learning Theory","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Balakrishna et al., 2010] Poornima Balakrishna, Rajesh Ganesan, and Lance Sherry. Accuracy of reinforcement learning algo- rithms for predicting aircraft taxi-out times: A case-study of tampa bay departures. Transportation Research Part C: Emerg- ing Technologies, 18(6):950-962, 2010.","title":"Accuracy of reinforcement learning algorithms for predicting aircraft taxi-out times: A case-study of tampa bay departures","authors":"Poornima Balakrishna, Rajesh Ganesan, Lance Sherry","year":"2010","venue":"Transportation Research Part C: Emerging Technologies","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Boyan, 1999] Justin A Boyan. Least-squares temporal difference learning. In ICML, pages 49-56.","title":"Least-squares temporal difference learning","authors":"Justin A Boyan","year":null,"venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Boyan, 2002] Justin A Boyan. Technical update: Least-squares temporal difference learning. Machine Learning, 49(2):233-246, 2002.","title":"Technical update: Least-squares temporal difference learning","authors":"Justin A Boyan","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bradtke and Barto, 1996] Steven J Bradtke and Andrew G Barto. Linear least-squares algorithms for temporal difference learning. Machine learning, 22(1-3):33-57, 1996.","title":"Linear least-squares algorithms for temporal difference learning","authors":"J Steven, Andrew G Bradtke, Barto","year":"1996","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dann et al., 2014] Christoph Dann, Gerhard Neumann, and Jan Peters. Policy evaluation with temporal differences: a survey and comparison. JMLR, 15(1):809-883, 2014.","title":"Policy evaluation with temporal differences: a survey and comparison","authors":"Dann","year":"2014","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dann, 2012] Christoph Dann. Algorithms for fast gradient tempo- ral difference learning. In Autonomous Learning Systems Semi- nar, TU Darmstadt, 2012.","title":"Algorithms for fast gradient temporal difference learning","authors":"Christoph Dann","year":"2012","venue":"Autonomous Learning Systems Seminar","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Farahmand and Szepesvari, 2011] Amir-massoud Farahmand and Csaba Szepesvari. Regularization in reinforcement learning. University of Alberta, 2011.","title":"Regularization in reinforcement learning","authors":"Amir-Massoud Farahmand, Csaba Szepesvari","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Frank et al., 2008] Jordan Frank, Shie Mannor, and Doina Precup. Reinforcement learning in the presence of rare events. In ICML, pages 336-343. ACM, 2008.","title":"Reinforcement learning in the presence of rare events","authors":"Frank","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gehring et al., 2016] Clement Gehring, Yangchen Pan, and Martha White. Incremental truncated lstd. In IJCAI, pages 1505- 1511. AAAI Press, 2016.","title":"Incremental truncated lstd","authors":"Clement Gehring, Yangchen Pan, Martha White","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/216"},{"position":11,"raw":"[Geist and Pietquin, 2013] Matthieu Geist and Olivier Pietquin. Al- gorithmic survey of parametric value function approximation. TNNLS, 24(6):845-867, 2013.","title":"Algorithmic survey of parametric value function approximation","authors":"Matthieu Geist, Olivier Pietquin","year":"2013","venue":"TNNLS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Geist and Scherrer, 2014] Matthieu Geist and Bruno Scherrer. Off- policy learning with eligibility traces: A survey. JMLR, 15(1):289-333, 2014.","title":"Offpolicy learning with eligibility traces: A survey","authors":"Scherrer Geist, Matthieu Geist, Bruno Scherrer","year":"2014","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ghavamzadeh et al., 2010] Mohammad Ghavamzadeh, Alessan- dro Lazaric, Odalric Maillard, and Rémi Munos. Lstd with ran- dom projections. In NIPS, pages 721-729, 2010.","title":"Lstd with random projections","authors":"Mohammad Ghavamzadeh, Alessandro Lazaric, Odalric Maillard, Rémi Munos","year":"2010","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kolter and Ng, 2009] J Zico Kolter and Andrew Y Ng. Regular- ization and feature selection in least-squares temporal difference learning. In ICML, pages 521-528. ACM, 2009.","title":"Regularization and feature selection in least-squares temporal difference learning","authors":"Ng ; ] J Zico Kolter, Andrew Y Kolter, Ng","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2018] Haifang Li, Yingce Xia, and Wensheng Zhang. Fi- nite sample analysis of lstd with random projections and eligibil- ity traces. arXiv preprint arXiv:1805.10005, 2018.","title":"Finite sample analysis of lstd with random projections and eligibility traces","authors":"Haifang Li, Yingce Xia, Wensheng Zhang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liang et al., 2016] Yitao Liang, Marlos C Machado, Erik Talvitie, and Michael Bowling. State of the art control of atari games us- ing shallow reinforcement learning. In AAMAS, pages 485-493. International Foundation for Autonomous Agents and Multiagent Systems, 2016.","title":"State of the art control of atari games using shallow reinforcement learning","authors":"Liang","year":"2016","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Maei, 2011] Hamid Reza Maei. Gradient temporal-difference learning algorithms. 2011.","title":"Gradient temporal-difference learning algorithms","authors":"Reza Hamid, Maei","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Maillard and Munos, 2009] Odalric Maillard and Rémi Munos. Compressed least-squares regression. In NIPS, pages 1213-1221, 2009.","title":"Compressed least-squares regression","authors":"Munos Maillard, Odalric Maillard, Rémi Munos","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mohri and Rostamizadeh, 2010] Mehryar Mohri and Afshin Ros- tamizadeh. Stability bounds for stationary ϕ-mixing and β- mixing processes. JMLR, 11(Feb):789-814, 2010.","title":"Stability bounds for stationary ϕ-mixing and βmixing processes","authors":"Mehryar Mohri, Afshin Rostamizadeh","year":"2010","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pan et al., 2017a] Yangchen Pan, Erfan Sadeqi Azer, and Martha White. Effective sketching methods for value function approxi- mation. arXiv preprint arXiv:1708.01298, 2017.","title":"Effective sketching methods for value function approximation","authors":"Pan","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pan et al., 2017b] Yangchen Pan, Adam M White, and Martha White. Accelerated gradient temporal difference learning. In AAAI, pages 2464-2470, 2017.","title":"Accelerated gradient temporal difference learning","authors":"Pan","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Puterman, 2014] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.","title":"Markov decision processes: discrete stochastic dynamic programming","authors":"Martin L Puterman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Riedmiller and Gabel, 2007] Martin Riedmiller and Thomas Gabel. On experiences in a complex and competitive gaming do- main: Reinforcement learning meets robocup. In Computational Intelligence and Games, 2007. CIG 2007. IEEE Symposium on, pages 17-23. IEEE, 2007.","title":"On experiences in a complex and competitive gaming domain: Reinforcement learning meets robocup","authors":"Gabel Riedmiller, Martin Riedmiller, Thomas Gabel","year":"2007","venue":"Computational Intelligence and Games","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Silver et al., 2007] D Silver, R Sutton, and M Müller. Reinforce- ment learning of local shape in the game of go. In IJCAI, pages 1053-1058, 2007.","title":"Reinforcement learning of local shape in the game of go","authors":"Silver, M Sutton, Müller","year":"2007","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2007/170"},{"position":25,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sutton et al., 2009] Richard S Sutton, Hamid R Maei, and Csaba Szepesvári. A convergent o(n) temporal-difference algorithm for off-policy learning with linear function approximation. In NIPS, pages 1609-1616, 2009.","title":"A convergent o(n) temporal-difference algorithm for off-policy learning with linear function approximation","authors":"Hamid R Richard S Sutton, Csaba Maei, Szepesvári","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sutton et al., 2014] Rich Sutton, Ashique Rupam Mahmood, Doina Precup, and Hado Hasselt. A new q (lambda) with interim forward view and monte carlo equivalence. In ICML, pages 568- 576, 2014.","title":"A new q (lambda) with interim forward view and monte carlo equivalence","authors":"Rich Sutton, Ashique Rupam Mahmood, Doina Precup, Hado Hasselt","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sutton, 1996] Richard S Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In NIPS, pages 1038-1044, 1996.","title":"Generalization in reinforcement learning: Successful examples using sparse coarse coding","authors":"Richard S Sutton","year":"1996","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tagorti and Scherrer, 2015] Manel Tagorti and Bruno Scherrer. On the rate of convergence and error bounds for lstd (λ). In ICML, pages 1521-1529, 2015.","title":"Manel Tagorti and Bruno Scherrer. On the rate of convergence and error bounds for lstd (λ)","authors":"Scherrer Tagorti","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tedrake et al., 2004] Russ Tedrake, Teresa Weirui Zhang, and H Sebastian Seung. Stochastic policy gradient reinforcement learning on a simple 3d biped. In In Proceedings of the IEEE International Conference on Intelligent Robots and Systems, vol- ume 3, pages 2849-2854. IEEE, 2004.","title":"Stochastic policy gradient reinforcement learning on a simple 3d biped","authors":"Russ Tedrake, Teresa Weirui Zhang, H Sebastian, Seung","year":"2004","venue":"Proceedings of the IEEE International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Tsitsiklis et al., 1997] John N Tsitsiklis, Benjamin Van Roy, et al. An analysis of temporal-difference learning with func- tion approximation. IEEE transactions on automatic control, 42(5):674-690, 1997.","title":"An analysis of temporal-difference learning with function approximation","authors":"Benjamin John N Tsitsiklis, Van Roy","year":"1997","venue":"IEEE transactions on automatic control","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yu, 2015] Huizhen Yu. On convergence of emphatic temporal- difference learning. In Conference on Learning Theory, pages 1724-1751, 2015.","title":"On convergence of emphatic temporaldifference learning","authors":"Yu ;, Huizhen Yu","year":"2015","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}