{"type":"ScholarlyArticle","title":"A Unified Approach for Multi-step Temporal-Difference Learning with Eligibility Traces in Reinforcement Learning","authors":[{"name":"Long Yang","slug":"long-yang","given":"Long","family":"Yang","affiliation":"College of Computer Science and Technology , Zhejiang University , China"},{"name":"Minhao Shi","slug":"minhao-shi","given":"Minhao","family":"Shi","affiliation":"College of Computer Science and Technology , Zhejiang University , China"},{"name":"Qian Zheng","slug":"qian-zheng","given":"Qian","family":"Zheng","affiliation":"College of Computer Science and Technology , Zhejiang University , China"},{"name":"Wenjia Meng","slug":"wenjia-meng","given":"Wenjia","family":"Meng","affiliation":"College of Computer Science and Technology , Zhejiang University , China"},{"name":"Gang Pan","slug":"gang-pan","given":"Gang","family":"Pan","affiliation":"College of Computer Science and Technology , Zhejiang University , China"}],"abstract":"Recently, a new multi-step temporal learning algorithm Q(σ) unifies n-step Tree-Backup (when σ = 0) and \r\nn-step Sarsa (when σ = 1) by introducing a sampling parameter σ. However, similar to other multi-step \r\ntemporal-difference learning algorithms, Q(σ) needs much memory consumption and computation time. Eligibility trace is an important mechanism to transform the off-line updates into efficient on-line ones which consume less memory and computation time. In this paper, we combine the original Q(σ) with eligibility traces and propose a new algorithm, called Qπ(σ,λ), where λ is trace-decay parameter. This new algorithm unifies Sarsa(λ) (when σ = 1) and Qπ (λ) (when σ = 0). Furthermore, we give an upper error bound of Qπ(σ,λ) policy evaluation algorithm. We prove that Qπ (σ, λ) control algorithm converges to the optimal value function exponentially. We also empirically compare it with conventional temporal-difference learning methods. Results show that, with an intermediate value of σ, Qπ(σ,λ) creates a mixture of the existing algorithms which learn the optimal value significantly faster than the extreme end (σ = 0, or 1).","doi":"10.24963/ijcai.2018/414","url":"https://ijcai.org/proceedings/2018/414","pdf":"https://ijcai.org/proceedings/2018/0414.pdf","markdown":"https://ijcai.org/proceedings/2018/414.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/414","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":414,"pages":{"first":2984,"last":2990},"keywords":["Uncertainty in AI: Markov Decision Processes","Machine Learning: Machine Learning","Machine Learning: Reinforcement Learning","Uncertainty in AI: Sequential Decision Making","Uncertainty in AI: Decision;Utility Theory"],"videos":[],"references":[{"position":1,"raw":"[Bertsekas and Tsitsiklis, 1996] Dimitri P. Bertsekas and John N. Tsitsiklis. Neuro Dynamic Programming. Athena scientific Belmont, MA, 1996.","title":"Neuro Dynamic Programming","authors":"Dimitri P Bertsekas, John N Tsitsiklis","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bertsekas et al., 2012] Dimitri P Bertsekas, Dimitri P Bert- sekas, Dimitri P Bertsekas, and Dimitri P Bertsekas. Dynamic programming and optimal control, volume 2. Athena scientific Belmont, MA, 2012.","title":"Dynamic programming and optimal control","authors":"Dimitri P Dimitri P Bertsekas, Dimitri P Bertsekas, Dimitri P Bertsekas, Bertsekas","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[De Asis et al., 2018] Kristopher De Asis, J Fernando Hernandez-Garcia, G Zacharias Holland, and Richard S Sutton. Multi-step reinforcement learning: A unifying al- gorithm. AAAI, 2018.","title":"Multi-step reinforcement learning: A unifying algorithm","authors":"Kristopher De Asis, Fernando Hernandez-Garcia, Zacharias Holland, Richard S Sutton","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dumke, 2017] Markus Dumke. Double q(σ) and q(σ, λ): Unifying reinforcement learning control algorithms. arXiv preprint arXiv:1711.01569, 2017.","title":"Double q(σ) and q(σ, λ): Unifying reinforcement learning control algorithms","authors":null,"year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Harutyunyan, 2016] Rémi Harutyunyan. Q (λ) with off- policy corrections. In Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, Octo- ber 19-21, 2016, Proceedings, volume 9925, page 305. Springer, 2016.","title":"Q (λ) with offpolicy corrections","authors":"Rémi Harutyunyan","year":"2016","venue":"Algorithmic Learning Theory: 27th International Conference, ALT 2016","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jaakkola et al., 1994] Tommi Jaakkola, Michael I Jordan, and Satinder P Singh. Convergence of stochastic iterative dynamic programming algorithms. In Advances in neural information processing systems, pages 703-710, 1994.","title":"Convergence of stochastic iterative dynamic programming algorithms","authors":"Tommi Jaakkola, Michael I Jordan, P Satinder, Singh","year":"1994","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kearns and Singh, 2000] Michael J Kearns and Satinder P Singh. Bias-variance error bounds for temporal difference updates. In COLT, pages 142-147, 2000.","title":"Bias-variance error bounds for temporal difference updates","authors":"J Michael, Kearns, P Satinder, Singh","year":"2000","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Munos et al., 2016] Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc Bellemare. Safe and efficient off- policy reinforcement learning. In Advances in Neural In- formation Processing Systems, pages 1054-1062, 2016.","title":"Safe and efficient offpolicy reinforcement learning","authors":"Rémi Munos, Tom Stepleton, Anna Harutyunyan, Marc Bellemare","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Rummery and Niranjan, 1994] Gavin A Rummery and Ma- hesan Niranjan. On-line Q-learning using connectionist systems, volume 37. University of Cambridge, Department of Engineering, 1994.","title":"On-line Q-learning using connectionist systems","authors":"A Gavin, Mahesan Rummery, Niranjan","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Seijen and Sutton, 2014] Harm Seijen and Rich Sutton. True online td (lambda). In International Conference on Machine Learning, pages 692-700, 2014.","title":"Harm Seijen and Rich Sutton. True online td (lambda)","authors":"Sutton Seijen","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Singh and Dayan, 1998] Satinder Singh and Peter Dayan. Analytical mean squared error curves for temporal differ- ence learning. Machine Learning, 32(1):5-40, 1998.","title":"Analytical mean squared error curves for temporal difference learning","authors":"Satinder Singh, Peter Dayan","year":"1998","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Singh and Sutton, 1996] Satinder P Singh and Richard S Sutton. Reinforcement learning with replacing eligibility traces. Machine learning, 22(1-3):123-158, 1996.","title":"Reinforcement learning with replacing eligibility traces","authors":"P Satinder, Richard S Singh, Sutton","year":"1996","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Singh et al., 2000] Satinder Singh, Tommi Jaakkola, Michael L Littman, and Csaba Szepesvári. Convergence results for single-step on-policy reinforcement-learning algorithms. Machine learning, 38(3):287-308, 2000.","title":"Convergence results for single-step on-policy reinforcement-learning algorithms","authors":"Satinder Singh, Tommi Jaakkola, Csaba Michael L Littman, Szepesvári","year":"2000","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, vol- ume 1. MIT press Cambridge, 1998.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sutton and Barto, 2017] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, vol- ume 1. MIT press Cambridge, 2017.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sutton, 1984] Richard S Sutton. Temporal credit assign- ment in reinforcement learning. PhD thesis, University of Massachusetts, 1984.","title":"Temporal credit assignment in reinforcement learning","authors":"Richard S Sutton","year":"1984","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sutton, 1988] Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3(1):9-44, 1988.","title":"Learning to predict by the methods of temporal differences","authors":"Richard S Sutton","year":"1988","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sutton, 1996] Richard S Sutton. Generalization in rein- forcement learning: Successful examples using sparse coarse coding. In Advances in neural information process- ing systems, pages 1038-1044, 1996.","title":"Generalization in reinforcement learning: Successful examples using sparse coarse coding","authors":"Richard S Sutton","year":"1996","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Van Seijen et al., 2009] Harm Van Seijen, Hado Van Has- selt, Shimon Whiteson, and Marco Wiering. A theoretical and empirical analysis of expected sarsa. In Adaptive Dy- namic Programming and Reinforcement Learning, 2009. ADPRL'09. IEEE Symposium on, pages 177-184. IEEE, 2009.","title":"A theoretical and empirical analysis of expected sarsa","authors":"Van Seijen","year":"2009","venue":"Adaptive Dynamic Programming and Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Van Seijen et al., 2016] Harm Van Seijen, A Rupam Mah- mood, Patrick M Pilarski, Marlos C Machado, and Richard S Sutton. True online temporal-difference learn- ing. Journal of Machine Learning Research, 17(145):1- 40, 2016.","title":"True online temporal-difference learning","authors":"Van Seijen","year":"2016","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Watkins, 1989] Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. PhD thesis, King's Col- lege, Cambridge, 1989.","title":"Learning from delayed rewards","authors":"Christopher John, Cornish Hellaby Watkins","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}