{"type":"ScholarlyArticle","title":"Conditions on Features for Temporal Difference-Like Methods to Converge","authors":[{"name":"Marcus Hutter","slug":"marcus-hutter","given":"Marcus","family":"Hutter","affiliation":"Australian National University"},{"name":"Samuel Yang-Zhao","slug":"samuel-yang-zhao","given":"Samuel","family":"Yang-Zhao","affiliation":"Australian National University"},{"name":"Sultan Javed Majeed","slug":"sultan-javed-majeed","given":"Sultan Javed","family":"Majeed","affiliation":"Australian National University"}],"abstract":"The convergence of many reinforcement learning (RL) algorithms with linear function approximation has been investigated extensively but most proofs assume that these methods converge to a unique solution. In this paper, we provide a complete characterization of non-uniqueness issues for a large class of reinforcement learning algorithms, simultaneously unifying many counter-examples to convergence in a theoretical framework. We achieve this by proving a new condition on features that can determine whether the convergence assumptions are valid or non-uniqueness holds. We consider a general class of RL methods, which we call natural algorithms, whose solutions are characterized as the fixed point of a projected Bellman equation. Our main result proves that natural algorithms converge to the correct solution if and only if all the value functions in the approximation space satisfy a certain shape. This implies that natural algorithms are, in general, inherently prone to converge to the wrong solution for most feature choices even if the value function can be represented exactly. Given our results, we show that state aggregation-based features are a safe choice for natural algorithms and also provide a condition for finding convergent algorithms under other feature constructions.","doi":"10.24963/ijcai.2019/357","url":"https://ijcai.org/proceedings/2019/357","pdf":"https://ijcai.org/proceedings/2019/0357.pdf","markdown":"https://ijcai.org/proceedings/2019/357.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/357","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":357,"pages":{"first":2570,"last":2577},"keywords":["Machine Learning: Feature Selection ; Learning Sparse Models","Machine Learning: Reinforcement Learning","Machine Learning: Learning Theory"],"videos":[],"references":[{"position":1,"raw":"[Baird, 1995] Leemon Baird. Residual Algorithms: Rein- forcement Learning with Function Approximation. Pro- ceedings of the Twelfth International Conference on Ma- chine Learning, pages 30-37, 1995.","title":"Residual Algorithms: Reinforcement Learning with Function Approximation","authors":"Leemon Baird","year":"1995","venue":"Proceedings of the Twelfth International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bertsekas, 1995] Dimitri P. Bertsekas. A Counterexample to Temporal Difference Learning. Neural Computation, 7:270-279, 1995.","title":"A Counterexample to Temporal Difference Learning","authors":"Dimitri P Bertsekas","year":"1995","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bertsekas, 2011] Dimitri P. Bertsekas. Dynamic Program- ming and Optimal Control 3rd Edition, Volume II. Mas- sachusetts Institute of Technology, 2011.","title":"Dynamic Programming and Optimal Control 3rd Edition","authors":"Dimitri P Bertsekas","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Boyan and Moore, 1995] Justin A. Boyan and Andrew W. Moore. Generalization in Reinforcement Learning: Safely Approximating the Value Function. pages 369-376, 1995.","title":"Generalization in Reinforcement Learning: Safely Approximating the Value Function","authors":"Justin A Boyan, Andrew W Moore","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gordon, 1995] Geoffrey J. Gordon. Stable Function Ap- proximation in Dynamic Programming. Proceedings of the Twelfth International Conference on Machine Learn- ing, pages 261-268, 1995.","title":"Stable Function Approximation in Dynamic Programming","authors":"Geoffrey J Gordon","year":"1995","venue":"Proceedings of the Twelfth International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hutter et al., 2019] Marcus Hutter, Samuel Yang-Zhao, and Sultan Javed Majeed. Conditions on Features for Tem- poral Difference-Like Methods to Converge. 2019. arXiv:1905.11702.","title":"Conditions on Features for Temporal Difference-Like Methods to Converge","authors":"Marcus Hutter, Samuel Yang-Zhao, Sultan Javed Majeed","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Maei, 2011] Hamid Reza Maei. Gradient Temporal- Difference Learning Algorithms. PhD thesis, University of Alberta, 2011.","title":"Gradient Temporal-Difference Learning Algorithms","authors":"Reza Hamid, Maei","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Scherrer, 2010] Bruno Scherrer. Should one compute the Temporal Difference fix point or minimize the Bellman Residual? The unified oblique projection view. 2010. arXiv:1011.4362.","title":"Should one compute the Temporal Difference fix point or minimize the Bellman Residual? The unified oblique projection view","authors":"Bruno Scherrer","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Sutton and Barto, 2018] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 2018.","title":"Reinforcement Learning: An Introduction","authors":"Richard S Sutton, Andrew G Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Sutton et al., 2009] Richard S. Sutton, Hamid Reza Maei, Doina Precup, Shalabh Bhatnagar, David Silver, Csaba Szepesvári, and Eric Wiewiora. Fast Gradient Descent Methods for Temporal-Difference Learning With Linear Function Approximation. Proceedings of the 26th Annual International Conference on Machine Learning, pages 993-1000, 2009.","title":"Fast Gradient Descent Methods for Temporal-Difference Learning With Linear Function Approximation","authors":"Richard S Sutton, Reza Hamid, Doina Maei, Shalabh Precup, David Bhatnagar, Csaba Silver, Eric Szepesvári, Wiewiora","year":"2009","venue":"Proceedings of the 26th Annual International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Sutton et al., 2016] Richard S. Sutton, Ashique Rupam Mahmood, and Martha White. An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning. Journal of Machine Learning Research, 17:1-29, 2016.","title":"An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning","authors":"Richard S Sutton, Ashique Rupam Mahmood, Martha White","year":"2016","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Tsitsiklis and Van Roy, 1996] John N. Tsitsiklis and Ben- jamin Van Roy. Feature-Based Methods for Large Scale Dynamic Programming. Machine Learning, 22:59-94, 1996.","title":"Feature-Based Methods for Large Scale Dynamic Programming","authors":"John N Tsitsiklis, Benjamin Van, Roy","year":"1996","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Tsitsiklis and Van Roy, 1997] John N. Tsitsiklis and Ben- jamin Van Roy. An Analysis of Temporal-Difference Learning with Function Approximation. IEEE Transac- tions on Automatic Control, 42(5):674-690, 1997.","title":"An Analysis of Temporal-Difference Learning with Function Approximation","authors":"John N Tsitsiklis, Benjamin Van, Roy","year":"1997","venue":"IEEE Transactions on Automatic Control","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}