{"type":"ScholarlyArticle","title":"SVRG for Policy Evaluation with Fewer Gradient Evaluations","authors":[{"name":"Zilun Peng","slug":"zilun-peng","given":"Zilun","family":"Peng","affiliation":"McGill University"},{"name":"Ahmed Touati","slug":"ahmed-touati","given":"Ahmed","family":"Touati","affiliation":"Mila , Université de Montréal"},{"name":"Pascal Vincent","slug":"pascal-vincent","given":"Pascal","family":"Vincent","affiliation":"Mila , Université de Montréal; Facebook AI Research"},{"name":"Doina Precup","slug":"doina-precup","given":"Doina","family":"Precup","affiliation":"McGill University"}],"abstract":"Stochastic variance-reduced gradient (SVRG) is an optimization method originally designed for tackling machine learning problems with a finite sum structure. SVRG was later shown to work for policy evaluation, a problem in reinforcement learning in which one aims to estimate the value function of a given policy. SVRG makes use of gradient estimates at two scales. At the slower scale, SVRG computes a full gradient over the whole dataset, which could lead to prohibitive computation costs. In this work, we show that two variants of SVRG for policy evaluation could significantly diminish the number of gradient calculations while preserving a linear convergence speed. More importantly, our theoretical result implies that one does not need to use the entire dataset in every epoch of SVRG when it is applied to policy evaluation with linear function approximation. Our experiments demonstrate large computational savings provided by the proposed methods.","doi":"10.24963/ijcai.2020/374","url":"https://ijcai.org/proceedings/2020/374","pdf":"https://ijcai.org/proceedings/2020/0374.pdf","markdown":"https://ijcai.org/proceedings/2020/374.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/374","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":374,"pages":{"first":2697,"last":2703},"keywords":["Machine Learning Applications: Applications of Reinforcement Learning","Machine Learning: Reinforcement Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25301/file_25301.m3u8","length_sec":280,"provider":"hls"}],"references":[{"position":1,"raw":"[Baird, 1995] Leemon Baird. Residual algorithms : Rein- forcement learning with function approximation. In Inter- national Conference on Machine Learning, 1995.","title":"Residual algorithms : Reinforcement learning with function approximation","authors":"Leemon Baird","year":"1995","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Balamurugan and Bach, 2016] P. Balamurugan and Francis Bach. Stochastic variance reduction methods for saddle- point problems. In Advances in Neural Information Pro- cessing Systems, 2016.","title":"Stochastic variance reduction methods for saddlepoint problems","authors":"P Balamurugan, Francis Bach","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bertsekas, 2011] Dimitri P Bertsekas. Temporal difference methods for general projected equations. IEEE Transac- tions on Automatic Control, 2011.","title":"Temporal difference methods for general projected equations","authors":"Dimitri P Bertsekas","year":"2011","venue":"IEEE Transactions on Automatic Control","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bhandari et al., 2018] Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. Conference on Learning Theory, 2018.","title":"A finite time analysis of temporal difference learning with linear function approximation","authors":"Jalaj Bhandari, Daniel Russo, Raghav Singal","year":"2018","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Boyan, 2002] Justin Boyan. Technical update: Least-squares temporal difference learning. Machine Learning, 49(2):233- 246, 2002.","title":"Technical update: Least-squares temporal difference learning","authors":"Justin Boyan, Boyan","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bradtke and Barto, 1996] Steven J. Bradtke and Andrew G. Barto. Linear least-squares algorithms for temporal differ- ence learning. Machine Learning, 22(1-3):33-57, 1996.","title":"Linear least-squares algorithms for temporal difference learning","authors":"Steven J Bradtke, Andrew G Barto","year":"1996","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Lud- wig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016. Dalal et al., 2019] Gal Dalal, Balazs Szorenyi, and Gugan Thoppe. A tale of two-timescale reinforcement learning with the tightest finite-time bound, 2019.","title":"A tale of two-timescale reinforcement learning with the tightest finite-time bound","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba, ; Dalal","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dann et al., 2014] Christoph Dann, Gerhard Neumann, and Jan Peters. Policy evaluation with temporal differences: a survey and comparison. Journal of Machine Learning Research, 15(1):809-883, 2014.","title":"Policy evaluation with temporal differences: a survey and comparison","authors":"Dann","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Defazio et al., 2014] Aaron Defazio, Francis Bach, and Si- mon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. In Advances in Neural Information Processing Systems, 2014.","title":"Saga: A fast incremental gradient method with support for non-strongly convex composite objectives","authors":"Aaron Defazio, Francis Bach, Simon Lacoste-Julien","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Du et al., 2017] Simon S. Du, Jianshu Chen, Lihong Li, Lin Xiao, and Dengyong Zhou. Stochastic variance reduction methods for policy evaluation. In International Conference on Machine Learning, 2017.","title":"Stochastic variance reduction methods for policy evaluation","authors":"Simon S Du, Jianshu Chen, Lihong Li, Lin Xiao, Dengyong Zhou","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Harikandeh et al., 2015] Reza Harikandeh, Mo- hamed Osama Ahmed, Alim Virani, Mark Schmidt, Jakub Konečný, and Scott Sallinen. Stop wasting my gradi- ents: Practical svrg. In Advances in Neural Information Processing Systems, 2015.","title":"Stop wasting my gradients: Practical svrg","authors":"Reza Harikandeh, Mohamed Osama Ahmed, Alim Virani, Mark Schmidt, Jakub Konečný, Scott Sallinen","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Johnson and Zhang, 2013] Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in Neural Information Processing Systems, 2013.","title":"Accelerating stochastic gradient descent using predictive variance reduction","authors":"Rie Johnson, Tong Zhang","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Korda and L.A., 2015] Nathaniel Korda and Prashanth L.A. On td(0) with function approximation: Concentration bounds and a centered variant with exponential conver- gence. In International Conference on Machine Learning, 2015.","title":"On td(0) with function approximation: Concentration bounds and a centered variant with exponential convergence","authors":"L A Korda, Nathaniel Korda, L A Prashanth","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lagoudakis and Parr, 2003] Michail G. Lagoudakis and Ronald Parr. Least-squares policy iteration. Journal of Machine Learning Research, pages 1107-1149, 2003.","title":"Least-squares policy iteration","authors":"Michail G Lagoudakis, Ronald Parr","year":"2003","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lei and Jordan, 2017] Lihua Lei and Michael I. Jordan. Less than a single pass: Stochastically controlled stochastic gra- dient method. In International Conference on Artificial Intelligence and Statistics, 2017.","title":"Less than a single pass: Stochastically controlled stochastic gradient method","authors":"Lihua Lei, Michael I Jordan","year":"2017","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2015] Bo Liu, Ji Liu, Mohammad Ghavamzadeh, Sridhar Mahadevan, and Marek Petrik. Finite-sample anal- ysis of proximal gradient td algorithms. In Conference on Uncertainty in Artificial Intelligence, 2015.","title":"Finite-sample analysis of proximal gradient td algorithms","authors":"Bo Liu, Ji Liu, Mohammad Ghavamzadeh, Marek Sridhar Mahadevan, Petrik","year":"2015","venue":"Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Papini et al., 2018] Matteo Papini, Damiano Binaghi, Giuseppe Canonaco, Matteo Pirotta, and Marcello Restelli. Stochastic variance-reduced policy gradient. International Conference on Machine Learning, 2018.","title":"Stochastic variance-reduced policy gradient","authors":"Matteo Papini, Damiano Binaghi, Giuseppe Canonaco, Matteo Pirotta, Marcello Restelli","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Peng et al., 2019] Zilun Peng, Ahmed Touati, Pascal Vin- cent, and Doina Precup. Svrg for policy evaluation with fewer gradient evaluations. CoRR, abs/1906.03704, 2019.","title":"Svrg for policy evaluation with fewer gradient evaluations","authors":"Peng","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998.","title":"Introduction to Reinforcement Learning","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sutton et al., 2008] Richard S. Sutton, Cesba Szepesvári, and Hamid Reza Maei. A convergent o(n) temporal- difference algorithm for off-policy learning with linear function approximation. In Advances in Neural Information Processing Systems, 2008.","title":"A convergent o(n) temporaldifference algorithm for off-policy learning with linear function approximation","authors":"Richard S Sutton, Cesba Szepesvári, Hamid Reza Maei","year":"2008","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sutton et al., 2009] Richard S. Sutton, Hamid Reza Maei, Doina Precup, Shalabh Bhatnagar, David Silver, Cesba Szepesvári, and Eric Wiewiora. Fast gradient-descent meth- ods for temporal-difference learning with linear function approximation. In International Conference on Machine Learning, pages 993-1000, 2009.","title":"Fast gradient-descent methods for temporal-difference learning with linear function approximation","authors":"Richard S Sutton, Reza Hamid, Doina Maei, Shalabh Precup, David Bhatnagar, Cesba Silver, Eric Szepesvári, Wiewiora","year":"2009","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutton, 1988] Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3(1):9- 44, 1988.","title":"Learning to predict by the methods of temporal differences","authors":"Richard S Sutton","year":"1988","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Touati et al., 2018] Ahmed Touati, Pierre-Luc Bacon, Doina Precup, and Pascal Vincent. Convergent tree backup and retrace with function approximation. In International Con- ference on Machine Learning, pages 4962-4971, 2018.","title":"Convergent tree backup and retrace with function approximation","authors":"Ahmed Touati, Pierre-Luc Bacon, Doina Precup, Pascal Vincent","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2017] Yue Wang, Wei Chen, Yuting Liu, Zhi- Ming Ma, and Tie-Yan Liu. Finite sample analysis of the gtd policy evaluation algorithms in markov setting. In Advances in Neural Information Processing Systems, page 5510-5519, 2017.","title":"Finite sample analysis of the gtd policy evaluation algorithms in markov setting","authors":"Yue Wang, Wei Chen, Yuting Liu, Zhi-Ming Ma, Tie-Yan Liu","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2019] Pan Xu, Felicia Gao, and Quanquan Gu. An improved convergence analysis of stochastic variance- reduced policy gradient. In Conference on Uncertainty in Artificial Intelligence, 2019.","title":"An improved convergence analysis of stochastic variancereduced policy gradient","authors":"Xu","year":"2019","venue":"Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2020] Tengyu Xu, Zhe Wang, Yi Zhou, and Ying- bin Liang. Reanalysis of variance reduced temporal differ- ence learning. In International Conference on Learning Representations, 2020.","title":"Reanalysis of variance reduced temporal difference learning","authors":"Tengyu Xu, Zhe Wang, Yi Zhou, Yingbin Liang","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}