{"type":"ScholarlyArticle","title":"Metatrace Actor-Critic: Online Step-Size Tuning by Meta-gradient Descent for Reinforcement Learning Control","authors":[{"name":"Kenny Young","slug":"kenny-young","given":"Kenny","family":"Young","affiliation":"Borealis AI , Edmonton , Alberta , Canada; University of Alberta , Edmonton , Alberta , Canada"},{"name":"Baoxiang Wang","slug":"baoxiang-wang","given":"Baoxiang","family":"Wang","affiliation":"Borealis AI , Edmonton , Alberta , Canada"},{"name":"Matthew E. Taylor","slug":"matthew-e-taylor","given":"Matthew E.","family":"Taylor","affiliation":"Borealis AI , Edmonton , Alberta , Canada"}],"abstract":"Reinforcement learning (RL) has had many successes, but significant hyperparameter tuning is commonly required to achieve good performance. Furthermore, when nonlinear function approximation is used, non-stationarity in the state representation can lead to learning instability.  A variety of techniques exist to combat this --- most notably experience replay or the use of parallel actors. These techniques stabilize learning by making the RL problem more similar to the supervised setting. However, they come at the cost of moving away from the RL problem as it is typically formulated, that is, a single agent learning online without maintaining a large database of training examples. To address these issues, we propose Metatrace, a meta-gradient descent based algorithm to tune the step-size online. Metatrace leverages the structure of eligibility traces, and works for both tuning a scalar step-size and a respective step-size for each parameter. We empirically evaluate Metatrace for actor-critic on the Arcade Learning Environment. Results show Metatrace can speed up learning, and improve performance in non-stationary settings.","doi":"10.24963/ijcai.2019/581","url":"https://ijcai.org/proceedings/2019/581","pdf":"https://ijcai.org/proceedings/2019/0581.pdf","markdown":"https://ijcai.org/proceedings/2019/581.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/581","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":581,"pages":{"first":4185,"last":4191},"keywords":["Machine Learning: Deep Learning","Machine Learning: Online Learning","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Bellemare et al., 2013] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"Yavar Marc G Bellemare, Joel Naddaf, Michael Veness, Bowling","year":"2013","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dabney, 2014] William C Dabney. Adaptive step-sizes for reinforcement learning. PhD thesis, University of Mas- sachusetts Amherst, 2014.","title":"Adaptive step-sizes for reinforcement learning","authors":"C William, Dabney","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Degris et al., 2012] Thomas Degris, Patrick M Pilarski, and Richard S Sutton. Model-free reinforcement learning with continuous action in practice. In American Control Con- ference (ACC), 2012, pages 2177-2182. IEEE, 2012.","title":"Model-free reinforcement learning with continuous action in practice","authors":"Thomas Degris, Patrick M Pilarski, Richard S Sutton","year":"2012","venue":"American Control Conference (ACC)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Elfwing et al., 2018] Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 2018.","title":"Sigmoid-weighted linear units for neural network function approximation in reinforcement learning","authors":"Stefan Elfwing, Eiji Uchibe, Kenji Doya","year":"2018","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Espeholt et al., 2018] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with impor- tance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561, 2018.","title":"Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures","authors":"Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kearney et al., 2019] Alex Kearney, Vivek Veeriah, Jaden Tavnik, Patrick M Pillarski, and Richard S Sutton. Learning feature relevance through step size adapta- tion in temporal-difference learning. arXiv preprint arxiv:1903.03252, 2019.","title":"Learning feature relevance through step size adaptation in temporal-difference learning","authors":"Alex Kearney, Vivek Veeriah, Jaden Tavnik, Patrick M Pillarski, Richard S Sutton","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Machado et al., 2017] Marlos C Machado, Marc G Belle- mare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environ- ment: Evaluation protocols and open problems for general agents. arXiv preprint arXiv:1709.06009, 2017.","title":"Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents","authors":"C Marlos, Marc G Machado, Erik Bellemare, Joel Talvitie, Matthew Veness, Michael Hausknecht, Bowling","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mahmood et al., 2012] Ashique Rupam Mahmood, Richard S Sutton, Thomas Degris, and Patrick M Pilarski. Tuning-free step-size adaptation. In Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on, pages 2121-2124. IEEE, 2012.","title":"Tuning-free step-size adaptation","authors":"Ashique Rupam Mahmood, Richard S Sutton, Thomas Degris, Patrick M Pilarski","year":"2012","venue":"Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In In- ternational conference on machine learning, pages 1928- 1937, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Schmidhuber, 1987] Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München., 1987.","title":"Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta","authors":"Jürgen Schmidhuber","year":"1987","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Schulman et al., 2015] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advan- tage estimation. arXiv preprint arXiv:1506.02438, 2015.","title":"Highdimensional continuous control using generalized advantage estimation","authors":"John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sutton et al., 2000] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approxi- mation. In Advances in neural information processing sys- tems, pages 1057-1063, 2000.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David A Richard S Sutton, Mcallester, P Satinder, Yishay Singh, Mansour","year":"2000","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sutton, 1992] Richard S Sutton. Adapting bias by gradi- ent descent: An incremental version of delta-bar-delta. In AAAI, pages 171-176, 1992.","title":"Adapting bias by gradient descent: An incremental version of delta-bar-delta","authors":"Richard S Sutton","year":"1992","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tieleman and Hinton, 2012] Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-RMSProp: Divide the gradient by a running average of its recent magnitude. Coursera: Neural networks for machine learning, 4(2):26-31, 2012.","title":"Lecture 6.5-RMSProp: Divide the gradient by a running average of its recent magnitude","authors":"Hinton Tieleman, Tijmen Tieleman, Geoffrey Hinton","year":"2012","venue":"Coursera: Neural networks for machine learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Van Hasselt et al., 2016] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with dou- ble q-learning. In AAAI, volume 2, page 5. Phoenix, AZ, 2016.","title":"Deep reinforcement learning with double q-learning","authors":"Van Hasselt","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Watkins and Dayan, 1992] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279- 292, 1992.","title":"Q-learning","authors":"Jch Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu et al., 2018] Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 2402- 2413.","title":"Meta-gradient reinforcement learning","authors":"Zhongwen Xu, David Hado P Van Hasselt, Silver","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"Curran Associates, Inc., 2018.","title":null,"authors":"Curran Associates, Inc","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}