{"type":"ScholarlyArticle","title":"Autoregressive Policies for Continuous Control Deep Reinforcement Learning","authors":[{"name":"Dmytro Korenkevych","slug":"dmytro-korenkevych","given":"Dmytro","family":"Korenkevych","affiliation":""},{"name":"A. Rupam Mahmood","slug":"a-rupam-mahmood","given":"A. Rupam","family":"Mahmood","affiliation":""},{"name":"Gautham Vasan","slug":"gautham-vasan","given":"Gautham","family":"Vasan","affiliation":""},{"name":"James Bergstra","slug":"james-bergstra","given":"James","family":"Bergstra","affiliation":""}],"abstract":"Reinforcement learning algorithms rely on exploration to discover new behaviors, which is typically achieved by following a stochastic policy. In continuous control tasks, policies with a Gaussian distribution have been widely adopted. Gaussian exploration however does not result in smooth trajectories that generally correspond to safe and rewarding behaviors in practical tasks. In addition, Gaussian policies do not result in an effective exploration of an environment and become increasingly inefficient as the action rate increases. This contributes to a low sample efficiency often observed in learning continuous control tasks. We introduce a family of stationary autoregressive (AR) stochastic processes to facilitate exploration in continuous control domains. We show that proposed processes possess two desirable features: subsequent process observations are temporally coherent with continuously adjustable degree of coherence, and the process stationary distribution is standard normal. We derive an autoregressive policy (ARP) that implements such processes maintaining the standard agent-environment interface. We show how ARPs can be easily used with the existing off-the-shelf learning algorithms. Empirically we demonstrate that using ARPs results in improved exploration and sample efficiency in both simulated and real world domains, and, furthermore, provides smooth exploration trajectories that enable safe operation of robotic hardware.","doi":"10.24963/ijcai.2019/382","url":"https://ijcai.org/proceedings/2019/382","pdf":"https://ijcai.org/proceedings/2019/0382.pdf","markdown":"https://ijcai.org/proceedings/2019/382.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/382","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":382,"pages":{"first":2754,"last":2762},"keywords":["Robotics: Learning in Robotics","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Andrychowicz et al., 2018] Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand ma- nipulation. arXiv preprint arXiv:1808.00177, 2018.","title":"Learning dexterous in-hand manipulation","authors":"Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob Mcgrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Benbrahim and Franklin, 1997] Hamid Benbrahim and Judy A Franklin. Biped dynamic walking using rein- forcement learning. Robotics and Autonomous Systems, 22(3-4):283-302, 1997.","title":"Hamid Benbrahim and Judy A Franklin. Biped dynamic walking using reinforcement learning","authors":"Franklin Benbrahim","year":"1997","venue":"Robotics and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brockwell et al., 2002] Peter J Brockwell, Richard A Davis, and Matthew V Calder. Introduction to time series and forecasting, volume 2. Springer, 2002.","title":"Introduction to time series and forecasting","authors":"J Peter, Richard A Brockwell, Matthew V Davis, Calder","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Burda et al., 2018] Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. arXiv preprint arXiv:1808.04355, 2018.","title":"Large-scale study of curiosity-driven learning","authors":"Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, Alexei A Efros","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fortunato et al., 2017] Meire Fortunato, Moham- mad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. arXiv preprint arXiv:1706.10295, 2017.","title":"Noisy networks for exploration","authors":"Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018.","title":"Soft actor-critic algorithms and applications","authors":"Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[van Hoof et al., 2017] Herke van Hoof, Daniel Tanneberg, and Jan Peters. Generalized exploration in policy search. Machine Learning, 106(9-10):1705-1724, 2017.","title":"Generalized exploration in policy search","authors":"Van Hoof","year":"2017","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kalashnikov et al., 2018] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vin- cent Vanhoucke, and Sergey Levine. Scalable deep rein- forcement learning for vision-based robotic manipulation. In Aude Billard, Anca Dragan, Jan Peters, and Jun Mo- rimoto, editors, Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pages 651-673. PMLR, 29-31 Oct 2018.","title":"Scalable deep reinforcement learning for vision-based robotic manipulation","authors":"Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, Sergey Levine","year":"2018","venue":"Proceedings of The 2nd Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous con- trol with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mahmood et al., 2018a] A Rupam Mahmood, Dmytro Ko- renkevych, Brent J Komer, and James Bergstra. Setting up a reinforcement learning task with a real-world robot. In 2018 IEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS), pages 4635-4640. IEEE, 2018.","title":"Setting up a reinforcement learning task with a real-world robot","authors":"Mahmood","year":"2018","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mahmood et al., 2018b] A. Rupam Mahmood, Dmytro Ko- renkevych, Gautham Vasan, William Ma, and James Bergstra. Benchmarking reinforcement learning algo- rithms on real-world robots. In CoRL, 2018.","title":"Benchmarking reinforcement learning algorithms on real-world robots","authors":"Mahmood","year":"2018","venue":"CoRL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Metz et al., 2017] Luke Metz, Julian Ibarz, Navdeep Jaitly, and James Davidson. Discrete sequential prediction of continuous actions for deep rl. arXiv preprint arXiv:1705.05035, 2017.","title":"Discrete sequential prediction of continuous actions for deep rl","authors":"Metz","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In In- ternational conference on machine learning, pages 1928- 1937, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ng et al., 1999] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, vol- ume 99, pages 278-287, 1999.","title":"Policy invariance under reward transformations: Theory and application to reward shaping","authors":"Daishi Andrew Y Ng, Stuart Harada, Russell","year":"1999","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Oudeyer et al., 2007] Pierre-Yves Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions on evo- lutionary computation, 11(2):265-286, 2007.","title":"Intrinsic motivation systems for autonomous mental development","authors":"Oudeyer","year":"2007","venue":"IEEE transactions on evolutionary computation","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pathak et al., 2017] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning (ICML), volume 2017, 2017.","title":"Curiosity-driven exploration by self-supervised prediction","authors":"Deepak Pathak, Pulkit Agrawal, Alexei A Efros, Trevor Darrell","year":"2017","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Peters and Schaal, 2007] Jan Peters and Stefan Schaal. Re- inforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th inter- national conference on Machine learning, pages 745-750. ACM, 2007.","title":"Reinforcement learning by reward-weighted regression for operational space control","authors":"Schaal Peters, Jan Peters, Stefan Schaal","year":"2007","venue":"Proceedings of the 24th international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Peters and Schaal, 2008] Jan Peters and Stefan Schaal. Re- inforcement learning of motor skills with policy gradients. Neural networks, 21(4):682-697, 2008.","title":"Reinforcement learning of motor skills with policy gradients","authors":"Schaal Peters, Jan Peters, Stefan Schaal","year":"2008","venue":"Neural networks","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Plappert et al., 2017] Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. arXiv preprint arXiv:1706.01905, 2017.","title":"Parameter space noise for exploration","authors":"Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Xi Richard Y Chen, Tamim Chen, Pieter Asfour, Marcin Abbeel, Andrychowicz","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Plappert et al., 2018] Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv:1802.09464, 2018.","title":"Multi-goal reinforcement learning: Challenging robotics environments and request for research","authors":"Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob Mcgrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Puterman, 2014] Martin L Puterman. Markov decision pro- cesses: discrete stochastic dynamic programming. John Wiley & Sons, 2014.","title":"Markov decision processes: discrete stochastic dynamic programming","authors":"Martin L Puterman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pages 1889-1897, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sutton et al., 2000] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approxi- mation. In Advances in neural information processing sys- tems, pages 1057-1063, 2000.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David A Richard S Sutton, Mcallester, P Satinder, Yishay Singh, Mansour","year":"2000","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tallec et al., 2019] Corentin Tallec, Léonard Blier, and Yann Ollivier. Making deep q-learning methods robust to time discretization. arXiv preprint arXiv:1901.09732, 2019.","title":"Making deep q-learning methods robust to time discretization","authors":"Corentin Tallec, Léonard Blier, Yann Ollivier","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Vinyals et al., 2017] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782, 2017.","title":"Starcraft ii: A new challenge for reinforcement learning","authors":"Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Sasha Alexander, Michelle Vezhnevets, Alireza Yeo, Heinrich Makhzani, John Küttler, Julian Agapiou, Schrittwieser","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wawrzynski, 2015] Pawel Wawrzynski. Control policy with autocorrelated noise in reinforcement learning for robotics. International Journal of Machine Learning and Computing, 5(2):91, 2015.","title":"Control policy with autocorrelated noise in reinforcement learning for robotics","authors":"Pawel Wawrzynski","year":"2015","venue":"International Journal of Machine Learning and Computing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}