{"type":"ScholarlyArticle","title":"Constrained Policy Improvement for Efficient Reinforcement Learning","authors":[{"name":"Elad Sarafian","slug":"elad-sarafian","given":"Elad","family":"Sarafian","affiliation":"Bar-Ilan University , Israel"},{"name":"Aviv Tamar","slug":"aviv-tamar","given":"Aviv","family":"Tamar","affiliation":"Technion , Israel"},{"name":"Sarit Kraus","slug":"sarit-kraus","given":"Sarit","family":"Kraus","affiliation":"Bar-Ilan University , Israel"}],"abstract":"We propose a policy improvement algorithm for Reinforcement Learning (RL) termed Rerouted Behavior Improvement (RBI). RBI is designed to take into account the evaluation errors of the Q-function. Such errors are common in RL when learning the Q-value from finite experience data. Greedy policies or even constrained policy optimization algorithms that ignore these errors may suffer from an improvement penalty (i.e., a policy impairment). To reduce the penalty, the idea of RBI is to attenuate rapid policy changes to actions that were rarely sampled. This approach is shown to avoid catastrophic performance degradation and reduce regret when learning from a batch of transition samples. Through a two-armed bandit example, we show that it also increases data efficiency when the optimal action has a high variance. We evaluate RBI in two tasks in the Atari Learning Environment: (1) learning from observations of multiple behavior policies and (2) iterative RL. Our results demonstrate the advantage of RBI over greedy policies and other constrained policy optimization algorithms both in learning from observations and in RL tasks.","doi":"10.24963/ijcai.2020/396","url":"https://ijcai.org/proceedings/2020/396","pdf":"https://ijcai.org/proceedings/2020/0396.pdf","markdown":"https://ijcai.org/proceedings/2020/396.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/396","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":396,"pages":{"first":2863,"last":2871},"keywords":["Machine Learning: Deep Reinforcement Learning","Machine Learning: Reinforcement Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24986/file_24986.m3u8","length_sec":282,"provider":"hls"}],"references":[{"position":1,"raw":"[Argall et al., 2009] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-483, 2009.","title":"A survey of robot learning from demonstration","authors":"Sonia Brenna D Argall, Manuela Chernova, Brett Veloso, Browning","year":"2009","venue":"Robotics and autonomous systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Auer et al., 2002] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bellemare et al., 2016] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pages 1471-1479, 2016.","title":"Unifying count-based exploration and intrinsic motivation","authors":"Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, Remi Munos","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dann and Brunskill, 2015] Christoph Dann and Emma Brun- skill. Sample complexity of episodic fixed-horizon rein- forcement learning. In Advances in Neural Information Processing Systems, pages 2818-2826, 2015.","title":"Christoph Dann and Emma Brunskill. Sample complexity of episodic fixed-horizon reinforcement learning","authors":"Brunskill Dann","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fruit et al., 2017] Ronan Fruit, Matteo Pirotta, Alessandro Lazaric, and Emma Brunskill. Regret minimization in mdps with options without prior knowledge. In Advances in Neural Information Processing Systems, pages 3166- 3176, 2017.","title":"Regret minimization in mdps with options without prior knowledge","authors":"Ronan Fruit, Matteo Pirotta, Alessandro Lazaric, Emma Brunskill","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fujimoto et al., 2018] Scott Fujimoto, Herke van Hoof, and Dave Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:1802.09477, 2018.","title":"Addressing function approximation error in actor-critic methods","authors":"Fujimoto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Garcıa and Fernández, 2015] Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforce- ment learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.","title":"Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning","authors":"Fernández Garcıa","year":"2015","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018.","title":"Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor","authors":"Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Hado V Hasselt. Double q-learning. In Ad- vances in Neural Information Processing Systems, pages 2613-2621, 2010.","title":"Double q-learning","authors":"V Hado, Hasselt","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hessel et al., 2017] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. arXiv preprint arXiv:1710.02298, 2017.","title":"Rainbow: Combining improvements in deep reinforcement learning","authors":"Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, David Silver","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hester et al., 2018] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Hor- gan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.","title":"Deep q-learning from demonstrations","authors":"Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Horgan et al., 2018] Dan Horgan, John Quan, David Bud- den, Gabriel Barth-Maron, Matteo Hessel, Hado Van Has- selt, and David Silver. Distributed prioritized experience replay. arXiv preprint arXiv:1803.00933, 2018.","title":"Distributed prioritized experience replay","authors":"Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, David Silver","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kakade and Langford, 2002] Sham Kakade and John Lang- ford. Approximately optimal approximate reinforcement learning. In ICML, volume 2, pages 267-274, 2002.","title":"Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning","authors":"Langford Kakade","year":"2002","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kearns and Singh, 2000] Michael J Kearns and Satinder P Singh. Bias-variance error bounds for temporal difference updates. In COLT, pages 142-147. Citeseer, 2000.","title":"Bias-variance error bounds for temporal difference updates","authors":"J Michael, Kearns, P Satinder, Singh","year":"2000","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krause and Ong, 2011] Andreas Krause and Cheng S Ong. Contextual gaussian process bandit optimization. In Ad- vances in Neural Information Processing Systems, pages 2447-2455, 2011.","title":"Contextual gaussian process bandit optimization","authors":"Ong ; ] Andreas Krause, Cheng S Krause, Ong","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kurin et al., 2017] Vitaly Kurin, Sebastian Nowozin, Katja Hofmann, Lucas Beyer, and Bastian Leibe. The atari grand challenge dataset. arXiv preprint arXiv:1705.10998, 2017.","title":"The atari grand challenge dataset","authors":"Sebastian Vitaly Kurin, Katja Nowozin, Lucas Hofmann, Bastian Beyer, Leibe","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Metelli et al., 2018] Alberto Maria Metelli, Matteo Papini, Francesco Faccio, and Marcello Restelli. Policy optimiza- tion via importance sampling. In Advances in Neural Infor- mation Processing Systems, pages 5442-5454, 2018.","title":"Policy optimization via importance sampling","authors":"Alberto Maria Metelli, Matteo Papini, Francesco Faccio, Marcello Restelli","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015. [","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"OpenAI, 2018 accessed May 2020] OpenAI. Openai five. https://blog.openai.com/openai-five/, 2018 -accessed May 2020.","title":"OpenAI. Openai five","authors":"Openai","year":"2018","venue":null,"doi":null,"url":"https://blog.openai.com/openai-five/,2018-","internal_url":null},{"position":20,"raw":"[Papini et al., 2019] Matteo Papini, Matteo Pirotta, and Mar- cello Restelli. Smoothing policies and safe policy gradients. arXiv preprint arXiv:1905.03231, 2019.","title":"Smoothing policies and safe policy gradients","authors":"Matteo Papini, Matteo Pirotta, Marcello Restelli","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pirotta et al., 2013a] Matteo Pirotta, Marcello Restelli, and Luca Bascetta. Adaptive step-size for policy gradient meth- ods. In Advances in Neural Information Processing Sys- tems, pages 1394-1402, 2013.","title":"Adaptive step-size for policy gradient methods","authors":"Pirotta","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pirotta et al., 2013b] Matteo Pirotta, Marcello Restelli, Alessio Pecorino, and Daniele Calandriello. Safe policy it- eration. In International Conference on Machine Learning, pages 307-315, 2013.","title":"Safe policy iteration","authors":"Pirotta","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Puterman, 2014] Martin L Puterman. Markov decision pro- cesses: discrete stochastic dynamic programming. John Wiley & Sons, 2014.","title":"Markov decision processes: discrete stochastic dynamic programming","authors":"Martin L Puterman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pages 1889-1897, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shalev-Shwartz et al., 2016] Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, rein- forcement learning for autonomous driving. arXiv preprint arXiv:1610.03295, 2016.","title":"Safe, multi-agent, reinforcement learning for autonomous driving","authors":"Shai Shalev-Shwartz, Shaked Shammah, Amnon Shashua","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017.","title":"Mastering the game of go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sutton and Barto, 2017] RS Sutton and AG Barto. Rein- forcement learning: An introduction, (complete draft), 2017.","title":"Reinforcement learning: An introduction","authors":"Barto ; Rs Sutton, A G Sutton, Barto","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sutton et al., 2000] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approx- imation. In Advances in neural information processing systems, pages 1057-1063, 2000.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David A Richard S Sutton, Mcallester, P Satinder, Yishay Singh, Mansour","year":"2000","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Thomas et al., 2015] Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High confidence policy improvement. In International Conference on Machine Learning, pages 2380-2388, 2015.","title":"High confidence policy improvement","authors":"Philip Thomas, Georgios Theocharous, Mohammad Ghavamzadeh","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Van Hasselt et al., 2016] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In AAAI, volume 2, page 5. Phoenix, AZ, 2016.","title":"Deep reinforcement learning with double q-learning","authors":"Van Hasselt","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Vanderbei and others, 2015] Robert J Vanderbei et al. Lin- ear programming. Springer, 2015.","title":"Linear programming","authors":"J Robert, Vanderbei","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Vuong et al., 2019] Quan Vuong, Yiming Zhang, and Keith W. Ross. SUPERVISED POLICY UPDATE. In In- ternational Conference on Learning Representations, 2019.","title":"SUPERVISED POLICY UPDATE","authors":"Quan Vuong, Yiming Zhang, Keith W Ross","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wang et al., 2015] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot, and Nando De Freitas. Dueling network architectures for deep reinforcement learn- ing. arXiv preprint arXiv:1511.06581, 2015.","title":"Dueling network architectures for deep reinforcement learning","authors":"Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot, Nando De Freitas","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Watkins and Dayan, 1992] Christopher JCH Watkins and Pe- ter Dayan. Q-learning. Machine learning, 8(3-4):279-292, 1992.","title":"Christopher JCH Watkins and Peter Dayan. Q-learning","authors":"Dayan Watkins","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}