{"type":"ScholarlyArticle","title":"On Principled Entropy Exploration in Policy Optimization","authors":[{"name":"Jincheng Mei","slug":"jincheng-mei","given":"Jincheng","family":"Mei","affiliation":"University of Alberta"},{"name":"Chenjun Xiao","slug":"chenjun-xiao","given":"Chenjun","family":"Xiao","affiliation":"University of Alberta"},{"name":"Ruitong Huang","slug":"ruitong-huang","given":"Ruitong","family":"Huang","affiliation":"Borealis AI Lab"},{"name":"Dale Schuurmans","slug":"dale-schuurmans","given":"Dale","family":"Schuurmans","affiliation":"University of Alberta"},{"name":"Martin Müller","slug":"martin-muller","given":"Martin","family":"Müller","affiliation":"University of Alberta"}],"abstract":"In this paper, we investigate Exploratory Conservative Policy Optimization (ECPO), a policy optimization strategy that improves exploration behavior while assuring monotonic progress in a principled objective. ECPO conducts maximum entropy exploration within a mirror descent framework, but updates policies using reversed KL projection. This formulation bypasses undesirable mode seeking behavior and avoids premature convergence to sub-optimal policies, while still supporting strong theoretical properties such as guaranteed policy improvement. Experimental evaluations demonstrate that the proposed method significantly improves practical exploration and surpasses the empirical performance of state-of-the art policy optimization methods in a set of benchmark tasks.","doi":"10.24963/ijcai.2019/434","url":"https://ijcai.org/proceedings/2019/434","pdf":"https://ijcai.org/proceedings/2019/0434.pdf","markdown":"https://ijcai.org/proceedings/2019/434.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/434","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":434,"pages":{"first":3130,"last":3136},"keywords":["Machine Learning Applications: Applications of Reinforcement Learning","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Abdolmaleki et al., 2018] Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy opti- misation. In ICLR, 2018.","title":"Maximum a posteriori policy optimisation","authors":"Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, Martin Riedmiller","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Beck and Teboulle, 2003] Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient meth- ods for convex optimization. Operations Research Letters, 31(3):167-175, 2003.","title":"Mirror descent and nonlinear projected subgradient methods for convex optimization","authors":"Amir Beck, Marc Teboulle","year":"2003","venue":"Operations Research Letters","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Lud- wig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Daniel et al., 2012] Christian Daniel, Gerhard Neumann, and Jan Peters. Hierarchical relative entropy policy search. In Artificial Intelligence and Statistics, pages 273-281, 2012.","title":"Hierarchical relative entropy policy search","authors":"Daniel","year":"2012","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deisenroth et al., 2013] Marc Peter Deisenroth, Gerhard Neumann, Jan Peters, et al. A survey on policy search for robotics. Foundations and Trends R in Robotics, 2(1- 2):1-142, 2013.","title":"A survey on policy search for robotics","authors":"Marc Peter Deisenroth, Gerhard Neumann, Jan Peters","year":"2013","venue":"Foundations and Trends R in Robotics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fox et al., 2015] Roy Fox, Ari Pakman, and Naftali Tishby. Taming the noise in reinforcement learning via soft updates. arXiv preprint arXiv:1512.08562, 2015.","title":"Taming the noise in reinforcement learning via soft updates","authors":"Roy Fox, Ari Pakman, Naftali Tishby","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fujimoto et al., 2018] Scott Fujimoto, Herke van Hoof, and Dave Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:1802.09477, 2018.","title":"Addressing function approximation error in actor-critic methods","authors":"Scott Fujimoto, Herke Van Hoof, Dave Meger","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Haarnoja et al., 2017] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learn- ing with deep energy-based policies. arXiv preprint arXiv:1702.08165, 2017.","title":"Reinforcement learning with deep energy-based policies","authors":"Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, Sergey Levine","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018.","title":"Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor","authors":"Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Levine, 2018] Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909, 2018.","title":"Reinforcement learning and control as probabilistic inference: Tutorial and review","authors":"Sergey Levine","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous con- trol with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Montgomery and Levine, 2016] William H Montgomery and Sergey Levine. Guided policy search via approxi- mate mirror descent. In Advances in Neural Information Processing Systems, pages 4008-4016, 2016.","title":"Guided policy search via approximate mirror descent","authors":"H William, Sergey Montgomery, Levine","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Murphy, 2012] Kevin P Murphy. Machine learning: a prob- abilistic perspective. Cambridge, MA, 2012.","title":"Machine learning: a probabilistic perspective","authors":"Kevin P Murphy","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nachum et al., 2017a] Ofir Nachum, Mohammad Norouzi, and Dale Schuurmans. Improving policy gradient by ex- ploring under-appreciated rewards. In ICLR, 2017.","title":"Improving policy gradient by exploring under-appreciated rewards","authors":"Nachum","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nachum et al., 2017b] Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap be- tween value and policy based reinforcement learning. In Advances in Neural Information Processing Systems, pages 2772-2782, 2017.","title":"Bridging the gap between value and policy based reinforcement learning","authors":"Nachum","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nachum et al., 2017c] Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Trust-pcl: An off-policy trust region method for continuous control. In ICLR, 2017. [Nemirovskii et al., 1983] Arkadii Nemirovskii, David Borisovich Yudin, and Edgar Ronald Daw- son. Problem complexity and method efficiency in optimization. 1983.","title":"Trust-pcl: An off-policy trust region method for continuous control","authors":"Nachum","year":"1983","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Neu et al., 2017] Gergely Neu, Anders Jonsson, and Vicenc ¸ Gómez. A unified view of entropy-regularized markov de- cision processes. arXiv preprint arXiv:1705.07798, 2017.","title":"A unified view of entropy-regularized markov decision processes","authors":"Gergely Neu, Anders Jonsson, Vicenc, Gómez","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Norouzi et al., 2016] Mohammad Norouzi, Samy Bengio, Navdeep Jaitly, Mike Schuster, Yonghui Wu, Dale Schu- urmans, et al. Reward augmented maximum likelihood for neural structured prediction. In Advances In Neural Information Processing Systems, pages 1723-1731, 2016.","title":"Reward augmented maximum likelihood for neural structured prediction","authors":"Mohammad Norouzi, Samy Bengio, Navdeep Jaitly, Mike Schuster, Yonghui Wu, Dale Schuurmans","year":"2016","venue":"Advances In Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Owen, 2013] Art B. Owen. Monte Carlo theory, methods and examples. 2013.","title":"Monte Carlo theory, methods and examples","authors":"Art B Owen","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Peters et al., 2010] Jan Peters, Katharina Mülling, and Yasemin Altun. Relative entropy policy search. In AAAI, pages 1607-1612. Atlanta, 2010.","title":"Relative entropy policy search","authors":"Peters","year":"2010","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pages 1889-1897, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schulman et al., 2017a] John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q- learning. arXiv preprint arXiv:1704.06440, 2017.","title":"Equivalence between policy gradients and soft qlearning","authors":"Schulman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schulman et al., 2017b] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"Schulman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sutton et al., 1998] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction. MIT press, 1998.","title":"Reinforcement learning: An introduction","authors":"Andrew G Richard S Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tangkaratt et al., 2017] Voot Tangkaratt, Abbas Abdol- maleki, and Masashi Sugiyama. Guide actor-critic for con- tinuous control. arXiv preprint arXiv:1705.07606, 2017.","title":"Guide actor-critic for continuous control","authors":"Voot Tangkaratt, Abbas Abdolmaleki, Masashi Sugiyama","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pages 5026-5033. IEEE, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Van Hoof et al., 2015] Herke Van Hoof, Jan Peters, and Ger- hard Neumann. Learning of non-parametric control policies with high-dimensional state features. In Artificial Intelli- gence and Statistics, pages 995-1003, 2015.","title":"Learning of non-parametric control policies with high-dimensional state features","authors":"Van Hoof","year":"2015","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Williams and Peng, 1991] Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241-268, 1991.","title":"Function optimization using connectionist reinforcement learning algorithms","authors":"J Ronald, Jing Williams, Peng","year":"1991","venue":"Connection Science","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine Learning, 8(3-4):229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}