{"type":"ScholarlyArticle","title":"Monte Carlo Tree Search for Policy Optimization","authors":[{"name":"Xiaobai Ma","slug":"xiaobai-ma","given":"Xiaobai","family":"Ma","affiliation":"Aeronautics and Astronautics Department , Stanford University"},{"name":"Katherine Driggs-Campbell","slug":"katherine-driggs-campbell","given":"Katherine","family":"Driggs-Campbell","affiliation":"Electrical and Computer Engineering Department , University of Illinois Urbana-Champaign"},{"name":"Zongzhang Zhang","slug":"zongzhang-zhang","given":"Zongzhang","family":"Zhang","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University"},{"name":"Mykel J. Kochenderfer","slug":"mykel-j-kochenderfer","given":"Mykel J.","family":"Kochenderfer","affiliation":"Aeronautics and Astronautics Department , Stanford University"}],"abstract":"Gradient-based methods are often used for policy optimization in deep reinforcement learning, despite being vulnerable to local optima and saddle points. Although gradient-free methods (e.g., genetic algorithms or evolution strategies) help mitigate these issues, poor initialization and local optima are still concerns in highly nonconvex spaces. This paper presents a method for policy optimization based on Monte-Carlo tree search and gradient-free optimization. Our method, called Monte-Carlo tree search for policy optimization (MCTSPO), provides a better exploration-exploitation trade-off through the use of the upper confidence bound heuristic. We demonstrate improved performance on reinforcement learning tasks with deceptive or sparse reward functions compared to popular gradient-based and deep genetic algorithm baselines.","doi":"10.24963/ijcai.2019/432","url":"https://ijcai.org/proceedings/2019/432","pdf":"https://ijcai.org/proceedings/2019/0432.pdf","markdown":"https://ijcai.org/proceedings/2019/432.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/432","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":432,"pages":{"first":3116,"last":3122},"keywords":["Planning and Scheduling: Markov Decisions Processes","Machine Learning: Deep Learning","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Baier and Cowling, 2018] Hendrik Baier and Peter I. Cowl- ing. Evolutionary MCTS for multi-action adversarial games. In IEEE Conference on Computational Intelli- gence and Games (CIG), pages 1-8. IEEE, 2018.","title":"Evolutionary MCTS for multi-action adversarial games","authors":"Cowling Baier, Hendrik Baier, I Peter","year":"2018","venue":"IEEE Conference on Computational Intelligence and Games (CIG)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI gym. arXiv:1606.01540, 2016.","title":"OpenAI gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chaslot et al., 2008a] Guillaume M.J.B Chaslot, Mark H.M. Winands, H. Jaap Van Den Herick, Jos W.H.M Uiterwijk, and Bruno Bouzy. Progressive strategies for Monte-Carlo tree search. New Mathematics and Natural Computation, 4(03):343-357, 2008.","title":"Progressive strategies for Monte-Carlo tree search","authors":"Chaslot","year":"2008","venue":"New Mathematics and Natural Computation","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chaslot et al., 2008b] Guillaume M.J.B. Chaslot, Mark H.M. Winands, and H. Jaap Van Den Herik. Parallel Monte-Carlo tree search. In International Conference on Computers and Games, pages 60-71, 2008.","title":"Parallel Monte-Carlo tree search","authors":"Chaslot","year":"2008","venue":"International Conference on Computers and Games","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chrabaszcz et al., 2018] Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. Back to basics: Bench- marking canonical evolution strategies for playing Atari. arXiv:1802.08842, 2018.","title":"Back to basics: Benchmarking canonical evolution strategies for playing Atari","authors":"Patryk Chrabaszcz, Ilya Loshchilov, Frank Hutter","year":"2018","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2018/197"},{"position":6,"raw":"[Couëtoux et al., 2011] Adrien Couëtoux, Jean-Baptiste Hoock, Nataliya Sokolovska, Olivier Teytaud, and Nico- las Bonnard. Continuous upper confidence trees. In Learning and Intelligent Optimization (LION), pages 433-445, 2011.","title":"Continuous upper confidence trees","authors":"Adrien Couëtoux, Jean-Baptiste Hoock, Nataliya Sokolovska, Olivier Teytaud, Nicolas Bonnard","year":"2011","venue":"Learning and Intelligent Optimization (LION)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Duan et al., 2016] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In Inter- national Conference on Machine Learning (ICML), pages 1329-1338, 2016.","title":"Benchmarking deep reinforcement learning for continuous control","authors":"Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel","year":"2016","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Geramifard et al., 2015] Alborz Geramifard, Christoph Dann, Robert H. Klein, William Dabney, and Jonathan P. How. RLPy: A value-function-based reinforcement learning framework for education and research. Journal of Machine Learning Research, 16:1573-1578, 2015.","title":"RLPy: A value-function-based reinforcement learning framework for education and research","authors":"Alborz Geramifard, Christoph Dann, Robert H Klein, William Dabney, Jonathan P How","year":"2015","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Grzes and Kudenko, 2009] Marek Grzes and Daniel Ku- denko. Theoretical and empirical analysis of reward shap- ing in reinforcement learning. In International Conference on Machine Learning and Applications, pages 337-344. IEEE, 2009.","title":"Theoretical and empirical analysis of reward shaping in reinforcement learning","authors":"Kudenko Grzes, Marek Grzes, Daniel Kudenko","year":"2009","venue":"International Conference on Machine Learning and Applications","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hessel et al., 2018] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Sil- ver. Rainbow: Combining improvements in deep rein- forcement learning. In AAAI Conference on Artificial In- telligence (AAAI), pages 3215-3222, 2018.","title":"Rainbow: Combining improvements in deep reinforcement learning","authors":"Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, David Silver","year":"2018","venue":"AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Keller and Helmert, 2013] Thomas Keller and Malte Helmert. Trial-based heuristic tree search for finite hori- zon MDPs. In International Conference on Automated Planning and Scheduling (ICAPS), pages 135-143, 2013.","title":"Trial-based heuristic tree search for finite horizon MDPs","authors":"Helmert Keller, Thomas Keller, Malte Helmert","year":"2013","venue":"International Conference on Automated Planning and Scheduling (ICAPS)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Khadka and Tumer, 2018] Shauharda Khadka and Kagan Tumer. Evolution-guided policy gradient in reinforcement learning. In Advances in Neural Information Processing Systems (NIPS), pages 1196-1208, 2018.","title":"Evolution-guided policy gradient in reinforcement learning","authors":"Tumer Khadka, Shauharda Khadka, Kagan Tumer","year":"2018","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Inter- national Conference on Learning Representations (ICLR), 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kochenderfer, 2015] Mykel J. Kochenderfer. Decision Making Under Uncertainty: Theory and Application. MIT Press, 2015.","title":"Decision Making Under Uncertainty: Theory and Application","authors":"J Mykel, Kochenderfer","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kocsis and Szepesvári, 2006] Levente Kocsis and Csaba Szepesvári. Bandit based Monte-Carlo planning. In Eu- ropean Conference on Machine Learning (ECML), pages 282-293, 2006.","title":"Bandit based Monte-Carlo planning","authors":"Szepesvári Kocsis, Levente Kocsis, Csaba Szepesvári","year":"2006","venue":"European Conference on Machine Learning (ECML)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lehman et al., 2018] Joel Lehman, Jay Chen, Jeff Clune, and Kenneth O. Stanley. Safe mutations for deep and recurrent neural networks through output gradients. In Genetic and Evolutionary Computation Conference, pages 117-124. ACM, 2018.","title":"Safe mutations for deep and recurrent neural networks through output gradients","authors":"Joel Lehman, Jay Chen, Jeff Clune, Kenneth O Stanley","year":"2018","venue":"Genetic and Evolutionary Computation Conference","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, K ; Andreas, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Moore, 1991] Andrew Moore. Efficient Memory-based Learning for Robot Control. PhD thesis, Carnegie Mel- lon University, 1991.","title":"Efficient Memory-based Learning for Robot Control","authors":"Andrew Moore","year":"1991","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Murthy and Raibert, 1984] Seshashayee S. Murthy and Marc H. Raibert. 3D balance in legged locomotion: Mod- eling and simulation for the one-legged case. ACM SIG- GRAPH Computer Graphics, 18(1):27-27, 1984.","title":"3D balance in legged locomotion: Modeling and simulation for the one-legged case","authors":"Seshashayee S Murthy, Marc H Raibert","year":"1984","venue":"ACM SIG-GRAPH Computer Graphics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Salimans et al., 2017] Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv:1703.03864, 2017.","title":"Evolution strategies as a scalable alternative to reinforcement learning","authors":"Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, Ilya Sutskever","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schulman et al., 2015a] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning (ICML), pages 1889-1897, 2015.","title":"Trust region policy optimization","authors":"Schulman","year":"2015","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schulman et al., 2015b] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advan- tage estimation. arXiv:1506.02438, 2015.","title":"Highdimensional continuous control using generalized advantage estimation","authors":"Schulman","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Such et al., 2017] Felipe Petroski Such, Vashisht Madha- van, Edoardo Conti, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv:1712.06567, 2017.","title":"Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning","authors":"Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O Stanley, Jeff Clune","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wawrzynski, 2007] Pawel Wawrzynski. Learning to con- trol a 6-degree-of-freedom walking robot. In EUROCON 2007-The International Conference on \" Computer as a Tool\", pages 698-705. IEEE, 2007.","title":"Learning to control a 6-degree-of-freedom walking robot","authors":"Pawel Wawrzynski","year":"2007","venue":"EUROCON 2007-The International Conference on \" Computer as a Tool","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wu et al., 2017] Yuhuai Wu, Elman Mansimov, Roger B. Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using Kronecker- factored approximation. In Advances in Neural Informa- tion Processing Systems (NIPS), pages 5279-5288, 2017.","title":"Scalable trust-region method for deep reinforcement learning using Kroneckerfactored approximation","authors":"Wu","year":"2017","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}