{"type":"ScholarlyArticle","title":"Multiple Policy Value Monte Carlo Tree Search","authors":[{"name":"Li-Cheng Lan","slug":"li-cheng-lan","given":"Li-Cheng","family":"Lan","affiliation":"Department of Computer Science , National Chiao Tung University , Taiwan"},{"name":"Wei Li","slug":"wei-li","given":"Wei","family":"Li","affiliation":"Department of Computer Science , National Chiao Tung University , Taiwan"},{"name":"Ting-Han Wei","slug":"ting-han-wei","given":"Ting-Han","family":"Wei","affiliation":"Department of Computer Science , National Chiao Tung University , Taiwan"},{"name":"I-Chen Wu","slug":"i-chen-wu","given":"I-Chen","family":"Wu","affiliation":"Department of Computer Science , National Chiao Tung University , Taiwan; Pervasive Artificial Intelligence Research (PAIR) Labs , Taiwan"}],"abstract":"Many of the strongest game playing programs use a combination of Monte Carlo tree search (MCTS) and deep neural networks (DNN), where the DNNs are used as policy or value evaluators. Given a limited budget, such as online playing or during the self-play phase of AlphaZero (AZ) training, a balance needs to be reached between accurate state estimation and more MCTS simulations, both of which are critical for a strong game playing agent. Typically, larger DNNs are better at generalization and accurate evaluation, while smaller DNNs are less costly, and therefore can lead to more MCTS simulations and bigger search trees with the same budget. This paper introduces a new method called the multiple policy value MCTS (MPV-MCTS), which combines multiple policy value neural networks (PV-NNs) of various sizes to retain advantages of each network, where two PV-NNs f_S and f_L are used in this paper. We show through experiments on the game NoGo that a combined f_S and f_L MPV-MCTS outperforms single PV-NN with policy value MCTS, called PV-MCTS. Additionally, MPV-MCTS also outperforms PV-MCTS for AZ training.","doi":"10.24963/ijcai.2019/653","url":"https://ijcai.org/proceedings/2019/653","pdf":"https://ijcai.org/proceedings/2019/0653.pdf","markdown":"https://ijcai.org/proceedings/2019/653.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/653","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Multidisciplinary Topics and Applications","track":"Main track","number":653,"pages":{"first":4704,"last":4710},"keywords":["Machine Learning: Reinforcement Learning","Heuristic Search and Game Playing: Heuristic Search and Machine Learning","Planning and Scheduling: Search in Planning and Scheduling","Heuristic Search and Game Playing: Combinatorial Search and Optimisation","Multidisciplinary Topics and Applications: Computer Games"],"videos":[],"references":[{"position":1,"raw":"[Browne et al., 2012] Cameron B Browne, Edward Pow- ley, Daniel Whitehouse, Simon M Lucas, Peter I Cowl- ing, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games, 4(1):1-43, 2012.","title":"A survey of monte carlo tree search methods","authors":"Edward Cameron B Browne, Daniel Powley, Whitehouse, M Simon, Peter I Lucas, Philipp Cowling, Stephen Rohlfshagen, Diego Tavener, Spyridon Perez, Simon Samothrakis, Colton","year":"2012","venue":"IEEE Transactions on Computational Intelligence and AI in games","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chou et al., 2011] C-W Chou, Olivier Teytaud, and S-J Yen. Revisiting monte-carlo tree search on a normal form game: Nogo. In European Conference on the Applications of Evolutionary Computation, pages 73-82. Springer, 2011.","title":"Revisiting monte-carlo tree search on a normal form game: Nogo","authors":"C-W Chou, Olivier Teytaud, S-J Yen","year":"2011","venue":"European Conference on the Applications of Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Clark and Storkey, 2015] Christopher Clark and Amos Storkey. Training deep convolutional neural networks to play go. In International Conference on Machine Learning, pages 1766-1774, 2015.","title":"Training deep convolutional neural networks to play go","authors":"Christopher Clark, Amos Storkey","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Coulom, 2006] Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In Interna- tional conference on computers and games, pages 72-83. Springer, 2006.","title":"Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search","authors":"Coulom","year":"2006","venue":"International conference on computers and games","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Elo, 1978] Arpad E Elo. The rating of chessplayers, past and present. Arco Pub., 1978.","title":"The rating of chessplayers, past and present","authors":"E Arpad, Elo","year":"1978","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Enzenberger et al., 2010] Markus Enzenberger, Mar- tin Muller, Broderick Arneson, and Richard Segal. Fuego-an open-source framework for board games and go engine based on monte carlo tree search. IEEE Transactions on Computational Intelligence and AI in Games, 2(4):259-270, 2010.","title":"Fuego-an open-source framework for board games and go engine based on monte carlo tree search","authors":"Enzenberger","year":"2010","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gao et al., 2018] Chao Gao, Martin Müller, and Ryan Hay- ward. Three-head neural network architecture for monte carlo tree search. In IJCAI, pages 3762-3768, 2018.","title":"Three-head neural network architecture for monte carlo tree search","authors":"Chao Gao, Martin Müller, Ryan Hayward","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/523"},{"position":8,"raw":"[Gelly and Silver, 2007] Sylvain Gelly and David Silver. Combining online and offline knowledge in uct. In Pro- ceedings of the 24th international conference on Machine learning, pages 273-280. ACM, 2007.","title":"Combining online and offline knowledge in uct","authors":"Silver Gelly, Sylvain Gelly, David Silver","year":"2007","venue":"Proceedings of the 24th international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gelly and Silver, 2011] Sylvain Gelly and David Silver. Monte-carlo tree search and rapid action value estima- tion in computer go. Artificial Intelligence, 175(11):1856- 1875, 2011.","title":"Monte-carlo tree search and rapid action value estimation in computer go","authors":"Silver Gelly, Sylvain Gelly, David Silver","year":"2011","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kocsis and Szepesvári, 2006] Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In Eu- ropean conference on machine learning, pages 282-293. Springer, 2006.","title":"Bandit based monte-carlo planning","authors":"Szepesvári Kocsis, Levente Kocsis, Csaba Szepesvári","year":"2006","venue":"European conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lan, 2016] Li-Cheng Lan. Hahanogo: An open source nogo program. https://github.com/lclan1024/HaHaNoGo, 2016.","title":"Hahanogo: An open source nogo program","authors":"Li-Cheng Lan","year":"2016","venue":null,"doi":null,"url":"https://github.com/lclan1024/HaHaNoGo","internal_url":null},{"position":14,"raw":"[Lanctot et al., 2014] Marc Lanctot, Mark HM Winands, Tom Pepels, and Nathan R Sturtevant. Monte carlo tree search with heuristic evaluations using implicit minimax backups. In 2014 IEEE Conference on Computational In- telligence and Games, pages 1-8. IEEE, 2014.","title":"Monte carlo tree search with heuristic evaluations using implicit minimax backups","authors":"Lanctot","year":"2014","venue":"IEEE Conference on Computational Intelligence and Games","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Müller, 2015] Martin Müller. Nogo history and compe- titions. https://webdocs.cs.ualberta.ca/ ∼ mmueller/nogo/ history.html, 2015.","title":"Nogo history and competitions","authors":"Martin Müller","year":"2015","venue":null,"doi":null,"url":"https://webdocs.cs.ualberta.ca/∼mmueller/nogo/history.html","internal_url":null},{"position":16,"raw":"[Pascutto, 2017] G.-C. Pascutto. Leela-zero. https://github. com/gcp/leela-zero, 2017.","title":"Leela-zero","authors":"G.-C Pascutto","year":null,"venue":null,"doi":null,"url":"https://github.com/gcp/leela-zero,2017","internal_url":null},{"position":17,"raw":"[Rosin, 2011] Christopher D Rosin. Multi-armed bandits with episode context. Annals of Mathematics and Artifi- cial Intelligence, 61(3):203-230, 2011.","title":"Multi-armed bandits with episode context","authors":"Christopher D Rosin","year":"2011","venue":"Annals of Mathematics and Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017.","title":"Mastering the game of go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Silver et al., 2018] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self- play. Science, 362(6419):1140-1144, 2018.","title":"A general reinforcement learning algorithm that masters chess, shogi, and go through selfplay","authors":"David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel","year":"2018","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tian and Zhu, 2015] Yuandong Tian and Yan Zhu. Better computer go player with neural network and long-term prediction. arXiv preprint arXiv:1511.06410, 2015.","title":"Better computer go player with neural network and long-term prediction","authors":"Yuandong Tian, Yan Zhu","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tian et al., 2019] Yuandong Tian, Jerry Ma*, Qucheng Gong*, Shubho Sengupta*, Zhuoyuan Chen, James Pinkerton, and C. Lawrence Zitnick. Elf opengo: An analysis and open reimplementation of alphazero. CoRR, abs/1902.04522, 2019.","title":"Elf opengo: An analysis and open reimplementation of alphazero","authors":"Yuandong Tian, Jerry Ma, *, Qucheng Gong, *, Shubho Sengupta, *, Zhuoyuan Chen, James Pinkerton, C Lawrence Zitnick","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}