{"type":"ScholarlyArticle","title":"Three-Head Neural Network Architecture for Monte Carlo Tree Search","authors":[{"name":"Chao Gao","slug":"chao-gao","given":"Chao","family":"Gao","affiliation":"University of Alberta"},{"name":"Martin Müller","slug":"martin-muller","given":"Martin","family":"Müller","affiliation":"University of Alberta"},{"name":"Ryan Hayward","slug":"ryan-hayward","given":"Ryan","family":"Hayward","affiliation":"University of Alberta"}],"abstract":"AlphaGo Zero pioneered the concept of two-head neural networks in Monte Carlo Tree Search (MCTS), where the policy output is used for prior action\r\nprobability and the state-value estimate is used for leaf node evaluation.\r\nWe propose a three-head neural net architecture with policy, state- and action-value outputs, which could lead to more efficient MCTS since \r\nneural leaf estimate can still be back-propagated in tree with delayed node expansion and evaluation. To effectively train \r\nthe newly introduced action-value head on the same game dataset as for two-head nets, we exploit the optimal relations between parent\r\nand children nodes for data augmentation and regularization. In our experiments for the game of Hex, the action-value head learning\r\nachieves similar error as the state-value prediction of a two-head architecture. The resulting neural net models are then combined with\r\nthe same Policy Value MCTS (PV-MCTS) implementation.  We show that, due to more efficient use of neural net evaluations, PV-MCTS with\r\nthree-head neural nets consistently performs better than the two-head ones, significantly outplaying the state-of-the-art player MoHex-CNN.","doi":"10.24963/ijcai.2018/523","url":"https://ijcai.org/proceedings/2018/523","pdf":"https://ijcai.org/proceedings/2018/0523.pdf","markdown":"https://ijcai.org/proceedings/2018/523.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/523","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Multidisciplinary Topics and Applications","track":"Main track","number":523,"pages":{"first":3762,"last":3768},"keywords":["Multidisciplinary Topics and Applications: Computer Games","Heuristic Search and Game Playing: Heuristic Search and Machine Learning","Machine Learning: Deep Learning","Heuristic Search and Game Playing: Heuristic Search"],"videos":[],"references":[{"position":1,"raw":"[Anthony et al., 2017] Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. arXiv preprint arXiv:1705.08439, 2017.","title":"Thinking fast and slow with deep learning and tree search","authors":"Thomas Anthony, Zheng Tian, David Barber","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arneson et al., 2010] Broderick Arneson, Ryan B Hay- ward, and Philip Henderson. Monte carlo tree search in Hex. IEEE Transactions on Computational Intelligence and AI in Games, 2(4):251-258, 2010.","title":"Monte carlo tree search in Hex","authors":"Ryan B Broderick Arneson, Philip Hayward, Henderson","year":"2010","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Browne et al., 2012] Cameron B Browne, Edward Pow- ley, Daniel Whitehouse, Simon M Lucas, Peter I Cowl- ing, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games, 4(1):1-43, 2012.","title":"A survey of monte carlo tree search methods","authors":"Edward Cameron B Browne, Daniel Powley, Whitehouse, M Simon, Peter I Lucas, Philipp Cowling, Stephen Rohlfshagen, Diego Tavener, Spyridon Perez, Simon Samothrakis, Colton","year":"2012","venue":"IEEE Transactions on Computational Intelligence and AI in games","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Clark and Storkey, 2015] Christopher Clark and Amos Storkey. Training deep convolutional neural networks to play Go. In International Conference on Machine Learning, pages 1766-1774, 2015.","title":"Training deep convolutional neural networks to play Go","authors":"Christopher Clark, Amos Storkey","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Coulom, 2006] Rémi Coulom. Efficient selectivity and backup operators in Monte-Carlo tree search. In Interna- tional conference on computers and games, pages 72-83. Springer, 2006.","title":"Rémi Coulom. Efficient selectivity and backup operators in Monte-Carlo tree search","authors":"Coulom","year":"2006","venue":"International conference on computers and games","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Enzenberger et al., 2010] Markus Enzenberger, Martin Müller, Broderick Arneson, and Richard Segal. Fuego- an open-source framework for board games and go engine based on monte carlo tree search. IEEE Trans- actions on Computational Intelligence and AI in Games, 2(4):259-270, 2010.","title":"Fuegoan open-source framework for board games and go engine based on monte carlo tree search","authors":"Markus Enzenberger, Martin Müller, Broderick Arneson, Richard Segal","year":"2010","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gao et al., 2017] Chao Gao, Ryan B Hayward, and Martin Müller. Move prediction using deep convolutional neural networks in Hex. IEEE Transactions on Games, 2017.","title":"Move prediction using deep convolutional neural networks in Hex","authors":"Chao Gao, Ryan B Hayward, Martin Müller","year":"2017","venue":"IEEE Transactions on Games","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gelly and Silver, 2007] Sylvain Gelly and David Silver. Combining online and offline knowledge in UCT. In Pro- ceedings of the 24th international conference on Machine learning, pages 273-280. ACM, 2007.","title":"Combining online and offline knowledge in UCT","authors":"Silver Gelly, Sylvain Gelly, David Silver","year":"2007","venue":"Proceedings of the 24th international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gelly and Silver, 2011] Sylvain Gelly and David Silver. Monte-Carlo tree search and rapid action value estimation in computer Go. Artificial Intelligence, 175(11):1856- 1875, 2011.","title":"Monte-Carlo tree search and rapid action value estimation in computer Go","authors":"Silver Gelly, Sylvain Gelly, David Silver","year":"2011","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016a] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"He","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2016b] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision, pages 630-645. Springer, 2016.","title":"Identity mappings in deep residual networks","authors":"He","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huang et al., 2013] Shih-Chieh Huang, Broderick Arneson, Ryan B Hayward, Martin Müller, and Jakub Pawlewicz. Mohex 2.0: a pattern-based MCTS Hex player. In Interna- tional Conference on Computers and Games, pages 60-71. Springer, 2013.","title":"Mohex 2.0: a pattern-based MCTS Hex player","authors":"Shih-Chieh Huang, Broderick Arneson, Ryan B Hayward, Martin Müller, Jakub Pawlewicz","year":"2013","venue":"International Conference on Computers and Games","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448-456, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jouppi and al, 2017] Norman P. Jouppi and et. al. In- datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th Annual International Sym- posium on Computer Architecture, ISCA '17, pages 1-12. ACM, 2017.","title":"Indatacenter performance analysis of a tensor processing unit","authors":"Norman P Jouppi","year":"2017","venue":"Proceedings of the 44th Annual International Symposium on Computer Architecture, ISCA '17","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Inter- national Conference on Learning Representations, 2014.","title":"Adam: A method for stochastic optimization","authors":"Ba ; ] Diederik Kingma, Jimmy Kingma, Ba","year":"2014","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kocsis and Szepesvári, 2006] Levente Kocsis and Csaba Szepesvári. Bandit based Monte-Carlo planning. In Eu- ropean conference on machine learning, pages 282-293. Springer, 2006.","title":"Bandit based Monte-Carlo planning","authors":"Szepesvári Kocsis, Levente Kocsis, Csaba Szepesvári","year":"2006","venue":"European conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Ge- offrey Hinton. Deep learning. Nature, 521(7553):436- 444, 2015.","title":"Deep learning","authors":"Yann Lecun, Yoshua Bengio, Geoffrey Hinton","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Littman, 1996] Michael Lederman Littman. Algorithms for sequential decision making. PhD thesis, Brown University Providence, RI, 1996.","title":"Algorithms for sequential decision making","authors":"Michael Lederman, Littman","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Maddison et al., 2015] Chris J Maddison, Aja Huang, Ilya Sutskever, and David Silver. Move evaluation in Go us- ing deep convolutional neural networks. In International Conference on Learning Representations, 2015.","title":"Move evaluation in Go using deep convolutional neural networks","authors":"Aja Chris J Maddison, Ilya Huang, David Sutskever, Silver","year":"2015","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nash, 1952] John Nash. Some games and machines for playing them. Technical Report D-1164, RAND Corpo- ration, Feb 1952.","title":"Some games and machines for playing them","authors":"John Nash","year":"1952","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pawlewicz et al., 2015] Jakub Pawlewicz, Ryan Hayward, Philip Henderson, and Broderick Arneson. Stronger vir- tual connections in Hex. IEEE Transactions on Computa- tional Intelligence and AI in Games, 7(2):156-166, 2015.","title":"Stronger virtual connections in Hex","authors":"Jakub Pawlewicz, Ryan Hayward, Philip Henderson, Broderick Arneson","year":"2015","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rosin, 2011] Christopher D Rosin. Multi-armed bandits with episode context. Annals of Mathematics and Artifi- cial Intelligence, 61(3):203-230, 2011.","title":"Multi-armed bandits with episode context","authors":"Christopher D Rosin","year":"2011","venue":"Annals of Mathematics and Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shannon, 1953] Claude E Shannon. Computers and au- tomata. Proceedings of the IRE, 41(10):1234-1241, 1953.","title":"Computers and automata","authors":"Claude E Shannon","year":"1953","venue":"Proceedings of the IRE","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of Go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature, 550(7676):354-359, 2017.","title":"Mastering the game of Go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George Van Den Driessche, Thore Graepel, Demis Hassabis","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tian and Zhu, 2015] Yuandong Tian and Yan Zhu. Better computer Go player with neural network and long-term prediction. In International Conference on Learning Rep- resentations, 2015.","title":"Better computer Go player with neural network and long-term prediction","authors":"Yuandong Tian, Yan Zhu","year":"2015","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}