{"type":"ScholarlyArticle","title":"Focused Depth-first Proof Number Search using Convolutional Neural Networks for the Game of Hex","authors":[{"name":"Chao Gao","slug":"chao-gao","given":"Chao","family":"Gao","affiliation":"University of Alberta"},{"name":"Martin Müller","slug":"martin-muller","given":"Martin","family":"Müller","affiliation":"University of Alberta"},{"name":"Ryan Hayward","slug":"ryan-hayward","given":"Ryan","family":"Hayward","affiliation":"University of Alberta"}],"abstract":"Proof Number search (PNS) is an effective algorithm for searching theoretical values on games with non-uniform branching factors. Focused depth-first proof number search (FDFPN) with dynamic widening was proposed for Hex where the branching factor is nearly uniform. However, FDFPN is fragile to its heuristic move ordering function. The recent advances of Convolutional Neural Networks (CNNs) have led to considerable progress in game playing. We investigate how to incorporate the strength of CNNs into solving, with application to the game of Hex. We describe FDFPN-CNN, a new focused DFPN search that uses convolutional neural networks. FDFPN-CNN integrates two CNNs trained from games played by expert players. The value approximation CNN provides reliable information for defining the widening size by estimating the value of the node to expand, while the policy CNN selects promising children nodes to the search. On 8x8 Hex, experimental results show FDFPN-CNN performs notably better than FDFPN, suggesting a promising direction for better solving Hex positions where learning from strong players is possible.","doi":"10.24963/ijcai.2017/513","url":"https://ijcai.org/proceedings/2017/513","pdf":"https://ijcai.org/proceedings/2017/0513.pdf","markdown":"https://ijcai.org/proceedings/2017/513.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/513","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Multidisciplinary Topics and Applications","track":"Main track","number":513,"pages":{"first":3668,"last":3674},"keywords":["Machine Learning: Deep Learning","Combinatorial & Heuristic Search: Heuristic Search","Multidisciplinary Topics and Applications: Computer Games"],"videos":[],"references":[{"position":1,"raw":"[Allis et al., 1994] L Victor Allis, Maarten van der Meulen, and H Jaap Van Den Herik. Proof-number search. Artifi- cial Intelligence, 66(1):91-124, 1994.","title":"Proof-number search","authors":"Allis Victor, Maarten Van Der Meulen, H Jaap Van Den, Herik","year":"1994","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anshelevich, 2002] Vadim V Anshelevich. A hierarchi- cal approach to computer Hex. Artificial Intelligence, 134(1):101-120, 2002.","title":"A hierarchical approach to computer Hex","authors":"V Vadim, Anshelevich","year":"2002","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Arneson et al., 2010] Broderick Arneson, Ryan B Hay- ward, and Philip Henderson. Monte carlo tree search in Hex. IEEE Transactions on Computational Intelligence and AI in Games, 2(4):251-258, 2010.","title":"Monte carlo tree search in Hex","authors":"Ryan B Broderick Arneson, Philip Hayward, Henderson","year":"2010","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cazenave, 2004] Tristan Cazenave. Generalized widening. In ECAI, volume 16, page 156, 2004.","title":"Generalized widening","authors":"Tristan Cazenave, Cazenave","year":"2004","venue":"ECAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Clark and Storkey, 2014] Christopher Clark and Amos Storkey. Teaching deep convolutional neural networks to play go. arXiv preprint arXiv:1412.3409, 2014.","title":"Teaching deep convolutional neural networks to play go","authors":"Christopher Clark, Amos Storkey","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"Hayward et al., ] Ryan Hayward, Jakub Pawlewicz, Kei Takada, and Tony van der Valk. Mohex wins 2015 Hex 11x11 and Hex 13x13 tournaments. ICGA Journal, (Preprint):1-6.","title":"Mohex wins 2015 Hex 11x11 and Hex 13x13 tournaments","authors":"Hayward","year":null,"venue":"ICGA Journal","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hayward et al., 2004] Ryan Hayward, Yngvi Björnsson, Michael Johanson, Morgan Kan, Nathan Po, and Jack van Rijswijck. Solving 7×7 hex: Virtual connections and game-state reduction. In Advances in Computer Games, pages 261-278. Springer, 2004.","title":"Solving 7×7 hex: Virtual connections and game-state reduction","authors":"Ryan Hayward, Yngvi Björnsson, Michael Johanson, Morgan Kan, Nathan Po, Jack Van Rijswijck","year":"2004","venue":"Advances in Computer Games","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Henderson et al., 2009] Philip Henderson, Broderick Arne- son, and Ryan B Hayward. Solving 8×8 Hex. In Proc. IJCAI, volume 9, pages 505-510. Citeseer, 2009.","title":"Solving 8×8 Hex","authors":"Philip Henderson, Broderick Arneson, Ryan B Hayward","year":"2009","venue":"Proc. IJCAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Henderson, 2010] Philip Thomas Henderson. Playing and solving the game of Hex. PhD thesis, University of Al- berta, 2010.","title":"Playing and solving the game of Hex","authors":"Philip Thomas, Henderson","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2013] Shih-Chieh Huang, Broderick Arneson, Ryan B Hayward, Martin Müller, and Jakub Pawlewicz. Mohex 2.0: a pattern-based MCTS hex player. In Interna- tional Conference on Computers and Games, pages 60-71. Springer, 2013.","title":"Mohex 2.0: a pattern-based MCTS hex player","authors":"Shih-Chieh Huang, Broderick Arneson, Ryan B Hayward, Martin Müller, Jakub Pawlewicz","year":"2013","venue":"International Conference on Computers and Games","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kishimoto and Müller, 2005] Akihiro Kishimoto and Mar- tin Müller. Search versus knowledge for solving life and death problems in Go. In AAAI, pages 1374-1379, 2005.","title":"Akihiro Kishimoto and Martin Müller. Search versus knowledge for solving life and death problems in Go","authors":"Müller Kishimoto","year":"2005","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kishimoto et al., 2012] Akihiro Kishimoto, Mark HM Winands, Martin Müller, and Jahn-Takeshi Saito. Game- tree search using proof numbers: The first twenty years. ICGA Journal, 35(3):131-156, 2012.","title":"Gametree search using proof numbers: The first twenty years","authors":"Akihiro Kishimoto, H M Mark, Martin Winands, Jahn-Takeshi Müller, Saito","year":"2012","venue":"ICGA Journal","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kishimoto, 2005] Akihiro Kishimoto. Correct and efficient search algorithms in the presence of repetitions. PhD the- sis, University of Alberta, 2005.","title":"Correct and efficient search algorithms in the presence of repetitions","authors":"Akihiro Kishimoto","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kishimoto, 2010] Akihiro Kishimoto. Dealing with infinite loops, underestimation, and overestimation of depth-first proof-number search. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, pages 108- 113. AAAI Press, 2010.","title":"Dealing with infinite loops, underestimation, and overestimation of depth-first proof-number search","authors":"Akihiro Kishimoto","year":"2010","venue":"Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural in- formation processing systems, pages 1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Krizhevsky","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Maddison et al., 2014] Chris J Maddison, Aja Huang, Ilya Sutskever, and David Silver. Move evaluation in Go us- ing deep convolutional neural networks. arXiv preprint arXiv:1412.6564, 2014.","title":"Move evaluation in Go using deep convolutional neural networks","authors":"Aja Chris J Maddison, Ilya Huang, David Sutskever, Silver","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[McAllester, 1988] David Allen McAllester. Conspiracy numbers for min-max search. Artificial Intelligence, 35(3):287-310, 1988.","title":"Conspiracy numbers for min-max search","authors":"David Allen, Mcallester","year":"1988","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nagai, 2002] Ayumu Nagai. Df-pn algorithm for search- ing AND/OR trees and its applications. PhD thesis, PhD thesis, Department of Information Science, University of Tokyo, 2002.","title":"Df-pn algorithm for searching AND/OR trees and its applications","authors":"Ayumu Nagai","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Norouzi et al., 2016] Mohammad Norouzi, Samy Bengio, Navdeep Jaitly, Mike Schuster, Yonghui Wu, Dale Schu- urmans, et al. Reward augmented maximum likelihood for neural structured prediction. In Advances In Neural Infor- mation Processing Systems, pages 1723-1731, 2016.","title":"Reward augmented maximum likelihood for neural structured prediction","authors":"Mohammad Norouzi, Samy Bengio, Navdeep Jaitly, Mike Schuster, Yonghui Wu, Dale Schuurmans","year":"2016","venue":"Advances In Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pawlewicz and Hayward, 2013] Jakub Pawlewicz and Ryan B Hayward. Scalable parallel DFPN search. In International Conference on Computers and Games, pages 138-150. Springer, 2013.","title":"Scalable parallel DFPN search","authors":"Hayward ; Pawlewicz, Jakub Pawlewicz, Ryan B Hayward","year":"2013","venue":"International Conference on Computers and Games","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pawlewicz et al., 2015] Jakub Pawlewicz, Ryan Hayward, Philip Henderson, and Broderick Arneson. Stronger Vir- tual Connections in Hex. IEEE Transactions on Computa- tional Intelligence and AI in Games, 7(2):156-166, 2015.","title":"Stronger Virtual Connections in Hex","authors":"Jakub Pawlewicz, Ryan Hayward, Philip Henderson, Broderick Arneson","year":"2015","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Reisch, 1981] Stefan Reisch. Hex ist PSPACE-vollständig. Acta Informatica, 15(2):167-191, 1981.","title":"Hex ist PSPACE-vollständig","authors":"Stefan Reisch","year":"1981","venue":"Acta Informatica","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schaeffer et al., 2007] Jonathan Schaeffer, Neil Burch, Yn- gvi Björnsson, Akihiro Kishimoto, Martin Müller, Robert Lake, Paul Lu, and Steve Sutphen. Checkers is solved. science, 317(5844):1518-1522, 2007.","title":"Checkers is solved","authors":"Jonathan Schaeffer, Neil Burch, Yngvi Björnsson, Akihiro Kishimoto, Martin Müller, Robert Lake, Paul Lu, Steve Sutphen","year":"2007","venue":"science","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of Go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tian and Zhu, 2015] Yuandong Tian and Yan Zhu. Better computer Go player with neural network and long-term prediction. arXiv preprint arXiv:1511.06410, 2015.","title":"Better computer Go player with neural network and long-term prediction","authors":"Yuandong Tian, Yan Zhu","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[van den Herik and Winands, 2008] H Jaap van den Herik and Mark HM Winands. Proof-number search and its vari- ants. In Oppositional Concepts in Computational Intelli- gence, pages 91-118. Springer, 2008.","title":"Proof-number search and its variants","authors":"Herik Van Den, ; H Jaap Winands, Van Den, Herik, H M Mark, Winands","year":"2008","venue":"Oppositional Concepts in Computational Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yoshizoe et al., 2007] Kazuki Yoshizoe, Akihiro Kishi- moto, and Martin Müller. Lambda Depth-First Proof Num- ber Search and Its Application to Go. In IJCAI, pages 2404-2409, 2007.","title":"Lambda Depth-First Proof Number Search and Its Application to Go","authors":"Kazuki Yoshizoe, Akihiro Kishimoto, Martin Müller","year":"2007","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2007/387"},{"position":29,"raw":"[Yoshizoe, 2008] Kazuki Yoshizoe. A new proof-number calculation technique for proof-number search. In Interna- tional Conference on Computers and Games, pages 135- 145. Springer, 2008.","title":"A new proof-number calculation technique for proof-number search","authors":"Kazuki Yoshizoe","year":"2008","venue":"International Conference on Computers and Games","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}