{"type":"ScholarlyArticle","title":"DeltaDou: Expert-level Doudizhu AI through Self-play","authors":[{"name":"Qiqi Jiang","slug":"qiqi-jiang","given":"Qiqi","family":"Jiang","affiliation":"SweetCode Inc , Beijing"},{"name":"Kuangzheng Li","slug":"kuangzheng-li","given":"Kuangzheng","family":"Li","affiliation":"SweetCode Inc , Beijing"},{"name":"Boyao Du","slug":"boyao-du","given":"Boyao","family":"Du","affiliation":"SweetCode Inc , Beijing"},{"name":"Hao Chen","slug":"hao-chen","given":"Hao","family":"Chen","affiliation":"SweetCode Inc , Beijing"},{"name":"Hai Fang","slug":"hai-fang","given":"Hai","family":"Fang","affiliation":"SweetCode Inc , Beijing"}],"abstract":"Artificial Intelligence has seen several breakthroughs in two-player \r\nperfect information game.  Nevertheless, Doudizhu, a three-player \r\nimperfect information game, is still quite challenging.  In this paper, \r\nwe present a Doudizhu AI by applying deep reinforcement learning from \r\ngames of self-play.  The algorithm combines an asymmetric MCTS on nodes \r\nof information set of each player, a policy-value network that \r\napproximates the policy and value on each decision node, and inference \r\non unobserved hands of other players by given policy.  Our results show \r\nthat self-play can significantly improve the performance of our agent in\r\n this multi-agent imperfect information game.  Even starting with a weak\r\n AI, our agent can achieve human expert level after days of self-play \r\nand training.","doi":"10.24963/ijcai.2019/176","url":"https://ijcai.org/proceedings/2019/176","pdf":"https://ijcai.org/proceedings/2019/0176.pdf","markdown":"https://ijcai.org/proceedings/2019/176.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/176","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Heuristic Search and Game Playing","track":"Main track","number":176,"pages":{"first":1265,"last":1271},"keywords":["Uncertainty in AI: Approximate Probabilistic Inference","Machine Learning: Reinforcement Learning","Heuristic Search and Game Playing: Game Playing and Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Brown and Sandholm, 2018] Noam Brown and Tuomas Sandholm. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science, 359(6374):418- 424, 2018.","title":"Superhuman AI for heads-up no-limit poker: Libratus beats top professionals","authors":"Sandholm Brown, Noam Brown, Tuomas Sandholm","year":"2018","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brown, 1951] George W Brown. Iterative solution of games by fictitious play. Activity analysis of production and allo- cation, 13(1):374-376, 1951.","title":"Iterative solution of games by fictitious play","authors":"George W Brown","year":"1951","venue":"Activity analysis of production and allocation","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Buro et al., 2009] Michael Buro, Jeffrey Richard Long, Timothy Furtak, and Nathan R Sturtevant. Improving state evaluation, inference, and search in trick-based card games. In IJCAI, pages 1407-1413, 2009.","title":"Improving state evaluation, inference, and search in trick-based card games","authors":"Michael Buro, Jeffrey Richard Long, Timothy Furtak, Nathan R Sturtevant","year":"2009","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2009/236"},{"position":4,"raw":"[Campbell et al., 2002] Murray Campbell, A Joseph Hoane Jr, and Feng-hsiung Hsu. Deep blue. Artificial intelligence, 134(1-2):57-83, 2002.","title":"Deep blue","authors":"Murray Campbell, Joseph Hoane, Feng-Hsiung Hsu","year":"2002","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cowling et al., 2012] Peter I Cowling, Edward J Powley, and Daniel Whitehouse. Information set monte carlo tree search. IEEE Transactions on Computational Intelligence and AI in Games, 4(2):120-143, 2012.","title":"Information set monte carlo tree search","authors":"Edward J Peter I Cowling, Daniel Powley, Whitehouse","year":"2012","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Furtak and Buro, 2013] Timothy Furtak and Michael Buro. Recursive monte carlo search for imperfect information games. In Computational Intelligence in Games (CIG), 2013 IEEE Conference on, pages 1-8. IEEE, 2013.","title":"Recursive monte carlo search for imperfect information games","authors":"Buro Furtak, Timothy Furtak, Michael Buro","year":"2013","venue":"Computational Intelligence in Games (CIG)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ginsberg, 1999] Matthew L Ginsberg. Gib: Steps toward an expert-level bridge-playing program. In IJCAI, pages 584-593.","title":"Gib: Steps toward an expert-level bridge-playing program","authors":"L Matthew, Ginsberg","year":null,"venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/1999/84"},{"position":8,"raw":"[Heinrich and Silver, 2015] Johannes Heinrich and David Silver. Smooth uct search in computer poker. In IJCAI, pages 554-560, 2015.","title":"Smooth uct search in computer poker","authors":"Silver Heinrich, Johannes Heinrich, David Silver","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/84"},{"position":9,"raw":"[Heinrich and Silver, 2016] Johannes Heinrich and David Silver. Deep reinforcement learning from self- play in imperfect-information games. arXiv preprint arXiv:1603.01121, 2016.","title":"Deep reinforcement learning from selfplay in imperfect-information games","authors":"Silver Heinrich, Johannes Heinrich, David Silver","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Heinrich et al., 2015] Johannes Heinrich, Marc Lanctot, and David Silver. Fictitious self-play in extensive-form games. In International Conference on Machine Learn- ing, pages 805-813, 2015.","title":"Fictitious self-play in extensive-form games","authors":"Johannes Heinrich, Marc Lanctot, David Silver","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Long et al., 2010] Jeffrey Richard Long, Nathan R Sturte- vant, Michael Buro, and Timothy Furtak. Understanding the success of perfect information monte carlo sampling in game tree search. In Twenty-Fourth AAAI Conference on Artificial Intelligence, 2010.","title":"Understanding the success of perfect information monte carlo sampling in game tree search","authors":"Jeffrey Richard, Long, Nathan R Sturtevant, Michael Buro, Timothy Furtak","year":"2010","venue":"Twenty-Fourth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Moravčík et al., 2017] Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisỳ, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artifi- cial intelligence in heads-up no-limit poker. Science, 356(6337):508-513, 2017.","title":"Deepstack: Expert-level artificial intelligence in heads-up no-limit poker","authors":"Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisỳ, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, Michael Bowling","year":"2017","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ponsen et al., 2010] Marc Ponsen, Geert Gerritsen, and Guillaume Chaslot. Integrating opponent models with monte-carlo tree search in poker. In Workshops at the Twenty-Fourth AAAI Conference on Artificial Intelligence, 2010.","title":"Integrating opponent models with monte-carlo tree search in poker","authors":"Ponsen","year":"2010","venue":"Workshops at the Twenty-Fourth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Powley et al., 2011] Edward J Powley, Daniel Whitehouse, and Peter I Cowling. Determinization in monte-carlo tree search for the card game dou di zhu. Proc. Artif. Intell. Simul. Behav, pages 17-24, 2011.","title":"Determinization in monte-carlo tree search for the card game dou di zhu","authors":"Powley","year":"2011","venue":"Proc. Artif. Intell. Simul. Behav","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Schaeffer et al., 2007] Jonathan Schaeffer, Neil Burch, Yn- gvi Björnsson, Akihiro Kishimoto, Martin Müller, Robert Lake, Paul Lu, and Steve Sutphen. Checkers is solved. science, 317(5844):1518-1522, 2007.","title":"Checkers is solved","authors":"Jonathan Schaeffer, Neil Burch, Yngvi Björnsson, Akihiro Kishimoto, Martin Müller, Robert Lake, Paul Lu, Steve Sutphen","year":"2007","venue":"science","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Silver et al., 2018] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self- play. Science, 362(6419):1140-1144, 2018.","title":"A general reinforcement learning algorithm that masters chess, shogi, and go through selfplay","authors":"David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel","year":"2018","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tesauro, 1995] Gerald Tesauro. Td-gammon: A self- teaching backgammon program. In Applications of Neural Networks, pages 267-285. Springer, 1995.","title":"Td-gammon: A selfteaching backgammon program","authors":"Gerald Tesauro","year":"1995","venue":"Applications of Neural Networks","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}