{"type":"ScholarlyArticle","title":"Generalized Mean Estimation in Monte-Carlo Tree Search","authors":[{"name":"Tuan Dam","slug":"tuan-dam","given":"Tuan","family":"Dam","affiliation":"Department of Computer Science , Technische Universität Darmstadt , Germany"},{"name":"Pascal Klink","slug":"pascal-klink","given":"Pascal","family":"Klink","affiliation":"Department of Computer Science , Technische Universität Darmstadt , Germany"},{"name":"Carlo D'Eramo","slug":"carlo-d-eramo","given":"Carlo","family":"D'Eramo","affiliation":"Department of Computer Science , Technische Universität Darmstadt , Germany"},{"name":"Jan Peters","slug":"jan-peters","given":"Jan","family":"Peters","affiliation":"Department of Computer Science , Technische Universität Darmstadt , Germany; Robot Learning Group , Max Planck Institute for Intelligent Systems , Tübingen , Germany"},{"name":"Joni Pajarinen","slug":"joni-pajarinen","given":"Joni","family":"Pajarinen","affiliation":"Department of Computer Science , Technische Universität Darmstadt , Germany; Computing Sciences , Tampere University , Finland"}],"abstract":"We consider Monte-Carlo Tree Search (MCTS) applied to Markov Decision Processes (MDPs) and Partially Observable MDPs (POMDPs), and the well-known Upper Confidence bound for Trees (UCT) algorithm. In UCT, a tree with nodes (states) and edges (actions) is incrementally built by the expansion of nodes, and the values of nodes are updated through a backup strategy based on the average value of child nodes. However, it has been shown that with enough samples the maximum operator yields more accurate node value estimates than averaging. Instead of settling for one of these value estimates, we go a step further proposing a novel backup strategy which uses the power mean operator, which computes a value between the average and maximum value. We call our new approach Power-UCT, and argue how the use of the power mean operator helps to speed up the learning in MCTS. We theoretically analyze our method providing guarantees of convergence to the optimum. Finally, we empirically demonstrate the effectiveness of our method in well-known MDP and POMDP benchmarks, showing significant improvement in performance and convergence speed w.r.t. state of the art algorithms.","doi":"10.24963/ijcai.2020/332","url":"https://ijcai.org/proceedings/2020/332","pdf":"https://ijcai.org/proceedings/2020/0332.pdf","markdown":"https://ijcai.org/proceedings/2020/332.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/332","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":332,"pages":{"first":2397,"last":2404},"keywords":["Machine Learning: Reinforcement Learning","Uncertainty in AI: Sequential Decision Making","Uncertainty in AI: Markov Decision Processes"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25163/file_25163.m3u8","length_sec":294,"provider":"hls"}],"references":[{"position":1,"raw":"[Auer et al., 2002] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bullen, 2013] Peter S Bullen. Handbook of means and their inequalities. Springer Science & Business Media, 2013.","title":"Handbook of means and their inequalities","authors":"Peter S Bullen","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chaslot et al., 2008] Guillaume Chaslot, Mark Winands, Jaap Van Den Herik, Jos Uiterwijk, and Bruno Bouzy. Pro- gressive strategies for monte-carlo tree search. New Math- ematics and Natural Computation, 4(03):343-357, 2008.","title":"Progressive strategies for monte-carlo tree search","authors":"Guillaume Chaslot, Mark Winands, Jaap Van Den, Jos Herik, Bruno Uiterwijk, Bouzy","year":"2008","venue":"New Mathematics and Natural Computation","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Childs et al., 2008] Benjamin E Childs, James H Brodeur, and Levente Kocsis. Transpositions and move groups in monte carlo tree search. In 2008 IEEE Symposium On Computational Intelligence and Games. IEEE, 2008.","title":"Transpositions and move groups in monte carlo tree search","authors":"James H Benjamin E Childs, Levente Brodeur, Kocsis","year":"2008","venue":"IEEE Symposium On Computational Intelligence and Games","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Coulom, 2006] Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In Interna- tional conference on computers and games, pages 72-83. Springer, 2006.","title":"Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search","authors":"Coulom","year":"2006","venue":"International conference on computers and games","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dam et al., 2019] Tuan Dam, Pascal Klink, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. Generalized mean estima- tion in monte-carlo tree search, 2019.","title":"Generalized mean estimation in monte-carlo tree search","authors":"Tuan Dam, Pascal Klink, Carlo D' Eramo, Jan Peters, Joni Pajarinen","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[D'Eramo et al., 2016] Carlo D'Eramo, Marcello Restelli, and Alessandro Nuara. Estimating maximum expected value through gaussian approximation. In International Conference on Machine Learning, 2016.","title":"Estimating maximum expected value through gaussian approximation","authors":"Carlo D' Eramo, Marcello Restelli, Alessandro Nuara","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gelly and Silver, 2007] Sylvain Gelly and David Silver. Combining online and offline knowledge in uct. In Pro- ceedings of the 24th international conference on Machine learning, pages 273-280. ACM, 2007.","title":"Combining online and offline knowledge in uct","authors":"Silver Gelly, Sylvain Gelly, David Silver","year":"2007","venue":"Proceedings of the 24th international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gelly and Wang, 2006] Sylvain Gelly and Yizao Wang. Ex- ploration exploitation in go: Uct for monte-carlo go. In NIPS: Neural Information Processing Systems Conference On-line trading of Exploration and Exploitation Work- shop, 2006.","title":"Exploration exploitation in go: Uct for monte-carlo go","authors":"Wang ; Gelly, Sylvain Gelly, Yizao Wang","year":"2006","venue":"NIPS: Neural Information Processing Systems Conference On-line trading of Exploration and Exploitation Workshop","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Hado V Hasselt. Double q-learning. In Ad- vances in Neural Information Processing Systems, pages 2613-2621, 2010.","title":"Double q-learning","authors":"V Hado, Hasselt","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Helmbold and Parker-Wood, 2009] David P Helmbold and Aleatha Parker-Wood. All-moves-as-first heuristics in monte-carlo go. In IC-AI, pages 605-610, 2009.","title":"All-moves-as-first heuristics in monte-carlo go","authors":"P David, Aleatha Helmbold, Parker-Wood","year":"2009","venue":"IC-AI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hoock et al., 2010] Jean-Baptiste Hoock, Chang-Shing Lee, Arpad Rimmel, Fabien Teytaud, Mei-Hui Wang, and Oliver Teytaud. Intelligent agents for the game of go. IEEE Computational Intelligence Magazine, 2010.","title":"Intelligent agents for the game of go","authors":"Jean-Baptiste Hoock, Chang-Shing Lee, Arpad Rimmel, Fabien Teytaud, Mei-Hui Wang, Oliver Teytaud","year":"2010","venue":"IEEE Computational Intelligence Magazine","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Khandelwal et al., 2016] Piyush Khandelwal, Elad Lieb- man, Scott Niekum, and Peter Stone. On the analysis of complex backup strategies in monte carlo tree search. In International Conference on Machine Learning, 2016.","title":"On the analysis of complex backup strategies in monte carlo tree search","authors":"Piyush Khandelwal, Elad Liebman, Scott Niekum, Peter Stone","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kocsis et al., 2006] Levente Kocsis, Csaba Szepesvári, and Jan Willemson. Improved monte-carlo search. 2006. [Kozelek, 2009] Tomáš Kozelek. Methods of mcts and the game arimaa. 2009.","title":"Improved monte-carlo search","authors":"Levente Kocsis, Csaba Szepesvári, Jan Willemson","year":"2006","venue":"Methods of mcts and the game arimaa","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lorentz, 2010] Richard J Lorentz. Improving monte-carlo tree search in havannah. In International Conference on Computers and Games, pages 105-115. Springer, 2010.","title":"Improving monte-carlo tree search in havannah","authors":"J Richard, Lorentz","year":"2010","venue":"International Conference on Computers and Games","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mansley et al., 2011] Chris Mansley, Ari Weinstein, and Michael Littman. Sample-based planning for continu- ous action markov decision processesgeneralizedfmean. In Twenty-First International Conference on Automated Planning and Scheduling, 2011.","title":"Sample-based planning for continuous action markov decision processesgeneralizedfmean","authors":"Chris Mansley, Ari Weinstein, Michael Littman","year":"2011","venue":"Twenty-First International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mitrinovic and Vasic, 1970] Dragoslav S Mitrinovic and Petar M Vasic. Analytic inequalities. Springer, 1970.","title":"Analytic inequalities","authors":"S Dragoslav, Mitrinovic, M Petar, Vasic","year":"1970","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rummery, 1995] Gavin Adrian Rummery. Problem solv- ing with reinforcement learning. PhD thesis, University of Cambridge Ph. D. dissertation, 1995.","title":"Problem solving with reinforcement learning","authors":"Adrian Gavin, Rummery","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Silver and Veness, 2010] David Silver and Joel Veness. Monte-carlo planning in large pomdps. In Advances in neural information processing systems, pages 2164-2172, 2010.","title":"Monte-carlo planning in large pomdps","authors":"Veness Silver, David Silver, Joel Veness","year":"2010","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Smith and Simmons, 2004] Trey Smith and Reid Simmons. Heuristic search value iteration for pomdps. In Proceed- ings of the 20th conference on Uncertainty in artificial in- telligence, pages 520-527. AUAI Press, 2004.","title":"Heuristic search value iteration for pomdps","authors":"Simmons Smith, Trey Smith, Reid Simmons","year":"2004","venue":"Proceedings of the 20th conference on Uncertainty in artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Smith and Winkler, 2006] James E Smith and Robert L Winkler. The optimizer's curse: Skepticism and postde- cision surprise in decision analysis. Management Science, 52(3):311-322, 2006.","title":"The optimizer's curse: Skepticism and postdecision surprise in decision analysis","authors":"E James, Robert L Smith, Winkler","year":"2006","venue":"Management Science","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tesauro et al., 2012] Gerald Tesauro, VT Rajan, and Richard Segal. Bayesian inference in monte-carlo tree search. arXiv preprint arXiv:1203.3519, 2012.","title":"Bayesian inference in monte-carlo tree search","authors":"Gerald Tesauro, Richard Vt Rajan, Segal","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Teytaud and Teytaud, 2010] Fabien Teytaud and Olivier Teytaud. On the huge benefit of decisive moves in monte-carlo tree search algorithms. In Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games, pages 359-364. IEEE, 2010.","title":"On the huge benefit of decisive moves in monte-carlo tree search algorithms","authors":"Fabien Teytaud, Olivier Teytaud","year":"2010","venue":"Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tom, 2010] David Tom. Investigating uct and rave: Steps towards a more robust method. 2010.","title":"Investigating uct and rave: Steps towards a more robust method","authors":"David Tom","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vodopivec et al., 2017] Tom Vodopivec, Spyridon Samoth- rakis, and Branko Ster. On monte carlo tree search and reinforcement learning. Journal of Artificial Intelligence Research, 60:881-936, 2017.","title":"On monte carlo tree search and reinforcement learning","authors":"Vodopivec","year":"2017","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Watkins, 1989] C. Watkins. Learning from Delayed Re- wards. PhD thesis, King's College, Cambridge, England, 1989.","title":"Learning from Delayed Rewards","authors":"C Watkins","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xiao et al., 2019] Chenjun Xiao, Ruitong Huang, Jincheng Mei, Dale Schuurmans, and Martin Müller. Maximum en- tropy monte-carlo planning. In Advances in Neural Infor- mation Processing Systems, pages 9516-9524, 2019.","title":"Maximum entropy monte-carlo planning","authors":"Xiao","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}