{"type":"ScholarlyArticle","title":"Search-based Reinforcement Learning through Bandit Linear Optimization","authors":[{"name":"Milan Peelman","slug":"milan-peelman","given":"Milan","family":"Peelman","affiliation":"Ghent University"},{"name":"Antoon Bronselaer","slug":"antoon-bronselaer","given":"Antoon","family":"Bronselaer","affiliation":"Ghent University"},{"name":"Guy De Tré","slug":"guy-de-tre","given":"Guy","family":"De Tré","affiliation":"Ghent University"}],"abstract":"The development of AlphaZero was a breakthrough in search-based reinforcement learning, by employing a given world model in a Monte-Carlo tree search (MCTS) algorithm to incrementally learn both an action policy and a value estimation. When extending this paradigm to the setting of simultaneous move games we find that the selection strategy of AlphaZero has theoretical shortcomings, including that convergence to a Nash equilibrium is not guaranteed. By analyzing these shortcomings, we find that the selection strategy corresponds to an approximated version of bandit linear optimization using Tsallis entropy regularization with α parameter set to zero, which is equivalent to log-barrier regularization. This observation allows us to refine the search method used by AlphaZero to obtain an algorithm that has theoretically optimal regret as well as superior empirical performance on our evaluation benchmark.","doi":"10.24963/ijcai.2022/469","url":"https://ijcai.org/proceedings/2022/469","pdf":"https://ijcai.org/proceedings/2022/0469.pdf","markdown":"https://ijcai.org/proceedings/2022/469.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/469","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":469,"pages":{"first":3380,"last":3386},"keywords":["Machine Learning: Reinforcement Learning","Agent-based and Multi-agent Systems: Algorithmic Game Theory","Search: Game Playing"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986570","length_sec":92,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984713","length_sec":360,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Abdolmaleki et al., 2018] Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Rémi Munos, Nicolas Heess, and Martin A. Riedmiller. Maximum a posteriori policy optimisation. CoRR, abs/1806.06920, 2018.","title":"Maximum a posteriori policy optimisation","authors":"Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Rémi Munos, Nicolas Heess, Martin A Riedmiller","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Abernethy et al., 2008] Jacob Abernethy, Elad Hazan, and Alexander Rakhlin. Competing in the dark: An efficient algorithm for bandit linear optimization. volume 3, pages 263-274, 01 2008.","title":"Competing in the dark: An efficient algorithm for bandit linear optimization","authors":"Jacob Abernethy, Elad Hazan, Alexander Rakhlin","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Abernethy et al., 2015] Jacob D. Abernethy, Chansoo Lee, and Ambuj Tewari. Fighting bandits with a new kind of smoothness. In NIPS, 2015.","title":"Fighting bandits with a new kind of smoothness","authors":"Jacob D Abernethy, Chansoo Lee, Ambuj Tewari","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Audibert and Bubeck, 2009] Jean-Yves Audibert and Sébastien Bubeck. Minimax Policies for Adversarial and Stochastic Bandits. In Proceedings of the 22nd Annual Conference on Learning Theory (COLT), January 2009. Edition: Proceedings of the 22nd Annual Conference on Learning Theory (COLT).","title":"Minimax Policies for Adversarial and Stochastic Bandits","authors":"Bubeck ; Jean-Yves Audibert, Sébastien Audibert, Bubeck","year":"2009","venue":"Proceedings of the 22nd Annual Conference on Learning Theory (COLT)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Auer et al., 2003] Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, and Robert E. Schapire. The nonstochastic mul- tiarmed bandit problem. SIAM J. Comput., 32(1):48-77, January 2003.","title":"The nonstochastic multiarmed bandit problem","authors":"Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, Robert E Schapire","year":"2003","venue":"SIAM J. Comput","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Blum and Mansour, 2007] Avrim Blum and Yishay Man- sour. Learning, regret minimization, and equilibria. Al- gorithmic Game Theory, 01 2007.","title":"Learning, regret minimization, and equilibria","authors":"Avrim Blum, Yishay Mansour","year":"2007","venue":"Algorithmic Game Theory","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bosansky et al., 2016] Branislav Bosansky, Viliam Lisy, Marc Lanctot, Jiri Cermak, and Mark Winands. Algo- rithms for computing strategies in two-player simultane- ous move games. Artificial Intelligence, 237, 04 2016.","title":"Algorithms for computing strategies in two-player simultaneous move games","authors":"Branislav Bosansky, Viliam Lisy, Marc Lanctot, Jiri Cermak, Mark Winands","year":"2016","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Foster et al., 2016] Dylan J. Foster, Zhiyuan Li, Thodoris Lykouris, Karthik Sridharan, and Éva Tardos. Fast conver- gence of common learning algorithms in games. CoRR, abs/1606.06244, 2016.","title":"Fast convergence of common learning algorithms in games","authors":"Dylan J Foster, Zhiyuan Li, Thodoris Lykouris, Karthik Sridharan, Éva Tardos","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Grill et al., 2020] Jean-Bastien Grill, Florent Altché, Yun- hao Tang, Thomas Hubert, Michal Valko, Ioannis Antonoglou, and Rémi Munos. Monte-carlo tree search as regularized policy optimization. CoRR, abs/2007.12509, 2020.","title":"Monte-carlo tree search as regularized policy optimization","authors":"Jean-Bastien Grill, Florent Altché, Yunhao Tang, Thomas Hubert, Michal Valko, Ioannis Antonoglou, Rémi Munos","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hart and Mas-Colell, 2001] Sergiu Hart and Andreu Mas- Colell. A reinforcement procedure leading to correlated equilibrium. Economic Essays, 01 2001.","title":"A reinforcement procedure leading to correlated equilibrium","authors":"Sergiu Hart, Andreu Mas-Colell","year":"2001","venue":"Economic Essays","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hazan, 2019] Elad Hazan. Introduction to online convex optimization. CoRR, abs/1909.05207, 2019.","title":"Introduction to online convex optimization","authors":"Elad Hazan","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kocsis and Szepesvári, 2006] Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. volume 2006, pages 282-293, 09 2006.","title":"Bandit based monte-carlo planning","authors":"Szepesvári Kocsis, Levente Kocsis, Csaba Szepesvári","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kovarík and Lisý, 2018] Vojtech Kovarík and Viliam Lisý. Analysis of hannan consistent selection for monte carlo tree search in simultaneous move games. CoRR, abs/1804.09045, 2018.","title":"Analysis of hannan consistent selection for monte carlo tree search in simultaneous move games","authors":"Lisý Kovarík, Vojtech Kovarík, Viliam Lisý","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kowalski and Miernik, 2020] Jakub Kowalski and Ra- doslaw Miernik. Evolutionary approach to collectible card game arena deckbuilding using active genes. CoRR, abs/2001.01326, 2020.","title":"Evolutionary approach to collectible card game arena deckbuilding using active genes","authors":"Miernik Kowalski, Jakub Kowalski, Radoslaw Miernik","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Mnih","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. CoRR, abs/1602.01783, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rosin, 2011] Christopher D. Rosin. Multi-armed bandits with episode context. Annals of Mathematics and Artifi- cial Intelligence, 61(3):203-230, March 2011.","title":"Multi-armed bandits with episode context","authors":"D Christopher, Rosin","year":"2011","venue":"Annals of Mathematics and Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schmid et al., 2021] Martin Schmid, Matej Moravcik, Neil Burch, Rudolf Kadlec, Joshua Davidson, Kevin Waugh, Nolan Bard, Finbarr Timbers, Marc Lanctot, Zach Hol- land, Elnaz Davoodi, Alden Christianson, and Michael Bowling. Player of games. CoRR, abs/2112.03178, 2021.","title":"Player of games","authors":"Martin Schmid, Matej Moravcik, Neil Burch, Rudolf Kadlec, Joshua Davidson, Kevin Waugh, Nolan Bard, Finbarr Timbers, Marc Lanctot, Zach Holland, Elnaz Davoodi, Alden Christianson, Michael Bowling","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schrittwieser et al., 2019] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy P. Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model. CoRR, abs/1911.08265, 2019.","title":"Mastering atari, go, chess and shogi by planning with a learned model","authors":"Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy P Lillicrap, David Silver","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Seldin and Lugosi, 2017] Yevgeny Seldin and Gábor Lu- gosi. An improved parametrization and analysis of the EXP3++ algorithm for stochastic and adversarial bandits. CoRR, abs/1702.06103, 2017.","title":"An improved parametrization and analysis of the EXP3++ algorithm for stochastic and adversarial bandits","authors":"Yevgeny Seldin, Gábor Lugosi","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shafiei et al., 2009] Mohammad Shafiei, Nathan R Sturte- vant, and Jonathan Schaeffer. Comparing uct versus cfr in simultaneous games. 2009.","title":"Comparing uct versus cfr in simultaneous games","authors":"Mohammad Shafiei, Nathan R Sturtevant, Jonathan Schaeffer","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Silver et al., 2017] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy P. Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. CoRR, abs/1712.01815, 2017.","title":"Mastering chess and shogi by self-play with a general reinforcement learning algorithm","authors":"David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy P Lillicrap, Karen Simonyan, Demis Hassabis","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tak et al., 2014] Mandy Tak, Marc Lanctot, and Mark Winands. Monte carlo tree search variants for simultane- ous move games. 08 2014.","title":"Monte carlo tree search variants for simultaneous move games","authors":"Mandy Tak, Marc Lanctot, Mark Winands","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tian et al., 2019] Yuandong Tian, Jerry Ma, Qucheng Gong, Shubho Sengupta, Zhuoyuan Chen, James Pinker- ton, and C. Lawrence Zitnick. ELF opengo: An anal- ysis and open reimplementation of alphazero. CoRR, abs/1902.04522, 2019.","title":"ELF opengo: An analysis and open reimplementation of alphazero","authors":"Yuandong Tian, Jerry Ma, Qucheng Gong, Shubho Sengupta, Zhuoyuan Chen, James Pinkerton, C Lawrence Zitnick","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tsallis, 1988] Constantino Tsallis. Possible generaliza- tion of boltzmann-gibbs statistics. Journal of Statistical Physics, 52:479-487, 07 1988.","title":"Constantino Tsallis. Possible generalization of boltzmann-gibbs statistics","authors":"Tsallis","year":"1988","venue":"Journal of Statistical Physics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wei and Luo, 2018] Chen-Yu Wei and Haipeng Luo. More adaptive algorithms for adversarial bandits. CoRR, abs/1801.03265, 2018.","title":"More adaptive algorithms for adversarial bandits","authors":"Chen-Yu Wei, Haipeng Luo","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zimmert and Seldin, 2018] Julian Zimmert and Yevgeny Seldin. An optimal algorithm for stochastic and adver- sarial bandits. CoRR, abs/1807.07623, 2018.","title":"Julian Zimmert and Yevgeny Seldin. An optimal algorithm for stochastic and adversarial bandits","authors":"Seldin Zimmert","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}