{"type":"ScholarlyArticle","title":"Algorithms or Actions? A Study in Large-Scale Reinforcement Learning","authors":[{"name":"Anderson Rocha Tavares","slug":"anderson-rocha-tavares","given":"Anderson Rocha","family":"Tavares","affiliation":"Computer Science Department - Universidade Federal de Minas Gerais"},{"name":"Sivasubramanian Anbalagan","slug":"sivasubramanian-anbalagan","given":"Sivasubramanian","family":"Anbalagan","affiliation":"School of Computing and Communications - Lancaster University"},{"name":"Leandro Soriano Marcolino","slug":"leandro-soriano-marcolino","given":"Leandro Soriano","family":"Marcolino","affiliation":"School of Computing and Communications - Lancaster University"},{"name":"Luiz Chaimowicz","slug":"luiz-chaimowicz","given":"Luiz","family":"Chaimowicz","affiliation":"Computer Science Department - Universidade Federal de Minas Gerais"}],"abstract":"Large state and action spaces are very challenging to reinforcement learning. However, in many domains there is a set of algorithms available, which estimate the best action given a state. Hence, agents can either directly learn a performance-maximizing mapping from states to actions, or from states to algorithms. We investigate several aspects of this dilemma, showing sufficient conditions for learning over algorithms to outperform over actions for a finite number of training iterations. We present synthetic experiments to further study such systems. Finally, we propose a function approximation approach, demonstrating the effectiveness of learning over algorithms in real-time strategy games.","doi":"10.24963/ijcai.2018/377","url":"https://ijcai.org/proceedings/2018/377","pdf":"https://ijcai.org/proceedings/2018/0377.pdf","markdown":"https://ijcai.org/proceedings/2018/377.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/377","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":377,"pages":{"first":2717,"last":2723},"keywords":["Uncertainty in AI: Markov Decision Processes","Machine Learning Applications: Game Playing","Machine Learning: Reinforcement Learning","Multidisciplinary Topics and Applications: Computer Games"],"videos":[],"references":[{"position":1,"raw":"[Barriga et al., 2015] Nicolas A. Barriga, Marius Stanescu, and Michael Buro. Puppet Search: Enhancing Scripted Behavior by Look-Ahead Search with Applications to Real-Time Strategy Games. In AIIDE, 2015.","title":"Puppet Search: Enhancing Scripted Behavior by Look-Ahead Search with Applications to Real-Time Strategy Games","authors":"Nicolas A Barriga, Marius Stanescu, Michael Buro","year":"2015","venue":"AIIDE","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Barriga et al., 2017] Nicolas A. Barriga, Marius Stanescu, and Michael Buro. Combining strategic learning and tactical search in real-time strategy games. In AIIDE, 2017.","title":"Combining strategic learning and tactical search in real-time strategy games","authors":"Nicolas A Barriga, Marius Stanescu, Michael Buro","year":"2017","venue":"AIIDE","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Barriga et al., 2018] Nicolas A. Barriga, Marius Stanescu, and Michael Buro. Game Tree Search Based on Non-Deterministic Action Scripts in Real-Time Strategy Games. IEEE Transactions on Games, 10(1):67-77, 2018.","title":"Game Tree Search Based on Non-Deterministic Action Scripts in Real-Time Strategy Games","authors":"Nicolas A Barriga, Marius Stanescu, Michael Buro","year":"2018","venue":"IEEE Transactions on Games","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bontrager et al., 2016] Philip Bontrager, Ahmed Khalifa, André Mendes, and Julian Togelius. Matching Games and Algorithms for General Video Game Playing. In AIIDE, 2016.","title":"Matching Games and Algorithms for General Video Game Playing","authors":"Philip Bontrager, Ahmed Khalifa, André Mendes, Julian Togelius","year":"2016","venue":"AIIDE","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Churchill and Buro, 2013] David Churchill and Michael Buro. Portfolio Greedy Search and Simulation for Large-Scale Combat in StarCraft. In CIG, 2013.","title":"Portfolio Greedy Search and Simulation for Large-Scale Combat in StarCraft","authors":"David Churchill, Michael Buro","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dietterich, 2000] Thomas G. Dietterich. Hierarchical reinforce- ment learning with the MAXQ value function decomposition. JAIR, 13:227-303, 2000.","title":"Hierarchical reinforcement learning with the MAXQ value function decomposition","authors":"G Thomas, Dietterich","year":"2000","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jiang et al., 2014] Albert Xin Jiang, Leandro Soriano Marcolino, Ariel D. Procaccia, Tuomas Sandholm, Nisarg Shah, and Milind Tambe. Diverse randomized agents vote to win. In NIPS, 2014.","title":"Diverse randomized agents vote to win","authors":"Albert Xin Jiang, Leandro Soriano Marcolino, Ariel D Procaccia, Tuomas Sandholm, Nisarg Shah, Milind Tambe","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lagoudakis and Littman, 2000] Michail G. Lagoudakis and Michael L. Littman. Algorithm selection using reinforcement learning. In ICML, 2000.","title":"Algorithm selection using reinforcement learning","authors":"Michail G Lagoudakis, Michael L Littman","year":"2000","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lelis, 2017] Levi H. S. Lelis. Stratified Strategy Selection for Unit Control in Real-Time Strategy Games. In IJCAI, 2017.","title":"Stratified Strategy Selection for Unit Control in Real-Time Strategy Games","authors":"H S Levi, Lelis","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/522"},{"position":10,"raw":"[Marcolino et al., 2013] Leandro Soriano Marcolino, Albert Xin Jiang, and Milind Tambe. Multi-agent team formation: diver- sity beats strength? In IJCAI, 2013.","title":"Multi-agent team formation: diversity beats strength?","authors":"Leandro Soriano Marcolino, Albert Xin Jiang, Milind Tambe","year":"2013","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2013/50"},{"position":11,"raw":"[Marcolino et al., 2014] Leandro Soriano Marcolino, Haifeng Xu, Albert Xin Jiang, Milind Tambe, and Emma Bowring. Give a hard problem to a diverse team: Exploring large action spaces. In AAAI, 2014.","title":"Give a hard problem to a diverse team: Exploring large action spaces","authors":"Leandro Soriano Marcolino, Haifeng Xu, Albert Xin Jiang, Milind Tambe, Emma Bowring","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ontañón, 2017] Santiago Ontañón. Combinatorial multi-armed ban- dits for real-time strategy games. JAIR, 58, 2017.","title":"Santiago Ontañón. Combinatorial multi-armed bandits for real-time strategy games","authors":"Ontañón","year":"2017","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ontañón and Buro, 2015] Santiago Ontañón and Michael Buro. Ad- versarial hierarchical-task network planning for complex real-time games. In IJCAI, 2015.","title":"Santiago Ontañón and Michael Buro. Adversarial hierarchical-task network planning for complex real-time games","authors":"Buro Ontañón","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ontañón et al., 2013] Santiago Ontañón, Gabriel Synnaeve, Al- berto Uriarte, Florian Richoux, David Churchill, and Mike Preuss. A survey of real-time strategy game AI research and competition in StarCraft. IEEE Transactions on Computational Intelligence and AI in Games (TCIAIG), 5(4):293-311, 2013.","title":"A survey of real-time strategy game AI research and competition in StarCraft","authors":"Ontañón","year":"2013","venue":"IEEE Transactions on Computational Intelligence and AI in Games (TCIAIG)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rice, 1976] John R. Rice. The algorithm selection problem. Ad- vances in computers, 15:65-118, 1976.","title":"The algorithm selection problem","authors":"John R Rice","year":"1976","venue":"Advances in computers","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rummery and Niranjan, 1994] Gavin A. Rummery and Mahesan Niranjan. On-line Q-learning using connectionist systems. Tech- nical report, Cambridge University, 1994.","title":"On-line Q-learning using connectionist systems","authors":"Gavin A Rummery, Mahesan Niranjan","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sailer et al., 2007] Frantisek Sailer, Michael Buro, and Marc Lanc- tot. Adversarial planning through strategy simulation. In CIG, 2007.","title":"Adversarial planning through strategy simulation","authors":"Frantisek Sailer, Michael Buro, Marc Lanctot","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.","title":"Reinforcement learning: An introduction","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sutton et al., 1999] Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for tempo- ral abstraction in reinforcement learning. Artificial Intelligence, 112:181-211, 1999.","title":"Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning","authors":"Richard S Sutton, Doina Precup, Satinder Singh","year":"1999","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tavares et al., 2016] Anderson Tavares, Hector Azpúrua, Amanda Santos, and Luiz Chaimowicz. Rock, Paper, StarCraft: Strategy Selection in Real-Time Strategy Games. In AIIDE, 2016.","title":"Rock, Paper, StarCraft: Strategy Selection in Real-Time Strategy Games","authors":"Anderson Tavares, Hector Azpúrua, Amanda Santos, Luiz Chaimowicz","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xu et al., 2008] Lin Xu, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. SATzilla: portfolio-based algorithm selection for SAT. JAIR, 32:565-606, 2008.","title":"SATzilla: portfolio-based algorithm selection for SAT","authors":"Lin Xu, Frank Hutter, H Holger, Kevin Hoos, Leyton-Brown","year":"2008","venue":"JAIR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}