{"type":"ScholarlyArticle","title":"Choosing Well Your Opponents: How to Guide the Synthesis of Programmatic Strategies","authors":[{"name":"Rubens O. Moraes","slug":"rubens-o-moraes","given":"Rubens O.","family":"Moraes","affiliation":"Departamento de Informática , Universidade Federal de Vic ¸osa , Brazil; Department of Computing Science , University of Alberta , Canada"},{"name":"David S. Aleixo","slug":"david-s-aleixo","given":"David S.","family":"Aleixo","affiliation":"Departamento de Informática , Universidade Federal de Vic ¸osa , Brazil"},{"name":"Lucas N. Ferreira","slug":"lucas-n-ferreira","given":"Lucas N.","family":"Ferreira","affiliation":"Department of Computing Science , University of Alberta , Canada; Alberta Machine Intelligence Institute (Amii)"},{"name":"Levi H. S. Lelis","slug":"levi-h-s-lelis","given":"Levi H. S.","family":"Lelis","affiliation":"Department of Computing Science , University of Alberta , Canada"}],"abstract":"This paper introduces Local Learner (2L), an algorithm for providing a set of reference strategies to guide the search for programmatic strategies in two-player zero-sum games. Previous learning algorithms, such as Iterated Best Response (IBR), Fictitious Play (FP), and Double-Oracle (DO), can be computationally expensive or miss important information for guiding search algorithms. 2L actively selects a set of reference strategies to improve the search signal. We empirically demonstrate the advantages of our approach while guiding a local search algorithm for synthesizing strategies in three games, including MicroRTS, a challenging real-time strategy game. Results show that 2L learns reference strategies that provide a stronger search signal than IBR, FP, and DO. We also simulate a tournament of MicroRTS, where a synthesizer using 2L outperformed the winners of the two latest MicroRTS competitions, which were programmatic strategies written by human programmers.","doi":"10.24963/ijcai.2023/539","url":"https://ijcai.org/proceedings/2023/539","pdf":"https://ijcai.org/proceedings/2023/0539.pdf","markdown":"https://ijcai.org/proceedings/2023/539.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/539","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":539,"pages":{"first":4847,"last":4854},"keywords":["Agent-based and Multi-agent Systems: MAS: Multi-agent learning","Search: S: Local search","Multidisciplinary Topics and Applications: MDA: Computer games","Machine Learning: ML: Symbolic methods"],"videos":[],"references":[{"position":1,"raw":"[Aguas et al., 2018] Javier Segovia Aguas, Sergio Jiménez, and Anders Jonsson. Computing hierarchical finite state controllers with classical planning. Journal of Artificial Intelligence Research, 62:755-797, 2018.","title":"Computing hierarchical finite state controllers with classical planning","authors":"Javier Segovia Aguas, Sergio Jiménez, Anders Jonsson","year":"2018","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aleixo and Lelis, 2023] David S. Aleixo and Levi H. S. Lelis. Show me the way! Bilevel search for synthesiz- ing programmatic strategies. In Proceedings of the AAAI Conference on Artificial Intelligence. AAAI Press, 2023.","title":"Show me the way! Bilevel search for synthesizing programmatic strategies","authors":"David S Aleixo, H S Levi, Lelis","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Anthony et al., 2020] Thomas Anthony, Tom Eccles, An- drea Tacchetti, János Kramár, Ian Gemp, Thomas Hudson, Nicolas Porcel, Marc Lanctot, Julien Pérolat, Richard Ev- erett, et al. Learning to play no-press diplomacy with best response policy iteration. Advances in Neural Information Processing Systems, 33:17987-18003, 2020.","title":"Learning to play no-press diplomacy with best response policy iteration","authors":"Thomas Anthony, Tom Eccles, Andrea Tacchetti, János Kramár, Ian Gemp, Thomas Hudson, Nicolas Porcel, Marc Lanctot, Julien Pérolat, Richard Everett","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bastani et al., 2018] O. Bastani, Y. Pu, and A. Solar- Lezama. Verifiable reinforcement learning via policy ex- traction. In Advances in Neural Information Processing Systems, pages 2499-2509, 2018.","title":"Verifiable reinforcement learning via policy extraction","authors":"O Bastani, Y Pu, A Solar-Lezama","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bonet et al., 2010] Blai Bonet, Héctor Palacios, and Héctor Geffner. Automatic derivation of finite-state machines for behavior control. In Proceedings of the AAAI Conference on Artificial Intelligence, page 1656-1659. AAAI Press, 2010.","title":"Automatic derivation of finite-state machines for behavior control","authors":"Blai Bonet, Héctor Palacios, Héctor Geffner","year":"2010","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brown, 1951] GW Brown. Iterative solution of games by fictitious play, 1951. Activity Analysis of Production and Allocation (TC Koopmans, Ed.), pages 374-376, 1951.","title":"Iterative solution of games by fictitious play","authors":"Gw Brown","year":"1951","venue":"Activity Analysis of Production and Allocation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Garey and Johnson, 1979] M. R. Garey and D. S. Johnson. Computers and Intractability: A Guide to the Theory of NP-Completeness. W. H. Freeman, 1979.","title":"Computers and Intractability: A Guide to the Theory of NP-Completeness","authors":"M R Garey, D S Johnson","year":"1979","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu and De Giacomo, 2013] Yuxiao Hu and Giuseppe De Giacomo. A generic technique for synthesizing bounded finite-state controllers. Proceedings of the International Conference on Automated Planning and Scheduling, 23(1):109-116, 2013.","title":"Yuxiao Hu and Giuseppe De Giacomo. A generic technique for synthesizing bounded finite-state controllers","authors":"De Hu, Giacomo","year":"2013","venue":"Proceedings of the International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lanctot et al., 2017] Marc Lanctot, Vinicius Zambaldi, Au- drunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game- theoretic approach to multiagent reinforcement learning. Advances in neural information processing systems, 30, 2017.","title":"A unified gametheoretic approach to multiagent reinforcement learning","authors":"Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, Thore Graepel","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lelis, 2021] Levi H. S. Lelis. Planning algorithms for zero- sum games with exponential action spaces: A unifying perspective. In Proceedings of the Twenty-Ninth Interna- tional Conference on International Joint Conferences on Artificial Intelligence, pages 4892-4898, 2021.","title":"Planning algorithms for zerosum games with exponential action spaces: A unifying perspective","authors":"H S Levi, Lelis","year":"2021","venue":"Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mariño and Toledo, 2022] Julian R. H. Mariño and Clau- dio Toledo. Evolving interpretable strategies for zero-sum games. Applied Soft Computing, page 108860, 2022.","title":"Evolving interpretable strategies for zero-sum games","authors":"R H Julian, Claudio Mariño, Toledo","year":"2022","venue":"Applied Soft Computing","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mariño et al., 2021] Julian R. H. Mariño, Rubens O. Moraes, Tassiana C. Oliveira, Claudio Toledo, and Levi H. S. Lelis. Programmatic strategies for real-time strategy games. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 35, pages 381-389, 2021.","title":"Programmatic strategies for real-time strategy games","authors":"R H Julian, Mariño, O Rubens, Moraes, C Tassiana, Claudio Oliveira, Levi H S Toledo, Lelis","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[McMahan et al., 2003] H Brendan McMahan, Geoffrey J Gordon, and Avrim Blum. Planning in the presence of cost functions controlled by an adversary. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pages 536-543, 2003.","title":"Planning in the presence of cost functions controlled by an adversary","authors":"Geoffrey J H Brendan Mcmahan, Avrim Gordon, Blum","year":"2003","venue":"Proceedings of the 20th International Conference on Machine Learning (ICML-03)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Medeiros et al., 2022] Leandro C. Medeiros, David S. Aleixo, and Levi H. S. Lelis. What can we learn even from the weakest? Learning sketches for programmatic strate- gies. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7761-7769. AAAI Press, 2022.","title":"What can we learn even from the weakest? Learning sketches for programmatic strategies","authors":"Leandro C Medeiros, David S Aleixo, Levi H S Lelis","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Muller et al., 2020] P Muller, S Omidshafiei, M Rowland, K Tuyls, J Pérolat, S Liu, D Hennes, L Marris, M Lanc- tot, E Hughes, et al. A generalized training approach for multiagent learning. In ICLR, pages 1-35. ICLR, 2020.","title":"A generalized training approach for multiagent learning","authors":"S Muller, M Omidshafiei, K Rowland, J Tuyls, S Pérolat, D Liu, L Hennes, M Marris, E Lanctot, Hughes","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ontañón et al., 2018] Santiago Ontañón, Nicolas A. Bar- riga, Cleyton R. Silva, Rubens O. Moraes, and Levi H. S. Lelis. The first microrts artificial intelligence competition. AI Magazine, 39(1), 2018.","title":"The first microrts artificial intelligence competition","authors":"Santiago Ontañón, Nicolas A Barriga, Cleyton R Silva, Rubens O Moraes, Levi H S Lelis","year":"2018","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ontañón, 2017] Santiago Ontañón. Combinatorial multi- armed bandits for real-time strategy games. Journal of Artificial Intelligence Research, 58:665-702, 2017.","title":"Santiago Ontañón. Combinatorial multiarmed bandits for real-time strategy games","authors":"Ontañón","year":"2017","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Srivastava et al., 2011] Siddharth Srivastava, Neil Immer- man, Shlomo Zilberstein, and Tianjiao Zhang. Directed search for generalized plans using classical planners. In Proceedings of the International Conference on Auto- mated Planning and Scheduling. AAAI, 2011.","title":"Directed search for generalized plans using classical planners","authors":"Siddharth Srivastava, Neil Immerman, Shlomo Zilberstein, Tianjiao Zhang","year":"2011","venue":"Proceedings of the International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Verma et al., 2018] A. Verma, V. Murali, R. Singh, P. Kohli, and S. Chaudhuri. Programmatically interpretable re- inforcement learning. In Proceedings of the Interna- tional Conference on Machine Learning, pages 5052- 5061, 2018.","title":"Programmatically interpretable reinforcement learning","authors":"A Verma, V Murali, R Singh, P Kohli, S Chaudhuri","year":"2018","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Verma et al., 2019] Abhinav Verma, Hoang Le, Yisong Yue, and Swarat Chaudhuri. Imitation-projected program- matic reinforcement learning. In Advances in Neural In- formation Processing Systems, volume 32, pages 1-12.","title":"Imitation-projected programmatic reinforcement learning","authors":"Abhinav Verma, Hoang Le, Yisong Yue, Swarat Chaudhuri","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"Curran Associates, Inc., 2019.","title":null,"authors":"Curran Associates, Inc","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}