{"type":"ScholarlyArticle","title":"Searching for Programmatic Policies in Semantic Spaces","authors":[{"name":"Rubens O. Moraes","slug":"rubens-o-moraes","given":"Rubens O.","family":"Moraes","affiliation":"Departamento de Informática , Universidade Federal de Vic ¸osa , Brazil"},{"name":"Levi H. S. Lelis","slug":"levi-h-s-lelis","given":"Levi H. S.","family":"Lelis","affiliation":"Department of Computing Science , University of Alberta , Canada; Alberta Machine Intelligence Institute (Amii)"}],"abstract":"Syntax-guided synthesis is commonly used to generate programs encoding policies. In this approach, the set of programs, that can be written in a domain-specific language defines the search space, and an algorithm searches within this space for programs that encode strong policies. In this paper, we propose an alternative method for synthesizing programmatic policies, where we search within an approximation of the language's semantic space. We hypothesized that searching in semantic spaces is more sample-efficient compared to syntax-based spaces. Our rationale is that the search is more efficient if the algorithm evaluates different agent behaviors as it searches through the space, a feature often missing in syntax-based spaces. This is because small changes in the syntax of a program often do not result in different agent behaviors. We define semantic spaces by learning a library of programs that present different agent behaviors. Then, we approximate the semantic space by defining a neighborhood function for local search algorithms, where we replace parts of the current candidate program with programs from the library. We evaluated our hypothesis in a real-time strategy game called MicroRTS. Empirical results support our hypothesis that searching in semantic spaces can be more sample-efficient than searching in syntax-based spaces.","doi":"10.24963/ijcai.2024/662","url":"https://ijcai.org/proceedings/2024/662","pdf":"https://ijcai.org/proceedings/2024/0662.pdf","markdown":"https://ijcai.org/proceedings/2024/662.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/662","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":662,"pages":{"first":5990,"last":5998},"keywords":["Multidisciplinary Topics and Applications: MTA: Game playing","Multidisciplinary Topics and Applications: MTA: Computer games"],"videos":[],"references":[{"position":1,"raw":"[Aguas et al., 2018] Javier Segovia Aguas, Sergio Jiménez, and Anders Jonsson. Computing hierarchical finite state controllers with classical planning. Journal of Artificial Intelligence Research, 62:755-797, 2018.","title":"Computing hierarchical finite state controllers with classical planning","authors":"Javier Segovia Aguas, Sergio Jiménez, Anders Jonsson","year":"2018","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aleixo and Lelis, 2023] David S. Aleixo and Levi H. S. Lelis. Show me the way! Bilevel search for synthesiz- ing programmatic strategies. In Proceedings of the AAAI Conference on Artificial Intelligence. AAAI Press, 2023.","title":"Show me the way! Bilevel search for synthesizing programmatic strategies","authors":"David S Aleixo, H S Levi, Lelis","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Alur et al., 2013] Rajeev Alur, Rastislav Bodik, Garvit Ju- niwal, Milo Martin, Mukund Raghothaman, Sanjit Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, and Abhishek Udupa. Syntax-guided synthesis. pages 1- 17, 10 2013.","title":"Syntax-guided synthesis","authors":"Rajeev Alur, Rastislav Bodik, Garvit Juniwal, Milo Martin, Mukund Raghothaman, Sanjit Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, Abhishek Udupa","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Barriga et al., 2017a] Nicolas Barriga, Marius Stanescu, and Michael Buro. Combining strategic learning and tac- tical search in real-time strategy games. In Proceedings of the AAAI Conference on Artificial Intelligence and Inter- active Digital Entertainment, pages 9-15. AAAI, 2017.","title":"Combining strategic learning and tactical search in real-time strategy games","authors":"Barriga","year":"2017","venue":"Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Barriga et al., 2017b] Nicolas Barriga, Marius Stanescu, and Michael Buro. Game tree search based on non- deterministic action scripts in real-time strategy games. IEEE Transactions on Computational Intelligence and AI in Games, pages 69-77, 2017.","title":"Game tree search based on nondeterministic action scripts in real-time strategy games","authors":"Barriga","year":"2017","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bastani et al., 2018] Osbert Bastani, Yewen Pu, and Ar- mando Solar-Lezama. Verifiable reinforcement learning via policy extraction. In Advances in Neural Information Processing Systems, pages 2499-2509, 2018.","title":"Verifiable reinforcement learning via policy extraction","authors":"Osbert Bastani, Yewen Pu, Armando Solar-Lezama","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bonet et al., 2010] Blai Bonet, Héctor Palacios, and Héctor Geffner. Automatic derivation of finite-state machines for behavior control. In Proceedings of the AAAI Conference on Artificial Intelligence, page 1656-1659. AAAI Press, 2010.","title":"Automatic derivation of finite-state machines for behavior control","authors":"Blai Bonet, Héctor Palacios, Héctor Geffner","year":"2010","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bowers et al., 2023] Matthew Bowers, Theo X. Olausson, Lionel Wong, Gabriel Grand, Joshua B. Tenenbaum, Kevin Ellis, and Armando Solar-Lezama. Top-down syn- thesis for library learning. Procedings of the ACM on Pro- gramming Languages, 7(POPL), 2023.","title":"Top-down synthesis for library learning","authors":"Matthew Bowers, Theo X Olausson, Lionel Wong, Gabriel Grand, Joshua B Tenenbaum, Kevin Ellis, Armando Solar-Lezama","year":"2023","venue":"Procedings of the ACM on Programming Languages","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cao et al., 2023] David Cao, Rose Kunkel, Chandrakana Nandi, Max Willsey, Zachary Tatlock, and Nadia Polikar- pova. Babble: Learning better abstractions with e-graphs and anti-unification. Procedings of the ACM on Program- ming Languages, 7(POPL), 2023.","title":"Babble: Learning better abstractions with e-graphs and anti-unification","authors":"David Cao, Rose Kunkel, Chandrakana Nandi, Max Willsey, Zachary Tatlock, Nadia Polikarpova","year":"2023","venue":"Procedings of the ACM on Programming Languages","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Carvalho et al., 2024] Tales Henrique Carvalho, Kenneth Tjhia, and Levi H. S. Lelis. Reclaiming the source of pro- grammatic policies: Programmatic versus latent spaces. In The Twelfth International Conference on Learning Repre- sentations, 2024.","title":"Reclaiming the source of programmatic policies: Programmatic versus latent spaces","authors":"Carvalho","year":"2024","venue":"The Twelfth International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dahlbom and Niklasson, 2006] Anders Dahlbom and Lars Niklasson. Goal-directed hierarchical dynamic scripting for rts games. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertain- ment, pages 21-28, 2006.","title":"Goal-directed hierarchical dynamic scripting for rts games","authors":"Niklasson Dahlbom, Anders Dahlbom, Lars Niklasson","year":"2006","venue":"Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ellis et al., 2023] Kevin Ellis, Lionel Wong, Maxwell Nye, Mathias Sabl-Meyer, Luc Cary, Lore Pozo, Luke Hewitt, Armando Solar-Lezama, and Joshua Tenenbaum. Dream- coder: growing generalizable, interpretable knowledge with wake-sleep bayesian program learning. Philosoph- ical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 381, 06 2023.","title":"Dreamcoder: growing generalizable, interpretable knowledge with wake-sleep bayesian program learning","authors":"Kevin Ellis, Lionel Wong, Maxwell Nye, Mathias Sabl-Meyer, Luc Cary, Lore Pozo, Luke Hewitt, Armando Solar-Lezama, Joshua Tenenbaum","year":null,"venue":"Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hu and De Giacomo, 2013] Yuxiao Hu and Giuseppe De Giacomo. A generic technique for synthesizing bounded finite-state controllers. Proceedings of the International Conference on Automated Planning and Scheduling, 23(1):109-116, 2013.","title":"Yuxiao Hu and Giuseppe De Giacomo. A generic technique for synthesizing bounded finite-state controllers","authors":"De Hu, Giacomo","year":"2013","venue":"Proceedings of the International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Husien and Schewe, 2016] Idress Husien and Sven Schewe. Program generation using simulated annealing and model checking. In Rocco De Nicola and Eva Kühn, editors, Soft- ware Engineering and Formal Methods, pages 155-171. Springer International Publishing, 2016.","title":"Idress Husien and Sven Schewe. Program generation using simulated annealing and model checking","authors":"Schewe Husien","year":"2016","venue":"Software Engineering and Formal Methods","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Inala et al., 2020] Jeevana Priya Inala, Osbert Bastani, Zenna Tavares, and Armando Solar-Lezama. Synthe- sizing programmatic policies that inductively generalize. In International Conference on Learning Representations. OpenReview.net, 2020.","title":"Synthesizing programmatic policies that inductively generalize","authors":"Jeevana Priya Inala, Osbert Bastani, Zenna Tavares, Armando Solar-Lezama","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Koza, 1992] J. R. Koza. Genetic Programming: On the Pro- gramming of Computers by Means of Natural Selection. MIT Press, Cambridge, MA, 1992.","title":"Genetic Programming: On the Programming of Computers by Means of Natural Selection","authors":"J R Koza","year":"1992","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lanctot et al., 2017] Marc Lanctot, Vinicius Zambaldi, Au- drunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game- theoretic approach to multiagent reinforcement learning. Advances in neural information processing systems, 30, 2017.","title":"A unified gametheoretic approach to multiagent reinforcement learning","authors":"Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, Thore Graepel","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lelis, 2021] Levi H. S. Lelis. Planning algorithms for zero- sum games with exponential action spaces: A unifying perspective. In Proceedings of the Twenty-Ninth Interna- tional Conference on International Joint Conferences on Artificial Intelligence, pages 4892-4898, 2021.","title":"Planning algorithms for zerosum games with exponential action spaces: A unifying perspective","authors":"H S Levi, Lelis","year":"2021","venue":"Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2023] Guan-Ting Liu, En-Pei Hu, Pu-Jen Cheng, Hung-Yi Lee, and Shao-Hua Sun. Hierarchical program- matic reinforcement learning via learning to compose pro- grams. arXiv preprint arXiv:2301.12950, 2023.","title":"Hierarchical programmatic reinforcement learning via learning to compose programs","authors":"Guan-Ting Liu, En-Pei Hu, Pu-Jen Cheng, Hung-Yi Lee, Shao-Hua Sun","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mariño et al., 2021] Julian R. H. Mariño, Rubens O. Moraes, Tassiana C. Oliveira, Claudio Toledo, and Levi H. S. Lelis. Programmatic strategies for real-time strategy games. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 35, pages 381-389, 2021.","title":"Programmatic strategies for real-time strategy games","authors":"R H Julian, Mariño, O Rubens, Moraes, C Tassiana, Claudio Oliveira, Levi H S Toledo, Lelis","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Medeiros et al., 2022] Leandro C. Medeiros, David S. Aleixo, and Levi H. S. Lelis. What can we learn even from the weakest? Learning sketches for programmatic strate- gies. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7761-7769. AAAI Press, 2022.","title":"What can we learn even from the weakest? Learning sketches for programmatic strategies","authors":"Leandro C Medeiros, David S Aleixo, Levi H S Lelis","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Moraes and Lelis, 2018] Rubens O. Moraes and Levi H. S. Lelis. Asymmetric action abstractions for multi-unit con- trol in adversarial real-time scenarios. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelli- gence. AAAI, 2018.","title":"Asymmetric action abstractions for multi-unit control in adversarial real-time scenarios","authors":"O Rubens, Moraes, H S Levi, Lelis","year":"2018","venue":"Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Moraes et al., 2023] Rubens O. Moraes, David S. Aleixo, Lucas N. Ferreira, and Levi H. S. Lelis. Choosing well your opponents: How to guide the synthesis of program- matic strategies. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 4847-4854, 2023.","title":"Choosing well your opponents: How to guide the synthesis of programmatic strategies","authors":"O Rubens, David S Moraes, Lucas N Aleixo, Levi H S Ferreira, Lelis","year":"2023","venue":"Proceedings of the International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2023/539"},{"position":24,"raw":"[Ontañón, 2017] Santiago Ontañón. Combinatorial multi- armed bandits for real-time strategy games. Journal of Artificial Intelligence Research, 58:665-702, 2017.","title":"Santiago Ontañón. Combinatorial multiarmed bandits for real-time strategy games","authors":"Ontañón","year":"2017","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ponsen and Spronck, 2004] Marc Ponsen and Pieter Spronck. Improving adaptive game AI with evolutionary learning. PhD thesis, Citeseer, 2004.","title":"Improving adaptive game AI with evolutionary learning","authors":"Marc Ponsen, Pieter Spronck","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Silva et al., 2018] Cleyton Silva, Rubens O Moraes, Levi HS Lelis, and Kobi Gal. Strategy generation for multi-unit real-time games via voting. IEEE Transactions on Games, 2018.","title":"Strategy generation for multi-unit real-time games via voting","authors":"Cleyton Silva, O Rubens, Levi Moraes, Kobi Hs Lelis, Gal","year":"2018","venue":"IEEE Transactions on Games","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Spronck et al., 2004] Pieter Spronck, Ida Sprinkhuizen- Kuyper, and Eric Postma. Online adaptation of game op- ponent ai with dynamic scripting. International Journal of Intelligent Games and Simulation, 3(1):45-53, 2004.","title":"Pieter Spronck, Ida Sprinkhuizen-Kuyper, and Eric Postma. Online adaptation of game opponent ai with dynamic scripting","authors":"Spronck","year":"2004","venue":"International Journal of Intelligent Games and Simulation","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Srivastava et al., 2011] Siddharth Srivastava, Neil Immer- man, Shlomo Zilberstein, and Tianjiao Zhang. Directed search for generalized plans using classical planners. In Proceedings of the International Conference on Auto- mated Planning and Scheduling. AAAI, 2011.","title":"Directed search for generalized plans using classical planners","authors":"Siddharth Srivastava, Neil Immerman, Shlomo Zilberstein, Tianjiao Zhang","year":"2011","venue":"Proceedings of the International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Trivedi et al., 2021] Dweep Trivedi, Jesse Zhang, Shao-Hua Sun, and Joseph J Lim. Learning to synthesize programs as interpretable and generalizable policies. Advances in neural information processing systems, 34:25146-25163, 2021.","title":"Learning to synthesize programs as interpretable and generalizable policies","authors":"Dweep Trivedi, Jesse Zhang, Shao-Hua Sun, Joseph J Lim","year":"2021","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Verma et al., 2018a] Abhinav Verma, Vijayaraghavan Mu- rali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaud- huri. Programmatically interpretable reinforcement learn- ing. In Proceedings of the International Conference on Machine Learning, pages 5052-5061, 2018.","title":"Programmatically interpretable reinforcement learning","authors":"Verma","year":"2018","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Verma et al., 2018b] Abhinav Verma, Vijayaraghavan Mu- rali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaud- huri. Programmatically interpretable reinforcement learn- ing. In International Conference on Machine Learning, pages 5045-5054. PMLR, 2018.","title":"Programmatically interpretable reinforcement learning","authors":"Verma","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Verma et al., 2019] Abhinav Verma, Hoang Le, Yisong Yue, and Swarat Chaudhuri. Imitation-projected program- matic reinforcement learning. In Advances in Neural In- formation Processing Systems, volume 32, pages 1-12.","title":"Imitation-projected programmatic reinforcement learning","authors":"Abhinav Verma, Hoang Le, Yisong Yue, Swarat Chaudhuri","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"Curran Associates, Inc., 2019.","title":null,"authors":"Curran Associates, Inc","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}