{"type":"ScholarlyArticle","title":"PG3: Policy-Guided Planning for Generalized Policy Generation","authors":[{"name":"Ryan Yang","slug":"ryan-yang","given":"Ryan","family":"Yang","affiliation":"Massachusetts Institute of Technology"},{"name":"Tom Silver","slug":"tom-silver","given":"Tom","family":"Silver","affiliation":"Massachusetts Institute of Technology"},{"name":"Aidan Curtis","slug":"aidan-curtis","given":"Aidan","family":"Curtis","affiliation":"Massachusetts Institute of Technology"},{"name":"Tomas Lozano-Perez","slug":"tomas-lozano-perez","given":"Tomas","family":"Lozano-Perez","affiliation":"Massachusetts Institute of Technology"},{"name":"Leslie Kaelbling","slug":"leslie-kaelbling","given":"Leslie","family":"Kaelbling","affiliation":"Massachusetts Institute of Technology"}],"abstract":"A longstanding objective in classical planning is to synthesize policies that generalize across multiple problems from the same domain. In this work, we study generalized policy search-based methods with a focus on the score function used to guide the search over policies. We demonstrate limitations of two score functions --- policy evaluation and plan comparison --- and propose a new approach that overcomes these limitations. The main idea behind our approach, Policy-Guided Planning for Generalized Policy Generalization (PG3), is that a candidate policy should be used to guide planning on training problems as a mechanism for evaluating that candidate. Theoretical results in a simplified setting give conditions under which PG3 is optimal or admissible. We then study a specific instantiation of policy search where planning problems are PDDL-based and policies are lifted decision lists. Empirical results in six domains confirm that PG3 learns generalized policies more efficiently and effectively than several baselines.","doi":"10.24963/ijcai.2022/650","url":"https://ijcai.org/proceedings/2022/650","pdf":"https://ijcai.org/proceedings/2022/0650.pdf","markdown":"https://ijcai.org/proceedings/2022/650.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/650","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Planning and Scheduling","track":"Main Track","number":650,"pages":{"first":4686,"last":4692},"keywords":["Machine Learning: Relational Learning","Search: Heuristic Search","Planning and Scheduling: Learning in Planning and Scheduling"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986771","length_sec":86,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984914","length_sec":352,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bonet and Geffner, 2015] Blai Bonet and Hector Geffner. Policies that generalize: Solving many planning problems with the same policy. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.","title":"Policies that generalize: Solving many planning problems with the same policy","authors":"Geffner ; ] Blai Bonet, Hector Bonet, Geffner","year":"2015","venue":"Twenty-Fourth International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2015/396"},{"position":2,"raw":"[Bonet and Geffner, 2018] Blai Bonet and Hector Geffner. Features, projections, and representation change for gen- eralized planning. CoRR, abs/1801.10055, 2018.","title":"Features, projections, and representation change for generalized planning","authors":"Geffner Bonet, Blai Bonet, Hector Geffner","year":"2018","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2018/649"},{"position":3,"raw":"[Bonet et al., 2019] Blai Bonet, Guillem Frances, and Hector Geffner. Learning features and abstract actions for com- puting generalized plans. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 33, pages 2703- 2710, 2019.","title":"Learning features and abstract actions for computing generalized plans","authors":"Blai Bonet, Guillem Frances, Hector Geffner","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fikes et al., 1972] Richard E Fikes, Peter E Hart, and Nils J Nilsson. Learning and executing generalized robot plans. Artificial intelligence, 3:251-288, 1972.","title":"Learning and executing generalized robot plans","authors":"Peter E Richard E Fikes, Nils J Hart, Nilsson","year":"1972","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Groshev et al., 2018] Edward Groshev, Maxwell Goldstein, Aviv Tamar, Siddharth Srivastava, and Pieter Abbeel. Learning generalized reactive policies using deep neural networks. In Proceedings of the International Conference on Automated Planning and Scheduling, 2018.","title":"Learning generalized reactive policies using deep neural networks","authors":"Edward Groshev, Maxwell Goldstein, Aviv Tamar, Siddharth Srivastava, Pieter Abbeel","year":"2018","venue":"Proceedings of the International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Illanes and McIlraith, 2019] León Illanes and Sheila A. McIlraith. Generalized planning via abstraction: Arbitrary numbers of objects. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):7610-7618, Jul. 2019.","title":"Generalized planning via abstraction: Arbitrary numbers of objects","authors":"Mcilraith Illanes, León Illanes, Sheila A Mcilraith","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jiménez and Jonsson, 2015] Sergio Jiménez and Anders Jonsson. Computing plans with control flow and pro- cedures using a classical planner. In Proceedings of the Eighth Annual Symposium on Combinatorial Search, SOCS-15, pages 62-69, 2015.","title":"Computing plans with control flow and procedures using a classical planner","authors":"Jonsson Jiménez, Sergio Jiménez, Anders Jonsson","year":"2015","venue":"Proceedings of the Eighth Annual Symposium on Combinatorial Search, SOCS-15","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jiménez et al., 2019] Sergio Jiménez, Javier Segovia- Aguas, and Anders Jonsson. A review of generalized planning. The Knowledge Engineering Review, 34:e5, 2019.","title":"A review of generalized planning","authors":"Sergio Jiménez, Javier Segovia-Aguas, Anders Jonsson","year":"2019","venue":"The Knowledge Engineering Review","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Karia and Srivastava, 2020] Rushang Karia and Siddharth Srivastava. Learning generalized relational heuristic networks for model-agnostic planning. arXiv preprint arXiv:2007.06702, 2020.","title":"Learning generalized relational heuristic networks for model-agnostic planning","authors":"Srivastava Karia","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Levesque, 2005] H. Levesque. Planning with loops. In IJ- CAI, 2005.","title":"Planning with loops","authors":"H Levesque","year":"2005","venue":"IJ-CAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Levine and Humphreys, 2003] John Levine and David Humphreys. Learning action strategies for planning domains using genetic programming. In Workshops on Applications of Evolutionary Computation, pages 684-695. Springer, 2003.","title":"Learning action strategies for planning domains using genetic programming","authors":"Humphreys ; ] Levine, John Levine, David Humphreys","year":"2003","venue":"Workshops on Applications of Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[McDermott et al., 1998] Drew McDermott, Malik Ghallab, Adele Howe, Craig Knoblock, Ashwin Ram, Manuela Veloso, Daniel Weld, and David Wilkins. Pddl-the plan- ning domain definition language. Technical report, Yale University, 1998.","title":"Pddl-the planning domain definition language","authors":"Drew Mcdermott, Malik Ghallab, Adele Howe, Craig Knoblock, Ashwin Ram, Manuela Veloso, Daniel Weld, David Wilkins","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mooney and Califf, 1995] Raymond J Mooney and Mary Elaine Califf. Induction of first-order decision lists: Results on learning the past tense of english verbs. Journal of Artificial Intelligence Research, 3:1-24, 1995.","title":"Induction of first-order decision lists: Results on learning the past tense of english verbs","authors":"Califf ; Mooney, J Raymond, Mary Elaine Mooney, Califf","year":"1995","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pong et al., 2018] Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models: Model-free deep rl for model-based control. arXiv preprint arXiv:1802.09081, 2018.","title":"Vitchyr Pong","authors":"Pong","year":"2018","venue":"Temporal difference models: Model-free deep rl for model-based control","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rivlin et al., 2020] Or Rivlin, Tamir Hazan, and Erez Karpas. Generalized planning with deep reinforcement learning. arXiv preprint arXiv:2005.02305, 2020.","title":"Generalized planning with deep reinforcement learning","authors":"Rivlin","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Schaul et al., 2015] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function ap- proximators. In International conference on machine learning, pages 1312-1320. PMLR, 2015.","title":"Universal value function approximators","authors":"Schaul","year":"2015","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Segovia-Aguas et al., 2018] Javier Segovia-Aguas, Sergio Jiménez, and Anders Jonsson. Computing hierarchical fi- nite state controllers with classical planning. Journal of Artificial Intelligence Research, 62:755-797, 2018.","title":"Computing hierarchical finite state controllers with classical planning","authors":"Javier Segovia-Aguas, Sergio Jiménez, Anders Jonsson","year":"2018","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Segovia-Aguas et al., 2021] Javier Segovia-Aguas, Sergio Jiménez, and Anders Jonsson. Generalized planning as heuristic search. In Proceedings of the International Conference on Automated Planning and Scheduling, vol- ume 31, pages 569-577, 2021.","title":"Generalized planning as heuristic search","authors":"Javier Segovia-Aguas, Sergio Jiménez, Anders Jonsson","year":"2021","venue":"Proceedings of the International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Silver and Chitnis, 2020] Tom Silver and Rohan Chitnis. Pddlgym: Gym environments from pddl problems. In International Conference on Automated Planning and Scheduling (ICAPS) PRL Wor kshop, 2020.","title":"Tom Silver and Rohan Chitnis. Pddlgym: Gym environments from pddl problems","authors":"Chitnis Silver","year":"2020","venue":"International Conference on Automated Planning and Scheduling (ICAPS) PRL Wor kshop","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Silver et al., 2021] Tom Silver, Rohan Chitnis, Aidan Curtis, Joshua Tenenbaum, Tomas Lozano-Perez, and Leslie Pack Kaelbling. Planning with learned object im- portance in large problem instances using graph neural net- works. In AAAI, 2021.","title":"Planning with learned object importance in large problem instances using graph neural networks","authors":"Tom Silver, Rohan Chitnis, Aidan Curtis, Joshua Tenenbaum, Tomas Lozano-Perez, Leslie Pack, Kaelbling","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Srivastava et al., 2011] Siddharth Srivastava, N. Immerman, S. Zilberstein, and Tianjiao Zhang. Directed search for generalized plans using classical planners. In ICAPS, 2011.","title":"Directed search for generalized plans using classical planners","authors":"Siddharth Srivastava, N Immerman, S Zilberstein, Tianjiao Zhang","year":"2011","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Srivastava, 2011] Siddharth Srivastava. Foundations and applications of generalized planning. AI Communications, 24(4):349-351, 2011.","title":"Foundations and applications of generalized planning","authors":"Siddharth Srivastava","year":"2011","venue":"AI Communications","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"Elly Zoe Winner. Learning Domain-Specific Planners from Example Plans. PhD thesis, Carnegie Mel- lon University, USA, 2008.","title":"Learning Domain-Specific Planners from Example Plans","authors":"Zoe Elly, Winner","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yoon et al., 2008] Sungwook Yoon, Alan Fern, and Robert Givan. Learning control knowledge for forward search planning. Journal of Machine Learning Research, 9(Apr):683-718, 2008.","title":"Learning control knowledge for forward search planning","authors":"Yoon","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}