{"type":"ScholarlyArticle","title":"Action Space Reduction for Planning Domains","authors":[{"name":"Harsha Kokel","slug":"harsha-kokel","given":"Harsha","family":"Kokel","affiliation":"IBM T.J. Watson Research Center , Yorktown Heights , USA"},{"name":"Junkyu Lee","slug":"junkyu-lee","given":"Junkyu","family":"Lee","affiliation":"IBM T.J. Watson Research Center , Yorktown Heights , USA"},{"name":"Michael Katz","slug":"michael-katz","given":"Michael","family":"Katz","affiliation":"IBM T.J. Watson Research Center , Yorktown Heights , USA"},{"name":"Kavitha Srinivas","slug":"kavitha-srinivas","given":"Kavitha","family":"Srinivas","affiliation":"IBM T.J. Watson Research Center , Yorktown Heights , USA"},{"name":"Shirin Sohrabi","slug":"shirin-sohrabi","given":"Shirin","family":"Sohrabi","affiliation":"IBM T.J. Watson Research Center , Yorktown Heights , USA"}],"abstract":"Planning tasks succinctly represent labeled transition systems, with each ground action corresponding to a label. This granularity, however, is not necessary for solving planning tasks and can be harmful, especially for model-free methods. In order to apply such methods, the label sets are often manually reduced. In this work, we propose automating this manual process. We characterize a valid label reduction for classical planning tasks and propose an automated way of obtaining such valid reductions by leveraging lifted mutex groups. Our experiments show a significant reduction in the action label space size across a wide collection of planning domains. We demonstrate the benefit of our automated label reduction in two separate use cases: improved sample complexity of model-free reinforcement learning algorithms and speeding up successor generation in lifted planning. The code and supplementary material are available at https://github.com/IBM/Parameter-Seed-Set.","doi":"10.24963/ijcai.2023/599","url":"https://ijcai.org/proceedings/2023/599","pdf":"https://ijcai.org/proceedings/2023/0599.pdf","markdown":"https://ijcai.org/proceedings/2023/599.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/599","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Planning and Scheduling","track":"Main Track","number":599,"pages":{"first":5394,"last":5401},"keywords":["Planning and Scheduling: PS: Theoretical foundations of planning"],"videos":[],"references":[{"position":1,"raw":"[Bäckström and Nebel, 1995] Christer Bäckström and Bern- hard Nebel. Complexity results for SAS + planning. Com- put. Intell., 11(4):625-655, 1995.","title":"Christer Bäckström and Bernhard Nebel","authors":"Nebel Bäckström","year":"1995","venue":"Comput. Intell","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bamford and Ovalle, 2021] Christopher Bamford and Al- varo Ovalle. Generalising discrete action spaces with con- ditional action trees. In 2021 IEEE Conference on Games (CoG), pages 1-8, 2021.","title":"Generalising discrete action spaces with conditional action trees","authors":"Christopher Bamford, Alvaro Ovalle","year":"2021","venue":"IEEE Conference on Games (CoG)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Boutilier et al., 2018] Craig Boutilier, Alon Cohen, Avinatan Hassidim, Yishay Mansour, Ofer Meshi, Martin Mladenov, and Dale Schuurmans. Planning and learning with stochastic action sets. In IJCAI, pages 4674-4682, 2018.","title":"Planning and learning with stochastic action sets","authors":"Craig Boutilier, Alon Cohen, Avinatan Hassidim, Yishay Mansour, Ofer Meshi, Martin Mladenov, Dale Schuurmans","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/650"},{"position":4,"raw":"[Corrêa et al., 2020] Augusto B. Corrêa, Florian Pommeren- ing, Malte Helmert, and Guillem Francès. Lifted succes- sor generation using query optimization techniques. In ICAPS, pages 80-89, 2020.","title":"Lifted successor generation using query optimization techniques","authors":"Augusto B Corrêa, Florian Pommerening, Malte Helmert, Guillem Francès","year":"2020","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Corrêa et al., 2021] Augusto B. Corrêa, Guillem Francès, Florian Pommerening, and Malte Helmert. Delete- relaxation heuristics for lifted classical planning. In ICAPS, pages 94-102, 2021.","title":"Deleterelaxation heuristics for lifted classical planning","authors":"Augusto B Corrêa, Guillem Francès, Florian Pommerening, Malte Helmert","year":"2021","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dzeroski et al., 2001] Saso Dzeroski, Luc De Raedt, and Kurt Driessens. Relational reinforcement learning. ML, 43(1/2):7-52, 2001.","title":"Relational reinforcement learning","authors":"Saso Dzeroski, Luc De Raedt, Kurt Driessens","year":"2001","venue":"ML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fern et al., 2006] Alan Fern, Sung Wook Yoon, and Robert Givan. Approximate policy iteration with a policy lan- guage bias: Solving relational markov decision processes. JAIR, 25:75-118, 2006.","title":"Approximate policy iteration with a policy language bias: Solving relational markov decision processes","authors":"Alan Fern, Sung Wook Yoon, Robert Givan","year":"2006","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fišer, 2020] Daniel Fišer. Lifted fact-alternating mutex groups and pruned grounding of classical planning prob- lems. In AAAI, pages 9835-9842, 2020.","title":"Lifted fact-alternating mutex groups and pruned grounding of classical planning problems","authors":"Daniel Fišer","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gefen and Brafman, 2011] Avitan Gefen and Ronen I. Braf- man. The minimal seed set problem. In ICAPS, number 1, pages 319-322, 2011.","title":"Brafman. The minimal seed set problem","authors":"Avitan Gefen, Ronen I","year":"2011","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gehring et al., 2022] Clement Gehring, Masataro Asai, Ro- han Chitnis, Tom Silver, Leslie Pack Kaelbling, Shirin Sohrabi, and Michael Katz. Reinforcement learning for classical planning: Viewing heuristics as dense reward generators. In ICAPS, pages 588-596, 2022.","title":"Reinforcement learning for classical planning: Viewing heuristics as dense reward generators","authors":"Clement Gehring, Masataro Asai, Rohan Chitnis, Tom Silver, Leslie Pack Kaelbling, Shirin Sohrabi, Michael Katz","year":"2022","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Geißer et al., 2020] Florian Geißer, David Speck, and Thomas Keller. Trial-based heuristic tree search for mdps with factored action spaces. In SOCS, pages 38-47, 2020.","title":"Trial-based heuristic tree search for mdps with factored action spaces","authors":"Geißer","year":"2020","venue":"SOCS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Guestrin et al., 2002] Carlos Guestrin, Michail G. Lagoudakis, and Ronald Parr. Coordinated reinforcement learning. In ICML, pages 227-234, 2002.","title":"Coordinated reinforcement learning","authors":"Carlos Guestrin, Michail G Lagoudakis, Ronald Parr","year":"2002","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Haslum, 2011] Patrik Haslum. Computing genome edit dis- tances using domain-independent planning. In Scheduling and Planning Applications woRKshop (SPARK) at ICAPS, pages 45-51, 2011.","title":"Computing genome edit distances using domain-independent planning","authors":"Patrik Haslum, Haslum","year":"2011","venue":"Scheduling and Planning Applications woRKshop (SPARK) at ICAPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Helmert et al., 2014] Malte Helmert, Patrik Haslum, Jörg Hoffmann, and Raz Nissim. Merge-and-shrink abstrac- tion: A method for generating lower bounds in factored state spaces. Journal of the ACM, 61(3):16:1-63, 2014.","title":"Merge-and-shrink abstraction: A method for generating lower bounds in factored state spaces","authors":"Malte Helmert, Patrik Haslum, Jörg Hoffmann, Raz Nissim","year":"2014","venue":"Journal of the ACM","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Helmert, 2006] Malte Helmert. The Fast Downward plan- ning system. JAIR, 26:191-246, 2006.","title":"The Fast Downward planning system","authors":"Malte Helmert, Helmert","year":"2006","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Helmert, 2009] Malte Helmert. Concise finite-domain rep- resentations for PDDL planning tasks. Artificial Intelli- gence, 173:503-535, 2009.","title":"Concise finite-domain representations for PDDL planning tasks","authors":"Malte Helmert, Helmert","year":"2009","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hoffman et al., 2020] Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, et al. Acme: A re- search framework for distributed reinforcement learning. CoRR, abs/2006.00979, 2020.","title":"Acme: A research framework for distributed reinforcement learning","authors":"Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Huang and Ontañón, 2022] Shengyi Huang and Santiago Ontañón. A closer look at invalid action masking in pol- icy gradient algorithms. In The Thirty Fifth International Florida Artificial Intelligence Research Society FLAIRS, 2022.","title":"A closer look at invalid action masking in policy gradient algorithms","authors":"Shengyi Huang, Santiago Ontañón","year":"2022","venue":"The Thirty Fifth International Florida Artificial Intelligence Research Society FLAIRS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kanervisto et al., 2020] Anssi Kanervisto, Christian Scheller, and Ville Hautamäki. Action space shaping in deep reinforcement learning. In 2020 IEEE Conference on Games (CoG), pages 479-486, 2020.","title":"Action space shaping in deep reinforcement learning","authors":"Anssi Kanervisto, Christian Scheller, Ville Hautamäki","year":"2020","venue":"IEEE Conference on Games (CoG)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lauer et al., 2021] Pascal Lauer, Álvaro Torralba, Daniel Fiser, Daniel Höller, Julia Wichlacz, and Jörg Hoffmann. Polynomial-time in PDDL input size: Making the delete relaxation feasible for lifted planning. In IJCAI, pages 4119-4126, 2021.","title":"Polynomial-time in PDDL input size: Making the delete relaxation feasible for lifted planning","authors":"Pascal Lauer, Álvaro Torralba, Daniel Fiser, Daniel Höller, Julia Wichlacz, Jörg Hoffmann","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/567"},{"position":21,"raw":"[Matloob and Soutchanski, 2016] Rami Matloob and Mikhail Soutchanski. Exploring organic synthesis with state-of-the-art planning techniques. In Scheduling and Planning Applications woRKshop (SPARK) at ICAPS, pages 52-61, 2016.","title":"Exploring organic synthesis with state-of-the-art planning techniques","authors":"Rami Matloob, Mikhail Soutchanski","year":"2016","venue":"Scheduling and Planning Applications woRKshop (SPARK) at ICAPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[McDermott, 2000] Drew V. McDermott. The 1998 AI plan- ning systems competition. AI Mag., 21(2):35-55, 2000.","title":"The 1998 AI planning systems competition","authors":"V Drew, Mcdermott","year":"2000","venue":"AI Mag","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Nelson, 2021] Mark J. Nelson. Estimates for the branching factors of atari games. In 2021 IEEE Conference on Games (CoG), pages 1-5, 2021.","title":"Estimates for the branching factors of atari games","authors":"Mark J Nelson","year":"2021","venue":"IEEE Conference on Games (CoG)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pazis and Lagoudakis, 2011] Jason Pazis and Michail G. Lagoudakis. Reinforcement learning in multidimensional continuous action spaces. In IEEE Symposium on Adap- tive Dynamic Programming And Reinforcement, pages 97- 104, 2011.","title":"Reinforcement learning in multidimensional continuous action spaces","authors":"Jason Pazis, Michail G Lagoudakis","year":"2011","venue":"IEEE Symposium on Adaptive Dynamic Programming And Reinforcement","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sievers et al., 2014] Silvan Sievers, Martin Wehrle, and Malte Helmert. Generalized label reduction for merge- and-shrink heuristics. In AAAI, pages 2358-2366, 2014.","title":"Generalized label reduction for mergeand-shrink heuristics","authors":"Sievers","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Silver and Chitnis, 2020] Tom Silver and Rohan Chitnis. Pddlgym: Gym environments from PDDL problems. CoRR, abs/2002.06432, 2020.","title":"Tom Silver and Rohan Chitnis. Pddlgym: Gym environments from PDDL problems","authors":"Chitnis Silver","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ullman, 1989] Jeffrey D. Ullman. Principles of Database and Knowledge-Base Systems, Volume II. Computer Sci- ence Press, 1989.","title":"Principles of Database and Knowledge-Base Systems","authors":"Jeffrey D Ullman","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}