{"type":"ScholarlyArticle","title":"Solving Hard AI Planning Instances Using Curriculum-Driven Deep Reinforcement Learning","authors":[{"name":"Dieqiao Feng","slug":"dieqiao-feng","given":"Dieqiao","family":"Feng","affiliation":"Department of Computer Science Cornell University"},{"name":"Carla Gomes","slug":"carla-gomes","given":"Carla","family":"Gomes","affiliation":"Department of Computer Science Cornell University"},{"name":"Bart Selman","slug":"bart-selman","given":"Bart","family":"Selman","affiliation":"Department of Computer Science Cornell University"}],"abstract":"Despite significant progress in general AI planning, certain domains remain out of reach of current AI planning systems. Sokoban is a PSPACE-complete planning task and represents one of the hardest domains for current AI planners. Even domain-specific specialized search methods fail quickly due to the exponential search complexity on hard instances. Our approach based on deep reinforcement learning augmented with a curriculum-driven method is the first one to solve hard instances within one day of training while other modern solvers cannot solve these instances within any reasonable time limit. In contrast to prior efforts, which use carefully handcrafted pruning techniques, our approach automatically uncovers domain structure. Our results reveal that deep RL provides a promising framework for solving previously unsolved AI planning problems, provided a proper training curriculum can be devised.","doi":"10.24963/ijcai.2020/304","url":"https://ijcai.org/proceedings/2020/304","pdf":"https://ijcai.org/proceedings/2020/0304.pdf","markdown":"https://ijcai.org/proceedings/2020/304.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/304","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":304,"pages":{"first":2198,"last":2205},"keywords":["Planning and Scheduling: Planning Algorithms","Machine Learning: Deep Reinforcement Learning","Heuristic Search and Game Playing: Heuristic Search and Machine Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25347/file_25347.m3u8","length_sec":286,"provider":"hls"}],"references":[{"position":1,"raw":"[Abel et al., 2015] David Abel, David Ellis Hershkowitz, Gabriel Barth-Maron, Stephen Brawner, Kevin O'Farrell, James MacGlashan, and Stefanie Tellex. Goal-based ac- tion priors. In Twenty-Fifth International Conference on Automated Planning and Scheduling, 2015.","title":"Goal-based action priors","authors":"David Abel, David Ellis Hershkowitz, Gabriel Barth-Maron, Stephen Brawner, O' Kevin, James Farrell, Stefanie Macglashan, Tellex","year":"2015","venue":"Twenty-Fifth International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2009] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learn- ing. In Proceedings of the 26th annual international con- ference on machine learning, pages 41-48, 2009.","title":"Curriculum learning","authors":"Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston","year":"2009","venue":"Proceedings of the 26th annual international conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blum and Furst, 1997] Avrim L Blum and Merrick L Furst. Fast planning through planning graph analysis. Artificial intelligence, 90(1-2):281-300, 1997.","title":"Fast planning through planning graph analysis","authors":"L Avrim, Merrick L Blum, Furst","year":"1997","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bylander, 1994] Tom Bylander. The computational com- plexity of propositional strips planning. Artificial Intel- ligence, 69(1-2):165-204, 1994.","title":"The computational complexity of propositional strips planning","authors":"Tom Bylander, Bylander","year":"1994","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Culberson, 1997] Joseph Culberson. Sokoban is pspace- complete. University of Alberta, Technical Report, TRID- ID TR97-02, 1997.","title":"Sokoban is pspacecomplete","authors":"Joseph Culberson","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fern et al., 2011] Alan Fern, Roni Khardon, and Prasad Tadepalli. The first learning track of the international plan- ning competition. Machine Learning, 84(1-2):81-107, 2011.","title":"The first learning track of the international planning competition","authors":"Alan Fern, Roni Khardon, Prasad Tadepalli","year":"2011","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fox and Long, 2003] Maria Fox and Derek Long. Pddl2. 1: An extension to pddl for expressing temporal planning do- mains. Journal of artificial intelligence research, 20:61- 124, 2003.","title":"Pddl2. 1: An extension to pddl for expressing temporal planning domains","authors":"Maria Fox, Derek Long","year":"2003","venue":"Journal of artificial intelligence research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gödel, 1931] Kurt Gödel. Über formal unentscheidbare sätze der principia mathematica und verwandter systeme i. Monatshefte für mathematik und physik, 38(1):173-198, 1931.","title":"Kurt Gödel. Über formal unentscheidbare sätze der principia mathematica und verwandter systeme i","authors":"Gödel","year":"1931","venue":"Monatshefte für mathematik und physik","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Groshev et al., 2018] Edward Groshev, Aviv Tamar, Maxwell Goldstein, Siddharth Srivastava, and Pieter Abbeel. Learning generalized reactive policies using deep neural networks. In 2018 AAAI Spring Symposium Series, 2018.","title":"Learning generalized reactive policies using deep neural networks","authors":"Groshev","year":"2018","venue":"AAAI Spring Symposium Series","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hearn and Demaine, 2005] Robert A Hearn and Erik D De- maine. Pspace-completeness of sliding-block puzzles and other problems through the nondeterministic constraint logic model of computation. Theoretical Computer Sci- ence, 343(1-2):72-96, 2005.","title":"Pspace-completeness of sliding-block puzzles and other problems through the nondeterministic constraint logic model of computation","authors":"A Robert, Erik D Hearn, Demaine","year":"2005","venue":"Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hoffmann, 2001] Jörg Hoffmann. Ff: The fast-forward planning system. AI magazine, 22(3):57-57, 2001.","title":"Ff: The fast-forward planning system","authors":"Jörg Hoffmann, Hoffmann","year":"2001","venue":"AI magazine","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Järvisalo et al., 2012] Matti Järvisalo, Daniel Le Berre, Olivier Roussel, and Laurent Simon. The international sat solver competitions. Ai Magazine, 33(1):89-92, 2012.","title":"The international sat solver competitions","authors":"Matti Järvisalo, Daniel Le Berre, Olivier Roussel, Laurent Simon","year":"2012","venue":"Ai Magazine","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Junghanns and Schaeffer, 2001] Andreas Junghanns and Jonathan Schaeffer. Sokoban: Improving the search with relevance cuts. Theoretical Computer Science, 252(1- 2):151-175, 2001.","title":"Sokoban: Improving the search with relevance cuts","authors":"Schaeffer Junghanns, Andreas Junghanns, Jonathan Schaeffer","year":"2001","venue":"Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kautz and Selman, 1998] Henry Kautz and Bart Selman. Blackbox: A new approach to the application of theorem proving to problem solving. In AIPS98 Workshop on Plan- ning as Combinatorial Search, volume 58260, pages 58- 60, 1998.","title":"Blackbox: A new approach to the application of theorem proving to problem solving","authors":"Selman Kautz, Henry Kautz, Bart Selman","year":"1998","venue":"AIPS98 Workshop on Planning as Combinatorial Search","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Koehler et al., 1997] Jana Koehler, Bernhard Nebel, Jörg Hoffmann, and Yannis Dimopoulos. Extending planning graphs to an adl subset. In European Conference on Plan- ning, pages 273-285. Springer, 1997.","title":"Extending planning graphs to an adl subset","authors":"Jana Koehler, Bernhard Nebel, Jörg Hoffmann, Yannis Dimopoulos","year":"1997","venue":"European Conference on Planning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lipovetzky, 2013] Nir Lipovetzky. Structure and inference in classical planning. PhD thesis, Universitat Pompeu Fabra, 2013.","title":"Structure and inference in classical planning","authors":"Nir Lipovetzky","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Marques-Silva and Sakallah, 1999] Joao P Marques-Silva and Karem A Sakallah. Grasp: A search algorithm for propositional satisfiability. IEEE Transactions on Com- puters, 48(5):506-521, 1999.","title":"Grasp: A search algorithm for propositional satisfiability","authors":"Joao P Marques-Silva, Karem A Sakallah","year":"1999","venue":"IEEE Transactions on Computers","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rosman and Ramamoorthy, 2012] Benjamin Rosman and Subramanian Ramamoorthy. What good are actions? accelerating learning using learned action priors. In 2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL), pages 1-6. IEEE, 2012.","title":"What good are actions? accelerating learning using learned action priors","authors":"Ramamoorthy Rosman, Benjamin Rosman, Subramanian Ramamoorthy","year":"2012","venue":"IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Selman et al., 1992] Bart Selman, Hector J Levesque, David G Mitchell, et al. A new method for solving hard satisfiability problems. In Aaai, volume 92, pages 440- 446. Citeseer, 1992.","title":"A new method for solving hard satisfiability problems","authors":"Bart Selman, Hector J Levesque, David G Mitchell","year":"1992","venue":"Aaai","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess-Che ; Marc, Lanctot","year":"2016","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Silver et al., 2017] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815, 2017.","title":"Mastering chess and shogi by self-play with a general reinforcement learning algorithm","authors":"David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vallati et al., 2015] Mauro Vallati, Lukas Chrpa, Marek Grześ, Thomas Leo McCluskey, Mark Roberts, Scott San- ner, et al. The 2014 international planning competition: Progress and trends. Ai Magazine, 36(3):90-98, 2015.","title":"The 2014 international planning competition: Progress and trends","authors":"Mauro Vallati, Lukas Chrpa, Marek Grześ, Thomas Leo Mccluskey, Mark Roberts, Scott Sanner","year":"2015","venue":"Ai Magazine","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Weber et al., 2017] Théophane Weber, Sébastien Racanière, David P Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdomenech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, et al. Imagination-augmented agents for deep reinforcement learning. arXiv preprint arXiv:1707.06203, 2017.","title":"Imagination-augmented agents for deep reinforcement learning","authors":"Weber","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Welle, 2003] James Welle. A comparison of parallelism in current ai planning systems. 2003.","title":"A comparison of parallelism in current ai planning systems","authors":"James Welle","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}