{"type":"ScholarlyArticle","title":"Incremental Learning of Planning Actions in Model-Based Reinforcement Learning","authors":[{"name":"Jun Hao Alvin Ng","slug":"jun-hao-alvin-ng","given":"Jun Hao Alvin","family":"Ng","affiliation":"Department of Computer Science , Heriot-Watt University; School of Informatics , University of Edinburgh"},{"name":"Ronald P. A. Petrick","slug":"ronald-p-a-petrick","given":"Ronald P. A.","family":"Petrick","affiliation":"Department of Computer Science , Heriot-Watt University"}],"abstract":"The soundness and optimality of a plan depends on the correctness of the domain model. Specifying complete domain models can be difficult when interactions between an agent and its environment are complex. We propose a model-based reinforcement learning (MBRL) approach to solve planning problems with unknown models. The model is learned incrementally over episodes using only experiences from the current episode which suits non-stationary environments. We introduce the novel concept of reliability as an intrinsic motivation for MBRL, and a method to learn from failure to prevent repeated instances of similar failures. Our motivation is to improve the learning efficiency and goal-directedness of MBRL. We evaluate our work with experimental results for three planning domains.","doi":"10.24963/ijcai.2019/443","url":"https://ijcai.org/proceedings/2019/443","pdf":"https://ijcai.org/proceedings/2019/0443.pdf","markdown":"https://ijcai.org/proceedings/2019/443.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/443","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":443,"pages":{"first":3195,"last":3201},"keywords":["Machine Learning: Reinforcement Learning","Planning and Scheduling: Planning Algorithms","Knowledge Representation and Reasoning: Action, Change and Causality"],"videos":[],"references":[{"position":1,"raw":"[Bengio et al., 2009] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proc. ICML, pages 41-48, 2009.","title":"Curriculum learning","authors":"Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston","year":"2009","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brafman and Tennenholtz, 2002] Ronen I Brafman and Moshe Tennenholtz. R-MAX -A general polynomial time algorithm for near-optimal reinforcement learning. JMLR, 3:213-231, 2002.","title":"R-MAX -A general polynomial time algorithm for near-optimal reinforcement learning","authors":"Ronen I Brafman, Moshe Tennenholtz","year":"2002","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chentanez et al., 2005] Nuttapong Chentanez, Andrew G Barto, and Satinder P Singh. Intrinsically motivated re- inforcement learning. In Proc. NIPS, pages 1281-1288, 2005.","title":"Intrinsically motivated reinforcement learning","authors":"Chentanez","year":"2005","venue":"Proc. NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cullen and Bryman, 1988] Jet Cullen and Alan Bryman. The knowledge acquisition bottleneck: time for reassess- ment? Expert Systems, 5(3):216-225, 1988.","title":"The knowledge acquisition bottleneck: time for reassessment?","authors":"Bryman Cullen, Jet Cullen, Alan Bryman","year":"1988","venue":"Expert Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Diuk et al., 2009] Carlos Diuk, Lihong Li, and Bethany R Leffler. The adaptive k-meteorologists problem and its ap- plication to structure learning and feature selection in rein- forcement learning. In Proc. ICML, pages 249-256, 2009.","title":"The adaptive k-meteorologists problem and its application to structure learning and feature selection in reinforcement learning","authors":"Carlos Diuk, Lihong Li, Bethany R Leffler","year":"2009","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gil, 1994] Yolanda Gil. Learning by experimentation: In- cremental refinement of incomplete planning domains. In Proc. ICML, pages 87-95, 1994.","title":"Learning by experimentation: Incremental refinement of incomplete planning domains","authors":"Yolanda Gil","year":"1994","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Guestrin et al., 2002] Carlos Guestrin, Relu Patrascu, and Dale Schuurmans. Algorithm-directed exploration for model-based reinforcement learning in factored mdps. In Proc. ICML, pages 235-242, 2002.","title":"Algorithm-directed exploration for model-based reinforcement learning in factored mdps","authors":"Carlos Guestrin, Relu Patrascu, Dale Schuurmans","year":"2002","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hester and Stone, 2017] Todd Hester and Peter Stone. In- trinsically motivated model learning for developing curi- ous robots. AIJ, 247:170-186, 2017.","title":"Intrinsically motivated model learning for developing curious robots","authors":"Stone Hester, Todd Hester, Peter Stone","year":"2017","venue":"AIJ","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kearns and Koller, 1999] Michael Kearns and Daphne Koller. Efficient reinforcement learning in factored MDPs. In Proc. IJCAI, pages 740-747, 1999.","title":"Efficient reinforcement learning in factored MDPs","authors":"Koller Kearns, Michael Kearns, Daphne Koller","year":"1999","venue":"Proc. IJCAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kolobov et al., 2012] Andrey Kolobov, Mausam, and Daniel S Weld. LRTDP vs. UCT for online probabilistic planning. In Proc. AAAI, page 1786-1792, 2012.","title":"LRTDP vs. UCT for online probabilistic planning","authors":"Andrey Kolobov, Daniel S Mausam, Weld","year":"2012","venue":"Proc. AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lang et al., 2012] Tobias Lang, Marc Toussaint, and Kris- tian Kersting. Exploration in relational domains for model- based reinforcement learning. JMLR, 13:3725-3768, 2012.","title":"Exploration in relational domains for modelbased reinforcement learning","authors":"Lang","year":"2012","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Little and Thiebaux, 2007] Iain Little and Sylvie Thiebaux. Probabilistic planning vs. replanning. In Proc. ICAPS Workshop on the International Planning Competition: Past, Present and Future, 2007.","title":"Probabilistic planning vs. replanning","authors":"Iain Little, Sylvie Thiebaux","year":"2007","venue":"Proc. ICAPS Workshop on the International Planning Competition: Past, Present and Future","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Martínez et al., 2016] David Martínez, Guillem Alenya, Carme Torras, Tony Ribeiro, and Katsumi Inoue. Learning relational dynamics of stochastic domains for planning. In Proc. ICAPS, page 235-243, 2016.","title":"Learning relational dynamics of stochastic domains for planning","authors":"David Martínez, Guillem Alenya, Carme Torras, Tony Ribeiro, Katsumi Inoue","year":"2016","venue":"Proc. ICAPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mourão et al., 2012] Kira Mourão, Luke S Zettlemoyer, Ronald P. A. Petrick, and Mark Steedman. Learning STRIPS operators from noisy and incomplete observa- tions. In Proc. UAI, pages 614-623, 2012.","title":"Learning STRIPS operators from noisy and incomplete observations","authors":"Kira Mourão, Luke S Zettlemoyer, P A Ronald, Mark Petrick, Steedman","year":"2012","venue":"Proc. UAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Oates and Cohen, 1996] Tim Oates and Paul R Cohen. Learning planning operators with conditional and proba- bilistic effects. In Proc. AAAI Spring Symposium on Plan- ning with Incomplete Information for Robot Problems, pages 86-94, 1996.","title":"Learning planning operators with conditional and probabilistic effects","authors":"Cohen Oates, Tim Oates, Paul R Cohen","year":"1996","venue":"Proc. AAAI Spring Symposium on Plan-ning with Incomplete Information for Robot Problems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pasula et al., 2007] Hanna M Pasula, Luke S Zettlemoyer, and Leslie Pack Kaelbling. Learning symbolic models of stochastic domains. JAIR, 29:309-352, 2007.","title":"Learning symbolic models of stochastic domains","authors":"Luke S Hanna M Pasula, Leslie Zettlemoyer, Kaelbling Pack","year":"2007","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rodrigues et al., 2010] Christophe Rodrigues, Pierre Gérard, and Céline Rouveirol. Incremental learning of relational action models in noisy environments. In Proc. ILP, pages 206-213, 2010.","title":"Incremental learning of relational action models in noisy environments","authors":"Christophe Rodrigues, Pierre Gérard, Céline Rouveirol","year":"2010","venue":"Proc. ILP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Walsh et al., 2010] Thomas J Walsh, Kaushik Subramanian, Michael L Littman, and Carlos Diuk. Generalizing appren- ticeship learning across hypothesis classes. In Proc. ICML, pages 1119-1126, 2010.","title":"Generalizing apprenticeship learning across hypothesis classes","authors":"J Thomas, Kaushik Walsh, Subramanian, Carlos Michael L Littman, Diuk","year":"2010","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang, 1995] Xuemei Wang. Learning by observation and practice: An incremental approach for planning operator acquisition. In Proc. ICML, pages 549-557, 1995.","title":"Learning by observation and practice: An incremental approach for planning operator acquisition","authors":"Xuemei Wang","year":"1995","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Younes and Littman, 2004] Håkan LS Younes and Michael L Littman. PPDDL1. 0: An extension to PDDL for expressing planning domains with probabilistic effects. Technical Report CMU-CS-04-162, 2004.","title":"PPDDL1. 0: An extension to PDDL for expressing planning domains with probabilistic effects","authors":"Littman ; Younes, L S Håkan, Younes, Michael L Littman","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Younes et al., 2005] Håkan LS Younes, Michael L Littman, David Weissman, and John Asmuth. The first probabilis- tic track of the International Planning Competition. JAIR, 24:851-887, 2005.","title":"The first probabilistic track of the International Planning Competition","authors":"L S Håkan, Younes, David Michael L Littman, John Weissman, Asmuth","year":"2005","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhuo et al., 2013] Hankz Hankui Zhuo, Tuan Anh Nguyen, and Subbarao Kambhampati. Refining incomplete plan- ning domain models through plan traces. In Proc. IJCAI, pages 2451-2458, 2013.","title":"Refining incomplete planning domain models through plan traces","authors":"Zhuo","year":"2013","venue":"Proc. IJCAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}