{"type":"ScholarlyArticle","title":"PEORL: Integrating Symbolic Planning and Hierarchical Reinforcement Learning for Robust Decision-Making","authors":[{"name":"Fangkai Yang","slug":"fangkai-yang","given":"Fangkai","family":"Yang","affiliation":"Maana Inc. , Bellevue , WA , USA"},{"name":"Daoming Lyu","slug":"daoming-lyu","given":"Daoming","family":"Lyu","affiliation":"Auburn University , Auburn , AL , USA"},{"name":"Bo Liu","slug":"bo-liu","given":"Bo","family":"Liu","affiliation":"Auburn University , Auburn , AL , USA"},{"name":"Steven Gustafson","slug":"steven-gustafson","given":"Steven","family":"Gustafson","affiliation":"Maana Inc. , Bellevue , WA , USA"}],"abstract":"Reinforcement learning and symbolic planning have both been used to build intelligent autonomous agents. Reinforcement learning relies on learning from interactions with real world, which often requires an unfeasibly large amount of experience. Symbolic planning relies on manually crafted symbolic knowledge, which may not be robust to domain uncertainties and changes. In this paper we present a unified framework PEORL that integrates symbolic planning with hierarchical reinforcement learning (HRL) to cope with decision-making in dynamic environment with uncertainties. Symbolic plans are used to guide the agent's task execution and learning, and the learned experience is fed back to symbolic knowledge to improve planning. This method leads to rapid policy search and robust symbolic plans in complex domains. The framework is tested on benchmark domains of HRL.","doi":"10.24963/ijcai.2018/675","url":"https://ijcai.org/proceedings/2018/675","pdf":"https://ijcai.org/proceedings/2018/0675.pdf","markdown":"https://ijcai.org/proceedings/2018/675.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/675","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Planning and Scheduling","track":"Main track","number":675,"pages":{"first":4860,"last":4866},"keywords":["Machine Learning Applications: Applications of Reinforcement Learning","Knowledge Representation and Reasoning: Action, Change and Causality","Machine Learning: Reinforcement Learning","Knowledge Representation and Reasoning: Knowledge Representation Languages","Planning and Scheduling: Applications of Planning","Knowledge Representation and Reasoning: Common-Sense Reasoning"],"videos":[],"references":[{"position":1,"raw":"[Banbara et al., 2017] M. Banbara, B. Kaufmann, M. Os- trowski, and T. Schaub. Clingcon: The next generation. TPLP, 17:408-461, 2017.","title":"Clingcon: The next generation","authors":"M Banbara, B Kaufmann, M Ostrowski, T Schaub","year":"2017","venue":"TPLP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Barto and Mahadevan, 2003] A. G. Barto and S. Mahade- van. Recent advances in hierarchical reinforcement learn- ing. Discrete Event Dynamic Systems, 13(1-2):41-77, Jan- uary 2003.","title":"Recent advances in hierarchical reinforcement learning","authors":"A G Barto, S Mahadevan","year":"2003","venue":"Discrete Event Dynamic Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2016] K. Chen, F. Yang, and X. Chen. Plan- ning with task-oriented knowledge acquisition for a ser- vice robot. In IJCAI, pages 812-818, 2016.","title":"Planning with task-oriented knowledge acquisition for a service robot","authors":"K Chen, F Yang, X Chen","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/120"},{"position":4,"raw":"[Cimatti et al., 2008] A. Cimatti, M. Pistore, and P. Traverso. Automated planning. In Frank van Harmelen, Vladimir Lifschitz, and Bruce Porter, editors, Handbook of Knowl- edge Representation. Elsevier, 2008.","title":"Automated planning","authors":"A Cimatti, M Pistore, P Traverso","year":"2008","venue":"Handbook of Knowledge Representation","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gebser et al., 2012] M. Gebser, B. Kaufmann, and T. Schaub. Conflict-driven answer set solving: From theory to practice. Artificial Intelligence, 187-188:52-89, 2012.","title":"Conflict-driven answer set solving: From theory to practice","authors":"M Gebser, B Kaufmann, T Schaub","year":"2012","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gelfond and Lifschitz, 1998] M. Gelfond and V. Lifschitz. Action languages. Electronic Transactions on Artificial Intelligence (ETAI), 1998.","title":"Action languages","authors":"M Gelfond, V Lifschitz","year":"1998","venue":"Electronic Transactions on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hanheide et al., 2015] M. Hanheide, M. Göbelbecker, G. S Horn, et al. Robot task planning and explanation in open and uncertain worlds. Artificial Intelligence, 2015.","title":"Robot task planning and explanation in open and uncertain worlds","authors":"M Hanheide, M Göbelbecker, G S Horn","year":"2015","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Helmert, 2006] M. Helmert. The fast downward planning system. JAIR, 26:191-246, 2006.","title":"The fast downward planning system","authors":"M Helmert","year":"2006","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hogg et al., 2010] C. Hogg, U. Kuter, and H. Munoz-Avila. Learning methods to generate good plans: Integrating htn learning and reinforcement learning. In AAAI, 2010.","title":"Learning methods to generate good plans: Integrating htn learning and reinforcement learning","authors":"C Hogg, U Kuter, H Munoz-Avila","year":"2010","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jiménez et al., 2013] S. Jiménez, F. Fernández, and D. Bor- rajo. Integrating planning, execution, and learning to im- prove plan execution. Computational Intelligence, 2013.","title":"Integrating planning, execution, and learning to improve plan execution","authors":"S Jiménez, F Fernández, D Borrajo","year":"2013","venue":"Computational Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Khandelwal et al., 2014] P. Khandelwal, F. Yang, M. Leonetti, V. Lifschitz, and P. Stone. Planning in Action Language BC while Learning Action Costs for Mobile Robots. In ICAPS, 2014.","title":"Planning in Action Language BC while Learning Action Costs for Mobile Robots","authors":"P Khandelwal, F Yang, M Leonetti, V Lifschitz, P Stone","year":"2014","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Khandelwal et al., 2017] P. Khandelwal, S. Zhang, J. Sinapov, M. Leonetti, J. Thomason, F. Yang, I. Gori, M. Svetlik, P. Khante, and V. Lifschitz, J. Aggarwal, R. Mooney, P. Stone. Bwibots: A platform for bridging the gap between ai and human-robot interaction research. IJRR, 36(5-7):635-659, 2017.","title":"Bwibots: A platform for bridging the gap between ai and human-robot interaction research","authors":"P Khandelwal, S Zhang, J Sinapov, M Leonetti, J Thomason, F Yang, I Gori, M Svetlik, P Khante, V Lifschitz, J Aggarwal, R Mooney, P Stone","year":"2017","venue":"IJRR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kulkarni et al., 2016] T. D. Kulkarni, K. Narasimhan, A. Saeedi, and J. Tenenbaum. Hierarchical deep reinforce- ment learning: Integrating temporal abstraction and intrin- sic motivation. In Advances in Neural Information Pro- cessing Systems, pages 3675-3683, 2016.","title":"Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation","authors":"T D Kulkarni, K Narasimhan, A Saeedi, J Tenenbaum","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lee et al., 2013] J. Lee, V. Lifschitz, and F. Yang. Action Language BC: A Preliminary Report. In International Joint Conference on Artificial Intelligence (IJCAI), 2013.","title":"Action Language BC: A Preliminary Report","authors":"J Lee, V Lifschitz, F Yang","year":"2013","venue":"International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Leonetti et al., 2012] M. Leonetti, L. Iocchi, and F. Pa- trizi. Automatic generation and learning of finite-state controllers. In International Conference on Artificial Intel- ligence: Methodology, Systems, and Applications, pages 135-144. Springer, 2012.","title":"Automatic generation and learning of finite-state controllers","authors":"M Leonetti, L Iocchi, F Patrizi","year":"2012","venue":"International Conference on Artificial Intelligence: Methodology, Systems, and Applications","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Leonetti et al., 2016] M. Leonetti, L. Iocchi, and P. Stone. A synthesis of automated planning and reinforcement learn- ing for efficient, robust decision-making. Artificial Intelli- gence, 241:103-130, 2016.","title":"A synthesis of automated planning and reinforcement learning for efficient, robust decision-making","authors":"M Leonetti, L Iocchi, P Stone","year":"2016","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lifschitz, 2008] V. Lifschitz. What is answer set program- ming? In Proceedings of the AAAI Conference on Artifi- cial Intelligence, pages 1594-1597. MIT Press, 2008.","title":"What is answer set programming?","authors":"V Lifschitz","year":"2008","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Machado et al., 2017] M. C Machado, M. G Bellemare, and M. Bowling. A laplacian framework for option discovery in reinforcement learning. In International Conference on Machine Learning, 2017.","title":"A laplacian framework for option discovery in reinforcement learning","authors":"M Machado, M G Bellemare, M Bowling","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mahadevan, 1996] S. Mahadevan. Average reward rein- forcement learning: Foundations, algorithms, and empir- ical results. Machine Learning, 22:159-195, 1996.","title":"Average reward reinforcement learning: Foundations, algorithms, and empirical results","authors":"S Mahadevan","year":"1996","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mannor et al., 2004] S. Mannor, I. Menache, A. Hoze, and U. Klein. Dynamic abstraction in reinforcement learning via clustering. In International Conference on Machine Learning, 2004.","title":"Dynamic abstraction in reinforcement learning via clustering","authors":"S Mannor, I Menache, A Hoze, U Klein","year":"2004","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[McCarthy and Hayes, 1969] J. McCarthy and P. Hayes. Some Philosophical Problems From The Standpoint of Ar- tificial Intelligence. In Machine Intelligence. Edinburgh University Press, 1969.","title":"Some Philosophical Problems From The Standpoint of Artificial Intelligence","authors":"J Mccarthy, P Hayes","year":"1969","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[McDermott et al., 1998] D. McDermott, M. Ghallab, A. Howe, C. Knoblock, A. Ram, M. Veloso, D. Weld, and D. Wilkins. Pddl-the planning domain definition language. 1998.","title":"Pddl-the planning domain definition language","authors":"D Mcdermott, M Ghallab, A Howe, C Knoblock, A Ram, M Veloso, D Weld, D Wilkins","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mnih et al., 2015] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human- level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Humanlevel control through deep reinforcement learning","authors":"V Mnih, K Kavukcuoglu, D Silver, A A Rusu, J Veness, M G Bellemare, A Graves, M Riedmiller, A K Fidjeland, G Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sutton and Barto, 1998] R. S. Sutton and A. G. Barto. Rein- forcement learning: An introduction. Cambridge Univer- sity Press, 1998.","title":"Reinforcement learning: An introduction","authors":"R S Sutton, A G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Parr and Russell, 1998] R. Parr and S. J. Russell. Reinforce- ment learning with hierarchies of machines. In Advances in neural information processing systems, pages 1043- 1049, 1998.","title":"Reinforcement learning with hierarchies of machines","authors":"R Parr, S J Russell","year":"1998","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ryan, 2002] M. R. Ryan. Using abstract models of be- haviours to automatically generate reinforcement learning hierarchies. In ICML, 2002.","title":"Using abstract models of behaviours to automatically generate reinforcement learning hierarchies","authors":"M R Ryan","year":"2002","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Schwartz, 1993] A. Schwartz. A reinforcement learning method for maximizing undiscounted rewards. In Proc. 10th International Conf. on Machine Learning.","title":"A reinforcement learning method for maximizing undiscounted rewards","authors":"A Schwartz","year":null,"venue":"Proc. 10th International Conf. on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"Morgan Kaufmann, San Francisco, CA, 1993.","title":null,"authors":"Morgan Kaufmann","year":"1993","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sutton et al., 1999] R. S. Sutton, D. Precup, and S. Singh. Between MDPs and semi-MDPs: A framework for tem- poral abstraction in reinforcement learning. Artificial in- telligence, 112(1-2):181-211, 1999.","title":"Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning","authors":"R S Sutton, D Precup, S Singh","year":"1999","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2017] S. Zhang, P. Khandelwal, and P. Stone. Dynamically constructed (po) mdps for adaptive robot planning. In AAAI, pages 3855-3863, 2017.","title":"Dynamically constructed (po) mdps for adaptive robot planning","authors":"S Zhang, P Khandelwal, P Stone","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}