{"type":"ScholarlyArticle","title":"Relational Abstractions for Generalized Reinforcement Learning on Symbolic Problems","authors":[{"name":"Rushang Karia","slug":"rushang-karia","given":"Rushang","family":"Karia","affiliation":"School of Computing and Augmented Intelligence , Arizona State University , U.S.A."},{"name":"Siddharth Srivastava","slug":"siddharth-srivastava","given":"Siddharth","family":"Srivastava","affiliation":"School of Computing and Augmented Intelligence , Arizona State University , U.S.A."}],"abstract":"Reinforcement learning in problems with symbolic state spaces is challenging due to the need for reasoning over long horizons. This paper presents a new approach that utilizes relational abstractions in conjunction with deep learning to learn a generalizable Q-function for such problems. The learned Q-function can be efficiently transferred to related problems that have different object names and object quantities, and thus, entirely different state spaces. We show that the learned, generalized Q-function can be utilized for zero-shot transfer to related problems without an explicit, hand-coded curriculum. Empirical evaluations on a range of problems show that our method facilitates efficient zero-shot transfer of learned knowledge to much larger problem instances containing many objects.","doi":"10.24963/ijcai.2022/435","url":"https://ijcai.org/proceedings/2022/435","pdf":"https://ijcai.org/proceedings/2022/0435.pdf","markdown":"https://ijcai.org/proceedings/2022/435.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/435","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":435,"pages":{"first":3135,"last":3142},"keywords":["Machine Learning: Reinforcement Learning","Planning and Scheduling: Learning in Planning and Scheduling","Machine Learning: Deep Reinforcement Learning","Uncertainty in AI: Sequential Decision Making"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986669","length_sec":82,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984812","length_sec":338,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Andrychowicz et al., 2017] M. Andrychowicz, D. Crow, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experi- ence replay. In NeurIPS, 2017.","title":"Hindsight experience replay","authors":"M Andrychowicz, D Crow, A Ray, J Schneider, R Fong, P Welinder, B Mcgrew, J Tobin, P Abbeel, W Zaremba","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baader et al., 2017] F. Baader, I. Horrocks, C. Lutz, and U. Sattler. An Introduction to Description Logic. Cam- bridge University Press, 2017.","title":"An Introduction to Description Logic","authors":"F Baader, I Horrocks, C Lutz, U Sattler","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio et al., 2009] Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In ICML, 2009.","title":"Curriculum learning","authors":"Y Bengio, J Louradour, R Collobert, J Weston","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bonet et al., 2019] B. Bonet, G. Francès, and H. Geffner. Learning features and abstract actions for computing gen- eralized plans. In AAAI, 2019.","title":"Learning features and abstract actions for computing generalized plans","authors":"B Bonet, G Francès, H Geffner","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cumby and Roth, 2002] C. Cumby and D. Roth. Learning with feature description logics. In ILP, 2002.","title":"Learning with feature description logics","authors":"C Cumby, D Roth","year":"2002","venue":"ILP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Das et al., 2020] S. Das, S. Natarajan, K. Roy, R. Parr, and K. Kersting. Fitted q-learning for relational domains. arXiv, abs/2006.05595, 2020.","title":"Fitted q-learning for relational domains","authors":"S Das, S Natarajan, K Roy, R Parr, K Kersting","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dzeroski et al., 2001] S. Dzeroski, L. D. Raedt, and K. Driessens. Relational reinforcement learning. Mach. Learn., 43(1/2):7-52, 2001.","title":"Relational reinforcement learning","authors":"S Dzeroski, L D Raedt, K Driessens","year":"2001","venue":"Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ferber et al., 2020] P. Ferber, M. Helmert, and J. Hoffmann. Reinforcement learning for planning heuristics. In ICAPS PRL workshop, 2020.","title":"Reinforcement learning for planning heuristics","authors":"P Ferber, M Helmert, J Hoffmann","year":"2020","venue":"ICAPS PRL workshop","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fern et al., 2006] A. Fern, S. W. Yoon, and R. Givan. Ap- proximate policy iteration with a policy language bias: Solving relational markov decision processes. J. Artif. In- tell. Res., 25:75-118, 2006.","title":"Approximate policy iteration with a policy language bias: Solving relational markov decision processes","authors":"A Fern, S W Yoon, R Givan","year":"2006","venue":"J. Artif. Intell. Res","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Francès et al., 2021] G. Francès, B. Bonet, and H. Geffner. Learning general planning policies from small examples without supervision. In AAAI, 2021.","title":"Learning general planning policies from small examples without supervision","authors":"G Francès, B Bonet, H Geffner","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Garg et al., 2019] S. Garg, A. Bajpai, and Mausam. Size in- dependent neural transfer for RDDL planning. In ICAPS, 2019.","title":"Size independent neural transfer for RDDL planning","authors":"S Garg, A Bajpai, Mausam","year":"2019","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Garg et al., 2020] S. Garg, A. Bajpai, and Mausam. Sym- bolic network: Generalized neural policies for relational mdps. In ICML, 2020.","title":"Symbolic network: Generalized neural policies for relational mdps","authors":"S Garg, A Bajpai, Mausam","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Groshev et al., 2018] E. Groshev, M. Goldstein, A. Tamar, S. Srivastava, and P. Abbeel. Learning generalized reactive policies using deep neural networks. In ICAPS, 2018.","title":"Learning generalized reactive policies using deep neural networks","authors":"E Groshev, M Goldstein, A Tamar, S Srivastava, P Abbeel","year":"2018","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Guestrin et al., 2001] C. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored mdps. In IJCAI, 2001.","title":"Max-norm projections for factored mdps","authors":"C Guestrin, D Koller, R Parr","year":"2001","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2001/93"},{"position":15,"raw":"[Guestrin et al., 2003] C. Guestrin, D. Koller, C. Gearhart, and N. Kanodia. Generalizing plans to new environments in relational mdps. In IJCAI, 2003.","title":"Generalizing plans to new environments in relational mdps","authors":"C Guestrin, D Koller, C Gearhart, N Kanodia","year":"2003","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2003/143"},{"position":16,"raw":"[Izhikevich et al., 2015] E. M. Izhikevich, J. H. Conway, and A. Seth. Game of life. Scholarpedia, 10(6):1816, 2015.","title":"Game of life","authors":"E M Izhikevich, J H Conway, A Seth","year":"2015","venue":"Scholarpedia","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Janisch et al., 2020] J. Janisch, T. Pevný, and V. Lisý. Sym- bolic relational deep reinforcement learning based on graph neural networks. arXiv, abs/2009.12462, 2020.","title":"Symbolic relational deep reinforcement learning based on graph neural networks","authors":"J Janisch, T Pevný, V Lisý","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Karia and Srivastava, 2021] R. Karia and S. Srivastava. Learning generalized relational heuristic networks for model-agnostic planning. In AAAI, 2021.","title":"Learning generalized relational heuristic networks for model-agnostic planning","authors":"R Karia, S Srivastava","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Karia and Srivastava, 2022] R. Karia and S. Srivastava. Re- lational abstractions for generalized reinforcement learn- ing on symbolic problems. arXiv, abs/2204.12665, 2022.","title":"Relational abstractions for generalized reinforcement learning on symbolic problems","authors":"R Karia, S Srivastava","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Keller and Eyerich, 2012] T. Keller and P. Eyerich. PROST: probabilistic planning based on UCT. In ICAPS, 2012.","title":"PROST: probabilistic planning based on UCT","authors":"T Keller, P Eyerich","year":"2012","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Khardon, 1999] R. Khardon. Learning action strategies for planning domains. Artif. Intell., 113(1-2):125-148, 1999.","title":"Learning action strategies for planning domains","authors":"R Khardon","year":"1999","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Kingma and Ba, 2015] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"D P Kingma, J Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Kokel et al., 2021] H. Kokel, A. Manoharan, S. Natarajan, B. Ravindran, and P. Tadepalli. Reprel: Integrating re- lational planning and reinforcement learning for effective abstraction. In ICAPS, 2021.","title":"Reprel: Integrating relational planning and reinforcement learning for effective abstraction","authors":"H Kokel, A Manoharan, S Natarajan, B Ravindran, P Tadepalli","year":"2021","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Long and Fox, 2003] D. Long and M. Fox. The 3rd interna- tional planning competition: Results and analysis. J. Artif. Intell. Res., 20:1-59, 2003.","title":"The 3rd international planning competition: Results and analysis","authors":"D Long, M Fox","year":"2003","venue":"J. Artif. Intell. Res","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Martín and Geffner, 2004] M. Martín and H. Geffner. Learning generalized policies from planning examples us- ing concept languages. Appl. Intell., 20(1):9-19, 2004.","title":"Learning generalized policies from planning examples using concept languages","authors":"M Martín, H Geffner","year":"2004","venue":"Appl. Intell","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Mnih et al., 2013] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. A. Ried- miller. Playing atari with deep reinforcement learning. arXiv, abs/1312.5602, 2013.","title":"Playing atari with deep reinforcement learning","authors":"V Mnih, K Kavukcuoglu, D Silver, A Graves, I Antonoglou, D Wierstra, M A Riedmiller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Narvekar and Stone, 2019] S. Narvekar and P. Stone. Learning curriculum policies for reinforcement learning. In AAMAS, 2019.","title":"Learning curriculum policies for reinforcement learning","authors":"S Narvekar, P Stone","year":"2019","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ng and Petrick, 2021a] J. H. A. Ng and R. Petrick. First- order function approximation for transfer learning in rela- tional mdps. In ICAPS PRL workshop, 2021.","title":"Firstorder function approximation for transfer learning in relational mdps","authors":"Petrick Ng, J H A Ng, R Petrick","year":"2021","venue":"ICAPS PRL workshop","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Ng and Petrick, 2021b] J. H. A. Ng and R. Petrick. Gener- alised task planning with first-order function approxima- tion. In CoRL, 2021.","title":"Generalised task planning with first-order function approximation","authors":"Petrick Ng, J H A Ng, R Petrick","year":"2021","venue":"CoRL","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Paszke et al., 2019] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. De- Vito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.","title":"Pytorch: An imperative style, high-performance deep learning library","authors":"A Paszke, S Gross, F Massa, A Lerer, J Bradbury, G Chanan, T Killeen, Z Lin, N Gimelshein, L Antiga, A Desmaison, A Kopf, E Yang, Z De-Vito, M Raison, A Tejani, S Chilamkurthy, B Steiner, L Fang, J Bai, S Chintala","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Rosenfeld et al., 2017] A. Rosenfeld, M. E. Taylor, and S. Kraus. Speeding up tabular reinforcement learning us- ing state-action similarities. In AAMAS, 2017.","title":"Speeding up tabular reinforcement learning using state-action similarities","authors":"A Rosenfeld, M E Taylor, S Kraus","year":"2017","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Sanner and Boutilier, 2005] S. Sanner and C. Boutilier. Ap- proximate linear programming for first-order mdps. In UAI, 2005.","title":"Approximate linear programming for first-order mdps","authors":"S Sanner, C Boutilier","year":"2005","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Sanner, 2010] S. Sanner. Relational dynamic influ- ence diagram language (RDDL): Language descrip- tion. http://users.cecs.anu.edu.au/ ∼ ssanner/IPPC 2011/ RDDL.pdf, 2010. Accessed: 2022-01-01.","title":"Relational dynamic influence diagram language (RDDL): Language description","authors":"S Sanner","year":"2010","venue":null,"doi":null,"url":"http://users.cecs.anu.edu.au/∼ssanner/IPPC2011/RDDL.pdf","internal_url":null},{"position":34,"raw":"[Sanner, 2011] S. Sanner. The 2011 international proba- bilistic planning competition. http://users.cecs.anu.edu.au/ ∼ ssanner/IPPC 2011/index.html, 2011. Accessed: 2022- 01-01.","title":"The 2011 international probabilistic planning competition","authors":"S Sanner","year":"2011","venue":null,"doi":null,"url":"http://users.cecs.anu.edu.au/∼ssanner/IPPC2011/index.html","internal_url":null},{"position":35,"raw":"[Sanner, 2014] S. Sanner. The 2014 international proba- bilistic planning competition. http://users.cecs.anu.edu.au/ ∼ ssanner/IPPC 2014/index.html, 2014. Accessed: 2022- 01-01.","title":"The 2014 international probabilistic planning competition","authors":"S Sanner","year":"2014","venue":null,"doi":null,"url":"http://users.cecs.anu.edu.au/∼ssanner/IPPC2014/index.html","internal_url":null},{"position":36,"raw":"[Srivastava et al., 2008] S. Srivastava, N. Immerman, and S. Zilberstein. Learning generalized plans using abstract counting. In AAAI, 2008.","title":"Learning generalized plans using abstract counting","authors":"S Srivastava, N Immerman, S Zilberstein","year":"2008","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Srivastava et al., 2010] S. Srivastava, N. Immerman, and S. Zilberstein. Computing applicability conditions for plans with loops. In ICAPS, 2010.","title":"Computing applicability conditions for plans with loops","authors":"S Srivastava, N Immerman, S Zilberstein","year":"2010","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Srivastava et al., 2011] S. Srivastava, S. Zilberstein, N. Im- merman, and H. Geffner. Qualitative numeric planning. In AAAI, 2011.","title":"Qualitative numeric planning","authors":"S Srivastava, S Zilberstein, N Immerman, H Geffner","year":"2011","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Srivastava et al., 2015] S. Srivastava, S. Zilberstein, A. Gupta, P. Abbeel, and S. Russell. Tractability of planning with loops. In AAAI, 2015.","title":"Tractability of planning with loops","authors":"S Srivastava, S Zilberstein, A Gupta, P Abbeel, S Russell","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Sutton and Barto, 1998] R. S. Sutton and A. G. Barto. Re- inforcement learning -an introduction. MIT Press, 1998.","title":"Reinforcement learning -an introduction","authors":"R S Sutton, A G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Tadepalli et al., 2004] P. Tadepalli, R. Givan, and K. Driessens. Relational reinforcement learning: An overiew. In ICML RRL workshop, 2004.","title":"Relational reinforcement learning: An overiew","authors":"P Tadepalli, R Givan, K Driessens","year":"2004","venue":"ICML RRL workshop","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Watkins, 1989] C. Watkins. Learning from Delayed Re- wards. PhD thesis, King's College, Cambridge, UK, 1989.","title":"Learning from Delayed Rewards","authors":"C Watkins","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Wu and Givan, 2007] J. Wu and R. Givan. Discovering re- lational domain features for probabilistic planning. In ICAPS, 2007.","title":"Discovering relational domain features for probabilistic planning","authors":"Givan Wu, ] J Wu, R Givan","year":"2007","venue":"ICAPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}