{"type":"ScholarlyArticle","title":"Leveraging Class Abstraction for Commonsense Reinforcement Learning via Residual Policy Gradient Methods","authors":[{"name":"Niklas Hopner","slug":"niklas-hopner","given":"Niklas","family":"Hopner","affiliation":"University of Amsterdam"},{"name":"Ilaria Tiddi","slug":"ilaria-tiddi","given":"Ilaria","family":"Tiddi","affiliation":"Vrije Universiteit Amsterdam"},{"name":"Herke van Hoof","slug":"herke-van-hoof","given":"Herke","family":"van Hoof","affiliation":"University of Amsterdam"}],"abstract":"Enabling reinforcement learning (RL) agents to leverage a knowledge base while learning from experience promises to advance RL in knowledge intensive domains. However, it has proven difficult to leverage knowledge that is not manually tailored to the environment. We propose to use the subclass relationships present in open-source knowledge graphs to abstract away from specific objects. We develop a residual policy gradient method that is able to integrate knowledge across different abstraction levels in the class hierarchy. Our method results in improved sample efficiency and generalisation to unseen objects in commonsense games, but we also investigate failure modes, such as excessive noise in the extracted class knowledge or environments with little class structure.","doi":"10.24963/ijcai.2022/423","url":"https://ijcai.org/proceedings/2022/423","pdf":"https://ijcai.org/proceedings/2022/0423.pdf","markdown":"https://ijcai.org/proceedings/2022/423.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/423","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":423,"pages":{"first":3050,"last":3056},"keywords":["Knowledge Representation and Reasoning: Common-Sense Reasoning","Machine Learning: Reinforcement Learning","Machine Learning: Deep Reinforcement Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987010","length_sec":92,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985153","length_sec":756,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Akat et al., 2020] Z. Akat, D. Balliet, M. de Rijke, and F. Dignum et al. A research agenda for hybrid intelligence: Augmenting human intellect with collaborative, adaptive, responsible, and ex- plainable artificial intelligence. Computer, 53(8):18-28, 2020.","title":"A research agenda for hybrid intelligence: Augmenting human intellect with collaborative, adaptive, responsible, and explainable artificial intelligence","authors":"Z Akat, D Balliet, M De Rijke, F Dignum","year":"2020","venue":"Computer","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Devlin et al., 2019] J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"J Devlin, M Chang, K Lee, K Toutanova","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Espeholt et al., 2018] L. Espeholt, H. Soyer, R. Munos, K. Si- monyan, and V. Mnih et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In ICML, 2018.","title":"Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures","authors":"L Espeholt, H Soyer, R Munos, K Simonyan, V Mnih","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Givan et al., 2003] R. Givan, T. Dean, and M. Greig. Equivalence notions and model minimization in markov decision processes. Artificial Intelligence, 2003.","title":"Equivalence notions and model minimization in markov decision processes","authors":"R Givan, T Dean, M Greig","year":"2003","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gou et al., 2021] Y. Gou, Y. Lei, L. Liu, Y. Dai, and C. Shen. Con- textualize knowledge bases with transformer for end-to-end task- oriented dialogue systems. In EMNLP, 2021.","title":"Contextualize knowledge bases with transformer for end-to-end taskoriented dialogue systems","authors":"Y Gou, Y Lei, L Liu, Y Dai, C Shen","year":"2021","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] J. He, J. Chen, X. He, J. Gao, and L. Li et al. Deep reinforcement learning with a natural language action space. In ACL, 2016.","title":"Deep reinforcement learning with a natural language action space","authors":"J He, J Chen, X He, J Gao, L Li","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huang et al., 2017] Z. Huang, J. Yang, F. v. Harmelen, and Q. Hu. Constructing knowledge graphs of depression. In International Conference on Health Information Science. Springer, 2017.","title":"Constructing knowledge graphs of depression","authors":"Z Huang, J Yang, F V Harmelen, Q Hu","year":"2017","venue":"International Conference on Health Information Science","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ilievski et al., 2021] F. Ilievski, P. A. Szekely, and B. Zhang. CSKG: the commonsense knowledge graph. In ESWC, 2021.","title":"CSKG: the commonsense knowledge graph","authors":"F Ilievski, P A Szekely, B Zhang","year":"2021","venue":"ESWC","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiang et al., 2020] M. Jiang, J. Luketina, N. Nardelli, P. Minervini, and P. HS Torr et al. Wordcraft: An environment for bench- marking commonsense agents. arXiv preprint arXiv:2007.09185, 2020.","title":"Wordcraft: An environment for benchmarking commonsense agents","authors":"M Jiang, J Luketina, N Nardelli, P Minervini, P Hs Torr","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kemertas and Aumentado-Armstrong, 2021] M. Kemertas and T. Aumentado-Armstrong. Towards robust bisimulation metric learning. CoRR, 2021.","title":"Towards robust bisimulation metric learning","authors":"M Kemertas, T Aumentado-Armstrong","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kirk et al., 2021] R. Kirk, A. Zhang, E. Grefenstette, and T. Rocktäschel. A survey of generalisation in deep reinforcement learning. CoRR, abs/2111.09794, 2021.","title":"A survey of generalisation in deep reinforcement learning","authors":"R Kirk, A Zhang, E Grefenstette, T Rocktäschel","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lehmann et al., 2015] J. Lehmann, R. Isele, M. Jakob, A. Jentzsch, and D. Kontokostas et al. Dbpedia-a large- scale, multilingual knowledge base extracted from wikipedia. Semantic web, 2015.","title":"Dbpedia-a largescale, multilingual knowledge base extracted from wikipedia","authors":"J Lehmann, R Isele, M Jakob, A Jentzsch, D Kontokostas","year":"2015","venue":"Semantic web","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2006] L. Li, T. J Walsh, and M. L Littman. Towards a unified theory of state abstraction for MDPs. ISAIM, 4:5, 2006.","title":"Towards a unified theory of state abstraction for MDPs","authors":"L Li, T J Walsh, M L Littman","year":"2006","venue":"ISAIM","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lillicrap et al., 2016] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, and et al. Continuous control with deep re- inforcement learning. In ICLR, 2016.","title":"Continuous control with deep reinforcement learning","authors":"T P Lillicrap, J J Hunt, A Pritzel, N Heess, T Erez","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Miller, 1995] G. A Miller. Wordnet: a lexical database for english. Communications of the ACM, 1995.","title":"Wordnet: a lexical database for english","authors":"G A Miller","year":"1995","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mnih et al., 2016] V. Mnih, A. P. Badia, M. Mirza, A. Graves, and T. P. Lillicrap et al. Asynchronous methods for deep reinforce- ment learning. In ICML, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"V Mnih, A P Badia, M Mirza, A Graves, T P Lillicrap","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Moon et al., 2019] S. Moon, P. Shah, A. Kumar, and R. Subba. Opendialkg: Explainable conversational reasoning with attention-based walks over knowledge graphs. In ACL, 2019.","title":"Opendialkg: Explainable conversational reasoning with attention-based walks over knowledge graphs","authors":"S Moon, P Shah, A Kumar, R Subba","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Murugesan et al., 2021] K. Murugesan, M. Atzeni, P. Kapanipathi, P. Shukla, and S. Kumaravel et al. Text-based RL agents with commonsense knowledge: New challenges, environments and baselines. In AAAI, 2021.","title":"Text-based RL agents with commonsense knowledge: New challenges, environments and baselines","authors":"K Murugesan, M Atzeni, P Kapanipathi, P Shukla, S Kumaravel","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nickel and Kiela, 2017] M. Nickel and D. Kiela. Poincaré embed- dings for learning hierarchical representations. NeurIPS, 2017.","title":"Poincaré embeddings for learning hierarchical representations","authors":"M Nickel, D Kiela","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pennington et al., 2014] J. Pennington, R. Socher, and C. D Man- ning. Glove: Global vectors for word representation. In EMNLP, 2014.","title":"Glove: Global vectors for word representation","authors":"J Pennington, R Socher, C D Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sap et al., 2019] M. Sap, R. Le Bras, E. Allaway, C. Bhagavatula, and N. Lourie et al. ATOMIC: an atlas of machine commonsense for if-then reasoning. In AAAI, 2019.","title":"ATOMIC: an atlas of machine commonsense for if-then reasoning","authors":"M Sap, R Le Bras, E Allaway, C Bhagavatula, N Lourie","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schulman et al., 2015] J. Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz. Trust region policy optimization. In ICML, 2015.","title":"Trust region policy optimization","authors":"J Schulman, S Levine, P Abbeel, M I Jordan, P Moritz","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Silver et al., 2016] D. Silver, A. Huang, C. J. Maddison, A. Guez, and L. Sifre et al. Mastering the game of go with deep neural networks and tree search. Nat., 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"D Silver, A Huang, C J Maddison, A Guez, L Sifre","year":"2016","venue":"Nat","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Speer and Lowry-Duda, 2017] R. Speer and J. Lowry-Duda. Con- ceptnet at semeval-2017 task 2: Extending word embeddings with multilingual relational knowledge. In SemEval@ACL, 2017.","title":"Conceptnet at semeval-2017 task 2: Extending word embeddings with multilingual relational knowledge","authors":"R Speer, J Lowry-Duda","year":"2017","venue":"SemEval@ACL","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Speer et al., 2017] R. Speer, J. Chin, and C. Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In AAAI, 2017.","title":"Conceptnet 5.5: An open multilingual graph of general knowledge","authors":"R Speer, J Chin, C Havasi","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vaezipoor et al., 2021] P. Vaezipoor, A. C. Li, R. T. Icarte, and S. A. McIlraith. Ltl2action: Generalizing LTL instructions for multi-task RL. In ICML, 2021.","title":"Ltl2action: Generalizing LTL instructions for multi-task RL","authors":"P Vaezipoor, A C Li, R T Icarte, S A Mcilraith","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[van Harmelen and ten Teije, 2019] F. van Harmelen and A. ten Teije. A boxology of design patterns for hybrid learning and rea- soning systems. In BNAIC, 2019.","title":"A boxology of design patterns for hybrid learning and reasoning systems","authors":"F Van Harmelen, A Teije","year":"2019","venue":"BNAIC","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Vaswani et al., 2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, and L. Jones et al. Attention is all you need. In NeurIPS, 2017.","title":"Attention is all you need","authors":"A Vaswani, N Shazeer, N Parmar, J Uszkoreit, L Jones","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Williams, 1992] R. J Williams. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Machine learning, 1992.","title":"Simple statistical gradientfollowing algorithms for connectionist reinforcement learning","authors":"R J Williams","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xie and Pu, 2021] Y. Xie and P. Pu. How commonsense knowl- edge helps with natural language tasks: A survey of recent re- sources and methodologies. CoRR, 2021.","title":"How commonsense knowledge helps with natural language tasks: A survey of recent resources and methodologies","authors":"Y Xie, P Pu","year":"2021","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yee, 2019] Eiling Yee. Abstraction and concepts: when, how, where, what and why? Language, Cognition and Neuroscience, 34(10):1257-1265, 2019.","title":"Abstraction and concepts: when, how, where, what and why? Language","authors":"Yee ;, Eiling Yee","year":"2019","venue":"Cognition and Neuroscience","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2020a] A. Zhang, C. Lyle, S. Sodhani, A. Filos, and M. Kwiatkowska et al. Invariant causal prediction for block- MDPs. In ICML, 2020.","title":"Invariant causal prediction for block-MDPs","authors":"Zhang","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2020b] M. Zhang, K. Ye, R. Hwa, and A. Kovashka. Story completion with explicit modeling of commonsense knowl- edge. In CVPR, 2020.","title":"Story completion with explicit modeling of commonsense knowledge","authors":"Zhang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}