{"type":"ScholarlyArticle","title":"Reasoning Like Human: Hierarchical Reinforcement Learning for Knowledge Graph Reasoning","authors":[{"name":"Guojia Wan","slug":"guojia-wan","given":"Guojia","family":"Wan","affiliation":"School of Computer Science , Institute of Artificial Intelligence , and National Engineering Research Center for Multimedia Software , Wuhan University , China"},{"name":"Shirui Pan","slug":"shirui-pan","given":"Shirui","family":"Pan","affiliation":"Faculty of Information Technology , Monash University , Australia"},{"name":"Chen Gong","slug":"chen-gong","given":"Chen","family":"Gong","affiliation":"School of Computer Science and Engineering , Nanjing University of Science and Technology , China; Department of Computing , Hong Kong Polytechnic University , Hong Kong , China"},{"name":"Chuan Zhou","slug":"chuan-zhou","given":"Chuan","family":"Zhou","affiliation":"Academy of Mathematics and Systems Science , Chinese Academy of Sciences , China"},{"name":"Gholamreza Haffari","slug":"gholamreza-haffari","given":"Gholamreza","family":"Haffari","affiliation":"Faculty of Information Technology , Monash University , Australia"}],"abstract":"Knowledge Graphs typically suffer from incompleteness. A popular approach to knowledge graph completion is to infer missing knowledge by multihop reasoning over the information found along other paths connecting a pair of entities. However, multi-hop reasoning is still challenging because the reasoning process usually experiences multiple semantic issue that a relation or an entity has multiple meanings. In order to deal with the situation, we propose a novel Hierarchical Reinforcement Learning framework to learn chains of reasoning from a Knowledge Graph automatically. Our framework is inspired by the hierarchical structure through which human handle cognitionally ambiguous cases. The whole reasoning process is decomposed into a hierarchy of two-level Reinforcement Learning policies for encoding historical information and learning structured action space. As a consequence, it is more feasible and natural for dealing with the multiple semantic issue. Experimental results show that our proposed model achieves substantial improvements in ambiguous relation tasks.","doi":"10.24963/ijcai.2020/267","url":"https://ijcai.org/proceedings/2020/267","pdf":"https://ijcai.org/proceedings/2020/0267.pdf","markdown":"https://ijcai.org/proceedings/2020/267.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/267","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Knowledge Representation and Reasoning","track":"Main track","number":267,"pages":{"first":1926,"last":1932},"keywords":["Machine Learning Applications: Applications of Reinforcement Learning","Knowledge Representation and Reasoning: Reasoning about Knowledge and Belief","Data Mining: Mining Graphs, Semi Structured Data, Complex Data"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25377/file_25377.m3u8","length_sec":280,"provider":"hls"}],"references":[{"position":1,"raw":"[Barto and Mahadevan, 2003] Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforce- ment learning. Discrete event dynamic systems, 13(1- 2):41-77, 2003.","title":"Recent advances in hierarchical reinforcement learning","authors":"G Andrew, Sridhar Barto, Mahadevan","year":"2003","venue":"Discrete event dynamic systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bordes et al., 2013] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In NeurIPS, pages 2787-2795, 2013.","title":"Translating embeddings for modeling multi-relational data","authors":"Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, Oksana Yakhnenko","year":"2013","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carlson et al., 2010] Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R Hruschka, and Tom M Mitchell. Toward an architecture for never-ending language learning. In AAAI, 2010.","title":"Toward an architecture for never-ending language learning","authors":"Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Tom M Estevam R Hruschka, Mitchell","year":"2010","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cohen, 2016] William W Cohen. Tensorlog: A differen- tiable deductive database. arXiv:1605.06523, 2016.","title":"Tensorlog: A differentiable deductive database","authors":"William W Cohen","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Colin et al., 2016] Thomas R Colin, Tony Belpaeme, An- gelo Cangelosi, and Nikolas Hemion. Hierarchical rein- forcement learning as creative problem solving. Robotics and Autonomous Systems, 86:196-206, 2016.","title":"Hierarchical reinforcement learning as creative problem solving","authors":"Tony Thomas R Colin, Angelo Belpaeme, Nikolas Cangelosi, Hemion","year":"2016","venue":"Robotics and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Das et al., 2018] Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, Luke Vilnis, Ishan Durugkar, Akshay Krishnamurthy, Alex Smola, and Andrew McCallum. Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning. In ICLR, 2018.","title":"Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning","authors":"Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, Luke Vilnis, Ishan Durugkar, Akshay Krishnamurthy, Alex Smola, Andrew Mccallum","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dettmers et al., 2018] Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In AAAI, 2018.","title":"Convolutional 2d knowledge graph embeddings","authors":"Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, Sebastian Riedel","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fang et al., 2020] Yixiang Fang, Xin Huang, Lu Qin, Ying Zhang, Wenjie Zhang, Reynold Cheng, and Xuemin Lin. A survey of community search over big graphs. The VLDB Journal, 29(1):353-392, 2020.","title":"A survey of community search over big graphs","authors":"Yixiang Fang, Xin Huang, Lu Qin, Ying Zhang, Wenjie Zhang, Reynold Cheng, Xuemin Lin","year":"2020","venue":"The VLDB Journal","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ji et al., 2020] Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and Philip S Yu. A survey on knowl- edge graphs: Representation, acquisition and applications. arXiv preprint arXiv:2002.00388, 2020.","title":"A survey on knowledge graphs: Representation, acquisition and applications","authors":"Ji","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Joshi et al., 2013] Salil Joshi, Diptesh Kanojia, and Push- pak Bhattacharyya. More than meets the eye: Study of human cognition in sense annotation. In NAACL, pages 733-738, 2013.","title":"More than meets the eye: Study of human cognition in sense annotation","authors":"Salil Joshi, Diptesh Kanojia, Pushpak Bhattacharyya","year":"2013","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kertkeidkachorn and Ichise, 2017] Natthawut Kertkeidka- chorn and Ryutaro Ichise. T2kg: An end-to-end system for creating knowledge graph from unstructured text. In AAAI, 2017.","title":"Natthawut Kertkeidkachorn and Ryutaro Ichise. T2kg: An end-to-end system for creating knowledge graph from unstructured text","authors":"Ichise Kertkeidkachorn","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lao et al., 2011] Ni Lao, Tom Mitchell, and William W Co- hen. Random walk inference and learning in a large scale knowledge base. In EMNLP, pages 529-539. Association for Computational Linguistics, 2011.","title":"Random walk inference and learning in a large scale knowledge base","authors":"Ni Lao, Tom Mitchell, William W Cohen","year":"2011","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[McCarthy, 1960] John McCarthy. Programs with common sense. RLE and MIT computation center, 1960.","title":"Programs with common sense","authors":"John Mccarthy","year":"1960","venue":"RLE and MIT computation center","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nickel et al., 2015] Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. A review of re- lational machine learning for knowledge graphs. Proceed- ings of the IEEE, 104(1):11-33, 2015.","title":"A review of relational machine learning for knowledge graphs","authors":"Nickel","year":"2015","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Purcell and Kiani, 2016] Braden A Purcell and Roozbeh Kiani. Hierarchical decision processes that operate over distinct timescales underlie choice and changes in strategy. PNAS, 113(31):E4531-E4540, 2016.","title":"Hierarchical decision processes that operate over distinct timescales underlie choice and changes in strategy","authors":"A Braden, Roozbeh Purcell, Kiani","year":"2016","venue":"PNAS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Quinlan, 1990] J. Ross Quinlan. Learning logical defini- tions from relations. Machine learning, 5(3):239-266, 1990.","title":"Learning logical definitions from relations","authors":"J, Ross Quinlan","year":"1990","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sarafyazd and Jazayeri, 2019] Morteza Sarafyazd and Mehrdad Jazayeri. Hierarchical reasoning by neural cir- cuits in the frontal cortex. Science, 364(6441):eaav8911, 2019.","title":"Hierarchical reasoning by neural circuits in the frontal cortex","authors":"Jazayeri Sarafyazd, Morteza Sarafyazd, Mehrdad Jazayeri","year":"2019","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shen et al., 2018] Yelong Shen, Jianshu Chen, Po-Sen Huang, Yuqing Guo, and Jianfeng Gao. M-walk: Learn- ing to walk over graphs using monte carlo tree search. In NeurIPS, pages 6786-6797, 2018.","title":"M-walk: Learning to walk over graphs using monte carlo tree search","authors":"Shen","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sutton et al., 2000] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approx- imation. In NeurIPS, pages 1057-1063, 2000.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David A Richard S Sutton, Mcallester, P Satinder, Yishay Singh, Mansour","year":"2000","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2017] Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of approaches and applications. TKDE, 29(12):2724-2743, 2017.","title":"Knowledge graph embedding: A survey of approaches and applications","authors":"Quan Wang, Zhendong Mao, Bin Wang, Li Guo","year":"2017","venue":"TKDE","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4):229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xiao et al., 2016] Han Xiao, Minlie Huang, and Xiaoyan Zhu. Transg: A generative model for knowledge graph embedding. In ACL, volume 1, pages 2316-2325, 2016.","title":"Transg: A generative model for knowledge graph embedding","authors":"Han Xiao, Minlie Huang, Xiaoyan Zhu","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xiong et al., 2017] Wenhan Xiong, Thien Hoang, and William Yang Wang. Deeppath: A reinforcement learn- ing method for knowledge graph reasoning. In EMNLP, pages 564-573, 2017.","title":"Deeppath: A reinforcement learning method for knowledge graph reasoning","authors":"Xiong","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2017] Fan Yang, Zhilin Yang, and William W Cohen. Differentiable learning of logical rules for knowl- edge base reasoning. In NeurIPS, pages 2319-2328, 2017.","title":"Differentiable learning of logical rules for knowledge base reasoning","authors":"Fan Yang, Zhilin Yang, William W Cohen","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ye et al., 2018] Mang Ye, Zheng Wang, Xiangyuan Lan, and Pong C Yuen. Visible thermal person re-identification via dual-constrained top-ranking. In IJCAI, volume 1, page 2, 2018.","title":"Visible thermal person re-identification via dual-constrained top-ranking","authors":"Mang Ye, Zheng Wang, Xiangyuan Lan, Pong C Yuen","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/152"},{"position":28,"raw":"[Yih et al., 2011] Wen-tau Yih, Kristina Toutanova, John C Platt, and Christopher Meek. Learning discriminative pro- jections for text similarity measures. In CoNLL, pages 247-256, 2011.","title":"Learning discriminative projections for text similarity measures","authors":"Wen-Tau Yih, Kristina Toutanova, John C Platt, Christopher Meek","year":"2011","venue":"CoNLL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}