{"type":"ScholarlyArticle","title":"Using Cases as Heuristics in Reinforcement Learning: A Transfer Learning Application","authors":[{"name":"Luiz A. Celiberto","slug":"luiz-a-celiberto","given":"Luiz A.","family":"Celiberto","affiliation":""},{"name":"Jackson Paul Matsuura","slug":"jackson-paul-matsuura","given":"Jackson Paul","family":"Matsuura","affiliation":""},{"name":"Ramón López de Mántaras","slug":"ramon-lopez-de-mantaras","given":"Ramón López","family":"de Mántaras","affiliation":""},{"name":"Reinaldo A. C. Bianchi","slug":"reinaldo-a-c-bianchi","given":"Reinaldo A. C.","family":"Bianchi","affiliation":""}],"abstract":"In this paper we propose to combine three AI techniques to speed up a Reinforcement Learning algorithm in a Transfer Learning problem: Case-based Reasoning, Heuristically Accelerated Reinforcement Learning and Neural Networks. To do so, we propose a new algorithm, called L3, which works in 3 stages: in the first stage, it uses Reinforcement Learning to learn how to perform one task, and stores the optimal policy for this problem as a case-base; in the second stage, it uses a Neural Network to map actions from one domain to actions in the other domain and; in the third stage, it uses the case-base learned in the first stage as heuristics to speed up the learning performance in a related, but different, task. The RL algorithm used in the first phase is the Q-learning and in the third phase is the recently proposed Case-based Heuristically Accelerated Q-learning. A set of empirical evaluations were conducted in transferring the learning between two domains, the Acrobot and the Robocup 3D: the policy learned during the solution of the Acrobot Problem is transferred and used to speed up the learning of stability policies for a humanoid robot in the Robocup 3D simulator. The results show that the use of this algorithm can lead to a significant improvement in the performance of the agent.","doi":"10.5591/978-1-57735-516-8/IJCAI11-206","url":"https://ijcai.org/proceedings/2011/206","pdf":"https://ijcai.org/proceedings/2011/0206.pdf","markdown":"https://ijcai.org/proceedings/2011/206.md","bibtex":"https://ijcai.org/proceedings/2011/bibtex/206","volume":{"slug":"2011","title":"Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2011","url":"https://ijcai.org/proceedings/2011/","year":2011,"location":"Barcelona, Catalonia, Spain","dates":"16–22 July 2011","editors":"Toby Walsh","isbn":"978-1-57735-512-0; 978-1-57735-513-7; 978-1-57735-514-4; 978-1-57735-515-1; 978-1-57735-516-8","publisher":"AAAI Press/International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":null,"track":"","number":206,"pages":{"first":1211,"last":1216},"keywords":[],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}