---
title: "Using Cases as Heuristics in Reinforcement Learning: A Transfer Learning Application"
authors:
  - "Luiz A. Celiberto"
  - "Jackson Paul Matsuura"
  - "Ramón López de Mántaras"
  - "Reinaldo A. C. Bianchi"
doi: "10.5591/978-1-57735-516-8/IJCAI11-206"
url: "https://ijcai.org/proceedings/2011/206"
pdf: "https://ijcai.org/proceedings/2011/0206.pdf"
venue: "Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence"
year: 2011
pages: "1211-1216"
track: ""
keywords:
---

# Using Cases as Heuristics in Reinforcement Learning: A Transfer Learning Application

Luiz A. Celiberto, Jackson Paul Matsuura, Ramón López de Mántaras, Reinaldo A. C. Bianchi

*Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence*, pages 1211-1216. DOI: [10.5591/978-1-57735-516-8/IJCAI11-206](https://doi.org/10.5591/978-1-57735-516-8/IJCAI11-206)

## Abstract

In this paper we propose to combine three AI techniques to speed up a Reinforcement Learning algorithm in a Transfer Learning problem: Case-based Reasoning, Heuristically Accelerated Reinforcement Learning and Neural Networks. To do so, we propose a new algorithm, called L3, which works in 3 stages: in the first stage, it uses Reinforcement Learning to learn how to perform one task, and stores the optimal policy for this problem as a case-base; in the second stage, it uses a Neural Network to map actions from one domain to actions in the other domain and; in the third stage, it uses the case-base learned in the first stage as heuristics to speed up the learning performance in a related, but different, task. The RL algorithm used in the first phase is the Q-learning and in the third phase is the recently proposed Case-based Heuristically Accelerated Q-learning. A set of empirical evaluations were conducted in transferring the learning between two domains, the Acrobot and the Robocup 3D: the policy learned during the solution of the Acrobot Problem is transferred and used to speed up the learning of stability policies for a humanoid robot in the Robocup 3D simulator. The results show that the use of this algorithm can lead to a significant improvement in the performance of the agent.