An Adaptive Architecture for Modular Q-Learning

Abstract

Reinforcement learning is a technique to learn suitable action policies that maximize utility, via the clue of reinforcement signals: reward or punishment. Q-l ea rning , a widely used reinforceme nt learning method, ha s been analyzed in much research on a utonomous agents. However , as the size of the problem space increases, agents need more comp utational resources and require more time to learn appropriate policies. Whitehead proposed an architecture called modular Q-learning, that decomposes the whole problem space into smaller subproblem spaces, and distribut es them among multiple modules. Thus, each module takes charge of part of the whole problem. In modular Q-learnin g, however, human designers have to deco mpo se the problem space, and create a suitable set of modules manually. Agents with such a fixed module architecture ca nnot adapt themselves to dynamic e nvironments. Here, we propose a new architecture for reinforceme nt lea rning ca ll ed AMQL (Automatic Modular Q-Learning), that enables agents to obtain a suitable set of modules by themselves using a selection m et hod. Through experiments, we show that agents can automatica ll y obtain s uit able modules to gain a reward. Furthermor e, we show that agents can adapt themselves to dynamic environments e ffi ciently, through recon str u ct ing modules.