{"type":"ScholarlyArticle","title":"An Adaptive Architecture for Modular Q-Learning","authors":[{"name":"Takayuki Kohri","slug":"takayuki-kohri","given":"Takayuki","family":"Kohri","affiliation":"Faculty of Science and Technology , Keio University 3-14-1 Hiyoshi , Kohoku-ku , Yokohama , 223 , JAPAN"},{"name":"Kei Matsubayashi","slug":"kei-matsubayashi","given":"Kei","family":"Matsubayashi","affiliation":"Faculty of Science and Technology , Keio University 3-14-1 Hiyoshi , Kohoku-ku , Yokohama , 223 , JAPAN"},{"name":"Mario Tokoro","slug":"mario-tokoro","given":"Mario","family":"Tokoro","affiliation":"Department of Computer Science ,"}],"abstract":"Reinforcement learning is a technique to learn suitable action policies that maximize utility, via the clue of reinforcement signals: reward or punishment. Q-l ea rning , a widely used reinforceme nt learning method, ha s been analyzed in much research on a utonomous agents. However , as the size of the problem space increases, agents need more comp utational resources and require more time to learn appropriate policies. Whitehead proposed an architecture called modular Q-learning, that decomposes the whole problem space into smaller subproblem spaces, and distribut es them among multiple modules. Thus, each module takes charge of part of the whole problem. In modular Q-learnin g, however, human designers have to deco mpo se the problem space, and create a suitable set of modules manually. Agents with such a fixed module architecture ca nnot adapt themselves to dynamic e nvironments. Here, we propose a new architecture for reinforceme nt lea rning ca ll ed AMQL (Automatic Modular Q-Learning), that enables agents to obtain a suitable set of modules by themselves using a selection m et hod. Through experiments, we show that agents can automatica ll y obtain s uit able modules to gain a reward. Furthermor e, we show that agents can adapt themselves to dynamic environments e ffi ciently, through recon str u ct ing modules.","doi":null,"url":"https://ijcai.org/proceedings/1997/118","pdf":"https://ijcai.org/proceedings/1997/0118.pdf","markdown":"https://ijcai.org/proceedings/1997/118.md","bibtex":"https://ijcai.org/proceedings/1997/bibtex/118","volume":{"slug":"1997","title":"Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 1997","url":"https://ijcai.org/proceedings/1997/","year":1997,"location":"Nagoya, Japan","dates":"","editors":"Martha E. Pollack","isbn":"","publisher":"International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":"Learning","track":"Learning","number":118,"pages":{"first":820,"last":824},"keywords":[],"videos":[],"references":[{"position":1,"raw":"[Benda et al., 1985] M. Benda, V. Jagannathan, and R. Dodhiawalla. On Optimal Cooperation of Knowl- edge Sources. Technical Report BCS-G2010-28, Boe- ing AI Center, 1985.","title":"On Optimal Cooperation of Knowledge Sources","authors":"M Benda, V Jagannathan, R Dodhiawalla","year":"1985","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dayan and Hinton, 1993] Peter Dayan and Geoffrey E. Hinton. Feudal Reinforcement Learning. In Advances in Neural Information Processing Systems 5, pages 271-278, 1993.","title":"Feudal Reinforcement Learning","authors":"Peter Dayan, Geoffrey E Hinton","year":"1993","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ono and Fukumoto, 1996] Norihiko Ono and Kenji Fukumoto. Multi-agent Reinforcement Learning: A Modular Approach. In Proceedings Second Interna- tional Conference on Multi-Agent Systems, pages 252- 258, 1996.","title":"Multi-agent Reinforcement Learning: A Modular Approach","authors":"Fukumoto Ono, Norihiko Ono, Kenji Fukumoto","year":"1996","venue":"Proceedings Second International Conference on Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Sabes and Jordan , 1996] Philip N. Sabes and Michael I. Jordan. Reinforcement learning by probability match- ing. In Advances of Neural Information Processing Systems 8, 1996.","title":"Reinforcement learning by probability matching","authors":"Philip N Sabes, Michael I Jordan","year":"1996","venue":"Advances of Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Singh, 1992] Satinder P. Singh. Reinforcement Learn- ing with a Hierarchy of Abstract Models. In Proceed- ings of the Tenth National Conference on Artificial Intelligence, pages 202-207, 1992.","title":"Reinforcement Learning with a Hierarchy of Abstract Models","authors":"P Satinder, Singh","year":"1992","venue":"Proceedings of the Tenth National Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Thrun and Schwartz, 1995] Sebastian Thrun and An- ton Schwartz. Finding Structure in Reinforcement Learning. In Advances in Neural Information Pro- cessing Systems 7, pages 385-392, 1995.","title":"Sebastian Thrun and Anton Schwartz. Finding Structure in Reinforcement Learning","authors":"Schwartz Thrun","year":"1995","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Watkins, 1989] C.J.C.H . Watkins. Learning from De- layed Rewards. PhD thesis, University of Cambridge, 1989.","title":"Learning from Delayed Rewards","authors":"C J C H Watkins","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Whitehead et al. , 1993] Steven White- head, Jonas Karlsson , and Josh Tenenberg. Learning Multiple Goal Behavior via Task Decomposition and Dynamic Policy Merging. In ROBOT LEARNING. Kiluwer Academic Press, 1993.","title":"Learning Multiple Goal Behavior via Task Decomposition and Dynamic Policy Merging","authors":"Steven Whitehead, Jonas Karlsson, Josh Tenenberg","year":"1993","venue":"ROBOT LEARNING","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}