{"type":"ScholarlyArticle","title":"State Abstraction Discovery from Irrelevant State Variables","authors":[{"name":"Nicholas K. Jong","slug":"nicholas-k-jong","given":"Nicholas K.","family":"Jong","affiliation":"Department of Computer Sciences University of Texas at Austin Austin , Texas 78712"},{"name":"Peter Stone","slug":"peter-stone","given":"Peter","family":"Stone","affiliation":"Department of Computer Sciences University of Texas at Austin Austin , Texas 78712"}],"abstract":"Abstraction is a powerful form of domain knowledge that allows reinforcement-learning agents to cope with complex environments, but in most cases a human must supply this knowledge. In the absence of such prior knowledge or a given model, we propose an algorithm for the automatic discovery of state abstraction from policies learned in one domain for use in other domains that have similar structure. To this end, we introduce a novel condition for state abstraction in terms of the relevance of state features to optimal behavior, and we exhibit statistical methods that detect this condition robustly. Finally, we show how to apply temporal abstraction to benefit safely from even partial state abstraction in the presence of generalization error.","doi":null,"url":"https://ijcai.org/proceedings/2005/228","pdf":"https://ijcai.org/proceedings/2005/0228.pdf","markdown":"https://ijcai.org/proceedings/2005/228.md","bibtex":"https://ijcai.org/proceedings/2005/bibtex/228","volume":{"slug":"2005","title":"Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2005","url":"https://ijcai.org/proceedings/2005/","year":2005,"location":"Edinburgh, Scotland, UK","dates":"","editors":"","isbn":"","publisher":"International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":null,"track":"","number":228,"pages":{"first":752,"last":756},"keywords":[],"videos":[],"references":[{"position":1,"raw":"[Cohen, 1995] William W. Cohen. Fast effective rule induction. In Proceedings of the Twelfth International Conference on Machine Learning, pages 115-123, 1995.","title":"Fast effective rule induction","authors":"William W Cohen","year":"1995","venue":"Proceedings of the Twelfth International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dean and Givan, 1997] Thomas Dean and Robert Givan. Model minimization in Markov decision processes. In Proceedings of the Fourteenth National Conference on Artificial Intelligence, 1997.","title":"Model minimization in Markov decision processes","authors":"Thomas Dean, Robert Givan","year":"1997","venue":"Proceedings of the Fourteenth National Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dearden et al., 1999] Richard Dearden, Nir Friedman, and David Andre. Model based Bayesian exploration. In Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence, pages 150-159, 1999.","title":"Model based Bayesian exploration","authors":"Richard Dearden, Nir Friedman, David Andre","year":"1999","venue":"Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Degroot, 1986] Morris H. Degroot. Probability and Statistics. Addison-Wesley Pub Co, 2nd edition, 1986.","title":"Probability and Statistics","authors":"H Morris, Degroot","year":"1986","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dietterich, 2000] Thomas G. Dietterich. Hierarchical reinforce- ment learning with the MAXQ value function decomposition. Journal of Artificial Intelligence Research, 13:227-303, 2000.","title":"Hierarchical reinforcement learning with the MAXQ value function decomposition","authors":"G Thomas, Dietterich","year":"2000","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Mannor et al., 2004] Shie Mannor, Ishai Menache, Amit Hoze, and Uri Klein. Dynamic abstraction in reinforcement learning via clustering. In Proceedings of the Twenty-First International Conference on Machine Learning, pages 560-567, 2004.","title":"Dynamic abstraction in reinforcement learning via clustering","authors":"Shie Mannor, Ishai Menache, Amit Hoze, Uri Klein","year":"2004","venue":"Proceedings of the Twenty-First International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[McCallum, 1995] Andrew Kachites McCallum. Reinforcement Learning with Selective Perception and Hidden State. PhD thesis, University of Rochester, 1995.","title":"Reinforcement Learning with Selective Perception and Hidden State","authors":"Andrew Kachites, Mccallum","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Özgür S ¸ims ¸ek and Barto, 2004] Özgür S ¸ims ¸ek and Andrew G. Barto. Using relative novelty to identify useful temporal abstrac- tions in reinforcement learning. In Proceedings of the Twenty- First International Conference on Machine Learning, pages 751- 758, 2004.","title":"Using relative novelty to identify useful temporal abstractions in reinforcement learning","authors":"S Özgür, ; ] Barto, S Özgür, Andrew G Barto","year":"2004","venue":"Proceedings of the Twenty-First International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ravindran and Barto, 2003] Balaraman Ravindran and Andrew G. Barto. SMDP homomorphisms: An algebraic approach to ab- straction in semi-Markov decision processes. In Proceedings of the Eighteenth International Joint Conference on Artificial Intel- ligence, 2003.","title":"SMDP homomorphisms: An algebraic approach to abstraction in semi-Markov decision processes","authors":"Balaraman Ravindran, Andrew G Barto","year":"2003","venue":"Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2003/144"},{"position":10,"raw":"[Strens, 2000] Malcolm Strens. A Bayesian framework for rein- forcement learning. In Proceedings of the Seventeenth Interna- tional Conference on Machine Learning, pages 943-950, 2000.","title":"A Bayesian framework for reinforcement learning","authors":"Malcolm Strens","year":"2000","venue":"Proceedings of the Seventeenth International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cam- bridge, MA, 1998.","title":"Reinforcement Learning: An Introduction","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Sutton et al., 1999] Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for tempo- ral abstraction in reinforcement learning. Artificial Intelligence, 112(1-2):181-211, 1999.","title":"Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning","authors":"Richard S Sutton, Doina Precup, Satinder Singh","year":"1999","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Thrun and Schwartz, 1995] Sebastian Thrun and Anton Schwartz. Finding structure in reinforcement learning. In Advances in Neu- ral Information Processing Systems 7, 1995.","title":"Finding structure in reinforcement learning","authors":"Schwartz Thrun, Sebastian Thrun, Anton Schwartz","year":"1995","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Watkins, 1989] Watkins. Learning From Delayed Rewards. PhD thesis, University of Cambridge, England, 1989.","title":"Learning From Delayed Rewards","authors":"Watkins","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}