{"type":"ScholarlyArticle","title":"On Overfitting and Asymptotic Bias in Batch Reinforcement Learning with Partial Observability (Extended Abstract)","authors":[{"name":"Vincent Francois-Lavet","slug":"vincent-francois-lavet","given":"Vincent","family":"François-Lavet","affiliation":"McGill University"},{"name":"Guillaume Rabusseau","slug":"guillaume-rabusseau","given":"Guillaume","family":"Rabusseau","affiliation":"Université de Montréal"},{"name":"Joelle Pineau","slug":"joelle-pineau","given":"Joelle","family":"Pineau","affiliation":"McGill University"},{"name":"Damien Ernst","slug":"damien-ernst","given":"Damien","family":"Ernst","affiliation":"Université de Liège"},{"name":"Raphael Fonteneau","slug":"raphael-fonteneau","given":"Raphael","family":"Fonteneau","affiliation":"Université de Liège"}],"abstract":"When an agent has limited information on its environment, the suboptimality of an RL algorithm can be decomposed into the sum of two terms: a term related to an asymptotic bias (suboptimality with unlimited data) and a term due to overfitting (additional suboptimality due to limited data). In the context of reinforcement learning with partial observability, this paper provides an analysis of the tradeoff between these two error sources. In particular, our theoretical analysis formally characterizes how a smaller state representation increases the asymptotic bias while decreasing the risk of overfitting.","doi":"10.24963/ijcai.2020/706","url":"https://ijcai.org/proceedings/2020/706","pdf":"https://ijcai.org/proceedings/2020/0706.pdf","markdown":"https://ijcai.org/proceedings/2020/706.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/706","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Journal track","track":"Journal track","number":706,"pages":{"first":5055,"last":5059},"keywords":["Knowledge Representation and Reasoning: Reasoning about Knowledge and Belief","Machine Learning: Reinforcement Learning","Planning and Scheduling: POMDPs","Machine Learning: Learning Theory"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24364/file_24364.m3u8","length_sec":299,"provider":"hls"}],"references":[{"position":1,"raw":"[Abel et al., 2016] David Abel, David Hershkowitz, and Michael Littman. Near optimal behavior via approxi- mate state abstraction. In Proceedings of The 33rd Inter- national Conference on Machine Learning, pages 2915- 2923, 2016.","title":"Near optimal behavior via approximate state abstraction","authors":"David Abel, David Hershkowitz, Michael Littman","year":"2016","venue":"Proceedings of The 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aberdeen et al., 2007] Douglas Aberdeen, Olivier Buffet, and Owen Thomas. Policy-gradients for PSRs and POMDPs. In Artificial Intelligence and Statistics, pages 3-10, 2007.","title":"Policy-gradients for PSRs and POMDPs","authors":"Douglas Aberdeen, Olivier Buffet, Owen Thomas","year":"2007","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cassandra et al., 1994] Anthony R Cassandra, Leslie Pack Kaelbling, and Michael L Littman. Acting optimally in partially observable stochastic domains. In Proceedings of the Twelfth AAAI National Conference on Artificial Intel- ligence, volume 94, pages 1023-1028, 1994.","title":"Acting optimally in partially observable stochastic domains","authors":"Leslie Anthony R Cassandra, Pack Kaelbling, Michael L Littman","year":"1994","venue":"Proceedings of the Twelfth AAAI National Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Farahmand, 2011] Amir-massoud Farahmand. Regulariza- tion in reinforcement learning. PhD thesis, University of Alberta, 2011.","title":"Regularization in reinforcement learning","authors":"Amir-Massoud Farahmand","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ferns et al., 2004] Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite Markov decision pro- cesses. In Proceedings of the 20th conference on Un- certainty in artificial intelligence, pages 162-169. AUAI Press, 2004.","title":"Metrics for finite Markov decision processes","authors":"Ferns","year":"2004","venue":"Proceedings of the 20th conference on Uncertainty in artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Franc ¸ois-Lavet et al., 2019] Vincent Franc ¸ois-Lavet, Guil- laume Rabusseau, Joelle Pineau, Damien Ernst, and Raphael Fonteneau. On overfitting and asymptotic bias in batch reinforcement learning with partial observability. Journal of Artificial Intelligence Research, 65:1-30, 2019.","title":"On overfitting and asymptotic bias in batch reinforcement learning with partial observability","authors":"Franc ¸ois-Lavet","year":"2019","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ghavamzadeh et al., 2015] Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar. Bayesian reinforcement learning: a survey. Foundations and Trends® in Machine Learning, 8(5-6):359-483, 2015.","title":"Bayesian reinforcement learning: a survey","authors":"Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, Aviv Tamar","year":"2015","venue":"Foundations and Trends® in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hutter, 2014] Marcus Hutter. Extreme state aggregation be- yond mdps. In International Conference on Algorithmic Learning Theory, pages 185-199. Springer, 2014.","title":"Extreme state aggregation beyond mdps","authors":"Marcus Hutter","year":"2014","venue":"International Conference on Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiang et al., 2015] Nan Jiang, Alex Kulesza, and Satinder Singh. Abstraction selection in model-based reinforce- ment learning. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 179- 188, 2015.","title":"Abstraction selection in model-based reinforcement learning","authors":"Nan Jiang, Alex Kulesza, Satinder Singh","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning (ICML-15)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kaelbling et al., 1998] Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intel- ligence, 101(1):99-134, 1998.","title":"Planning and acting in partially observable stochastic domains","authors":"Leslie Pack, Kaelbling Michael L Littman, Anthony R Cassandra","year":"1998","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lange et al., 2012] Sascha Lange, Thomas Gabel, and Mar- tin Riedmiller. Batch reinforcement learning. In Rein- forcement learning, pages 45-73. Springer, 2012.","title":"Batch reinforcement learning","authors":"Sascha Lange, Thomas Gabel, Martin Riedmiller","year":"2012","venue":"Reinforcement learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Littman and Sutton, 2002] Michael L Littman and Richard S Sutton. Predictive representations of state. In Advances in neural information processing systems, pages 1555-1561, 2002.","title":"Predictive representations of state","authors":"L Michael, Richard S Littman, Sutton","year":"2002","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[McCallum, 1996] Andrew Kachites McCallum. Reinforce- ment learning with selective perception and hidden state. PhD thesis, University of Rochester, 1996.","title":"Reinforcement learning with selective perception and hidden state","authors":"Andrew Kachites, Mccallum","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pineau et al., 2003] Joelle Pineau, Geoff Gordon, and Se- bastian Thrun. Point-based value iteration: An anytime al- gorithm for POMDPs. In Proceedings of the 18th interna- tional joint conference on Artificial intelligence, volume 3, pages 1025-1032, 2003.","title":"Point-based value iteration: An anytime algorithm for POMDPs","authors":"Pineau","year":"2003","venue":"Proceedings of the 18th international joint conference on Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ross et al., 2011] Stéphane Ross, Joelle Pineau, Brahim Chaib-draa, and Pierre Kreitmann. A Bayesian approach for learning and planning in partially observable Markov decision processes. The Journal of Machine Learning Re- search, 12:1729-1770, 2011.","title":"A Bayesian approach for learning and planning in partially observable Markov decision processes","authors":"Stéphane Ross, Joelle Pineau, Brahim Chaib-Draa, Pierre Kreitmann","year":"2011","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Singh et al., 2004] Satinder Singh, Michael R James, and Matthew R Rudary. Predictive state representations: A new theory for modeling dynamical systems. In Proceed- ings of the 20th conference on Uncertainty in artificial in- telligence, pages 512-519. AUAI Press, 2004.","title":"Predictive state representations: A new theory for modeling dynamical systems","authors":"Satinder Singh, Michael R James, Matthew R Rudary","year":"2004","venue":"Proceedings of the 20th conference on Uncertainty in artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sondik, 1978] Edward J Sondik. The optimal control of par- tially observable Markov processes over the infinite hori- zon: Discounted costs. Operations research, 26(2):282- 304, 1978.","title":"The optimal control of partially observable Markov processes over the infinite horizon: Discounted costs","authors":"J Edward, Sondik","year":"1978","venue":"Operations research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, vol- ume 1. MIT press Cambridge, 1998.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}