{"type":"ScholarlyArticle","title":"Organizing Experience: a Deeper Look at Replay Mechanisms for Sample-Based Planning in Continuous State Domains","authors":[{"name":"Yangchen Pan","slug":"yangchen-pan","given":"Yangchen","family":"Pan","affiliation":"University of Alberta"},{"name":"Muhammad Zaheer","slug":"muhammad-zaheer","given":"Muhammad","family":"Zaheer","affiliation":"University of Alberta"},{"name":"Adam White","slug":"adam-white","given":"Adam","family":"White","affiliation":"University of Alberta"},{"name":"Andrew Patterson","slug":"andrew-patterson","given":"Andrew","family":"Patterson","affiliation":"University of Alberta"},{"name":"Martha White","slug":"martha-white","given":"Martha","family":"White","affiliation":"University of Alberta"}],"abstract":"Model-based strategies for control are critical to obtain sample efficient learning. Dyna is a planning paradigm that naturally interleaves learning and planning, by simulating one-step experience to update the action-value function. This elegant planning strategy has been mostly explored in the tabular setting. The aim of this paper is to revisit sample-based planning, in stochastic and continuous domains with learned models. We first highlight the flexibility afforded by a model over Experience Replay (ER). Replay-based methods can be seen as stochastic planning methods that repeatedly sample from a buffer of recent agent-environment interactions and perform updates to improve data efficiency. We show that a model, as opposed to a replay buffer, is particularly useful for specifying which states to sample from during planning, such as predecessor states that propagate information in reverse from a state more quickly. We introduce a semi-parametric model learning approach, called Reweighted Experience Models (REMs), that makes it simple to sample next states or predecessors. We demonstrate that REM-Dyna exhibits similar advantages over replay-based methods in learning in continuous state problems, and that the performance gap grows when moving to stochastic domains, of increasing size.","doi":"10.24963/ijcai.2018/666","url":"https://ijcai.org/proceedings/2018/666","pdf":"https://ijcai.org/proceedings/2018/0666.pdf","markdown":"https://ijcai.org/proceedings/2018/666.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/666","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Planning and Scheduling","track":"Main track","number":666,"pages":{"first":4794,"last":4800},"keywords":["Planning and Scheduling: Markov Decisions Processes","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Alain et al., 2016] Guillaume Alain, Yoshua Bengio, Li Yao, Jason Yosinski, Éric Thibodeau-Laufer, Saizheng Zhang, and Pascal Vincent. GSNs: generative stochastic networks. Information and Inference: A Journal of the IMA, 2016.","title":"GSNs: generative stochastic networks","authors":"Guillaume Alain, Yoshua Bengio, Li Yao, Jason Yosinski, Éric Thibodeau-Laufer, Saizheng Zhang, Pascal Vincent","year":"2016","venue":"Information and Inference: A Journal of the IMA","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bagnell and Schneider, 2001] J A Bagnell and J G Schneider. Au- tonomous helicopter control using reinforcement learning policy search methods. In IEEE International Conference on Robotics and Automation, 2001.","title":"Autonomous helicopter control using reinforcement learning policy search methods","authors":"Bagnell, ; ] J A Schneider, J G Bagnell, Schneider","year":"2001","venue":"IEEE International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Barreto et al., 2011] A Barreto, D Precup, and J Pineau. Reinforce- ment Learning using Kernel-Based Stochastic Factorization. In Advances in Neural Information Processing Systems, 2011.","title":"Reinforcement Learning using Kernel-Based Stochastic Factorization","authors":"D Barreto, Precup, Pineau","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Barreto et al., 2014] A Barreto, J Pineau, and D Precup. Policy Iteration Based on Stochastic Factorization. Journal of Artificial Intelligence Research, 2014.","title":"Policy Iteration Based on Stochastic Factorization","authors":"J Barreto, Pineau, Precup","year":"2014","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Barreto et al., 2016] A Barreto, R Beirigo, J Pineau, and D Precup. Incremental Stochastic Factorization for Online Reinforcement Learning. In AAAI Conference on Artificial Intelligence, 2016.","title":"Incremental Stochastic Factorization for Online Reinforcement Learning","authors":"R Barreto, J Beirigo, Pineau, Precup","year":"2016","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Deisenroth and Rasmussen, 2011] M Deisenroth and C E Ras- mussen. PILCO: A model-based and data-efficient approach to policy search. In International Conference on Machine Learning, 2011.","title":"PILCO: A model-based and data-efficient approach to policy search","authors":"M Deisenroth, C E Rasmussen","year":"2011","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[French, 1999] R M French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4):128-135, 1999.","title":"Catastrophic forgetting in connectionist networks","authors":"R M French","year":"1999","venue":"Trends in Cognitive Sciences","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goodfellow et al., 2013] I J Goodfellow, M Mirza, D Xiao, A Courville, and Y Bengio. An empirical investigation of catas- trophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013.","title":"An empirical investigation of catastrophic forgetting in gradient-based neural networks","authors":"I J Goodfellow, M Mirza, D Xiao, Y Courville, Bengio","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goodfellow et al., 2014] I J Goodfellow, J Pouget-Abadie, MehMdi Mirza, B Xu, D Warde-Farley, S Ozair, A C Courville, and Y Bengio. Generative Adversarial Nets. In Advances in Neural Information Processing Systems, 2014.","title":"Generative Adversarial Nets","authors":"I J Goodfellow, J Pouget-Abadie, Mehmdi Mirza, B Xu, D Warde-Farley, S Ozair, A C Courville, Y Bengio","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Grunewalder et al., 2012] Steffen Grunewalder, Guy Lever, Luca Baldassarre, Massi Pontil, and Arthur Gretton. Modelling transi- tion dynamics in MDPs with RKHS embeddings. In International Conference on Machine Learning, 2012.","title":"Modelling transition dynamics in MDPs with RKHS embeddings","authors":"Steffen Grunewalder, Guy Lever, Luca Baldassarre, Massi Pontil, Arthur Gretton","year":"2012","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gu et al., 2016] Shixiang Gu, Timothy P Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous Deep Q-Learning with Model- based Acceleration. In International Conference on Machine Learning, 2016.","title":"Continuous Deep Q-Learning with Modelbased Acceleration","authors":"Gu","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hamilton et al., 2014] W L Hamilton, M M Fard, and J Pineau. Efficient learning and planning with compressed predictive states. Journal of Machine Learning Research, 2014.","title":"Efficient learning and planning with compressed predictive states","authors":"M M W L Hamilton, Fard, Pineau","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Holmes et al., 2007] Michael P Holmes, Alexander G Gray, and Charles L Isbell. Fast Nonparametric Conditional Density Estima- tion. Uncertainty in AI, 2007.","title":"Fast Nonparametric Conditional Density Estimation","authors":"Alexander G Michael P Holmes, Charles L Gray, Isbell","year":"2007","venue":"Uncertainty in AI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kveton and Theocharous, 2012] B Kveton and G Theocharous. Kernel-Based Reinforcement Learning on Representative States. AAAI Conference on Artificial Intelligence, 2012.","title":"Kernel-Based Reinforcement Learning on Representative States","authors":"B Kveton, G Theocharous","year":"2012","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lever et al., 2016] Guy Lever, John Shawe-Taylor, Ronnie Stafford, and Csaba Szepesvári. Compressed Conditional Mean Embed- dings for Model-Based Reinforcement Learning. In AAAI Confer- ence on Artificial Intelligence, 2016.","title":"Compressed Conditional Mean Embeddings for Model-Based Reinforcement Learning","authors":"Guy Lever, John Shawe-Taylor, Ronnie Stafford, Csaba Szepesvári","year":"2016","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin, 1992] Long-Ji Lin. Self-Improving Reactive Agents Based On Reinforcement Learning, Planning and Teaching. Machine Learning, 1992.","title":"Self-Improving Reactive Agents Based On Reinforcement Learning, Planning and Teaching","authors":"Long-Ji Lin","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mairal et al., 2014] Julien Mairal, Piotr Koniusz, Zaid Harchaoui, and Cordelia Schmid. Convolutional Kernel Networks. Advances in Neural Information Processing Systems, 2014.","title":"Convolutional Kernel Networks","authors":"Julien Mairal, Piotr Koniusz, Zaid Harchaoui, Cordelia Schmid","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[McCloskey and Cohen, 1989] Michael McCloskey and Neal J Co- hen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of learning and motiva- tion, 24:109-165, 1989.","title":"Catastrophic interference in connectionist networks: The sequential learning problem","authors":"Cohen Mccloskey, Michael Mccloskey, Neal J Cohen","year":"1989","venue":"Psychology of learning and motivation","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Moore and Atkeson, 1993] Andrew W Moore and Christopher G Atkeson. Prioritized sweeping: Reinforcement learning with less data and less time. Machine learning, 13(1):103-130, 1993.","title":"Prioritized sweeping: Reinforcement learning with less data and less time","authors":"Andrew W, Moore, Christopher G Atkeson","year":"1993","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ormoneit and Sen, 2002] Dirk Ormoneit and Śaunak Sen. Kernel- Based Reinforcement Learning. Machine Learning, 2002.","title":"Dirk Ormoneit and Śaunak Sen. Kernel-Based Reinforcement Learning","authors":"Sen Ormoneit","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Peng and Williams, 1993] Jing Peng and Ronald J Williams. Effi- cient Learning and Planning Within the Dyna Framework. Adap- tive behavior, 1993.","title":"Efficient Learning and Planning Within the Dyna Framework","authors":"Jing Peng, Ronald J Williams","year":"1993","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pires and Szepesvári, 2016] Bernardo Avila Pires and Csaba Szepesvári. Policy Error Bounds for Model-Based Reinforcement Learning with Factored Linear Models. In Annual Conference on Learning Theory, 2016.","title":"Policy Error Bounds for Model-Based Reinforcement Learning with Factored Linear Models","authors":"Bernardo Avila, Pires, Csaba Szepesvári","year":"2016","venue":"Annual Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schaul et al., 2016] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized Experience Replay. In International Conference on Learning Representations, 2016.","title":"Prioritized Experience Replay","authors":"Tom Schaul, John Quan, Ioannis Antonoglou, David Silver","year":"2016","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schlegel et al., 2017] Matthew Schlegel, Yangchen Pan, Jiecao Chen, and Martha White. Adapting Kernel Representations Online Using Submodular Maximization. In International Conference on Machine Learning, 2017.","title":"Adapting Kernel Representations Online Using Submodular Maximization","authors":"Matthew Schlegel, Yangchen Pan, Jiecao Chen, Martha White","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sohn et al., 2015] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning Structured Output Representation using Deep Condi- tional Generative Models. In Advances in Neural Information Processing Systems, 2015.","title":"Learning Structured Output Representation using Deep Conditional Generative Models","authors":"Sohn","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Strehl and Littman, 2008] A. Strehl and M Littman. An analysis of model-based Interval Estimation for Markov Decision Processes. Journal of Computer and System Sciences, 2008.","title":"An analysis of model-based Interval Estimation for Markov Decision Processes","authors":"Littman Strehl, A Strehl, Littman","year":"2008","venue":"Journal of Computer and System Sciences","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sutton and Barto, 1998] R.S. Sutton and A G Barto. Reinforcement Learning: An Introduction. MIT press, 1998.","title":"Reinforcement Learning: An Introduction","authors":"R S Sutton, A G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sutton et al., 2008] R Sutton, C Szepesvári, A Geramifard, and M Bowling. Dyna-style planning with linear function approxima- tion and prioritized sweeping. In Conference on Uncertainty in Artificial Intelligence, 2008.","title":"Dyna-style planning with linear function approximation and prioritized sweeping","authors":"C Sutton, Szepesvári, M Geramifard, Bowling","year":"2008","venue":"Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sutton, 1991] R.S. Sutton. Integrated modeling and control based on reinforcement learning and dynamic programming. In Ad- vances in Neural Information Processing Systems, 1991.","title":"Integrated modeling and control based on reinforcement learning and dynamic programming","authors":"R S Sutton","year":"1991","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Talvitie, 2017] Erik Talvitie. Self-Correcting Models for Model- Based Reinforcement Learning. In AAAI Conference on Artificial Intelligence, 2017.","title":"Self-Correcting Models for Model-Based Reinforcement Learning","authors":"Erik Talvitie, Talvitie","year":"2017","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Van Hoof et al., 2015] H Van Hoof, J. Peters, and G Neu- mann. Learning of Non-Parametric Control Policies with High- Dimensional State Features. AI and Statistics, 2015.","title":"Learning of Non-Parametric Control Policies with High-Dimensional State Features","authors":"J Van Hoof, G Peters, Neumann","year":"2015","venue":"AI and Statistics","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[van Seijen and Sutton, 2015] H van Seijen and R.S. Sutton. A deeper look at planning as learning from replay. In International Conference on Machine Learning, 2015.","title":"A deeper look at planning as learning from replay","authors":"H Van Seijen, R S Sutton","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[White, 2017] Martha White. Unifying task specification in rein- forcement learning. In International Conference on Machine Learning, 2017.","title":"Unifying task specification in reinforcement learning","authors":"Martha White","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yao et al., 2014] Hengshuai Yao, Csaba Szepesvári, Bernardo Avila Pires, and Xinhua Zhang. Pseudo-MDPs and factored linear action models. In ADPRL, 2014.","title":"Pseudo-MDPs and factored linear action models","authors":"Hengshuai Yao, Csaba Szepesvári, Bernardo Avila Pires, Xinhua Zhang","year":"2014","venue":"ADPRL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}