{"type":"ScholarlyArticle","title":"Planning with Expectation Models","authors":[{"name":"Yi Wan","slug":"yi-wan","given":"Yi","family":"Wan","affiliation":"Reinforcement Learning and Artificial Intelligence Laboratory , University of Alberta {wan6 ,"},{"name":"Muhammad Zaheer","slug":"muhammad-zaheer","given":"Muhammad","family":"Zaheer","affiliation":"Reinforcement Learning and Artificial Intelligence Laboratory , University of Alberta {wan6 ,"},{"name":"Adam White","slug":"adam-white","given":"Adam","family":"White","affiliation":"Reinforcement Learning and Artificial Intelligence Laboratory , University of Alberta {wan6 ,"},{"name":"Martha White","slug":"martha-white","given":"Martha","family":"White","affiliation":"Reinforcement Learning and Artificial Intelligence Laboratory , University of Alberta {wan6 ,"},{"name":"Richard S. Sutton","slug":"richard-s-sutton","given":"Richard S.","family":"Sutton","affiliation":"Reinforcement Learning and Artificial Intelligence Laboratory , University of Alberta {wan6 ,"}],"abstract":"Distribution and sample models are two popular model choices in model-based reinforcement learning (MBRL). However, learning these models can be intractable, particularly when the state and action spaces are large. Expectation models, on the other hand, are relatively easier to learn due to their compactness and have also been widely used for deterministic environments. For stochastic environments, it is not obvious how expectation models can be used for planning as they only partially characterize a distribution. In this paper, we propose a sound way of using approximate expectation models for MBRL. In particular, we 1) show that planning with an expectation model is equivalent to planning with a distribution model if the state value function is linear in state features, 2) analyze two common parametrization choices for approximating the expectation: linear and non-linear expectation models, 3) propose a sound model-based policy evaluation algorithm and present its convergence results, and 4) empirically demonstrate the effectiveness of the proposed planning algorithm.","doi":"10.24963/ijcai.2019/506","url":"https://ijcai.org/proceedings/2019/506","pdf":"https://ijcai.org/proceedings/2019/0506.pdf","markdown":"https://ijcai.org/proceedings/2019/506.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/506","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":506,"pages":{"first":3649,"last":3655},"keywords":["Machine Learning: Reinforcement Learning","Planning and Scheduling: Planning Algorithms"],"videos":[],"references":[{"position":1,"raw":"[Baird, 1995] Leemon Baird. Residual algorithms: Rein- forcement learning with function approximation. In Ma- chine Learning Proceedings 1995, pages 30-37. Elsevier, 1995.","title":"Residual algorithms: Reinforcement learning with function approximation","authors":"Leemon Baird","year":"1995","venue":"Machine Learning Proceedings","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chua et al., 2018] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforce- ment learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pages 4759-4770, 2018.","title":"Deep reinforcement learning in a handful of trials using probabilistic dynamics models","authors":"Chua","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chung et al., 2018] Wesley Chung, Somjit Nath, Ajin Joseph, and Martha White. Two-timescale networks for nonlinear value function approximation. 2018.","title":"Two-timescale networks for nonlinear value function approximation","authors":"Wesley Chung, Somjit Nath, Ajin Joseph, Martha White","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Coulom, 2006] Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In Interna- tional conference on computers and games, pages 72-83. Springer, 2006.","title":"Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search","authors":"Coulom","year":"2006","venue":"International conference on computers and games","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deisenroth and Rasmussen, 2011] Marc Deisenroth and Carl E Rasmussen. Pilco: A model-based and data- efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11), pages 465-472, 2011.","title":"Pilco: A model-based and dataefficient approach to policy search","authors":"Rasmussen Deisenroth, Marc Deisenroth, Carl E Rasmussen","year":"2011","venue":"Proceedings of the 28th International Conference on machine learning (ICML-11)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ghiassian et al., 2018] Sina Ghiassian, Andrew Patterson, Martha White, Richard S. Sutton, and Adam White. On- line off-policy prediction. CoRR, abs/1811.02597, 2018.","title":"Online off-policy prediction","authors":"Sina Ghiassian, Andrew Patterson, Martha White, Richard S Sutton, Adam White","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Glorot and Bengio, 2010] Xavier Glorot and Yoshua Ben- gio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth in- ternational conference on artificial intelligence and statis- tics, pages 249-256, 2010.","title":"Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks","authors":"Bengio Glorot","year":"2010","venue":"Proceedings of the thirteenth international conference on artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jaderberg et al., 2016] Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397, 2016.","title":"Reinforcement learning with unsupervised auxiliary tasks","authors":"Jaderberg","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Konidaris et al., 2011] George Konidaris, Sarah Osentoski, and Philip Thomas. Value function approximation in rein- forcement learning using the fourier basis. In Twenty-fifth AAAI conference on artificial intelligence, 2011.","title":"Value function approximation in reinforcement learning using the fourier basis","authors":"George Konidaris, Sarah Osentoski, Philip Thomas","year":"2011","venue":"Twenty-fifth AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Le et al., 2017] Lei Le, Raksha Kumaraswamy, and Martha White. Learning sparse representations in reinforce- ment learning with sparse coding. arXiv preprint arXiv:1707.08316, 2017.","title":"Learning sparse representations in reinforcement learning with sparse coding","authors":"Lei Le, Raksha Kumaraswamy, Martha White","year":"2017","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2017/287"},{"position":11,"raw":"[Leibfried et al., 2016] Felix Leibfried, Nate Kushman, and Katja Hofmann. A deep learning approach for joint video frame and reward prediction in atari games. arXiv preprint arXiv:1611.07078, 2016.","title":"A deep learning approach for joint video frame and reward prediction in atari games","authors":"Felix Leibfried, Nate Kushman, Katja Hofmann","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Modayil et al., 2014] Joseph Modayil, Adam White, and Richard S Sutton. Multi-timescale nexting in a reinforce- ment learning robot. Adaptive Behavior, 22(2):146-160, 2014.","title":"Multi-timescale nexting in a reinforcement learning robot","authors":"Joseph Modayil, Adam White, Richard S Sutton","year":"2014","venue":"Adaptive Behavior","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Moore and Atkeson, 1993] Andrew W Moore and Christo- pher G Atkeson. Prioritized sweeping: Reinforcement learning with less data and less time. Machine learning, 13(1):103-130, 1993.","title":"Prioritized sweeping: Reinforcement learning with less data and less time","authors":"Andrew W, Moore, Christopher G Atkeson","year":"1993","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Oh et al., 2015] Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. In Advances in Neural Information Processing Systems, pages 2863-2871, 2015.","title":"Action-conditional video prediction using deep networks in atari games","authors":"Oh","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pan et al., 2018] Yangchen Pan, Muhammad Zaheer, Adam White, Andrew Patterson, and Martha White. Organiz- ing experience: a deeper look at replay mechanisms for sample-based planning in continuous state domains. arXiv preprint arXiv:1806.04624, 2018.","title":"Organizing experience: a deeper look at replay mechanisms for sample-based planning in continuous state domains","authors":"Yangchen Pan, Muhammad Zaheer, Adam White, Andrew Patterson, Martha White","year":"2018","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2018/666"},{"position":16,"raw":"[Parr et al., 2008] Ronald Parr, Lihong Li, Gavin Taylor, Christopher Painter-Wakefield, and Michael L Littman. An analysis of linear models, linear value-function ap- proximation, and feature selection for reinforcement learn- ing. In Proceedings of the 25th international conference on Machine learning, pages 752-759. ACM, 2008.","title":"An analysis of linear models, linear value-function approximation, and feature selection for reinforcement learning","authors":"Ronald Parr, Lihong Li, Gavin Taylor, Christopher Painter-Wakefield, Michael L Littman","year":"2008","venue":"Proceedings of the 25th international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sutton et al., 1999] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181-211, 1999.","title":"Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning","authors":"Doina Richard S Sutton, Satinder Precup, Singh","year":"1999","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sutton et al., 2009] Richard S Sutton, Hamid Reza Maei, Doina Precup, Shalabh Bhatnagar, David Silver, Csaba Szepesvári, and Eric Wiewiora. Fast gradient-descent methods for temporal-difference learning with linear func- tion approximation. In Proceedings of the 26th Annual In- ternational Conference on Machine Learning, pages 993- 1000. ACM, 2009.","title":"Fast gradient-descent methods for temporal-difference learning with linear function approximation","authors":"Hamid Richard S Sutton, Reza Maei, Doina Precup, Shalabh Bhatnagar, David Silver, Csaba Szepesvári, Eric Wiewiora","year":"2009","venue":"Proceedings of the 26th Annual International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sutton et al., 2011] Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time ar- chitecture for learning knowledge from unsupervised sen- sorimotor interaction. In The 10th International Con- ference on Autonomous Agents and Multiagent Systems- Volume 2, pages 761-768. International Foundation for Autonomous Agents and Multiagent Systems, 2011.","title":"Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction","authors":"Joseph Richard S Sutton, Michael Modayil, Thomas Delp, Patrick M Degris, Adam Pilarski, Doina White, Precup","year":"2011","venue":"The 10th International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sutton et al., 2012] Richard S Sutton, Csaba Szepesvári, Alborz Geramifard, and Michael P Bowling. Dyna-style planning with linear function approximation and priori- tized sweeping. arXiv preprint arXiv:1206.3285, 2012.","title":"Dyna-style planning with linear function approximation and prioritized sweeping","authors":"Csaba Richard S Sutton, Alborz Szepesvári, Michael P Geramifard, Bowling","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutton, 1991] R.S. Sutton. Integrated modeling and control based on reinforcement learning and dynamic program- ming. In Advances in Neural Information Processing Sys- tems, 1991.","title":"Integrated modeling and control based on reinforcement learning and dynamic programming","authors":"R S Sutton","year":"1991","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sutton, 1996] Richard S Sutton. Generalization in rein- forcement learning: Successful examples using sparse coarse coding. In Advances in neural information process- ing systems, pages 1038-1044, 1996.","title":"Generalization in reinforcement learning: Successful examples using sparse coarse coding","authors":"Richard S Sutton","year":"1996","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[White and others, 2015] Adam White et al. Developing a predictive approach to knowledge. 2015.","title":"Developing a predictive approach to knowledge","authors":"Adam White","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu, 2010] Huizhen Yu. Convergence of least squares tem- poral difference methods under general conditions. In ICML, pages 1207-1214, 2010.","title":"Convergence of least squares temporal difference methods under general conditions","authors":"Yu ;, Huizhen Yu","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}