{"type":"ScholarlyArticle","title":"Hill Climbing on Value Estimates for Search-control in Dyna","authors":[{"name":"Yangchen Pan","slug":"yangchen-pan","given":"Yangchen","family":"Pan","affiliation":"Department of Computing Science , University of Alberta , Canada"},{"name":"Hengshuai Yao","slug":"hengshuai-yao","given":"Hengshuai","family":"Yao","affiliation":"Huawei HiSilicon , Canada"},{"name":"Amir-massoud Farahmand","slug":"amir-massoud-farahmand","given":"Amir-massoud","family":"Farahmand","affiliation":"Vector Institute , Canada; Department of Computer Science , University of Toronto , Canada"},{"name":"Martha White","slug":"martha-white","given":"Martha","family":"White","affiliation":"Department of Computing Science , University of Alberta , Canada"}],"abstract":"Dyna is an architecture for model based reinforcement learning (RL), where simulated experience from a model is used to update policies or value functions. A key component of Dyna is search control, the mechanism to generate the state and action from which the agent queries the model, which remains largely unexplored. In this work, we propose to generate such states by using the trajectory obtained from Hill Climbing (HC) the current estimate of the value function. This has the effect of propagating value from high value regions and of preemptively updating value estimates of the regions that the agent is likely to visit next. We derive a noisy projected natural gradient algorithm for hill climbing, and highlight a connection to Langevin dynamics. We provide an empirical demonstration on four classical domains that our algorithm, HC Dyna, can obtain significant sample efficiency improvements. We study the properties of different sampling distributions for search control, and find that there appears to be a benefit specifically from using the samples generated by climbing on current value estimates from low value to high value region.","doi":"10.24963/ijcai.2019/445","url":"https://ijcai.org/proceedings/2019/445","pdf":"https://ijcai.org/proceedings/2019/0445.pdf","markdown":"https://ijcai.org/proceedings/2019/445.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/445","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":445,"pages":{"first":3209,"last":3215},"keywords":["Machine Learning: Reinforcement Learning","Planning and Scheduling: Other approaches to planning","Machine Learning: Deep Learning","Planning and Scheduling: Model-Based Reasoning"],"videos":[],"references":[{"position":1,"raw":"[Adam et al., 2012] Sander Adam, Lucian Busoniu, and Robert Babuska. Experience replay for real-time reinforce- ment learning control. IEEE Transactions on Systems, Man, and Cybernetics, pages 201-212, 2012.","title":"Experience replay for real-time reinforcement learning control","authors":"Sander Adam, Lucian Busoniu, Robert Babuska","year":"2012","venue":"IEEE Transactions on Systems, Man, and Cybernetics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amari and Douglas, 1998] Shun-Ichi Amari and Scott C. Douglas. Why natural gradient? IEEE International Con- ference on Acoustics, Speech and Signal Processing, pages 1213-1216, 1998.","title":"Why natural gradient? IEEE International Conference on Acoustics, Speech and Signal Processing","authors":"Douglas ; Shun-Ichi Amari, Scott C Amari, Douglas","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Amari, 1998] Shun-Ichi Amari. Natural gradient works effi- ciently in learning. Neural Computation, 10(2):251-276, 1998.","title":"Shun-Ichi Amari. Natural gradient works efficiently in learning","authors":"Amari","year":"1998","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chiang et al., 1987] Tzuu-Shuh Chiang, Chii-Ruey Hwang, and Shuenn Jyi Sheu. Diffusion for global optimization in R n . SIAM Journal on Control and Optimization, pages 737-753, 1987.","title":"Diffusion for global optimization in R n","authors":"Chiang","year":"1987","venue":"SIAM Journal on Control and Optimization","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Corneil et al., 2018] Dane S. Corneil, Wulfram Gerstner, and Johanni Brea. Efficient model-based deep reinforce- ment learning with variational state tabulation. ICML, pages 1049-1058, 2018.","title":"Efficient model-based deep reinforcement learning with variational state tabulation","authors":"Dane S Corneil, Wulfram Gerstner, Johanni Brea","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Durmus and Moulines, 2017] Alain Durmus and Eric Moulines. Nonasymptotic convergence analysis for the unadjusted Langevin algorithm. The Annals of Applied Probability, pages 1551-1587, 2017.","title":"Nonasymptotic convergence analysis for the unadjusted Langevin algorithm","authors":"Moulines Durmus, Alain Durmus, Eric Moulines","year":"2017","venue":"The Annals of Applied Probability","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gu et al., 2016] Shixiang Gu, Timothy P. Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous Deep Q- Learning with Model-based Acceleration. In ICML, pages 2829-2838, 2016.","title":"Continuous Deep Q-Learning with Model-based Acceleration","authors":"Gu","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Holland et al., 2018] G. Zacharias Holland, Erik Talvitie, and Michael Bowling. The effect of planning shape on dyna-style planning in high-dimensional state spaces. CoRR, abs/1806.01825, 2018.","title":"The effect of planning shape on dyna-style planning in high-dimensional state spaces","authors":"G Zacharias Holland, Erik Talvitie, Michael Bowling","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lillicrap et al., 2016] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin, 1992] Long-Ji Lin. Self-Improving Reactive Agents Based On Reinforcement Learning, Planning and Teaching. Machine Learning, 1992.","title":"Self-Improving Reactive Agents Based On Reinforcement Learning, Planning and Teaching","authors":"Long-Ji Lin","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, and et al. Human-level control through deep reinforcement learning. Nature, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Moore and Atkeson, 1993] Andrew W. Moore and Christo- pher G. Atkeson. Prioritized sweeping: Reinforcement learning with less data and less time. Machine learning, pages 103-130, 1993.","title":"Prioritized sweeping: Reinforcement learning with less data and less time","authors":"Andrew W Moore, Christopher G Atkeson","year":"1993","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pan et al., 2018] Yangchen Pan, Muhammad Zaheer, Adam White, Andrew Patterson, and Martha White. Organizing experience: a deeper look at replay mechanisms for sample- based planning in continuous state domains. In IJCAI, pages 4794-4800, 2018.","title":"Organizing experience: a deeper look at replay mechanisms for samplebased planning in continuous state domains","authors":"Yangchen Pan, Muhammad Zaheer, Adam White, Andrew Patterson, Martha White","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Peng and Williams, 1993] Jing Peng and Ronald J Williams. Efficient learning and planning within the Dyna framework. Adaptive behavior, 1993.","title":"Efficient learning and planning within the Dyna framework","authors":"Jing Peng, Ronald J Williams","year":"1993","venue":"Adaptive behavior","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Peng et al., 2018] Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, Kam-Fai Wong, and Shang-Yu Su. Deep Dyna-Q: Integrating planning for task-completion dialogue policy learning. In Annual Meeting of the Association for Computational Linguistics, pages 2182-2192, 2018.","title":"Deep Dyna-Q: Integrating planning for task-completion dialogue policy learning","authors":"Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, Kam-Fai Wong, Shang-Yu Su","year":"2018","venue":"Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Roberts, 1996] Richard L. Roberts, Gareth O.and Tweedie. Exponential convergence of langevin distributions and their discrete approximations. Bernoulli, pages 341-363, 1996.","title":"Exponential convergence of langevin distributions and their discrete approximations","authors":"Richard L Roberts, O Gareth, Tweedie","year":"1996","venue":"Bernoulli","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Santos et al., 2012] Matilde Santos, José Antonio Martín H., Victoria López, and Guillermo Botella. Dyna-H: A heuristic planning reinforcement learning algorithm applied to role- playing game strategy decision systems. Knowledge-Based Systems, 32:28-36, 2012.","title":"Dyna-H: A heuristic planning reinforcement learning algorithm applied to roleplaying game strategy decision systems","authors":"Matilde Santos, José Antonio Martín, H, Victoria López, Guillermo Botella","year":"2012","venue":"Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schaul et al., 2016] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized Experience Re- play. In ICLR, 2016.","title":"Prioritized Experience Replay","authors":"Tom Schaul, John Quan, Ioannis Antonoglou, David Silver","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Silver et al., 2014] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. De- terministic policy gradient algorithms. In ICML, pages I-387-I-395, 2014.","title":"Deterministic policy gradient algorithms","authors":"David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, Martin Riedmiller","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sutton and Barto, 2018] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018.","title":"Reinforcement Learning: An Introduction","authors":"Richard S Sutton, Andrew G Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sutton et al., 2008] Richard S. Sutton, Csaba Szepesvári, Al- borz Geramifard, and Michael Bowling. Dyna-style plan- ning with linear function approximation and prioritized sweeping. In UAI, pages 528-536, 2008.","title":"Dyna-style planning with linear function approximation and prioritized sweeping","authors":"Richard S Sutton, Csaba Szepesvári, Alborz Geramifard, Michael Bowling","year":"2008","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutton, 1990] Richard S. Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In ML, 1990.","title":"Integrated architectures for learning, planning, and reacting based on approximating dynamic programming","authors":"Richard S Sutton","year":"1990","venue":"ML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sutton, 1991] Richard S. Sutton. Dyna, an integrated ar- chitecture for learning, planning, and reacting. SIGART Bulletin, 2(4):160-163, 1991.","title":"Dyna, an integrated architecture for learning, planning, and reacting","authors":"Richard S Sutton","year":"1991","venue":"SIGART Bulletin","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Thomas et al., 2016] Philip Thomas, Bruno Castro Silva, Christoph Dann, and Emma Brunskill. Energetic natural gradient descent. In ICML, pages 2887-2895, 2016.","title":"Energetic natural gradient descent","authors":"Philip Thomas, Bruno Castro Silva, Christoph Dann, Emma Brunskill","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tsitsiklis, 1994] John N. Tsitsiklis. Asynchronous stochastic approximation and Q-learning. Machine Learning, pages 185-202, 1994.","title":"Asynchronous stochastic approximation and Q-learning","authors":"John N Tsitsiklis","year":"1994","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[van Seijen and Sutton, 2015] Harm van Seijen and Richard S. Sutton. A deeper look at planning as learning from replay. In ICML, pages 2314-2322, 2015.","title":"Harm van Seijen and Richard S. Sutton. A deeper look at planning as learning from replay","authors":"Sutton Van Seijen","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wawrzyński and Tanwani, 2013] Paweł Wawrzyński and Ajay Kumar Tanwani. Autonomous reinforcement learning with experience replay. Neural Networks, pages 156-167, 2013.","title":"Autonomous reinforcement learning with experience replay","authors":"Paweł Wawrzyński, Ajay Kumar, Tanwani","year":"2013","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Welling and Teh, 2011] Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient Langevin dynam- ics. In ICML, pages 681-688, 2011.","title":"Bayesian learning via stochastic gradient Langevin dynamics","authors":"Teh ; ] Max Welling, Yee Whye Welling, Teh","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}