{"type":"ScholarlyArticle","title":"Assumed Density Filtering Q-learning","authors":[{"name":"Heejin Jeong","slug":"heejin-jeong","given":"Heejin","family":"Jeong","affiliation":"University of Pennsylvania , Philadelphia , PA 19104"},{"name":"Clark Zhang","slug":"clark-zhang","given":"Clark","family":"Zhang","affiliation":"University of Pennsylvania , Philadelphia , PA 19104"},{"name":"George J. Pappas","slug":"george-j-pappas","given":"George J.","family":"Pappas","affiliation":"University of Pennsylvania , Philadelphia , PA 19104"},{"name":"Daniel D. Lee","slug":"daniel-d-lee","given":"Daniel D.","family":"Lee","affiliation":"Cornell Tech , New York , NY 10044"}],"abstract":"While off-policy temporal difference (TD) methods have widely been used in reinforcement learning due to their efficiency and simple implementation, their Bayesian counterparts have not been utilized as frequently. One reason is that the non-linear max operation in the Bellman optimality equation makes it difficult to define conjugate distributions over the value functions. In this paper, we introduce a novel Bayesian approach to off-policy TD methods, called as ADFQ, which updates beliefs on state-action values, Q, through an online Bayesian inference method known as Assumed Density Filtering. We formulate an efficient closed-form solution for the value update by approximately estimating analytic parameters of the posterior of the Q-beliefs. Uncertainty measures in the beliefs not only are used in exploration but also provide a natural regularization for the value update considering all next available actions. ADFQ converges to Q-learning as the uncertainty measures of the Q-beliefs decrease and improves common drawbacks of other Bayesian RL algorithms such as computational complexity. We extend ADFQ with a neural network.  Our empirical results demonstrate that ADFQ outperforms comparable algorithms on various Atari 2600 games, with drastic improvements in highly stochastic domains or domains with a large action space.","doi":"10.24963/ijcai.2019/362","url":"https://ijcai.org/proceedings/2019/362","pdf":"https://ijcai.org/proceedings/2019/0362.pdf","markdown":"https://ijcai.org/proceedings/2019/362.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/362","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":362,"pages":{"first":2607,"last":2613},"keywords":["Uncertainty in AI: Approximate Probabilistic Inference","Machine Learning: Reinforcement Learning","Machine Learning: Probabilistic Machine Learning","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Azizzadenesheli et al., 2018] Kamyar Azizzadenesheli, Emma Brunskill, and Animashree Anandkumar. Efficient exploration through bayesian deep q-networks. arXiv preprint arXiv:1802.04412, 2018.","title":"Efficient exploration through bayesian deep q-networks","authors":"Kamyar Azizzadenesheli, Emma Brunskill, Animashree Anandkumar","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellemare et al., 2017] Marc G. Bellemare, Will Dabney, , and Rémi Munos. A distributional perspective on reinforce- ment learning. In Proceedings of the 34th International Conference on Machine Learning, 2017.","title":"A distributional perspective on reinforcement learning","authors":"G Marc, Will Bellemare, Rémi Dabney, Munos","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Boyen and Koller, 1998] Xavier Boyen and Daphne Koller. Tractable inference for complex stochastic processes. In Proceedings of the 14th conference on Uncertainty in Arti- ficial Intelligence, Berkeley, CA, 1998.","title":"Tractable inference for complex stochastic processes","authors":"Koller Boyen, Xavier Boyen, Daphne Koller","year":"1998","venue":"Proceedings of the 14th conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Lud- wig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chowdhary et al., 2014] Girish Chowdhary, Miao Liu, Robert Grande, Thomas Walsh, Jonathan How, and Lawrence Carin. Off-policy reinforcement learning with gaussian process. IEEE/CAA Journal of Automatica Sinica, 1(3):227-238, 2014.","title":"Off-policy reinforcement learning with gaussian process","authors":"Girish Chowdhary, Miao Liu, Robert Grande, Thomas Walsh, Jonathan How, Lawrence Carin","year":"2014","venue":"IEEE/CAA Journal of Automatica Sinica","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dearden et al., 1998] Richard Dearden, Nir Friedman, and Stuart Russell. Bayesian q-learning. In AAAI/IAAI, pages 761-768, 1998.","title":"Bayesian q-learning","authors":"Richard Dearden, Nir Friedman, Stuart Russell","year":"1998","venue":"AAAI/IAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dearden et al., 1999] Richard Dearden, Nir Friedman, and David Andre. Model based bayesian exploration. In Pro- ceedings of the 15th conference on uncertainty in artificial intelligence, pages 150-159, 1999.","title":"Model based bayesian exploration","authors":"Richard Dearden, Nir Friedman, David Andre","year":"1999","venue":"Proceedings of the 15th conference on uncertainty in artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Duff, 2002] Michael Duff. Optimal learning: Computational procedures for bayes-adaptive markov decision processes.","title":"Optimal learning: Computational procedures for bayes-adaptive markov decision processes","authors":"Michael Duff","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Engel et al., 2003] Yaakov Engel, Shie Mannor, and Ron Meir. Bayes meets bellman: The gaussian process approach to temporal difference learning. In Proceedings of the 20th International Conference on Machine Learning, 2003.","title":"Bayes meets bellman: The gaussian process approach to temporal difference learning","authors":"Yaakov Engel, Shie Mannor, Ron Meir","year":"2003","venue":"Proceedings of the 20th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Engel et al., 2005] Yaakov Engel, Shie Mannor, and Ron Meir. Reinforcement learning with gaussian processes. In Proceedings of the 22nd International Conference on Machine Learning, pages 201-208, 2005.","title":"Reinforcement learning with gaussian processes","authors":"Engel","year":"2005","venue":"Proceedings of the 22nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Geist and Pietquin, 2010] Matthieu Geist and Olivier Pietquin. Kalman temporal differences. Journal of Artificial Intelligence Research, 39:483-532, 2010.","title":"Matthieu Geist and Olivier Pietquin. Kalman temporal differences","authors":"Pietquin Geist","year":"2010","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Guez et al., 2012] Arthur Guez, David Silver, and Peter Dayan. Efficient bayes-adaptive reinforcement learning using sample-based search. In Advances in Neural Infor- mation Processing Systems, pages 1071-1079, 2012.","title":"Efficient bayes-adaptive reinforcement learning using sample-based search","authors":"Arthur Guez, David Silver, Peter Dayan","year":"2012","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Harutyunyan et al., 2016] Anna Harutyunyan, Marc G. Bellemare, Tom Stepleton, and Rémi Munos. Q (λ) with off-policy corrections. In International Conference on Algo- rithmic Learning Theory, pages 305-320. Springer, 2016.","title":"Q (λ) with off-policy corrections","authors":"Anna Harutyunyan, Marc G Bellemare, Tom Stepleton, Rémi Munos","year":"2016","venue":"International Conference on Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hasselt et al., 2016] Hado V. Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In 13th AAAI Conference on Artificial Intelli- gence, 2016. [Hasselt, 2010]","title":"Deep reinforcement learning with double q-learning","authors":"V Hado, Arthur Hasselt, David Guez, Silver","year":"2010","venue":"13th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"Hado V. Hasselt. Double q-learning. In Ad- vances in Neural Information Processing Systems, 2010.","title":"Double q-learning","authors":"V Hado, Hasselt","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kaelbling et al., 1996] Leslie P. Kaelbling, Michael L. Littman, and Andrew W. Moore. Reinforcement learn- ing: A survey. Journal of Artificial Intelligence Research, 4:237-285, 1996.","title":"Reinforcement learning: A survey","authors":"Leslie P Kaelbling, Michael L Littman, Andrew W Moore","year":"1996","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Maybeck, 1982] Peter S. Maybeck. Stochastic models, esti- mation and control. Academic Press, chapter 12.7, 1982.","title":"Stochastic models, estimation and control","authors":"Peter S Maybeck","year":"1982","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep rein- forcement learning. In Advances in Neural Information Processing Systems, Deep Learning Workshop, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":"Advances in Neural Information Processing Systems, Deep Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[O'Donoghue et al., 2017] Brendan O'Donoghue, Ian Os- band, Rémi Munos, and Volodymyr Mnih. The uncer- tainty bellman equation and exploration. arXiv preprint arXiv:1709.05380, 2017.","title":"The uncertainty bellman equation and exploration","authors":"O' Brendan, Ian Donoghue, Rémi Osband, Volodymyr Munos, Mnih","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Opper, 1999] Manfred Opper. A bayesian approach to online learning. On-Line Learning in Neural Networks, 1999.","title":"A bayesian approach to online learning","authors":"Manfred Opper, Opper","year":"1999","venue":"On-Line Learning in Neural Networks","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Osband et al., 2013] Ian Osband, Daniel Russo, and Ben- jamin V. Roy. (more) efficient reinforcement learning via posterior sampling. In Advances in Neural Information Processing Systems, pages 3003-3011, 2013.","title":"more) efficient reinforcement learning via posterior sampling","authors":"Ian Osband, Daniel Russo, Benjamin V Roy","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Osband et al., 2014] Ian Osband, Benjamin V. Roy, and Zheng Wen. Generalization and exploration via randomized value functions. arXiv preprint arXiv:1402.0631, 2014.","title":"Generalization and exploration via randomized value functions","authors":"Ian Osband, Benjamin V Roy, Zheng Wen","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Osband et al., 2016] Ian Osband, Charles Blundell, Alexan- der Pritzel, and Benjamin V. Roy. Deep exploration via bootstrapped dqn. In Advances in Neural Information Pro- cessing Systems, pages 4026-4034, 2016.","title":"Deep exploration via bootstrapped dqn","authors":"Ian Osband, Charles Blundell, Alexander Pritzel, Benjamin V Roy","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Poupart et al., 2006] Pascal Poupart, Nikos Vlassis, Jesse Hoey, and Kevin Regan. An analytic solution to discrete bayesian reinforcement learning. In Proceedings of the 23rd International Conference on Machine Learning, 2006.","title":"An analytic solution to discrete bayesian reinforcement learning","authors":"Pascal Poupart, Nikos Vlassis, Jesse Hoey, Kevin Regan","year":"2006","venue":"Proceedings of the 23rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Schaul et al., 2015] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience re- play. arXiv preprint arXiv:1511.05952, 2015.","title":"Prioritized experience replay","authors":"Tom Schaul, John Quan, Ioannis Antonoglou, David Silver","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Thompson, 1933] William R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 1933.","title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples","authors":"R William, Thompson","year":"1933","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tsitsiklis, 2002] John N. Tsitsiklis. On the convergence of optimistic policy iteration. Journal of Machine Learning Research, 3:59-72, 2002.","title":"On the convergence of optimistic policy iteration","authors":"John N Tsitsiklis","year":"2002","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Watkins and Dayan, 1992] Christopher J.C.H Watkins and Peter Dayan. Q-learning. In Machine Learning, pages 279-292, 1992.","title":"Q-learning","authors":"J C Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Ziebart, 2010] Brian D. Ziebart. Modeling purposeful adap- tive behavior with the principle of maximum causal entropy. 2010.","title":"Modeling purposeful adaptive behavior with the principle of maximum causal entropy","authors":"Brian D Ziebart","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}