{"type":"ScholarlyArticle","title":"Exploration by Distributional Reinforcement Learning","authors":[{"name":"Yunhao Tang","slug":"yunhao-tang","given":"Yunhao","family":"Tang","affiliation":"Columbia University IEOR"},{"name":"Shipra Agrawal","slug":"shipra-agrawal","given":"Shipra","family":"Agrawal","affiliation":"Columbia University IEOR"}],"abstract":"We propose a framework based on distributional reinforcement learning and recent attempts to combine Bayesian parameter updates with deep reinforcement learning. We show that our proposed framework conceptually unifies multiple previous methods in exploration. We also derive a practical algorithm that achieves efficient exploration on challenging control tasks.","doi":"10.24963/ijcai.2018/376","url":"https://ijcai.org/proceedings/2018/376","pdf":"https://ijcai.org/proceedings/2018/0376.pdf","markdown":"https://ijcai.org/proceedings/2018/376.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/376","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":376,"pages":{"first":2710,"last":2716},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning: Neural Networks"],"videos":[],"references":[{"position":1,"raw":"[Azizzadenesheli et al., 2017] Kamyar Azizzadenesheli, Emma Brunskill, and Animashree Anandkumar. Ef- ficient exploration through bayesian deep q networks. Symposium on Deep Reinforcement Learning, NIPS, 2017.","title":"Efficient exploration through bayesian deep q networks","authors":"Kamyar Azizzadenesheli, Emma Brunskill, Animashree Anandkumar","year":"2017","venue":"Symposium on Deep Reinforcement Learning, NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellemare et al., 2017] Marc G. Bellemare, Will Dabney, and Remi Munos. A distributional perspective on rein- forcement learning. International Conference on Machine Learning, 2017.","title":"A distributional perspective on reinforcement learning","authors":"G Marc, Will Bellemare, Remi Dabney, Munos","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blei et al., 2017] David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Associ- ation, Volume 112 -Issue 518, 2017.","title":"Variational inference: A review for statisticians","authors":"David M Blei, Alp Kucukelbir, Jon D Mcauliffe","year":"2017","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. Arxiv: 1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dearden et al., 1998] Richard Dearden, Nir Friedman, and Stuart Russel. Bayesian q learning. American Association for Artificial Intelligence (AAAI), 1998.","title":"Bayesian q learning","authors":"Richard Dearden, Nir Friedman, Stuart Russel","year":"1998","venue":"American Association for Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Duan et al., 2016] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. Interna- tional Conference on Machine Learning, 2016.","title":"Benchmarking deep reinforcement learning for continuous control","authors":"Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fortunato et al., 2017] Meire Fortunato, Moham- mad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Ilivier Pietquin, Charles Blundell, and Shane Legg. Noisy network for exploration. arXiv:1706.10295, 2017.","title":"Noisy network for exploration","authors":"Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Ilivier Pietquin, Charles Blundell, Shane Legg","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Henderson et al., 2017] Peter Henderson, Thang Doan, Ri- ashat Islam, and David Meger. Bayesian policy gradients via alpha divergence dropout inference. 2nd Workshop on Bayesian Deep Learning, NIPS, 2017.","title":"Bayesian policy gradients via alpha divergence dropout inference. 2nd Workshop on Bayesian Deep Learning","authors":"Peter Henderson, Thang Doan, Riashat Islam, David Meger","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Levine et al., 2016] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End to end training of deep visuomotor policies. Journal of Machine Learning Re- search, 2016.","title":"End to end training of deep visuomotor policies","authors":"Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel","year":"2016","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lillicrap et al., 2016] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yu- val Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. International Conference on Learning Representations, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra","year":"2016","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lipton et al., 2016] Zachary C. Lipton, Xiujun Li, Jianfeng Gao, Lihong Li, Faisal Ahmed, and Li Deng. Effi- cient dialogue policy learning with bbq-networks. ArXiv: 1608.05081, 2016.","title":"Efficient dialogue policy learning with bbq-networks","authors":"Zachary C Lipton, Xiujun Li, Jianfeng Gao, Lihong Li, Faisal Ahmed, Li Deng","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. NIPS workshop in Deep Learning, 2013. [Moerland et al., 2017]","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":"NIPS workshop in Deep Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"Thomas M. Moerland, Joost Broekens, and Catholijn M. Jonker. Efficient exploration with double uncertain value networks. Symposium on Deep Reinforcement Learning, NIPS, 2017.","title":"Efficient exploration with double uncertain value networks","authors":"M Thomas, Joost Moerland, Catholijn M Broekens, Jonker","year":"2017","venue":"Symposium on Deep Reinforcement Learning, NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Morimura et al., 2010] Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Nonparametric return distribution approximation for reinforcement learning. ICML, 2010.","title":"Nonparametric return distribution approximation for reinforcement learning","authors":"Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, Toshiyuki Tanaka","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Morimura et al., 2012] Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Parametric return density estimation for reinforcement learning. UAI, 2012.","title":"Parametric return density estimation for reinforcement learning","authors":"Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, Toshiyuki Tanaka","year":"2012","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Osband and Van Roy, 2015] Ian Osband and Benjamin Van Roy. Bootstrapped thompson sampling and deep explo- ration. arXiv:1507:00300, 2015.","title":"Bootstrapped thompson sampling and deep exploration","authors":"Van Osband, Ian Roy, Benjamin Osband, Roy Van","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Osband et al., 2013] Ian Osband, Daniel Russo, and Ben- jamin Van Roy. (more) efficient reinforcement learning via posterior sampling. Arxiv: 1306.0940, 2013.","title":"efficient reinforcement learning via posterior sampling","authors":"Ian Osband, Daniel Russo, Benjamin Van, Roy","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Osband et al., 2016] Ian Osband, Charles Blundell, Alexan- der Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. arXiv:1602.04621, 2016.","title":"Deep exploration via bootstrapped dqn","authors":"Ian Osband, Charles Blundell, Alexander Pritzel, Benjamin Van, Roy","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Osband et al., 2017] Ian Osband, daniel Russo, Zheng Wen, and Benjamin Van Roy. Deep exploration via randomized value functions. arXiv: 1703.07608, 2017.","title":"Deep exploration via randomized value functions","authors":"Ian Osband, Zheng Daniel Russo, Benjamin Wen, Roy Van","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Plappert et al., 2016] Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. In- ternational Conference on Learning Representation, 2016.","title":"Parameter space noise for exploration","authors":"Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, Marcin Andrychowicz","year":"2016","venue":"ternational Conference on Learning Representation","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ranganath et al., 2014] Rejesh Ranganath, Sean Gerrish, and David M. Blei. Black box variational inference. Pro- ceedings of the 17th International Conference on Artificial Intelligence and Statistics (AISTATS), 2014.","title":"Black box variational inference","authors":"Ranganath","year":"2014","venue":"Proceedings of the 17th International Conference on Artificial Intelligence and Statistics (AISTATS)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Russo, 2017] Daniel Russo. Tutorial on thompson sam- pling. arxiv, 2017.","title":"Tutorial on thompson sampling","authors":"Daniel Russo","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization. International Conference on Machine Learning, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Philipp Moritz, Michael I Jordan, Pieter Abbeel","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tang and Kucukelbir, 2017] Yunhao Tang and Alp Ku- cukelbir. Variational deep q network. 2nd Workshop on Bayesian Deep Learning, NIPS, 2017.","title":"Variational deep q network. 2nd Workshop on Bayesian Deep Learning","authors":"Yunhao Tang, Alp Kucukelbir","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Thompson, 1933] William R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, Vol. 25, No. 3/4, 1933.","title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples","authors":"R William, Thompson","year":"1933","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. Mujoco: A physics engine for model-based con- trol. International Conference on Intelligent Robots, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"International Conference on Intelligent Robots","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tsitsiklis and Van Roy, 1996] John N. Tsitsiklis and Ben- jamin Van Roy. Feature based methods for large scale dynamic programming. Machine Learning, 1996.","title":"Feature based methods for large scale dynamic programming","authors":"John N Tsitsiklis, Benjamin Van, Roy","year":"1996","venue":"Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}