{"type":"ScholarlyArticle","title":"Hashing over Predicted Future Frames for Informed Exploration of Deep Reinforcement Learning","authors":[{"name":"Haiyan Yin","slug":"haiyan-yin","given":"Haiyan","family":"Yin","affiliation":"Nanyang Technological University , Singapore"},{"name":"Jianda Chen","slug":"jianda-chen","given":"Jianda","family":"Chen","affiliation":"Nanyang Technological University , Singapore"},{"name":"Sinno Jialin Pan","slug":"sinno-jialin-pan","given":"Sinno Jialin","family":"Pan","affiliation":"Nanyang Technological University , Singapore"}],"abstract":"In deep reinforcement learning (RL) tasks, an efficient exploration mechanism should be able to encourage an agent to take actions that lead to less frequent states which may yield higher accumulative future return. However, both knowing about the future and evaluating the frequentness of states are non-trivial tasks, especially for deep RL domains, where a state is represented by high-dimensional image frames. In this paper, we propose a novel informed exploration framework for deep RL, where we build the capability for an RL agent to predict over the future transitions and evaluate the frequentness for the predicted future frames in a meaningful manner. To this end, we train a deep prediction model to predict future frames given a state-action pair, and a convolutional autoencoder model to hash over the seen frames. In addition, to utilize the counts derived from the seen frames to evaluate the frequentness for the predicted frames, we tackle the challenge of matching the predicted future frames and their corresponding seen frames at the latent feature level. In this way, we derive a reliable metric for evaluating the novelty of the future direction pointed by each action, and hence inform the agent to explore the least frequent one.","doi":"10.24963/ijcai.2018/420","url":"https://ijcai.org/proceedings/2018/420","pdf":"https://ijcai.org/proceedings/2018/0420.pdf","markdown":"https://ijcai.org/proceedings/2018/420.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/420","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":420,"pages":{"first":3026,"last":3032},"keywords":["Machine Learning: Deep Learning","Machine Learning Applications: Applications of Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Achiam and Sastry, 2017] Joshua Achiam and Shankar Sas- try. Surprise-based intrinsic motivation for deep reinforce- ment learning. arXiv preprint arXiv:1703.01732, 2017.","title":"Joshua Achiam and Shankar Sastry. Surprise-based intrinsic motivation for deep reinforcement learning","authors":"Sastry Achiam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellemare et al., 2013] M. G. Bellemare, Y. Naddaf, J. Ve- ness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Arti- ficial Intelligence Research, 47:253-279, Jun 2013.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"M G Bellemare, Y Naddaf, J Veness, M Bowling","year":"2013","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bellemare et al., 2016] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In NIPS, pages 1471-1479, 2016.","title":"Unifying count-based exploration and intrinsic motivation","authors":"Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, Remi Munos","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Blundell et al., 2017] Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis. Model-free episodic control. In ICML, 2017.","title":"Model-free episodic control","authors":"Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, Demis Hassabis","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Charikar, 2002] Moses S Charikar. Similarity estimation techniques from rounding algorithms. In STOC, pages 380-388. ACM, 2002.","title":"Similarity estimation techniques from rounding algorithms","authors":"Moses S Charikar, Charikar","year":"2002","venue":"STOC","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Houthooft et al., 2016] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In NIPS, pages 1109-1117, 2016.","title":"Vime: Variational information maximizing exploration","authors":"Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, Pieter Abbeel","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kingma and Ba, 2014] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kingma and Welling, 2014] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lillicrap et al., 2016] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Martin et al., 2017] Jarryd Martin, Suraj Narayanan Sasiku- mar, Tom Everitt, and Marcus Hutter. Count-based explo- ration in feature space for reinforcement learning. In IJCAI, 2017.","title":"Count-based exploration in feature space for reinforcement learning","authors":"Jarryd Martin, Suraj Narayanan Sasikumar, Tom Everitt, Marcus Hutter","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/344"},{"position":11,"raw":"[Mnih et al., 2015] V. Mnih, K. Kavukcuoglu, D. Silver, A. a Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. K. Fidjeland, G. Ostrovski, S. Petersen, A. Sadik C. Beattie, I. Antonoglou, D. Kumaran H. King, D. Wier- stra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 2015.","title":"Human-level control through deep reinforcement learning","authors":"V Mnih, K Kavukcuoglu, D Silver, A Rusu, J Veness, M G Bellemare, A Graves, M Riedmiller, A K Fidjeland, G Ostrovski, S Petersen, A Sadik, C Beattie, I Antonoglou, D Kumaran, H King, D Wierstra, S Legg, D Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In ICML, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Oh et al., 2015] Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. In NIPS, pages 2845-2853. 2015.","title":"Action-conditional video prediction using deep networks in atari games","authors":"Oh","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Osband et al., 2016] Ian Osband, Charles Blundell, Alexan- der Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In NIPS, pages 4026-4034, 2016.","title":"Deep exploration via bootstrapped dqn","authors":"Ian Osband, Charles Blundell, Alexander Pritzel, Benjamin Van, Roy","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ostrovski et al., 2017] Georg Ostrovski, Marc G Bellemare, Aaron van den Oord, and Rémi Munos. Count-based ex- ploration with neural density models. In ICML, 2017.","title":"Count-based exploration with neural density models","authors":"Georg Ostrovski, Aaron Marc G Bellemare, Rémi Van Den Oord, Munos","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Parisotto et al., 2016] Emilio Parisotto, Jimmy Ba, and Rus- lan Salakhutdinov. Actor-mimic deep multitask and transfer reinforcement learning. In ICLR, 2016.","title":"Actor-mimic deep multitask and transfer reinforcement learning","authors":"Emilio Parisotto, Jimmy Ba, Ruslan Salakhutdinov","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pathak et al., 2017] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In ICML, 2017.","title":"Curiosity-driven exploration by self-supervised prediction","authors":"Deepak Pathak, Pulkit Agrawal, Alexei A Efros, Trevor Darrell","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rusu et al., 2016] Andrei A. Rusu, Sergio Gomez Col- menarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. In ICLR, 2016.","title":"Policy distillation","authors":"Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, Raia Hadsell","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schaul et al., 2016] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience re- play. In ICLR, 2016.","title":"Prioritized experience replay","authors":"Tom Schaul, John Quan, Ioannis Antonoglou, David Silver","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Stadie et al., 2015] Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.00814, 2015.","title":"Incentivizing exploration in reinforcement learning with deep predictive models","authors":"Sergey Bradly C Stadie, Pieter Levine, Abbeel","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Strehl and Littman, 2008] Alexander L Strehl and Michael L Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8):1309-1331, 2008.","title":"An analysis of model-based interval estimation for markov decision processes","authors":"L Alexander, Strehl, Michael L Littman","year":"2008","venue":"Journal of Computer and System Sciences","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tang et al., 2017] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schul- man, Filip De Turck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. In NIPS, 2017.","title":"# exploration: A study of count-based exploration for deep reinforcement learning","authors":"Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, Pieter Abbeel","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Thrun, 1992] Sebastian B Thrun. Efficient exploration in reinforcement learning. 1992.","title":"Efficient exploration in reinforcement learning","authors":"Sebastian B Thrun","year":"1992","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2016] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot, and Nando de Freitas. Du- eling network architectures for deep reinforcement learning. In ICML, pages 1995-2003, 2016.","title":"Dueling network architectures for deep reinforcement learning","authors":"Ziyu Wang, Tom Schaul, Matteo Hessel, Marc Hado Van Hasselt, Nando Lanctot, De Freitas","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yin and Pan, 2017] Haiyan Yin and Sinno Jialin Pan. Knowl- edge transfer for deep reinforcement learning with hierar- chical experience replay. In AAAI, pages 1640-1646, 2017.","title":"Knowledge transfer for deep reinforcement learning with hierarchical experience replay","authors":"Haiyan Yin, Sinno Jialin Pan","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}