{"type":"ScholarlyArticle","title":"DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewards","authors":[{"name":"Shanchuan Wan","slug":"shanchuan-wan","given":"Shanchuan","family":"Wan","affiliation":"The University of Tokyo"},{"name":"Yujin Tang","slug":"yujin-tang","given":"Yujin","family":"Tang","affiliation":"Google Research , Brain Team"},{"name":"Yingtao Tian","slug":"yingtao-tian","given":"Yingtao","family":"Tian","affiliation":"Google Research , Brain Team"},{"name":"Tomoyuki Kaneko","slug":"tomoyuki-kaneko","given":"Tomoyuki","family":"Kaneko","affiliation":"The University of Tokyo"}],"abstract":"Exploration is a fundamental aspect of reinforcement learning (RL), and its effectiveness is a deciding factor in the performance of RL algorithms, especially when facing sparse extrinsic rewards. Recent studies have shown the effectiveness of encouraging exploration with intrinsic rewards estimated from novelties in observations. However, there is a gap between the novelty of an observation and an exploration, as both the stochasticity in the environment and the agent's behavior may affect the observation. To evaluate exploratory behaviors accurately, we propose DEIR, a novel method in which we theoretically derive an intrinsic reward with a conditional mutual information term that principally scales with the novelty contributed by agent explorations, and then implement the reward with a discriminative forward model. Extensive experiments on both standard and advanced exploration tasks in MiniGrid show that DEIR quickly learns a better policy than the baselines. Our evaluations on ProcGen demonstrate both the generalization capability and the general applicability of our intrinsic reward.","doi":"10.24963/ijcai.2023/477","url":"https://ijcai.org/proceedings/2023/477","pdf":"https://ijcai.org/proceedings/2023/0477.pdf","markdown":"https://ijcai.org/proceedings/2023/477.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/477","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":477,"pages":{"first":4289,"last":4298},"keywords":["Machine Learning: ML: Reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Agarwal et al., 2021] Rishabh Agarwal, Marlos C. Machado, Pablo Samuel Castro, and Marc G. Belle- mare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. arXiv preprint arXiv:2101.05265, 2021.","title":"Contrastive behavioral similarity embeddings for generalization in reinforcement learning","authors":"Rishabh Agarwal, Marlos C Machado, Pablo Samuel Castro, Marc G Bellemare","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alain and Bengio, 2016] Guillaume Alain and Yoshua Ben- gio. Understanding intermediate layers using linear clas- sifier probes. arXiv preprint arXiv:1610.01644, 2016.","title":"Understanding intermediate layers using linear classifier probes","authors":"Alain, Bengio","year":"2016","venue":"Guillaume Alain and Yoshua Bengio","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Badia et al., 2020] Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, et al. Never give up: Learning directed exploration strategies. In International Conference on Learning Representations, 2020.","title":"Never give up: Learning directed exploration strategies","authors":"Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Baker et al., 2019] Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent au- tocurricula. In International Conference on Learning Rep- resentations, 2019.","title":"Emergent tool use from multi-agent autocurricula","authors":"Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob Mcgrew, Igor Mordatch","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bellemare et al., 2016] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.","title":"Unifying count-based exploration and intrinsic motivation","authors":"Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, Remi Munos","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Blundell et al., 2016] Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z. Leibo, Jack Rae, Daan Wierstra, and Demis Hass- abis. Model-free episodic control. arXiv preprint arXiv:1606.04460, 2016.","title":"Model-free episodic control","authors":"Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, Demis Hassabis","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bretagnolle and Huber, 1978] Jean Bretagnolle and Cather- ine Huber. Estimation des densités: risque minimax. Séminaire de probabilités de Strasbourg, 12:342-363, 1978.","title":"Jean Bretagnolle and Catherine Huber. Estimation des densités: risque minimax","authors":"Huber Bretagnolle","year":"1978","venue":"Séminaire de probabilités de Strasbourg","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Burda et al., 2019] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random net- work distillation. In Seventh International Conference on Learning Representations, 2019.","title":"Exploration by random network distillation","authors":"Yuri Burda, Harrison Edwards, Amos Storkey, Oleg Klimov","year":"2019","venue":"Seventh International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chevalier-Boisvert et al., 2018] Maxime Chevalier- Boisvert, Lucas Willems, and Suman Pal. Min- imalistic gridworld environment for openai gym. https://github.com/maximecb/gym-minigrid, 2018. Accessed: 2023-05-18.","title":"Minimalistic gridworld environment for openai gym","authors":"Maxime Chevalier-Boisvert, Lucas Willems, Suman Pal","year":"2023","venue":null,"doi":null,"url":"https://github.com/maximecb/gym-minigrid,2018","internal_url":null},{"position":10,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the proper- ties of neural machine translation: Encoder-decoder ap- proaches. Syntax, Semantics and Structure in Statistical Translation, page 103, 2014.","title":"On the properties of neural machine translation: Encoder-decoder approaches","authors":"Cho","year":"2014","venue":"Syntax, Semantics and Structure in Statistical Translation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Cobbe et al., 2019] Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Procgen benchmark. https: //github.com/openai/procgen, 2019. Accessed: 2023-05- 18.","title":"Procgen benchmark","authors":"Karl Cobbe, Christopher Hesse, Jacob Hilton, John Schulman","year":"2019","venue":null,"doi":null,"url":"https://github.com/openai/procgen","internal_url":null},{"position":12,"raw":"[Cobbe et al., 2020] Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International Con- ference on Machine Learning, pages 2048-2056. PMLR, 2020.","title":"Leveraging procedural generation to benchmark reinforcement learning","authors":"Karl Cobbe, Chris Hesse, Jacob Hilton, John Schulman","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Cobbe et al., 2021] Karl W. Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In International Conference on Machine Learning, pages 2020-2027. PMLR, 2021.","title":"Phasic policy gradient","authors":"Karl W Cobbe, Jacob Hilton, Oleg Klimov, John Schulman","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Espeholt et al., 2018] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Con- ference on Machine Learning, pages 1407-1416. PMLR, 2018.","title":"Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures","authors":"Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Goyal et al., 2017] Anirudh Goyal, Alessandro Sordoni, Marc-Alexandre Côté, Nan Rosemary Ke, and Yoshua Bengio. Z-forcing: Training stochastic recurrent networks. In Advances in neural information processing systems, volume 30, 2017.","title":"Z-forcing: Training stochastic recurrent networks","authors":"Anirudh Goyal, Alessandro Sordoni, Marc-Alexandre Côté, Nan Rosemary Ke, Yoshua Bengio","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Gulrajani et al., 2017] Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. PixelVAE: A latent vari- able model for natural images. In International Confer- ence on Learning Representations, 2017.","title":"PixelVAE: A latent variable model for natural images","authors":"Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, Aaron Courville","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hafner et al., 2020] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.","title":"Dream to control: Learning behaviors by latent imagination","authors":"Hafner","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Laskin et al., 2020] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning. In International Con- ference on Machine Learning, pages 5639-5650. PMLR, 2020.","title":"Curl: Contrastive unsupervised representations for reinforcement learning","authors":"Michael Laskin, Aravind Srinivas, Pieter Abbeel","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, K ; Andreas, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Murphy, 2022] Kevin P. Murphy. Probabilistic Machine Learning: An introduction. MIT Press, 2022.","title":"Probabilistic Machine Learning: An introduction","authors":"Kevin P Murphy","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ostrovski et al., 2017] Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pages 2721-2730. PMLR, 2017.","title":"Count-based exploration with neural density models","authors":"Georg Ostrovski, Aäron Marc G Bellemare, Rémi Oord, Munos","year":"2017","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pathak et al., 2017] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, pages 2778-2787.","title":"Curiosity-driven exploration by self-supervised prediction","authors":"Deepak Pathak, Pulkit Agrawal, Alexei A Efros, Trevor Darrell","year":null,"venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"PMLR, 2017.","title":null,"authors":"Pmlr","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Pritzel et al., 2017] Alexander Pritzel, Benigno Uria, Sri- ram Srinivasan, Adria Puigdomenech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. In International Con- ference on Machine Learning, pages 2827-2836. PMLR, 2017.","title":"Neural episodic control","authors":"Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, Charles Blundell","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Raffin et al., 2021] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021.","title":"Stable-baselines3: Reliable reinforcement learning implementations","authors":"Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, Noah Dormann","year":"2021","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Raileanu and Fergus, 2021] Roberta Raileanu and Rob Fer- gus. Decoupling value and policy for generalization in re- inforcement learning. In International Conference on Ma- chine Learning, pages 8787-8798. PMLR, 2021.","title":"Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforcement learning","authors":"Fergus Raileanu","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Rosenblatt, 1958] Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organiza- tion in the brain. Psychological review, 65(6):386, 1958.","title":"The perceptron: a probabilistic model for information storage and organization in the brain","authors":"Frank Rosenblatt","year":"1958","venue":"Psychological review","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Rumelhart et al., 1986] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning Internal Rep- resentations by Error Propagation. MIT Press, 1986.","title":"Learning Internal Representations by Error Propagation","authors":"David E Rumelhart, Geoffrey E Hinton, Ronald J Williams","year":"1986","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Scott, 2010] Steven L. Scott. A modern bayesian look at the multi-armed bandit. Applied Stochastic Models in Busi- ness and Industry, 26(6):639-658, 2010.","title":"A modern bayesian look at the multi-armed bandit","authors":"Steven L Scott","year":"2010","venue":"Applied Stochastic Models in Business and Industry","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Sutton and Barto, 2018] Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, 2nd edition, 2018.","title":"Introduction to Reinforcement Learning","authors":"Richard S Sutton, Andrew G Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Tang et al., 2017] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schul- man, Filip De Turck, and Pieter Abbeel. #Exploration: A study of count-based exploration for deep reinforcement learning. In Advances in Neural Information Processing Systems, pages 4-9, 2017.","title":"#Exploration: A study of count-based exploration for deep reinforcement learning","authors":"Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, Pieter Abbeel","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Tsybakov, 2008] Alexandre B. Tsybakov. Introduction to nonparametric estimation. In Springer Series in Statistics, 2008.","title":"Introduction to nonparametric estimation","authors":"B Alexandre, Tsybakov","year":"2008","venue":"Springer Series in Statistics","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2021] Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E. Gonzalez, and Yuandong Tian. NovelD: A simple yet effective explo- ration criterion. In Advances in Neural Information Pro- cessing Systems, 2021.","title":"NovelD: A simple yet effective exploration criterion","authors":"Zhang","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}