{"type":"ScholarlyArticle","title":"Scaling Goal-based Exploration via Pruning Proto-goals","authors":[{"name":"Akhil Bagaria","slug":"akhil-bagaria","given":"Akhil","family":"Bagaria","affiliation":"Brown University , Providence , RI , USA"},{"name":"Tom Schaul","slug":"tom-schaul","given":"Tom","family":"Schaul","affiliation":"DeepMind , London , UK"}],"abstract":"One of the gnarliest challenges in reinforcement learning (RL) is exploration that scales to vast domains, where novelty-, or coverage-seeking behaviour falls short. Goal-directed, purposeful behaviours are able to overcome this, but rely on a good goal space. The core challenge in goal discovery is finding the right balance between generality (not hand-crafted) and tractability (useful, not too many). Our approach explicitly seeks the middle ground, enabling the human designer to specify a vast but meaningful proto-goal space, and an autonomous discovery process to refine this to a narrower space of controllable, reachable, novel, and relevant goals. The effectiveness of goal-conditioned exploration with the latter is then demonstrated in three challenging environments.","doi":"10.24963/ijcai.2023/384","url":"https://ijcai.org/proceedings/2023/384","pdf":"https://ijcai.org/proceedings/2023/0384.pdf","markdown":"https://ijcai.org/proceedings/2023/384.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/384","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":384,"pages":{"first":3451,"last":3460},"keywords":["Machine Learning: ML: Reinforcement learning","Machine Learning: ML: Deep reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Abel, 2020] David Abel. A Theory of Abstraction in Rein- forcement Learning. PhD thesis, Brown University, 2020.","title":"A Theory of Abstraction in Reinforcement Learning","authors":"David Abel","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Andrychowicz et al., 2017] Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems 30, 2017, pages 5048-5058, 2017.","title":"Hindsight experience replay","authors":"Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob Mcgrew, Josh Tobin, Pieter Abbeel, Wojciech Zaremba","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Auer, 2002] Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3(Nov):397-422, 2002.","title":"Using confidence bounds for exploitation-exploration trade-offs","authors":"Peter Auer","year":"2002","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bacon et al., 2017] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceed- ings of the AAAI Conference on Artificial Intelligence, vol- ume 31, 2017.","title":"The option-critic architecture","authors":"Bacon","year":"2017","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Badia et al., 2020] Adrià P. Badia, Bilal Piot, Steven Kap- turowski, Pablo Sprechmann, Alex Vitvitskyi, Zhao- han Daniel Guo, and Charles Blundell. Agent57: Out- performing the Atari human benchmark. In International Conference on Machine Learning, pages 507-517, 2020.","title":"Agent57: Outperforming the Atari human benchmark","authors":"P Adrià, Bilal Badia, Steven Piot, Pablo Kapturowski, Alex Sprechmann, Zhaohan Vitvitskyi, Charles Daniel Guo, Blundell","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bagaria and Konidaris, 2020] Akhil Bagaria and George Konidaris. Option discovery using deep skill chaining. In International Conference on Learning Representations, 2020.","title":"Option discovery using deep skill chaining","authors":"Akhil Bagaria, George Konidaris","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bagaria et al., 2021a] Akhil Bagaria, Jason K. Senthil, and George Konidaris. Skill discovery for exploration and planning using deep skill graphs. In International Confer- ence on Machine Learning, pages 521-531. PMLR, 2021.","title":"Skill discovery for exploration and planning using deep skill graphs","authors":"Bagaria","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bagaria et al., 2021b] Akhil Bagaria, Jason K. Senthil, Matthew Slivinski, and George Konidaris. Robustly learn- ing composable options in deep reinforcement learning. In IJCAI, pages 2161-2169, 2021.","title":"Robustly learning composable options in deep reinforcement learning","authors":"Bagaria","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/298"},{"position":9,"raw":"[Barreto et al., 2019] André Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Aygün, Philippe Hamel, Daniel Toyama, Shibl Mourad, David Silver, and Doina Precup. The option keyboard: Combining skills in rein- forcement learning. Advances in Neural Information Pro- cessing Systems, 32, 2019.","title":"The option keyboard: Combining skills in reinforcement learning","authors":"André Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Aygün, Philippe Hamel, Daniel Toyama, Shibl Mourad, David Silver, Doina Precup","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Bellemare et al., 2016] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.","title":"Unifying count-based exploration and intrinsic motivation","authors":"Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, Remi Munos","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Burda et al., 2019a] Yuri Burda, Harrison Edwards, Deepak Pathak, Amos J. Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study of curiosity-driven learning. In International Conference on Learning Representations, ICLR, 2019.","title":"Large-scale study of curiosity-driven learning","authors":"Burda","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Burda et al., 2019b] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random net- work distillation. In International Conference on Learning Representations, 2019.","title":"Exploration by random network distillation","authors":"Burda","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Campos et al., 2020] Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Con- ference on Machine Learning, 2020.","title":"Explore, discover and learn: Unsupervised discovery of state-covering skills","authors":"Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-I Nieto, Jordi Torres","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Chentanez et al., 2004] Nuttapong Chentanez, Andrew Barto, and Satinder Singh. Intrinsically motivated re- inforcement learning. Advances in neural information processing systems, 17, 2004.","title":"Intrinsically motivated reinforcement learning","authors":"Chentanez","year":"2004","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Colas et al., 2022] Cédric Colas, Tristan Karch, Olivier Sigaud, and Pierre-Yves Oudeyer. Autotelic agents with intrinsically motivated goal-conditioned reinforce- ment learning: a short survey. Journal of Artificial Intelli- gence Research, 74:1159-1199, 2022.","title":"Autotelic agents with intrinsically motivated goal-conditioned reinforcement learning: a short survey","authors":"Cédric Colas, Tristan Karch, Olivier Sigaud, Pierre-Yves Oudeyer","year":"2022","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Dietterich, 1998] Thomas Dietterich. The MAXQ method for hierarchical reinforcement learning. In ICML, vol- ume 98, pages 118-126, 1998.","title":"The MAXQ method for hierarchical reinforcement learning","authors":"Thomas Dietterich","year":"1998","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ecoffet et al., 2021] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580-586, 2021.","title":"First return, then explore","authors":"Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, Jeff Clune","year":"2021","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Eysenbach et al., 2018] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In Inter- national Conference on Learning Representations, 2018.","title":"Diversity is all you need: Learning skills without a reward function","authors":"Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey Levine","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Forestier et al., 2022] Sébastien Forestier, Rémy Portelas, Yoan Mollard, and Pierre-Yves Oudeyer. Intrinsically motivated goal exploration processes with automatic cur- riculum learning. Journal of Machine Learning Research (JMLR), 2022.","title":"Intrinsically motivated goal exploration processes with automatic curriculum learning","authors":"Sébastien Forestier, Rémy Portelas, Yoan Mollard, Pierre-Yves Oudeyer","year":"2022","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Fortunato et al., 2018] Meire Fortunato, Mohammad G. Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy net- works for exploration. International Conference on Learn- ing Representations, ICLR, 2018.","title":"Noisy networks for exploration","authors":"Meire Fortunato, Mohammad G Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, Shane Legg","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Gershman, 2017] Samuel J. Gershman. On the blessing of abstraction. Quarterly Journal of Experimental Psychol- ogy, 70:361 -365, 2017.","title":"On the blessing of abstraction","authors":"J Samuel, Gershman","year":"2017","venue":"Quarterly Journal of Experimental Psychology","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ghavamzadeh et al., 2010] Mohammad Ghavamzadeh, Alessandro Lazaric, Odalric Maillard, and Rémi Munos. LSTD with random projections. Advances in Neural Information Processing Systems, 23, 2010.","title":"LSTD with random projections","authors":"Mohammad Ghavamzadeh, Alessandro Lazaric, Odalric Maillard, Rémi Munos","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Gregor et al., 2016] Karol Gregor, Danilo J. Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.07507, 2016.","title":"Variational intrinsic control","authors":"Karol Gregor, Danilo J Rezende, Daan Wierstra","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Jaderberg et al., 2017] Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. 5th Inter- national Conference on Learning Representations, ICLR, 2017.","title":"Reinforcement learning with unsupervised auxiliary tasks","authors":"Jaderberg","year":"2017","venue":"5th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Kaplan and Oudeyer, 2004] Frédéric Kaplan and Pierre- Yves Oudeyer. Maximizing learning progress: an internal reward system for development. In Embodied artificial in- telligence, pages 259-270. Springer, 2004.","title":"Maximizing learning progress: an internal reward system for development","authors":"Oudeyer Kaplan, Frédéric Kaplan, Pierre-Yves Oudeyer","year":"2004","venue":"Embodied artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Kapturowski et al., 2018] Steven Kapturowski, Georg Os- trovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning repre- sentations, 2018.","title":"Recurrent experience replay in distributed reinforcement learning","authors":"Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, Will Dabney","year":"2018","venue":"International conference on learning representations","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Konidaris and Barto, 2009] George Konidaris and Andrew Barto. Skill discovery in continuous reinforcement learn- ing domains using skill chaining. Advances in neural in- formation processing systems, 22, 2009.","title":"Skill discovery in continuous reinforcement learning domains using skill chaining","authors":"Barto Konidaris, George Konidaris, Andrew Barto","year":"2009","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Konidaris, 2019] George Konidaris. On the necessity of ab- straction. Current opinion in behavioral sciences, 29:1-7, 2019.","title":"George Konidaris. On the necessity of abstraction","authors":"Konidaris","year":"2019","venue":"Current opinion in behavioral sciences","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Küttler et al., 2020] Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Ed- ward Grefenstette, and Tim Rocktäschel. The NetHack learning environment. Advances in Neural Information Processing Systems, 33:7671-7684, 2020.","title":"The NetHack learning environment","authors":"Küttler","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Lagoudakis and Parr, 2003] Michail G. Lagoudakis and Ronald Parr. Least-squares policy iteration. The Journal of Machine Learning Research, 2003.","title":"Least-squares policy iteration","authors":"Michail G Lagoudakis, Ronald Parr","year":"2003","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Lillicrap et al., 2016] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yu- val Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, ICLR, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra","year":"2016","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Lin, 1993] Long-Ji Lin. Reinforcement learning for robots using neural networks. Technical report, Carnegie-Mellon Univ Pittsburgh PA School of Computer Science, 1993.","title":"Reinforcement learning for robots using neural networks","authors":"Long-Ji Lin","year":"1993","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Liu et al., 2022] Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Prob- lems and solutions. arXiv preprint arXiv:2201.08299, 2022.","title":"Goal-conditioned reinforcement learning: Problems and solutions","authors":"Liu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2022/770"},{"position":35,"raw":"[Lopes et al., 2012] Manuel Lopes, Tobias Lang, Marc Tou- ssaint, and Pierre-Yves Oudeyer. Exploration in model- based reinforcement learning by empirically estimating learning progress. Advances in neural information pro- cessing systems, 25, 2012.","title":"Exploration in modelbased reinforcement learning by empirically estimating learning progress","authors":"Manuel Lopes, Tobias Lang, Marc Toussaint, Pierre-Yves Oudeyer","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Machado et al., 2017] Marlos C. Machado, Marc G. Belle- mare, and Michael Bowling. A Laplacian framework for option discovery in reinforcement learning. In Inter- national Conference on Machine Learning, pages 2295- 2304. PMLR, 2017.","title":"A Laplacian framework for option discovery in reinforcement learning","authors":"C Marlos, Marc G Machado, Michael Bellemare, Bowling","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Mohamed and Rezende, 2015] Shakir Mohamed and Danilo Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. Ad- vances in neural information processing systems, 28, 2015.","title":"Variational information maximisation for intrinsically motivated reinforcement learning","authors":"Shakir Mohamed, Danilo Rezende","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Osband et al., 2016] Ian Osband, Charles Blundell, Alexan- der Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. Advances in neural information pro- cessing systems, 29, 2016.","title":"Deep exploration via bootstrapped DQN","authors":"Ian Osband, Charles Blundell, Alexander Pritzel, Benjamin Van, Roy","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Osband et al., 2018] Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforce- ment learning. Advances in Neural Information Process- ing Systems, 31, 2018.","title":"Randomized prior functions for deep reinforcement learning","authors":"Ian Osband, John Aslanides, Albin Cassirer","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Ostrovski et al., 2021] Georg Ostrovski, Pablo Samuel Cas- tro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. In Advances in Neural In- formation Processing Systems, volume 34, pages 23283- 23295, 2021.","title":"Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning","authors":"Ostrovski","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Pathak et al., 2017] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, 2017.","title":"Curiosity-driven exploration by self-supervised prediction","authors":"Deepak Pathak, Pulkit Agrawal, Alexei A Efros, Trevor Darrell","year":"2017","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Pîslar et al., 2022] Miruna Pîslar, David Szepesvari, Georg Ostrovski, Diana Borsa, and Tom Schaul. When should agents explore? In International Conference on Learning Representations (ICLR), 2022.","title":"When should agents explore?","authors":"Miruna Pîslar, David Szepesvari, Georg Ostrovski, Diana Borsa, Tom Schaul","year":"2022","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Pitis et al., 2020] Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain ex- ploration for long horizon multi-goal reinforcement learn- ing. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, volume 119, pages 7750-7761, 2020.","title":"Maximum entropy gain exploration for long horizon multi-goal reinforcement learning","authors":"Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, Jimmy Ba","year":"2020","venue":"Proceedings of the 37th International Conference on Machine Learning, ICML 2020","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Plappert et al., 2018] Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. International Conference on Learning Representations, ICLR, 2018.","title":"Parameter space noise for exploration","authors":"Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, Marcin Andrychowicz","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Pong et al., 2019] Vitchyr H. Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew- fit: State-covering self-supervised reinforcement learning. Proceedings of the 37th International Conference on Ma- chine Learning, ICML, 2019.","title":"Skewfit: State-covering self-supervised reinforcement learning","authors":"H Vitchyr, Murtaza Pong, Steven Dalal, Ashvin Lin, Shikhar Nair, Sergey Bahl, Levine","year":"2019","venue":"Proceedings of the 37th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Raileanu and Rocktäschel, 2020] Roberta Raileanu and Tim Rocktäschel. RIDE: Rewarding impact-driven exploration for procedurally-generated environments. In International Conference on Learning Representations, 2020.","title":"Roberta Raileanu and Tim Rocktäschel. RIDE: Rewarding impact-driven exploration for procedurally-generated environments","authors":"Rocktäschel Raileanu","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Randløv and Alstrøm, 1998] Jette Randløv and Preben Al- strøm. Learning to drive a bicycle using reinforcement learning and shaping. In ICML, volume 98, pages 463- 471. Citeseer, 1998.","title":"Learning to drive a bicycle using reinforcement learning and shaping","authors":"Alstrøm Randløv","year":"1998","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Ring, 1994] Mark B. Ring. Continual learning in reinforce- ment environments. PhD thesis, University of Texas at Austin, Texas, 1994.","title":"Continual learning in reinforcement environments","authors":"B Mark, Ring","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Samvelyan et al., 2021] Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holder, Minqi Jiang, Eric Ham- bro, Fabio Petroni, Heinrich Küttler, Edward Grefenstette, and Tim Rocktäschel. MiniHack the planet: A sandbox for open-ended reinforcement learning research. In NeurIPS Datasets and Benchmarks Track, 2021.","title":"MiniHack the planet: A sandbox for open-ended reinforcement learning research","authors":"Robert Mikayel Samvelyan, Vitaly Kirk, Jack Kurin, Minqi Parker-Holder, Eric Jiang, Fabio Hambro, Heinrich Petroni, Edward Küttler, Tim Grefenstette, Rocktäschel","year":"2021","venue":"NeurIPS Datasets and Benchmarks Track","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Schaul and Ring, 2013] Tom Schaul and Mark B. Ring. Better generalization with forecasts. In IJCAI, pages 1656-1662, 2013.","title":"Better generalization with forecasts","authors":"Ring Schaul, Tom Schaul, Mark B Ring","year":"2013","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2013/246"},{"position":51,"raw":"[Schaul et al., 2015] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function ap- proximators. In Proceedings of the 32nd International Conference on Machine Learning, ICML, volume 37, pages 1312-1320, 2015.","title":"Universal value function approximators","authors":"Schaul","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning, ICML","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Schaul et al., 2019] Tom Schaul, Diana Borsa, Joseph Mo- dayil, and Razvan Pascanu. Ray interference: a source of plateaus in deep reinforcement learning. arXiv preprint arXiv:1904.11455, 2019.","title":"Ray interference: a source of plateaus in deep reinforcement learning","authors":"Schaul","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Schmidhuber, 1991] Jürgen Schmidhuber. Curious model- building control systems. In Proc. international joint con- ference on neural networks, pages 1458-1463, 1991.","title":"Curious modelbuilding control systems","authors":"Jürgen Schmidhuber","year":"1991","venue":"Proc. international joint conference on neural networks","doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Schmidhuber, 2010] Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE transactions on autonomous mental development, 2(3):230-247, 2010.","title":"Formal theory of creativity, fun, and intrinsic motivation (1990-2010)","authors":"Jürgen Schmidhuber","year":"2010","venue":"IEEE transactions on autonomous mental development","doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Strehl and Littman, 2008] Alexander L. Strehl and Michael L. Littman. An analysis of model-based in- terval estimation for Markov decision processes. Journal of Computer and System Sciences, 74(8):1309-1331, 2008.","title":"An analysis of model-based interval estimation for Markov decision processes","authors":"Alexander L Strehl, Michael L Littman","year":"2008","venue":"Journal of Computer and System Sciences","doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Sutton and Barto, 2018] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"Richard S Sutton, Andrew G Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":57,"raw":"[Sutton et al., 1999] Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, pages 181-211, 1999.","title":"Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning","authors":"Richard S Sutton, Doina Precup, Satinder Singh","year":"1999","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":58,"raw":"[Sutton et al., 2011] Richard S. Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M. Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time ar- chitecture for learning knowledge from unsupervised sen- sorimotor interaction. In The 10th International Con- ference on Autonomous Agents and Multiagent Systems- Volume 2, pages 761-768, 2011.","title":"Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction","authors":"Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, Doina Precup","year":"2011","venue":"The 10th International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":59,"raw":"[Sutton et al., 2022a] Richard S. Sutton, Michael H. Bowl- ing, and Patrick M. Pilarski. The Alberta plan for AI re- search. arXiv preprint arXiv:2208.11173, 2022.","title":"The Alberta plan for AI research","authors":"Sutton","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":60,"raw":"[Sutton et al., 2022b] Richard S. Sutton, Marlos C. Machado, G. Zacharias Holland, David Szepesvari Timbers, Finbarr Timbers, Brian Tanner, and Adam White. Reward-respecting subtasks for model-based reinforcement learning. arXiv preprint arXiv:2202.03466, 2022.","title":"Reward-respecting subtasks for model-based reinforcement learning","authors":"Sutton","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":61,"raw":"[Taiga et al., 2020] Adrien Ali Taiga, William Fedus, Mar- los C. Machado, Aaron Courville, and Marc G. Bellemare. On bonus based exploration methods in the arcade learn- ing environment. In International Conference on Learning Representations, 2020.","title":"On bonus based exploration methods in the arcade learning environment","authors":"Adrien Ali Taiga, William Fedus, Marlos C Machado, Aaron Courville, Marc G Bellemare","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":62,"raw":"[Tasse et al., 2022] Geraud N. Tasse, Benjamin Rosman, and Steven James. World value functions: Knowledge rep- resentation for learning and planning. arXiv preprint arXiv:2206.11940, 2022.","title":"World value functions: Knowledge representation for learning and planning","authors":"N Geraud, Benjamin Tasse, Steven Rosman, James","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":63,"raw":"[Teikari, 2019] Arvi Teikari. Baba is you. Game [PC], Hempuli Oy, Finland, 2019.","title":"Baba is you. Game","authors":"Arvi Teikari","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":64,"raw":"[Veeriah et al., 2019] Vivek Veeriah, Matteo Hessel, Zhong- wen Xu, Janarthanan Rajendran, Richard L. Lewis, Jun- hyuk Oh, Hado van Hasselt, David Silver, and Satinder Singh. Discovery of useful questions as auxiliary tasks. Advances in Neural Information Processing Systems, 32, 2019.","title":"Discovery of useful questions as auxiliary tasks","authors":"Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Janarthanan Rajendran, Richard L Lewis, Junhyuk Oh, David Hado Van Hasselt, Satinder Silver, Singh","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":65,"raw":"[Veeriah et al., 2021] Vivek Veeriah, Tom Zahavy, Matteo Hessel, Zhongwen Xu, Junhyuk Oh, Iurii Kemaev, Hado van Hasselt, David Silver, and Satinder Singh. Discovery of options via meta-learned subgoals. Advances in Neural Information Processing Systems, 34:29861-29873, 2021.","title":"Discovery of options via meta-learned subgoals","authors":"Vivek Veeriah, Tom Zahavy, Matteo Hessel, Zhongwen Xu, Junhyuk Oh, Iurii Kemaev, David Hado Van Hasselt, Satinder Silver, Singh","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}