{"type":"ScholarlyArticle","title":"Autonomous Exploration for Navigating in MDPs Using Blackbox RL Algorithms","authors":[{"name":"Pratik Gajane","slug":"pratik-gajane","given":"Pratik","family":"Gajane","affiliation":"Eindhoven University of Technology"},{"name":"Peter Auer","slug":"peter-auer","given":"Peter","family":"Auer","affiliation":"Montanuniversität Leoben"},{"name":"Ronald Ortner","slug":"ronald-ortner","given":"Ronald","family":"Ortner","affiliation":"Montanuniversität Leoben"}],"abstract":"We consider the problem of navigating in a Markov decision process where extrinsic rewards are either absent or ignored. In this setting, the objective is to learn policies to reach all the states that are reachable within a given number of steps (in expectation) from a starting state. We introduce a novel meta-algorithm which can use any online reinforcement learning algorithm (with appropriate regret guarantees) as a black-box. Our algorithm demonstrates a method for transforming the output of online algorithms to a batch setting. We prove an upper bound on the sample complexity of our algorithm in terms of the regret bound of the used black-box RL algorithm. Furthermore, we provide experimental results to validate the effectiveness of our algorithm and correctness of our theoretical results.","doi":"10.24963/ijcai.2023/413","url":"https://ijcai.org/proceedings/2023/413","pdf":"https://ijcai.org/proceedings/2023/0413.pdf","markdown":"https://ijcai.org/proceedings/2023/413.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/413","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":413,"pages":{"first":3714,"last":3722},"keywords":["Machine Learning: ML: Reinforcement learning","Machine Learning: ML: Learning theory"],"videos":[],"references":[{"position":1,"raw":"[Achiam and Sastry, 2017] Joshua Achiam and Shankar Sas- try. Surprise-based intrinsic motivation for deep reinforce- ment learning. CoRR, abs/1703.01732, 2017.","title":"Joshua Achiam and Shankar Sastry. Surprise-based intrinsic motivation for deep reinforcement learning","authors":"Sastry Achiam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agarwal et al., 2020] Alekh Agarwal, Mikael Henaff, Sham Kakade, and Wen Sun. Pc-pg: Policy cover directed explo- ration for provable policy gradient learning. In Advances in Neural Information Processing Systems, volume 33, 2020.","title":"Pc-pg: Policy cover directed exploration for provable policy gradient learning","authors":"Alekh Agarwal, Mikael Henaff, Sham Kakade, Wen Sun","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Azar et al., 2019] Mohammad Gheshlaghi Azar, Bilal Piot, Bernardo A. Pires, Jean-Bastien Grill, Florent Altché, and Rémi Munos. World discovery models. CoRR, abs/1902.07685, 2019.","title":"World discovery models","authors":"Mohammad Gheshlaghi Azar, Bilal Piot, Bernardo A Pires, Jean-Bastien Grill, Florent Altché, Rémi Munos","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Azuma, 1967] Kazuoki Azuma. Weighted sums of cer- tain dependent random variables. Tohoku Math. J. (2), 19(3):357-367, 1967.","title":"Weighted sums of certain dependent random variables","authors":"Kazuoki Azuma","year":"1967","venue":"Tohoku Math. J","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Baranes and Oudeyer, 2009] A. Baranes and P.-Y. Oudeyer. R-IAC: Robust intrinsically motivated exploration and ac- tive learning. IEEE Transactions on Autonomous Mental Development, 1:155-169, 2009.","title":"R-IAC: Robust intrinsically motivated exploration and active learning","authors":"A Baranes, P.-Y Oudeyer","year":"2009","venue":"IEEE Transactions on Autonomous Mental Development","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bartlett and Tewari, 2009] Peter L. Bartlett and Ambuj Tewari. Regal: A regularization based algorithm for reinforcement learning in weakly communicating mdps. In Proceedings of the Twenty-Fifth Conference on Un- certainty in Artificial Intelligence, UAI '09, page 35-42, 2009.","title":"Regal: A regularization based algorithm for reinforcement learning in weakly communicating mdps","authors":"Peter L Bartlett, Ambuj Tewari","year":"2009","venue":"Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI '09","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Burda et al., 2019] Yuri Burda, Harrison Edwards, Deepak Pathak, Amos J. Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study of curiosity-driven learning. In ICLR, 2019.","title":"Large-scale study of curiosity-driven learning","authors":"Yuri Burda, Harrison Edwards, Deepak Pathak, Amos J Storkey, Trevor Darrell, Alexei A Efros","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cai et al., 2022] Haoyuan Cai, Tengyu Ma, and Simon Du. Near-optimal algorithms for autonomous exploration and multi-goal stochastic shortest path. In Kamalika Chaud- huri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 2434-2456. PMLR, 17-23 Jul 2022.","title":"Near-optimal algorithms for autonomous exploration and multi-goal stochastic shortest path","authors":"Cai","year":"2022","venue":"Proceedings of the 39th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cesa-Bianchi et al., 1997] Nicolò Cesa-Bianchi, Yoav Fre- und, David Haussler, David P. Helmbold, Robert E. Schapire, and Manfred K. Warmuth. How to use expert advice. J. ACM, page 427-485, May 1997.","title":"How to use expert advice","authors":"Nicolò Cesa-Bianchi, Yoav Freund, David Haussler, David P Helmbold, Robert E Schapire, Manfred K Warmuth","year":"1997","venue":"J. ACM","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Cesa-Bianchi et al., 2006] Nicolò Cesa-Bianchi, Alex Con- coni, and Claudio Gentile. On the generalization ability of on-line learning algorithms. IEEE Trans. Inf. Theor., 50(9):2050-2057, September 2006.","title":"On the generalization ability of on-line learning algorithms","authors":"Cesa-Bianchi","year":"2006","venue":"IEEE Trans. Inf. Theor","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ecoffet et al., 2019] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. CoRR, abs/1901.10995, 2019.","title":"Go-explore: a new approach for hard-exploration problems","authors":"Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, Jeff Clune","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ecoffet et al., 2020] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. First return then explore. CoRR, abs/2004.12919, 2020.","title":"First return then explore","authors":"Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, Jeff Clune","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Florensa et al., 2018] Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal gener- ation for reinforcement learning agents. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1515-1528. PMLR, 10-15 Jul 2018.","title":"Automatic goal generation for reinforcement learning agents","authors":"Carlos Florensa, David Held, Xinyang Geng, Pieter Abbeel","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Freund and Schapire, 1999] Yoav Freund and Robert E. Schapire. Large margin classification using the perceptron algorithm. Mach. Learn., page 277-296, 1999.","title":"Large margin classification using the perceptron algorithm","authors":"Yoav Freund, Robert E Schapire","year":"1999","venue":"Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Fruit et al., 2020] Ronan Fruit, Matteo Pirotta, and Alessan- dro Lazaric. Improved analysis of UCRL2 with empirical bernstein inequality. CoRR, abs/2007.05456, 2020.","title":"Improved analysis of UCRL2 with empirical bernstein inequality","authors":"Ronan Fruit, Matteo Pirotta, Alessandro Lazaric","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Gajane et al., 2019] Pratik Gajane, Ronald Ortner, Peter Auer, and Csaba Szepesvári. Autonomous explo- ration for navigating in non-stationary cmps. CoRR, abs/1910.08446, 2019.","title":"Autonomous exploration for navigating in non-stationary cmps","authors":"Pratik Gajane, Ronald Ortner, Peter Auer, Csaba Szepesvári","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Gottlieb et al., 2013] Jacqueline Gottlieb, Pierre-Yves Oudeyer, Manuel Lopes, and Adrien Baranes. Information-seeking, curiosity, and attention: com- putational and neural mechanisms. Trends in cognitive sciences, 17(11):585-593, 2013.","title":"Information-seeking, curiosity, and attention: computational and neural mechanisms","authors":"Jacqueline Gottlieb, Pierre-Yves Oudeyer, Manuel Lopes, Adrien Baranes","year":"2013","venue":"Trends in cognitive sciences","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Haber et al., 2018] Nick Haber, Damian Mrowca, Stephanie Wang, Li F Fei-Fei, and Daniel L Yamins. Learning to play with intrinsically-motivated, self-aware agents. In Advances in Neural Information Processing Systems, pages 8388-8399, 2018.","title":"Learning to play with intrinsically-motivated, self-aware agents","authors":"Nick Haber, Damian Mrowca, Stephanie Wang, Daniel L Li F Fei-Fei, Yamins","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Hazan et al., 2019] Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, ICML, volume 97, pages 2681- 2691, 2019.","title":"Provably efficient maximum entropy exploration","authors":"Elad Hazan, Sham Kakade, Karan Singh, Abby Van Soest","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Helmbold and Warmuth, 1995] David P. Helmbold and Manfred K. Warmuth. On weak learning. J. Comput. Syst. Sci., page 551-573, June 1995.","title":"On weak learning","authors":"David P Helmbold, Manfred K Warmuth","year":"1995","venue":"J. Comput. Syst. Sci","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Hoeffding, 1963] Wassily Hoeffding. Probability inequali- ties for sums of bounded random variables. Journal of the American Statistical Association, 58(301):13-30, 1963.","title":"Wassily Hoeffding. Probability inequalities for sums of bounded random variables","authors":"Hoeffding","year":"1963","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Houthooft et al., 2016] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Variational information maximizing exploration. In NIPS 2016 Deep Learning Symposium, 2016.","title":"Variational information maximizing exploration","authors":"Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, Pieter Abbeel","year":"2016","venue":"NIPS 2016 Deep Learning Symposium","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Jaksch et al., 2010] Thomas Jaksch, Ronald Ortner, and Pe- ter Auer. Near-optimal regret bounds for reinforce- ment learning. Journal of Machine Learning Research, 11(51):1563-1600, 2010.","title":"Near-optimal regret bounds for reinforcement learning","authors":"Thomas Jaksch, Ronald Ortner, Peter Auer","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Jin et al., 2020] Chi Jin, Akshay Krishnamurthy, Max Sim- chowitz, and Tiancheng Yu. Reward-free exploration for reinforcement learning. In Proceedings of the 37th Inter- national Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4870- 4879. PMLR, 2020.","title":"Reward-free exploration for reinforcement learning","authors":"Jin","year":"2020","venue":"Proceedings of the 37th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Kaplan and Oudeyer, 2003] Frédéric Kaplan and Pierre- Yves Oudeyer. Motivational principles for visual know- how development. Lund University Cognitive Studies, 101:73-80, 2003.","title":"Motivational principles for visual knowhow development","authors":"Oudeyer Kaplan, Frédéric Kaplan, Pierre-Yves Oudeyer","year":"2003","venue":"Lund University Cognitive Studies","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Lim and Auer, 2012] Shiau Hong Lim and Peter Auer. Au- tonomous exploration for navigating in mdps. In Proceed- ings of the 25th Annual Conference on Learning Theory, volume 23 of Proceedings of Machine Learning Research, pages 40.1-40.24, 2012.","title":"Autonomous exploration for navigating in mdps","authors":"Hong Shiau, Peter Lim, Auer","year":"2012","venue":"Proceedings of the 25th Annual Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Littlestone, 1989] Nick Littlestone. From on-line to batch learning. In Proceedings of the Second Annual Work- shop on Computational Learning Theory, COLT '89, page 269-284, 1989.","title":"From on-line to batch learning","authors":"Nick Littlestone","year":"1989","venue":"Proceedings of the Second Annual Workshop on Computational Learning Theory, COLT '89","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Lopes et al., 2012] Manuel Lopes, Tobias Lang, Marc Tou- ssaint, and Pierre-Yves Oudeyer. Exploration in model- based reinforcement learning by empirically estimating learning progress. In Advances in neural information pro- cessing systems, pages 206-214, 2012.","title":"Exploration in modelbased reinforcement learning by empirically estimating learning progress","authors":"Manuel Lopes, Tobias Lang, Marc Toussaint, Pierre-Yves Oudeyer","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Ostrovski et al., 2017] Georg Ostrovski, Marc G Bellemare, Aäron van den Oord, and Rémi Munos. Count-based ex- ploration with neural density models. In Proceedings of the 34th International Conference on Machine Learning, pages 2721-2730, 2017.","title":"Count-based exploration with neural density models","authors":"Georg Ostrovski, Aäron Marc G Bellemare, Rémi Van Den Oord, Munos","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Oudeyer and Kaplan, 2007] Pierre-Yves Oudeyer and Frédéric Kaplan. What is intrinsic motivation? a typology of computational approaches. Frontiers in neurorobotics, 1, 2007.","title":"What is intrinsic motivation? a typology of computational approaches","authors":"Kaplan ; Pierre-Yves Oudeyer, Frédéric Oudeyer, Kaplan","year":"2007","venue":"Frontiers in neurorobotics","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Pathak et al., 2017] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 16-17, 2017.","title":"Curiosity-driven exploration by self-supervised prediction","authors":"Deepak Pathak, Pulkit Agrawal, Alexei A Efros, Trevor Darrell","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Schmidhuber, 2010] Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). Autonomous Mental Development, IEEE Transactions on, 2:230-247, 2010.","title":"Formal theory of creativity, fun, and intrinsic motivation (1990-2010","authors":"Jürgen Schmidhuber","year":"2010","venue":"Autonomous Mental Development","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Shalev-Shwartz, 2012] Shai Shalev-Shwartz. Online learn- ing and online convex optimization. Found. Trends Mach. Learn., 4(2):107-194, February 2012.","title":"Online learning and online convex optimization","authors":"Shai Shalev-Shwartz","year":"2012","venue":"Found. Trends Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Singh et al., 2004] Satinder P. Singh, Andrew G. Barto, and Nuttapong Chentanez. Intrinsically motivated reinforce- ment learning. In NIPS, 2004.","title":"Intrinsically motivated reinforcement learning","authors":"P Satinder, Andrew G Singh, Nuttapong Barto, Chentanez","year":"2004","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Singh et al., 2010] Satinder P. Singh, Richard L. Lewis, An- drew G. Barto, and Jonathan Sorg. Intrinsically moti- vated reinforcement learning: An evolutionary perspec- tive. IEEE T. Autonomous Mental Development, 2:70-82, 2010.","title":"Intrinsically motivated reinforcement learning: An evolutionary perspective","authors":"P Satinder, Richard L Singh, Andrew G Lewis, Jonathan Barto, Sorg","year":"2010","venue":"IEEE T. Autonomous Mental Development","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Tarbouriech et al., 2019] Jean Tarbouriech, Evrard Garcelon, Michal Valko, Matteo Pirotta, and Alessan- dro Lazaric. No-regret exploration in goal-oriented reinforcement learning. CoRR, abs/1912.03517, 2019.","title":"No-regret exploration in goal-oriented reinforcement learning","authors":"Jean Tarbouriech, Evrard Garcelon, Michal Valko, Matteo Pirotta, Alessandro Lazaric","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Tarbouriech et al., 2020] Jean Tarbouriech, Matteo Pirotta, Michal Valko, and Alessandro Lazaric. Improved sam- ple complexity for incremental autonomous exploration in mdps. In Advances in Neural Information Processing Sys- tems 33 pre-proceedings (NeurIPS 2020), 2020.","title":"Improved sample complexity for incremental autonomous exploration in mdps","authors":"Tarbouriech","year":"2020","venue":"Advances in Neural Information Processing Systems 33 pre-proceedings","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Weng et al., 2001] Juyang Weng, James McClelland, Alex Pentland, Olaf Sporns, Ida Stockman, Mriganka Sur, and Esther Thelen. Autonomous mental development by robots and animals. Science, 291(5504):599-600, 2001.","title":"Autonomous mental development by robots and animals","authors":"Weng","year":"2001","venue":"Science","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}