{"type":"ScholarlyArticle","title":"Goal-Conditioned Reinforcement Learning: Problems and Solutions","authors":[{"name":"Minghuan Liu","slug":"minghuan-liu","given":"Minghuan","family":"Liu","affiliation":"Shanghai Jiao Tong University"},{"name":"Menghui Zhu","slug":"menghui-zhu","given":"Menghui","family":"Zhu","affiliation":"Shanghai Jiao Tong University"},{"name":"Weinan Zhang","slug":"weinan-zhang","given":"Weinan","family":"Zhang","affiliation":"Shanghai Jiao Tong University"}],"abstract":"Goal-conditioned reinforcement learning (GCRL), related to a set of complex RL problems, trains an agent to achieve different goals under particular scenarios. Compared to the standard RL solutions that learn a policy solely depending on the states or observations, GCRL additionally requires the agent to make decisions according to different goals.\r\nIn this survey, we provide a comprehensive overview of the challenges and algorithms for GCRL. Firstly, we answer what the basic problems are studied in this field. Then, we explain how goals are represented and present how existing solutions are designed from different points of view. Finally, we make the conclusion and discuss potential future prospects that recent researches focus on.","doi":"10.24963/ijcai.2022/770","url":"https://ijcai.org/proceedings/2022/770","pdf":"https://ijcai.org/proceedings/2022/0770.pdf","markdown":"https://ijcai.org/proceedings/2022/770.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/770","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Survey Track","track":"Survey Track","number":770,"pages":{"first":5502,"last":5511},"keywords":["Survey Track: Machine Learning","Survey Track: Robotics","Survey Track: Planning and Scheduling"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987280","length_sec":85,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985423","length_sec":727,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Ahn et al., 2022] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Haus- man, Alex Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.","title":"Do as i can, not as i say: Grounding language in robotic affordances","authors":"Anthony Michael Ahn, Noah Brohan, Yevgen Brown, Omar Chebotar, Byron Cortes, Chelsea David, Keerthana Finn, Karol Gopalakrishnan, Alex Hausman, Herzog","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Akakzia et al., 2021] Ahmed Akakzia, Cédric Colas, Pierre-Yves Oudeyer, Mohamed Chetouani, and Olivier Sigaud. Grounding language to autonomously-acquired skills via goal generation. ICLR, 2021.","title":"Grounding language to autonomously-acquired skills via goal generation","authors":"Ahmed Akakzia, Cédric Colas, Pierre-Yves Oudeyer, Mohamed Chetouani, Olivier Sigaud","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Andrychowicz et al., 2017] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. NeurIPS, 2017.","title":"Hindsight experience replay","authors":"Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob Mcgrew, Josh Tobin, Pieter Abbeel, Wojciech Zaremba","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bai et al., 2019] Chenjia Bai, Peng Liu, Wei Zhao, and Xi- anglong Tang. Guided goal generation for hindsight multi- goal reinforcement learning. Neurocomputing, 2019.","title":"Guided goal generation for hindsight multigoal reinforcement learning","authors":"Bai","year":"2019","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Beattie et al., 2016] Charles Beattie, Joel Z Leibo, et al. Deepmind lab. CoRR abs/1612.03801, 2016.","title":"Deepmind lab","authors":"Charles Beattie, Joel Z Leibo","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brafman et al., 2018] Ronen Brafman, Giuseppe De Gia- como, and Fabio Patrizi. Ltlf/ldlf non-markovian rewards. In Proceedings of the AAAI Conference on Artificial Intel- ligence, volume 32, 2018.","title":"Ltlf/ldlf non-markovian rewards","authors":"Ronen Brafman, Giuseppe De, Giacomo, Fabio Patrizi","year":"2018","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, et al. Openai gym. CoRR abs/1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Campero et al., 2021] Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua B Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with amigo: Adversar- ially motivated intrinsic goals. ICLR, 2021.","title":"Learning with amigo: Adversarially motivated intrinsic goals","authors":"Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua B Tenenbaum, Tim Rocktäschel, Edward Grefenstette","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chan et al., 2019] Harris Chan, Yuhuai Wu, Jamie Kiros, Sanja Fidler, and Jimmy Ba. Actrce: Augmenting expe- rience via teacher's advice for multi-goal reinforcement learning. CoRR abs/1902.04546, 2019.","title":"Actrce: Augmenting experience via teacher's advice for multi-goal reinforcement learning","authors":"Harris Chan, Yuhuai Wu, Jamie Kiros, Sanja Fidler, Jimmy Ba","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chane-Sane et al., 2021] Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforce- ment learning with imagined subgoals. In ICML, 2021.","title":"Goal-conditioned reinforcement learning with imagined subgoals","authors":"Chane-Sane","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Charlesworth and Montana, 2020] Henry Charlesworth and Giovanni Montana. Plangan: Model-based planning with sparse rewards and multiple goals. NeurIPS, 2020.","title":"Plangan: Model-based planning with sparse rewards and multiple goals","authors":"Montana Charlesworth, Henry Charlesworth, Giovanni Montana","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Chebotar et al., 2021] Yevgen Chebotar, Karol Hausman, et al. Actionable models: Unsupervised offline reinforce- ment learning of robotic skills. ICML, 2021.","title":"Actionable models: Unsupervised offline reinforcement learning of robotic skills","authors":"Yevgen Chebotar, Karol Hausman","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Chen et al., 2021] Lili Chen, Kevin Lu, Aravind Ra- jeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. De- cision transformer: Reinforcement learning via sequence modeling. CoRR abs/2106.01345, 2021.","title":"Decision transformer: Reinforcement learning via sequence modeling","authors":"Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, Igor Mordatch","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Chitnis et al., 2021] Rohan Chitnis, Tom Silver, Joshua Tenenbaum, Leslie Pack Kaelbling, and Tomas Lozano- Perez. Glib: Efficient exploration for relational model- based reinforcement learning via goal-literal babbling. AAAI, 2021.","title":"Glib: Efficient exploration for relational modelbased reinforcement learning via goal-literal babbling","authors":"Rohan Chitnis, Tom Silver, Joshua Tenenbaum, Leslie Pack Kaelbling, Tomas Lozano-Perez","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Colas et al., 2020a] Cédric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Clément Moulin-Frier, Pe- ter Ford Dominey, and Pierre-Yves Oudeyer. Language as a cognitive tool to imagine goals in curiosity-driven ex- ploration. NeurIPS, 2020.","title":"Language as a cognitive tool to imagine goals in curiosity-driven exploration","authors":"Colas","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Colas et al., 2020b] Cédric Colas, Tristan Karch, Olivier Sigaud, and Pierre-Yves Oudeyer. Intrinsically motivated goal-conditioned reinforcement learning: a short survey. arXiv preprint arXiv:2012.09830, 2020.","title":"Intrinsically motivated goal-conditioned reinforcement learning: a short survey","authors":"Colas","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Colas et al., 2021] Cédric Colas, Boris Hejblum, Sébastien Rouillon, et al. Epidemioptim: A toolbox for the optimiza- tion of control policies in epidemiological models. JAIR, 2021.","title":"Epidemioptim: A toolbox for the optimization of control policies in epidemiological models","authors":"Cédric Colas, Boris Hejblum, Sébastien Rouillon","year":"2021","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Durugkar et al., 2021] Ishan Durugkar, Mauricio Tec, Scott Niekum, and Peter Stone. Adversarial intrinsic motivation for reinforcement learning. CoRR abs/2105.13345, 2021.","title":"Adversarial intrinsic motivation for reinforcement learning","authors":"Ishan Durugkar, Mauricio Tec, Scott Niekum, Peter Stone","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Eysenbach et al., 2019a] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. ICLR, 2019.","title":"Diversity is all you need: Learning skills without a reward function","authors":"Eysenbach","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Eysenbach et al., 2019b] Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. NeurIPS, 2019.","title":"Search on the replay buffer: Bridging planning and reinforcement learning","authors":"Eysenbach","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Eysenbach et al., 2020] Benjamin Eysenbach, Xinyang Geng, Sergey Levine, and Ruslan Salakhutdinov. Rewrit- ing history with inverse rl: Hindsight inference for policy improvement. NeurIPS, 2020.","title":"Rewriting history with inverse rl: Hindsight inference for policy improvement","authors":"Benjamin Eysenbach, Xinyang Geng, Sergey Levine, Ruslan Salakhutdinov","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Fang et al., 2019] Meng Fang, Tianyi Zhou, Yali Du, Lei Han, and Zhengyou Zhang. Curriculum-guided hindsight experience replay. NeurIPS, 2019.","title":"Curriculum-guided hindsight experience replay","authors":"Meng Fang, Tianyi Zhou, Yali Du, Lei Han, Zhengyou Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Florensa et al., 2018] Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. ICML, 2018. [Forestier and Oudeyer, 2016]","title":"Automatic goal generation for reinforcement learning agents","authors":"Carlos Florensa, David Held, Xinyang Geng, Pieter Abbeel","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"Sébastien Forestier and Pierre-Yves Oudeyer. Modular active curiosity-driven discovery of tool use. IROS, 2016.","title":"Modular active curiosity-driven discovery of tool use","authors":"Sébastien Forestier, Pierre-Yves Oudeyer","year":"2016","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ghosh et al., 2021] Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated super- vised learning. ICLR, 2021.","title":"Learning to reach goals via iterated supervised learning","authors":"Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, Sergey Levine","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Hartikainen et al., 2019] Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical distance learning for semi-supervised and unsupervised skill discovery. ICLR, 2019.","title":"Dynamical distance learning for semi-supervised and unsupervised skill discovery","authors":"Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, Sergey Levine","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Jiang et al., 2019] Yiding Jiang, Shixiang Gu, Kevin Mur- phy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning. NeurIPS, 2019.","title":"Language as an abstraction for hierarchical deep reinforcement learning","authors":"Jiang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Johnson et al., 2016] Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentation. 2016.","title":"The malmo platform for artificial intelligence experimentation","authors":"Matthew Johnson, Katja Hofmann, Tim Hutton, David Bignell","year":"2016","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2016/643"},{"position":29,"raw":"[Khazatsky et al., 2021] Alexander Khazatsky, Ashvin Nair, Daniel Jing, and Sergey Levine. What can i do here? learn- ing new skills by imagining visual affordances. ICRA, 2021.","title":"What can i do here? learning new skills by imagining visual affordances","authors":"Alexander Khazatsky, Ashvin Nair, Daniel Jing, Sergey Levine","year":"2021","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Kuang et al., 2020] Yingyi Kuang, Abraham Itzhak Wein- berg, George Vogiatzis, and Diego R Faria. Goal density- based hindsight experience prioritization for multi-goal robot manipulation reinforcement learning. RO-MAN, 2020.","title":"Goal densitybased hindsight experience prioritization for multi-goal robot manipulation reinforcement learning","authors":"Yingyi Kuang, Abraham Itzhak Weinberg, George Vogiatzis, Diego R Faria","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Kumar et al., 2019] Aviral Kumar, Xue Bin Peng, and Sergey Levine. Reward-conditioned policies. CoRR abs/1912.13465, 2019.","title":"Reward-conditioned policies","authors":"Aviral Kumar, Xue Bin Peng, Sergey Levine","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Lai et al., 2020] Yaqing Lai, Wufan Wang, Yunjie Yang, Ji- hong Zhu, and Minchi Kuang. Hindsight planner. AAMAS, 2020.","title":"Hindsight planner. AAMAS","authors":"Yaqing Lai, Wufan Wang, Yunjie Yang, Jihong Zhu, Minchi Kuang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Lee et al., 2020] Lisa Lee, Benjamin Eysenbach, Ruslan Salakhutdinov, Shixiang Shane Gu, and Chelsea Finn. Weakly-supervised reinforcement learning for control- lable behavior. NeurIPS, 2020.","title":"Weakly-supervised reinforcement learning for controllable behavior","authors":"Lisa Lee, Benjamin Eysenbach, Ruslan Salakhutdinov, Shixiang Shane Gu, Chelsea Finn","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Luketina et al., 2019] Jelena Luketina, Nantas Nardelli, et al. A survey of reinforcement learning informed by nat- ural language. IJCAI, 2019.","title":"A survey of reinforcement learning informed by natural language","authors":"Luketina","year":"2019","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2019/880"},{"position":35,"raw":"[McCarthy and Redmond, 2021] Robert McCarthy and Stephen J Redmond. Imaginary hindsight experience replay: Curious model-based learning for sparse reward tasks. CoRR abs/2110.02414, 2021.","title":"Imaginary hindsight experience replay: Curious model-based learning for sparse reward tasks","authors":"Robert Mccarthy, Stephen J Redmond","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Mendonca et al., 2021] Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Discovering and achieving goals via world models. CoRR abs/2110.09514, 2021.","title":"Discovering and achieving goals via world models","authors":"Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, Deepak Pathak","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Nair and Finn, 2020] Suraj Nair and Chelsea Finn. Hierar- chical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation. ICLR, 2020.","title":"Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation","authors":"Finn Nair, Suraj Nair, Chelsea Finn","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Nair et al., 2018a] Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. ICRA, 2018.","title":"Overcoming exploration in reinforcement learning with demonstrations","authors":"Nair","year":"2018","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Nair et al., 2018b] Ashvin V Nair, Vitchyr Pong, Mur- taza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. NeurIPS, 2018.","title":"Visual reinforcement learning with imagined goals","authors":"Nair","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Nair et al., 2020a] Ashvin Nair, Shikhar Bahl, Alexander Khazatsky, Vitchyr Pong, Glen Berseth, and Sergey Levine. Contextual imagined goals for self-supervised robotic learning. CoRL, 2020.","title":"Contextual imagined goals for self-supervised robotic learning","authors":"Nair","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Nair et al., 2020b] Suraj Nair, Silvio Savarese, and Chelsea Finn. Goal-aware prediction: Learning to model what mat- ters. ICML, 2020.","title":"Goal-aware prediction: Learning to model what matters","authors":"Nair","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Niemueller et al., 2019] Tim Niemueller, Till Hofmann, and Gerhard Lakemeyer. Goal reasoning in the clips ex- ecutive for integrated planning and execution. In Proceed- ings of the International Conference on Automated Plan- ning and Scheduling, volume 29, pages 754-763, 2019.","title":"Goal reasoning in the clips executive for integrated planning and execution","authors":"Niemueller","year":"2019","venue":"Proceedings of the International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Paul et al., 2019] Sujoy Paul, Jeroen Vanbaar, and Amit Roy-Chowdhury. Learning from trajectories via subgoal discovery. NeurIPS, 2019.","title":"Learning from trajectories via subgoal discovery","authors":"Sujoy Paul, Jeroen Vanbaar, Amit Roy-Chowdhury","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Pitis et al., 2020] Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain ex- ploration for long horizon multi-goal reinforcement learn- ing. ICML, 2020.","title":"Maximum entropy gain exploration for long horizon multi-goal reinforcement learning","authors":"Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, Jimmy Ba","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Plappert et al., 2018] Matthias Plappert, Marcin Andrychowicz, et al. Multi-goal reinforcement learn- ing: Challenging robotics environments and request for research. CoRR abs/1802.09464, 2018.","title":"Multi-goal reinforcement learning: Challenging robotics environments and request for research","authors":"Matthias Plappert, Marcin Andrychowicz","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Pong et al., 2020] Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew- fit: State-covering self-supervised reinforcement learning. ICML, 2020.","title":"Skewfit: State-covering self-supervised reinforcement learning","authors":"Murtaza Vitchyr H Pong, Steven Dalal, Ashvin Lin, Shikhar Nair, Sergey Bahl, Levine","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Ren et al., 2019] Zhizhou Ren, Kefan Dong, Yuan Zhou, Qiang Liu, and Jian Peng. Exploration via hindsight goal generation. NeurIPS, 2019.","title":"Exploration via hindsight goal generation","authors":"Ren","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Roberts et al., 2018] Mark Roberts, Daniel Borrajo, Michael Cox, and Neil Yorke-Smith. Special issue on goal reasoning. AI Communications, 31(2):115-116, 2018.","title":"Special issue on goal reasoning","authors":"Mark Roberts, Daniel Borrajo, Michael Cox, Neil Yorke-Smith","year":"2018","venue":"AI Communications","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Rolf and Steil, 2013] Matthias Rolf and Jochen J Steil. Ef- ficient exploratory learning of inverse kinematics on a bionic elephant trunk. IEEE transactions on neural net- works and learning systems, 25(6):1147-1160, 2013.","title":"Efficient exploratory learning of inverse kinematics on a bionic elephant trunk","authors":"Rolf, Steil, Matthias Rolf, Jochen J Steil","year":"2013","venue":"IEEE transactions on neural networks and learning systems","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Santucci et al., 2016] Vieri Giuliano Santucci, Gianluca Baldassarre, and Marco Mirolli. Grail: a goal-discovering robotic architecture for intrinsically-motivated learning. IEEE Transactions on Cognitive and Developmental Sys- tems, 8(3):214-231, 2016.","title":"Grail: a goal-discovering robotic architecture for intrinsically-motivated learning","authors":"Giuliano Vieri, Gianluca Santucci, Marco Baldassarre, Mirolli","year":"2016","venue":"IEEE Transactions on Cognitive and Developmental Systems","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Schaul et al., 2015] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function ap- proximators. ICML, 2015.","title":"Universal value function approximators","authors":"Schaul","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Schmidhuber, 2019] Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards-just map them to actions. CoRR abs/1912.02875, 2019.","title":"Reinforcement learning upside down: Don't predict rewards-just map them to actions","authors":"Juergen Schmidhuber","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Shah et al., 2021] Dhruv Shah, Benjamin Eysenbach, Nicholas Rhinehart, and Sergey Levine. Rapid explo- ration for open-world navigation with latent goal models. arXiv preprint arXiv:2104.05859, 2021.","title":"Rapid exploration for open-world navigation with latent goal models","authors":"Dhruv Shah, Benjamin Eysenbach, Nicholas Rhinehart, Sergey Levine","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Sharma et al., 2020] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. ICLR, 2020.","title":"Dynamicsaware unsupervised discovery of skills","authors":"Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, Karol Hausman","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Srinivas et al., 2018] Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal plan- ning networks: Learning generalizable representations for visuomotor control. ICML, 2018.","title":"Universal planning networks: Learning generalizable representations for visuomotor control","authors":"Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, Chelsea Finn","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Srivastava et al., 2019] Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojciech Jaśkowski, and Jürgen Schmidhuber. Training agents using upside-down rein- forcement learning. CoRR abs/1912.02877, 2019.","title":"Training agents using upside-down reinforcement learning","authors":"Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojciech Jaśkowski, Jürgen Schmidhuber","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":57,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":58,"raw":"[Tang and Kucukelbir, 2021] Yunhao Tang and Alp Ku- cukelbir. Hindsight expectation maximization for goal- conditioned reinforcement learning. In AISTATS, pages 2863-2871, 2021.","title":"Hindsight expectation maximization for goalconditioned reinforcement learning","authors":"Yunhao Tang, Alp Kucukelbir","year":"2021","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":59,"raw":"[Tassa et al., 2020] Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess. dm control: Software and tasks for continuous con- trol, 2020.","title":"dm control: Software and tasks for continuous control","authors":"Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":60,"raw":"[Tian et al., 2021] Stephen Tian, Suraj Nair, Frederik Ebert, Sudeep Dasari, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Model-based visual planning with self- supervised functional distances. ICLR, 2021.","title":"Model-based visual planning with selfsupervised functional distances","authors":"Stephen Tian, Suraj Nair, Frederik Ebert, Sudeep Dasari, Benjamin Eysenbach, Chelsea Finn, Sergey Levine","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":61,"raw":"[Trott et al., 2019] Alexander Trott, Stephan Zheng, Caim- ing Xiong, and Richard Socher. Keeping your distance: Solving sparse reward tasks using self-balancing shaped rewards. NeurIPS, 2019.","title":"Keeping your distance: Solving sparse reward tasks using self-balancing shaped rewards","authors":"Alexander Trott, Stephan Zheng, Caiming Xiong, Richard Socher","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":62,"raw":"[Warde-Farley et al., 2019] David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. ICLR, 2019.","title":"Unsupervised control through non-parametric discriminative rewards","authors":"David Warde-Farley, Tom Van De Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, Volodymyr Mnih","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":63,"raw":"[Wu et al., 2018] Yifan Wu, George Tucker, and Ofir Nachum. The laplacian in rl: Learning representations with efficient approximations. ICLR, 2018.","title":"The laplacian in rl: Learning representations with efficient approximations","authors":"Yifan Wu, George Tucker, Ofir Nachum","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":64,"raw":"[Zhang et al., 2020] Yunzhi Zhang, Pieter Abbeel, and Ler- rel Pinto. Automatic curriculum learning through value disagreement. NeurIPS, 2020.","title":"Automatic curriculum learning through value disagreement","authors":"Zhang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":65,"raw":"[Zhang et al., 2021] Lunjun Zhang, Ge Yang, and Bradly C Stadie. World model as a graph: Learning latent landmarks for planning. ICML, 2021.","title":"World model as a graph: Learning latent landmarks for planning","authors":"Zhang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":66,"raw":"[Zhu et al., 2021] Menghui Zhu, Minghuan Liu, Jian Shen, Zhicheng Zhang, Sheng Chen, Weinan Zhang, Deheng Ye, Yong Yu, Qiang Fu, and Wei Yang. Mapgo: Model- assisted policy optimization for goal-oriented tasks. IJCAI, 2021.","title":"Mapgo: Modelassisted policy optimization for goal-oriented tasks","authors":"Menghui Zhu, Minghuan Liu, Jian Shen, Zhicheng Zhang, Sheng Chen, Weinan Zhang, Deheng Ye, Yong Yu, Qiang Fu, Wei Yang","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}