{"type":"ScholarlyArticle","title":"Automatic Curriculum Learning For Deep RL: A Short Survey","authors":[{"name":"Rémy Portelas","slug":"remy-portelas","given":"Rémy","family":"Portelas","affiliation":"Inria , France"},{"name":"Cédric Colas","slug":"cedric-colas","given":"Cédric","family":"Colas","affiliation":"Inria , France"},{"name":"Lilian Weng","slug":"lilian-weng","given":"Lilian","family":"Weng","affiliation":"OpenAI , USA"},{"name":"Katja Hofmann","slug":"katja-hofmann","given":"Katja","family":"Hofmann","affiliation":"Microsoft Research , UK"},{"name":"Pierre-Yves Oudeyer","slug":"pierre-yves-oudeyer","given":"Pierre-Yves","family":"Oudeyer","affiliation":"Inria , France"}],"abstract":"Automatic Curriculum Learning (ACL) has become a cornerstone of recent successes in Deep Reinforcement Learning (DRL). These methods shape the learning trajectories of agents by challenging them with tasks adapted to their capacities. In recent years, they have been used to improve sample efficiency and asymptotic performance, to organize exploration, to encourage generalization or to solve sparse reward problems, among others. To do so, ACL mechanisms can act on many aspects of learning problems. They can optimize domain randomization for Sim2Real transfer, organize task presentations in multi-task robotic settings, order sequences of opponents in multi-agent scenarios, etc. The ambition of this work is dual: 1) to present a compact and accessible introduction to the Automatic Curriculum Learning literature and 2) to draw a bigger picture of the current state of the art in ACL to encourage the cross-breeding of existing concepts and the emergence of new ideas.","doi":"10.24963/ijcai.2020/671","url":"https://ijcai.org/proceedings/2020/671","pdf":"https://ijcai.org/proceedings/2020/0671.pdf","markdown":"https://ijcai.org/proceedings/2020/671.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/671","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Survey track","track":"Survey track","number":671,"pages":{"first":4819,"last":4825},"keywords":["Machine Learning: general","Agent-based and Multi-agent Systems: general"],"videos":[],"references":[{"position":1,"raw":"[Andrychowicz et al., 2017] Marcin Andrychowicz et al. Hind- sight experience replay. In NeurIPS.","title":"Hindsight experience replay","authors":"Marcin Andrychowicz","year":null,"venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baker et al., 2019] Bowen Baker et al. Emergent tool use from multi-agent autocurricula. arXiv.","title":"Emergent tool use from multi-agent autocurricula","authors":"Baker","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bansal et al., 2017] Trapit Bansal et al. Emergent complexity via multi-agent competition. arXiv.","title":"Emergent complexity via multi-agent competition","authors":"Trapit Bansal","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Baranes and Oudeyer, 2011] A. Baranes and P. Oudeyer. The interaction of maturational constraints and intrinsic motiva- tions in active motor development. In ICDL.","title":"The interaction of maturational constraints and intrinsic motivations in active motor development","authors":"A Baranes, P Oudeyer","year":null,"venue":"ICDL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Baranes and Oudeyer, 2013] Adrien Baranes and Pierre-Yves Oudeyer. Active learning of inverse models with intrinsically motivated goal exploration in robots. Robot. Auton. Syst.","title":"Active learning of inverse models with intrinsically motivated goal exploration in robots","authors":"Oudeyer Baranes, Adrien Baranes, Pierre-Yves Oudeyer","year":"2013","venue":"Robot. Auton. Syst","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bellemare et al., 2016] Marc Bellemare et al. Unifying count- based exploration and intrinsic motivation. In NeurIPS. [Bengio et al., 2009]","title":"Unifying countbased exploration and intrinsic motivation","authors":"Bellemare","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"Yoshua Bengio et al. Curriculum learning. In ICML.","title":"Curriculum learning","authors":"Yoshua Bengio","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Burda et al., 2019] Yuri Burda et al. Exploration by random network distillation. ICLR.","title":"Exploration by random network distillation","authors":"Yuri Burda","year":null,"venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cideron et al., 2019] Geoffrey Cideron et al. Self-educated language agent with hindsight experience replay for instruc- tion following. arXiv.","title":"Self-educated language agent with hindsight experience replay for instruction following","authors":"Cideron","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Cobbe et al., 2018] Karl Cobbe et al. Quantifying generaliza- tion in reinforcement learning. arXiv. [Colas et al., 2019] Cédric Colas et al. CURIOUS: Intrinsi- cally motivated modular multi-goal reinforcement learning. In ICML.","title":"CURIOUS: Intrinsically motivated modular multi-goal reinforcement learning","authors":"Cobbe","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Colas et al., 2020] Cédric Colas et al. Language as a cognitive tool to imagine goals in curiosity-driven exploration. arXiv.","title":"Language as a cognitive tool to imagine goals in curiosity-driven exploration","authors":"Colas","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Czarnecki et al., 2018] Wojciech Czarnecki et al. Mix & match agent curricula for reinforcement learning. In ICML.","title":"Mix & match agent curricula for reinforcement learning","authors":"Wojciech Czarnecki","year":null,"venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Czarnecki et al., 2019] Wojciech Marian Czarnecki et al. Dis- tilling policy distillation. arXiv.","title":"Distilling policy distillation","authors":"Marian Wojciech, Czarnecki","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Elman, 1993] Jeffrey L. Elman. Learning and development in neural networks: the importance of starting small. Cognition.","title":"Learning and development in neural networks: the importance of starting small","authors":"Jeffrey L Elman","year":null,"venue":"Cognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Eysenbach et al., 2018] Benjamin Eysenbach et al. Diversity is all you need: Learning skills without a reward function. arXiv.","title":"Diversity is all you need: Learning skills without a reward function","authors":"Benjamin Eysenbach","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Fang et al., 2019] Meng Fang et al. Curriculum-guided hind- sight experience replay. In NeurIPS.","title":"Curriculum-guided hindsight experience replay","authors":"Meng Fang","year":null,"venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Florensa et al., 2017] Carlos Florensa et al. Reverse curricu- lum generation for reinforcement learning. CoRL.","title":"Reverse curriculum generation for reinforcement learning","authors":"Carlos Florensa","year":null,"venue":"CoRL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Florensa et al., 2018] Carlos Florensa et al. Automatic goal generation for reinforcement learning agents. In ICML.","title":"Carlos Florensa et al. Automatic goal generation for reinforcement learning agents","authors":"Florensa","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Forestier et al., 2017] Sébastien Forestier et al. Intrinsically motivated goal exploration processes with automatic curricu- lum learning. arXiv.","title":"Intrinsically motivated goal exploration processes with automatic curriculum learning","authors":"Forestier","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Fournier et al., 2018] Pierre Fournier et al. Accuracy-based curriculum learning in deep reinforcement learning. arXiv.","title":"Accuracy-based curriculum learning in deep reinforcement learning","authors":"Fournier","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Franc ¸ois-Lavet et al., 2015] Vincent Franc ¸ois-Lavet et al. How to discount deep reinforcement learning: Towards new dy- namic strategies. arXiv.","title":"Vincent Franc ¸ois-Lavet et al. How to discount deep reinforcement learning: Towards new dynamic strategies","authors":"Franc ¸ois-Lavet","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ha, 2019] David Ha. Reinforcement learning for improving agent design. Arti. Life. [Hacohen and Weinshall, 2019] Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. In Kamalika Chaudhuri and Ruslan Salakhut- dinov, editors, ICML.","title":"On the power of curriculum learning in training deep networks","authors":"David Ha, ; Arti, Guy Life, Daphna Hacohen, Weinshall","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Hernandez et al., 2019] D. Hernandez et al. A generalized framework for self-play training. In IEEE CoG. [","title":"A generalized framework for self-play training","authors":"D Hernandez","year":null,"venue":"IEEE CoG","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Horgan et al., 2018] Dan Horgan et al. Distributed prioritized experience replay. arXiv.","title":"Distributed prioritized experience replay","authors":"Dan Horgan","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ivanovic et al., 2018] Boris Ivanovic et al. Barc: Backward reachability curriculum for robotic reinforcement learning. ICRA.","title":"Barc: Backward reachability curriculum for robotic reinforcement learning","authors":"Ivanovic","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Jabri et al., 2019] Allan Jabri et al. Unsupervised curricula for visual meta-reinforcement learning. In NeurIPS.","title":"Unsupervised curricula for visual meta-reinforcement learning","authors":"Allan Jabri","year":null,"venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Jaderberg et al., 2016] Max Jaderberg et al. Reinforcement learning with unsupervised auxiliary tasks. arXiv.","title":"Max Jaderberg et al. Reinforcement learning with unsupervised auxiliary tasks","authors":"Jaderberg","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Krueger and Dayan, 2009] Kai A. Krueger and Peter Dayan. Flexible shaping: How learning in small steps helps. Cog- nition.","title":"Flexible shaping: How learning in small steps helps","authors":"Kai A Krueger, Peter Dayan","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Lair et al., 2019] Nicolas Lair et al. Language grounding through social interactions and curiosity-driven multi-goal learning. arXiv.","title":"Language grounding through social interactions and curiosity-driven multi-goal learning","authors":"Lair","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Laversanne-Finot et al., 2018] Adrien Laversanne-Finot et al. Curiosity driven exploration of learned disentangled goal spaces. [Lesort et al., 2019] Timothée Lesort et al. Continual learning for robotics. arXiv.","title":"Adrien Laversanne-Finot et al. Curiosity driven exploration of learned disentangled goal spaces","authors":"Laversanne-Finot","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Lin, 1992] Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Mach. lear.","title":"Self-improving reactive agents based on reinforcement learning, planning and teaching","authors":"Long-Ji Lin","year":null,"venue":"Mach. lear","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Lopes and Oudeyer, 2012] Manuel Lopes and Pierre-Yves Oudeyer. The strategic student approach for life-long explo- ration and learning. In ICDL.","title":"The strategic student approach for life-long exploration and learning","authors":"Manuel Lopes, Pierre-Yves Oudeyer","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Matiisen et al., 2017] Tambet Matiisen et al. Teacher-student curriculum learning. IEEE TNNLS.","title":"Tambet Matiisen et al. Teacher-student curriculum learning","authors":"Matiisen","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Mehta et al., 2019] Bhairav Mehta et al. Active domain ran- domization. CoRL.","title":"Active domain randomization","authors":"Mehta","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Moore and Atkeson, 1993] Andrew W Moore and Christo- pher G Atkeson. Prioritized sweeping: Reinforcement learn- ing with less data and less time. Mach. learn.","title":"Prioritized sweeping: Reinforcement learning with less data and less time","authors":"Andrew W, Moore, Christopher G Atkeson","year":null,"venue":"Mach. learn","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Mysore et al., 2018] S. Mysore et al. Reward-guided curricu- lum for robust reinforcement learning. preprint.","title":"Reward-guided curriculum for robust reinforcement learning","authors":"S Mysore","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Narasimhan et al., 2015] Karthik Narasimhan et al. Language understanding for text-based games using deep reinforcement learning. arXiv.","title":"Language understanding for text-based games using deep reinforcement learning","authors":"Karthik Narasimhan","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[OpenAI et al., 2019] OpenAI et al. Solving rubik's cube with a robot hand. arXiv.","title":"Solving rubik's cube with a robot hand","authors":"Openai","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Oudeyer et al., 2007] Pierre-Yves Oudeyer et al. Intrinsic mo- tivation systems for autonomous mental development. IEEE trans. on evolutionary comp. [Pathak et al., 2017] Deepak Pathak et al. Curiosity-driven ex- ploration by self-supervised prediction. In CVPR.","title":"Intrinsic motivation systems for autonomous mental development","authors":"Pierre-Yves Oudeyer","year":"2017","venue":"IEEE trans. on evolutionary comp","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Pathak et al., 2019] Deepak Pathak et al. Self-supervised ex- ploration via disagreement. arXiv.","title":"Self-supervised exploration via disagreement","authors":"Deepak Pathak","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Pinto et al., 2017] Lerrel Pinto et al. Robust adversarial rein- forcement learning. arXiv.","title":"Lerrel Pinto et al. Robust adversarial reinforcement learning","authors":"Pinto","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Pong et al., 2019] Vitchyr H. Pong et al. Skew-fit: State- covering self-supervised reinforcement learning. arXiv.","title":"Skew-fit: Statecovering self-supervised reinforcement learning","authors":"H Vitchyr, Pong","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Portelas et al., 2019] Rémy Portelas et al. Teacher algorithms for curriculum learning of deep rl in continuously parameter- ized environments. CoRL.","title":"Rémy Portelas et al. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments","authors":"Portelas","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Racanière et al., 2019] Sébastien Racanière et al. Automated curricula through setter-solver interactions. arXiv.","title":"Automated curricula through setter-solver interactions","authors":"Sébastien Racanière","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Risi and Togelius, 2019] Sebastian Risi and Julian Togelius. Procedural content generation: From automatically generat- ing game levels to increasing generality in machine learning. arXiv.","title":"Procedural content generation: From automatically generating game levels to increasing generality in machine learning","authors":"Togelius Risi, Sebastian Risi, Julian Togelius","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Salimans and Chen, 2018] Tim Salimans and Richard Chen. Learning montezuma's revenge from a single demonstration. NeurIPS.","title":"Learning montezuma's revenge from a single demonstration","authors":"Chen ; Salimans, Tim Salimans, Richard Chen","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Schaul et al., 2015a] Tom Schaul et al. Universal value func- tion approximators. In ICML.","title":"Universal value function approximators","authors":"Schaul","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Schaul et al., 2015b] Tom Schaul et al. Prioritized experience replay. arXiv.","title":"Prioritized experience replay","authors":"Schaul","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Schmidhuber, 1991] Jürgen Schmidhuber. Curious model- building control systems. In IJCNN. IEEE.","title":"Curious modelbuilding control systems","authors":"Jürgen Schmidhuber","year":null,"venue":"IJCNN. IEEE","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Selfridge et al., 1985] Oliver G Selfridge et al. Training and tracking in robotics. In IJCAI.","title":"Training and tracking in robotics","authors":"Oliver G Selfridge","year":null,"venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/1985/130"},{"position":51,"raw":"[Shyam et al., 2018] Pranav Shyam et al. Model-based active exploration. arXiv.","title":"Model-based active exploration","authors":"Pranav Shyam","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Silver et al., 2017] David Silver et al. Mastering the game of go without human knowledge. Nature. [Sukhbaatar et al., 2017] Sainbayar Sukhbaatar et al. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv.","title":"Intrinsic motivation and automatic curricula via asymmetric self-play","authors":"David Silver","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Taylor and Stone, 2009] Matthew E. Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A sur- vey. JMLR.","title":"Transfer learning for reinforcement learning domains: A survey","authors":"Matthew E Taylor, Peter Stone","year":null,"venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Turing, 1950] Alan M Turing. Computing machinery and in- telligence.","title":"Computing machinery and intelligence","authors":"Alan M Turing","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Vinyals et al., 2019] Oriol Vinyals et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature.","title":"Grandmaster level in StarCraft II using multi-agent reinforcement learning","authors":"Vinyals","year":"2019","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":57,"raw":"[Wang et al., 2019] Rui Wang et al. Paired open-ended trail- blazer (POET): endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv.","title":"Paired open-ended trailblazer (POET): endlessly generating increasingly complex and diverse learning environments and their solutions","authors":"Rui Wang","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":58,"raw":"[Zha et al., 2019] Daochen Zha et al. Experience replay opti- mization. arXiv.","title":"Experience replay optimization","authors":"Zha","year":"2019","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2019/589"},{"position":59,"raw":"[Zhao and Tresp, 2018] Rui Zhao and Volker Tresp. Energy- based hindsight experience prioritization. arXiv.","title":"Energybased hindsight experience prioritization","authors":"Rui Zhao, Volker Tresp","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":60,"raw":"[Zhao and Tresp, 2019] Rui Zhao and Volker Tresp. Curiosity- driven experience prioritization via density estimation. arXiv.","title":"Curiositydriven experience prioritization via density estimation","authors":"Rui Zhao, Volker Tresp","year":null,"venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}