{"type":"ScholarlyArticle","title":"Transferable Curricula through Difficulty Conditioned Generators","authors":[{"name":"Sidney Tio","slug":"sidney-tio","given":"Sidney","family":"Tio","affiliation":"Singapore Management University"},{"name":"Pradeep Varakantham","slug":"pradeep-varakantham","given":"Pradeep","family":"Varakantham","affiliation":"Singapore Management University"}],"abstract":"Advancements in reinforcement learning (RL) have demonstrated superhuman performance in complex tasks such as Starcraft, Go, Chess etc. However, knowledge transfer from Artificial ``Experts\" to humans remain a significant challenge. A promising avenue for such transfer would be the use of curricula. Recent methods in curricula generation focuses on training RL agents efficiently, yet such methods rely on surrogate measures to track student progress, and are not suited for training robots in the real world (or more ambitiously humans). In this paper, we introduce a method named Parameterized Environment Response Model (PERM) that shows promising results in training RL agents in parameterized environments. Inspired by Item Response Theory, PERM seeks to model difficulty of environments and ability of RL agents directly. Given that RL agents and humans are trained more efficiently under the ``zone of proximal development\", our method generates a curriculum by matching the difficulty of an environment to the current ability of the student.  In addition, PERM can be trained offline and does not employ non-stationary measures of student ability, making it suitable for transfer between students. We demonstrate PERM's ability to represent the environment parameter space, and training with RL agents with PERM produces a strong performance in deterministic environments. Lastly, we show that our method is transferable between students, without any sacrifice in training quality.","doi":"10.24963/ijcai.2023/543","url":"https://ijcai.org/proceedings/2023/543","pdf":"https://ijcai.org/proceedings/2023/0543.pdf","markdown":"https://ijcai.org/proceedings/2023/543.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/543","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":543,"pages":{"first":4883,"last":4891},"keywords":["Multidisciplinary Topics and Applications: MDA: Game playing","Humans and AI: HAI: Computer-aided education","Multidisciplinary Topics and Applications: MDA: Social sciences"],"videos":[],"references":[{"position":1,"raw":"[Arulkumaran et al., 2019] Kai Arulkumaran, Antoine Cully, and Julian Togelius. Alphastar: An evolutionary computation perspective. In Proceedings of the genetic and evolutionary computation conference companion, pages 314-315, 2019.","title":"Alphastar: An evolutionary computation perspective","authors":"Kai Arulkumaran, Antoine Cully, Julian Togelius","year":"2019","venue":"Proceedings of the genetic and evolutionary computation conference companion","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Berner et al., 2019] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforce- ment learning. arXiv preprint arXiv:1912.06680, 2019.","title":"Dota 2 with large scale deep reinforcement learning","authors":"Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dennis et al., 2020] Michael Dennis, Natasha Jaques, Eu- gene Vinitsky, Alexandre Bayen, Stuart Russell, An- drew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment de- sign. Advances in neural information processing systems, 33:13049-13061, 2020.","title":"Emergent complexity and zero-shot transfer via unsupervised environment design","authors":"Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, Sergey Levine","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Du et al., 2022] Yuqing Du, Pieter Abbeel, and Aditya Grover. It takes four to tango: Multiagent selfplay for automatic curriculum generation. arXiv preprint arXiv:2202.10608, 2022.","title":"It takes four to tango: Multiagent selfplay for automatic curriculum generation","authors":"Yuqing Du, Pieter Abbeel, Aditya Grover","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Embretson and Reise, 2013] Susan E Embretson and Steven P Reise. Item response theory. Psychology Press, 2013.","title":"Item response theory","authors":"E Susan, Steven P Embretson, Reise","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Foglino et al., 2019] Francesco Foglino, Christiano Coletto Christakou, and Matteo Leonetti. An optimization frame- work for task sequencing in curriculum learning. In 2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), pages 207-214. IEEE, 2019.","title":"An optimization framework for task sequencing in curriculum learning","authors":"Francesco Foglino, Christiano Coletto Christakou, Matteo Leonetti","year":"2019","venue":"Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Harlen, 2001] Wynne Harlen. The Assessment of Scien- tific Literacy in the OECD/PISA Project, pages 49-60. Springer Netherlands, Dordrecht, 2001.","title":"The Assessment of Scientific Literacy in the OECD/PISA Project","authors":"Wynne Harlen","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hartshorne et al., 2018] Joshua K Hartshorne, Joshua B Tenenbaum, and Steven Pinker. A critical period for sec- ond language acquisition: Evidence from 2/3 million en- glish speakers. Cognition, 177:263-277, 2018.","title":"A critical period for second language acquisition: Evidence from 2/3 million english speakers","authors":"Joshua B Joshua K Hartshorne, Steven Tenenbaum, Pinker","year":"2018","venue":"Cognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jiang et al., 2021a] Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-guided adversarial environment design. Advances in Neural Information Processing Sys- tems, 34:1884-1897, 2021.","title":"Replay-guided adversarial environment design","authors":"Jiang","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jiang et al., 2021b] Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In Interna- tional Conference on Machine Learning, pages 4940- 4950. PMLR, 2021.","title":"Prioritized level replay","authors":"Jiang","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2023a] Dexun Li, Wenjun Li, and Pradeep Varakantham. Diversity induced environment design via self-play. arXiv preprint arXiv:2302.02119, 2023.","title":"Diversity induced environment design via self-play","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2023b] Wenjun Li, Pradeep Varakantham, and Dexun Li. Effective diversity in unsupervised environment design. arXiv preprint arXiv:2301.08025, 2023.","title":"Effective diversity in unsupervised environment design","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Łuniewska et al., 2016] Magdalena Łuniewska, Ewa Haman, Sharon Armon-Lotem, Bartłomiej Etenkowski, Frenette Southwood, Darinka Anelković, Elma Blom, Tessel Boerma, Shula Chiat, Pascale Engel de Abreu, et al. Ratings of age of acquisition of 299 words across 25 languages: Is there a cross-linguistic order of words? Behavior research methods, 48(3):1154-1177, 2016.","title":"Ratings of age of acquisition of 299 words across 25 languages: Is there a cross-linguistic order of words?","authors":"Magdalena Łuniewska, Ewa Haman, Sharon Armon-Lotem, Bartłomiej Etenkowski, Frenette Southwood, Darinka Anelković, Elma Blom, Tessel Boerma, Shula Chiat, Pascale Engel De Abreu","year":"2016","venue":"Behavior research methods","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Moravec, 1988] Hans Moravec. The mind children: The fu- ture of robot and human intelligence. Harvard University Press, 1988.","title":"The mind children: The future of robot and human intelligence","authors":"Hans Moravec, Moravec","year":"1988","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Narvekar et al., 2016] Sanmit Narvekar, Jivko Sinapov, Matteo Leonetti, and Peter Stone. Source task creation for curriculum learning. In Proceedings of the 2016 inter- national conference on autonomous agents & multiagent systems, pages 566-574, 2016.","title":"Source task creation for curriculum learning","authors":"Narvekar","year":"2016","venue":"Proceedings of the 2016 international conference on autonomous agents & multiagent systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Narvekar et al., 2017] Sanmit Narvekar, Jivko Sinapov, and Peter Stone. Autonomous task sequencing for customized curriculum design in reinforcement learning. In IJCAI, pages 2536-2542, 2017.","title":"Autonomous task sequencing for customized curriculum design in reinforcement learning","authors":"Narvekar","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/353"},{"position":19,"raw":"[Narvekar et al., 2020] Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning do- mains: A framework and survey. The Journal of Machine Learning Research, 21(1):7382-7431, 2020.","title":"Curriculum learning for reinforcement learning domains: A framework and survey","authors":"Narvekar","year":"2020","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Parker-Holder et al., 2022] Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foer- ster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. arXiv preprint arXiv:2203.01302, 2022.","title":"Evolving curricula with regret-based environment design","authors":"Parker-Holder","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Racaniere et al., 2019] Sebastien Racaniere, Andrew K Lampinen, Adam Santoro, David P Reichert, Vlad Firoiu, and Timothy P Lillicrap. Automated curric- ula through setter-solver interactions. arXiv preprint arXiv:1909.12892, 2019.","title":"Automated curricula through setter-solver interactions","authors":"Sebastien Racaniere, Adam Andrew K Lampinen, David P Santoro, Vlad Reichert, Timothy P Firoiu, Lillicrap","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rasch, 1993] Georg Rasch. Probabilistic models for some intelligence and attainment tests. ERIC, 1993.","title":"Probabilistic models for some intelligence and attainment tests","authors":"Georg Rasch, Rasch","year":"1993","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Samejima, 1974] Fumiko Samejima. Normal ogive model on the continuous response level in the multidimensional latent space. Psychometrika, 39(1):111-121, 1974.","title":"Normal ogive model on the continuous response level in the multidimensional latent space","authors":"Fumiko Samejima, Samejima","year":"1974","venue":"Psychometrika","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schulman et al., 2015] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advan- tage estimation. arXiv preprint arXiv:1506.02438, 2015.","title":"Highdimensional continuous control using generalized advantage estimation","authors":"John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Silver et al., 2017] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815, 2017.","title":"Mastering chess and shogi by self-play with a general reinforcement learning algorithm","authors":"David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sutton et al., 1998] Richard S Sutton, Andrew G Barto, et al. Introduction to reinforcement learning, volume 135. MIT press Cambridge, 1998.","title":"Introduction to reinforcement learning","authors":"Andrew G Richard S Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tobin et al., 2017] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural net- works from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.","title":"Domain randomization for transferring deep neural networks from simulation to the real world","authors":"Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, Pieter Abbeel","year":"2017","venue":"IEEE/RSJ international conference on intelligent robots and systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Vygotsky and Cole, 1978] Lev Semenovich Vygotsky and Michael Cole. Mind in society: Development of higher psychological processes. Harvard university press, 1978.","title":"Lev Semenovich Vygotsky and Michael Cole. Mind in society: Development of higher psychological processes","authors":"Cole Vygotsky","year":"1978","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wu et al., 2020] Mike Wu, Richard L Davis, Benjamin W Domingue, Chris Piech, and Noah Goodman. Variational item response theory: Fast, accurate, and expressive. arXiv preprint arXiv:2002.00276, 2020.","title":"Variational item response theory: Fast, accurate, and expressive","authors":"Mike Wu, Richard L Davis, Benjamin W Domingue, Chris Piech, Noah Goodman","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhuang et al., 2022] Yan Zhuang, Qi Liu, Zhenya Huang, Zhi Li, Shuanghong Shen, and Haiping Ma. Fully adap- tive framework: Neural computerized adaptive testing for online education. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 4734-4742, 2022.","title":"Fully adaptive framework: Neural computerized adaptive testing for online education","authors":"Yan Zhuang, Qi Liu, Zhenya Huang, Zhi Li, Shuanghong Shen, Haiping Ma","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}