{"type":"ScholarlyArticle","title":"Generalization through Diversity: Improving Unsupervised Environment Design","authors":[{"name":"Wenjun Li","slug":"wenjun-li","given":"Wenjun","family":"Li","affiliation":"Singapore Management University"},{"name":"Pradeep Varakantham","slug":"pradeep-varakantham","given":"Pradeep","family":"Varakantham","affiliation":"Singapore Management University"},{"name":"Dexun Li","slug":"dexun-li","given":"Dexun","family":"Li","affiliation":"Singapore Management University"}],"abstract":"Agent decision making using Reinforcement Learning (RL) heavily relies on either a model or simulator of the environment (e.g., moving in an 8x8 maze with three rooms, playing Chess on an 8x8 board). Due to this dependence, small changes in the environment (e.g., positions of obstacles in the maze, size of the board) can severely affect the effectiveness of the policy learned by the agent. To that end, existing work has proposed training RL agents on an adaptive curriculum of environments (generated automatically) to improve performance on out-of-distribution (OOD) test scenarios. Specifically, existing research has employed the potential for the agent to learn in an environment (captured using Generalized Advantage Estimation, GAE) as the key factor to select the next environment(s) to train the agent. However, such a mechanism can select similar environments (with a high potential to learn) thereby making agent training redundant on all but one of those environments. To that end, we provide a principled approach to adaptively identify diverse environments based on a novel distance measure relevant to environment design. We empirically demonstrate the versatility and effectiveness of our method in comparison to multiple leading approaches for unsupervised environment design on three distinct benchmark problems used in literature.","doi":"10.24963/ijcai.2023/601","url":"https://ijcai.org/proceedings/2023/601","pdf":"https://ijcai.org/proceedings/2023/0601.pdf","markdown":"https://ijcai.org/proceedings/2023/601.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/601","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Planning and Scheduling","track":"Main Track","number":601,"pages":{"first":5411,"last":5419},"keywords":["Planning and Scheduling: PS: Search in planning and scheduling","Machine Learning: ML: Deep reinforcement learning","Planning and Scheduling: PS: POMDPs"],"videos":[],"references":[{"position":1,"raw":"[Agarwal et al., 2021a] Rishabh Agarwal, Marlos C Machado, Pablo Samuel Castro, and Marc G Bellemare. Contrastive behavioral similarity embeddings for gen- eralization in reinforcement learning. arXiv preprint arXiv:2101.05265, 2021.","title":"Contrastive behavioral similarity embeddings for generalization in reinforcement learning","authors":"Agarwal","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agarwal et al., 2021b] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Belle- mare. Deep reinforcement learning at the edge of the sta- tistical precipice. Advances in neural information process- ing systems, 34:29304-29320, 2021.","title":"Deep reinforcement learning at the edge of the statistical precipice","authors":"Agarwal","year":"2021","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Akkaya et al., 2019] Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113, 2019.","title":"Solving rubik's cube with a robot hand","authors":"Akkaya","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cobbe et al., 2019] Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying general- ization in reinforcement learning. In International Con- ference on Machine Learning, pages 1282-1289. PMLR, 2019.","title":"Quantifying generalization in reinforcement learning","authors":"Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, John Schulman","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cobbe et al., 2020] Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International con- ference on machine learning, pages 2048-2056. PMLR, 2020.","title":"Leveraging procedural generation to benchmark reinforcement learning","authors":"Karl Cobbe, Chris Hesse, Jacob Hilton, John Schulman","year":"2020","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dennis et al., 2020] Michael Dennis, Natasha Jaques, Eu- gene Vinitsky, Alexandre Bayen, Stuart Russell, An- drew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment de- sign. Advances in neural information processing systems, 33:13049-13061, 2020.","title":"Emergent complexity and zero-shot transfer via unsupervised environment design","authors":"Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, Sergey Levine","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Flamary et al., 2021] Remi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z Alaya, Aurelie Bois- bunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, et al. Pot: Python optimal transport. J. Mach. Learn. Res., 22(78):1- 8, 2021.","title":"Pot: Python optimal transport","authors":"Remi Flamary, Nicolas Courty, Alexandre Gramfort, Z Mokhtar, Aurelie Alaya, Stanislas Boisbunon, Laetitia Chambon, Adrien Chapel, Kilian Corenflos, Nemo Fatras, Fournier","year":"2021","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hong et al., 2018] Zhang-Wei Hong, Tzu-Yun Shann, Shih- Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun-Yi Lee. Diversity-driven exploration strategy for deep reinforce- ment learning. Advances in neural information processing systems, 31, 2018.","title":"Diversity-driven exploration strategy for deep reinforcement learning","authors":"Zhang-Wei Hong, Tzu-Yun Shann, Shih-Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, Chun-Yi Lee","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jakobi, 1997] Nick Jakobi. Evolutionary robotics and the radical envelope-of-noise hypothesis. Adaptive behavior, 6(2):325-368, 1997.","title":"Evolutionary robotics and the radical envelope-of-noise hypothesis","authors":"Nick Jakobi","year":"1997","venue":"Adaptive behavior","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jiang et al., 2021a] Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-guided adversarial environment design. Advances in Neural Information Processing Sys- tems, 34:1884-1897, 2021.","title":"Replay-guided adversarial environment design","authors":"Jiang","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jiang et al., 2021b] Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In Interna- tional Conference on Machine Learning, pages 4940- 4950. PMLR, 2021.","title":"Prioritized level replay","authors":"Jiang","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kantorovich, 1942] LV Kantorovich. Kantorovich: On the transfer of masses. In Dokl. Akad. Nauk. SSSR, 1942.","title":"LV Kantorovich. Kantorovich: On the transfer of masses","authors":"Kantorovich","year":"1942","venue":"Dokl. Akad. Nauk. SSSR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lupu et al., 2021] Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In International Conference on Machine Learning, pages 7204-7213. PMLR, 2021.","title":"Trajectory diversity for zero-shot coordination","authors":"Andrei Lupu, Brandon Cui, Hengyuan Hu, Jakob Foerster","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mehta et al., 2020] Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J Pal, and Liam Paull. Active domain randomization. In Conference on Robot Learning, pages 1162-1176. PMLR, 2020.","title":"Active domain randomization","authors":"Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J Pal, Liam Paull","year":"2020","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mirhoseini et al., 2021] Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Azade Nazi, et al. A graph placement method- ology for fast chip design. Nature, 594(7862):207-212, 2021.","title":"A graph placement methodology for fast chip design","authors":"Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Azade Nazi","year":"2021","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Monge, 1781] Gaspard Monge. Mémoire sur la théorie des déblais et des remblais. Mem. Math. Phys. Acad. Royale Sci., pages 666-704, 1781.","title":"Mémoire sur la théorie des déblais et des remblais","authors":"Gaspard Monge","year":"1781","venue":"Mem. Math. Phys. Acad. Royale Sci.","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Morimoto and Doya, 2005] Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural computation, 17(2):335-359, 2005.","title":"Robust reinforcement learning","authors":"Doya Morimoto, Jun Morimoto, Kenji Doya","year":"2005","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Parker-Holder et al., 2020] Jack Parker-Holder, Aldo Pac- chiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning. Advances in Neural Information Processing Sys- tems, 33:18050-18062, 2020.","title":"Effective diversity in population based reinforcement learning","authors":"Parker-Holder","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Parker-Holder et al., 2022] Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foer- ster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. arXiv preprint arXiv:2203.01302, 2022.","title":"Evolving curricula with regret-based environment design","authors":"Parker-Holder","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pinto et al., 2017] Lerrel Pinto, James Davidson, and Ab- hinav Gupta. Supervision via competition: Robot adver- saries for learning tasks. In 2017 IEEE International Con- ference on Robotics and Automation (ICRA), pages 1601- 1608. IEEE, 2017.","title":"Supervision via competition: Robot adversaries for learning tasks","authors":"Pinto","year":"2017","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Portelas et al., 2020] Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yves Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously param- eterized environments. In Conference on Robot Learning, pages 835-853. PMLR, 2020.","title":"Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments","authors":"Rémy Portelas, Cédric Colas, Katja Hofmann, Pierre-Yves Oudeyer","year":"2020","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sadeghi and Levine, 2016] Fereshteh Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without a single real image. arXiv preprint arXiv:1611.04201, 2016.","title":"Cad2rl: Real single-image flight without a single real image","authors":"Levine Sadeghi, Fereshteh Sadeghi, Sergey Levine","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schulman et al., 2015] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advan- tage estimation. arXiv preprint arXiv:1506.02438, 2015.","title":"Highdimensional continuous control using generalized advantage estimation","authors":"John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Song et al., 2016] Jinhua Song, Yang Gao, Hao Wang, and Bo An. Measuring the distance between finite markov de- cision processes. In Proceedings of the 2016 international conference on autonomous agents & multiagent systems, pages 468-476, 2016.","title":"Measuring the distance between finite markov decision processes","authors":"Jinhua Song, Yang Gao, Hao Wang, Bo An","year":"2016","venue":"Proceedings of the 2016 international conference on autonomous agents & multiagent systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tobin et al., 2017] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural net- works from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.","title":"Domain randomization for transferring deep neural networks from simulation to the real world","authors":"Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, Pieter Abbeel","year":"2017","venue":"IEEE/RSJ international conference on intelligent robots and systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2019a] Hao Wang, Shaokang Dong, and Ling Shao. Measuring structural similarities in finite mdps. In IJCAI, pages 3684-3690, 2019.","title":"Measuring structural similarities in finite mdps","authors":"Wang","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/511"},{"position":30,"raw":"[Wang et al., 2019b] Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and di- verse learning environments and their solutions. arXiv preprint arXiv:1901.01753, 2019.","title":"Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions","authors":"Wang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2018] Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint arXiv:1806.07937, 2018.","title":"A dissection of overfitting and generalization in continuous reinforcement learning","authors":"Amy Zhang, Nicolas Ballas, Joelle Pineau","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}