{"type":"ScholarlyArticle","title":"Multi-objective Optimization-based Selection for Quality-Diversity by Non-surrounded-dominated Sorting","authors":[{"name":"Ren-Jian Wang","slug":"ren-jian-wang","given":"Ren-Jian","family":"Wang","affiliation":"State Key Laboratory for Novel Software Technology , Nanjing University , Nanjing , China"},{"name":"Ke Xue","slug":"ke-xue","given":"Ke","family":"Xue","affiliation":"State Key Laboratory for Novel Software Technology , Nanjing University , Nanjing , China"},{"name":"Haopu Shang","slug":"haopu-shang","given":"Haopu","family":"Shang","affiliation":"State Key Laboratory for Novel Software Technology , Nanjing University , Nanjing , China"},{"name":"Chao Qian","slug":"chao-qian","given":"Chao","family":"Qian","affiliation":"State Key Laboratory for Novel Software Technology , Nanjing University , Nanjing , China"},{"name":"Haobo Fu","slug":"haobo-fu","given":"Haobo","family":"Fu","affiliation":"Tencent AI Lab , Shenzhen , China"},{"name":"Qiang Fu","slug":"qiang-fu","given":"Qiang","family":"Fu","affiliation":"Tencent AI Lab , Shenzhen , China"}],"abstract":"Quality-Diversity (QD) algorithms, a subset of evolutionary algorithms, maintain an archive (i.e., a set of solutions) and simulate the natural evolution process through iterative selection and reproduction, with the goal of generating a set of high-quality and diverse solutions. Though having found many successful applications in reinforcement learning, QD algorithms often select the parent solutions uniformly at random, which lacks selection pressure and may limit the performance. Recent studies have treated each type of behavior of a solution as an objective, and selected the parent solutions based on Multi-objective Optimization (MO), which is a natural idea, but has not lead to satisfactory performance as expected. This paper gives the reason for the first time, and then proposes a new MO-based selection method by non-surrounded-dominated sorting (NSS), which considers all possible directions of the behaviors, and thus can generate diverse solutions over the whole behavior space. By combining NSS with the most widespread QD algorithm, MAP-Elites, we perform experiments on synthetic functions and several complex tasks (i.e., QDGym, robotic arm, and Mario environment generation), showing that NSS achieves better performance than not only other MO-based selection methods but also state-of-the-art selection methods in QD.","doi":"10.24963/ijcai.2023/482","url":"https://ijcai.org/proceedings/2023/482","pdf":"https://ijcai.org/proceedings/2023/0482.pdf","markdown":"https://ijcai.org/proceedings/2023/482.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/482","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":482,"pages":{"first":4335,"last":4343},"keywords":["Machine Learning: ML: Reinforcement learning","Machine Learning: ML: Evolutionary learning","Search: S: Evolutionary computation"],"videos":[],"references":[{"position":1,"raw":"[Allard et al., 2022] Maxime Allard, Simón C Smith, Kon- stantinos Chatzilygeroudis, Bryan Lim, and Antoine Cully. Online damage recovery for physical robots with hierarchical quality-diversity. arXiv:2210.09918, 2022.","title":"Online damage recovery for physical robots with hierarchical quality-diversity","authors":"Maxime Allard, C Simón, Konstantinos Smith, Bryan Chatzilygeroudis, Antoine Lim, Cully","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bäck, 1996] Thomas Bäck. Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms. Oxford University Press, 1996.","title":"Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms","authors":"Thomas Bäck","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bhatt et al., 2022] Varun Bhatt, Bryon Tjanaka, Matthew C. Fontaine, and Stefanos Nikolaidis. Deep surrogate assisted generation of environments. In Advances in Neural Infor- mation Processing Systems 35 (NeurIPS), New Orleans, LA, 2022.","title":"Deep surrogate assisted generation of environments","authors":"Varun Bhatt, Bryon Tjanaka, Matthew C Fontaine, Stefanos Nikolaidis","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chalumeau et al., 2023] Felix Chalumeau, Raphael Boige, Bryan Lim, Valentin Macé, Maxime Allard, Arthur Flajo- let, Antoine Cully, and Thomas Pierrot. Neuroevolution is a competitive alternative to reinforcement learning for skill discovery. In The 11th International Conference on Learning Representations (ICLR), Kigali, Rwanda, 2023.","title":"Neuroevolution is a competitive alternative to reinforcement learning for skill discovery","authors":"Felix Chalumeau, Raphael Boige, Bryan Lim, Valentin Macé, Maxime Allard, Arthur Flajolet, Antoine Cully, Thomas Pierrot","year":"2023","venue":"The 11th International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chatzilygeroudis et al., 2021] Konstantinos Chatzilyger- oudis, Antoine Cully, Vassilis Vassiliades, and Jean- Baptiste Mouret. Quality-diversity optimization: A novel branch of stochastic optimization. In Black Box Optimization, Machine Learning, and No-Free Lunch Theorems, pages 109-135. Springer, 2021.","title":"Quality-diversity optimization: A novel branch of stochastic optimization","authors":"Konstantinos Chatzilygeroudis, Antoine Cully, Vassilis Vassiliades, Jean-Baptiste Mouret","year":"2021","venue":"Black Box Optimization, Machine Learning, and No-Free Lunch Theorems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cully and Demiris, 2018] Antoine Cully and Yiannis Demiris. Quality and diversity optimization: A unifying modular framework. IEEE Transactions on Evolutionary Computation, 22(2):245-259, 2018.","title":"Quality and diversity optimization: A unifying modular framework","authors":"Demiris Cully, Antoine Cully, Yiannis Demiris","year":"2018","venue":"IEEE Transactions on Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cully et al., 2015] Antoine Cully, Jeff Clune, Danesh Tara- pore, and Jean-Baptiste Mouret. Robots that can adapt like animals. Nature, 521(7553):503-507, 2015.","title":"Robots that can adapt like animals","authors":"Antoine Cully, Jeff Clune, Danesh Tarapore, Jean-Baptiste Mouret","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Deb et al., 2002] Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, and T. Meyarivan. A fast and elitist multiobjec- tive genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation, 6(2):182-197, 2002.","title":"A fast and elitist multiobjective genetic algorithm: NSGA-II","authors":"Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, T Meyarivan","year":"2002","venue":"IEEE Transactions on Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Deb, 2011] Kalyanmoy Deb. Multi-objective optimisation using evolutionary algorithms: An introduction. In Multi- objective Evolutionary Optimisation for Product Design and Manufacturing, pages 3-34. Springer, 2011.","title":"Multi-objective optimisation using evolutionary algorithms: An introduction","authors":"Deb Kalyanmoy","year":"2011","venue":"Multiobjective Evolutionary Optimisation for Product Design and Manufacturing","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Do et al., 2022] Anh Do, Mingyu Guo, Aneta Neumann, and Frank Neumann. Analysis of evolutionary diversity optimization for permutation problems. ACM Transac- tions on Evolutionary Learning and Optimization, 2(3):1- 27, 2022.","title":"Analysis of evolutionary diversity optimization for permutation problems","authors":"Anh Do, Mingyu Guo, Aneta Neumann, Frank Neumann","year":"2022","venue":"ACM Transactions on Evolutionary Learning and Optimization","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ecoffet et al., 2021] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580-586, 2021.","title":"First return, then explore","authors":"Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, Jeff Clune","year":"2021","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Eysenbach et al., 2018] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In The 6th International Conference on Learning Represen- tations (ICLR), Vancouver, Canada, 2018.","title":"Diversity is all you need: Learning skills without a reward function","authors":"Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey Levine","year":"2018","venue":"The 6th International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Flageat et al., 2023] Manon Flageat, Felix Chalumeau, and Antoine Cully. Empirical analysis of PGA-MAP-Elites for neuroevolution in uncertain domains. ACM Transactions on Evolutionary Learning and Optimization, 2023.","title":"Empirical analysis of PGA-MAP-Elites for neuroevolution in uncertain domains","authors":"Flageat","year":"2023","venue":"ACM Transactions on Evolutionary Learning and Optimization","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Fontaine and Nikolaidis, 2021] Matthew Fontaine and Ste- fanos Nikolaidis. Differentiable quality diversity. In Advances in Neural Information Processing Systems 34 (NeurIPS), pages 10040-10052, Virtual, 2021.","title":"Matthew Fontaine and Stefanos Nikolaidis. Differentiable quality diversity","authors":"Nikolaidis Fontaine","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Fontaine et al., 2021] Matthew C. Fontaine, Ruilin Liu, Ahmed Khalifa, Jignesh Modi, Julian Togelius, Amy K. Hoover, and Stefanos Nikolaidis. Illuminating mario scenes in the latent space of a generative adversarial net- work. In Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI), pages 5922-5930, Virtual, 2021.","title":"Illuminating mario scenes in the latent space of a generative adversarial network","authors":"C Matthew, Ruilin Fontaine, Ahmed Liu, Jignesh Khalifa, Julian Modi, Amy K Togelius, Stefanos Hoover, Nikolaidis","year":"2021","venue":"Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kumar et al., 2020] Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured MaxEnt RL. In Advances in Neural Information Processing Systems 34 (NeurIPS), pages 8198-8210, Vancouver, Canada, 2020.","title":"One solution is not all you need: Few-shot extrapolation via structured MaxEnt RL","authors":"Saurabh Kumar, Aviral Kumar, Sergey Levine, Chelsea Finn","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lehman and Stanley, 2011] Joel Lehman and Kenneth O. Stanley. Evolving a diversity of virtual creatures through novelty search and local competition. In Proceedings of the 13th ACM Genetic and Evolutionary Computation Conference (GECCO), pages 211-218, Dublin, Ireland, 2011.","title":"Evolving a diversity of virtual creatures through novelty search and local competition","authors":"Stanley ; Lehman, Joel Lehman, Kenneth O Stanley","year":"2011","venue":"Proceedings of the 13th ACM Genetic and Evolutionary Computation Conference (GECCO)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lupu et al., 2021] Andrei Lupu, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In Proceedings of the 38th International Conference on Machine Learning (ICML), pages 7204-7213, Virtual, 2021.","title":"Trajectory diversity for zero-shot coordination","authors":"Andrei Lupu, Hengyuan Hu, Jakob Foerster","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Miao et al., 2022] Jiayu Miao, Tianze Zhou, Kun Shao, Ming Zhou, Weinan Zhang, Jianye Hao, Yong Yu, and Jun Wang. Promoting quality and diversity in population- based reinforcement learning via hierarchical trajectory space exploration. In Proceedings of the 39th IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 7544-7550, Philadelphia, PA, 2022.","title":"Promoting quality and diversity in populationbased reinforcement learning via hierarchical trajectory space exploration","authors":"Jiayu Miao, Tianze Zhou, Kun Shao, Ming Zhou, Weinan Zhang, Jianye Hao, Yong Yu, Jun Wang","year":"2022","venue":"Proceedings of the 39th IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mouret and Clune, 2015] Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv:1504.04909, 2015.","title":"Illuminating search spaces by mapping elites","authors":"Clune ; Jean-Baptiste Mouret, Jeff Mouret, Clune","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nikfarjam et al., 2022] Adel Nikfarjam, Amirhossein Moosavi, Aneta Neumann, and Frank Neumann. Com- puting high-quality solutions for the patient admission scheduling problem using evolutionary diversity optimisa- tion. In Proceedings of the 17th International Conference on Parallel Problem Solving from Nature (PPSN), pages 250-264, Dortmund, Germany, 2022.","title":"Computing high-quality solutions for the patient admission scheduling problem using evolutionary diversity optimisation","authors":"Nikfarjam","year":"2022","venue":"Proceedings of the 17th International Conference on Parallel Problem Solving from Nature (PPSN)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nilsson and Cully, 2021] Olle Nilsson and Antoine Cully. Policy gradient assisted MAP-Elites. In Proceedings of the 23th ACM Genetic and Evolutionary Computation Confer- ence (GECCO), page 866-875, Lille, France, 2021.","title":"Policy gradient assisted MAP-Elites","authors":"Olle Nilsson, Antoine Cully","year":"2021","venue":"Proceedings of the 23th ACM Genetic and Evolutionary Computation Conference (GECCO)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Parker-Holder et al., 2020] Jack Parker-Holder, Aldo Pac- chiano, Krzysztof M. Choromanski, and Stephen J. Roberts. Effective diversity in population based reinforce- ment learning. In Advances in Neural Information Pro- cessing Systems 34 (NeurIPS), pages 18050-18062, Van- couver, Canada, 2020.","title":"Effective diversity in population based reinforcement learning","authors":"Parker-Holder","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pugh et al., 2016] Justin K. Pugh, Lisa B. Soros, and Ken- neth O. Stanley. Quality diversity: A new frontier for evo- lutionary computation. Frontiers Robotics AI, 3:40, 2016.","title":"Quality diversity: A new frontier for evolutionary computation","authors":"Justin K Pugh, Lisa B Soros, Kenneth O Stanley","year":"2016","venue":"Frontiers Robotics AI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Salehi et al., 2022] Achkan Salehi, Alexandre Coninx, and Stephane Doncieux. Few-shot quality-diversity optimiza- tion. IEEE Robotics and Automation Letters, 7(2):4424- 4431, 2022.","title":"Few-shot quality-diversity optimization","authors":"Salehi","year":"2022","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sheikh et al., 2022] Hassam Sheikh, Kizza Frisbee, and Mariano Phielipp. DNS: Determinantal point process based neural network sampler for ensemble reinforcement learning. In Proceedings of the 39th International Confer- ence on Machine Learning (ICML), pages 19731-19746, Baltimore, MD, 2022.","title":"DNS: Determinantal point process based neural network sampler for ensemble reinforcement learning","authors":"Hassam Sheikh, Kizza Frisbee, Mariano Phielipp","year":"2022","venue":"Proceedings of the 39th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Shen et al., 2020] Ruimin Shen, Yan Zheng, Jianye Hao, Zhaopeng Meng, Yingfeng Chen, Changjie Fan, and Yang Liu. Generating behavior-diverse game ais with evolution- ary multi-objective deep reinforcement learning. In Pro- ceedings of the 29th International Joint Conference on Ar- tificial Intelligence (IJCAI), pages 3371-3377, Yokohama, Japan, 2020.","title":"Generating behavior-diverse game ais with evolutionary multi-objective deep reinforcement learning","authors":"Ruimin Shen, Yan Zheng, Jianye Hao, Zhaopeng Meng, Yingfeng Chen, Changjie Fan, Yang Liu","year":"2020","venue":"Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2020/466"},{"position":28,"raw":"[Tjanaka et al., 2022] Bryon Tjanaka, Matthew C. Fontaine, Julian Togelius, and Stefanos Nikolaidis. Approximating gradients for differentiable quality diversity in reinforce- ment learning. In Proceedings of the 24th ACM Genetic and Evolutionary Computation Conference (GECCO), page 1102-1111, Boston, MA, 2022.","title":"Approximating gradients for differentiable quality diversity in reinforcement learning","authors":"Bryon Tjanaka, Matthew C Fontaine, Julian Togelius, Stefanos Nikolaidis","year":"2022","venue":"Proceedings of the 24th ACM Genetic and Evolutionary Computation Conference (GECCO)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Villin et al., 2021] Victor Villin, Naoki Masuyama, and Yusuke Nojima. Effects of different optimization formula- tions in evolutionary reinforcement learning on diverse be- havior generation. In Proceedings of the 7th IEEE Sympo- sium Series on Computational Intelligence (SSCI), pages 01-08, Orlando, FL, 2021.","title":"Effects of different optimization formulations in evolutionary reinforcement learning on diverse behavior generation","authors":"Victor Villin, Naoki Masuyama, Yusuke Nojima","year":"2021","venue":"Proceedings of the 7th IEEE Symposium Series on Computational Intelligence (SSCI)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2022] Yutong Wang, Ke Xue, and Chao Qian. Evolutionary diversity optimization with clustering-based selection for reinforcement learning. In The 10th Inter- national Conference on Learning Representations (ICLR), Virtual, 2022.","title":"Evolutionary diversity optimization with clustering-based selection for reinforcement learning","authors":"Yutong Wang, Ke Xue, Chao Qian","year":"2022","venue":"The 10th International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xue et al., 2022] Ke Xue, Yutong Wang, Lei Yuan, Cong Guan, Chao Qian, and Yang Yu. Heteroge- neous multi-agent zero-shot coordination by coevolution. arXiv:2208.04957, 2022.","title":"Heterogeneous multi-agent zero-shot coordination by coevolution","authors":"Ke Xue, Yutong Wang, Lei Yuan, Cong Guan, Yang Chao Qian, Yu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yu et al., 2023] Chao Yu, Jiaxuan Gao, Weilin Liu, Botian Xu, Hao Tang, Jiaqi Yang, Yu Wang, and Yi Wu. Learning zero-shot cooperation with humans, assuming humans are biased. In The 11th International Conference on Learning Representations (ICLR), Kigali, Rwanda, 2023.","title":"Learning zero-shot cooperation with humans, assuming humans are biased","authors":"Chao Yu, Jiaxuan Gao, Weilin Liu, Botian Xu, Hao Tang, Jiaqi Yang, Yu Wang, Yi Wu","year":"2023","venue":"The 11th International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yuan et al., 2023a] Lei Yuan, Feng Chen, Zongzhang Zhang, and Yang Yu. Communication-robust multi-agent learning by adaptable auxiliary multi-agent adversary gen- eration. arXiv:2305.05116, 2023.","title":"Communication-robust multi-agent learning by adaptable auxiliary multi-agent adversary generation","authors":"Yuan","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yuan et al., 2023b] Lei Yuan, Ziqian Zhang, Ke Xue, Hao Yin, Feng Chen, Cong Guan, Lihe Li, Chao Qian, and Yang Yu. Robust multi-agent coordination via evolution- ary generation of auxiliary adversarial attackers. In Pro- ceedings of the 37th AAAI Conference on Artificial Intelli- gence (AAAI), Washington, DC, 2023.","title":"Robust multi-agent coordination via evolutionary generation of auxiliary adversarial attackers","authors":"Yuan","year":"2023","venue":"Proceedings of the 37th AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang et al., 2023] Ziqian Zhang, Lei Yuan, Lihe Li, Ke Xue, Chengxing Jia, Cong Guan, Chao Qian, and Yang Yu. Fast teammate adaptation in the presence of sudden policy change. In Preoceedings of the 39th Conference on Uncertainty in Artificial Intelligence (UAI), Pittsburgh, PA, 2023.","title":"Fast teammate adaptation in the presence of sudden policy change","authors":"Ziqian Zhang, Lei Yuan, Lihe Li, Ke Xue, Chengxing Jia, Cong Guan, Yang Chao Qian, Yu","year":"2023","venue":"Preoceedings of the 39th Conference on Uncertainty in Artificial Intelligence (UAI)","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhou et al., 2019] Zhi-Hua Zhou, Yang Yu, and Chao Qian. Evolutionary Learning: Advances in Theories and Algo- rithms. Springer, 2019.","title":"Evolutionary Learning: Advances in Theories and Algorithms","authors":"Zhi-Hua Zhou, Yang Yu, Chao Qian","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}