{"type":"ScholarlyArticle","title":"An Evolution Strategy with Progressive Episode Lengths for Playing Games","authors":[{"name":"Lior Fuks","slug":"lior-fuks","given":"Lior","family":"Fuks","affiliation":"University of Freiburg , Germany"},{"name":"Noor Awad","slug":"noor-awad","given":"Noor","family":"Awad","affiliation":"University of Freiburg , Germany"},{"name":"Frank Hutter","slug":"frank-hutter","given":"Frank","family":"Hutter","affiliation":"University of Freiburg , Germany"},{"name":"Marius Lindauer","slug":"marius-lindauer","given":"Marius","family":"Lindauer","affiliation":"University of Freiburg , Germany"}],"abstract":"Recently, Evolution Strategies (ES) have been successfully applied to solve problems commonly addressed by reinforcement learning (RL). Due to the simplicity of ES approaches, their runtime is often dominated by the RL-task at hand (e.g., playing a game). In this work, we introduce Progressive Episode Lengths (PEL) as a new technique and incorporate it with ES. The main objective is to allow the agent to play short and easy tasks with limited lengths, and then use the gained knowledge to further solve long and hard tasks with progressive lengths. Hence allowing the agent to perform many function evaluations and find a good solution for short time horizons before adapting the strategy to tackle larger time horizons. We evaluated PEL on a subset of Atari games from OpenAI Gym, showing that it can substantially improve the optimization speed, stability and final score of canonical ES. Specifically, we show average improvements of 80% (32%) after 2 hours (10 hours) compared to canonical ES.","doi":"10.24963/ijcai.2019/172","url":"https://ijcai.org/proceedings/2019/172","pdf":"https://ijcai.org/proceedings/2019/0172.pdf","markdown":"https://ijcai.org/proceedings/2019/172.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/172","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Heuristic Search and Game Playing","track":"Main track","number":172,"pages":{"first":1234,"last":1240},"keywords":["Heuristic Search and Game Playing: Game Playing and Machine Learning","Machine Learning Applications: Game Playing"],"videos":[],"references":[{"position":1,"raw":"[Alvernaz and Togelius, 2017] S. Alvernaz and J. Togelius. Autoencoder-augmented neuroevolution for visual doom playing. In Proc. of IEEE CIG, pages 1-8, 2017.","title":"Autoencoder-augmented neuroevolution for visual doom playing","authors":"S Alvernaz, J Togelius","year":"2017","venue":"Proc. of IEEE CIG","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2009] Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In Proc. of ICML, pages 41-48, 2009.","title":"Curriculum learning","authors":"Y Bengio, J Louradour, R Collobert, J Weston","year":"2009","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brockman et al., 2016] G. Brockman, V. Cheung, L. Pet- tersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv:1606.01540 [cs.LG], 2016.","title":"Openai gym","authors":"G Brockman, V Cheung, L Pettersson, J Schneider, J Schulman, J Tang, W Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chrabaszcz et al., 2018] P. Chrabaszcz, I. Loshchilov, and F. Hutter. Back to basics: Benchmarking canonical evolu- tion strategies for playing atari. In Proc. of IJCAI, pages 1419-1426, 2018.","title":"Back to basics: Benchmarking canonical evolution strategies for playing atari","authors":"P Chrabaszcz, I Loshchilov, F Hutter","year":"2018","venue":"Proc. of IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/197"},{"position":5,"raw":"[Clevert et al., 2016] D.-A. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). In Proc. of ICLR, 2016.","title":"Fast and accurate deep network learning by exponential linear units (elus)","authors":"D.-A Clevert, T Unterthiner, S Hochreiter","year":"2016","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Conti et al., 2018] E. Conti, V. Madhavan, F. Such, J. Lehman, K. Stanley, and J. Clune. Improving ex- ploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents. arXiv:1712.06560 [cs.AI], 2018.","title":"Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents","authors":"E Conti, V Madhavan, F Such, J Lehman, K Stanley, J Clune","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cuccu et al., 2018] G. Cuccu, J. Togelius, and P. Cudre- Mauroux. Evolutionary generative adversarial networks. arXiv:1803.00657 [cs.LG], 2018.","title":"Evolutionary generative adversarial networks","authors":"G Cuccu, J Togelius, P Cudre-Mauroux","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Falkner et al., 2018] S. Falkner, A. Klein, and F. Hutter. BOHB: robust and efficient hyperparameter optimization at scale. In Proc. of ICML, pages 1436-1445, 2018.","title":"BOHB: robust and efficient hyperparameter optimization at scale","authors":"S Falkner, A Klein, F Hutter","year":"2018","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiang et al., 2015] L. Jiang, D. Meng, Q. Zhao, S. Shan, and A. Hauptmann. Self-paced curriculum learning. In Proc. of AAAI, pages 2694-2700, 2015.","title":"Self-paced curriculum learning","authors":"L Jiang, D Meng, Q Zhao, S Shan, A Hauptmann","year":"2015","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Karnin et al., 2013] Z. Karnin, T. Koren, and O. Somekh. Almost optimal exploration in multi-armed bandits. In Proc. of ICML, pages 1238-1246, 2013.","title":"Almost optimal exploration in multi-armed bandits","authors":"Z Karnin, T Koren, O Somekh","year":"2013","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[LaPorte et al., 2015] G. LaPorte, J. Branke, and C.-H. Chen. Adaptive parent population sizing in evolution strategies. Evolutionary Computation, 23:397-420, 2015.","title":"Adaptive parent population sizing in evolution strategies","authors":"G Laporte, J Branke, C.-H Chen","year":"2015","venue":"Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2017] L. Li, K. Jamieson, G. DeSalvo, A. Ros- tamizadeh, and A. Talwalkar. Hyperband: A novel bandit- based approach to hyperparameter optimization. Journal of Machine Learning Research, 18:185:1-185:52, 2017.","title":"Hyperband: A novel banditbased approach to hyperparameter optimization","authors":"L Li, K Jamieson, G Desalvo, A Rostamizadeh, A Talwalkar","year":"2017","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Miller et al., 1989] G. Miller, P. Todd, and S. Hegde. De- signing neural networks using genetic algorithms. In Proc. of ICGA, pages 379-384, 1989.","title":"Designing neural networks using genetic algorithms","authors":"G Miller, P Todd, S Hegde","year":"1989","venue":"Proc. of ICGA","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2013] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Ried- miller. Playing atari with deep reinforcement learning. arXiv:1312.5602 [cs.LG], 2013.","title":"Playing atari with deep reinforcement learning","authors":"V Mnih, K Kavukcuoglu, D Silver, A Graves, I Antonoglou, D Wierstra, M Riedmiller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2015] V. Mnih, K. Kavukcuoglu, D. Silver, A. Rusu, J. Veness, M. Bellemare, A. Graves, M. Ried- miller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beat- tie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"V Mnih, K Kavukcuoglu, D Silver, A Rusu, J Veness, M Bellemare, A Graves, M Riedmiller, A Fidjeland, G Ostrovski, S Petersen, C Beattie, A Sadik, I Antonoglou, H King, D Kumaran, D Wierstra, S Legg, D Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Narvekar, 2017] S. Narvekar. Curriculum learning in rein- forcement learning. In Proc. of IJCAI, pages 5195-5196, 2017.","title":"Curriculum learning in reinforcement learning","authors":"S Narvekar","year":"2017","venue":"Proc. of IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/757"},{"position":17,"raw":"[Poulsen et al., 2017] A. Poulsen, M. Thorhauge, M. Funch, and S. Risi. DLNE: A hybridization of deep learning and neuroevolution for visual control. In Proc. of IEEE CIG, pages 256-263, 2017.","title":"DLNE: A hybridization of deep learning and neuroevolution for visual control","authors":"A Poulsen, M Thorhauge, M Funch, S Risi","year":"2017","venue":"Proc. of IEEE CIG","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Real et al., 2017] E. Real, S. Moore, A. Selle, S. Saxena, Y. Suematsu, J. Tan, Q. Le, and A. Kurakin. Large-scale evolution of image classifiers. In Proc. of ICML, pages 2902-2911, 2017.","title":"Large-scale evolution of image classifiers","authors":"E Real, S Moore, A Selle, S Saxena, Y Suematsu, J Tan, Q Le, A Kurakin","year":"2017","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rechenberg, 1973] I. Rechenberg. Evolutionsstrategie opti- mierung technisher systeme nach prinzipien der biologis- chen evolution. PhD thesis, Frommann-Holzboog, 1973.","title":"Evolutionsstrategie optimierung technisher systeme nach prinzipien der biologischen evolution","authors":"; I Rechenberg, Rechenberg","year":"1973","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Risi and Togelius, 2017] S. Risi and J. Togelius. Neuroevo- lution in games: State of the art and open challenges. IEEE Trans. Comput. Intellig. and AI in Games, 9(1):25- 41, 2017.","title":"Neuroevolution in games: State of the art and open challenges","authors":"S Risi, J Togelius","year":"2017","venue":"IEEE Trans. Comput. Intellig. and AI in Games","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Salimans et al., 2016] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. In Pros. of NIPS, 2016.","title":"Improved techniques for training gans","authors":"T Salimans, I Goodfellow, W Zaremba, V Cheung, A Radford, X Chen","year":"2016","venue":"Pros. of NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Salimans et al., 2017] T. Salimans, J. Ho, X. Chen, and S. Sidor amd I. Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv:1703.03864 [stat.ML], 2017.","title":"Evolution strategies as a scalable alternative to reinforcement learning","authors":"T Salimans, J Ho, X Chen, S Sidor, I Sutskever","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schulman et al., 2017] J. Schulman, S. Levine, P. Moritz, M. Jordan, and P. Abbeel. Trust region policy optimiza- tion. arXiv:1502.05477 [cs.LG], 2017.","title":"Trust region policy optimization","authors":"J Schulman, S Levine, P Moritz, M Jordan, P Abbeel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Silver et al., 2017] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. Driessche, T. Graepel, and Demis Hassabis. Master- ing the game of go without human knowledge. Nature, 550:354-359, 2017.","title":"Mastering the game of go without human knowledge","authors":"D Silver, J Schrittwieser, K Simonyan, I Antonoglou, A Huang, A Guez, T Hubert, L Baker, M Lai, A Bolton, Y Chen, T Lillicrap, F Hui, L Sifre, G Driessche, T Graepel, Demis Hassabis","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Such et al., 2017] F. Such, V. Madhavan, E. Conti, J. Lehman, K. Stanley, and J. Clune. Deep neuroevolu- tion: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. CoRR, abs/1712.06567, 2017.","title":"Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning","authors":"F Such, V Madhavan, E Conti, J Lehman, K Stanley, J Clune","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Swersky et al., 2013] K. Swersky, J. Snoek, and R. Adams. Multi-task bayesian optimization. In Proc. of NIPS, pages 2004-2012, 2013.","title":"Multi-task bayesian optimization","authors":"K Swersky, J Snoek, R Adams","year":"2013","venue":"Proc. of NIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wierstra et al., 2014] D. Wierstra, T. Schaul, T. Glasmach- ers, Y. Sun, J. Peters, and J. Schmidhuber. Natural evolu- tion strategies. JMLR, 15:949-980, 2014.","title":"Natural evolution strategies","authors":"D Wierstra, T Schaul, T Glasmachers, Y Sun, J Peters, J Schmidhuber","year":"2014","venue":"JMLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}