{"type":"ScholarlyArticle","title":"Playing Atari with Six Neurons (Extended Abstract)","authors":[{"name":"Giuseppe Cuccu","slug":"giuseppe-cuccu","given":"Giuseppe","family":"Cuccu","affiliation":"eXascale Infolab , Department of Computer Science , University of Fribourg , Switzerland"},{"name":"Julian Togelius","slug":"julian-togelius","given":"Julian","family":"Togelius","affiliation":"Game Innovation Lab , Tandon School of Engineering , New York University , NY , USA"},{"name":"Philippe Cudré-Mauroux","slug":"philippe-cudre-mauroux","given":"Philippe","family":"Cudre-Mauroux","affiliation":"eXascale Infolab , Department of Computer Science , University of Fribourg , Switzerland"}],"abstract":"Deep reinforcement learning applied to vision-based problems like Atari games maps pixels directly to actions; internally, the deep neural network bears the responsibility of both extracting useful information and making decisions based on it. By separating image processing from decision-making, one could better understand the complexity of each task, as well as potentially find smaller policy representations that are easier for humans to understand and may generalize better. To this end, we propose a new method for learning policies and compact state representations separately but simultaneously for policy approximation in reinforcement learning. State representations are generated by an encoder based on two novel algorithms: Increasing Dictionary Vector Quantization makes the encoder capable of growing its dictionary size over time, to address new observations; and Direct Residuals Sparse Coding encodes observations by aiming  for highest information inclusion. We test our system on a selection of Atari games using tiny neural networks of only 6 to 18 neurons (depending on the game's controls). These are still capable of achieving results comparable---and occasionally superior---to state-of-the-art techniques which use two orders of magnitude more neurons.","doi":"10.24963/ijcai.2020/651","url":"https://ijcai.org/proceedings/2020/651","pdf":"https://ijcai.org/proceedings/2020/0651.pdf","markdown":"https://ijcai.org/proceedings/2020/651.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/651","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":651,"pages":{"first":4711,"last":4715},"keywords":["Machine Learning: Unsupervised Learning","Machine Learning: Reinforcement Learning","Machine Learning Applications: Game Playing"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25140/file_25140.m3u8","length_sec":248,"provider":"hls"}],"references":[{"position":1,"raw":"[Bellemare et al., 2013] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"Yavar Marc G Bellemare, Joel Naddaf, Michael Veness, Bowling","year":"2013","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Conti et al., 2018] Edoardo Conti, Vashisht Madhavan, Fe- lipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty- seeking agents. In Advances in Neural Information Pro- cessing Systems (NIPS), pages 5032-5043, 2018.","title":"Improving exploration in evolution strategies for deep reinforcement learning via a population of noveltyseeking agents","authors":"Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, Jeff Clune","year":"2018","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cuccu et al., 2019] Giuseppe Cuccu, Julian Togelius, and Philippe Cudré-Mauroux. Playing Atari with six neurons. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 998- 1006. International Foundation for Autonomous Agents and Multiagent Systems, 2019.","title":"Playing Atari with six neurons","authors":"Giuseppe Cuccu, Julian Togelius, Philippe Cudré-Mauroux","year":"2019","venue":"Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Floreano et al., 2008] Dario Floreano, Peter Dürr, and Clau- dio Mattiussi. Neuroevolution: from architectures to learn- ing. Evolutionary Intelligence, 1(1):47-62, 2008.","title":"Neuroevolution: from architectures to learning","authors":"Dario Floreano, Peter Dürr, Claudio Mattiussi","year":"2008","venue":"Evolutionary Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Glasmachers et al., 2010] Tobias Glasmachers, Tom Schaul, Sun Yi, Daan Wierstra, and Jürgen Schmidhuber. Exponential natural evolution strategies. In Proceedings of the 12th annual conference on Genetic and evolutionary computation, pages 393-400. ACM, 2010.","title":"Exponential natural evolution strategies","authors":"Tobias Glasmachers, Tom Schaul, Sun Yi, Daan Wierstra, Jürgen Schmidhuber","year":"2010","venue":"Proceedings of the 12th annual conference on Genetic and evolutionary computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hausknecht et al., 2014] Matthew Hausknecht, Joel Lehman, Risto Miikkulainen, and Peter Stone. A neu- roevolution approach to general atari game playing. IEEE Transactions on Computational Intelligence and AI in Games, 6(4):355-366, 2014.","title":"A neuroevolution approach to general atari game playing","authors":"Matthew Hausknecht, Joel Lehman, Risto Miikkulainen, Peter Stone","year":"2014","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hessel et al., 2017] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dab- ney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep re- inforcement learning. arXiv preprint arXiv:1710.02298, 2017.","title":"Rainbow: Combining improvements in deep reinforcement learning","authors":"Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, David Silver","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Igel, 2003] Christian Igel. Neuroevolution for reinforce- ment learning using evolution strategies. In Evolutionary Computation, 2003. CEC'03. The 2003 Congress on, vol- ume 4, pages 2588-2595. IEEE, 2003.","title":"Christian Igel. Neuroevolution for reinforcement learning using evolution strategies","authors":"Igel","year":"2003","venue":"CEC'03. The 2003 Congress on","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Justesen et al., 2019] Niels Justesen, Philip Bontrager, Ju- lian Togelius, and Sebastian Risi. Deep learning for video game playing. IEEE Transactions on Games, 2019.","title":"Deep learning for video game playing","authors":"Niels Justesen, Philip Bontrager, Julian Togelius, Sebastian Risi","year":"2019","venue":"IEEE Transactions on Games","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Risi and Togelius, 2017] Sebastian Risi and Julian Togelius. Neuroevolution in games: State of the art and open chal- lenges. IEEE Transactions on Computational Intelligence and AI in Games, 9(1):25-41, 2017.","title":"Neuroevolution in games: State of the art and open challenges","authors":"Togelius Risi, Sebastian Risi, Julian Togelius","year":"2017","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Salimans et al., 2017] Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864, 2017.","title":"Evolution strategies as a scalable alternative to reinforcement learning","authors":"Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, Ilya Sutskever","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Stanley and Miikkulainen, 2002] Kenneth O Stanley and Risto Miikkulainen. Evolving neural networks through augmenting topologies. Evolutionary computation, 10(2):99-127, 2002.","title":"Evolving neural networks through augmenting topologies","authors":"O Kenneth, Risto Stanley, Miikkulainen","year":"2002","venue":"Evolutionary computation","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Such et al., 2017] Felipe Petroski Such, Vashisht Madha- van, Edoardo Conti, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Deep neuroevolution: Genetic algo- rithms are a competitive alternative for training deep neu- ral networks for reinforcement learning. arXiv preprint arXiv:1712.06567, 2017.","title":"Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning","authors":"Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O Stanley, Jeff Clune","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Yannakakis and Togelius, 2018] Georgios N. Yannakakis and Julian Togelius. Artificial Intelligence and Games. Springer, 2018. http://gameaibook.org.","title":"Artificial Intelligence and Games","authors":"Georgios N Yannakakis, Julian Togelius","year":"2018","venue":null,"doi":null,"url":"http://gameaibook.org","internal_url":null},{"position":17,"raw":"[Yao, 1999] Xin Yao. Evolving artificial neural networks. Proceedings of the IEEE, 87(9):1423-1447, 1999.","title":"Evolving artificial neural networks","authors":"Xin Yao","year":"1999","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}