{"type":"ScholarlyArticle","title":"Emergent Tangled Program Graphs in Multi-Task Learning","authors":[{"name":"Stephen Kelly","slug":"stephen-kelly","given":"Stephen","family":"Kelly","affiliation":"Dalhousie University"},{"name":"Malcolm Heywood","slug":"malcolm-heywood","given":"Malcolm","family":"Heywood","affiliation":"Dalhousie University"}],"abstract":"We propose a Genetic Programming (GP) framework to address high-dimensional Multi-Task Reinforcement Learning (MTRL) through emergent modularity. A bottom-up process is assumed in which multiple programs self-organize into collective decision-making entities, or teams, which then further develop into multi-team policy graphs, or Tangled Program Graphs (TPG). The framework learns to play three Atari video games simultaneously, producing a single control policy that matches or exceeds leading results from (game-specific) deep reinforcement learning in each game. More importantly, unlike the representation assumed for deep learning, TPG policies start simple and adaptively complexify through interaction with the task environment, resulting in agents that are exceedingly simple, operating in real-time without specialized hardware support such as GPUs.","doi":"10.24963/ijcai.2018/740","url":"https://ijcai.org/proceedings/2018/740","pdf":"https://ijcai.org/proceedings/2018/0740.pdf","markdown":"https://ijcai.org/proceedings/2018/740.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/740","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Sister Conferences Best Papers","track":"Best Sister Conferences","number":740,"pages":{"first":5294,"last":5298},"keywords":["Machine Learning: Reinforcement Learning","Multidisciplinary Topics and Applications: Computer Games","Machine Learning: Transfer, Adaptation, Multi-task Learning"],"videos":[],"references":[{"position":1,"raw":"[Bellemare et al., 2012] Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 2012.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"G Marc, Yavar Bellemare, Joel Naddaf, Michael Veness, Bowling","year":"2012","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brameier and Banzhaf, 2007] Markus Brameier and Wolf- gang Banzhaf. Linear Genetic Programming. Springer, 1st edition, 2007.","title":"Markus Brameier and Wolfgang","authors":"Banzhaf ; Brameier, Banzhaf","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Doucette et al., 2012] John A. Doucette, Peter Lichodzijew- ski, and Malcolm I. Heywood. Hierarchical task decom- position through symbiosis in reinforcement learning. In Proceedings of the ACM Genetic and Evolutionary Com- putation Conference, pages 97-104, 2012.","title":"Hierarchical task decomposition through symbiosis in reinforcement learning","authors":"John A Doucette, Peter Lichodzijewski, Malcolm I Heywood","year":"2012","venue":"Proceedings of the ACM Genetic and Evolutionary Computation Conference","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hausknecht et al., 2014] Matthew Hausknecht, Joel Lehman, Risto Miikkulainen, and Peter Stone. A neu- roevolution approach to general Atari game playing. IEEE Transactions on Computational Intelligence and AI in Games, 6(4):355-366, 2014.","title":"A neuroevolution approach to general Atari game playing","authors":"Matthew Hausknecht, Joel Lehman, Risto Miikkulainen, Peter Stone","year":"2014","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Kelly and Heywood, 2014] Stephen Kelly and Malcolm I. Heywood. On diversity, teaming, and hierarchical poli- cies: Observations from the keepaway soccer task. In European Conference on Genetic Programming, volume 8599 of LNCS, pages 75-86. Springer, 2014.","title":"On diversity, teaming, and hierarchical policies: Observations from the keepaway soccer task","authors":"Stephen Kelly, Malcolm I Heywood","year":"2014","venue":"European Conference on Genetic Programming","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kelly and Heywood, 2015] Stephen Kelly and Malcolm I. Heywood. Knowledge transfer from keepaway soccer to half-field offense through program symbiosis: Building simple programs for a complex task. In Proceedings of the ACM Genetic and Evolutionary Computation Conference, pages 1143-1150, 2015.","title":"Knowledge transfer from keepaway soccer to half-field offense through program symbiosis: Building simple programs for a complex task","authors":"Stephen Kelly, Malcolm I Heywood","year":"2015","venue":"Proceedings of the ACM Genetic and Evolutionary Computation Conference","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kelly and Heywood, 2017a] Stephen Kelly and Malcolm I. Heywood. Emergent tangled graph representations for Atari game playing agents. In European Conference on Genetic Programming, volume 10196 of LNCS, pages 64- 79, 2017.","title":"Emergent tangled graph representations for Atari game playing agents","authors":"Heywood ; Kelly, Stephen Kelly, Malcolm I Heywood","year":"2017","venue":"European Conference on Genetic Programming","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kelly and Heywood, 2017b] Stephen Kelly and Malcolm I. Heywood. Multi-task learning in atari video games with emergent tangled program graphs. In Proceedings of the ACM Genetic and Evolutionary Computation Conference, 2017.","title":"Multi-task learning in atari video games with emergent tangled program graphs","authors":"Heywood ; Kelly, Stephen Kelly, Malcolm I Heywood","year":"2017","venue":"Proceedings of the ACM Genetic and Evolutionary Computation Conference","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kelly et al., 2012] Stephen Kelly, Peter Lichodzijewski, and Malcolm I. Heywood. On run time libraries and hierarchi- cal symbiosis. In IEEE Congress on Evolutionary Compu- tation, pages 3245-3252, 2012.","title":"On run time libraries and hierarchical symbiosis","authors":"Stephen Kelly, Peter Lichodzijewski, Malcolm I Heywood","year":"2012","venue":"IEEE Congress on Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kelly et al., 2018] Stephen Kelly, Robert Smith, and Mal- colm I. Heywood. Emergent policy discovery for visual reinforcement learning through tangled program graphs: A tutorial. In Genetic Programming Theory and Practice XVI. Springer, 2018.","title":"Emergent policy discovery for visual reinforcement learning through tangled program graphs: A tutorial","authors":"Kelly","year":"2018","venue":"Genetic Programming Theory and Practice XVI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kelly, 2018] Stephen Kelly. Scaling genetic programming to challenging reinforcement tasks through emergent mod- ularity. PhD thesis, Faculty of Computer Science, Dal- housie University, 2018.","title":"Scaling genetic programming to challenging reinforcement tasks through emergent modularity","authors":"Stephen Kelly","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kirkpatrick et al., 2016] James Kirkpatrick, Razvan Pas- canu, Neil Rabinowitz, Joel Veness, Guillaume Des- jardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Had- sell. Overcoming catastrophic forgetting in neural net- works. arXiv preprint 1612.00796, 2016.","title":"Overcoming catastrophic forgetting in neural networks","authors":"James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, Raia Hadsell","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lichodzijewski and Heywood, 2010] Peter Lichodzijewski and Malcolm I. Heywood. Symbiosis, complexification and simplicity under GP. In Proceedings of the ACM Genetic and Evolutionary Computation Conference, pages 853-860, 2010.","title":"Peter Lichodzijewski and Malcolm I. Heywood. Symbiosis, complexification and simplicity under GP","authors":"Heywood Lichodzijewski","year":"2010","venue":"Proceedings of the ACM Genetic and Evolutionary Computation Conference","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lichodzijewski and Heywood, 2011] Peter Lichodzijewski and Malcolm I. Heywood. The Rubik cube and GP tem- poral sequence learning: an initial study. In Genetic Pro- gramming Theory and Practice VIII, chapter 3, pages 35- 54. Springer, 2011.","title":"Peter Lichodzijewski and Malcolm I. Heywood. The Rubik cube and GP temporal sequence learning: an initial study","authors":"Heywood Lichodzijewski","year":"2011","venue":"Genetic Programming Theory and Practice VIII","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Machado et al., 2017] Marlos C. Machado, Marc G. Belle- mare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environ- ment: Evaluation protocols and open problems for general agents. arXiv preprint 1709.06009, 2017.","title":"Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents","authors":"C Marlos, Marc G Machado, Erik Bellemare, Joel Talvitie, Matthew Veness, Michael Hausknecht, Bowling","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel. Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Has- sabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Naddaf, 2010] Yavar Naddaf. Game-independent AI agents for playing Atari 2600 console games. Masters thesis, Uni- versity of Alberta, 2010.","title":"Game-independent AI agents for playing Atari 2600 console games","authors":"Yavar Naddaf, Naddaf","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nolfi, 1997] Stefano Nolfi. Using emergent modularity to develop control systems for mobile robots. Adaptive be- havior, 5(3-4):343-363, 1997.","title":"Using emergent modularity to develop control systems for mobile robots","authors":"Stefano Nolfi","year":"1997","venue":"Adaptive behavior","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[van Steenkiste et al., 2016] Sjoerd van Steenkiste, Jan Koutník, Kurt Driessens, and Jürgen Schmidhuber. A wavelet-based encoding for neuroevolution. In Pro- ceedings of the Genetic and Evolutionary Computation Conference, pages 517-524, 2016.","title":"A wavelet-based encoding for neuroevolution","authors":"Van Steenkiste","year":"2016","venue":"Proceedings of the Genetic and Evolutionary Computation Conference","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yannakakis and Togelius, 2015] Georgios N. Yannakakis and Julian Togelius. A panorama of artificial and com- putational intelligence in games. IEEE Transactions on Computational Intelligence and AI in Games, 7(4):317- 335, 2015.","title":"A panorama of artificial and computational intelligence in games","authors":"Georgios N Yannakakis, Julian Togelius","year":"2015","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}