{"type":"ScholarlyArticle","title":"Curriculum Learning for Cumulative Return Maximization","authors":[{"name":"Francesco Foglino","slug":"francesco-foglino","given":"Francesco","family":"Foglino","affiliation":"University of Leeds , United Kingdom"},{"name":"Christiano Coletto Christakou","slug":"christiano-coletto-christakou","given":"Christiano","family":"Coletto Christakou","affiliation":"University of Leeds , United Kingdom"},{"name":"Ricardo Luna Gutierrez","slug":"ricardo-luna-gutierrez","given":"Ricardo","family":"Luna Gutierrez","affiliation":"University of Leeds , United Kingdom"},{"name":"Matteo Leonetti","slug":"matteo-leonetti","given":"Matteo","family":"Leonetti","affiliation":"University of Leeds , United Kingdom"}],"abstract":"Curriculum learning has been successfully used in reinforcement learning to accelerate the learning process, through knowledge transfer between tasks of increasing complexity. Critical tasks, in which suboptimal exploratory actions must be minimized, can benefit from curriculum learning, and its ability to shape exploration through transfer. We propose a task sequencing algorithm maximizing the cumulative return, that is, the return obtained by the agent across all the learning episodes. By maximizing the cumulative return, the agent not only aims at achieving high rewards as fast as possible, but also at doing so while limiting suboptimal actions. We experimentally compare our task sequencing algorithm to several popular metaheuristic algorithms for combinatorial optimization, and show that it achieves significantly better performance on the problem of cumulative return maximization. Furthermore, we validate our algorithm on a critical task, optimizing a home controller for a micro energy grid.","doi":"10.24963/ijcai.2019/320","url":"https://ijcai.org/proceedings/2019/320","pdf":"https://ijcai.org/proceedings/2019/0320.pdf","markdown":"https://ijcai.org/proceedings/2019/320.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/320","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":320,"pages":{"first":2308,"last":2314},"keywords":["Machine Learning: Deep Learning","Machine Learning: Developmental Learning","Machine Learning: Reinforcement Learning","Machine Learning: Transfer, Adaptation, Multi-task Learning"],"videos":[],"references":[{"position":1,"raw":"[Asada et al., 1996] Minoru Asada, Shoichi Noda, Sukoya Tawaratsumida, and Koh Hosoda. Purposive behavior ac- quisition for a real robot by vision-based reinforcement learning. Machine learning, 23(2-3):279-303, 1996.","title":"Purposive behavior acquisition for a real robot by vision-based reinforcement learning","authors":"Minoru Asada, Shoichi Noda, Sukoya Tawaratsumida, Koh Hosoda","year":"1996","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baranes and Oudeyer, 2013] Adrien Baranes and Pierre- Yves Oudeyer. Active learning of inverse models with in- trinsically motivated goal exploration in robots. Robotics and Autonomous Systems, 61(1):49-73, 2013.","title":"Active learning of inverse models with intrinsically motivated goal exploration in robots","authors":"Oudeyer Baranes, Adrien Baranes, Pierre-Yves Oudeyer","year":"2013","venue":"Robotics and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Da Silva and Costa, 2018] Felipe Leno Da Silva and Anna Helena Reali Costa. Object-oriented curriculum genera- tion for reinforcement learning. In International Confer- ence on Autonomous Agents and Multiagent Systems (AA- MAS), 2018.","title":"Felipe Leno Da Silva and Anna Helena Reali Costa. Object-oriented curriculum generation for reinforcement learning","authors":"Da Silva, Costa","year":"2018","venue":"International Conference on Autonomous Agents and Multiagent Systems (AA-MAS)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Da Silva and Costa, 2019] Felipe Leno Da Silva and Anna Helena Reali Costa. A survey on transfer learning for mul- tiagent reinforcement learning systems. Journal of Artifi- cial Intelligence Research, 64:645-703, 2019.","title":"Felipe Leno Da Silva and Anna Helena Reali Costa. A survey on transfer learning for multiagent reinforcement learning systems","authors":"Da Silva, Costa","year":"2019","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dorigo et al., 1991] Marco Dorigo, Vittorio Maniezzo, and Alberto Colorni. The ant system: An autocatalytic opti- mizing process, 1991.","title":"The ant system: An autocatalytic optimizing process","authors":"Marco Dorigo, Vittorio Maniezzo, Alberto Colorni","year":"1991","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Florensa et al., 2018] Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal gener- ation for reinforcement learning agents. In International Conference on Machine Learning (ICML), 2018.","title":"Automatic goal generation for reinforcement learning agents","authors":"Carlos Florensa, David Held, Xinyang Geng, Pieter Abbeel","year":"2018","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Foglino et al., 2019] Francesco Foglino, Christiano Co- letto Christakou, and Matteo Leonetti. An optimization framework for task sequencing in curriculum learning. In Proceedings of 9th Joint IEEE International Conference on Development and Learning and on Epigenetic Robotics (ICDL-EpiRob), 2019.","title":"An optimization framework for task sequencing in curriculum learning","authors":"Francesco Foglino, Christiano Coletto Christakou, Matteo Leonetti","year":"2019","venue":"Proceedings of 9th Joint IEEE International Conference on Development and Learning and on Epigenetic Robotics (ICDL-EpiRob)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Glover, 1989] Fred Glover. Tabu search-part i. ORSA Journal on computing, 1(3):190-206, 1989.","title":"Fred Glover. Tabu search-part i","authors":"Glover","year":"1989","venue":"ORSA Journal on computing","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goldberg, 1989] David E Goldberg. Genetic algorithms in search. Optimization, and MachineLearning, 1989.","title":"Genetic algorithms in search","authors":"David E Goldberg","year":"1989","venue":"Optimization, and MachineLearning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jaksch et al., 2010] Thomas Jaksch, Ronald Ortner, and Pe- ter Auer. Near-optimal regret bounds for reinforce- ment learning. Journal of Machine Learning Research, 11(Apr):1563-1600, 2010.","title":"Near-optimal regret bounds for reinforcement learning","authors":"Thomas Jaksch, Ronald Ortner, Peter Auer","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lazaric, 2012] Alessandro Lazaric. Transfer in reinforce- ment learning: a framework and a survey. In Reinforce- ment Learning, pages 143-173. Springer, 2012.","title":"Transfer in reinforcement learning: a framework and a survey","authors":"Alessandro Lazaric","year":"2012","venue":"Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Narvekar et al., 2016] Sanmit Narvekar, Jivko Sinapov, Matteo Leonetti, and Peter Stone. Source task creation for curriculum learning. In International Conference on Au- tonomous Agents and Multiagent Systems (AAMAS), 2016.","title":"Source task creation for curriculum learning","authors":"Narvekar","year":"2016","venue":"International Conference on Autonomous Agents and Multiagent Systems (AAMAS)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Narvekar et al., 2017] Sanmit Narvekar, Jivko Sinapov, and Peter Stone. Autonomous task sequencing for customized curriculum design in reinforcement learning. In (IJCAI), The 2017 International Joint Conference on Artificial In- telligence, 2017.","title":"Autonomous task sequencing for customized curriculum design in reinforcement learning","authors":"Narvekar","year":"2017","venue":"The 2017 International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2017/353"},{"position":14,"raw":"[Riedmiller et al., 2018] Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom van de Wiele, Vlad Mnih, Nicolas Heess, and Jost To- bias Springenberg. Learning by playing solving sparse re- ward tasks from scratch. In Proceedings of the 35th Inter- national Conference on Machine Learning, pages 4344- 4353, 2018.","title":"Learning by playing solving sparse reward tasks from scratch","authors":"Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van De Wiele, Vlad Mnih, Nicolas Heess, Jost","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rusu et al., 2016] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Had- sell. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016.","title":"Progressive neural networks","authors":"Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, Raia Hadsell","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ruvolo and Eaton, 2013] Paul Ruvolo and Eric Eaton. Ella: An efficient lifelong learning algorithm. In In Proceedings of the 30th International Conference on Machine Learn- ing, pages 507-515, 2013.","title":"Ella: An efficient lifelong learning algorithm","authors":"Eaton ; ] Ruvolo, Ruvolo Paul, Eric Eaton","year":"2013","venue":"Proceedings of the 30th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shao et al., 2018] K. Shao, Y. Zhu, and D. Zhao. Starcraft micromanagement with reinforcement learning and cur- riculum transfer learning. IEEE Transactions on Emerging Topics in Computational Intelligence, pages 1-12, 2018.","title":"Starcraft micromanagement with reinforcement learning and curriculum transfer learning","authors":"K Shao, Y Zhu, D Zhao","year":"2018","venue":"IEEE Transactions on Emerging Topics in Computational Intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sukhbaatar et al., 2018] Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. In International Conference on Learning Representations (ICLR), 2018.","title":"Intrinsic motivation and automatic curricula via asymmetric self-play","authors":"Sukhbaatar","year":"2018","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Svetlik et al., 2017] M Svetlik, M Leonetti, J Sinapov, R Shah, N Walker, and P Stone. Automatic curriculum graph generation for reinforcement learning agents. In Thirty-First AAAI Conference on Artificial Intelligence. Association for the Advancement of Artificial Intelligence, 2017.","title":"Automatic curriculum graph generation for reinforcement learning agents","authors":"M Svetlik, J Leonetti, R Sinapov, N Shah, Walker, Stone","year":"2017","venue":"Thirty-First AAAI Conference on Artificial Intelligence. Association for the Advancement of Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Taylor and Stone, 2009] Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(Jul):1633-1685, 2009.","title":"Transfer learning for reinforcement learning domains: A survey","authors":"E Matthew, Peter Taylor, Stone","year":"2009","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wilson et al., 2007] Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learn- ing: a hierarchical bayesian approach. In Proceedings of the 24th international conference on Machine learning, pages 1015-1022. ACM, 2007.","title":"Multi-task reinforcement learning: a hierarchical bayesian approach","authors":"Aaron Wilson, Alan Fern, Soumya Ray, Prasad Tadepalli","year":"2007","venue":"Proceedings of the 24th international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu and Tian, 2017] Yuxin Wu and Yuandong Tian. Train- ing agent for first-person shooter game with actor-critic curriculum learning. In International Conference on Learning Representations (ICLR), 2017.","title":"Training agent for first-person shooter game with actor-critic curriculum learning","authors":"Tian ; ] Yuxin Wu, Yuandong Wu, Tian","year":"2017","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2018] Yuechen Wu, Wei Zhang, and Ke Song. Master-slave curriculum design for reinforcement learn- ing. In Proceedings of the Twenty-Seventh Interna- tional Joint Conference on Artificial Intelligence, IJCAI- 18, pages 1523-1529. International Joint Conferences on Artificial Intelligence Organization, 7 2018.","title":"Master-slave curriculum design for reinforcement learning","authors":"Wu","year":"2018","venue":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18","doi":null,"url":null,"internal_url":"/proceedings/2018/211"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}