{"type":"ScholarlyArticle","title":"Autonomous Task Sequencing for Customized Curriculum Design in Reinforcement Learning","authors":[{"name":"Sanmit Narvekar","slug":"sanmit-narvekar","given":"Sanmit","family":"Narvekar","affiliation":"Department of Computer Science , University of Texas at Austin"},{"name":"Jivko Sinapov","slug":"jivko-sinapov","given":"Jivko","family":"Sinapov","affiliation":"Department of Computer Science , University of Texas at Austin"},{"name":"Peter Stone","slug":"peter-stone","given":"Peter","family":"Stone","affiliation":"Department of Computer Science , University of Texas at Austin"}],"abstract":"Transfer learning is a method where an agent reuses knowledge learned in a source task to improve learning on a target task. Recent work has shown that transfer learning can be extended to the idea of curriculum learning, where the agent incrementally accumulates knowledge over a sequence of tasks (i.e. a curriculum). In most existing work, such curricula have been constructed manually. Furthermore, they are fixed ahead of time, and do not adapt to the progress or abilities of the agent. In this paper, we formulate the design of a curriculum as a Markov Decision Process, which directly models the accumulation of knowledge as an agent interacts with tasks, and propose a method that approximates an execution of an optimal policy in this MDP to produce an agent-specific curriculum. We use our approach to automatically sequence tasks for 3 agents with varying sensing and action capabilities in an experimental domain, and show that our method produces curricula customized for each agent that improve performance relative to learning from scratch or using a different agent's curriculum.","doi":"10.24963/ijcai.2017/353","url":"https://ijcai.org/proceedings/2017/353","pdf":"https://ijcai.org/proceedings/2017/0353.pdf","markdown":"https://ijcai.org/proceedings/2017/353.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/353","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":353,"pages":{"first":2536,"last":2542},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning: Transfer, Adaptation, Multi-task Learning"],"videos":[],"references":[{"position":1,"raw":"[Ammar et al., 2014a] Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, and Matthew Taylor. Online multi-task learn- ing for policy gradient methods. In Proceedings of the 31st International Conference on Machine Learning (ICML- 14), pages 1206-1214, 2014.","title":"Online multi-task learning for policy gradient methods","authors":"Ammar","year":"2014","venue":"Proceedings of the 31st International Conference on Machine Learning (ICML-14)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ammar et al., 2014b] Haitham Bou Ammar, Eric Eaton, Matthew E Taylor, Decebal Constantin Mocanu, Kurt Driessens, Gerhard Weiss, and Karl Tuyls. An automated measure of mdp similarity for transfer in reinforcement learning. In Workshops at the Twenty-Eighth AAAI Con- ference on Artificial Intelligence, 2014.","title":"An automated measure of mdp similarity for transfer in reinforcement learning","authors":"Ammar","year":"2014","venue":"Workshops at the Twenty-Eighth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio et al., 2009] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Confer- ence on Machine Learning, pages 41-48. ACM, 2009.","title":"Curriculum learning","authors":"Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston","year":"2009","venue":"Proceedings of the 26th Annual International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fachantidis et al., 2013] Anestis Fachantidis, Ioannis Parta- las, Grigorios Tsoumakas, and Ioannis Vlahavas. Transfer- ring task models in reinforcement learning agents. Neuro- computing, 107:23-32, 2013.","title":"Transferring task models in reinforcement learning agents","authors":"Anestis Fachantidis, Ioannis Partalas, Grigorios Tsoumakas, Ioannis Vlahavas","year":"2013","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fernández et al., 2010] Fernando Fernández, Javier García, and Manuela Veloso. Probabilistic policy reuse for inter- task transfer learning. Robotics and Autonomous Systems, 58(7):866 -871, 2010.","title":"Probabilistic policy reuse for intertask transfer learning","authors":"Fernando Fernández, Javier García, Manuela Veloso","year":"2010","venue":"Robotics and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ferns et al., 2011] Norm Ferns, Prakash Panangaden, and Doina Precup. Bisimulation metrics for continuous markov decision processes. SIAM Journal on Computing, 40(6):1662-1714, 2011.","title":"Bisimulation metrics for continuous markov decision processes","authors":"Ferns","year":"2011","venue":"SIAM Journal on Computing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ferns et al., 2012] Norm Ferns, Pablo Samuel Castro, Doina Precup, and Prakash Panangaden. Methods for computing state similarity in markov decision processes. arXiv preprint arXiv:1206.6836, 2012.","title":"Norm Ferns, Pablo Samuel Castro","authors":"Ferns","year":"2012","venue":"Methods for computing state similarity in markov decision processes","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Isele et al., 2016] David Isele, Mohammad Rostami, and Eric Eaton. Using task features for zero-shot knowledge transfer in lifelong learning. In Proceedings of the 25th In- ternational Joint Conference on Artificial Intelligence (IJ- CAI), pages 1620-1626, 2016.","title":"Using task features for zero-shot knowledge transfer in lifelong learning","authors":"David Isele, Mohammad Rostami, Eric Eaton","year":"2016","venue":"Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJ-CAI)","doi":null,"url":null,"internal_url":"/proceedings/2016/232"},{"position":9,"raw":"[Konidaris and Barto, 2007] George Konidaris and An- drew G Barto. Building portable options: Skill transfer in reinforcement learning. In IJCAI, volume 7, pages 895-900, 2007.","title":"George Konidaris and Andrew G Barto. Building portable options: Skill transfer in reinforcement learning","authors":"Barto Konidaris","year":"2007","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lazaric et al., 2008] A. Lazaric, M. Restelli, and A. Bonar- ini. Transfer of samples in batch reinforcement learn- ing. In Proceedings of the Twenty-Fifth Annual Inter- national Conference on Machine Learning (ICML-2008), pages 544-551, Helsinki, Finalnd, July 2008.","title":"Transfer of samples in batch reinforcement learning","authors":"A Lazaric, M Restelli, A Bonarini","year":"2008","venue":"Proceedings of the Twenty-Fifth Annual International Conference on Machine Learning (ICML-2008)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lazaric, 2011] A. Lazaric. Transfer in reinforcement learn- ing: a framework and a survey. In Reinforcement Learn- ing: State of the Art. Springer, 2011.","title":"Transfer in reinforcement learning: a framework and a survey","authors":"A Lazaric","year":"2011","venue":"Reinforcement Learning: State of the Art","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Narvekar et al., 2016] Sanmit Narvekar, Jivko Sinapov, Matteo Leonetti, and Peter Stone. Source task creation for curriculum learning. In Proceedings of the 15th Interna- tional Conference on Autonomous Agents and Multiagent Systems (AAMAS 2016), May 2016.","title":"Source task creation for curriculum learning","authors":"Narvekar","year":"2016","venue":"Proceedings of the 15th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2016)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Peng et al., 2016] Bei Peng, James MacGlashan, Robert Loftin, Michael L. Littman, David L. Roberts, and Matthew E. Taylor. An empirical study of non-expert cur- riculum design for machine learners. In In Proceedings of the IJCAI Interactive Machine Learning Workshop, 2016.","title":"An empirical study of non-expert curriculum design for machine learners","authors":"Bei Peng, James Macglashan, Robert Loftin, Michael L Littman, David L Roberts, Matthew E Taylor","year":"2016","venue":"Proceedings of the IJCAI Interactive Machine Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Perkins et al., 1999] Theodore J Perkins, Doina Precup, et al. Using options for knowledge transfer in reinforce- ment learning. University of Massachusetts, Amherst, MA, USA, Tech. Rep, 1999.","title":"Using options for knowledge transfer in reinforcement learning","authors":"J Theodore, Doina Perkins, Precup","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sinapov et al., 2015] Jivko Sinapov, Sanmit Narvekar, Mat- teo Leonetti, and Peter Stone. Learning inter-task transfer- ability in the absence of target task samples. In Proceed- ings of the 2015 ACM Conference on Autonomous Agents and Multi-Agent Systems (AAMAS). ACM, 2015.","title":"Learning inter-task transferability in the absence of target task samples","authors":"Jivko Sinapov, Sanmit Narvekar, Matteo Leonetti, Peter Stone","year":"2015","venue":"Proceedings of the 2015 ACM Conference on Autonomous Agents and Multi-Agent Systems (AAMAS)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Soni and Singh, 2006] Vishal Soni and Satinder Singh. Us- ing homomorphisms to transfer options across continuous reinforcement learning domains. In Proceedings of Amer- ican Association for Artificial Intelligence (AAAI), 2006.","title":"Using homomorphisms to transfer options across continuous reinforcement learning domains","authors":"Vishal Soni, Satinder Singh","year":"2006","venue":"Proceedings of American Association for Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sutton and Barto, 1998] Richard Sutton and Andrew Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.","title":"Reinforcement Learning: An Introduction","authors":"Barto Sutton, Richard Sutton, Andrew Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Svetlik et al., 2017] Maxwell Svetlik, Matteo Leonetti, Jivko Sinapov, Rishi Shah, Nick Walker, and Peter Stone. Automatic curriculum graph generation for reinforcement learning agents. In Proceedings of the 31st AAAI Confer- ence on Artificial Intelligence (AAAI), February 2017.","title":"Automatic curriculum graph generation for reinforcement learning agents","authors":"Maxwell Svetlik, Matteo Leonetti, Jivko Sinapov, Rishi Shah, Nick Walker, Peter Stone","year":"2017","venue":"Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Taylor and Stone, 2005] Matthew E. Taylor and Peter Stone. Behavior transfer for value-function-based rein- forcement learning. In The Fourth International Joint Con- ference on Autonomous Agents and Multiagent Systems, pages 53-59, New York, NY, July 2005. ACM Press.","title":"Behavior transfer for value-function-based reinforcement learning","authors":"Matthew E Taylor, Peter Stone","year":"2005","venue":"The Fourth International Joint Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Taylor and Stone, 2009] Matthew E. Taylor and Peter Stone. Transfer learning for reinforcement learning do- mains: A survey. Journal of Machine Learning Research, 10(1):1633-1685, 2009.","title":"Transfer learning for reinforcement learning domains: A survey","authors":"Matthew E Taylor, Peter Stone","year":"2009","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Taylor et al., 2007] Matthew E. Taylor, Peter Stone, and Yaxin Liu. Transfer learning via inter-task mappings for temporal difference learning. Journal of Machine Learn- ing Research, 8(1):2125-2167, 2007.","title":"Transfer learning via inter-task mappings for temporal difference learning","authors":"Matthew E Taylor, Peter Stone, Yaxin Liu","year":"2007","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wilson et al., 2007] Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learn- ing: a hierarchical bayesian approach. In Proceedings of the 24th International Conference on Machine Learning, pages 1015-1022. ACM, 2007.","title":"Multi-task reinforcement learning: a hierarchical bayesian approach","authors":"Aaron Wilson, Alan Fern, Soumya Ray, Prasad Tadepalli","year":"2007","venue":"Proceedings of the 24th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}