{"type":"ScholarlyArticle","title":"Master-Slave Curriculum Design for Reinforcement Learning","authors":[{"name":"Yuechen Wu","slug":"yuechen-wu","given":"Yuechen","family":"Wu","affiliation":"School of Control Science and Engineering , Shandong University"},{"name":"Wei Zhang","slug":"wei-zhang","given":"Wei","family":"Zhang","affiliation":"School of Control Science and Engineering , Shandong University"},{"name":"Ke Song","slug":"ke-song","given":"Ke","family":"Song","affiliation":"School of Control Science and Engineering , Shandong University"}],"abstract":"Curriculum learning is often introduced as a leverage to improve the agent training for complex tasks, where the goal is to generate a sequence of easier subasks for an agent to train on, such that final performance or learning speed is improved. However, conventional curriculum is mainly designed for one agent with fixed action space and sequential simple-to-hard training manner. Instead, we present a novel curriculum learning strategy by introducing the concept of master-slave agents and enabling flexible action setting for agent training. Multiple agents, referred as master agent for the target task and slave agents for the subtasks, are trained concurrently within different action spaces by sharing a perception network with an asynchronous strategy. Extensive evaluation on the VizDoom platform demonstrates the joint learning of master agent and slave agents mutually benefit each other. Significant improvement is obtained over A3C in terms of learning speed and performance.","doi":"10.24963/ijcai.2018/211","url":"https://ijcai.org/proceedings/2018/211","pdf":"https://ijcai.org/proceedings/2018/0211.pdf","markdown":"https://ijcai.org/proceedings/2018/211.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/211","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Heuristic Search and Game Playing","track":"Main track","number":211,"pages":{"first":1523,"last":1529},"keywords":["Heuristic Search and Game Playing: Game Playing and Machine Learning","Heuristic Search and Game Playing: General Game Playing and General Video Game Playing","Machine Learning: Reinforcement Learning","Heuristic Search and Game Playing: Game Playing"],"videos":[],"references":[{"position":1,"raw":"[Andreas et al., 2016] Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. arXiv preprint arXiv:1611.01796, 2016.","title":"Modular multitask reinforcement learning with policy sketches","authors":"Jacob Andreas, Dan Klein, Sergey Levine","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2009] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, pages 41-48, 2009.","title":"Curriculum learning","authors":"Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Busoniu et al., 2007] Lucian Busoniu, Robert Babuska, and Bart De Schutter. Multi-agent reinforcement learning: A survey. In International Conference on Control, Automa- tion, Robotics and Vision, pages 1-6, 2007.","title":"Multi-agent reinforcement learning: A survey","authors":"Lucian Busoniu, Robert Babuska, Bart De Schutter","year":"2007","venue":"International Conference on Control, Automation, Robotics and Vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Florensa et al., 2017] Carlos Florensa, David Held, Markus Wulfmeier, and Pieter Abbeel. Reverse curriculum gen- eration for reinforcement learning. arXiv preprint arX- iv:1707.05300, 2017.","title":"Reverse curriculum generation for reinforcement learning","authors":"Carlos Florensa, David Held, Markus Wulfmeier, Pieter Abbeel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jaderberg et al., 2016] Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397, 2016.","title":"Reinforcement learning with unsupervised auxiliary tasks","authors":"Jaderberg","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jiang et al., 2015] Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G Hauptmann. Self-paced curriculum learning. In AAAI, page 6, 2015.","title":"Self-paced curriculum learning","authors":"Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, Alexander G Hauptmann","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Karpathy and Van De Panne, 2012] Andrej Karpathy and Michiel Van De Panne. Curriculum learning for motor skills. In Canadian Conference on Artificial Intelligence, pages 325-330, 2012.","title":"Curriculum learning for motor skills","authors":"Van Karpathy, Andrej De Panne, Michiel Karpathy, Van De Panne","year":"2012","venue":"Canadian Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kempka et al., 2016] Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaśkowski. Vizdoom: A doom-based ai research platform for visu- al reinforcement learning. In Computational Intelligence and Games (CIG), 2016 IEEE Conference on, pages 1-8, 2016.","title":"Vizdoom: A doom-based ai research platform for visual reinforcement learning","authors":"Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, Wojciech Jaśkowski","year":"2016","venue":"Computational Intelligence and Games (CIG)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kulkarni et al., 2016] Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In NIPS, pages 3675-3683, 2016.","title":"Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation","authors":"D Tejas, Karthik Kulkarni, Ardavan Narasimhan, Josh Saeedi, Tenenbaum","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kumar et al., 2010] M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In NIPS, pages 1189-1197, 2010.","title":"Self-paced learning for latent variable models","authors":"Benjamin M Pawan Kumar, Daphne Packer, Koller","year":"2010","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Levine et al., 2016] Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learn- ing hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research, 2016.","title":"Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection","authors":"Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, Deirdre Quillen","year":"2016","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2009] Hui Li, Xuejun Liao, and Lawrence Carin. Multi-task reinforcement learning in partially observable stochastic environments. Journal of Machine Learning Re- search, 2009.","title":"Multi-task reinforcement learning in partially observable stochastic environments","authors":"Hui Li, Xuejun Liao, Lawrence Carin","year":"2009","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In ICML, pages 1928-1937, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Narvekar et al., 2016] Sanmit Narvekar, Jivko Sinapov, Matteo Leonetti, and Peter Stone. Source task creation for curriculum learning. In AAMAS, pages 566-574, 2016.","title":"Source task creation for curriculum learning","authors":"Narvekar","year":"2016","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Narvekar et al., 2017] Sanmit Narvekar, Jivko Sinapov, and Peter Stone. Autonomous task sequencing for customized curriculum design in reinforcement learning. In IJCAI, pages 2536-2542, 2017.","title":"Autonomous task sequencing for customized curriculum design in reinforcement learning","authors":"Narvekar","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/353"},{"position":17,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press Cambridge, 1998.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Svetlik et al., 2017] Maxwell Svetlik, Matteo Leonetti, Jivko Sinapov, Rishi Shah, Nick Walker, and Peter Stone. Automatic curriculum graph generation for reinforcement learning agents. In AAAI, pages 2590-2596, 2017.","title":"Automatic curriculum graph generation for reinforcement learning agents","authors":"Maxwell Svetlik, Matteo Leonetti, Jivko Sinapov, Rishi Shah, Nick Walker, Peter Stone","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tessler et al., 2017] Chen Tessler, Shahar Givony, Tom Za- havy, Daniel J Mankowitz, and Shie Mannor. A deep hi- erarchical approach to lifelong learning in minecraft. In AAAI, volume 3, page 6, 2017.","title":"A deep hierarchical approach to lifelong learning in minecraft","authors":"Chen Tessler, Shahar Givony, Tom Zahavy, Daniel J Mankowitz, Shie Mannor","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wilson et al., 2007] Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learn- ing: a hierarchical bayesian approach. In ICML, pages 1015-1022, 2007.","title":"Multi-task reinforcement learning: a hierarchical bayesian approach","authors":"Aaron Wilson, Alan Fern, Soumya Ray, Prasad Tadepalli","year":"2007","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wu and Tian, 2017] Yuxin Wu and Yuandong Tian. Train- ing agent for first-person shooter game with actor-critic curriculum learning. ICLR, 2017.","title":"Training agent for first-person shooter game with actor-critic curriculum learning","authors":"Tian ; ] Yuxin Wu, Yuandong Wu, Tian","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yang et al., 2017] Zhaoyang Yang, Kathryn Merrick, Hus- sein Abbass, and Lianwen Jin. Multi-task deep reinforce- ment learning for continuous action control. In IJCAI, pages 3301-3307, 2017.","title":"Multi-task deep reinforcement learning for continuous action control","authors":"Yang","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/461"},{"position":23,"raw":"[Zhang et al., 2018a] Wei Zhang, Qi Chen, Weidong Zhang, and Xuanyu He. Long-range terrain perception using con- volutional neural networks. Neurocomputing, 275:781- 787, 2018.","title":"Long-range terrain perception using convolutional neural networks","authors":"Zhang","year":"2018","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2018b] Wei Zhang, Weidong Zhang, Kan Liu, and Jason Gu. A feature descriptor based on local normal- ized difference for real-world texture classification. IEEE Transactions on Multimedia, 20(4):880-888, 2018.","title":"A feature descriptor based on local normalized difference for real-world texture classification","authors":"Zhang","year":"2018","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}