{"type":"ScholarlyArticle","title":"Meta Reinforcement Learning with Task Embedding and Shared Policy","authors":[{"name":"Lin Lan","slug":"lin-lan","given":"Lin","family":"Lan","affiliation":"MOE NSKEY Lab , Xi'an Jiaotong University , China"},{"name":"Zhenguo Li","slug":"zhenguo-li","given":"Zhenguo","family":"Li","affiliation":"Huawei Noah's Ark Lab"},{"name":"Xiaohong Guan","slug":"xiaohong-guan","given":"Xiaohong","family":"Guan","affiliation":"MOE NSKEY Lab , Xi'an Jiaotong University , China; Shenzhen Research School , Xi'an Jiaotong University , China; Department of Automation and NLIST Lab , Tsinghua University , China"},{"name":"Pinghui Wang","slug":"pinghui-wang","given":"Pinghui","family":"Wang","affiliation":"Shenzhen Research School , Xi'an Jiaotong University , China; MOE NSKEY Lab , Xi'an Jiaotong University , China"}],"abstract":"Despite significant progress, deep reinforcement learning (RL) suffers from data-inefficiency and limited generalization. Recent efforts apply meta-learning to learn a meta-learner from a set of RL tasks such that a novel but related task could be solved quickly. Though specific in some ways, different tasks in meta-RL are generally similar at a high level. However, most meta-RL methods do not explicitly and adequately model the specific and shared information among different tasks, which limits their ability to learn training tasks and to generalize to novel tasks. In this paper, we propose to capture the shared information on the one hand and meta-learn how to quickly abstract the specific information about a task on the other hand. Methodologically, we train an SGD meta-learner to quickly optimize a task encoder for each task, which generates a task embedding based on past experience. Meanwhile, we learn a policy which is shared across all tasks and conditioned on task embeddings. Empirical results on four simulated tasks demonstrate that our method has better learning capacity on both training and novel tasks and attains up to 3 to 4 times higher returns compared to baselines.","doi":"10.24963/ijcai.2019/387","url":"https://ijcai.org/proceedings/2019/387","pdf":"https://ijcai.org/proceedings/2019/0387.pdf","markdown":"https://ijcai.org/proceedings/2019/387.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/387","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":387,"pages":{"first":2794,"last":2800},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning: Transfer, Adaptation, Multi-task Learning"],"videos":[],"references":[{"position":1,"raw":"[Al-Shedivat et al., 2017] Maruan Al-Shedivat, Trapit Bansal, Yuri Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. arXiv preprint arXiv:1710.03641, 2017.","title":"Maruan Al-Shedivat","authors":"Al-Shedivat","year":"2017","venue":"Continuous adaptation via meta-learning in nonstationary and competitive environments","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Clavera et al., 2018] Ignasi Clavera, Anusha Nagabandi, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt: Meta-learning for model- based control. arXiv preprint arXiv:1803.11347, 2018.","title":"Learning to adapt: Meta-learning for modelbased control","authors":"Ignasi Clavera, Anusha Nagabandi, Ronald S Fearing, Pieter Abbeel, Sergey Levine, Chelsea Finn","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Duan et al., 2016] Yan Duan, John Schulman, Xi Chen, Pe- ter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RL 2 : Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779, 2016.","title":"RL 2 : Fast reinforcement learning via slow reinforcement learning","authors":"Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, Pieter Abbeel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv:1703.03400, 2017.","title":"Model-agnostic meta-learning for fast adaptation of deep networks","authors":"Finn","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Florensa et al., 2017] Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical rein- forcement learning. In ICLR, 2017.","title":"Stochastic neural networks for hierarchical reinforcement learning","authors":"Carlos Florensa, Yan Duan, Pieter Abbeel","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Frans et al., 2018] Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. In ICLR, 2018.","title":"Meta learning shared hierarchies","authors":"Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, John Schulman","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gupta et al., 2018a] Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta- learning for reinforcement learning. arXiv preprint arXiv:1806.04640, 2018.","title":"Unsupervised metalearning for reinforcement learning","authors":"Gupta","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gupta et al., 2018b] Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In NIPS, 2018.","title":"Metareinforcement learning of structured exploration strategies","authors":"Gupta","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Houthooft et al., 2018] Rein Houthooft, Richard Y Chen, Phillip Isola, Bradly C Stadie, Filip Wolski, Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. arXiv preprint arXiv:1802.04821, 2018.","title":"Evolved policy gradients","authors":"Rein Houthooft, Phillip Richard Y Chen, Bradly C Isola, Filip Stadie, Jonathan Wolski, Pieter Ho, Abbeel","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kang and Feng, 2018] Bingyi Kang and Jiashi Feng. Trans- ferable meta learning across domains. In UAI, 2018.","title":"Transferable meta learning across domains","authors":"Feng Kang, Bingyi Kang, Jiashi Feng","year":"2018","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Levine et al., 2016] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. JMLR, 2016.","title":"End-to-end training of deep visuomotor policies","authors":"Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel","year":"2016","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2017] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few shot learning. arXiv preprint arXiv:1707.09835, 2017.","title":"Meta-sgd: Learning to learn quickly for few shot learning","authors":"Zhenguo Li, Fengwei Zhou, Fei Chen, Hang Li","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous con- trol with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ma et al., 2018] Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In SIGKDD, 2018.","title":"Modeling task relationships in multi-task learning with multi-gate mixture-of-experts","authors":"Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, Ed H Chi","year":"2018","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mishra et al., 2017] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. Meta-learning with temporal convolutions. arXiv preprint arXiv:1707.03141, 2017.","title":"Meta-learning with temporal convolutions","authors":"Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, Pieter Abbeel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep rein- forcement learning. arXiv preprint arXiv:1312.5602, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ruder, 2017] Sebastian Ruder. An overview of multi- task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017.","title":"An overview of multitask learning in deep neural networks","authors":"Sebastian Ruder","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Julian Driessche, Ioannis Schrittwieser, Veda Antonoglou, Marc Panneershelvam, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Snell et al., 2017] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NIPS, 2017.","title":"Prototypical networks for few-shot learning","authors":"Jake Snell, Kevin Swersky, Richard Zemel","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Stadie et al., 2018] Bradly C Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some considerations on learning to explore via meta-reinforcement learning. arXiv preprint arXiv:1803.01118, 2018.","title":"Some considerations on learning to explore via meta-reinforcement learning","authors":"Ge Bradly C Stadie, Rein Yang, Xi Houthooft, Yan Chen, Yuhuai Duan, Pieter Wu, Ilya Abbeel, Sutskever","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sutton et al., 1998] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction. MIT press, 1998.","title":"Reinforcement learning: An introduction","authors":"Andrew G Richard S Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tirinzoni et al., 2018] Andrea Tirinzoni, Rafael Rodriguez Sanchez, and Marcello Restelli. Transfer of value functions via variational methods. In NIPS, 2018.","title":"Transfer of value functions via variational methods","authors":"Andrea Tirinzoni, Rafael Rodriguez Sanchez, Marcello Restelli","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. Mujoco: A physics engine for model-based control. In IROS, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine Learning, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2018a] Kelvin Xu, Ellis Ratner, Anca Dragan, Sergey Levine, and Chelsea Finn. Learning a prior over in- tent via meta-inverse reinforcement learning. arXiv preprint arXiv:1805.12573, 2018.","title":"Learning a prior over intent via meta-inverse reinforcement learning","authors":"Xu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xu et al., 2018b] Tianbing Xu, Qiang Liu, Liang Zhao, Wei Xu, and Jian Peng. Learning to explore with meta-policy gradient. arXiv preprint arXiv:1803.05044, 2018.","title":"Learning to explore with meta-policy gradient","authors":"Xu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xu et al., 2018c] Zhongwen Xu, Hado van Hasselt, and David Silver. Meta-gradient reinforcement learning. arXiv preprint arXiv:1805.09801, 2018.","title":"Meta-gradient reinforcement learning","authors":"Xu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}