{"type":"ScholarlyArticle","title":"Exploring Parameter Space with Structured Noise for Meta-Reinforcement Learning","authors":[{"name":"Hui Xu","slug":"hui-xu","given":"Hui","family":"Xu","affiliation":"University of Electronic Science and Technology of China"},{"name":"Chong Zhang","slug":"chong-zhang","given":"Chong","family":"Zhang","affiliation":"Tencent Robotics X"},{"name":"Jiaxing Wang","slug":"jiaxing-wang","given":"Jiaxing","family":"Wang","affiliation":"Institute of Automation , Chinese Academy of Sciences; University of Chinese Academy of Sciences"},{"name":"Deqiang Ouyang","slug":"deqiang-ouyang","given":"Deqiang","family":"Ouyang","affiliation":"University of Electronic Science and Technology of China"},{"name":"Yu Zheng","slug":"yu-zheng","given":"Yu","family":"Zheng","affiliation":"Tencent Robotics X"},{"name":"Jie Shao","slug":"jie-shao","given":"Jie","family":"Shao","affiliation":"University of Electronic Science and Technology of China; Sichuan Artificial Intelligence Research Institute"}],"abstract":"Efficient exploration is a major challenge in Reinforcement Learning (RL) and has been studied extensively. However, for a new task existing methods explore either by taking actions that maximize task agnostic objectives (such as information gain) or applying a simple dithering strategy (such as noise injection), which might not be effective enough. In this paper, we investigate whether previous learning experiences can be leveraged to guide exploration of current new task. To this end, we propose a novel Exploration with Structured Noise in Parameter Space (ESNPS) approach. ESNPS utilizes meta-learning and directly uses meta-policy parameters, which contain prior knowledge, as structured noises to perturb the base model for effective exploration in new tasks. Experimental results on four groups of tasks: cheetah velocity, cheetah direction, ant velocity and ant direction demonstrate the superiority of ESNPS against a number of competitive baselines.","doi":"10.24963/ijcai.2020/436","url":"https://ijcai.org/proceedings/2020/436","pdf":"https://ijcai.org/proceedings/2020/0436.pdf","markdown":"https://ijcai.org/proceedings/2020/436.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/436","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":436,"pages":{"first":3153,"last":3159},"keywords":["Machine Learning: Deep Reinforcement Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25126/file_25126.m3u8","length_sec":283,"provider":"hls"}],"references":[{"position":1,"raw":"[Bellemare et al., 2016] Marc G. Bellemare, Sriram Srini- vasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count-based exploration and intrin- sic motivation. In NIPS, pages 1471-1479, 2016.","title":"Unifying count-based exploration and intrinsic motivation","authors":"G Marc, Sriram Bellemare, Georg Srinivasan, Tom Ostrovski, David Schaul, Rémi Saxton, Munos","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brafman and Tennenholtz, 2001] Ronen I. Brafman and Moshe Tennenholtz. R-MAX -A general polynomial time algorithm for near-optimal reinforcement learning. In IJCAI, pages 953-958, 2001.","title":"R-MAX -A general polynomial time algorithm for near-optimal reinforcement learning","authors":"Ronen I Brafman, Moshe Tennenholtz","year":"2001","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2001/130"},{"position":3,"raw":"[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, pages 1126-1135, 2017.","title":"Model-agnostic meta-learning for fast adaptation of deep networks","authors":"Finn","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Finn et al., 2018] Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In NeurIPS, pages 9537-9548, 2018.","title":"Probabilistic model-agnostic meta-learning","authors":"Finn","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fortunato et al., 2018] Meire Fortunato, Mohammad Ghesh- laghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Os- band, Alex Graves, Volodymyr Mnih, Rémi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In ICLR, 2018.","title":"Noisy networks for exploration","authors":"Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Osband, Alex Graves, Volodymyr Mnih, Rémi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, Shane Legg","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Grant et al., 2018] Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas L. Griffiths. Recasting gradient- based meta-learning as hierarchical bayes. In ICLR, 2018.","title":"Recasting gradientbased meta-learning as hierarchical bayes","authors":"Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, Thomas L Griffiths","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gupta et al., 2018] Abhishek Gupta, Russell Mendonca, Yuxuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In NeurIPS, pages 5307-5316, 2018.","title":"Metareinforcement learning of structured exploration strategies","authors":"Abhishek Gupta, Russell Mendonca, Yuxuan Liu, Pieter Abbeel, Sergey Levine","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hochreiter et al., 2001] Sepp Hochreiter, A. Steven Younger, and Peter R. Conwell. Learning to learn using gradient de- scent. In ICANN, pages 87-94, 2001.","title":"Learning to learn using gradient descent","authors":"A Steven Sepp Hochreiter, Peter R Younger, Conwell","year":"2001","venue":"ICANN","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Houthooft et al., 2016] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: variational information maximizing exploration. In NIPS, pages 1109-1117, 2016.","title":"VIME: variational information maximizing exploration","authors":"Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, Pieter Abbeel","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hu et al., 2019] Wenqing Hu, Chris Junchi Li, Lei Li, and Jian-Guo Liu. Near-optimal reinforcement learning in poly- nomial time. Annals of Mathematical Sciences and Applica- tions, 4(1):3-32, 2019.","title":"Near-optimal reinforcement learning in polynomial time","authors":"Wenqing Hu, Chris Junchi Li, Lei Li, Jian-Guo Liu","year":"2019","venue":"Annals of Mathematical Sciences and Applications","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jin et al., 2017] Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M. Kakade, and Michael I. Jordan. How to escape sad- dle points efficiently. In ICML, pages 1724-1732, 2017.","title":"How to escape saddle points efficiently","authors":"Jin","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kearns and Singh, 2002] Michael J. Kearns and Satinder P. Singh. Near-optimal reinforcement learning in polynomial time. Machine Learning, 49(2-3):209-232, 2002.","title":"Near-optimal reinforcement learning in polynomial time","authors":"J Michael, Kearns, P Satinder, Singh","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Welling, 2014] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lan et al., 2019] Lin Lan, Zhenguo Li, Xiaohong Guan, and Pinghui Wang. Meta reinforcement learning with task embed- ding and shared policy. In IJCAI, pages 2794-2800, 2019.","title":"Meta reinforcement learning with task embedding and shared policy","authors":"Lin Lan, Zhenguo Li, Xiaohong Guan, Pinghui Wang","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/387"},{"position":15,"raw":"[Lemke et al., 2015] Christiane Lemke, Marcin Budka, and Bogdan Gabrys. Metalearning: a survey of trends and tech- nologies. Artif. Intell. Rev., 44(1):117-130, 2015.","title":"Metalearning: a survey of trends and technologies","authors":"Lemke","year":"2015","venue":"Artif. Intell. Rev","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2017] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few shot learning. arXiv preprint, arXiv:1707.09835, 2017.","title":"Meta-sgd: Learning to learn quickly for few shot learning","authors":"Zhenguo Li, Fengwei Zhou, Fei Chen, Hang Li","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mishra et al., 2017] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. Meta-learning with temporal convolutions. arXiv preprint, arXiv:1707.03141, 2017.","title":"Meta-learning with temporal convolutions","authors":"Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, Pieter Abbeel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mitchell and Thrun, 1992] Tom M. Mitchell and Sebastian Thrun. Explanation-based neural network learning for robot control. In NIPS, pages 287-294, 1992.","title":"Explanation-based neural network learning for robot control","authors":"Tom M Mitchell, Sebastian Thrun","year":"1992","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nichol et al., 2018] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint, arXiv:1803.02999, 2018.","title":"On first-order meta-learning algorithms","authors":"Alex Nichol, Joshua Achiam, John Schulman","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Osband et al., 2016] Ian Osband, Charles Blundell, Alexan- der Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. In NIPS, pages 4026-4034, 2016.","title":"Deep exploration via bootstrapped DQN","authors":"Ian Osband, Charles Blundell, Alexander Pritzel, Benjamin Van, Roy","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pathak et al., 2017] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven explo- ration by self-supervised prediction. In CVPR Workshops, pages 488-489, 2017.","title":"Curiosity-driven exploration by self-supervised prediction","authors":"Deepak Pathak, Pulkit Agrawal, Alexei A Efros, Trevor Darrell","year":"2017","venue":"CVPR Workshops","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Plappert et al., 2018] Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. In ICLR, 2018.","title":"Parameter space noise for exploration","authors":"Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, Marcin Andrychowicz","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rückstieß et al., 2008] Thomas Rückstieß, Martin Felder, and Jürgen Schmidhuber. State-dependent exploration for policy gradient methods. In ECML/PKDD, pages 234-249, 2008.","title":"State-dependent exploration for policy gradient methods","authors":"Thomas Rückstieß, Martin Felder, Jürgen Schmidhuber","year":"2008","venue":"ECML/PKDD","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, pages 1889-1897, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael I Jordan, Philipp Moritz","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sun et al., 2011] Yi Sun, Faustino J. Gomez, and Jürgen Schmidhuber. Planning to be surprised: Optimal bayesian exploration in dynamic environments. In AGI, pages 41-51, 2011.","title":"Planning to be surprised: Optimal bayesian exploration in dynamic environments","authors":"Yi Sun, Faustino J Gomez, Jürgen Schmidhuber","year":"2011","venue":"AGI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sutton and Barto, 2018] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 2018.","title":"Reinforcement Learning: An Introduction","authors":"Richard S Sutton, Andrew G Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tang et al., 2017] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. #exploration: A study of count-based exploration for deep reinforcement learning. In NIPS, pages 2753-2762, 2017.","title":"#exploration: A study of count-based exploration for deep reinforcement learning","authors":"Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, Pieter Abbeel","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. Mujoco: A physics engine for model-based con- trol. In IROS, pages 5026-5033, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Vilalta and Drissi, 2002] Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artif. Intell. Rev., 18(2):77-95, 2002.","title":"A perspective view and survey of meta-learning","authors":"Ricardo Vilalta, Youssef Drissi","year":"2002","venue":"Artif. Intell. Rev","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Vinyals et al., 2016] Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In NIPS, pages 3630-3638, 2016.","title":"Matching networks for one shot learning","authors":"Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, Daan Wierstra","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yoon et al., 2018] Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In NeurIPS, pages 7343-7353, 2018.","title":"Bayesian model-agnostic meta-learning","authors":"Yoon","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}