{"type":"ScholarlyArticle","title":"Reinforcement Learning Experience Reuse with Policy Residual Representation","authors":[{"name":"WenJi Zhou","slug":"wenji-zhou","given":"WenJi","family":"Zhou","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing , China"},{"name":"Yang Yu","slug":"yang-yu","given":"Yang","family":"Yu","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing , China"},{"name":"Yingfeng Chen","slug":"yingfeng-chen","given":"Yingfeng","family":"Chen","affiliation":"NetEase Fuxi AI Lab , Hangzhou , China"},{"name":"Kai Guan","slug":"kai-guan","given":"Kai","family":"Guan","affiliation":"NetEase Fuxi AI Lab , Hangzhou , China"},{"name":"Tangjie Lv","slug":"tangjie-lv","given":"Tangjie","family":"Lv","affiliation":"NetEase Fuxi AI Lab , Hangzhou , China"},{"name":"Changjie Fan","slug":"changjie-fan","given":"Changjie","family":"Fan","affiliation":"NetEase Fuxi AI Lab , Hangzhou , China"},{"name":"Zhi-Hua Zhou","slug":"zhi-hua-zhou","given":"Zhi-Hua","family":"Zhou","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing , China"}],"abstract":"Experience reuse is key to sample-efficient reinforcement learning. One of the critical issues is how the experience is represented and stored. Previously, the experience can be stored in the forms of features, individual models, and the average model, each lying at a different granularity. However, new tasks may require experience across multiple granularities. In this paper, we propose the policy residual representation (PRR) network, which can extract and store multiple levels of experience. PRR network is trained on a set of tasks with a multi-level architecture, where a module in each level corresponds to a subset of the tasks. Therefore, the PRR network represents the experience in a spectrum-like way. When training on a new task, PRR can provide different levels of experience for accelerating the learning. We experiment with the PRR network on a set of grid world navigation tasks, locomotion tasks, and fighting tasks in a video game. The results show that the PRR network leads to better reuse of experience and thus outperforms some state-of-the-art approaches.","doi":"10.24963/ijcai.2019/618","url":"https://ijcai.org/proceedings/2019/618","pdf":"https://ijcai.org/proceedings/2019/0618.pdf","markdown":"https://ijcai.org/proceedings/2019/618.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/618","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":618,"pages":{"first":4447,"last":4453},"keywords":["Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Barreto et al., 2017] André Barreto, Will Dabney, Rémi Munos, Jonathan J. Hunt, Tom Schaul, David Silver, and Hado P. van Hasselt. Successor features for transfer in reinforcement learning. In Advances in Neural Informa- tion Processing Systems 30 (NIPS'17), pages 4058-4068, Long Beach, CA, 2017.","title":"Successor features for transfer in reinforcement learning","authors":"André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, David Silver, Hado P Van Hasselt","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bou-Ammar et al., 2012] Haitham Bou-Ammar, Karl Tuyls, Matthew E. Taylor, Kurt Driessens, and Gerhard Weiss. Reinforcement learning transfer via sparse coding. In Proceedings of 11th International Conference on Au- tonomous Agents and Multiagent Systems (AAMAS'12), pages 383-390, Valencia, Spain, 2012.","title":"Reinforcement learning transfer via sparse coding","authors":"Haitham Bou-Ammar, Karl Tuyls, Matthew E Taylor, Kurt Driessens, Gerhard Weiss","year":"2012","venue":"Proceedings of 11th International Conference on Autonomous Agents and Multiagent Systems (AAMAS'12)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th Interna- tional Conference on Machine Learning (ICML'17), pages 1126-1135, Sydney, Australia, 2017.","title":"Model-agnostic meta-learning for fast adaptation of deep networks","authors":"Finn","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning (ICML'17)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Frans et al., 2018] Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. In Proceedings of the 6th International Con- ference on Learning Representations (ICLR'18), pages 1- 8, Vancouver, Canada, 2018.","title":"Meta learning shared hierarchies","authors":"Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, John Schulman","year":"2018","venue":"Proceedings of the 6th International Conference on Learning Representations (ICLR'18)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Guo et al., 2014] Xiaoxiao Guo, Satinder P. Singh, Honglak Lee, Richard L. Lewis, and Xiaoshi Wang. Deep learning for real-time Atari game play using offline Monte-Carlo tree search planning. In Advances in Neural Informa- tion Processing Systems 27 (NIPS'14), pages 3338-3346, Montreal, Quebec, Canada, 2014.","title":"Deep learning for real-time Atari game play using offline Monte-Carlo tree search planning","authors":"Xiaoxiao Guo, P Satinder, Honglak Singh, Richard L Lee, Xiaoshi Lewis, Wang","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gupta et al., 2018] Abhishek Gupta, Russell Mendonca, Yuxuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In Advances in Neural Information Processing Systems 31 (NIPS'18), pages 5307-5316, Montréal, Canada., 2018.","title":"Metareinforcement learning of structured exploration strategies","authors":"Abhishek Gupta, Russell Mendonca, Yuxuan Liu, Pieter Abbeel, Sergey Levine","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2018] Yi-Qi Hu, Yang Yu, and Zhi-Hua Zhou. Ex- perienced optimization with reusable directional model for hyper-parameter search. In Proceedings of the Twenty- Seventh International Joint Conference on Artificial Intel- ligence (IJCAI'18), pages 2276-2282, Stockholm, Swe- den, 2018.","title":"Experienced optimization with reusable directional model for hyper-parameter search","authors":"Yi-Qi Hu, Yang Yu, Zhi-Hua Zhou","year":"2018","venue":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI'18)","doi":null,"url":null,"internal_url":"/proceedings/2018/315"},{"position":8,"raw":"[John Schulman, 2017] Prafulla Dhariwal Alec Radford Oleg Klimov John Schulman, Filip Wolski. Proximal pol- icy optimization algorithms. arXiv, abs/1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"Prafulla Dhariwal, Alec Radford, Oleg Klimov, John Schulman, Filip Wolski","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lackey and Meltzer, 1971] R. B. Lackey and D. Meltzer. A simplified definition of Walsh functions. IEEE Transac- tions on Computers, C-20:211-213, 1971.","title":"A simplified definition of Walsh functions","authors":"R B Lackey, D Meltzer","year":"1971","venue":"IEEE Transactions on Computers","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Levine et al., 2016] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Re- search, 17:1-40, 2016.","title":"End-to-end training of deep visuomotor policies","authors":"Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel","year":"2016","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lillicrap et al., 2015] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yu- val Tassa, David Silver, and Daan Wierstra. Contin- uous control with deep reinforcement learning. arXiv, abs/1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Has- sabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Mnih","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd In- ternational Conference on Machine Learning (ICML'15), pages 1889-1897, Lille, France, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael I Jordan, Philipp Moritz","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning (ICML'15)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J. Mad- dison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Ve- davyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mas- tering the game of Go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of Go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, Demis Hassabis","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning -An Introduction. MIT Press, 1998.","title":"Reinforcement Learning -An Introduction","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. MuJoCo: A physics engine for model-based control. In Proceedings of the 2012 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems, pages 5026-5033, Vilamoura, Portugal, 2012.","title":"MuJoCo: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Yu et al., 2018] Yang Yu, Shi-Yong Chen, Qing Da, and Zhi-Hua Zhou. Reusable reinforcement learning via shal- low trails. IEEE Transactions on Neural Networks and Learning Systems, 29(6):2204-2215, 2018.","title":"Reusable reinforcement learning via shallow trails","authors":"Yang Yu, Shi-Yong Chen, Qing Da, Zhi-Hua Zhou","year":"2018","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Zhang et al., 2018] Chao Zhang, Yang Yu, and Zhi-Hua Zhou. Learning environmental calibration actions for policy self-evolution. In Proceedings of the 27th Inter- national Joint Conference on Artificial Intelligence (IJ- CAI'18), pages 3061-3067, Stockholm, Sweden, 2018.","title":"Learning environmental calibration actions for policy self-evolution","authors":"Chao Zhang, Yang Yu, Zhi-Hua Zhou","year":"2018","venue":"Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJ-CAI'18)","doi":null,"url":null,"internal_url":"/proceedings/2018/425"},{"position":19,"raw":"[Zhou, 2016] Z.-H. Zhou. Learnware: On the future of ma- chine learning. Frontiers of Computer Science, 10(4):589- 590, 2016.","title":"Learnware: On the future of machine learning","authors":"Z.-H Zhou","year":"2016","venue":"Frontiers of Computer Science","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}