{"type":"ScholarlyArticle","title":"Efficient Deep Reinforcement Learning via Adaptive Policy Transfer","authors":[{"name":"Tianpei Yang","slug":"tianpei-yang","given":"Tianpei","family":"Yang","affiliation":"College of Intelligence and Computing , Tianjin University; Noah's Ark Lab , Huawei"},{"name":"Jianye Hao","slug":"jianye-hao","given":"Jianye","family":"Hao","affiliation":"College of Intelligence and Computing , Tianjin University; Noah's Ark Lab , Huawei; Tianjin Key Lab of Machine Learning"},{"name":"Zhaopeng Meng","slug":"zhaopeng-meng","given":"Zhaopeng","family":"Meng","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Zongzhang Zhang","slug":"zongzhang-zhang","given":"Zongzhang","family":"Zhang","affiliation":"Nanjing University"},{"name":"Yujing Hu","slug":"yujing-hu","given":"Yujing","family":"Hu","affiliation":"Fuxi AI Lab in Netease"},{"name":"Yingfeng Chen","slug":"yingfeng-chen","given":"Yingfeng","family":"Chen","affiliation":"Fuxi AI Lab in Netease"},{"name":"Changjie Fan","slug":"changjie-fan","given":"Changjie","family":"Fan","affiliation":"Fuxi AI Lab in Netease"},{"name":"Weixun Wang","slug":"weixun-wang","given":"Weixun","family":"Wang","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Wulong Liu","slug":"wulong-liu","given":"Wulong","family":"Liu","affiliation":"Noah's Ark Lab , Huawei"},{"name":"Zhaodong Wang","slug":"zhaodong-wang","given":"Zhaodong","family":"Wang","affiliation":"JD Digits"},{"name":"Jiajie Peng","slug":"jiajie-peng","given":"Jiajie","family":"Peng","affiliation":"College of Intelligence and Computing , Tianjin University"}],"abstract":"Transfer learning has shown great potential to accelerate Reinforcement Learning (RL) by leveraging prior knowledge from past learned policies of relevant tasks. Existing approaches either transfer previous knowledge by explicitly computing similarities between tasks or select appropriate source policies to provide guided explorations. However, how to directly optimize the target policy by alternatively utilizing knowledge from appropriate source policies without explicitly measuring the similarities is currently missing. In this paper, we propose a novel Policy Transfer Framework (PTF) by taking advantage of this idea. PTF learns when and which source policy is the best to reuse for the target policy and when to terminate it by modeling multi-policy transfer as an option learning problem. PTF can be easily combined with existing DRL methods and experimental results show it significantly accelerates RL and surpasses state-of-the-art policy transfer methods in terms of learning efficiency and final performance in both discrete and continuous action spaces.","doi":"10.24963/ijcai.2020/428","url":"https://ijcai.org/proceedings/2020/428","pdf":"https://ijcai.org/proceedings/2020/0428.pdf","markdown":"https://ijcai.org/proceedings/2020/428.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/428","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":428,"pages":{"first":3094,"last":3100},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning: Transfer, Adaptation, Multi-task Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25189/file_25189.m3u8","length_sec":279,"provider":"hls"}],"references":[{"position":1,"raw":"[Bacon et al., 2017] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceed- ings of the 31st AAAI Conference on Artificial Intelligence, pages 1726-1734, 2017.","title":"The option-critic architecture","authors":"Bacon","year":"2017","venue":"Proceedings of the 31st AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brunskill and Li, 2014] Emma Brunskill and Lihong Li. PAC-inspired option discovery in lifelong reinforcement learning. In Proceedings of International Conference on Machine Learning, pages 316-324, 2014.","title":"PAC-inspired option discovery in lifelong reinforcement learning","authors":"Li ; Brunskill, Emma Brunskill, Lihong Li","year":"2014","venue":"Proceedings of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brys et al., 2015] Tim Brys, Anna Harutyunyan, Matthew E Taylor, and Ann Nowé. Policy transfer using reward shap- ing. In Proceedings of International Conference on Au- tonomous Agents and Multiagent Systems, pages 181-188, 2015.","title":"Policy transfer using reward shaping","authors":"Tim Brys, Anna Harutyunyan, Matthew E Taylor, Ann Nowé","year":"2015","venue":"Proceedings of International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fernández and Veloso, 2006] Fernando Fernández and Manuela Veloso. Probabilistic policy reuse in a rein- forcement learning agent. In Proceedings of International Conference on Autonomous Agents and Multiagent Systems, pages 720-727, 2006.","title":"Probabilistic policy reuse in a reinforcement learning agent","authors":"Fernando Fernández, Manuela Veloso","year":"2006","venue":"Proceedings of International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jaderberg et al., 2017] Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Don- ahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dun- ning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural net- works. CoRR, abs/1711.09846, 2017.","title":"Population based training of neural networks","authors":"Jaderberg","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Laroche and Barlier, 2017] Romain Laroche and Merwan Barlier. Transfer reinforcement learning with shared dy- namics. In Proceedings of the 31st AAAI Conference on Artificial Intelligence, pages 2147-2153, 2017.","title":"Transfer reinforcement learning with shared dynamics","authors":"Romain Laroche, Merwan Barlier","year":"2017","venue":"Proceedings of the 31st AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Li and Zhang, 2018] Siyuan Li and Chongjie Zhang. An op- timal online method of selecting source policies for rein- forcement learning. In Proceedings of AAAI Conference on Artificial Intelligence, pages 3562-3570, 2018.","title":"An optimal online method of selecting source policies for reinforcement learning","authors":"Li, Zhang ; ] Siyuan Li, Chongjie Zhang","year":"2018","venue":"Proceedings of AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2019] Siyuan Li, Fangda Gu, Guangxiang Zhu, and Chongjie Zhang. Context-aware policy reuse. In Proceedings of the 18th International Conference on Au- tonomous Agents and Multiagent Systems, pages 989-997, 2019.","title":"Context-aware policy reuse","authors":"Siyuan Li, Fangda Gu, Guangxiang Zhu, Chongjie Zhang","year":"2019","venue":"Proceedings of the 18th International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lillicrap et al., 2016] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Proceedings of Interna- tional Conference on Learning Representations, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2016","venue":"Proceedings of International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mann et al., 2014] Timothy Mann, Daniel Mankowitz, and Shie Mannor. Time-regularized interrupting options. In Proceedings of International Conference on Machine Learning, pages 1350-1358, 2014.","title":"Time-regularized interrupting options","authors":"Timothy Mann, Daniel Mankowitz, Shie Mannor","year":"2014","venue":"Proceedings of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Has- sabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin A Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In Proceedings of International Conference on Machine Learning, pages 1928-1937, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Mnih","year":"2016","venue":"Proceedings of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Rajendran et al., 2017] Janarthanan Rajendran, Aravind S Lakshminarayanan, Mitesh M Khapra, P Prasanna, and Balaraman Ravindran. Attend, adapt and transfer: Atten- tive deep architecture for adaptive transfer from multiple sources in the same domain. In Proceedings of Interna- tional Conference on Learning Representations, 2017.","title":"Attend, adapt and transfer: Attentive deep architecture for adaptive transfer from multiple sources in the same domain","authors":"Janarthanan Rajendran, Aravind S Lakshminarayanan, M Mitesh, P Khapra, Balaraman Prasanna, Ravindran","year":"2017","venue":"Proceedings of International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rusu et al., 2016] Andrei A Rusu, Sergio Gomez Col- menarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. In Proceedings of International Conference on Learning Representations, 2016.","title":"Policy distillation","authors":"Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, Raia Hadsell","year":"2016","venue":"Proceedings of International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Schmitt et al., 2018] Simon Schmitt, Jonathan J. Hudson, Augustin Zídek, Simon Osindero, Carl Doersch, Woj- ciech M. Czarnecki, Joel Z. Leibo, Heinrich Küttler, An- drew Zisserman, Karen Simonyan, and S. M. Ali Eslami. Kickstarting deep reinforcement learning. arXiv preprint arXiv:1803.03835, 2018.","title":"Kickstarting deep reinforcement learning","authors":"Simon Schmitt, Jonathan J Hudson, Augustin Zídek, Simon Osindero, Carl Doersch, Wojciech M Czarnecki, Joel Z Leibo, Heinrich Küttler, Andrew Zisserman, Karen Simonyan, S M Ali Eslami","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Song et al., 2016] Jinhua Song, Yang Gao, Hao Wang, and Bo An. Measuring the distance between finite Markov decision processes. In Proceedings of International Con- ference on Autonomous Agents and Multiagent Systems, pages 468-476, 2016.","title":"Measuring the distance between finite Markov decision processes","authors":"Jinhua Song, Yang Gao, Hao Wang, Bo An","year":"2016","venue":"Proceedings of International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 1998.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sutton et al., 1999] Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1):181 -211, 1999.","title":"Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning","authors":"Richard S Sutton, Doina Precup, Satinder Singh","year":"1999","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tassa et al., 2018] Yuval Tassa, Yotam Doron, Alistair Mul- dal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillicrap, and Martin A. Riedmiller. DeepMind Control Suite. CoRR, abs/1801.00690, 2018.","title":"DeepMind Control Suite","authors":"Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego De Las, David Casas, Abbas Budden, Josh Abdolmaleki, Andrew Merel, Timothy P Lefrancq, Martin A Lillicrap, Riedmiller","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Thomas, 2014] Philip Thomas. Bias in natural actor-critic algorithms. In Proceedings of International Conference on Machine Learning, pages 441-448, 2014.","title":"Bias in natural actor-critic algorithms","authors":"Philip Thomas","year":"2014","venue":"Proceedings of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. Mujoco: A physics engine for model-based con- trol. In Proceedings of the International Conference on Intelligent Robots and Systems, pages 5026-5033, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"Proceedings of the International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}