{"type":"ScholarlyArticle","title":"MULTIPOLAR: Multi-Source Policy Aggregation for Transfer Reinforcement Learning between Diverse Environmental Dynamics","authors":[{"name":"Mohammadamin Barekatain","slug":"mohammadamin-barekatain","given":"Mohammadamin","family":"Barekatain","affiliation":"Technical University of Munich"},{"name":"Ryo Yonetani","slug":"ryo-yonetani","given":"Ryo","family":"Yonetani","affiliation":"Technical University of Munich"},{"name":"Masashi Hamaya","slug":"masashi-hamaya","given":"Masashi","family":"Hamaya","affiliation":"Technical University of Munich"}],"abstract":"Transfer reinforcement learning (RL) aims at improving the learning efficiency of an agent by exploiting knowledge from other source agents trained on relevant tasks. However, it remains challenging to transfer knowledge between different environmental dynamics without having access to the source environments. In this work, we explore a new challenge in transfer RL, where only a set of source policies collected under diverse unknown dynamics is available for learning a target task efficiently. To address this problem, the proposed approach, MULTI-source POLicy AggRegation (MULTIPOLAR), comprises two key techniques. We learn to aggregate the actions provided by the source policies adaptively to maximize the target task performance. Meanwhile, we learn an auxiliary network that predicts residuals around the aggregated actions, which ensures the target policy's expressiveness even when some of the source policies perform poorly. We demonstrated the effectiveness of MULTIPOLAR through an extensive experimental evaluation across six simulated environments ranging from classic control problems to challenging robotics simulations, under both continuous and discrete action spaces. The demo videos and code are available on the project webpage: https://omron-sinicx.github.io/multipolar/.","doi":"10.24963/ijcai.2020/430","url":"https://ijcai.org/proceedings/2020/430","pdf":"https://ijcai.org/proceedings/2020/0430.pdf","markdown":"https://ijcai.org/proceedings/2020/430.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/430","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":430,"pages":{"first":3108,"last":3116},"keywords":["Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Deep Reinforcement Learning","Machine Learning: Reinforcement Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/26864/file_26864.m3u8","length_sec":976,"provider":"hls"}],"references":[{"position":1,"raw":"[Bacon et al., 2017] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The Option-Critic Architecture. In AAAI, 2017.","title":"The Option-Critic Architecture","authors":"Bacon","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bello et al., 2016] Irwan Bello, Hieu Pham, Quoc V Le, Mohammad Norouzi, and Samy Bengio. Neural combi- natorial optimization with reinforcement learning. arXiv preprint arXiv:1611.09940, 2016.","title":"Neural combinatorial optimization with reinforcement learning","authors":"Irwan Bello, Hieu Pham, V Quoc, Mohammad Le, Samy Norouzi, Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018] Tao Chen, Adithyavairavan Murali, and Abhinav Gupta. Hardware Conditioned Policies for Multi- Robot Transfer Learning. In NeurIPS, 2018.","title":"Hardware Conditioned Policies for Multi-Robot Transfer Learning","authors":"Tao Chen, Adithyavairavan Murali, Abhinav Gupta","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Clavera et al., 2019] Ignasi Clavera, Anusha Nagabandi, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning. In ICLR, 2019.","title":"Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning","authors":"Ignasi Clavera, Anusha Nagabandi, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, Chelsea Finn","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Devin et al., 2017] Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning Mod- ular Neural Network Policies for Multi-Task and Multi- Robot Transfer. In ICRA, 2017.","title":"Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfer","authors":"Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, Sergey Levine","year":"2017","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fernández and Veloso, 2006] Fernando Fernández and Manuela Veloso. Probabilistic Policy Reuse in a Reinforcement Learning Agent. In AAMAS, 2006.","title":"Probabilistic Policy Reuse in a Reinforcement Learning Agent","authors":"Fernando Fernández, Manuela Veloso","year":"2006","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Franc ¸ois-Lavet et al., 2018] Vincent Franc ¸ois-Lavet, Peter Henderson, Riashat Islam, Marc G. Bellemare, and Joelle Pineau. An Introduction to Deep Reinforcement Learn- ing. Foundations and Trends in Machine Learning, 11(3- 4):219-354, 2018.","title":"An Introduction to Deep Reinforcement Learning","authors":"Franc ¸ois-Lavet","year":"2018","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Frans et al., 2018] Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta Learning Shared Hierarchies. In ICLR, 2018.","title":"Meta Learning Shared Hierarchies","authors":"Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, John Schulman","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off- Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In ICML, 2018.","title":"Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor","authors":"Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Henderson et al., 2018] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep Reinforcement Learning That Matters. In AAAI, 2018.","title":"Deep Reinforcement Learning That Matters","authors":"Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hill et al., 2018] Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable Baselines. https://github.com/ hill-a/stable-baselines, 2018.","title":"Stable Baselines","authors":"Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu","year":"2018","venue":null,"doi":null,"url":"https://github.com/hill-a/stable-baselines","internal_url":null},{"position":12,"raw":"[Johannink et al., 2019] Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, and Sergey Levine. Residual Reinforcement Learning for Robot Con- trol. In ICRA, 2019.","title":"Residual Reinforcement Learning for Robot Control","authors":"Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, Sergey Levine","year":"2019","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kulkarni et al., 2016] Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation. In NeurIPS, 2016.","title":"Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation","authors":"D Tejas, Karthik Kulkarni, Ardavan Narasimhan, Josh Saeedi, Tenenbaum","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lazaric et al., 2008] Alessandro Lazaric, Marcello Restelli, and Andrea Bonarini. Transfer of Samples in Batch Rein- forcement Learning. In ICML, 2008.","title":"Transfer of Samples in Batch Reinforcement Learning","authors":"Alessandro Lazaric, Marcello Restelli, Andrea Bonarini","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li and Zhang, 2018] Siyuan Li and Chongjie Zhang. An Optimal Online Method of Selecting Source Policies for Reinforcement Learning. In AAAI, 2018.","title":"An Optimal Online Method of Selecting Source Policies for Reinforcement Learning","authors":"Li, Zhang ; ] Siyuan Li, Chongjie Zhang","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2019] Siyuan Li, Fangda Gu, Guangxiang Zhu, and Chongjie Zhang. Context-Aware Policy Reuse. In AAMAS, 2019.","title":"Context-Aware Policy Reuse","authors":"Siyuan Li, Fangda Gu, Guangxiang Zhu, Chongjie Zhang","year":"2019","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mankowitz et al., 2018] Daniel J Mankowitz, Timothy A Mann, Pierre-Luc Bacon, Doina Precup, and Shie Mannor. Learning Robust Options. In AAAI, 2018.","title":"Learning Robust Options","authors":"Timothy A Daniel J Mankowitz, Pierre-Luc Mann, Doina Bacon, Shie Precup, Mannor","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Osa et al., 2019] Takayuki Osa, Voot Tangkaratt, and Masashi Sugiyama. Hierarchical Reinforcement Learn- ing via Advantage-Weighted Information Maximization. In ICLR, 2019.","title":"Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization","authors":"Takayuki Osa, Voot Tangkaratt, Masashi Sugiyama","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Parisotto et al., 2016] Emilio Parisotto, Jimmy Ba, and Rus- lan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. In ICLR, 2016.","title":"Actor-mimic: Deep multitask and transfer reinforcement learning","authors":"Parisotto","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rajendran et al., 2017] Janarthanan Rajendran, Aravind S Lakshminarayanan, Mitesh M Khapra, P Prasanna, and Balaraman Ravindran. Attend, adapt and transfer: Atten- tive deep architecture for adaptive transfer from multiple sources in the same domain. In ICLR, 2017.","title":"Attend, adapt and transfer: Attentive deep architecture for adaptive transfer from multiple sources in the same domain","authors":"Janarthanan Rajendran, Aravind S Lakshminarayanan, M Mitesh, P Khapra, Balaraman Prasanna, Ravindran","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rosman et al., 2016] Benjamin Rosman, Majd Hawasly, and Subramanian Ramamoorthy. Bayesian Policy Reuse. Machine Learning, 104(1):99-127, 2016.","title":"Bayesian Policy Reuse","authors":"Benjamin Rosman, Majd Hawasly, Subramanian Ramamoorthy","year":"2016","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal Policy Optimization Algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Silver et al., 2018] Tom Silver, Kelsey Allen, Josh Tenen- baum, and Leslie Kaelbling. Residual Policy Learning. arXiv preprint arXiv:1812.06298, 2018.","title":"Residual Policy Learning","authors":"Tom Silver, Kelsey Allen, Josh Tenenbaum, Leslie Kaelbling","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Song et al., 2016] Jinhua Song, Yang Gao, Hao Wang, and Bo An. Measuring the Distance Between Finite Markov Decision Processes. In AAMAS, 2016.","title":"Measuring the Distance Between Finite Markov Decision Processes","authors":"Jinhua Song, Yang Gao, Hao Wang, Bo An","year":"2016","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, 1st edition, 1998.","title":"Introduction to Reinforcement Learning","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sutton et al., 1999] Richard S Sutton, Doina Precup, and Satinder Singh. Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning. Artificial intelligence, 112(1-2):181-211, 1999.","title":"Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning","authors":"Doina Richard S Sutton, Satinder Precup, Singh","year":"1999","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tan et al., 2018] Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-Real: Learning Agile Loco- motion For Quadruped Robots. In RSS, 2018.","title":"Sim-to-Real: Learning Agile Locomotion For Quadruped Robots","authors":"Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, Vincent Vanhoucke","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tange, 2018] Ole Tange. GNU Parallel 2018.","title":"GNU Parallel","authors":"Ole Tange","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"Ole Tange, 2018.","title":null,"authors":"Ole Tange","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tirinzoni et al., 2018] Andrea Tirinzoni, Andrea Sessa, Matteo Pirotta, and Marcello Restelli. Importance Weighted Transfer of Samples in Reinforcement Learning. In ICML, 2018.","title":"Importance Weighted Transfer of Samples in Reinforcement Learning","authors":"Andrea Tirinzoni, Andrea Sessa, Matteo Pirotta, Marcello Restelli","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Vanschoren, 2018] Joaquin Vanschoren. Meta-Learning: A Survey. arXiv preprint arXiv:1810.03548, 2018.","title":"Joaquin Vanschoren. Meta-Learning: A Survey","authors":"Vanschoren","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2019] Hao Wang, Shaokang Dong, and Ling Shao. Measuring Structural Similarities in Finite MDPs. In IJCAI, 2019.","title":"Measuring Structural Similarities in Finite MDPs","authors":"Hao Wang, Shaokang Dong, Ling Shao","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/511"},{"position":33,"raw":"[Yu et al., 2019] Wenhao Yu, C. Karen Liu, and Greg Turk. Policy Transfer with Strategy Optimization. In ICLR, 2019.","title":"Policy Transfer with Strategy Optimization","authors":"Wenhao Yu, C Karen Liu, Greg Turk","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zheng et al., 2018] Yan Zheng, Zhaopeng Meng, Jianye Hao, Zongzhang Zhang, Tianpei Yang, and Changjie Fan. A Deep Bayesian Policy Reuse Approach against Non- Stationary Agents. In NeurIPS, 2018.","title":"A Deep Bayesian Policy Reuse Approach against Non-Stationary Agents","authors":"Yan Zheng, Zhaopeng Meng, Jianye Hao, Zongzhang Zhang, Tianpei Yang, Changjie Fan","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}