{"type":"ScholarlyArticle","title":"Multi-policy Grounding and Ensemble Policy Learning for Transfer Learning with Dynamics Mismatch","authors":[{"name":"Hyun-Rok Lee","slug":"hyun-rok-lee","given":"Hyun-Rok","family":"Lee","affiliation":"University of Toronto; Inha University"},{"name":"Ram Ananth Sreenivasan","slug":"ram-ananth-sreenivasan","given":"Ram Ananth","family":"Sreenivasan","affiliation":"University of Toronto"},{"name":"Yeonjeong Jeong","slug":"yeonjeong-jeong","given":"Yeonjeong","family":"Jeong","affiliation":"LG AI Research; York University"},{"name":"Jongseong Jang","slug":"jongseong-jang","given":"Jongseong","family":"Jang","affiliation":"LG AI Research"},{"name":"Dongsub Shim","slug":"dongsub-shim","given":"Dongsub","family":"Shim","affiliation":"LG AI Research"},{"name":"Chi-Guhn Lee","slug":"chi-guhn-lee","given":"Chi-Guhn","family":"Lee","affiliation":"University of Toronto"}],"abstract":"We propose a new transfer learning algorithm between tasks with different dynamics. The proposed algorithm solves an Imitation from Observation problem (IfO) to ground the source environment to the target task before learning an optimal policy in the grounded environment. The learned policy is deployed in the target task without additional training. A particular feature of our algorithm is the employment of multiple rollout policies during training with a goal to ground the environment more globally; hence, it is named as Multi-Policy Grounding (MPG). The quality of final policy is further enhanced via ensemble policy learning. We demonstrate the superiority of the proposed algorithm analytically and numerically. Numerical studies show that the proposed multi-policy approach allows comparable grounding with single policy approach with a fraction of target samples, hence the algorithm is able to maintain the quality of obtained policy even as the number of interactions with the target environment becomes extremely small.","doi":"10.24963/ijcai.2022/440","url":"https://ijcai.org/proceedings/2022/440","pdf":"https://ijcai.org/proceedings/2022/0440.pdf","markdown":"https://ijcai.org/proceedings/2022/440.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/440","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":440,"pages":{"first":3171,"last":3177},"keywords":["Machine Learning: Deep Reinforcement Learning","Machine Learning: Ensemble Methods","Machine Learning: Generative Adverserial Networks","Machine Learning: Reinforcement Learning","Machine Learning: Multi-task and Transfer Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986821","length_sec":95,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984964","length_sec":359,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Barekatain et al., 2019] Mohammadamin Barekatain, Ryo Yonetani, and Masashi Hamaya. Multipolar: Multi- source policy aggregation for transfer reinforcement learn- ing between diverse environmental dynamics. ArXiv, abs/1909.13111, 2019.","title":"Multipolar: Multisource policy aggregation for transfer reinforcement learning between diverse environmental dynamics","authors":"Barekatain","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brys et al., 2015] Tim Brys, Anna Harutyunyan, Matthew E Taylor, and Ann Nowé. Policy transfer using reward shap- ing. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pages 181-188, 2015.","title":"Policy transfer using reward shaping","authors":"Tim Brys, Anna Harutyunyan, Matthew E Taylor, Ann Nowé","year":"2015","venue":"Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chebotar et al., 2019] Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world expe- rience. In 2019 International Conference on Robotics and Automation (ICRA), pages 8973-8979. IEEE, 2019.","title":"Closing the sim-to-real loop: Adapting simulation randomization with real world experience","authors":"Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, Dieter Fox","year":"2019","venue":"2019 International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chua et al., 2018] Kurtland Chua, R. Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in a handful of trials using probabilistic dynamics mod- els. In NeurIPS, 2018.","title":"Deep reinforcement learning in a handful of trials using probabilistic dynamics models","authors":"R Kurtland Chua, Rowan Calandra, Sergey Mcallister, Levine","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Desai et al., 2020] Siddharth Desai, Ishan Durugkar, Haresh Karnan, Garrett Warnell, Josiah Hanna, Peter Stone, and AI Sony. An imitation from observation approach to transfer learning with dynamics mismatch. Advances in Neural Information Processing Systems, 33, 2020.","title":"An imitation from observation approach to transfer learning with dynamics mismatch","authors":"Siddharth Desai, Ishan Durugkar, Haresh Karnan, Garrett Warnell, Josiah Hanna, Peter Stone, A I Sony","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Eysenbach et al., 2020] Benjamin Eysenbach, Swapnil Asawa, Shreyas Chaudhari, Ruslan Salakhutdinov, and Sergey Levine. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. arXiv preprint arXiv:2006.13916, 2020.","title":"Off-dynamics reinforcement learning: Training for transfer with domain classifiers","authors":"Benjamin Eysenbach, Swapnil Asawa, Shreyas Chaudhari, Ruslan Salakhutdinov, Sergey Levine","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Farchy et al., 2013] Alon Farchy, Samuel Barrett, Patrick MacAlpine, and Peter Stone. Humanoid robots learning to walk faster: From the real world to simulation and back. In Proceedings of the 2013 international conference on Au- tonomous agents and multi-agent systems, pages 39-46, 2013.","title":"Humanoid robots learning to walk faster: From the real world to simulation and back","authors":"Alon Farchy, Samuel Barrett, Patrick Macalpine, Peter Stone","year":"2013","venue":"Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gimelfarb et al., 2020] Michael Gimelfarb, Scott Sanner, and Chi-Guhn Lee. Contextual policy reuse using deep mixture models. arXiv preprint arXiv:2003.00203, 2020.","title":"Contextual policy reuse using deep mixture models","authors":"Michael Gimelfarb, Scott Sanner, Chi-Guhn Lee","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hanna and Stone, 2017] Josiah Hanna and Peter Stone. Grounded action transformation for robot learning in sim- ulation. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 31, 2017.","title":"Grounded action transformation for robot learning in simulation","authors":"Stone Hanna, Josiah Hanna, Peter Stone","year":"2017","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hill et al., 2018] Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https:// github.com/hill-a/stable-baselines, 2018. Accessed: 2022- 05-23.","title":"Stable baselines","authors":"Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu","year":"2018","venue":null,"doi":null,"url":"https://github.com/hill-a/stable-baselines","internal_url":null},{"position":12,"raw":"[Jakobi, 1997] Nick Jakobi. Evolutionary robotics and the radical envelope-of-noise hypothesis. Adaptive behavior, 6(2):325-368, 1997.","title":"Evolutionary robotics and the radical envelope-of-noise hypothesis","authors":"Nick Jakobi","year":"1997","venue":"Adaptive behavior","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Killian et al., 2017] Taylor W Killian, Samuel Daulton, George Konidaris, and Finale Doshi-Velez. Robust and efficient transfer learning with hidden parameter markov decision processes. In Advances in neural information pro- cessing systems, pages 6250-6261, 2017.","title":"Robust and efficient transfer learning with hidden parameter markov decision processes","authors":"Samuel Taylor W Killian, George Daulton, Finale Konidaris, Doshi-Velez","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kurutach et al., 2018] Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model- ensemble trust-region policy optimization. arXiv preprint arXiv:1802.10592, 2018.","title":"Modelensemble trust-region policy optimization","authors":"Kurutach","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lee et al., 2021] Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learn- ing, pages 6131-6141. PMLR, 2021.","title":"Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning","authors":"Lee","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Perez et al., 2020] Christian F Perez, Felipe Petroski Such, and Theofanis Karaletsos. Generalized hidden parameter mdps transferable model-based rl in a handful of trials. arXiv preprint arXiv:2002.03072, 2020.","title":"Generalized hidden parameter mdps transferable model-based rl in a handful of trials","authors":"Felipe Petroski Christian F Perez, Theofanis Such, Karaletsos","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pinto et al., 2017] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial re- inforcement learning. In International Conference on Ma- chine Learning, pages 2817-2826, 2017.","title":"Robust adversarial reinforcement learning","authors":"Pinto","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Plisnier et al., 2019] Hélène Plisnier, Denis Steckelmacher, Diederik M. Roijers, and Ann Nowé. Transfer reinforce- ment learning across environment dynamics with multiple advisors. CEUR-WS, 2019.","title":"Transfer reinforcement learning across environment dynamics with multiple advisors","authors":"Hélène Plisnier, Denis Steckelmacher, M Diederik, Ann Roijers, Nowé","year":"2019","venue":"CEUR-WS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Raffin, 2018] Antonin Raffin. Rl baselines zoo. https: //github.com/araffin/rl-baselines-zoo, 2018. Accessed: 2022-05-23.","title":"Rl baselines zoo","authors":"Antonin Raffin","year":"2022","venue":null,"doi":null,"url":"https://github.com/araffin/rl-baselines-zoo,2018","internal_url":null},{"position":20,"raw":"[Rajeswaran et al., 2017] Aravind Rajeswaran, Sarvjeet Ghotra, Sergey Levine, and Balaraman Ravindran. Epopt: Learning robust neural network policies using model ensembles. ArXiv, abs/1610.01283, 2017.","title":"Epopt: Learning robust neural network policies using model ensembles","authors":"Aravind Rajeswaran, Sarvjeet Ghotra, Sergey Levine, Balaraman Ravindran","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Taylor and Stone, 2009] Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(Jul):1633-1685, 2009.","title":"Transfer learning for reinforcement learning domains: A survey","authors":"E Matthew, Peter Taylor, Stone","year":"2009","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Torabi et al., 2018] Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. arXiv preprint arXiv:1807.06158, 2018.","title":"Generative adversarial imitation from observation","authors":"Faraz Torabi, Garrett Warnell, Peter Stone","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2020] Yue Wang, Yuting Liu, Wei Chen, Zhi- Ming Ma, and Tie-Yan Liu. Target transfer q-learning and its convergence analysis. Neurocomputing, 2020.","title":"Target transfer q-learning and its convergence analysis","authors":"Yue Wang, Yuting Liu, Wei Chen, Zhi-Ming Ma, Tie-Yan Liu","year":"2020","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2020a] Jiachen Yang, Brenden Petersen, Hongyuan Zha, and Daniel Faissol. Single episode policy transfer in reinforcement learning. In International Conference on Learning Representations, 2020.","title":"Single episode policy transfer in reinforcement learning","authors":"Yang","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2020b] Tianpei Yang, Jianye Hao, Zhaopeng Meng, Zongzhang Zhang, Yujing Hu, Yingfeng Chen, Changjie Fan, Weixun Wang, Zhaodong Wang, and Jia- jie Peng. Efficient deep reinforcement learning through policy transfer. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Sys- tems, pages 2053-2055, 2020.","title":"Efficient deep reinforcement learning through policy transfer","authors":"Yang","year":"2020","venue":"Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2020] Qiang Zhang, Tete Xiao, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Learning cross- domain correspondence for control with dynamics cycle- consistency. arXiv preprint arXiv:2012.09811, 2020.","title":"Learning crossdomain correspondence for control with dynamics cycleconsistency","authors":"Qiang Zhang, Tete Xiao, Alexei A Efros, Lerrel Pinto, Xiaolong Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}