{"type":"ScholarlyArticle","title":"Sharing Experience in Multitask Reinforcement Learning","authors":[{"name":"Tung-Long Vuong","slug":"tung-long-vuong","given":"Tung-Long","family":"Vuong","affiliation":"HMI Lab , UET , Vietnam National University , Hanoi , Vietnam"},{"name":"Do-Van Nguyen","slug":"do-van-nguyen","given":"Do-Van","family":"Nguyen","affiliation":"HMI Lab , UET , Vietnam National University , Hanoi , Vietnam"},{"name":"Tai-Long Nguyen","slug":"tai-long-nguyen","given":"Tai-Long","family":"Nguyen","affiliation":"HMI Lab , UET , Vietnam National University , Hanoi , Vietnam"},{"name":"Cong-Minh Bui","slug":"cong-minh-bui","given":"Cong-Minh","family":"Bui","affiliation":"HMI Lab , UET , Vietnam National University , Hanoi , Vietnam"},{"name":"Hai-Dang Kieu","slug":"hai-dang-kieu","given":"Hai-Dang","family":"Kieu","affiliation":"HMI Lab , UET , Vietnam National University , Hanoi , Vietnam"},{"name":"Viet-Cuong Ta","slug":"viet-cuong-ta","given":"Viet-Cuong","family":"Ta","affiliation":"HMI Lab , UET , Vietnam National University , Hanoi , Vietnam"},{"name":"Quoc-Long Tran","slug":"quoc-long-tran","given":"Quoc-Long","family":"Tran","affiliation":"HMI Lab , UET , Vietnam National University , Hanoi , Vietnam"},{"name":"Thanh-Ha Le","slug":"thanh-ha-le","given":"Thanh-Ha","family":"Le","affiliation":"HMI Lab , UET , Vietnam National University , Hanoi , Vietnam"}],"abstract":"In multitask reinforcement learning, tasks often have sub-tasks that share the same solution, even though the overall tasks are different. If the shared-portions could be effectively identified, then the learning process could be improved since all the samples between tasks in the shared space could be used. In this paper, we propose a Sharing Experience Framework (SEF) for simultaneously training of multiple tasks. In SEF, a confidence sharing agent uses task-specific rewards from the environment to identify similar parts that should be shared across tasks and defines those parts as shared-regions between tasks. The shared-regions are expected to guide task-policies sharing their experience during the learning process. The experiments highlight that our framework improves the performance and the stability of learning task-policies, and is possible to help task-policies avoid local optimums.","doi":"10.24963/ijcai.2019/505","url":"https://ijcai.org/proceedings/2019/505","pdf":"https://ijcai.org/proceedings/2019/0505.pdf","markdown":"https://ijcai.org/proceedings/2019/505.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/505","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":505,"pages":{"first":3642,"last":3648},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Andreas et al., 2016] Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. CoRR, abs/1611.01796, 2016.","title":"Modular multitask reinforcement learning with policy sketches","authors":"Jacob Andreas, Dan Klein, Sergey Levine","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Borsa et al., 2016] Diana Borsa, Thore Graepel, and John Shawe-Taylor. Learning shared representations in multi- task reinforcement learning. CoRR, abs/1603.02041, 2016.","title":"Learning shared representations in multitask reinforcement learning","authors":"Diana Borsa, Thore Graepel, John Shawe-Taylor","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Devin et al., 2017] C. Devin, A. Gupta, T. Darrell, P. Abbeel, and S. Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 2169-2176, May 2017.","title":"Learning modular neural network policies for multi-task and multi-robot transfer","authors":"C Devin, A Gupta, T Darrell, P Abbeel, S Levine","year":"2017","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ferns et al., 2011] N. Ferns, P. Panangaden, and D. Precup. Bisimulation metrics for continuous markov decision pro- cesses. SIAM Journal on Computing, 40(6):1662-1714, 2011.","title":"Bisimulation metrics for continuous markov decision processes","authors":"N Ferns, P Panangaden, D Precup","year":"2011","venue":"SIAM Journal on Computing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Graves et al., 2013] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. In Acoustics, speech and signal pro- cessing (icassp), 2013 ieee international conference on, pages 6645-6649. IEEE, 2013.","title":"Speech recognition with deep recurrent neural networks","authors":"Alex Graves, Abdel-Rahman Mohamed, Geoffrey Hinton","year":"2013","venue":"Acoustics, speech and signal processing (icassp), 2013 ieee international conference on","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Grounds and Kudenko, 2008] M. Grounds and D. Kudenko. Parallel reinforcement learning with linear function ap- proximation. In Adaptive Agents and Multi-Agent Systems III. Adaptation and Multi-Agent Learning, pages 60-74. Springer, 2008.","title":"Parallel reinforcement learning with linear function approximation","authors":"M Grounds, D Kudenko","year":"2008","venue":"Adaptive Agents and Multi-Agent Systems III. Adaptation and Multi-Agent Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gupta et al., 2017] Abhishek Gupta, Coline Devin, Yuxuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learn- ing. CoRR, abs/1703.02949, 2017.","title":"Learning invariant feature spaces to transfer skills with reinforcement learning","authors":"Abhishek Gupta, Coline Devin, Yuxuan Liu, Pieter Abbeel, Sergey Levine","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hausman et al., 2018] K. Hausman, J.T. Springenberg, Z. Wang, N. Heess, and M. Riedmiller. Learning an em- bedding space for transferable robot skills. In Interna- tional Conference on Learning Representations, 2018.","title":"Learning an embedding space for transferable robot skills","authors":"K Hausman, J T Springenberg, Z Wang, N Heess, M Riedmiller","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kakade and Langford, 2002] Sham Kakade and John Lang- ford. Approximately optimal approximate reinforcement learning. In ICML, volume 2, pages 267-274, 2002.","title":"Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning","authors":"Langford Kakade","year":"2002","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Konidaris et al., 2011] G. Konidaris, S. Osentoski, and P. Thomas. Value function approximation in reinforcement learning using the fourier basis. In AAAI, volume 6, page 7, 2011.","title":"Value function approximation in reinforcement learning using the fourier basis","authors":"G Konidaris, S Osentoski, P Thomas","year":"2011","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krizhevsky et al., 2012] A. Krizhevsky, I. Sutskever, and G.E. Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"A Krizhevsky, I Sutskever, G E Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[LeCun et al., 2015] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. nature, 521(7553):436, 2015.","title":"Deep learning","authors":"Y Lecun, Y Bengio, G Hinton","year":"2015","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lillicrap et al., 2015] T.P. Lillicrap, J.J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wier- stra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"T P Lillicrap, J J Hunt, A Pritzel, N Heess, T Erez, Y Tassa, D Silver, D Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2015] V. Mnih, K. Kavukcuoglu, D. Silver, A.A. Rusu, J. Veness, M.G. Bellemare, A. Graves, M. Riedmiller, A.K. Fidjeland, G. Ostrovski, and S. Pe- tersen. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"V Mnih, K Kavukcuoglu, D Silver, A A Rusu, J Veness, M G Bellemare, A Graves, M Riedmiller, A K Fidjeland, G Ostrovski, S Petersen","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2016] V. Mnih, A.P. Badia, M. Mirza, Lil- licrap T. Graves, A., T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep rein- forcement learning. In International conference on ma- chine learning, pages 1928-1937, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"V Mnih, A P Badia, M Mirza, Lillicrap T Graves, A, T Harley, D Silver, K Kavukcuoglu","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Parisotto et al., 2015] E. Parisotto, J.L. Ba, and R. Salakhut- dinov. Actor-mimic: Deep multitask and transfer rein- forcement learning. CoRR, abs/1511.06342, 2015.","title":"Actor-mimic: Deep multitask and transfer reinforcement learning","authors":"E Parisotto, J L Ba, R Salakhutdinov","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rusu et al., 2015] A.A. Rusu, S.G. Colmenarejo, C. Gul- cehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, and R. Hadsell. Policy distillation. CoRR, abs/1511.06295, 2015.","title":"Policy distillation","authors":"A A Rusu, S G Colmenarejo, C Gulcehre, G Desjardins, J Kirkpatrick, R Pascanu, V Mnih, K Kavukcuoglu, R Hadsell","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rusu et al., 2016] A.A. Rusu, N.C. Rabinowitz, G. Des- jardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pas- canu, and R. Hadsell. Progressive neural networks. CoRR, abs/1606.04671, 2016.","title":"Progressive neural networks","authors":"A A Rusu, N C Rabinowitz, G Desjardins, H Soyer, J Kirkpatrick, K Kavukcuoglu, R Pascanu, R Hadsell","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael I Jordan, and Philipp Moritz. Trust region policy optimization. In Icml, volume 37, pages 1889-1897, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Philipp Michael I Jordan, Moritz","year":"2015","venue":"Icml","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schulman et al., 2017] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"J Schulman, F Wolski, P Dhariwal, A Radford, O Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Introduction to reinforcement learning, volume 135. MIT press Cambridge, 1998.","title":"Introduction to reinforcement learning","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutton and S, 1988] Sutton and Richard S. Learning to pre- dict by the methods of temporal differences. Machine learning, 3(1):9-44, 1988.","title":"Learning to predict by the methods of temporal differences","authors":"S ; Sutton, Richard S Sutton","year":"1988","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Teh et al., 2017] Y. Teh, V. Bapst, W.M. Czarnecki, J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, and R. Pascanu. Dis- tral: Robust multitask reinforcement learning. CoRR, abs/1707.04175, 2017.","title":"Distral: Robust multitask reinforcement learning","authors":"Y Teh, V Bapst, W M Czarnecki, J Quan, J Kirkpatrick, R Hadsell, N Heess, R Pascanu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Watkins and Dayan, 1992] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279- 292, 1992.","title":"Q-learning","authors":"Jch Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wilson et al., 2007] A. Wilson, A. Fern, S. Ray, and P. Tadepalli. Multi-task reinforcement learning: A hier- archical bayesian approach. In Proceedings of the 24th In- ternational Conference on Machine Learning, ICML '07, pages 1015-1022, New York, NY, USA, 2007. ACM.","title":"Multi-task reinforcement learning: A hierarchical bayesian approach","authors":"A Wilson, A Fern, S Ray, P Tadepalli","year":"2007","venue":"Proceedings of the 24th International Conference on Machine Learning, ICML '07","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2017] Z. Yang, K.E. Merrick, H.A. Abbass, and L. Jin. Multi-task deep reinforcement learning for contin- uous action control. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 3301-3307, 2017.","title":"Multi-task deep reinforcement learning for continuous action control","authors":"Z Yang, K E Merrick, H A Abbass, L Jin","year":"2017","venue":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17","doi":null,"url":null,"internal_url":"/proceedings/2017/461"},{"position":27,"raw":"[Zhang et al., 2017] J. Zhang, J.T. Springenberg, J. Boedecker, and W. Burgard. Deep reinforcement learning with successor features for navigation across similar environments. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2371-2378, Sept 2017.","title":"Deep reinforcement learning with successor features for navigation across similar environments","authors":"Zhang","year":"2017","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}