{"type":"ScholarlyArticle","title":"Independent Skill Transfer for Deep Reinforcement Learning","authors":[{"name":"Qiangxing Tian","slug":"qiangxing-tian","given":"Qiangxing","family":"Tian","affiliation":"Zhejiang University , Hangzhou , China; School of Engineering , Westlake University , Hangzhou , China"},{"name":"Guanchu Wang","slug":"guanchu-wang","given":"Guanchu","family":"Wang","affiliation":"School of Engineering , Westlake University , Hangzhou , China"},{"name":"Jinxin Liu","slug":"jinxin-liu","given":"Jinxin","family":"Liu","affiliation":"Zhejiang University , Hangzhou , China; School of Engineering , Westlake University , Hangzhou , China"},{"name":"Donglin Wang","slug":"donglin-wang","given":"Donglin","family":"Wang","affiliation":"School of Engineering , Westlake University , Hangzhou , China"},{"name":"Yachen Kang","slug":"yachen-kang","given":"Yachen","family":"Kang","affiliation":"School of Engineering , Westlake University , Hangzhou , China"}],"abstract":"Recently, diverse primitive skills have been learned by adopting the entropy as intrinsic reward, which further shows that new practical skills can be produced by combining a variety of primitive skills. This is essentially skill transfer, very useful for learning high-level skills but quite challenging due to the low efficiency of transferring primitive skills. In this paper, we propose a novel efficient skill transfer method, where we learn independent skills and only independent components of skills are transferred instead of the whole set of skills. More concretely, independent components of skills are obtained through independent component analysis (ICA), which always have a smaller amount (or lower dimension) compared with their mixtures. With a lower dimension, independent skill transfer (IST) exhibits a higher efficiency on learning a given task. Extensive experiments including three robotic tasks demonstrate the effectiveness and high efficiency of our proposed IST method in comparison to direct primitive-skill transfer and conventional reinforcement learning.","doi":"10.24963/ijcai.2020/401","url":"https://ijcai.org/proceedings/2020/401","pdf":"https://ijcai.org/proceedings/2020/0401.pdf","markdown":"https://ijcai.org/proceedings/2020/401.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/401","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":401,"pages":{"first":2901,"last":2907},"keywords":["Machine Learning: Deep Reinforcement Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24486/file_24486.m3u8","length_sec":296,"provider":"hls"}],"references":[{"position":1,"raw":"[Bach and Jordan, 2002] Francis R Bach and Michael I Jor- dan. Kernel independent component analysis. Journal of machine learning research, 3(Jul):1-48, 2002.","title":"Kernel independent component analysis","authors":"R Francis, Michael I Bach, Jordan","year":"2002","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bell and Sejnowski, 1995] Anthony J Bell and Terrence J Sejnowski. An information-maximization approach to blind separation and blind deconvolution. Neural compu- tation, 7(6):1129-1159, 1995.","title":"An information-maximization approach to blind separation and blind deconvolution","authors":"J Anthony, Terrence J Bell, Sejnowski","year":"1995","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio, 2017] Yoshua Bengio. The consciousness prior. arXiv preprint arXiv:1709.08568, 2017.","title":"The consciousness prior","authors":"Yoshua Bengio","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bryant and Yarnold, 1995] Fred B Bryant and Paul R Yarnold. Principal-components analysis and exploratory and confirmatory factor analysis. 1995.","title":"Principal-components analysis and exploratory and confirmatory factor analysis","authors":"B Fred, Paul R Bryant, Yarnold","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Coros et al., 2009] Stelian Coros, Philippe Beaudoin, and Michiel Van de Panne. Robust task-based control poli- cies for physics-based characters. In ACM Transactions on Graphics (TOG), volume 28, page 170. ACM, 2009.","title":"Robust task-based control policies for physics-based characters","authors":"Stelian Coros, Philippe Beaudoin, Michiel Van De Panne","year":"2009","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Eysenbach et al., 2018] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070, 2018.","title":"Diversity is all you need: Learning skills without a reward function","authors":"Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey Levine","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Frans et al., 2017] Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. arXiv preprint arXiv:1710.09767, 2017.","title":"Meta learning shared hierarchies","authors":"Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, John Schulman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gregor et al., 2016] Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.07507, 2016.","title":"Variational intrinsic control","authors":"Karol Gregor, Danilo Jimenez Rezende, Daan Wierstra","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Haarnoja et al., 2018a] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018.","title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","authors":"Haarnoja","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Haarnoja et al., 2018b] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018.","title":"Soft actor-critic algorithms and applications","authors":"Haarnoja","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hausknecht and Stone, 2015] Matthew Hausknecht and Pe- ter Stone. Deep reinforcement learning in parameterized action space. arXiv preprint arXiv:1511.04143, 2015.","title":"Deep reinforcement learning in parameterized action space","authors":"Matthew Hausknecht, Peter Stone","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hyvärinen and Oja, 2000] Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applica- tions. Neural networks, 13(4-5):411-430, 2000.","title":"Independent component analysis: algorithms and applications","authors":"Oja Hyvärinen, Aapo Hyvärinen, Erkki Oja","year":"2000","venue":"Neural networks","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kulkarni et al., 2016] Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Ad- vances in neural information processing systems, pages 3675-3683, 2016.","title":"Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation","authors":"D Tejas, Karthik Kulkarni, Ardavan Narasimhan, Josh Saeedi, Tenenbaum","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mohamed and Rezende, 2015] Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in neural information processing systems, pages 2125-2133, 2015.","title":"Variational information maximisation for intrinsically motivated reinforcement learning","authors":"Shakir Mohamed, Danilo Jimenez, Rezende","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Oja and Yuan, 2006] Erkki Oja and Zhijian Yuan. The fas- tica algorithm revisited: Convergence analysis. IEEE transactions on neural networks, 17:1370-81, 12 2006.","title":"The fastica algorithm revisited: Convergence analysis","authors":"Yuan ; ] Erkki Oja, Zhijian Oja, Yuan","year":"2006","venue":"IEEE transactions on neural networks","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Peng et al., 2019] Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. Mcp: Learn- ing composable hierarchical control with multiplicative compositional policies. arXiv preprint arXiv:1905.09808, 2019.","title":"Mcp: Learning composable hierarchical control with multiplicative compositional policies","authors":"Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, Sergey Levine","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Peters et al., 2017] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foun- dations and learning algorithms. MIT press, 2017.","title":"Elements of causal inference: foundations and learning algorithms","authors":"Peters","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sahni et al., 2017] Himanshu Sahni, Saurabh Kumar, Farhan Tejani, and Charles Isbell. Learning to compose skills. arXiv preprint arXiv:1711.11289, 2017.","title":"Learning to compose skills","authors":"Himanshu Sahni, Saurabh Kumar, Farhan Tejani, Charles Isbell","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schulman et al., 2015] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advan- tage estimation. arXiv preprint arXiv:1506.02438, 2015.","title":"Highdimensional continuous control using generalized advantage estimation","authors":"John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sharma et al., 2019] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. arXiv preprint arXiv:1907.01657, 2019.","title":"Dynamicsaware unsupervised discovery of skills","authors":"Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, Karol Hausman","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Singh et al., 2019] Avi Singh, Larry Yang, Kristian Har- tikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineer- ing. arXiv preprint arXiv:1904.07854, 2019.","title":"End-to-end robotic reinforcement learning without reward engineering","authors":"Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, Sergey Levine","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wold et al., 1987] Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37-52, 1987.","title":"Principal component analysis","authors":"Svante Wold, Kim Esbensen, Paul Geladi","year":"1987","venue":"Chemometrics and intelligent laboratory systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhu et al., 2017] Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA), pages 3357-3364. IEEE, 2017.","title":"Target-driven visual navigation in indoor scenes using deep reinforcement learning","authors":"Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, Ali Farhadi","year":"2017","venue":"IEEE international conference on robotics and automation (ICRA)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}