{"type":"ScholarlyArticle","title":"Tensor Based Knowledge Transfer Across Skill Categories for Robot Control","authors":[{"name":"Chenyang Zhao","slug":"chenyang-zhao","given":"Chenyang","family":"Zhao","affiliation":"University of Edinburgh , School of Informatics , Edinburgh , United Kingdom"},{"name":"Timothy M. Hospedales","slug":"timothy-m-hospedales","given":"Timothy M.","family":"Hospedales","affiliation":"University of Edinburgh , School of Informatics , Edinburgh , United Kingdom"},{"name":"Freek Stulp","slug":"freek-stulp","given":"Freek","family":"Stulp","affiliation":"German Aerospace Center (DLR) , Institute of Robotics and Mechatronics , Wessling , Germany"},{"name":"Olivier Sigaud","slug":"olivier-sigaud","given":"Olivier","family":"Sigaud","affiliation":"Sorbonne Universités , UPMC Univ Paris 06 , CNRS UMR 7222 , ISIR , F-75005 Paris , France"}],"abstract":"Advances in hardware and learning for control are enabling robots to perform increasingly dextrous and dynamic control tasks. These skills typically require a prohibitive amount of exploration for reinforcement learning, and so are commonly achieved by imitation learning from manual demonstration. The costly non-scalable nature of manual demonstration has motivated work into skill generalisation, e.g., through contextual policies and options. Despite good results, existing work along these lines is limited to generalising across variants of one skill such as throwing an object to different locations. In this paper we go significantly further and investigate generalisation across qualitatively different classes of control skills. In particular, we introduce a class of neural network controllers that can realise four distinct skill classes: reaching, object throwing, casting, and ball-in-cup. By factorising the weights of the neural network, we are able to extract transferrable latent skills, that enable dramatic acceleration of learning in cross-task transfer. With a suitable curriculum, this allows us to learn challenging dextrous control tasks like ball-in-cup from scratch with pure reinforcement learning.","doi":"10.24963/ijcai.2017/484","url":"https://ijcai.org/proceedings/2017/484","pdf":"https://ijcai.org/proceedings/2017/0484.pdf","markdown":"https://ijcai.org/proceedings/2017/484.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/484","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":484,"pages":{"first":3462,"last":3468},"keywords":["Robotics and Vision: Developmental Robotics","Machine Learning: Transfer, Adaptation, Multi-task Learning"],"videos":[],"references":[{"position":1,"raw":"[Ammar et al., 2014] Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, and Matthew Taylor. Online multi-task learning for policy gradient methods. In ICML, 2014.","title":"Online multi-task learning for policy gradient methods","authors":"Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, Matthew Taylor","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Argall et al., 2009] Brenna D. Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robot. Auton. Syst., 57(5):469-483, May 2009.","title":"A survey of robot learning from demonstration","authors":"Brenna D Argall, Sonia Chernova, Manuela Veloso, Brett Browning","year":"2009","venue":"Robot. Auton. Syst","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio et al., 2009] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, 2009.","title":"Curriculum learning","authors":"Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deisenroth et al., 2014] M. P. Deisenroth, P. Englert, J. Pe- ters, and D. Fox. Multi-task policy search for robotics. In ICRA, pages 3876-3881, May 2014.","title":"Multi-task policy search for robotics","authors":"M P Deisenroth, P Englert, J Peters, D Fox","year":"2014","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Isele et al., 2016] David Isele, Jose Marcio Luna, Eric Eaton, Gabriel V. de la Cruz, James Irwin, Brandon Kalla- her, and Matthew E. Taylor. Lifelong learning for distur- bance rejection on mobile robots. In IROS, 2016.","title":"Lifelong learning for disturbance rejection on mobile robots","authors":"David Isele, Jose, Marcio Luna, Eric Eaton, V Gabriel, James De La Cruz, Brandon Irwin, Matthew E Kallaher, Taylor","year":"2016","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kern et al., 2004] Stefan Kern, Sibylle D Müller, Nikolaus Hansen, Dirk Büche, Jiri Ocenasek, and Petros Koumout- sakos. Learning probability distributions in continuous evolutionary algorithms-a comparative review. Natural Computing, 3(1):77-112, 2004.","title":"Learning probability distributions in continuous evolutionary algorithms-a comparative review","authors":"Stefan Kern, Sibylle D Müller, Nikolaus Hansen, Dirk Büche, Jiri Ocenasek, Petros Koumoutsakos","year":"2004","venue":"Natural Computing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kober and Peters, 2010] Jens Kober and Jan Peters. Policy search for motor primitives in robotics. Machine Learning, 84(1):171-203, 2010.","title":"Policy search for motor primitives in robotics","authors":"Peters Kober, Jens Kober, Jan Peters","year":"2010","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kober et al., 2010] J. Kober, E. Oztop, and J. Peters. Rein- forcement learning to adjust robot movements to new sit- uations. In Robotics: Science and Systems (R:SS), 2010.","title":"Reinforcement learning to adjust robot movements to new situations","authors":"J Kober, E Oztop, J Peters","year":"2010","venue":"Robotics: Science and Systems (R:SS)","doi":null,"url":null,"internal_url":"/proceedings/2011/441"},{"position":9,"raw":"[Kormushev et al., 2010] P. Kormushev, S. Calinon, and D. G. Caldwell. Robot motor skill coordination with em- based reinforcement learning. In IROS, 2010.","title":"Robot motor skill coordination with embased reinforcement learning","authors":"P Kormushev, S Calinon, D G Caldwell","year":"2010","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kumar and Daume III, 2012] Abhishek Kumar and Hal Daume III. Learning task grouping and overlap in multi- task learning. In ICML, 2012.","title":"Learning task grouping and overlap in multitask learning","authors":"Daume Iii ; Kumar, Hal Kumar, Iii Daume","year":"2012","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kupcsik et al., 2013] Andras Gabor Kupcsik, Marc Peter Deisenroth, Jan Peters, and Gerhard Neumann. Data- efficient generalization of robot skills with contextual pol- icy search. In AAAI, 2013.","title":"Dataefficient generalization of robot skills with contextual policy search","authors":"Andras Gabor Kupcsik, Marc Peter Deisenroth, Jan Peters, Gerhard Neumann","year":"2013","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lathauwer et al., 2000] L. D. Lathauwer, B. D. Moor, and J. Vandewalle. A multilinear singular value decomposi- tion. In SIAM Journal on Matrix Analysis and Applica- tions, 2000.","title":"A multilinear singular value decomposition","authors":"L D Lathauwer, B D Moor, J Vandewalle","year":"2000","venue":"SIAM Journal on Matrix Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lazaric and Ghavamzadeh, 2010] Alessandro Lazaric and Mohammad Ghavamzadeh. Bayesian multi-task reinforce- ment learning. In ICML, 2010.","title":"Bayesian multi-task reinforcement learning","authors":"Alessandro Lazaric, Mohammad Ghavamzadeh","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lebedev et al., 2015] Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine- tuned cp-decomposition. In ICLR, 2015.","title":"Speeding-up convolutional neural networks using finetuned cp-decomposition","authors":"Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, Victor Lempitsky","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Luck et al., 2014] K. S. Luck, G. Neumann, E. Berger, J. Peters, and H. B. Amor. Latent space policy search for robotics. In IROS, 2014. [Mokhtari and Benallegue, 2004] Abdellah Mokhtari and Abdelaziz Benallegue. Dynamic feedback controller of euler angles and wind parameters estimation for a quadro- tor unmanned aerial vehicle. In ICRA, 2004.","title":"Dynamic feedback controller of euler angles and wind parameters estimation for a quadrotor unmanned aerial vehicle","authors":"K S Luck, G Neumann, E Berger, J Peters, H B Amor, Abdellah Mokhtari, Abdelaziz Benallegue","year":"2004","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Parisotto et al., 2016] Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. In ICLR, 2016.","title":"Actor-mimic: Deep multitask and transfer reinforcement learning","authors":"Emilio Parisotto, Jimmy Lei Ba, Ruslan Salakhutdinov","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Peters and Schaal, 2008] Jan Peters and Stefan Schaal. Re- inforcement learning of motor skills with policy gradients. Neural Networks, 21(4):682 -697, 2008.","title":"Reinforcement learning of motor skills with policy gradients","authors":"Schaal Peters, Jan Peters, Stefan Schaal","year":"2008","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ring, 1998] Mark B Ring. Child: A first step towards con- tinual learning. Machine Learning, pages 261-292, 1998.","title":"Child: A first step towards continual learning","authors":"B Mark, Ring","year":"1998","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ruvolo and Eaton, 2013a] Paul Ruvolo and Eric Eaton. Ac- tive task selection for lifelong machine learning. In AAAI, 2013.","title":"Active task selection for lifelong machine learning","authors":"Eaton Ruvolo, Paul Ruvolo, Eric Eaton","year":"2013","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ruvolo and Eaton, 2013b] Paul Ruvolo and Eric Eaton. Ella: An efficient lifelong learning algorithm. In ICML, 2013.","title":"Ella: An efficient lifelong learning algorithm","authors":"Eaton Ruvolo, Paul Ruvolo, Eric Eaton","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schaal et al., 2005] Stefan Schaal, Jan Peters, Jun Nakan- ishi, and Auke Ijspeert. Learning movement primitives. In International Symposium on Robotics Research, 2005.","title":"Learning movement primitives","authors":"Stefan Schaal, Jan Peters, Jun Nakanishi, Auke Ijspeert","year":"2005","venue":"International Symposium on Robotics Research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Selfridge et al., 1985] Oliver G Selfridge, Richard S Sutton, and Andrew G Barto. Training and tracking in robotics. In IJCAI, pages 670-672, 1985.","title":"Training and tracking in robotics","authors":"Richard S Oliver G Selfridge, Andrew G Sutton, Barto","year":"1985","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/1985/130"},{"position":23,"raw":"[Stulp and Sigaud, 2013] F. Stulp and O. Sigaud. Robot skill learning: From reinforcement learning to evolution strate- gies. Paladyn, Journal of Behavioral Robotics, 4(1):49- 61, 2013.","title":"Robot skill learning: From reinforcement learning to evolution strategies","authors":"F Stulp, O Sigaud","year":"2013","venue":"Paladyn, Journal of Behavioral Robotics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Stulp et al., 2013] F. Stulp, G. Raiola, A. Hoarau, S. Ivaldi, and O. Sigaud. Learning compact parameterized skills with a single regression. In Humanoids, 2013.","title":"Learning compact parameterized skills with a single regression","authors":"F Stulp, G Raiola, A Hoarau, S Ivaldi, O Sigaud","year":"2013","venue":"Humanoids","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Stulp et al., 2014] Freek Stulp, Laura Herlant, Antoine Hoarau, and Gennaro Raiola. Simultaneous on-line dis- covery and improvement of robotic skill options. In IROS, 2014.","title":"Simultaneous on-line discovery and improvement of robotic skill options","authors":"Freek Stulp, Laura Herlant, Antoine Hoarau, Gennaro Raiola","year":"2014","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Taylor and Stone, 2009] Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(Jul):1633-1685, 2009.","title":"Transfer learning for reinforcement learning domains: A survey","authors":"E Matthew, Peter Taylor, Stone","year":"2009","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Thrun, 1996] Sebastian Thrun. Is learning the n-th thing any easier than learning the first? In NIPS, 1996.","title":"Is learning the n-th thing any easier than learning the first?","authors":"Sebastian Thrun, Thrun","year":"1996","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tucker, 1966] Ledyard R Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3):279- 311, 1966.","title":"Some mathematical notes on three-mode factor analysis","authors":"R Ledyard, Tucker","year":"1966","venue":"Psychometrika","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wimalawarne et al., 2014] Kishan Wimalawarne, Masashi Sugiyama, and Ryota Tomioka. Multitask learning meets tensor factorization: task imputation via convex optimiza- tion. In NIPS, 2014.","title":"Multitask learning meets tensor factorization: task imputation via convex optimization","authors":"Kishan Wimalawarne, Masashi Sugiyama, Ryota Tomioka","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang and Hospedales, 2015] Yongxin Yang and Timo- thy M. Hospedales. A unified perspective on multi-domain and multi-task learning. In ICLR, 2015.","title":"A unified perspective on multi-domain and multi-task learning","authors":"Yang, Hospedales ;, Yongxin Yang, Timothy M Hospedales","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang and Hospedales, 2017] Yongxin Yang and Timo- thy M. Hospedales. Deep multi-task representation learning: A tensor factorisation approach. ICLR, 2017.","title":"Deep multi-task representation learning: A tensor factorisation approach","authors":"Yang, Hospedales ;, Yongxin Yang, Timothy M Hospedales","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}