{"type":"ScholarlyArticle","title":"Using Task Features for Zero-Shot Knowledge Transfer in Lifelong Learning","authors":[{"name":"David Isele","slug":"david-isele","given":"David","family":"Isele","affiliation":"University of Pennsylvania , Philadelphia , PA , USA"},{"name":"Mohammad Rostami","slug":"mohammad-rostami","given":"Mohammad","family":"Rostami","affiliation":"University of Pennsylvania , Philadelphia , PA , USA"},{"name":"Eric Eaton","slug":"eric-eaton","given":"Eric","family":"Eaton","affiliation":"University of Pennsylvania , Philadelphia , PA , USA"}],"abstract":"Knowledge transfer between tasks can improve the performance of learned models, but requires an accurate estimate of the inter-task relationships to identify the relevant knowledge to transfer. These inter-task relationships are typically estimated based on training data for each task, which is inefficient in lifelong learning settings where the goal is to learn each consecutive task rapidly from as little data as possible. To reduce this burden, we develop a lifelong reinforcement learning method based on coupled dictionary learning that incorporates high-level task descriptors to model the inter-task relationships. We show that using task descriptors improves the performance of the learned task policies, providing both theoretical justification for the benefit and empirical demonstration of the improvement across a variety of dynamical control problems. Given only the descriptor for a new task, the lifelong learner is also able to accurately predict the task policy through zero-shot learning using the coupled dictionary, eliminating the need to pause to gather training data before addressing the task.","doi":null,"url":"https://ijcai.org/proceedings/2016/232","pdf":"https://ijcai.org/proceedings/2016/0232.pdf","markdown":"https://ijcai.org/proceedings/2016/232.md","bibtex":"https://ijcai.org/proceedings/2016/bibtex/232","volume":{"slug":"2016","title":"Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2016","url":"https://ijcai.org/proceedings/2016/","year":2016,"location":"New York, NY, USA","dates":"9–15 July 2016","editors":"Subbarao Kambhampati","isbn":"978-1-57735-770-4; 978-1-57735-771-1","publisher":"AAAI Press / International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":null,"track":"","number":232,"pages":{"first":1620,"last":1625},"keywords":[],"videos":[],"references":[{"position":1,"raw":"[Ando & Zhang, 2005] Rie Kubota Ando & Tong Zhang. A frame- work for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6:1817- 1853, 2005.","title":"A framework for learning predictive structures from multiple tasks and unlabeled data","authors":"Rie Kubota, Ando, Tong Zhang","year":"2005","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bakker & Heskes, 2003] Bart Bakker & Tom Heskes. Task clus- tering and gating for Bayesian multi-task learning. Journal of Machine Learning Research, 4:83-99, 2003.","title":"Task clustering and gating for Bayesian multi-task learning","authors":"& Bakker, Bart Heskes, Tom Bakker, Heskes","year":"2003","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baxter, 2000] Jonathan Baxter. A model of inductive bias learning. Journal of Artificial Intelligence Research, 12:149-198, 2000.","title":"A model of inductive bias learning","authors":"Jonathan Baxter","year":"2000","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ben-David et al., 2007] Shai Ben-David, John Blitzer, Koby Crammer, & Fernando Pereira. Analysis of representations for domain adaptation. Neural Information Processing Systems, 19:137-144, 2007.","title":"Analysis of representations for domain adaptation","authors":"Shai Ben-David, John Blitzer, Koby Crammer, Fernando Pereira","year":"2007","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bickel et al., 2009] Steffen Bickel, Christoph Sawade, & Tobias Scheffer. Transfer learning by distribution matching for targeted advertising. Neural Information Processing Systems, 2009.","title":"Transfer learning by distribution matching for targeted advertising","authors":"Steffen Bickel, Christoph Sawade, Tobias Scheffer","year":"2009","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bonilla et al., 2007] Edwin Bonilla, Felix Agakov, & Christopher Williams. Kernel multi-task learning using task-specific features. Int. Conference on Artificial Intelligence and Statistics, 2007.","title":"Kernel multi-task learning using task-specific features","authors":"Edwin Bonilla, Felix Agakov, Christopher Williams","year":"2007","venue":"Int. Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bou Ammar et al., 2014] Haitham Bou Ammar, Eric Eaton, & Paul Ruvolo. Online multi-task learning for policy gradient meth- ods. International Conference on Machine Learning, 2014.","title":"Online multi-task learning for policy gradient methods","authors":"Haitham Bou Ammar, Eric Eaton, Paul Ruvolo","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bou Ammar et al., 2015] Haitham Bou Ammar, Eric Eaton, Jose Marcio Luna, & Paul Ruvolo. Autonomous cross-domain knowl- edge transfer in lifelong policy gradient reinforcement learning. International Joint Conference on Artificial Intelligence, 2015.","title":"Autonomous cross-domain knowledge transfer in lifelong policy gradient reinforcement learning","authors":"Haitham Bou Ammar, Eric Eaton, Jose, Marcio Luna, Paul Ruvolo","year":"2015","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2015/471"},{"position":9,"raw":"[Bouabdallah & Siegwart, 2005] Samir Bouabdallah & Roland Siegwart. Backstepping and sliding-mode techniques applied to an indoor micro quadrotor. IEEE International Conference on Robotics and Automation., 2005.","title":"Backstepping and sliding-mode techniques applied to an indoor micro quadrotor","authors":"Samir Bouabdallah, Roland Siegwart","year":"2005","venue":"IEEE International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Caruana, 1997] Rich Caruana. Multitask Learning. Machine Learning, 28:41-75, 1997.","title":"Multitask Learning","authors":"Caruana","year":"1997","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Donoho & Huo, 2001] David Donoho & Xiaoming Huo. Uncer- tainty principles and ideal atomic decomposition. IEEE Transac- tions on Information Theory, 47(7):2845-2862, 2001.","title":"Uncertainty principles and ideal atomic decomposition","authors":"& Donoho, ; Huo, David Donoho, Xiaoming Huo","year":"2001","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Donoho et al., 2006] David Donoho, Michael Elad, & Vladimir Temlyakov. Stable recovery of sparse overcomplete representa- tions in the presence of noise. IEEE Transactions on Information Theory, 52(1):6-18, 2006.","title":"Stable recovery of sparse overcomplete representations in the presence of noise","authors":"David Donoho, Michael Elad, Vladimir Temlyakov","year":"2006","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kober & Peters, 2009] Jens Kober & Jan Peters. Policy search for motor primitives in robotics. Neural Information Processing Sys- tems, pp. 849-856, 2009.","title":"Policy search for motor primitives in robotics","authors":"& Kober, Jens Peters, Jan Kober, Peters","year":"2009","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kumar & Daumé, 2012] Abhishek Kumar & Hal Daumé. Learn- ing task grouping and overlap in multi-task learning. Interna- tional Conference on Machine Learning, 2012.","title":"Learning task grouping and overlap in multi-task learning","authors":"Daumé Kumar, Abhishek Kumar, Hal Daumé","year":"2012","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lazaric & Ghavamzadeh, 2010] Alessandro Lazaric & Moham- mad Ghavamzadeh. Bayesian multi-task reinforcement learning. International Conference on Machine Learning, 2010.","title":"Bayesian multi-task reinforcement learning","authors":"Alessandro Lazaric, Mohammad Ghavamzadeh","year":"2010","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Maurer et al., 2013] Andreas Maurer, Massimiliano Pontil, & Bernardino Romera-Paredes. Sparse coding for multitask and transfer learning. Int. Conference on Machine Learning, 2013. [Negahban et al., 2009] Sahand Negahban, Bin Yu, Martin Wain- wright, & Pradeep Ravikumar. A unified framework for high- dimensional analysis of m-estimators with decomposable regu- larizers. Neural Information Processing Systems, 2009.","title":"Sparse coding for multitask and transfer learning","authors":"Andreas Maurer, Massimiliano Pontil, Bernardino Romera-Paredes","year":"2009","venue":"A unified framework for highdimensional analysis of m-estimators with decomposable regularizers","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Oyen & Lane, 2012] Diane Oyen & Terran Lane. Leveraging do- main knowledge in multitask Bayesian network structure learn- ing. AAAI Conference on Artificial Intelligence, 2012.","title":"Leveraging domain knowledge in multitask Bayesian network structure learning","authors":"Lane ; Oyen, Diane Oyen, Terran Lane","year":"2012","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Palatucci et al., 2009] Mark Palatucci, Geoffrey Hinton, Dean Pomerleau, & Tom Mitchell. Zero-shot learning with semantic output codes. Neural Information Processing Systems, 2009.","title":"Zero-shot learning with semantic output codes","authors":"Mark Palatucci, Geoffrey Hinton, Dean Pomerleau, Tom Mitchell","year":"2009","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Parameswaran & Weinberger, 2010] Shibin Parameswaran & Kil- ian Q. Weinberger. Large margin multi-task metric learning. Neu- ral Information Processing Systems, pp. 1867-1875, 2010.","title":"Large margin multi-task metric learning","authors":"& Parameswaran, Q Weinberger ; Shibin Parameswaran & Kilian, Weinberger","year":"2010","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, & Christopher Manning. GloVe: Global vectors for word repre- sentation. Empiricial Methods in Natural Language Processing, 12:1532-1543, 2014.","title":"GloVe: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"Empiricial Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Peters & Schaal, 2008] Jan Peters & Stefan Schaal. Natural actor- critic. Neurocomputing, 71(7):1180-1190, 2008.","title":"Natural actorcritic","authors":"Schaal Peters, Jan Peters, Stefan Schaal","year":"2008","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Romera-Paredes & Torr, 2015] Bernardino Romera-Paredes & Philip Torr. An embarrassingly simple approach to zero-shot learning. International Conference on Machine Learning, 2015.","title":"Bernardino Romera-Paredes & Philip Torr. An embarrassingly simple approach to zero-shot learning","authors":"Romera-Paredes & Torr","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ruvolo & Eaton, 2013] Paul Ruvolo & Eric Eaton. ELLA: an ef- ficient lifelong learning algorithm. International Conference on Machine Learning, 2013.","title":"ELLA: an efficient lifelong learning algorithm","authors":"& Ruvolo, Paul Eaton, Eric Ruvolo, Eaton","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sinapov et al., 2015] Jivko Sinapov, Sanmit Narvekar, Matteo Leonetti, & Peter Stone. Learning inter-task transferability in the absence of target task samples. International Conference on Autonomous Agents and Multiagent Systems, 2015.","title":"Learning inter-task transferability in the absence of target task samples","authors":"Jivko Sinapov, Sanmit Narvekar, Matteo Leonetti, Peter Stone","year":"2015","venue":"International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Socher et al., 2013] Richard Socher, Milind Ganjoo, Christopher Manning, & Andrew Ng. Zero-shot learning through cross-modal transfer. Neural Information Processing Systems, 2013.","title":"Zero-shot learning through cross-modal transfer","authors":"Richard Socher, Milind Ganjoo, Christopher Manning, Andrew Ng","year":"2013","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sutton et al., 1999] Richard Sutton, David McAllester, Satinder Singh, & Yishay Mansour. Policy gradient methods for reinforce- ment learning with function approximation. Neural Information Processing Systems, 99:1057-1063, 1999.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"Richard Sutton, David Mcallester, Satinder Singh, Yishay Mansour","year":"1999","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Taylor & Stone, 2009] Matthew E. Taylor & Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10:1633-1685, 2009.","title":"Transfer learning for reinforcement learning domains: A survey","authors":"Matthew E Taylor, Peter Stone","year":"2009","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Thrun, 1996] Sebastian Thrun. Is learning the n-th thing any easier than learning the first? Neural Information Processing Systems, pp. 640-646, 1996.","title":"Is learning the n-th thing any easier than learning the first? Neural Information Processing Systems","authors":"Sebastian Thrun, Thrun","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2014] Qifan Wang, Lingyun Ruan, & Luo Si. Adap- tive knowledge transfer for multiple instance learning in image classification. AAAI Conference on Artificial Intelligence, 2014.","title":"Adaptive knowledge transfer for multiple instance learning in image classification","authors":"Wang","year":"2014","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Williams, 1992] Ronald Williams. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229-256, 1992.","title":"Simple statistical gradientfollowing algorithms for connectionist reinforcement learning","authors":"Ronald Williams","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang et al., 2010] Jianchao Yang, John Wright, Thomas Huang, & Yi Ma. Image super-resolution via sparse representation. IEEE Transactions on Image Processing, 19(11):2861-2873, 2010.","title":"Image super-resolution via sparse representation","authors":"Jianchao Yang, John Wright, Thomas Huang, Yi Ma","year":"2010","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yu et al., 2014] Zhou Yu, Fei Wu, Yi Yang, Qi Tian, Jiebo Luo, & Yueting Zhuang. Discriminative coupled dictionary hashing for fast cross-media retrieval. International ACM SIGIR Conference on Research & Development in Information Retrieval, 2014.","title":"Discriminative coupled dictionary hashing for fast cross-media retrieval","authors":"Zhou Yu, Fei Wu, Yi Yang, Qi Tian, Jiebo Luo, Yueting Zhuang","year":"2014","venue":"International ACM SIGIR Conference on Research & Development in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhuang et al., 2013] Yue Ting Zhuang, Yan Fei Wang, Fei Wu, Yin Zhang, & Wei Ming Lu. Supervised coupled dictionary learning with group structures for multi-modal retrieval. AAAI Conference on Artificial Intelligence, 2013.","title":"Supervised coupled dictionary learning with group structures for multi-modal retrieval","authors":"Yue Ting, Zhuang, Yan Fei Wang, Fei Wu, Yin Zhang, Ming Wei, Lu","year":"2013","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}