{"type":"ScholarlyArticle","title":"Understanding How Feature Structure Transfers in Transfer Learning","authors":[{"name":"Tongliang Liu","slug":"tongliang-liu","given":"Tongliang","family":"Liu","affiliation":"Hong Kong University of Science and Technology , Hong Kong"},{"name":"Qiang Yang","slug":"qiang-yang","given":"Qiang","family":"Yang","affiliation":"‡ UBTech Sydney AI Institute and SIT , FEIT , The University of Sydney , Australia"},{"name":"Dacheng Tao","slug":"dacheng-tao","given":"Dacheng","family":"Tao","affiliation":"Hong Kong University of Science and Technology , Hong Kong"}],"abstract":"Transfer learning transfers knowledge across domains to improve the learning performance. Since feature structures generally represent the common knowledge across different domains, they can be transferred successfully even though the labeling functions across domains differ arbitrarily. However, theoretical justification for this success has remained elusive. In this paper, motivated by self-taught learning, we regard a set of bases as a feature structure of a domain if the bases can (approximately) reconstruct any observation in this domain. We propose a general analysis scheme to theoretically justify that if the source and target domains share similar feature structures, the source domain feature structure is transferable to the target domain, regardless of the change of the labeling functions across domains. The transferred structure is interpreted to function as a regularization matrix which benefits the learning process of the target domain task. We prove that such transfer enables the corresponding learning algorithms to be uniformly stable. Specifically, we illustrate the existence of feature structure transfer in two well-known transfer learning settings: domain adaptation and learning to learn.","doi":"10.24963/ijcai.2017/329","url":"https://ijcai.org/proceedings/2017/329","pdf":"https://ijcai.org/proceedings/2017/0329.pdf","markdown":"https://ijcai.org/proceedings/2017/329.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/329","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra, IIIA-CSIC","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":329,"pages":{"first":2365,"last":2371},"keywords":["Machine Learning: Learning Theory","Machine Learning: Transfer, Adaptation, Multi-task Learning"],"videos":[],"references":[{"position":1,"raw":"[Ando and Zhang, 2005] Rie Kubota Ando and Tong Zhang. A framework for learning predictive structures from multi- ple tasks and unlabeled data. Journal of Machine Learning Research, 6:1817-1853, 2005.","title":"A framework for learning predictive structures from multiple tasks and unlabeled data","authors":"Rie Kubota, Ando, Tong Zhang","year":"2005","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baxter, 2000] Jonathan Baxter. A model of inductive bias learning. Journal of Artificial Intelligence Research, 12(1):149-198, 2000.","title":"A model of inductive bias learning","authors":"Jonathan Baxter","year":"2000","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ben-David et al., 2007] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In NIPS, 2007.","title":"Analysis of representations for domain adaptation","authors":"Shai Ben-David, John Blitzer, Koby Crammer, Fernando Pereira","year":"2007","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ben-David et al., 2010] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine Learning, 79(1-2):151-175, 2010.","title":"A theory of learning from different domains","authors":"Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, Jennifer Wortman Vaughan","year":"2010","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Blitzer et al., 2008] John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman. Learn- ing bounds for domain adaptation. In NIPS, 2008.","title":"Learning bounds for domain adaptation","authors":"John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, Jennifer Wortman","year":"2008","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bousquet and Elisseeff, 2002] Olivier Bousquet and André Elisseeff. Stability and generalization. Journal of Machine Learning Research, 2:499-526, 2002.","title":"Stability and generalization","authors":"Elisseeff Bousquet, Olivier Bousquet, André Elisseeff","year":"2002","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Evgeniou and Pontil, 2004] Theodoros Evgeniou and Mas- similiano Pontil. Regularized multi-task learning. In SIGKDD, 2004.","title":"Theodoros Evgeniou and Massimiliano Pontil. Regularized multi-task learning","authors":"Pontil Evgeniou","year":"2004","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gong et al., 2016] Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transfer- able components. In ICML, pages 2839-2848, 2016.","title":"Domain adaptation with conditional transferable components","authors":"Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, Bernhard Schölkopf","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2006] Jiayuan Huang, Arthur Gretton, Karsten M Borgwardt, Bernhard Schölkopf, and Alex J Smola. Correcting sample selection bias by unlabeled data. In NIPS, 2006.","title":"Correcting sample selection bias by unlabeled data","authors":"Jiayuan Huang, Arthur Gretton, Karsten M Borgwardt, Bernhard Schölkopf, Alex J Smola","year":"2006","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kakade and Tewari, 2009] Sham M Kakade and Ambuj Tewari. On the generalization ability of online strongly convex programming algorithms. In NIPS, 2009.","title":"On the generalization ability of online strongly convex programming algorithms","authors":"M Sham, Ambuj Kakade, Tewari","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kuzborskij and Orabona, 2013] Ilja Kuzborskij and Francesco Orabona. Stability and hypothesis transfer learning. In ICML, 2013.","title":"Stability and hypothesis transfer learning","authors":"Orabona Kuzborskij, Ilja Kuzborskij, Francesco Orabona","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2015] Ya Li, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Multi-task model and feature joint learning. In IJCAI, pages 3643-3649, 2015.","title":"Multi-task model and feature joint learning","authors":"Ya Li, Xinmei Tian, Tongliang Liu, Dacheng Tao","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2016] Hongfu Liu, Ming Shao, and Yun Fu. Structure-preserved multi-source domain adaptation. In ICDM, pages 1059-1064, 2016.","title":"Structure-preserved multi-source domain adaptation","authors":"Hongfu Liu, Ming Shao, Yun Fu","year":"2016","venue":"ICDM","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2017a] Tongliang Liu, Gábor Lugosi, Gergely Neu, and Dacheng Tao. Algorithmic stability and hypoth- esis complexity. arXiv preprint arXiv:1702.08712, 2017.","title":"Algorithmic stability and hypothesis complexity","authors":"Liu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2017b] Tongliang Liu, Dacheng Tao, Mingli Song, and Stephen J Maybank. Algorithm-dependent gen- eralization bounds for multi-task learning. IEEE trans- actions on pattern analysis and machine intelligence, 39(2):227-241, 2017.","title":"Algorithm-dependent generalization bounds for multi-task learning","authors":"Liu","year":"2017","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Long et al., 2015] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In ICML, pages 97-105, 2015.","title":"Learning transferable features with deep adaptation networks","authors":"Mingsheng Long, Yue Cao, Jianmin Wang, Michael Jordan","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Luo et al., 2014] Yong Luo, Tongliang Liu, Dacheng Tao, and Chao Xu. Decomposition-based transfer distance met- ric learning for image classification. IEEE Transactions on Image Processing, 23(9):3789-3801, 2014.","title":"Decomposition-based transfer distance metric learning for image classification","authors":"Yong Luo, Tongliang Liu, Dacheng Tao, Chao Xu","year":"2014","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mansour et al., 2009] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. In COLT, 2009.","title":"Domain adaptation: Learning bounds and algorithms","authors":"Yishay Mansour, Mehryar Mohri, Afshin Rostamizadeh","year":"2009","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Maurer et al., 2013] Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. Sparse coding for mul- titask and transfer learning. In ICML, 2013.","title":"Sparse coding for multitask and transfer learning","authors":"Andreas Maurer, Massimiliano Pontil, Bernardino Romera-Paredes","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Maurer, 2009] Andreas Maurer. Transfer bounds for linear feature learning. Machine Learning, 75(3):327-350, 2009.","title":"Transfer bounds for linear feature learning","authors":"Andreas Maurer","year":"2009","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mohri et al., 2012] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT Press, 2012.","title":"Foundations of machine learning","authors":"Mehryar Mohri, Afshin Rostamizadeh, Ameet Talwalkar","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pan and Yang, 2010] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowl- edge and Data Engineering, 22(10):1345-1359, 2010.","title":"A survey on transfer learning","authors":"Jialin Sinno, Qiang Pan, Yang","year":"2010","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pentina and Lampert, 2014] Anastasia Pentina and Christoph H Lampert. A PAC-bayesian bound for lifelong learning. In ICML, 2014.","title":"A PAC-bayesian bound for lifelong learning","authors":"Anastasia Pentina, Christoph H Lampert","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Raina et al., 2007] Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer, and Andrew Y Ng. Self-taught learning: Transfer learning from unlabeled data. In ICML, 2007.","title":"Self-taught learning: Transfer learning from unlabeled data","authors":"Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer, Andrew Y Ng","year":"2007","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shao et al., 2014] Ming Shao, Dmitry Kit, and Yun Fu. Generalized transfer subspace learning through low-rank constraint. International Journal of Computer Vision, 109(1-2):74-93, 2014.","title":"Generalized transfer subspace learning through low-rank constraint","authors":"Ming Shao, Dmitry Kit, Yun Fu","year":"2014","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shao et al., 2016] Ming Shao, Zhengming Ding, Handong Zhao, and Yun Fu. Spectral bisection tree guided deep adaptive exemplar autoencoder for unsupervised domain adaptation. In AAAI, 2016.","title":"Spectral bisection tree guided deep adaptive exemplar autoencoder for unsupervised domain adaptation","authors":"Ming Shao, Zhengming Ding, Handong Zhao, Yun Fu","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Si et al., 2010] Si Si, Dacheng Tao, and Bo Geng. Breg- man divergence-based regularization for transfer subspace learning. IEEE Transactions on Knowledge and Data En- gineering, 22(7):929-942, 2010.","title":"Bregman divergence-based regularization for transfer subspace learning","authors":"Si Si, Dacheng Tao, Bo Geng","year":"2010","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang and Schneider, 2014] Xuezhi Wang and Jeff Schnei- der. Flexible transfer learning under support and model shift. In NIPS, 2014.","title":"Xuezhi Wang and Jeff Schneider. Flexible transfer learning under support and model shift","authors":"Schneider Wang","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2013] Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adap- tation under target and conditional shift. In ICML, 2013.","title":"Domain adaptation under target and conditional shift","authors":"Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, Zhikun Wang","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2015] Kun Zhang, Mingming Gong, and Bern- hard Scholkopf. Multi-source domain adaptation: a causal view. In AAAI, pages 3150-3157, 2015.","title":"Multi-source domain adaptation: a causal view","authors":"Kun Zhang, Mingming Gong, Bernhard Scholkopf","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}