{"type":"ScholarlyArticle","title":"Completely Heterogeneous Transfer Learning with Attention - What And What Not To Transfer","authors":[{"name":"Seungwhan Moon","slug":"seungwhan-moon","given":"Seungwhan","family":"Moon","affiliation":"Language Technologies Institute School of Computer Science Carnegie Mellon University"},{"name":"Jaime Carbonell","slug":"jaime-carbonell","given":"Jaime","family":"Carbonell","affiliation":"Language Technologies Institute School of Computer Science Carnegie Mellon University"}],"abstract":"We study a transfer learning framework where source and target datasets are heterogeneous in both feature and label spaces. Specifically, we do not assume explicit relations between source and target tasks a priori, and thus it is crucial to determine what and what not to transfer from source knowledge. Towards this goal, we define a new heterogeneous transfer learning approach that (1) selects and attends to an optimized subset of source samples to transfer knowledge from, and (2) builds a unified transfer network that learns from both source and target knowledge. This method, termed \"Attentional Heterogeneous Transfer\", along with a newly proposed unsupervised transfer loss, improve upon the previous state-of-the-art approaches on extensive simulations as well as a challenging hetero-lingual text classification task.","doi":"10.24963/ijcai.2017/349","url":"https://ijcai.org/proceedings/2017/349","pdf":"https://ijcai.org/proceedings/2017/0349.pdf","markdown":"https://ijcai.org/proceedings/2017/349.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/349","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":349,"pages":{"first":2508,"last":2514},"keywords":["Machine Learning: Neural Networks","Machine Learning: Deep Learning","Machine Learning: Transfer, Adaptation, Multi-task Learning","Natural Language Processing: Text Classification"],"videos":[],"references":[{"position":1,"raw":"[Amini et al., 2009] Massih Amini, Nicolas Usunier, and Cyril Goutte. Learning from multiple partially observed views-an application to multilingual text categorization. In NIPS, pages 28-36, 2009.","title":"Learning from multiple partially observed views-an application to multilingual text categorization","authors":"Massih Amini, Nicolas Usunier, Cyril Goutte","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bordes et al., 2013] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, pages 2787-2795, 2013.","title":"Translating embeddings for modeling multi-relational data","authors":"Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, Oksana Yakhnenko","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chan et al., 2015] William Chan, Navdeep Jaitly, Quoc V Le, and Oriol Vinyals. Listen, attend and spell. arXiv preprint arXiv:1508.01211, 2015.","title":"Listen, attend and spell","authors":"William Chan, Navdeep Jaitly, Oriol Quoc V Le, Vinyals","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dai et al., 2008] Wenyuan Dai, Yuqiang Chen, Gui-Rong Xue, Qiang Yang, and Yong Yu. Translated learning: Transfer learning across different feature spaces. In NIPS, pages 353-360, 2008.","title":"Translated learning: Transfer learning across different feature spaces","authors":"Wenyuan Dai, Yuqiang Chen, Gui-Rong Xue, Qiang Yang, Yong Yu","year":"2008","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dhillon et al., 2011] Paramveer Dhillon, Dean P Foster, and Lyle H Ungar. Multi-view learning of word embeddings via cca. In NIPS, pages 199-207, 2011.","title":"Multi-view learning of word embeddings via cca","authors":"Dhillon","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Duan et al., 2012] Lixin Duan, Dong Xu, and Ivor Tsang. Learning with augmented features for heterogeneous do- main adaptation. ICML, 2012.","title":"Learning with augmented features for heterogeneous domain adaptation","authors":"Lixin Duan, Dong Xu, Ivor Tsang","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Frome et al., 2013] Andrea Frome, Greg Corrado, Jon Shlens, Samy Bengio, Jeffrey Dean, Marc'Aurelio Ran- zato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In NIPS, 2013.","title":"Devise: A deep visual-semantic embedding model","authors":"Andrea Frome, Greg Corrado, Jon Shlens, Samy Bengio, Jeffrey Dean, Marc'aurelio Ranzato, Tomas Mikolov","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gupta and Ratinov, 2008] Rakesh Gupta and Lev-Arie Rati- nov. Text categorization with knowledge transfer from het- erogeneous data sources. In AAAI, pages 842-847, 2008.","title":"Text categorization with knowledge transfer from heterogeneous data sources","authors":"Rakesh Gupta, Lev-Arie Ratinov","year":"2008","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hendricks et al., 2016] Lisa Anne Hendricks, Subhashini Venugopalan, Marcus Rohrbach, Raymond Mooney, Kate Saenko, and Trevor Darrell. Deep compositional cap- tioning: Describing novel object categories without paired training data. CVPR, 2016.","title":"Deep compositional captioning: Describing novel object categories without paired training data","authors":"Anne Lisa, Subhashini Hendricks, Marcus Venugopalan, Raymond Rohrbach, Kate Mooney, Trevor Saenko, Darrell","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2007] Jiayuan Huang, Arthur Gretton, Karsten M Borgwardt, Bernhard Schölkopf, and Alex J Smola. Correcting sample selection bias by unlabeled data. In NIPS, 2007.","title":"Correcting sample selection bias by unlabeled data","authors":"Jiayuan Huang, Arthur Gretton, Karsten M Borgwardt, Bernhard Schölkopf, Alex J Smola","year":"2007","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kodirov et al., 2015] Elyor Kodirov, Tao Xiang, Zhenyong Fu, and Shaogang Gong. Unsupervised domain adaptation for zero-shot learning. In ICCV, 2015.","title":"Unsupervised domain adaptation for zero-shot learning","authors":"Kodirov","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lewis et al., 2004] David D Lewis, Yiming Yang, Tony G Rose, and Fan Li. Rcv1: A new benchmark collection for text categorization research. Journal of machine learning research, 5(Apr):361-397, 2004.","title":"Rcv1: A new benchmark collection for text categorization research","authors":"Yiming David D Lewis, Tony G Yang, Fan Rose, Li","year":"2004","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Long and Wang, 2015] Mingsheng Long and Jianmin Wang. Learning transferable features with deep adapta- tion networks. ICML, 2015.","title":"Learning transferable features with deep adaptation networks","authors":"Mingsheng Long, Jianmin Wang","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mensink et al., 2014] Thomas Mensink, Efstratios Gavves, and Cees GM Snoek. Costa: Co-occurrence statistics for zero-shot classification. In CVPR, 2014. [Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Cor- rado, and Jeffrey Dean. Efficient estimation of word rep- resentations in vector space. ICLR, 2013.","title":"Costa: Co-occurrence statistics for zero-shot classification","authors":"Thomas Mensink, Efstratios Gavves, G M Cees, ; Snoek, Mikolov","year":"2013","venue":"Efficient estimation of word representations in vector space","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Miller, 1995] George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-41, 1995.","title":"Wordnet: a lexical database for english","authors":"George A Miller","year":"1995","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Moon and Carbonell, 2016] Seungwhan Moon and Jaime Carbonell. Proactive transfer learning for heterogeneous feature and label spaces. ECML-PKDD, 2016.","title":"Seungwhan Moon and Jaime Carbonell. Proactive transfer learning for heterogeneous feature and label spaces","authors":"Carbonell Moon","year":"2016","venue":"ECML-PKDD","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nickel et al., 2015] Maximilian Nickel, Lorenzo Rosasco, and Tomaso Poggio. Holographic embeddings of knowl- edge graphs. arXiv preprint arXiv:1510.04935, 2015.","title":"Holographic embeddings of knowledge graphs","authors":"Maximilian Nickel, Lorenzo Rosasco, Tomaso Poggio","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rosenstein et al., 2005] Michael T Rosenstein, Zvika Marx, Leslie Pack Kaelbling, and Thomas G Dietterich. To trans- fer or not to transfer. In NIPS 2005 Workshop on Inductive Transfer: 10 Years Later, volume 2, page 7, 2005.","title":"To transfer or not to transfer","authors":"Zvika Michael T Rosenstein, Leslie Marx, Thomas G Pack Kaelbling, Dietterich","year":"2005","venue":"NIPS 2005 Workshop on Inductive Transfer: 10 Years Later","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Socher et al., 2013] Richard Socher, Milind Ganjoo, Christopher D. Manning, and Andrew Y. Ng. Zero Shot Learning Through Cross-Modal Transfer. In NIPS. 2013.","title":"Zero Shot Learning Through Cross-Modal Transfer","authors":"Richard Socher, Milind Ganjoo, Christopher D Manning, Andrew Y Ng","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sukhbaatar et al., 2015] Sainbayar Sukhbaatar, Jason We- ston, Rob Fergus, et al. End-to-end memory networks. In NIPS, pages 2440-2448, 2015.","title":"End-to-end memory networks","authors":"Sukhbaatar","year":"2015","venue":"Sainbayar Sukhbaatar","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sun et al., 2015] Qian Sun, Mohammad Amin, Baoshi Yan, Craig Martell, Vita Markman, Anmol Bhasin, and Jieping Ye. Transfer learning for bilingual content classification. In KDD, pages 2147-2156, 2015.","title":"Transfer learning for bilingual content classification","authors":"Qian Sun, Mohammad Amin, Baoshi Yan, Craig Martell, Vita Markman, Anmol Bhasin, Jieping Ye","year":"2015","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2014] Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by translating on hyperplanes. In AAAI, pages 1112-1119.","title":"Knowledge graph embedding by translating on hyperplanes","authors":"Zhen Wang, Jianwen Zhang, Jianlin Feng, Zheng Chen","year":null,"venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2015] Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi-view represen- tation learning. ICML, 2015.","title":"On deep multi-view representation learning","authors":"Wang","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Weston et al., 2011] Jason Weston, Samy Bengio, and Nicolas Usunier. Wsabie: Scaling up to large vocabulary image annotation. In IJCAI'11, 2011.","title":"Wsabie: Scaling up to large vocabulary image annotation","authors":"Jason Weston, Samy Bengio, Nicolas Usunier","year":"2011","venue":"IJCAI'11","doi":null,"url":null,"internal_url":"/proceedings/2011/460"},{"position":25,"raw":"[Xiao and Guo, 2015] Min Xiao and Yuhong Guo. Semi- supervised subspace co-projection for multi-class hetero- geneous domain adaptation. In ECMLPKDD. 2015.","title":"Semisupervised subspace co-projection for multi-class heterogeneous domain adaptation","authors":"Xiao, Guo ; ] Min, Xiao, Yuhong Guo","year":"2015","venue":"ECMLPKDD","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044, 2(3):5, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S Zemel, Yoshua Bengio","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang and Saligrama, 2015] Ziming Zhang and Venkatesh Saligrama. Zero-shot learning via semantic similarity em- bedding. In ICCV, 2015.","title":"Zero-shot learning via semantic similarity embedding","authors":"Ziming Zhang, Venkatesh Saligrama","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou et al., 2014] Joey Tianyi Zhou, Sinno Jialin Pan, Ivor W. Tsang, and Yan Yan. Hybrid heterogeneous trans- fer learning through deep learning. AAAI, 2014.","title":"Hybrid heterogeneous transfer learning through deep learning","authors":"Joey Tianyi Zhou, Sinno Jialin Pan, Ivor W Tsang, Yan Yan","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2016] Joey Zhou, Sinno Pan, Ivor Tsang, and Shen-Shyang Ho. Transfer learning for cross-language text categorization through active correspondences con- struction. AAAI, 2016.","title":"Transfer learning for cross-language text categorization through active correspondences construction","authors":"Joey Zhou, Sinno Pan, Ivor Tsang, Shen-Shyang Ho","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}