{"type":"ScholarlyArticle","title":"Theoretic Analysis and Extremely Easy Algorithms for Domain Adaptive Feature Learning","authors":[{"name":"Wenhao Jiang","slug":"wenhao-jiang","given":"Wenhao","family":"Jiang","affiliation":"Tencent AI Lab , P. R. China"},{"name":"Cheng Deng","slug":"cheng-deng","given":"Cheng","family":"Deng","affiliation":"Xidian University , P. R. China"},{"name":"Wei Liu","slug":"wei-liu","given":"Wei","family":"Liu","affiliation":"Tencent AI Lab , P. R. China"},{"name":"Feiping Nie","slug":"feiping-nie","given":"Feiping","family":"Nie","affiliation":"Northwestern Polytechnical University , P. R. China"},{"name":"Fu-lai Chung","slug":"fu-lai-chung","given":"Fu-lai","family":"Chung","affiliation":"Hong Kong Polytechnic University , Hong Kong"},{"name":"Heng Huang","slug":"heng-huang","given":"Heng","family":"Huang","affiliation":"University of Texas at Arlington , USA; Xidian University , P. R. China"}],"abstract":"Domain adaptation problems arise in a variety of applications, where a training dataset from the source domain and a test dataset from the target domain typically follow different distributions. The primary difficulty in designing effective learning models to solve such problems lies in how to bridge the gap between the source and target distributions. In this paper, we provide comprehensive analysis of feature learning algorithms used in conjunction with linear classifiers for domain adaptation. Our analysis shows that in order to achieve good adaptation performance, the second moments of the source domain distribution and target domain distribution should be similar. Based on our new analysis, a novel extremely easy feature learning algorithm for domain adaptation is proposed. Furthermore, our algorithm is extended by leveraging multiple layers, leading to another feature learning algorithm. We evaluate the effectiveness of the proposed algorithms in terms of domain adaptation tasks on Amazon review and spam datasets from the ECML/PKDD 2006 discovery challenge.","doi":"10.24963/ijcai.2017/272","url":"https://ijcai.org/proceedings/2017/272","pdf":"https://ijcai.org/proceedings/2017/0272.pdf","markdown":"https://ijcai.org/proceedings/2017/272.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/272","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":272,"pages":{"first":1958,"last":1964},"keywords":["Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Feature Selection/Construction"],"videos":[],"references":[{"position":1,"raw":"[Ajakan et al., 2014] Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc ¸ois Laviolette, and Mario Marchand. Domain-adversarial neural networks. arXiv preprint arXiv:1412.4446, 2014.","title":"Domain-adversarial neural networks","authors":"Hana Ajakan, Pascal Germain, Hugo Larochelle, Mario Franc ¸ois Laviolette, Marchand","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ben-david et al., 2006] S. Ben-david, J. Blitzer, K. Cram- mer, and F. Pereira. Analysis of representations for domain adaptation. In NIPS, pages 137-144. 2006.","title":"Analysis of representations for domain adaptation","authors":"S Ben-David, J Blitzer, K Crammer, F Pereira","year":"2006","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ben-David et al., 2010] Shai Ben-David, Tyler Lu, Teresa Luu, and Dávid Pál. Impossibility theorems for domain adaptation. In International Conference on Artificial Intel- ligence and Statistics, pages 129-136, 2010.","title":"Impossibility theorems for domain adaptation","authors":"Shai Ben-David, Tyler Lu, Teresa Luu, Dávid Pál","year":"2010","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bickel, 2008] S. Bickel. ECML-PKDD discovery challenge 2006 overview. In ECML-PKDD Discovery Challenge Workshop, pages 1-9, 2008.","title":"ECML-PKDD discovery challenge 2006 overview","authors":"S Bickel","year":"2008","venue":"ECML-PKDD Discovery Challenge Workshop","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Blitzer et al., 2006] J. Blitzer, R. McDonald, and F. Pereira. Domain adaptation with structural correspondence learn- ing. In EMNLP, pages 120-128, 2006.","title":"Domain adaptation with structural correspondence learning","authors":"J Blitzer, R Mcdonald, F Pereira","year":"2006","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Blitzer et al., 2007] J. Blitzer, M. Dredze, and F. Pereira. Biographies, Bollywood, Boom-boxes and Blenders: Do- main Adaptation for Sentiment Classification. In ACL, pages 440-447, June 2007.","title":"Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification","authors":"J Blitzer, M Dredze, F Pereira","year":"2007","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Blitzer et al., 2008] John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman. Learn- ing bounds for domain adaptation. In Advances in Neural Information Processing Systems 20, pages 129-136. 2008.","title":"Learning bounds for domain adaptation","authors":"John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, Jennifer Wortman","year":"2008","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Blitzer et al., 2011] J. Blitzer, D. Foster, and S. Kakade. Do- main adaptation with coupled subspaces. In International Conference on Artificial Intelligence and Statistics, pages 173-181, 2011.","title":"Domain adaptation with coupled subspaces","authors":"J Blitzer, D Foster, S Kakade","year":"2011","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2011a] M. Chen, Y. Chen, and K. Q. Wein- berger. Automatic feature decomposition for single view co-training. In ICML, pages 953-960, 2011.","title":"Automatic feature decomposition for single view co-training","authors":"Chen","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen et al., 2011b] M. Chen, K. Q. Weinberger, and J. Blitzer. Co-Training for domain adaptation. In NIPS 24, pages 2456-2464. 2011.","title":"Co-Training for domain adaptation","authors":"Chen","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Chen et al., 2012] M. Chen, Z. Xu, F. Sha, and K. Q. Wein- berger. Marginalized denoising autoencoders for domain adaptation. In ICML, pages 767-774, 2012.","title":"Marginalized denoising autoencoders for domain adaptation","authors":"M Chen, Z Xu, F Sha, K Q Weinberger","year":"2012","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Duan et al., 2012] Lixin Duan, Ivor W Tsang, and Dong Xu. Domain transfer multiple kernel learning. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 34(3):465-479, 2012.","title":"Domain transfer multiple kernel learning","authors":"Lixin Duan, Ivor W Tsang, Dong Xu","year":"2012","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ganin and Lempitsky, 2015] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backprop- agation. In ICML-15, pages 1180-1189, 2015.","title":"Unsupervised domain adaptation by backpropagation","authors":"Yaroslav Ganin, Victor Lempitsky","year":"2015","venue":"ICML-15","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Germain et al., 2013] Pascal Germain, Amaury Habrard, Emilie Morvant, et al. A PAC-bayesian approach for do- main adaptation with specialization to linear classifiers. In ICML, pages 738-746, 2013.","title":"A PAC-bayesian approach for domain adaptation with specialization to linear classifiers","authors":"Pascal Germain, Amaury Habrard, Emilie Morvant","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Glorot et al., 2011] X. Glorot, A. Bordes, and Y. Bengio. Domain adaptation for large-scale sentiment classification: A deep learning approach. In ICML, pages 513-520, 2011.","title":"Domain adaptation for large-scale sentiment classification: A deep learning approach","authors":"X Glorot, A Bordes, Y Bengio","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Gong et al., 2012] B. Gong, Y. Shi, F. Sha, and K. Grauman. Geodesic flow kernel for unsupervised domain adaptation. In CVPR, pages 2066-2073. IEEE, 2012.","title":"Geodesic flow kernel for unsupervised domain adaptation","authors":"B Gong, Y Shi, F Sha, K Grauman","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.","title":"Deep Learning","authors":"Ian Goodfellow, Yoshua Bengio, Aaron Courville","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Heckman, 1979] James J Heckman. Sample selection bias as a specification error. Econometrica: Journal of the econometric society, pages 153-161, 1979.","title":"Sample selection bias as a specification error","authors":"J James, Heckman","year":"1979","venue":"Econometrica: Journal of the econometric society","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Horn and Johnson, 1991] R.A. Horn and C.R. Johnson. Topics in Matrix Analysis. Cambridge University Press, 1991.","title":"Topics in Matrix Analysis","authors":"R A Horn, C R Johnson","year":"1991","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Horn and Johnson, 2012] Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.","title":"Matrix analysis","authors":"A Roger, Charles R Horn, Johnson","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Huang et al., 2007] J. Huang, A. J. Smola, A. Gretton, K. M. Borgwardt, and B. Schölkopf. Correcting sample selection bias by unlabeled data. In NIPS 19, pages 601- 608. MIT Press, Cambridge, MA, 2007.","title":"Correcting sample selection bias by unlabeled data","authors":"J Huang, A J Smola, A Gretton, K M Borgwardt, B Schölkopf","year":"2007","venue":"NIPS 19","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2012] Wei Liu, Jun Wang, and Shih-Fu Chang. Robust and scalable graph-based semisupervised learning. Proceedings of the IEEE, 100(9):2624-2638, 2012.","title":"Robust and scalable graph-based semisupervised learning","authors":"Wei Liu, Jun Wang, Shih-Fu Chang","year":"2012","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mansour et al., 2009] Y. Mansour, M. Mohri, and A. Ros- tamizadeh. Domain adaptation with multiple sources. In NIPS 21, pages 1041-1048. 2009.","title":"Domain adaptation with multiple sources","authors":"Y Mansour, M Mohri, A Rostamizadeh","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shimodaira, 2000] Hidetoshi Shimodaira. Improving pre- dictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.","title":"Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function","authors":"Shimodaira","year":"2000","venue":"Journal of statistical planning and inference","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sun et al., 2015] Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. arXiv preprint arXiv:1511.05547, 2015.","title":"Return of frustratingly easy domain adaptation","authors":"Sun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2011] Hua Wang, Feiping Nie, Heng Huang, and Chris Ding. Dyadic transfer learning for cross-domain image classification. In 2011 IEEE International Confer- ence on Computer Vision (ICCV), pages 551-556. IEEE, 2011.","title":"Dyadic transfer learning for cross-domain image classification","authors":"Hua Wang, Feiping Nie, Heng Huang, Chris Ding","year":"2011","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zadrozny, 2004] Bianca Zadrozny. Learning and evaluating classifiers under sample selection bias. In ICML, pages 114-122, 2004.","title":"Learning and evaluating classifiers under sample selection bias","authors":"Bianca Zadrozny","year":"2004","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang and Oles, 2001] Tong Zhang and Frank J Oles. Text categorization based on regularized linear classification methods. Information retrieval, 4(1):5-31, 2001.","title":"Text categorization based on regularized linear classification methods","authors":"Tong Zhang, Frank J Oles","year":"2001","venue":"Information retrieval","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2013] Chao Zhang, Lei Zhang, and Jieping Ye. Generalization bounds for domain adaptation. CoRR, abs/1304.1574, 2013.","title":"Generalization bounds for domain adaptation","authors":"Chao Zhang, Lei Zhang, Jieping Ye","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhong et al., 2010] Erheng Zhong, Wei Fan, Qiang Yang, Olivier Verscheure, and Jiangtao Ren. Cross validation framework to choose amongst models and datasets for transfer learning. In ECML PKDD, pages 547-562, 2010.","title":"Cross validation framework to choose amongst models and datasets for transfer learning","authors":"Erheng Zhong, Wei Fan, Qiang Yang, Olivier Verscheure, Jiangtao Ren","year":"2010","venue":"ECML PKDD","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}