{"type":"ScholarlyArticle","title":"Cross-modal Common Representation Learning by Hybrid Transfer Network","authors":[{"name":"Xin Huang","slug":"xin-huang","given":"Xin","family":"Huang","affiliation":"Institute of Computer Science and Technology , Peking University , Beijing 100871 , China"},{"name":"Yuxin Peng","slug":"yuxin-peng","given":"Yuxin","family":"Peng","affiliation":"Institute of Computer Science and Technology , Peking University , Beijing 100871 , China"},{"name":"Mingkuan Yuan","slug":"mingkuan-yuan","given":"Mingkuan","family":"Yuan","affiliation":"Institute of Computer Science and Technology , Peking University , Beijing 100871 , China"}],"abstract":"DNN-based cross-modal retrieval is a research hotspot to retrieve across different modalities as image and text, but existing methods often face the challenge of insufficient cross-modal training data. In single-modal scenario, similar problem is usually relieved by transferring knowledge from large-scale auxiliary datasets (as ImageNet). Knowledge from such single-modal datasets is also very useful for cross-modal retrieval, which can provide rich general semantic information that can be shared across different modalities. However, it is challenging to transfer useful knowledge from single-modal (as image) source domain to cross-modal (as image/text) target domain. Knowledge in source domain cannot be directly transferred to both two different modalities in target domain, and the inherent cross-modal correlation contained in target domain provides key hints for cross-modal retrieval which should be preserved during transfer process. This paper proposes Cross-modal Hybrid Transfer Network (CHTN) with two subnetworks: Modal-sharing transfer subnetwork utilizes the modality in both source and target domains as a bridge, for transferring knowledge to both two modalities simultaneously; Layer-sharing correlation subnetwork preserves the inherent cross-modal semantic correlation to further adapt to cross-modal retrieval task. Cross-modal data can be converted to common representation by CHTN for retrieval, and comprehensive experiment on 3 datasets shows its effectiveness.","doi":"10.24963/ijcai.2017/263","url":"https://ijcai.org/proceedings/2017/263","pdf":"https://ijcai.org/proceedings/2017/0263.pdf","markdown":"https://ijcai.org/proceedings/2017/263.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/263","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":263,"pages":{"first":1893,"last":1900},"keywords":["Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Cao et al., 2016] Zhangjie Cao, Mingsheng Long, and Qiang Yang. Transitive hashing network for heteroge- neous multimedia retrieval. arXiv, abs/1608.04307, 2016.","title":"Transitive hashing network for heterogeneous multimedia retrieval","authors":"Cao","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen and Zhang, 2013] Zheng Chen and Weixiong Zhang. Domain adaptation with topical correspondence learning. IJCAI, pages 1280-1286, 2013.","title":"Domain adaptation with topical correspondence learning","authors":"Chen, Zhang, Zheng Chen, Weixiong Zhang","year":"2013","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2013/193"},{"position":3,"raw":"[Chua et al., 2009] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. Nus- wide: a real-world web image database from national uni- versity of singapore. ACM-CIVR, 2009.","title":"Nuswide: a real-world web image database from national university of singapore","authors":"Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, Yantao Zheng","year":"2009","venue":"ACM-CIVR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. CVPR, pages 248-255, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Fei-Fei Li","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Feng et al., 2014] Fangxiang Feng, Xiaojie Wang, and Ruifan Li. Cross-modal retrieval with correspondence au- toencoder. ACM-MM, pages 7-16, 2014.","title":"Cross-modal retrieval with correspondence autoencoder","authors":"Feng","year":"2014","venue":"ACM-MM","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Glorot et al., 2011] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Domain adaptation for large-scale sen- timent classification: A deep learning approach. ICML, pages 513-520, 2011.","title":"Domain adaptation for large-scale sentiment classification: A deep learning approach","authors":"Xavier Glorot, Antoine Bordes, Yoshua Bengio","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gretton et al., 2012] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13:723-773, 2012.","title":"A kernel two-sample test","authors":"Arthur Gretton, Karsten M Borgwardt, J Malte, Bernhard Rasch, Alexander Schölkopf, Smola","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gupta et al., 2016] Saurabh Gupta, Judy Hoffman, and Ji- tendra Malik. Cross modal distillation for supervision transfer. CVPR, pages 2827-2836, 2016.","title":"Cross modal distillation for supervision transfer","authors":"Saurabh Gupta, Judy Hoffman, Jitendra Malik","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hardoon et al., 2004] David R. Hardoon, Sándor Szedmák, and John Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Neural Computation, 16(12):2639-2664, 2004.","title":"Canonical correlation analysis: An overview with application to learning methods","authors":"David R Hardoon, Sándor Szedmák, John Shawe-Taylor","year":"2004","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hotelling, 1936] Harold Hotelling. Relations between two sets of variates. Biometrika, pages 321-377, 1936.","title":"Harold Hotelling. Relations between two sets of variates","authors":"Hotelling","year":"1936","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kang et al., 2015] Cuicui Kang, Shiming Xiang, Shengcai Liao, Changsheng Xu, and Chunhong Pan. Learning consistent feature representation for cross-modal multime- dia retrieval. IEEE Transactions on Multimedia (TMM), 17(3):370-381, 2015.","title":"Learning consistent feature representation for cross-modal multimedia retrieval","authors":"Kang","year":"2015","venue":"IEEE Transactions on Multimedia (TMM)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim et al., 2012] Jungi Kim, Jinseok Nam, and Iryna Gurevych. Learning semantics with deep belief network for cross-language information retrieval. International Committee on Computational Linguistic (ICCL), pages 579-588, 2012.","title":"Learning semantics with deep belief network for cross-language information retrieval","authors":"Kim","year":"2012","venue":"International Committee on Computational Linguistic (ICCL)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. NIPS, pages 1106-1114, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2003] Dongge Li, Nevenka Dimitrova, Mingkun Li, and Ishwar K Sethi. Multimedia content processing through cross-modal association. ACM-MM, pages 604- 611, 2003.","title":"Multimedia content processing through cross-modal association","authors":"Li","year":"2003","venue":"ACM-MM","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Long et al., 2015] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable fea- tures with deep adaptation networks. ICML, pages 97-105, 2015.","title":"Learning transferable features with deep adaptation networks","authors":"Mingsheng Long, Yue Cao, Jianmin Wang, Michael I Jordan","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ngiam et al., 2011] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Mul- timodal deep learning. ICML, pages 689-696, 2011.","title":"Multimodal deep learning","authors":"Ngiam","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pan and Yang, 2010] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowl- edge and Data Engineering (TKDE), 22(10):1345-1359, 2010.","title":"A survey on transfer learning","authors":"Jialin Sinno, Qiang Pan, Yang","year":"2010","venue":"IEEE Transactions on Knowledge and Data Engineering (TKDE)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Peng et al., 2016] Yuxin Peng, Xin Huang, and Jinwei Qi. Cross-media shared representation by hierarchical learn- ing with multiple deep networks. IJCAI, pages 3846-3853, 2016.","title":"Cross-media shared representation by hierarchical learning with multiple deep networks","authors":"Yuxin Peng, Xin Huang, Jinwei Qi","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/541"},{"position":19,"raw":"[Peng, 2015] Yuxin Peng. Adaptive sampling with optimal cost for class-imbalance learning. AAAI, pages 2921- 2927, 2015.","title":"Adaptive sampling with optimal cost for class-imbalance learning","authors":"Yuxin Peng, Peng","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rashtchian et al., 2010] C. Rashtchian, P. Young, M. Ho- dosh, and J. Hockenmaier. Collecting image annotations using amazon's mechanical turk. North American Chap- ter of the Association for Computational Linguistics, pages 139-147, 2010.","title":"Collecting image annotations using amazon's mechanical turk","authors":"C Rashtchian, P Young, M Hodosh, J Hockenmaier","year":"2010","venue":"North American Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rasiwasia et al., 2010] Nikhil Rasiwasia, Jose Costa Pereira, Emanuele Coviello, Gabriel Doyle, Gert RG Lanckriet, Roger Levy, and Nuno Vasconcelos. A new approach to cross-modal multimedia retrieval. ACM-MM, pages 251-260, 2010.","title":"A new approach to cross-modal multimedia retrieval","authors":"Nikhil Rasiwasia, Jose Costa Pereira, Emanuele Coviello, Gabriel Doyle, Gert Rg Lanckriet, Roger Levy, Nuno Vasconcelos","year":"2010","venue":"ACM-MM","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Samdani and Yih, 2011] Rajhans Samdani and Wen-tau Yih. Domain adaptation with ensemble of feature groups. IJCAI, pages 1458-1464, 2011.","title":"Domain adaptation with ensemble of feature groups","authors":"Rajhans Samdani, Wen-Tau Yih","year":"2011","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2011/246"},{"position":23,"raw":"[Srivastava and Salakhutdinov, 2012] Nitish Srivastava and Ruslan Salakhutdinov. Learning representations for multi- modal data with deep belief nets. ICML Workshop, 2012.","title":"Learning representations for multimodal data with deep belief nets","authors":"Nitish Srivastava, Ruslan Salakhutdinov","year":"2012","venue":"ICML Workshop","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tang et al., 2015] Jinhui Tang, Zechao Li, Meng Wang, and Ruizhen Zhao. Neighborhood discriminant hashing for large-scale image retrieval. IEEE Transactions on Image Processing (TIP), 24(9):2827-2840, 2015.","title":"Neighborhood discriminant hashing for large-scale image retrieval","authors":"Jinhui Tang, Zechao Li, Meng Wang, Ruizhen Zhao","year":"2015","venue":"IEEE Transactions on Image Processing (TIP)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tsai et al., 2016] Yao-Hung Hubert Tsai, Yi-Ren Yeh, and Yu-Chiang Frank Wang. Learning cross-domain land- marks for heterogeneous domain adaptation. CVPR, pages 5081-5090, 2016.","title":"Learning cross-domain landmarks for heterogeneous domain adaptation","authors":"Yao-Hung Hubert Tsai, Yi-Ren Yeh, Yu-Chiang Frank, Wang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2015] Daixin Wang, Peng Cui, Mingdong Ou, and Wenwu Zhu. Deep multimodal hashing with orthogo- nal regularization. IJCAI, pages 2291-2297, 2015.","title":"Deep multimodal hashing with orthogonal regularization","authors":"Wang","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/324"},{"position":27,"raw":"[Wei et al., 2017] Yunchao Wei, Yao Zhao, Canyi Lu, Shikui Wei, Luoqi Liu, Zhenfeng Zhu, and Shuicheng Yan. Cross-modal retrieval with CNN visual features: A new baseline. IEEE Transactions on Cybernetics (TCYB), 47(2):449-460, 2017.","title":"Cross-modal retrieval with CNN visual features: A new baseline","authors":"Yunchao Wei, Yao Zhao, Canyi Lu, Shikui Wei, Luoqi Liu, Zhenfeng Zhu, Shuicheng Yan","year":"2017","venue":"IEEE Transactions on Cybernetics (TCYB)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yang et al., 2015] Xiaoshan Yang, Tianzhu Zhang, and Changsheng Xu. Cross-domain feature learning in mul- timedia. IEEE Transactions on Multimedia (TMM), 17(1):64-78, 2015.","title":"Cross-domain feature learning in multimedia","authors":"Yang","year":"2015","venue":"IEEE Transactions on Multimedia (TMM)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yosinski et al., 2014] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? NIPS, pages 3320-3328, 2014.","title":"How transferable are features in deep neural networks? NIPS","authors":"Jason Yosinski, Jeff Clune, Yoshua Bengio, Hod Lipson","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhai et al., 2014] Xiaohua Zhai, Yuxin Peng, and Jianguo Xiao. Learning cross-media joint representation with sparse and semi-supervised regularization. IEEE Trans- actions on Circuits and Systems for Video Technology (TCSVT), 24(6):965-978, 2014.","title":"Learning cross-media joint representation with sparse and semi-supervised regularization","authors":"Xiaohua Zhai, Yuxin Peng, Jianguo Xiao","year":"2014","venue":"IEEE Transactions on Circuits and Systems for Video Technology (TCSVT)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}