{"type":"ScholarlyArticle","title":"Semi-Supervised Multi-Modal Learning with Incomplete Modalities","authors":[{"name":"Yang Yang","slug":"yang-yang","given":"Yang","family":"Yang","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"De-Chuan Zhan","slug":"de-chuan-zhan","given":"De-Chuan","family":"Zhan","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"Xiang-Rong Sheng","slug":"xiang-rong-sheng","given":"Xiang-Rong","family":"Sheng","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"Yuan Jiang","slug":"yuan-jiang","given":"Yuan","family":"Jiang","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"}],"abstract":"In real world applications, data are often with multiple modalities. Researchers proposed the multi-modal learning approaches for integrating the information from different modalities. Most of the previous multi-modal methods assume that training examples are with complete modalities. However, due to the failures of data collection, self-deficiencies and other various reasons, multi-modal examples are usually with incomplete feature representation in real applications. In this paper, the incomplete feature representation issues in multi-modal learning are named as incomplete modalities, and we propose a semi-supervised multi-modal learning method aimed at this incomplete modal issue (SLIM). SLIM can utilize the extrinsic information from unlabeled data against the insufficiencies brought by the incomplete modal issues in a semi-supervised scenario. Besides, the proposed SLIM forms the problem into a unified framework which can be treated as a classifier or clustering learner, and integrate the intrinsic consistencies and extrinsic unlabeled information. As SLIM can extract the most discriminative predictors for each modality, experiments on 15 real world multi-modal datasets validate the effectiveness of our method.","doi":"10.24963/ijcai.2018/416","url":"https://ijcai.org/proceedings/2018/416","pdf":"https://ijcai.org/proceedings/2018/0416.pdf","markdown":"https://ijcai.org/proceedings/2018/416.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/416","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":416,"pages":{"first":2998,"last":3004},"keywords":["Machine Learning: Semi-Supervised Learning","Machine Learning: Multi-instance;Multi-label;Multi-view learning"],"videos":[],"references":[{"position":1,"raw":"[Arora et al., 2016] Raman Arora, Poorya Mianjy, and Teodor V. Marinov. Stochastic optimization for multiview representation learning using partial least squares. In Pro- ceedings of the 33nd International Conference on Machine Learning, pages 1786-1794, New York City, NY, 2016.","title":"Stochastic optimization for multiview representation learning using partial least squares","authors":"Raman Arora, Poorya Mianjy, Teodor V Marinov","year":"2016","venue":"Proceedings of the 33nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bisson and Grimal, 2012] Gilles Bisson and Clement Gri- mal. Co-clustering of multi-view datasets: A paralleliz- able approach. In Proceedings of the 12th IEEE Interna- tional Conference on Data Mining, pages 828-833, Brus- sels, Belgium, 2012.","title":"Co-clustering of multi-view datasets: A parallelizable approach","authors":"Grimal Bisson, Gilles Bisson, Clement Grimal","year":"2012","venue":"Proceedings of the 12th IEEE International Conference on Data Mining","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cai et al., 2013] Xiao Cai, Feiping Nie, Weidong Cai, and Heng Huang. Heterogeneous image features integration via multi-modal semi-supervised learning model. In Pro- ceedings of the IEEE International Conference on Com- puter Vision, pages 1737-1744, Sydney, Australia, 2013.","title":"Heterogeneous image features integration via multi-modal semi-supervised learning model","authors":"Xiao Cai, Feiping Nie, Weidong Cai, Heng Huang","year":"2013","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ding and He, 2005] Chris H. Q. Ding and Xiaofeng He. On the equivalence of nonnegative matrix factorization and spectral clustering. In Proceedings of the 2005 SIAM In- ternational Conference on Data Mining, pages 606-610, Newport Beach, CA, 2005.","title":"On the equivalence of nonnegative matrix factorization and spectral clustering","authors":"H Q Chris, Xiaofeng Ding, He","year":"2005","venue":"Proceedings of the 2005 SIAM International Conference on Data Mining","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Eaton et al., 2010] Eric Eaton, Marie desJardins, and Sara Jacob. Multi-view clustering with constraint propagation for learning with an incomplete mapping between views. In Proceedings of the 19th ACM Conference on Informa- tion and Knowledge Management, pages 389-398, On- tario, Canada, 2010.","title":"Multi-view clustering with constraint propagation for learning with an incomplete mapping between views","authors":"Eric Eaton, Marie Desjardins, Sara Jacob","year":"2010","venue":"Proceedings of the 19th ACM Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Guo, 2013] Yuhong Guo. Convex subspace representation learning from multi-view data. In Proceedings of the 27th AAAI Conference on Artificial Intelligence, pages 387- 393, Bellevue, Washington, 2013.","title":"Convex subspace representation learning from multi-view data","authors":"Yuhong Guo","year":"2013","venue":"Proceedings of the 27th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Iwata and Yamada, 2016] Tomoharu Iwata and Makoto Ya- mada. Multi-view anomaly detection via robust probabilis- tic latent variable models. In Advances in Neural Informa- tion Processing Systems 29, pages 1136-1144, Barcelona, Spain, 2016.","title":"Multi-view anomaly detection via robust probabilistic latent variable models","authors":"Yamada Iwata, Tomoharu Iwata, Makoto Yamada","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Karasuyama and Mamitsuka, 2013] Masayuki Karasuyama and Hiroshi Mamitsuka. Multiple graph label propagation by sparse integration. IEEE Trans. Neural Netw. Learning Syst., 24(12):1999-2012, 2013.","title":"Multiple graph label propagation by sparse integration","authors":"Masayuki Karasuyama, Hiroshi Mamitsuka","year":"2013","venue":"IEEE Trans. Neural Netw. Learning Syst","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2014] Shao-Yuan Li, Yuan Jiang, and Zhi-Hua Zhou. Partial multi-view clustering. In Proceedings of the 28th AAAI Conference on Artificial Intelligence, pages 1968-1974, Quebec, Canada, 2014.","title":"Partial multi-view clustering","authors":"Shao-Yuan, Yuan Li, Zhi-Hua Jiang, Zhou","year":"2014","venue":"Proceedings of the 28th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin et al., 2010] Zhouchen Lin, Minming Chen, and Yi Ma. The augmented lagrange multiplier method for ex- act recovery of corrupted low-rank matrices. CoRR, abs/1009.5055, 2010.","title":"The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices","authors":"Lin","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2014] Han Liu, Lie Wang, and Tuo Zhao. Multi- variate regression with calibration. In Advances in Neural Information Processing Systems 27, pages 127-135, Que- bec, Canada, 2014.","title":"Multivariate regression with calibration","authors":"Han Liu, Lie Wang, Tuo Zhao","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Sen et al., 2008] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Gallagher, and Tina Eliassi- Rad. Collective classification in network data. AI Mag- azine, 29(3):93-106, 2008.","title":"Collective classification in network data","authors":"Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Gallagher, Tina Eliassi-Rad","year":"2008","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Shao et al., 2015] Weixiang Shao, Lifang He, and Philip S. Yu. Multiple incomplete views clustering via weighted nonnegative matrix factorization with l 2,1 regularization. In Proceedings of the European Conference Machine Learning and Knowledge Discovery in Databases, pages 318-334, Porto, Portugal, 2015.","title":"Multiple incomplete views clustering via weighted nonnegative matrix factorization with l 2,1 regularization","authors":"Weixiang Shao, Lifang He, Philip S Yu","year":"2015","venue":"Proceedings of the European Conference Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Shao et al., 2016] Weixiang Shao, Lifang He, Chun-Ta Lu, and Philip S. Yu. Online multi-view clustering with in- complete views. In Proceedings of the IEEE International Conference on Big Data, pages 1012-1017, Washington DC, 2016.","title":"Online multi-view clustering with incomplete views","authors":"Weixiang Shao, Lifang He, Chun-Ta Lu, Philip S Yu","year":"2016","venue":"Proceedings of the IEEE International Conference on Big Data","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Trivedi et al., 2010] Anusua Trivedi, Piyush Rai, Hal Daume, and Scott L Duvall. Multiview clustering with incomplete views. 2010.","title":"Multiview clustering with incomplete views","authors":"Trivedi","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wang et al., 2013] Hua Wang, Feiping Nie, and Heng Huang. Multi-View Clustering and Feature Learning via Structured Sparsity. In Proceedings of the 30th Interna- tional Conference on Machine Learning, pages 352-360, Atlanta, GA, 2013.","title":"Multi-View Clustering and Feature Learning via Structured Sparsity","authors":"Hua Wang, Feiping Nie, Heng Huang","year":"2013","venue":"Proceedings of the 30th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Xu et al., 2015] Chang Xu, Dacheng Tao, and Chao Xu. Multi-view learning with incomplete views. IEEE Trans. Image Processing, 24(12):5812-5825, 2015.","title":"Multi-view learning with incomplete views","authors":"Chang Xu, Dacheng Tao, Chao Xu","year":"2015","venue":"IEEE Trans. Image Processing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Yang et al., 2015] Yang Yang, Han-Jia Ye, De-Chuan Zhan, and Yuan Jiang. Auxiliary information regularized ma- chine for multiple modality feature learning. In Proceed- ings of the 24th International Joint Conference on Arti- ficial Intelligence, pages 1033-1039, Buenos Aires, Ar- gentina, 2015.","title":"Auxiliary information regularized machine for multiple modality feature learning","authors":"Yang Yang, Han-Jia Ye, Yuan De-Chuan Zhan, Jiang","year":"2015","venue":"Proceedings of the 24th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2015/150"},{"position":19,"raw":"[Yang et al., 2016] Yang Yang, De-Chuan Zhan, and Yuan Jiang. Learning by actively querying strong modal fea- tures. In Proceedings of the 25th International Joint Con- ference on Artificial Intelligence, pages 2280-2286, New York, NY, 2016.","title":"Learning by actively querying strong modal features","authors":"Yang Yang, Yuan De-Chuan Zhan, Jiang","year":"2016","venue":"Proceedings of the 25th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2016/325"},{"position":20,"raw":"[Yang et al., 2017] Erkun Yang, Cheng Deng, Wei Liu, Xi- anglong Liu, Dacheng Tao, and Xinbo Gao. Pairwise re- lationship guided deep hashing for cross-modal retrieval. In Proceedings of the 31st AAAI Conference on Artificial Intelligence, pages 1618-1625, San Francisco, California, 2017.","title":"Pairwise relationship guided deep hashing for cross-modal retrieval","authors":"Erkun Yang, Cheng Deng, Wei Liu, Xianglong Liu, Dacheng Tao, Xinbo Gao","year":"2017","venue":"Proceedings of the 31st AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ye et al., 2015] Han-Jia Ye, De-Chuan Zhan, Yuan Miao, Yuan Jiang, and Zhi-Hua Zhou. Rank consistency based multi-view learning: A privacy-preserving approach. In Proceedings of the 24th ACM International Conference on Information and Knowledge Management, pages 991- 1000, Melbourne, Australia, 2015.","title":"Rank consistency based multi-view learning: A privacy-preserving approach","authors":"Han-Jia Ye, Yuan De-Chuan Zhan, Yuan Miao, Zhi-Hua Jiang, Zhou","year":"2015","venue":"Proceedings of the 24th ACM International Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yin et al., 2017] Qiyue Yin, Shu Wu, and Liang Wang. Uni- fied subspace learning for incomplete and unlabeled multi- view data. Pattern Recognition, 67:313-327, 2017.","title":"Unified subspace learning for incomplete and unlabeled multiview data","authors":"Qiyue Yin, Shu Wu, Liang Wang","year":"2017","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhao et al., 2016] Handong Zhao, Hongfu Liu, and Yun Fu. Incomplete multi-modal visual data grouping. In Proceed- ings of the 25th International Joint Conference on Artifi- cial Intelligence, pages 2392-2398, New York, NY, 2016.","title":"Incomplete multi-modal visual data grouping","authors":"Handong Zhao, Hongfu Liu, Yun Fu","year":"2016","venue":"Proceedings of the 25th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2016/341"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}