{"type":"ScholarlyArticle","title":"Comprehensive Semi-Supervised Multi-Modal Learning","authors":[{"name":"Yang Yang","slug":"yang-yang","given":"Yang","family":"Yang","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University"},{"name":"Ke-Tao Wang","slug":"ke-tao-wang","given":"Ke-Tao","family":"Wang","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University"},{"name":"De-Chuan Zhan","slug":"de-chuan-zhan","given":"De-Chuan","family":"Zhan","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University"},{"name":"Hui Xiong","slug":"hui-xiong","given":"Hui","family":"Xiong","affiliation":"Rutgers University"},{"name":"Yuan Jiang","slug":"yuan-jiang","given":"Yuan","family":"Jiang","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University"}],"abstract":"Multi-modal learning refers to the process of learning a precise model to represent the joint representations of different modalities. Despite its promise for multi-modal learning, the co-regularization method is based on the consistency principle with a sufficient assumption, which usually does not hold for real-world multi-modal data. Indeed, due to the modal insufficiency in real-world applications, there are divergences among heterogeneous modalities. This imposes a critical challenge for multi-modal learning. To this end, in this paper, we propose a novel Comprehensive Multi-Modal Learning (CMML) framework, which can strike a balance between the consistency and divergency modalities by considering the insufficiency in one unified framework. Specifically, we utilize an instance level attention mechanism to weight the sufficiency for each instance on different modalities. Moreover, novel diversity regularization and robust consistency metrics are designed for discovering insufficient modalities. Our empirical studies show the superior performances of CMML on real-world data in terms of various criteria.","doi":"10.24963/ijcai.2019/568","url":"https://ijcai.org/proceedings/2019/568","pdf":"https://ijcai.org/proceedings/2019/0568.pdf","markdown":"https://ijcai.org/proceedings/2019/568.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/568","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":568,"pages":{"first":4092,"last":4098},"keywords":["Machine Learning: Multi-instance;Multi-label;Multi-view learning","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Balcan et al., 2004] Maria-Florina Balcan, Avrim Blum, and Ke Yang. Co-training and expansion: Towards bridg- ing theory and practice. In NIPS, pages 89-96, British Columbia, Canada, 2004.","title":"Co-training and expansion: Towards bridging theory and practice","authors":"Maria-Florina Balcan, Avrim Blum, Ke Yang","year":"2004","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blum and Mitchell, 1998] Avrim Blum and Tom M. Mitchell. Combining labeled and unlabeled data with co-training. In COLT, pages 92-100, Madison, Wisconsin, 1998.","title":"Combining labeled and unlabeled data with co-training","authors":"Avrim Blum, Tom M Mitchell","year":"1998","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brefeld et al., 2006] Ulf Brefeld, Thomas Gartner, Tobias Scheffer, and Stefan Wrobel. Efficient co-regularised least squares regression. In ICML, pages 137-144, Pittsburgh, Pennsylvania, 2006.","title":"Efficient co-regularised least squares regression","authors":"Brefeld","year":"2006","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chua et al., 2009] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. NUS- WIDE: a real-world web image database from national u- niversity of singapore. In CIVR, Santorini Island, Greece, 2009.","title":"NUS-WIDE: a real-world web image database from national university of singapore","authors":"Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, Yantao Zheng","year":"2009","venue":"CIVR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dasgupta et al., 2001] Sanjoy Dasgupta, Michael L. Littman, and David A. McAllester. PAC generalization bounds for co-training. In NIPS, pages 375-382, British Columbia, Canada, 2001.","title":"PAC generalization bounds for co-training","authors":"Sanjoy Dasgupta, Michael L Littman, David A Mcallester","year":"2001","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Escalante et al., 2010] Hugo Jair Escalante, Carlos A. Her- nandez, Jesus A. Gonzalez, Aurelio Lopez-Lopez, Manuel Montes-y-Gomez, Eduardo F. Morales, Luis Enrique Su- car, Luis Villasenor Pineda, and Michael Grubinger. The segmented and annotated IAPR TC-12 benchmark. CVIU, 114(4):419-428, 2010.","title":"The segmented and annotated IAPR TC-12 benchmark","authors":"Escalante","year":"2010","venue":"CVIU","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Feng and Zhou, 2017] Ji Feng and Zhi-Hua Zhou. Deep MIML network. In AAAI, pages 1884-1890, San Fran- cisco, California, 2017.","title":"Deep MIML network","authors":"Ji Feng, Zhi-Hua Zhou","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. arXiv preprint arXiv:1512.03385, 2015.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huber and others, 1964] Peter J Huber et al. Robust estima- tion of a location parameter. AMS, 35(1):73-101, 1964.","title":"Robust estimation of a location parameter","authors":"J Peter, Huber","year":"1964","venue":"AMS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huiskes and Lew, 2008] Mark J. Huiskes and Michael S. Lew. The MIR flickr retrieval evaluation. In SIGMM, pages 39-43, British Columbia, Canada, 2008.","title":"The MIR flickr retrieval evaluation","authors":"Mark J Huiskes, Michael S Lew","year":"2008","venue":"SIGMM","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Ge- offrey E. Hinton. Deep learning. Nature, 521(7553):436- 444, 2015.","title":"Deep learning","authors":"Yann Lecun, Yoshua Bengio, Geoffrey E Hinton","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2012] Nan Li, Yang Yu, and Zhi-Hua Zhou. Di- versity regularized ensemble pruning. In ECML/PKDD, pages 330-345, Bristol, UK, 2012.","title":"Diversity regularized ensemble pruning","authors":"Nan Li, Yang Yu, Zhi-Hua Zhou","year":"2012","venue":"ECML/PKDD","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge J. Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C. Lawrence Zitnick. Microsoft COCO: com- mon objects in context. In ECCV, pages 740-755, Zurich, Switzerland, 2014.","title":"Microsoft COCO: common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge J Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, C Lawrence Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sridharan and Kakade, 2008] Karthik Sridharan and Sham M. Kakade. An information theoretic framework for multi-view learning. In COLT, pages 403-414, Helsinki, Finland, 2008.","title":"An information theoretic framework for multi-view learning","authors":"Karthik Sridharan, M Sham, Kakade","year":"2008","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Tao et al., 2018] Hong Tao, Chenping Hou, Xinwang Liu, Tongliang Liu, Dongyun Yi, and Jubo Zhu. Reliable multi- view clustering. In AAAI, pages 4123-4130, New Orleans, Louisiana, 2018.","title":"Reliable multiview clustering","authors":"Tao","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wang and Zhou, 2013] Wei Wang and Zhi-Hua Zhou. Co- training with insufficient views. In ACML, pages 467-482, Canberra, Australia, 2013.","title":"Cotraining with insufficient views","authors":"Wei Wang, Zhi-Hua Zhou","year":"2013","venue":"ACML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2013] Hua Wang, Feiping Nie, and Heng Huang. Multi-View Clustering and Feature Learning vi- a Structured Sparsity. In ICML, pages 352-360, Atlanta, GA, 2013.","title":"Multi-View Clustering and Feature Learning via Structured Sparsity","authors":"Hua Wang, Feiping Nie, Heng Huang","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Xu et al., 2013] Chang Xu, Dacheng Tao, and Chao Xu. A survey on multi-view learning. CoRR, abs/1304.5634, 2013.","title":"A survey on multi-view learning","authors":"Chang Xu, Dacheng Tao, Chao Xu","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yang et al., 2015] Yang Yang, Han-Jia Ye, De-Chuan Zhan, and Yuan Jiang. Auxiliary information regularized ma- chine for multiple modality feature learning. In IJCAI, pages 1033-1039, Buenos Aires, Argentina, 2015.","title":"Auxiliary information regularized machine for multiple modality feature learning","authors":"Yang Yang, Han-Jia Ye, Yuan De-Chuan Zhan, Jiang","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/150"},{"position":20,"raw":"[Yang et al., 2018a] Yang Yang, Yi-Feng Wu, De-Chuan Zhan, Zhi-Bin Liu, and Yuan Jiang. Complex object clas- sification: A multi-modal multi-instance multi-label deep network with optimal transport. In KDD, pages 2594- 2603, London, UK, 2018.","title":"Complex object classification: A multi-modal multi-instance multi-label deep network with optimal transport","authors":"Yang","year":"2018","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2018b] Yang Yang, De-Chuan Zhan, Xiang- Rong Sheng, and Yuan Jiang. Semi-supervised multi- modal learning with incomplete modalities. In IJCAI, pages 2998-3004, Stockholm, Sweden, 2018.","title":"Semi-supervised multimodal learning with incomplete modalities","authors":"Yang","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ye et al., 2016] Han-Jia Ye, De-Chuan Zhan, Xiaolin Li, Zhen-Chuan Huang, and Yuan Jiang. College student scholarships and subsidies granting: A multi-modal multi- label approach. In ICDM, pages 559-568, Barcelona, S- pain, 2016.","title":"College student scholarships and subsidies granting: A multi-modal multilabel approach","authors":"Han-Jia Ye, Xiaolin De-Chuan Zhan, Zhen-Chuan Li, Yuan Huang, Jiang","year":"2016","venue":"ICDM","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yu et al., 2011] Yang Yu, Yu-Feng Li, and Zhi-Hua Zhou. Diversity regularized machine. In IJCAI, pages 1603- 1608, Catalonia, Spain, 2011.","title":"Diversity regularized machine","authors":"Yang Yu, Yu-Feng Li, Zhi-Hua Zhou","year":"2011","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2011/269"},{"position":24,"raw":"[Zhang and Zhou, 2011] Min-Ling Zhang and Zhi-Hua Zhou. Cotrade: Confident co-training with data editing. TSMC, 41(6):1612-1626, 2011.","title":"Cotrade: Confident co-training with data editing","authors":"Zhou ; ] Min-Ling Zhang, Zhi-Hua Zhang, Zhou","year":"2011","venue":"TSMC","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}