Semi-Supervised Multi-Modal Learning with Incomplete Modalities

1 Introduction

Multi-modal learning becomes attractive as the development of data collection technicals, and can be widely used in applications with relatively independent data sources, e.g., biological data with gene expression, array-comparative genomic hybridization, single-nucleotide polymorphism and methylation. Multi-modal learning approaches can utilize information from multiple modalities, among which information from each single modality can complement each other to improve the generalization ability of the whole learner, e.g., clustering [Arora et al., 2016; Iwata and Yamada, 2016], classification [Yang et al., 2015; 2016], regression [Yang et al., 2017]. It is notable that the mainstreammulti-modal learning approaches assume that training examples are with complete modalities.

Nevertheless, the assumption mentioned above is excessive, since there are many reasons for insufficiencies or incompleteness, including data collection failures from the damage of data sensors, data corruption by network communication, data privacy policies, etc., e.g., in web pages classification with document/image representations, documents and images are two modalities, yet some web pages only have document or image information; for user identification in cross-network, the user profile features, content information or linkage information can be regarded as multiple modalities, yet some users only have one or partial modalities due to personal preference or privacy issues. Existing multi-modal learning approaches cannot directly be applied on the incomplete modal situation. With some straight forward strategies, e.g., removing the examples only with partial modal features, filling in the incomplete modal features with missing data techniques, current multi-modal learning approaches can be executed, yet the model trained will clearly loses information and introduces extra noises.

Aiming at the incomplete modal issues, there are some preliminary investigations. Trivedi et al. [2010] proposed a partial modal approach, which uses one modal kernel matrix as the similarity matrix and completes the missing modal kernel using Laplacian regularization; Shao et al. [2016] proposed an online multi-modal clustering algorithm OMVC to learn the latent feature matrices for each individual incomplete modality and pushes them towards a common consensus; Zhao et al. [2016] proposed an unsupervised method which well handles the incomplete multi-modal data by transforming the original and incomplete data to a new and complete representation in a latent space. These methods mainly focus on making full use of the inherent information, i.e., the consistencies between multiple modalities. In this paper, we consider the defects of insufficient information caused by the incompleteness among modalities should be remedied by supplementing extrinsic information.

Transductive multi-modal learning methods are proposed for utilizing extrinsic information from test sets, e.g., Karasuyama and Mamitsuka [2013] proposed a new method SMGI, integrating multiple graphs for label propagation, which appeals to the sparsity of graph weights and can easily eliminate irrelevant graphs; Cai et al. [2013] proposed a novel approach to integrate heterogeneous features by performing multi-modal semi-supervised classification on unlabeled as well as unsegmented instances, which learns a common shared class indicator matrix and weights for different modalities. However, these transductive methods are difficult to extend to classification under the incomplete modal setting with unseen test data.

Different to above solutions, we propose a novel Semisupervised multi-modal Learning approach with the Incomplete Modal data (SLIM). SLIM utilizes the intrinsic modal consistencies and extrinsic unlabeled information in one unified framework as well as can perform in both transductive and inductive configurations. Consequently, SLIM is with wider applicable range and can be applied in both classification and clustering tasks. Besides, SLIM can also learn the most discriminative classifiers for each modality separately. Meanwhile, noting that real world datasets are always with noise and outliers resulting in unreliable solution, the squareroot loss is used for robust weight learning for each modality. Finally, more discriminative classifiers and robust clustering performance can be achieved in SLIM. We empirically investigate the effectiveness of SLIM, and it achieves significantly better performance on various tasks.

In the following of this paper, we start with a brief review of related works. Then the proposed SLIM approach and the experimental results. After that, we conclude the paper.

2 RelatedWork

The exploitation of multiple modal learning has attracted much attention recently. In this paper, our method integrates the intrinsic consistencies and extrinsic unlabeled information in a semi-supervised scenario with incomplete multiple modal data, which can be treated as a classifier or clustering learner. Therefore, we consider our work related to multimodal learning and semi-supervised learning.

Most of the previous multi-modal methods assume that training examples are with complete modalities. However, multi-modal examples are usually with incomplete feature representation in real applications. Therefore, many researchers have devoted to handling the incomplete modal data recently. Li et al. [2014] established a latent subspace where the instances corresponding to the same example in different modalities are close to each other, and similar instances in the same modality should be well grouped; Shao et al. [2015] proposed the MIC (Multi-Incomplete-view Clustering), an algorithm based on weighted nonnegative matrix factorization with L2,1 regularization, which learns the latent feature matrices for all the modalities and generating a consensus matrix so that minimize the difference between each modality and the consensus matrix; Xu et al. [2015] proposed an effective algorithm to accomplish multi-modal learning with incomplete modalities by assuming that different modalities are generated from a shared subspace, which exploits the connections between multiple modalities, enabling the incomplete modalities to be restored with the help of the complete modalities. However, these methods mainly focus on the inherent information, i.e., the consistencies between multiple modalities or the data structures among multiple modalities. In this paper, we consider that the defects of insufficient information caused by the incompleteness among modalities should be remedied by supplementing extrinsic information instead.

Transductive multi-modal learning, as a matter of fact, utilizes the extrinsic information from test sets. Eaton et al. [2010] proposed a constrained clustering that can operate with an incomplete mapping, which propagates given pairwise constraints using a local similarity measure to those instances that can be mapped to other modalities; Yin et al. [2017] proposed a novel subspace learning framework for incomplete and unlabeled multi-modal data, which directly optimizes the class indicator matrix, the inter-modal and intra-modal data similarities are preserved to enhance the model. These multi-modal learning approaches with incomplete modal information partially incorporate with the semisupervised learning techniques to relax the issues introduced by modality incompleteness. However, these approaches are under the configuration of transductive learning and are difficult to extend on unseen test data.

In this paper, we propose a novel approach named Semisupervised multi-modal Learning with the Incomplete Modal information (SLIM), which utilizes the intrinsic modal consistencies and extrinsic unlabeled information in one unified framework, and can perform under both transductive and inductive configurations. As a consequence, SLIM can be applied in both classification and clustering tasks. Besides, considering that different modalities have various noise levels, we utilize the square-root loss rather than learning the weight for each modality. Finally, more discriminative classifiers and robust clustering performance can be achieved in SLIM.

3 ProposedMethod

In our incomplete multiple modal learning setting, an instance is characterized by multiple modal features while only with one unified label. Suppose we are given a dataset with N examples and K modalities. The i-th instance xi of k-th modality can be represented as xik ∈ Rdk , where dk is the dimension of the k-th modality. Each instance may has complete or partial modalities as shown in Fig. 1. Without any loss of generality, suppose we have Nc homogeneous examples with complete modal features, meanwhile, we have Nk heterogeneous instances for each modality. Thus, the incomplete modal example set can be represented as D = {Xc,X1,X2, · · · ,XK}, where Xc = {(xi1, xi2, · · · , xiK)}Nc i=1 ∈ RNc×d denotes the examples present in all modalities, d = d1 + d2 + · · · + dK, Xk ∈ RNk×dk denotes the incomplete examples present in the k-th modality. While under the semi-supervised learning scenario, we assume that there are l labeled examples including complete or incomplete examples. For labeled examples, the label of example xi can be represented as yi ∈ {1, · · · ,C}, C is the class number, and the labeled example sets can be represented as Θl. The goal of SLIM is to cluster the N examples into their corresponding clusters, while learning discriminative predictors for each modal prediction.

3.1 The Formulation of SLIM

In this section, we will describe the SLIM in detail. In incomplete modal learning, SLIM aims to utilize the intrinsic modal consistencies and extrinsic unlabeled information in one unified framework, and can perform in both transductive and inductive configurations. Specifically, SLIM can be decomposed into two targets: first, with the incomplete modal examples, we aim to learn the predictors to classify the test instances of each modality accurately. Besides, we wish to cluster the unlabeled instances by modeling a joint transformed matrix factorization problem with respect to each modal similarity matrix and the learned predictions, and pushes them towards a consensus. Thus, SLIM can be defined as:

There are K modalities, the first term ˆLk(Fk, F) denotes the loss of classification of the k-th modality, Fk and F are the classification results and the labels of all the instances on k-th modality to be learned. The second term, ˜Lk( ˆXk, F) considers both the intrinsic and extrinsic information for incomplete modalities. More specifically, it models with a joint transformed matrix factorization problem, here ˆXk ∈ RN×dk is the matrix of the k-th modality with missing rows filling with zeros. In detail, we treat the incomplete similarity matrix of each modal and the learned predictions as a transformed matrix factorization problem, and wish to keep the consistency between them, λ2 > 0 is a balance parameter.

Specifically, objective function ˆLk on the k-th modality in Eq. 1 can be generally represented as the form: min Fk ˆLk(Fk, F) = ℓ(Fk, F) + λ1 2 r(Fk). Here r(Fk) is the

regularizer for modal-specific classifier. λ1 2 is a scalar coefficient to balance the weights of the two terms. Eq. 1 indicates the classifier in ℓ(·) and the prediction results F for instances are connected. Without any loss of generality, the loss function ℓ(Fk, F) can take any convex forms, and we use square loss and linear classifier here for simplicity:

Where Wk ∈ Rdk×C is the linear classifier, bk ∈ RC is the bias for current predictor, 1 is the all one vector, ⊙ represents element wise product operator, Pk ∈ RN×C is the indicator matrix, where [Pk]i,· = 1 iff i-th instance is complete on k-th modality, otherwise [Pk]i,· = 0. In multi-class cases, we expand the label yi for instance xi to a vector with C elements, where yi,j = 1 indicates the i-th instance is with label j, otherwise, yi,j = 0, similarly, F ∈ RN×C denotes the predictions of all instances to be learned, ηk is the number of the complete examples of k-th modality.

The intrinsic information, i.e., the consistencies between the indicator matrix of different modalities are one of the most prominent information for relief the insufficiencies from modality incompletenesses. Therefore, the consistency loss can be treated as the main component for the 2nd term ˜Lk: ‖RΩ(Mk) - RΩ(Y Y ⊤)‖2F, where Mk ∈ RN×N is the similarity matrix of the labeled examples of k-th modality. [RΩ(Mk)]i,j = [Mk]i,j iff i-th instance and j-th instance have complete entries on k-th modality, otherwise [RΩ(Mk)]i,j = 0, and RΩ(Y Y ⊤) has the same definition, Y denotes the label matrix of the labeled examples.

However, in the semi-supervised scenario, more extrinsic information can be involved for better modeling. In this paper, we treat all examples, both label and unlabeled data, with labels as F, and ˜Lk can be reformulated as:

note that here each similarity matrix only has ηk × ηk realvalued entries and we fill the rest entries with zeros. The constraint F Θl = Y restricts the prediction on labeled data as same as the ground truth to avoid collapsing of predictions, Θl here is the index set of the labeled data. In addition, we constrain the predicted values into the same range as true labels by 0 ≤ F ≤ 1 to maintain the intrinsic consistencies. It is notable that the Eq. 3 closely relates to a kernel Kmeans and laplacian-based spectral clustering in a wild condition [Ding and He, 2005], which implies that the whole approach (consisted with this term) can be also applied in clustering tasks.

In addition, it is also notable that real world data always contain noise and outlying entries that result in the unreliable similarity matrix, which will impair the final performance. Previous multi-modal learning methods usually weight different modalities or instances against the affections introduced by noises for ensuring the robustness. However, in semisupervised learning scenario, there are only insufficient number of labeled data for weighting parameters tuning, and as a matter of fact, the affections of noises become one of the barriers for modeling robustly. In this paper, we further employ the square-root loss function instead of the least squares function in Eq. 3 to reduce the affections from noisy data. This solution can be regarded as a weighted regularized least squares form of the original one, where the weight for each modality is: 1 ηk‖RΩ(Mk)-RΩ(FF ⊤)‖F due to [Liu et al., 2014]. This modification can calibrate each modality by considering the different noise levels of all modalities and increases the robustness of the 2nd term in SLIM:

Without loss of generality, we can combine Eq. 2 and Eq. 4 in a unified framework and yield the whole SLIM model:

3.2 Solutions

In this section, we mainly focus on the methodology for addressing the optimization of SLIM represented as Eq. 5 which is convex to Wk, bk yet not a jointly convex problem. An alternative descent algorithm is considered to be utilized for solving this problem, nevertheless, further derivations successfully show that the alternative descent approach is with closed form solutions for some key parameters.

First, it clearly shows that the optimal solution of bk is with closed-form when Wk and F are fixed,

Substitute Eq. 6 into Eq. 5, we can simplify Eq. 5 as:

s.t. 0 ≤ F ≤ 1, F Θl = Y,

where Ck = I - 1 ηk 11⊤ ⊙Pk. Then we can find that the Wk is also with closed-form when F is fixed:

where Ak = ( ˆX⊤ k C ⊤ k Ck ˆXk + ηkλ1I)-1 , Bk = ˆX⊤ k C ⊤ k . Combining Eq. 8 and Eq. 7, we can rewrite the Eq. 7 as:

where tr(·) is the matrix trace operator, H = ∑Kk=1 ΠΓk [CkC ⊤ k B⊤ k A⊤k ( λ1 2 AkBk + 1 2ηk BkB⊤ k AkBk - 1 ηk Bk) + 1 2ηk C ⊤ k Ck], where Γk = {γ1, γ2, · · · , γηk } represents the index set of the complete instances of k-th modality. ΠΓk (A) represents the rows and columns in Γk of matrix A are set as 0. And we can use the project sub-gradient method to optimize Eq. 9 for simplicity.

With the parameters Wk, bk solved in closed-form, we can solve the F with the projected sub-gradient of Eq. 10.

4 Experiments

Data Sets: In this paper, we conduct experiments on 7 two modalities datasets and 8 multiple modalities datasets. In detail, two modal datasets include: Movie dataset is extracted from IMDb, which has 617 movies of 17 genres, and there are two data matrices describing the same movies, i.e., keywords matrix and actors matrix. The main goal is to find the genre of the movies; Citeseer dataset [Sen et al., 2008] is originally made of 4 modalities, i.e., content, inbound, outbound, cites, on the same documents. We follow [Bisson and Grimal, 2012] to choose the content and cites modalities in our experiment. WebKB dataset [Sen et al., 2008] contains webpages collected from 4 universities: Cornell, Texas, Wisconsin and Washington, which have 5 categories, i.e., student, project, course, stuff and faculty. Multiple modal datasets include: NewsGroup dataset [Bisson and Grimal, 2012] is of 6 groups extracted from the 20 Newsgroup datasets, i.e., M2, M5, M10, NG1, NG2, NG3. Every group contains 10 subsets, and we choose the first subset for all 6 groups in our experiment, i.e., News-M2, News-M5, News-M10, News-NG1, News-NG2 and News-NG3, respectively. 3-Source Text data (3Sources)(http://mlg.ucd.ie/datasets/3sources.html) is collected from three online news sources: BBC, Reuters, and Guardian. The description sketches of datasets, including the number of classes, the number of examples and modalities together with the feature numbers are summarized in Table 1.

We run each compared method 30 times for the 15 datasets. For all datasets, we randomly select 70% for training and the remains are for test. For both the training set and test set. As in [Li et al., 2014], in each split, we randomly select 10% to 90% examples, with 20% as interval, as homogeneous examples with complete modality, and the remains are incomplete instances, i.e., in WebKB datasets, they are described by either the content or the citation modality, but not both. For all the examples, we randomly choose 30% as the labeled data, and the left 70% as unlabeled ones. In the training phase, the parameters λ1 and λ2 are selected by 5-fold cross validation from {10-5 , 10-4 , · · · , 104 , 105 } with further splittings on the training datasets only, i.e., there is no overlap between the test set and the validation set for parameter picking up. Empirically, when the variations between the objective value of Eq. 9 is less than 10-6 in iteration, we treat SLIM converged. The average accuracy and std. of predictions are recorded for indicating the classification performance, and the NMI and Purity are recorded as clustering performance. For all compared methods, the parameters are tuned best.

Compared Approaches: Our method solves the problem of semi-supervised clustering and classification with incomplete modality. Thus, to evaluate the performance of our proposed approach, for semi-supervised clustering task, we choose 3 state-of-the-art multi-modal methods: ConvexSub [Guo, 2013]; PVC [Li et al., 2014]; MIC [Shao et al., 2015], considering the limitation of the clustering compared method, we first learn a latent representation of the original data and then using the semi-supervised K-means to get the clustering result. For classification task, we compare with the WNH [Wang et al., 2013], RANC [Ye et al., 2015], MVLIL [Xu et al., 2015]. For compared methods which can't handle incomplete examples, i.e., ConvexSub, WNH, RANC, for fair comparison, we are facilitated with the ALM (Augmented Lagrange Multipliers) [Lin et al., 2010] matrix completion method by first filling in the missing information.

4.1 Experiment Results

Semi-Supervised Clustering/Classification

To demonstrate the effectiveness of our proposed method. For all datasets, we fix the incomplete ratio of the multi-modal data as 90%, and record the clustering results NMI, purity of the SLIM and compared methods in Table 2, Table 3, and prediction accuracies (avg.± std.) in Table 4. It is notable that PVC method can only leverage two modalities, so we have not compared with PVC for multi-modal datasets.

From the Table 2 and Table 3, it reveals that for both two modal datasets and multiple modal datasets, SLIM almost consistently achieve the significant superior clustering performance on either purity or NMI comparing to all other methods, except for Wisconsin on purity and Movie on NMI. It can be owing to that in SLIM, the similarity matrices of all datasets are initialized with cosine similarity for more robust generalization, rather than task specific similarity matrix construction method. Besides, from the Table 4, it can be observed that SLIM also achieve the best performance on most datasets except Reuters. This phenomenon clearly reveals the effectiveness of considering the high order consistencies between each modal similarity matrix and the learned predictions, consequently, we can learn the most discriminative predictors for each modality, while acquiring better results.

Influence of Number of Incomplete Multi-Modal Data

In order to explore the influence of the ratio of the incomplete modalities on performance, extensive experiments are conducted. In this section, the parameters in each investigation are fixed as the optimal values selected in above investigations, the λ1 and λ2 in SLIM are set 1, while the ratio of the incomplete data varies in {90%, 70%, · · · , 10%}, with 20%

as interval. Due to the page limits, results on 4 datasets, i.e., Texas, Movie, News-M2, News-NG1, and the results of NMI, purity and accuracy are recorded in Fig. 2, Fig. 3 and Fig. 4. From these figures, it clearly shows that SLIM achieves the best on all datasets. Besides, we can also find that SLIM achieves superiorities from high incomplete ratio, and the performance of SLIM increases faster than compared methods as incomplete ratio decreasing.

5 Conclusion

This paper focus on the issues of incomplete modalities in multi-modal learning. Previous mainstream solutions alleviate the affections of incomplete modal issues via utilizing the intrinsic information from the data structures or prediction consistencies among multiple modalities. A few of multimodal learning methods consider making use of the auxiliary information from test data, and thus form transductive solutions which cannot be applied on unseen data. In this paper, we proposed a novel multi-modal learning approach, SLIM, with more extrinsic information exploited from unlabeled data in a semi-supervised scenario, and yielded an inductive learner which consequently can be applied in general multi-modal circumstances. By leveraging the intrinsic and extrinsic information together, SLIM possesses a unified framework which is closely related to classification and semisupervised clustering. Therefore, SLIM can be easily adopted to either classification or clustering tasks. Besides, by incorporating in square-root loss, SLIM becomes less sensitive to data noise and ensures the robustness of the whole solution. Experimental evaluations on real-world applications demonstrate the superiority of our proposed method over the compared methods. How to extend the scalability with improved performance and theoretical analysis on incomplete multimodal learning can be interesting future work.

Acknowledgments

This work was supported by National Key R&D Program of China (SQ2018YFB100002) NSFC (61773198, 61632004, 6163000043).