Deep Correlated Predictive Subspace Learning for Incomplete Multi-View Semi-Supervised Classification

1 Introduction

Data can be represented by multiple views in many real-world applications. For instance, an image can be described by different types of features such as LBP, SIFT, and color histogram. Different views usually provide complementary information and leveraging multi-view data is beneficial to improving the overall learning performance. However, some views may not contain complete information in real tasks (e.g., some Web pages contain both image and text information while some may only contain text information). The incomplete data may lead to performance degradation or even failure of the conventional multi-view methods.

Up to now, several approaches have been developed for the incomplete multi-view unsupervised or semi-supervised learning. For two-view incomplete data clustering, some methods [Li et al., 2014; Zhao et al., 2016; Xu et al., 2018] adopt matrix factorization model to seek a common latent subspace, in which the samples of different views are constrained to have the same representation. To conduct clustering with more than two views, MIC [Shao et al., 2015]

∗Corresponding Author

adopts a co-regularized method to push the learned subspace of all the views into a common consensus. DAIMC [Hu and Chen, 2018] constructs a consistent data representation with l2,1-norm to reduce the influence of missing views. A spectral clustering based method is proposed [Wen et al., 2019] to learn the consensus representation and the similarity graphs for all views. In addition to unsupervised learning, semisupervised learning methods for incomplete multi-view data are developed recently. SLIM [Yang et al., 2018] learns the classifiers by leveraging the intrinsic view consistency and extrinsic unlabeled information, which is further used to predict the class of unlabeled samples. To predict multiple labels from incomplete views, iMVWL [Tan et al., 2018] learns a consistent subspace by considering the label correlations which can reinforce the prediction results.

While some successful incomplete multi-view learning methods have been proposed, there remains room for practical improvements and additional theoretical understanding. First, most of the incomplete multi-view learning methods are based on shallow models, which cannot learn robust representation for data with complex distributions. Second, since some samples may lack of feature descriptions in some views, the data correlation is an important clue to make the data complement each other and improve the discriminating power of data representation. Third, different views have shared and independent information, omitting such shared and private nature of multi-view data would limit the performance of classification.

To solve the aforementioned problems, we propose a Deep Correlated Predictive Subspace Learning (DCPSL) method for incomplete multi-view semi-supervised classification. We integrate semi-supervised deep matrix factorization, correlated subspace learning and multi-view class label prediction into a unified objective function to jointly learn the deep correlated predictive subspace and the shared and private label predictors. The proposed DCPSL is able to learn the proper subspace representation that is suitable for class label prediction, which can further improve the classification performance. The discriminating power of the learned subspace representation is guaranteed from three aspects. First, the label information is incorporated into deep matrix factorization model which makes the learned representation achieve obvious category structure. Second, data correlation is utilized to make the incomplete multi-view data complement each other, which further enhances the effectiveness of the subspace. Third, we introduce the shared and private label predictors for classification so that the complementarity information of the learned multi-view subspace can be leveraged and more accurate class labels can be predicted. The experiments conducted on several multi-view datasets verify the effectiveness of our method. The main contributions of this work are summarized as follows:

• We extract abstract and high-level multi-view representation by semi-supervised deep matrix factorization model. By encoding the label information into the representation, we can significantly improve the discriminating power of the subspace representation.
• We learn low-rank subspaces from deep matrix factorization model, where data correlation can be effectively extracted. Then the data correlation is further used to enhance the effectiveness of the learned subspace representation.
• We propose to divide the label predictor into shared and private parts, which can effectively utilize multiview complementary information and improve the performance of class label prediction.

2 ProposedMethod

2.1 Preliminaries

For complete multi-view data with V views, sample xi is composed of V features {x(1) i , x(2) i , ..., x(V ) i }, where x(v) i ∈ Rdv and dv is the feature dimension of the v-th view. Due to the incomplete data problem, the features of some views may be missing for xi. Let τ denote the index of views that xi appears in, then xi is compose of {x(j) i }j∈τ . Given n samples, incomplete multi-view data can be denoted by a set of matrices {X(v) ∈ Rdv×nv }Vv=1 , where X(v) consists of nv samples appear in the v-th view. We have nv ≤ n due to the multi-view data missing problem. In semi-supervised setting, some samples have labels while others do not. We use Y = [y1, y2, ..., yl] ∈ {0, 1}c×l to represent the label matrix, where l is the number of labeled samples and c is the class number. For a labeled sample xj, if it belongs to the i-th class, Yij = 1; otherwise Yij = 0. Our objective is to predict the class labels of the unlabeled samples based on the incomplete multi-view feature matrices {X(v) }Vv=1 and the label matrix Y .

To capture complex data distributions, deep Semi-NMF [Trigeorgis et al., 2014] is proposed to learn the inherent attributes and higher-level feature representation. It decomposes data matrix X into m layers as

where Zi ∈ Rki-1×ki and Hm ∈ Rkm×n are the loading matrix and coefficient matrix, respectively. ki is the dimension of the i-th layer, and (a)+ = max(0, a) corresponds to the hinge operation.

Subspace clustering [Vidal, 2011; Elhamifar and Vidal, 2013] assumes that data are drawn from different subspaces, and its goal is to cluster data according to their underlying subspaces. The low-rank representation (LRR) method [Liu et al., 2013] learns a low-rank subspace representation and achieves promising clustering performance. Given a data matrix X ∈ Rd×n , LRR solves the self-representation problem by finding the lowest rank representation of all data as

where S ∈ Rn×n is the learned low-rank subspace representation of data X, E is the error matrix, || · ||∗ denotes the nuclear norm of a matrix, which equals to the sum of its singular values. The learned subspace S is capable of capturing the correlations of data samples.

2.2 ProblemFormulation

There are two main problems in semi-supervised incomplete multi-view learning: i) how to obtain the proper multi-view data representation for classification, and ii) how to predict the class label of unlabeled samples. To this end, we first introduce deep correlated subspace learning to properly represent multi-view data, and then introduce multi-view shared and private label prediction. Finally, the ultimate objective function integrates the two subproblems to obtain both proper subspace representation and accurate classification results.

Deep Correlated Subspace Learning

To learn the proper multi-view representation for semisupervised classification, three factors should be considered. First, the class label information should be used to guide the representation learning. Second, data correlations contain abundant descriptions of relations between data, which can be used to enhance the effectiveness of representation. Third, data samples may be produced by complex data distributions. Compared to shallow models, deep models can better learn the inherent attributes and higher-level data representation. In the light of the above points, we introduce the following deep correlated subspace learning objective to obtain data repre-

where m is the number of layers. Z (v) i and H(v) m are the learned loading matrix and coefficient matrix for the v-th view, respectively. Inspired by [Liu et al., 2012], the label constraint matrix P (v) ∈ R(nv-lv+c)×nv is used to guide the learning of H(v) m , where c is the class number and lv is number of labeled samples in the v-th view. For example, given samples {x(v) 1 , x(v) 2 , ..., x(v) nv } from the v-th view, x(v) 1 , x(v) 2

belong to class 1, x(v) 3 , x(v) 4 belong to class 2, and the other samples are unlabeled, then P (v) is defined as

where Inv-4 ∈ R(nv-4)×(nv-4) is an identity matrix. The new data representation becomes H(v) m P (v) where the label information can be well preserved. It is easy to check that if x(v) i and x(v) j have the same label, then (H(v) m P (v) )(:,i) = (H(v) m P (v) )(:,j).

Although H(v) m P (v) achieves high-level feature representation and preserves the label information, it does not take advantage of the data correlations, which may affect its discriminating power. We further learn the low-rank subspace S(v) ∈ Rnv×nv from H(v) m P (v) to reveal the intrinsic data correlations. Through self-representation learning, each element in S(v) i well captures the relations between two samples. Then we can use H(v) m P (v)S(v) i as the enhanced data subspace representation to achieve better prediction.

Multi-view Shared and Private Label Prediction

Given the enhanced data subspace presentation H(v) m P (v)S(v) i ∈ Rkm×nv and label matrix Y ∈ Rc×l , the class labels of unlabeled samples should be predicted. We adopt linear regression model for label prediction due to its convenience and effectiveness. It should be noted that different views share some common information while retaining some independent information. Only by considering the shared and private nature of multi-view data can we make accurate predictions of data labels. Hence, we propose multi-view shared and private label prediction as follows

where F ∈ Rc×n is the predicted label matrix, πl is the index set of the labeled samples. Fπl = Y restricts the prediction on labeled data as same as the ground truth. The label predictor for the v-th view can be denoted by W (v) = Ws + W (v) p , where Ws ∈ Rc×km is the shared part that used for all the views and W (v) p ∈ Rc×km is the private part that used only for the v-th view. We impose l1-norm regularization on W (v) p to make it adaptively capture the private components of the v-th view. Different views adopt different label predictors so that multi-view complementary information can be leveraged to generate more accurate results.

To cope with the incomplete multi-view data, the alignment matrix Q(v) ∈ Rn×nv is introduced which represents the correspondence between nv samples appear in the v-th view and all the n samples. For example, if the three samples from the v-th view, i.e., H(v) m P (v)S(v) i ∈ Rkm×3 , correspond to the 2nd, 3rd and 5-th sample in F , respectively, then Q(v) ∈ Rn×3 is constructed as

Our method can flexibly deal with the incomplete multi-view data, so that all the samples from different views can be leveraged for label prediction effectively. After obtaining F, any unlabeled sample can be assigned to a class, e.g., the class of the unlabeled sample xi is determined by argmaxj Fji.

Ultimate Objective Function

The ultimate objective function is formed by combining the above sub-problems. By jointly conduct deep correlated subspace learning and multi-view shared and private label predictor learning, the ultimate objective function is proposed, which is

where,

We introduce λ to control the weight of label prediction term. By solving the ultimate objective function J, our method can jointly learn subspace representation, label predictors and the predicted label matrix F, so that more effective data representation and accurate classification results can be obtained.

3 Optimization

In this section, we provide more details on solving the ultimate objective function (7) by the iterative block coordinate descent algorithm, where only one variable is updated while the others are fixed in each iteration. Specifically, we pretrain each layer and obtain initial Z (v) i and H(v) m . Then, each variable such as Z (v) i , H(v) m , S(v) , Ws, W (v) p and F is updated. The details of pre-training and derivations of update rules are presented as follows. Algorithm 2 presents the algorithm for solving problem (7).

3.1 Pre-training of Deep Semi-NMF

The latent factors Z (v) i and H(v) i in Deep Semi-NMF should be pre-trained before solving the other variables. For each view v = 1, ..., nv, the first layer is learned by X(v) ≈ Z (v) 1 H(v) 1 , where Z (v) 1 ∈ Rdv×k1 and H(v) 1 ∈ Rk1×n . After that, coefficient matrix H(v) 1 is further decomposed by H(v) 1 ≈ Z (v) 2 H(v) 2 , where Z (v) 2 ∈ Rk1×k2 and H(v) 2 ∈ Rk2×n . This process is continued until all the layers are pretrained, i.e., H(v) 2 ≈ Z (v) 3 H(v) 3 , · · · ,H(v) m-1 ≈ Z (v) m H(v) m .

3.2 Updating Z (v) i

To solve Z (v) i , let the partial derivative ∂(J)/∂(Z (v) i ) = 0, and then the update rule can be obtained by Z (v) i ← Ψ†X(v)Ω† , where Ψ = Z (v) 1 · · · Z (v) i-1 and Ω = Z (v) i+1 · · · Z (v) m H(v) m P (v) . (·)† is the Moore-Penrose pseudoinverse operator and A† = (AT A)-1AT .

3.3 Updating H(v) m

To solve H(v) m , we take the related parts from J and obtain

where we introduce W (v) = Ws +W (v) p . Using the standard Semi-NMF optimization method, we can derive the following multiplicative update rule,

where

The notation of [·]pos and [·]neg in (11) and (12) denote the operation that replaces all the negative and positive elements in the matrix by 0, respectively, and can be defined as

3.4 Updating S(v)

Keeping the parts that are related to S(v) i from ultimate objective function J, we obtain the following problem

which can be solved by alternating direction method of multipliers (ADMM) [Liu et al., 2013]. The augmented Lagrangian function of problem (13) is,

where {Bi}nv i=1 are the auxiliary variables, {Ri}nv i=1 are the Lagrange multipliers. lR+(·) is defined as

We set the partial derivative ∂(L(S(v) ,B1,B2,B3,B4))/ ∂(S(v) ) = 0 to solve S(v) as

where ηi = Bi -Ri, ξ (v) = H(v) m P (v) and I is the identity matrix. By using the similar updating method, we solve B1 and B3 as

We use Singular Value Thresholding operator [Cai et al., 2010] to solve B2. Let Θτ(A) = UΛτV T, where A = UΛτV T is the singular value decomposition, and Λτ(a) = sgn(a)max(|a| - τ, 0) is the shrinkage operator. Then the update rule for B2 is B2 ← Θα/µ(S(v) + R2). Considering the non-negative constraint, B4 can be solved by B4 ← max(S(v) +R4, 0). Furthermore, Lagrangian multipliers R1, R2, R3 and R4 should also be updated. Algorithm 1 summarizes the ADMM optimization procedure for solving S(v) .

3.5 UpdatingWs andW (v) p

In this subproblem, we fix the related parts of Ws and W (v) p from J and optimize the Ws and W (v) p . The optimization problem becomes

We can update Ws and W (v) p by solving ridge regression and lasso regression problems. Ws can be updated by the closed form solution, and W (v) p can be updated by the standard coordinate descent method [Lange, 2008].

3.6 Updating F

Considering the equality constraint Fπl = Y imposed on F, we introduce a penalty to arrive at the following equivalent objective function to solve F ,

where U ∈ {0, 1}n×l is the correspondence matrix. If the i-th column in F corresponds to the j-th column in Y , then Uij = 1, and Uik = 0, k ̸= j. η > 0 is used to control the equality constraint, which should be large enough to ensure the equality constraint satisfied. Let the partial derivative

Algorithm 1: The algorithm to solve S(v) .

Input: H(v) m , P (v) , W (v) , F, Q(v) , α, λ, µ. 1 Initialization: ∀j,Bj = Rj = 0 2 while not converged do 3 Update S(v) i by (15); 4 Update B1,B2,B3,B4 in Sec. 3.4; 5 Update the Lagrange multipliers: 6 R1 ←R1 - (B1 -H(v) m P (v)S(v) ); 7 R2 ←R2 - (B2 -S(v) ); 8 R3 ←R3 - (B3 -W (v)H(v) m P (v)S(v) ); 9 R4 ←R4 - (B4 -S(v) ); 10 end

Algorithm 2: The learning procedure of our approach.

Input: {X(v) , P (v) ,Q(v) }Vv=1 , m, λ, α, β1, β2. 1 Initialize Z (v) i , H(v) m by pre-training. 2 while not converged do 3 for v = 1, ..., V do 4 Update {Z (v) i }mi=1 in Sec. 3.2; 5 Update H(v) m by (10); 6 Update S(v) by Algorithm 1; 7 Update W (v) p by solving (18); 8 end 9 Update Ws by solving (18); 10 Update F by (20); 11 end

∂(J(F ))/∂(F ) = 0, the following update rule can be derived to solve F

3.7 Complexity Analysis

Pre-training and fine-tuning are the two stages of our algorithm. The computational complexity for pre-training is of order O(ndk + nk2 + kn2 ), where n is the number of samples, d is the maximum dimension of multi-view data, and k is the maximum dimension of all the layers. All the variables are updated in the fine-tuning stage. The complexity for finetuning is of order O(ndk + nk2 + kn2 + n3 + ck2 + cn2 ). The proposed algorithm is efficient and achieves comparative complexity with NMF and subspace clustering methods.

4 Experiments

4.1 Experimental Setup

Datasets. Four commonly used multi-view datasets are used to evaluate the proposed method. NUS [Chua et al., 2009] is a web image dataset for object recognition. We adopt 31 categories and choose 100 images for each class. Five features

(b) Sensitivity analysis of m on each dataset

Figure 1: Parameter analysis w.r.t. β1, β2 and m.

are extracted to represent images. SUN [Xiao et al., 2016] contains 899 categories and 130519 images. We randomly choose 30 classes, and each class has 100 images. Five types of visual features are adopted as different views. Caltech [Li et al., 2007] is an object recognition data set containing 101 categories of images. We select the widely used 20 classes and get 1230 images. Five features are extracted from the images. Flowers [Nilsback and Zisserman, 2006] is composed of 17 flower categories, with 80 images for each category. Each image is described by three views using color, shape, and texture features. For each dataset, 70% data are randomly sampled for training and the remaining 30% data are used for testing. To create incomplete multi-view data scenarios, we randomly remove ε% samples from each view and ensures that each sample appears in at least one view.

Methods and parameter setting. We compare six state-ofthe-art multi-view learning methods to demonstrate the effectiveness of our method: AMGL [Nie et al., 2016], MLAN [Feiping Nie and Li, 2018], MLHR [Yang et al., 2013], GLCC [Zhang and Zhang, 2016], MVAR [Tao et al., 2017], iMVWL [Tan et al., 2018]. The first five methods are conventional multi-view semi-supervised learning methods which are designed for complete view data. For a fair comparison, we adopt the matrix completion method [Lin et al., 2010] by filling the missing information and then conduct classification for these methods. The parameters of all the compared methods are set as suggested in the corresponding papers. The parameters of our method are determined by five fold cross-validation. λ and α are selected from {10-3 , 10-2 ,..., 102 }. β1 and β2 are selected from {0.005, 0.01,..., 50}. All the experiments are repeated ten times and the averaged performance are reported. For the evaluation metric, we follow [Feiping Nie and Li, 2018] and use accuracy for performance evaluation, which calculates the proportion of the correctly classified samples.

4.2 Experimental Results

Semi-supervised Classification. To illustrate the semisupervised classification performance of DCPSL, we fix the incomplete rate of the multi-view data as ε% = 50%, and present the classification accuracy of all the methods with different percentages of labeled samples in Figure 2. In can be observed that DCPSL achieves better classification accuracy compared to all the other methods on each dataset. The largest performance improvements on the four datasets are: 2.9%, 3.3%, 3.7% and 3.4%, respectively. The classification results clearly verify that the learned deep correlated predictive subspace is helpful to improve the class label prediction performance for incomplete multi-view data. The conventional multi-view methods such as AMGL, MLAN, MLHR, GLCC, MVAR fail to achieve good performance due to they cannot well handle the incomplete multi-view data. Since DCPSL can jointly learn the proper subspace representation and discriminative label predictors for incomplete multi-view data, our method outperforms the other methods with different percentages of labeled samples.

Influence of Incomplete Rate. To evaluate the influence of incomplete rate ε% on classification, we conduct classification experiments by changing the incomplete rate ε% from {0, 10%, 30%, 50%, 70%} while fixing the percentage of labeled samples to 0.3. The classification results are shown in Figure 3. It clearly shows that DCPSL performs better than the other methods on each dataset. Due to the influence of missing views, the performance of all the methods are declined with the increase of ε%. In contrast, DCPSL achieves better performance than the others by effectively leveraging both data correlation and multi-view complementary information of the incomplete data.

Parameter Analysis. We conduct the sensitivity analysis with several critical parameters β1, β2, and m. ε% is set to 0.5, and the percentage of labeled samples is set to 0.3. The experimental results of β1 and β2 on NUS dataset are shown in Figure 1(a). DCPSL obtains competitive performance when β1 = {0.05, ..., 5} and β2 = {0.1, ..., 10}. The sensitivity analysis of the number of layers m are shown in Figure 1(b). Better classification results can be obtained when m = {2, 3} for most of the datasets. The shallow model (m = 1) fails to learn the discriminative subspace representation, so its classification performance is limited.

5 Conclusion

In this paper, a deep correlated predictive subspace learning method (DCPSL) is developed for incomplete multiview semi-supervised classification. Our method is capable of jointly leveraging the data correlations and multi-view complementary information, which is achieved by integrating deep correlated predictive subspace learning and multiview shared and private label prediction into a unified objective function. Compared with the state-of-the-art multi-view semi-supervised learning methods, DCPSL can better handle the incomplete multi-view data and achieves competitive classification results on various practical datasets.

Acknowledgments

This work was supported in part by National Natural Science Foundation of China: 61802028, 61532006, 61772083 and 61877006, in part by the Fundamental Research Funds for the Central University (No. 2018RC44), in part by US NSF IIS-1816227.