Adversarial Incomplete Multi-view Clustering

1 Introduction

Many real-world datasets contain multiple kinds of features. Such datasets are called multi-view data. For example, photos shared on an online social network have both visual features and tag descriptions. Different views exhibit consistency and complementary properties of the same data, leading to extensive research on multi-view learning. Synthesizing multiview features leads to a more comprehensive description of the data, which could benefit many tasks such as classification [Xu et al., 2018a] and clustering [De Sa et al., 2010]. This work is concerned with multi-view clustering, which integrates multiple views to help identify essential grouping structure in an unsupervised manner.

∗Corresponding author

Most of the previous studies on multi-view clustering always assume that all of the views are complete. However, in real-life cases some views could be missing for some data instances. For instance, in Twitter, tweets can contain text, images and videos, but only a part of them contain all the three views. As a result, the lack of partial views makes most multi-view clustering methods inevitably degenerate or even fail. In this paper, we are focused on this Incomplete Multiview Clustering (IMC) problem.

The prevalent solutions to the IMC problem mainly depend on Non-negative Matrix Factorization (NMF) [Li et al., 2014; Zhao et al., 2016; Xu et al., 2018b]. NMF based methods learn a common latent space for complete instances and private latent representations for incomplete instances. Nevertheless, they cannot be easily extended to handle more than two incomplete views. Weighted NMF based approaches [Shao et al., 2015; Hu and Chen, 2018; 2019] first fill in the missing values by average feature values or matrix completion methods, and then handle the problem with the help of weighted NMF by giving filled data lower weights than original data. However, such a simple padding scheme is invalid when the missing ratio is large. Meanwhile, these methods cannot fully capture the hidden information of the missing data for consensus representation learning because the missing data is not well recovered.

Recently, some IMC methods considering missing data inference have been proposed. In [Wen et al., 2019], Wen et al. proposed to learn the missing views of instances jointly with the NMF framework. However, this factorization model corresponds to a shallow projection of the data and may not well handle the dependence and discrepancy among different views. Wang et al. proposed a deep IMC method [Wang et al., 2018] based on Generative Adversarial Network (GAN) [Goodfellow et al., 2014]. It uses one view to generate the missing data of the other view by Cycle GAN [Zhu et al., 2017]. This method can only handle two-view data because the cycle generation framework is dedicated to the two-view case.

To eliminate the above limitations and drawbacks, we propose a new IMC framework, named Adversarial Incomplete Multi-view Clustering (AIMC). As shown in Figure 1, the encoders learn the aligned subspaces z(v) 's of multiple views, where we require the same dimension of the subspaces to represent the same high-level feature. We use average pooling to learn common representation z, which synthesizes the consistent and complementary information from all the views. Then for each view v, both z(v) and z are used separately to reconstruct x(v) via the same decoder gv. The motivation of this design is twofold: (1) We need to align the subspaces of z(v) 's through z by forcing them to be similar. The naive idea of minimizing the distance between each z(v) and z would force one view to exhibit complementary high-level features in other views. By putting them in the same role for reconstructing x(v) , we implicitly force them to be similar. (2) When view v is missing, we can still employ z for a good inference of x(v) using consistent information among views. To obtain good high-level features and inference of missing data, we train the framework by three losses: an elementwise reconstruction loss, an adversarial loss and a KullbackLeibler (KL) aligned clustering loss. The reconstruction loss is standard for high-level subspace learning, but ignores the correlations between data dimensions. Hence, we train additional discriminators Dv to distinguish the original data and reconstructed data. The decoders will get a deeper semantic understanding when the discriminators cannot distinguish them [Pathak et al., 2016], making decoders good generators for missing data inference. The KL aligned clustering loss is imposed on complete data only, which tries to make the latent distributions for all the views consistent and compact cluster-wise. After training, we infer missing data and apply AIMC once more on the generated complete data to obtain the updated common representation z for clustering.

The major contribution of this work is a novel method for the IMC problem with the consideration of more accurately inferring missing information by the adversarial encoderdecoder pipeline. Meanwhile, the encoders learn high-level common representation via multi-layer non-linear transformations. Another contribution is that we extend the KL aligned clustering loss [Xie et al., 2016] to the multi-view case to simultaneously learn aligned high-level subspaces and capture better clustering structures for all the views. Finally, we empirically evaluate AIMC on three real world datasets and show its superiority over state-of-art baseline methods.

2 RelatedWork

In the section, we briefly review two lines of related work, incomplete multi-view clustering and unsupervised multi-view deep learning.

2.1 Incomplete Multi-viewClustering (IMC)

Li et al. [Li et al., 2014] present the first NMF based IMC method, PVC, which learns common representations for complete instances and private latent representations for incomplete instances with the same basis matrices. Then these common and private representations are used together for clustering. Inspired by this work, [Zhao et al., 2016] integrated PVC and manifold learning to learn the global structure of multi-view data. Nevertheless, these methods can only deal with two-view data, limiting their application scope. Weighted NMF based approaches, such as [Shao et al., 2015; Hu and Chen, 2018; 2019], were proposed to deal with more than two views by filling in the missing data and giving them lower weights. However, they still failed to capture the hidden distribution of missing data. Wen et al. [Wen et al., 2019] added an error matrix to compensate the missing data. The original incomplete data matrix and the error matrix were combined to form a completed data matrix in a matrix factorization architecture. The common latent representation and error matrix (i.e. missing data) were jointly optimized. Although this work tried to capture the hidden distribution of missing data, it was still based on matrix factorization which corresponds to a shallow projection. Such a simple model cannot well handle complex relationships between low-level features of multi-view data.

2.2 UnsupervisedMulti-viewDeep Learning

Inspired by deep learning [Hinton and Salakhutdinov, 2006], different deep models were proposed recently for multi-view learning. There are three main categories. The first category is based on Canonical Correlation Analysis (CCA) [Hotelling, 1936], which finds linear projections of two views that are maximally correlated. Andrew et al. [Andrew et al., 2013] proposed deep extension of CCA (Deep CCA) that simultaneously learned two deep nonlinear mappings of two views. The second one is based on autoencoder. Ngiam et al. [Ngiam et al., 2011] explored extracting shared representations by training a two-view deep autoencoder which aimed to best reconstruct the two-view input. Shahroudy et al. [Shahroudy et al., 2018] introduced autoencoder-based network to analyze the multi-view (RGB+Deep features) videos. Wang et al. [Wang et al., 2015] found that CCA-based approaches tended to promote autoencoder-based approaches and proposed a deep model that combined CCA and autoencoder. Recently, there exist several approaches utlizing GAN [Goodfellow et al., 2014] for multi-view learning such as CycleGAN [Zhu et al., 2017] and StarGAN [Choi et al., 2018]. However, those methods mainly focused on cross-domain data generation. Wang [Wang et al., 2018] proposed Consistent GAN for the two-view IMC problem. It uses one view to generate the missing data of the other view, and then performs clustering on the generated complete data. However, it is dedicate to the two-view case, while our AIMC can be applied to IMC problems with an arbitrary number of views.

3 The Method

In this section, we present Adversarial Incomplete Multi-view Clustering (AIMC) in detail, together with its implementation.

3.1 Notations and Problem Statement

In the incomplete multi-view clustering setting, an instance is characterized by multiple views and may have complete or partial views as shown in Figure 2. Suppose we are given a dataset with H views, N complete instances, and ˜ N incomplete instances. We use x(v) n /˜x(v) ˜n ∈ Rdv (v = 1, . . . ,H) to denote the feature vector for the v-th view of the n-th/˜n-th instance in the set of complete/incomplete instances, respectively, where dv is the dimensionality of the v-th view. An indicator matrix M ∈ RH× ˜ N for incomplete instances is defined as:

where each column of M encodes the view status (present/absent) for the corresponding instance. Since there are missing views, for every ˜n we have ∑Hv=1 Mv˜n < H.

Our goal is to group all the N + ˜ N instances into K clusters.

3.2 Adversarial Encoder-Decoder Pipeline

As shown in Figure 1, the overall architecture is an encoderdecoder pipeline which consists of H encoders {fv}Hv=1 , H decoders {gv}Hv=1 and H discriminators {Dv}Hv=1 . The encoders aim to obtain high-level latent representations and the common latent representation. The decoders take these representations to reconstruct the original data and also generate missing views of incomplete instances. The discriminators are added to assist in training this model for better inferring missing data. After obtaining the predicted missing data, the encoders use the generated complete data to calculate the common representation, which can capture the hidden information of the missing data. Details regarding each component will be elaborated as below.

Encoder

The v-th encoder fv is dedicated to learning aligned subspace z(v) ∈ Rc for all the input data of the v-th view. The same latent space dimensionality c is shared among the H views. The latent representations {z(v) }Hv=1 need to be aligned. To this end, we add the average pooling layer, which averages all the latent representations for different views, to get the common representation z. One feature in the common subspace will be activated when it is activated in at least one view. The activation intensity depends on the number of activated views. I.e., consistent features tend to have higher intensity than complementary features. To achieve better subspace alignment and inference of missing data, we put each z(v) and z in the same role (i.e. input to the decoder) for reconstructing the corresponding x(v) .

Decoder

We now discuss the second half of our pipeline, the decoders {gv}Hv=1 , which reconstruct the original data and generate the missing views of incomplete instances using the common latent representations. ˆx(v) , the output of gv, is the reconstruction of the original data x(v) (or the generated missing data). The traditional element-wise mean squared error is utilized to evaluate reconstruction. Nevertheless, this objective does not explicitly try to capture the correlations between data dimensions which could be important for describing the underlying distribution of a view. To capture such correlations for better missing data inference, we employ the discriminators {Dv}Hv=1 . For each view, the original data x(v) and the reconstructed data ˆx(v) with their labels (True and False respectively) are used to train the discriminator Dv. Each pair (gv,Dv) forms a adversarial relationship, leading to a deeper semantic understanding of the data, and consequently better generation of missing data. This framework explicitly supplements the missing data to learn a more complete common representation for incomplete multi-view data.

3.3 Loss Function

We train our AIMC model by the original complete instances {x(v) n } and incomplete instances {˜x(v) ˜n }. The goal is to learn common representation for multi-view data and to precisely reconstruct the original data simultaneously. We now describe different components of our loss function.

Reconstruction Loss

We use L2 distance as our element-wise reconstruction loss function, LR:

where z(v) n /˜z(v) ˜n is the latent representation of the v-th view and the n-th/˜n-th instance of complete/incomplete instances. zn/˜z˜n is the common latent representation of the complete/incomplete instances. These latent vectors are calculated as in Eq. (3).

The first term of LR is the reconstruction criterion utilizing the common latent representation. It aims to learn the bidirectional mapping between the original data space and the common embedding space. The second term denotes viewspecific reconstruction, which implicitly forces the latent representation of each view close to the common representation by feeding them into the same decoder network. The last two terms are designed in the same notion for the incomplete part. However, the L2 loss focuses on each data dimension separately while ignores the correlations between data dimensions. In the next, we detail the adversarial loss in our model to alleviate this problem.

Adversarial Loss

The adversarial loss is based on idea of GAN [Goodfellow et al., 2014]. Traditional GAN consist of two parts, generator G and discriminator D. G maps the data w from noise distribution Pw to data distribution Pdata. The training procedure is a two-player game where D tries to distinguish the ground truth data and the output of G, while G tries to confuse D by generating data as "real" as possible. The objective for GAN can be formulated as follows:

In our model, the problem of multi-view reconstruction is modeled in the adversarial setting to learn the conditional distributions Pdata(v)(x(v) |⋆), Pdata(v)(˜x(v) |⋆), where ⋆ can be common latent representations z, ˜z, or view-specific latent representations z(v) , ˜z(v) . Specifically, we adapt the traditional GAN framework to multi-view reconstruction by treating decoders as generators, i.e., g ≜ G. The adversarial loss for reconstructing complete data x(v) by common latent representation z is

Similarly, we consider the adversarial loss for reconstructing complete data x(v) by view-specific latent representation z(v) . For incomplete data ˜x(v) , the adversarial loss is designed in the same manner. We omit the details due to the forms are similar to Eq. (5). Hence the overall adversarial loss for our model, LA, is the sum of the above four terms.

The decoders are trained to reconstruct the original data (x(v) and ˜x(v) ) through the L2 reconstruction loss LR and the adversarial loss LA. The reconstruction loss tries to reproduce original data as accurately as possible, while the adversarial loss tries to capture the underlying data distribution. Combining these two loss functions would lead to robust encoders and decoders for high-level representation projection and missing data inference.

KL Aligned Clustering Loss

According to the aforementioned analysis, we use average pooling to get the common representation of multiple views. In order to further polish the latent representations for multiview data, we propose the KL aligned clustering loss on complete instances, which minimizes KL divergence between the clustering distributions calculated by z(v) 's and emphasized target distribution calculated by z. Following [Xie et al., 2016], we model the probability of a complete data instance n being assigned to cluster k using the Student's t-distribution [Maaten and Hinton, 2008], producing a distribution for each view:

where δ is the number of degrees of freedom for the Student's t-distribution. Following Xie et al., we set δ to 1. µ(v) k is the clustering centroid of the k-th cluster in the latent space for the v-th view. The centroids are initialized by K-means and updated by Stochastic Gradient Descent (SGD) [Xie et al., 2016]. q(v) nk can be interpreted as the probability of assigning instance n to cluster k (i.e., a soft assignment) for the v-th view.

Similarly, we calculate the soft assignment distribution qnk for the common latent representation zn. In order to improve cluster compactness, we put more emphasis on data points assigned with high confidence by calculate the emphasized target distribution pnk as follows:

where fk = ∑n qnk are soft cluster frequencies. It first squares qnk and then normalizes it by frequency per cluster. Hence, high probabilities will be emphasized. We define the objective function as a KL divergence between the soft assignments q(v) nk and the emphasized target distribution pnk as

The distribution gap of different views will be reduced gradually by optimizing LD, leading to more compatible latent representations across different views. Meanwhile, the clusters are iteratively refined, which helps obtain a better clustering structure of the common representation z.

Joint Loss

By synthesizing the above objectives, the overall optimization problem of AIMC is formulated as

where α, β > 0 are hyper-parameters.

3.4 Implementation

The whole process of using AIMC for clustering is summarized as below.

Step 1: Training AIMC model. We use the original complete instances {x(v) n } and incomplete instances {˜x(v) ˜n } to train AIMC by optimizing Eq. (9). We use the adaptive moment (Adam) optimizer to train our model and set the learning rate to 0.0001. Our model is implemented by PyTorch and run on Ubuntu Linux 16.04.

Step 2: Generating missing data. In this step, we put the common representation ˜z˜n calculated through Eq.(3) into the trained decoder network {gv}Hv=1 to generate the missing views of incomplete instances:

Step 3: Calculating the common representation using the generated complete data. We combine the original incomplete instances {˜x(v) ˜n } and the generated missing data {ˆx(v) ˜n } to compute the common representations {ˆz(v) ˜n } for the incomplete part:

In this way, the model can capture the hidden information of the missing data. Finally, we apply K-means on the learned common representations {{zn}Nn=1 , {ˆz˜n} ˜ N˜n=1 } to quantitatively test AIMC's performance on data clustering.

4 Experiments

We evaluate the clustering performance of AIMC on three datasets. Important statistics are summarized in Table 1 and a brief introduction of the datasets is presented below.

Reuters [Amini et al., 2009] consists of 111740 documents written in 5 languages of 6 categories represented as TFIDF vectors. We utilize documents written in English, French and German as three views. For each category, we randomly choose 500 documents. Totally 3000 documents are used. BDGP [Cai et al., 2012] contains 2500 instances about drosophila embryos of 5 categories. We utilize 1000D lateral visual vector, 500D dorsal visual vector and 79D texture feature vector as three views. We use all instances in our experiment. Youtube [Omid et al., 2013] contains 92457 instances from 31 categories, each described by 13 feature types. We sample 500 instances from each category. We select 512D vision feature, 2000D audio feature and 1000D text feature as three views.

We compare AIMC with the following baseline algorithms. Due to the missing data, we cannot directly perform the algorithms that are only applicable on complete data. Following [Hu and Chen, 2018], we first fill the missing data with the average feature values for each view, and then perform these algorithms. Best Single View (BSV) clusters on each view, and reports the best result. Concat concatenates feature vectors of different views to apply K-means. Deep Multi-view SemiNMF (DMSNMF) [Zhao et al., 2017] is an deep method for complete multi-view data clustering. Doubly Aligned Incomplete Multi-view Clustering (DAIMC) [Hu and Chen, 2018] is a weighted NMF based IMC method. Unified Embedding Alignment Framework (UEAF) [Wen et al., 2019] is the state-of-the-art IMC method with missing data inference.

As in [Hu and Chen, 2018], we randomly select ˜ N instances as incomplete data and randomly remove some views from each of them. The Missing Rate (MR) ˜ N N+ ˜ N is from 0 to 0.5. The clustering Accuracy and Normalized Mutual Information (NMI) are used to evaluate clustering performance [Shao et al., 2015]. All of the hyper-parameters of these methods are selected through grid-search.

4.1 Results

Figure 3, 4, 5 show the clustering performance of AIMC and baseline methods. First, the performance of IMC methods drop more slowly than others with the increase of missing rate. We can see that IMC methods are more effective for the IMC problem. Second, when the missing rate is 0, deep methods (AIMC, DMSNMF) outperform shallow ones. This is intuitive since the deep model could better handle the dependence and discrepancy among different views by hierarchical modeling. Thirdly, AIMC and UEAF consistently outperforms DAIMC on the two datasets, which indicates that capturing the hidden distribution of the missing data can greatly facilitate the IMC. We use t-test with significance level 0.05 to test the significance of performance difference. Results show that AIMC significantly outperforms all the baseline methods.

4.2 Analysis

In this subsection, we will analyze AIMC from two perspectives, i.e., parameter setting and convergence analysis.

The AIMC method contains two hyper-parameters: α and β. Here we explore their impact to performance on the BDGP dataset. We set missing rate as 0.2, and report the accuracy by varying α and β in the set {10-4 , 10-3 , 10-2 , 10-1 , 1, 10}. Fig 6(a) shows the results. We find a general pattern: the performance curves first go up and then go down when increasing the hyper-parameters. This indicates the discriminators and KL aligned clustering term are useful for IMC. Based on the results, we set α = 0.001, β = 0.01 in other experiments. Note that we follow the experimental methodology employed by previous works [Li et al., 2014; Zhao et al., 2016] that set hyper-parameters by looking at performance on the BDGP test set. The results for BDGP could be overly optimistic. However, the hyper-parameters of AIMC and all the baselines are tuned in exactly the same way, so it is a fair comparison. Moreover, AIMC was run with the same hyper-parameter setting on the three datasets and achieved good performance on all of them, which means AIMC is not very sensitive with respect to the hyper-parameters.

Fig 6(b) shows the curves of loss value, Accuracy and NMI against the number of epochs for AIMC. We set the missing rate as 0.5 on BDGP dataset. We find at the beginning the loss value drops and the performance increases rapidly. The optimization procedure of AIMC typically converges in around 5 epochs.

5 Conclusion

In this paper, we proposed a Adversarial Incomplete Multiview Clustering (AIMC) method for IMC with an arbitrary number of views. AIMC tries to seek a common high-level representation for incomplete multi-view data. It also tries to capture hidden information of the missing data by missing data inference via the element-wise reconstruction and the GAN. Experimental results on three real-world datasets confirmed the effectiveness of AIMC compared to state-ofthe-art incomplete multi-view clustering methods.

Acknowledgments

This research was supported by the National Natural Science Foundation of China (Grant Nos. 61672409, 61522206, 61876144, 61876145), the Major Basic Research Project of Shaanxi Province (Grant No. 2017ZDJC-31), Shaanxi Province Science Fund for Distinguished Young Scholars (Grant No. 2018JC-016), and the Fundamental Research Funds for the Central Universities (Grant Nos. JB190301, JB190305)