---
title: "Deep Latent Low-Rank Fusion Network for Progressive Subspace Discovery"
authors:
  - "Zhao Zhang"
  - "Jiahuan Ren"
  - "Zheng Zhang"
  - "Guangcan Liu"
doi: "10.24963/ijcai.2020/383"
url: "https://ijcai.org/proceedings/2020/383"
pdf: "https://ijcai.org/proceedings/2020/0383.pdf"
venue: "Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence"
year: 2020
pages: "2762-2768"
track: "Main track"
keywords:
  - "Machine Learning: Unsupervised Learning"
  - "Machine Learning: Clustering"
  - "Machine Learning: Deep Learning"
---

# Deep Latent Low-Rank Fusion Network for Progressive Subspace Discovery

Zhao Zhang, Jiahuan Ren, Zheng Zhang, Guangcan Liu

*Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence*, pages 2762-2768. DOI: [10.24963/ijcai.2020/383](https://doi.org/10.24963/ijcai.2020/383)

## Abstract

Low-rank representation is powerful for recover-ing and clustering the subspace structures, but it cannot obtain deep hierarchical information due to the single-layer mode. In this paper, we present a new and effective strategy to extend the sin-gle-layer latent low-rank models into multi-ple-layers, and propose a new and progressive Deep Latent Low-Rank Fusion Network (DLRF-Net) to uncover deep features and struc-tures embedded in input data. The basic idea of DLRF-Net is to refine features progressively from the previous layers by fusing the subspaces in each layer, which can potentially obtain accurate fea-tures and subspaces for representation. To learn deep information, DLRF-Net inputs shallow fea-tures of the last layers into subsequent layers. Then, it recovers the deeper features and hierar-chical information by congregating the projective subspaces and clustering subspaces respectively in each layer. Thus, one can learn hierarchical sub-spaces, remove noise and discover the underlying clean subspaces. Note that most existing latent low-rank coding models can be extended to multi-layers using DLRF-Net. Extensive results show that our network can deliver enhanced perfor-mance over other related frameworks.

## Full text

## 1 Introduction

Representation learning is always a fundamental problem to obtain the underlying explanatory factors and features for the subsequent data classification or clustering tasks [Chen et al., 2019 and 2020] [Zhang et al., 2016 and 2019]. Representation learning is still challenging in reality due to the complexity and diversity of data [Lu et al., 2019] [Su et al., 2019] [Acharya et al., 2019] [Ding et al., 2018].

Since most real data can be characterized using low-rank subspaces, low-rank coding methods can recover the underlying subspaces and obtain notable features [Liu et al., 2019] [Ren et al., 2019]. Low-Rank Representation (LRR) [Liu et al., 2013] is one of the most classical algorithms to discover multi-subspaces, but it is essentially a transductive method failing to handle new data efficiently. To address the out-ofsample issue, Inductive Robust Principal Component Analysis (IRPCA) [Bao et al., 2012] was recently proposed seeking a low-rank projection to map samples into underlying subspaces. To enable a solution for subspace segmentation and feature extraction, Latent LRR (LatLRR) [Liu et al., 2011], was proposed, which decomposed data into a principal feature part, a salient feature part and a sparse error. Although LatLRR resolves the insufficient sampling issue and obtains enhanced performance over LRR, it still suffers from a high computational cost due to using Nuclear-norm to approximate the rank function to constrain the subspaces, while the computation of the Nuclear-norm needs the time-consuming Singular Value Decomposition of matrices at each iteration, especially for large-scale datasets. To improve the efficiency, a Frobenius-norm based LatLRR (FLLRR) [Yu et al., 2018] was proposed, which approximates the rank function using Frobenius-norm. But the Frobenius-norm is sensitive to noise and outliers, which may produce inaccurate representations.

It is noteworthy that the above algorithms have a common drawback, i.e., they are "shallow" models using single-layer structures. As a result, they cannot obtain deep information and subspaces. But due to the strong representation ability of the deep neural networks [Kim et al., 2019], deep low-rank coding models equipped with carefully designed hierarchical structures should be able to obtain the enhanced performance. In fact, researchers have also designed some deep low-rank coding models, such as Weakly-supervised Deep Nonnegative Low-rank Model (WDNL) [Li et al., 2017a], which finds the intrinsic relations between images and tags by removing noise or irrelevant tags, but it is unclear how to handle images directly and the results are usually incomplete. Another deep model is Deep Low-Rank Subspace Ensemble (DLRSE) [Xue et al., 2019], where the Frobenius-norm is used as a low-rank constraint. DLRSE uses the deep matrix factorization to learn the diverse hierarchical structures and obtains the low-rank representations from extracted factors. But DLRSE is originally proposed for multi-view clustering, which clearly differs from our task. To cluster big data effectively, a Projective Low-rank Sub-space Clustering via Learning Deep Encoder (PLrSC) [Li et al., 2017] has been recently proposed.

Figure 1: Comparison between the mechanisms of traditional latent low-rank coding methods (a) and our DLRF-Net (b).

![Image](https://ijcai.org/proceedings/2020/fulltext/383/image_000000_c34014484a13571fdea0685d08a42ee548366e07b0bec38e7c3bd9d7fc4547d2.png)

PLrSC uses a small database randomly sampled from a big dataset and train a deep encoder, and then a deep encoder is used to compute the low-rank representations of all samples. However, it will be not easy to determine the sampling frequency based on different real datasets in practice.

In this paper, we mainly propose a general and progressive deep low-rank fusion network that can unfold existing latent low-rank methods into multilayers for the hierarchical representation and deep subspace discovery. The major contributions of this paper are summarized as follows:

(1) Technically, a simple yet effective progressive deep representation learning model termed Deep Latent Low-Rank Fusion Network (DLRF-Net) is derived. DLRF-Net can learn deep hierarchical information and subspaces from input data. The advantage of this practice is that multiple layer low-rank coding structures can deliver rich and useful hidden hierarchical information that has a great potential in learning more powerful deep representation and subspace structures. To be specific, DLRF-Net in each layer aims to refine the features and subspaces progressively from previous layers by fusion, i.e., it recovers deep hierarchical information by respectively congregating the projective and clustering subspaces in each layer to produce accurate results. We compare the traditional single-layer low-rank models with DLRF-Net in Figure 1.

- (2) The network of DLRF-Net is simple, general and easy to extend. Specifically, many existing latent low-rank coding models such as LatLRR and FLLRR can be easily extended from single-layer to multiple-layers using DLRF-Net. In this paper, we mainly explain our basic idea rather than deriving a complex formulation, two simple deep network models are constructed based on embedding LatLRR and FLLRR into DLRF-Net as examples for the multi-layer low-rank coding, which we call Nuclear-norm based DLRF-Net (nDLRF-Net) and Frobenius-norm based fast DLRF-Net (fDLRF-Net).

(3) Extensive simulations on public databases demonstrate that both nDLRF-Net and fDLRF-Net can deliver enhanced performance than the related single-layer models. That is, the multilayer idea of DLRF-Net is feasible and effective.

## 2 Related Work

We describe the closely-related low-rank coding algorithms.

## 2.1 LatLRR and FLLRR

Given a data matrix   1 2 , , n N N X x x x    , where n i x  is a sample represented using an n-dimensional vector and N

is the number of samples, then LatLRR improves LRR using the unobserved hidden data XH to extend the dictionary and overcome the insufficient data sampling issue. Specifically, LatLRR considers the following coding formulation:

<!-- formula-not-decoded -->

where   rank is rank function and XO is the observed data matrix. Supposing that XO and XH are sampled from the same collection of low-rank subspaces, by using the Nuclear-norm to approximate the rank function and using sparse L1-norm on error term E, LatLRR recovers the hidden effects by

<!-- formula-not-decoded -->

where * Z is the Nuclear-norm of Z [Liu et al., 2013] [Xie et al., 2017], i.e., the sum of its singular values, XZ and LX are principal features and salient features respectively, and  is a positive scalar. Since LatLRR uses the Nuclear-norm constraints on Z and L, the SVD process is involved, which is time-consuming. Note that Frobenius-norm F can also be used as the convex surrogate of the rank function [Yu et al., 2018]. Besides, the optimization of Frobenius-norm is very efficient. The objective function of FLLRR is defined as

<!-- formula-not-decoded -->

from which multi-subspace structures can be recovered by Z and the notable features can be extracted using L.

## 2.2 Projective Low-rank Subspace Clustering via Learning Deep Encoder (PLrSC)

Assuming that ' 1 , ,        i k n N Y Y Y Y is a big dataset and over-sufficiently drawn from a union of k subspaces, where N is the number of samples in all subspaces. PLrSC assumes that 1 , , i k X X X X      n N   is a small dataset sampled randomly from Y and X is still sufficient. First, PLrSC learns a non-iterative deep encoder   ; de f X  , where is the learning parameter to approximate low-rank representations. Then, the deep encoder is utilized to obtain the low-rank codes for replacing the costly non-linear Singular Value Thresholding (SVT) [Cai et al., 2010] operations. Thus, the predictive lowrank decomposition of PLrSC can be written as follows:

<!-- formula-not-decoded -->

where n N E   denotes a sparse error,  is the regularization parameter for E,  is a control parameter for approximation term   2 ; de F Z f X   .         2 ; = M i de f X g W g W g W X  is a deep encoder with M layers, where   g is an activation function (e.g., sigmoid or ReLU).  is a learning parameter set, where     2 1 1 2 , , , , M M l l l l M W W        . i l denotes the number of the units in the i-th layer ( 1 l d  and M l N  ). Then, PLrSC employs the alternating direction algorithm (ADM) [Liu et al., 2013] and a gradient descent algorithm (GD) [Li et al., 2015] to optimize the above problem. Finally, PLrSC applies the landmark-based spectral clustering (LSC) algorithm to cluster the big dataset Y [Li et al., 2017].

. . .

Figure 2: The framework and flow-chart of our proposed DLRF-Net algorithm.

![Image](https://ijcai.org/proceedings/2020/fulltext/383/image_000001_b6985132f2acb5163fa8b1d88a96fb5fa071861126a4df660a2c9067e94fdf6c.png)

## 3 Proposed DLRF-Net

## 3.1 Objective Function

DLRF-Net designs a hierarchical and progressive approach, i.e., the representation in the uncovered subspaces are learnt layer by layer. That is, the deep principal features XZ0Z1…Zl-1 and deep salient features Ll-1…L1L0X from the (l-1)-th layer are fed into the l-th layer, which are further decomposed into a deep principal feature part, a deep salient feature part and a deep sparse error. The whole framework of our DLRF-Net is shown in Figure 2. Assuming that DLRF-Net has M layers, the decomposition process of our DLRF-Net framework in the l-th layer can be presented as follows:

<!-- formula-not-decoded -->

where 1 l Z and 2 l Z are the deep coefficient matrices, 1 l L and 2 l L denote the deep projection matrices that are learned from XZ0Z1…Zl-1 and Ll-1…L1L0X in the l-th layer, respectively. Note that Z0 and L0 are included to simplify the descriptions, which are set to the identity matrices, i.e., the input of the first layer is the original data. It should be noted that for the optimization in the l-th layer, Z0, Z1, …, Zl-1, L0, L1, …, Ll-1 are known variables that are updated in the last layer. As such, intuitively from the multilayer leaning process, deep principal features XZ0Z1…Zl-1 and deep salient features Ll-1…L1L0X are learnt progressively from different layers, i.e., extracting fine-grained features from layer to layer.

Finally, deep principal features XZ0Z1…Zl and salient features Ll…L1L0X in the l-th layer can be obtained as

<!-- formula-not-decoded -->

The above subspace fusion operation can potentially make the learned representations more accurate by fusing feature information from deep principal and salient features in previous layers. The above averaging operation can also prevent the feature information loss and balance the information from deep principal and salient features in each layer. As such, we have the following model for DLRF-Net in the l-th layer:

<!-- formula-not-decoded -->

where p is the matrix p-norm, which can be Nuclear-norm or squared Frobenius-norm. l  is a positive tunable parameter that replies on the noise level of data [Liu et al., 2011]. We name the Nuclear-norm based DLRF-Net as nDLRF-Net and name the squared Frobenius-norm based DLRF-Net as fDLRF-Net. The objective function of fDLRF-Net can then be defined as follows for deep subspace discovery:

<!-- formula-not-decoded -->

Next, we show the optimization procedures of DLRF-Net.

## 4 Optimization

We mainly describe the optimization of fDLRF-Net in detail, as the optimization of nDLRF-Net is similar. Since Zl, Pl and El depend on each other, we update them alternately. We use the inexact Augmented Lagrange Multiplier (Inexact ALM) algorithm [Lin et al., 2009] for efficiency.

Algorithm 1 Solving Eq.(9) by Inexact ALM (l-th layer) Inputs: Reconstructed data 1 l A  , tunable parameters , . l l   Initialization:           0 0 0 0 0 1 2 1 2 1 0, 0, 0, 0, 0, 0       l l l l l t Z Z L L E ,       0 0 0 2 1 2 6 0 6 7 max 0, 0, 0, 10 , =10 , =1.12, 10            l l l E Y Y .

## While not converged do

1. Update the coefficients sub-matrices   1 1  t l Z and   1 2  t l Z by using Eq.(12-1e), and obtain       1 1 1 1 2 2      t t t l l l Z Z Z ;
3. Update the sparse errors   1 1  t l E and   1 2  t l E by Eq.(16-17);
2. Update the projection sub-matrices   1 1  t l L and   1 2  t l L by using Eq.(14-15), and obtain       1 1 1 1 2 2      t t t l l l L L L ;
4. Update the Lagrange multipliers   1 1  t l Y and   1 2  t l Y ;
5. Update the parameter l by   1 max =min ,     t t l l ;
6. Check for convergence: Suppose  1 1 1 1 1 1 ,        l l l l l l P P Z L P E  2 2 2 1 1 1          l l l l l l S S Z L S E , stop; else 1   t t . End while

Output: * 1 * 1 , .     t t l l l l Z Z P P

To simplify the descriptions of optimization, we train the model layer by layer. To learn features in the l-th layer (l=1, 2,…, M), the target function can be defined as

<!-- formula-not-decoded -->

Denote by 1 0 1 ...    l l P XZ Z and 1 1 0 ... l l S L L X   = two auxiliary matrices, the Lagrange function of Eq.(9) can be obtained as

<!-- formula-not-decoded -->

where 1 l Y and 2 l Y are Lagrange multipliers, and l denotes a positive parameter. Then, fDLRF-Net updates the variables by solving . Note that the optimization procedures of our fDLRF-Net in the l-th layer can then be detailed as follows:

Fix others, update Zl: For the optimization of l Z , we need to solve 1 l Z and 2 l Z . By removing the irrelevant terms, we can update 1 l Z and 2 l Z by the following reduced problem:

<!-- formula-not-decoded -->

where 1 1 1 1 1 l l l l l P L P E        and 2 2 1 1 1 l l l l l S L S E        . We first show the optimization of 1 l Z . By taking the derivative of   1 2 ,  l l Z Z w.r.t. 1 l Z and zeroing the derivative, we can infer the coefficients matrix 1 l Z at the (t+1)-th iteration as follows:

<!-- formula-not-decoded -->

where     1 1 1 1 1        t t t l l l l l P L P E . Similar to the optimization of 1 l Z , we can infer   1 2 t l Z  in the (t+1)-th iteration as

<!-- formula-not-decoded -->

where     2 2 1 1 1        t t t l l l l l S L S E . After optimizing the   1 1  t l Z and   1 2  t l Z , we can obtain       1 1 1 1 2 2      t t t l l l Z Z Z .

Fix others, update Ll: By removing the irrelevant terms from , taking the derivatives of   1 2 ,  l l L L w.r.t 1 l L and 2 l L , , and zeroing the derivatives, we can similarly obtain

<!-- formula-not-decoded -->

<!-- formula-not-decoded -->

where     1 1 1 1 1 1 1 t t t l l l l l P P Z E        + and     1 2 2 1 1 1 1         t l l l l l t t S S Z E . After optimizing the projection matrices   1 1  t l L and   1 2  t l L , we can obtain       1 1 1 1 2 2      t t t l l l L L L .

Fix others, update 1 l E and 2 l E : By taking the derivative of Lagrange function w.r.t. 1 l E and 2 l E respectively and zeroing the derivatives, we can infer 1 l E and 2 l E as

<!-- formula-not-decoded -->

<!-- formula-not-decoded -->

which can be easily solved by the shrinkage operator [Lin et al., 2009], where 1 l and 2 l are auxiliary matrices defined as       1 1 1 1 1 1 1 1 t t t t l l l l l l l P Z L P Y         + and   1 2 2 1      t l l l S Z     1 2 2 1    + t t t l l l l L S Y . For complete presentation of our model, we summarized the optimization procedures of solving the sub-problem of Eq.(9) in the l-th layer in Algorithm 1.

## 5 Discussion

## 5.1 Relationship Analysis

We mainly discuss the relations of our DLRF-Net to LatLRR and FLLRR. To facilitate the analysis, we consider the special case that l=1.We first express this special case as

<!-- formula-not-decoded -->

Since Z0 and L0 are initialized to the identity matrices in the optimization, the two constraints are the same. As such, it is clear that when we use the Frobenius-norm to constrain the matrices Z1 and L1, the problem identifies FLLRR; while we use the Nuclear-norm as constraints, the resulting problem is identical to LatLRR. That is, both FLLRR and LatLRR are the special causes of our DLRF-Net framework.

## 5.2 Computational Time Complexity

We analyze the complexity of each layer of our deep models. For fDLRF-Net, SVD is not used and the major computation is updating the matrices Zl and Ll. Thus, the time complexity of Algorithm 1 is equal to that of FLLRR. Thus, it is easy to infer that the total time complexity of fDLRF-Net is M times that of each layer, where M is the number of layers, which is usually a small value. For nDLRF-Net, the time complexity of each layer is the same as that of regular LatLRR and the total time complexity is M times that of each layer.

Table 1: Descriptions of used image datasets.

| Dataset Name   |   # Samples |   # Dim |   # Classes |
|----------------|-------------|---------|-------------|
| CMU PIE        |       11554 |    1024 |          68 |
| UMIST face     |        1012 |    1024 |          20 |
| Fashion MNIST  |       70000 |     784 |          10 |

## 6 Experimental Results and Analysis

We conduct experiments to evaluate the effectiveness of our fDLRF-Net and nDLRF-Net, and show the comparison results with other related methods, including FLLRR, LatLRR, LRR, Robust LatLRR (rLatLRR) [Zhang et al., 2014b], Laplacian Regularized LRR (rLRR) [Zhang et al., 2014a], Sa-LatLRR [Wang et al., 2018] and PLrSC [Li et al., 2017b]. Three real image databases are involved, including two face datasets (i.e., CMU PIE [Sim et al., 2003], UMIST [Graham et al., 1998]) and the Fashion MNIST database [Xiao et al., 2017]. The details of used databases are described in Table 1. We follow the common procedure to resize each face image into 32×32 pixels and images of the Fashion MNIST dataset are resized into 28×28 pixels. We perform all experiments on a PC with Intel (R) Core (TM) i7-7700 CPU @ 3.6 GHz 8G.

## 6.1 Visual Image Analysis by Visualization

Visualization of coefficient matrix Z. To represent data appropriately, Z should have a block-diagonal structure. Each block denotes the coefficients for certain subject so that each sample can be reconstructed by the samples of one class as much as possible. We follow [Liu et al., 2011] to construct 10 independent subspaces  10 1 i i S  and apply this artificial data for DLRF-Net. The visualization of Z in the first four layers are illustrated in Figure 3. We see that all coefficient matrices Z have block-diagonal structures. But compared with the 1-th layer, the results of other layers have less noise and wrong inter-class connections. It can also be found that the subspace structures of Z are improved progressively, i.e., the learned structures from the 3-rd and 4-th layers are better that from the 2-nd layer. But the difference of the structures of Zs in the 3-rd and 4-th layers is small, i.e., our DLRF-Net can remove noise contained in features and recovery the subspaces of Z by using small number of layers. That is, the structures of the coefficient matrix Z will not become better, even though we use more layers involving high computational cost. In all the simulations, the results of fDLRF-Net and nDLRF-Net in the 1-th layer corresponds to FLLRR and LatLRR, respectively.

Visualization of the recovered features XZ. We evaluate fDLRF-Net and nDLRF-Net by visualizing recovered deep features XZ. Given a data matrix X, DLRF-Net decomposes it into principal features XZ, salient features LX and a sparse error E in each layer. CMU PIE face dataset is used. This face image database contains 68 persons with 41368 images under varying poses, illuminations and facial expressions. 170 near frontal images per person are employed for CMU PIE, which contains five near frontal poses (C05, C07, C09, C27, and C29). To evaluate the robustness properties, random Gaussian noise with variance 500 is included into the image data. Some original images, noisy images and recovered principal features in the first three layers are shown in Figure 4. We see find that fDLRF-Net and nDLRF-Net can effectively remove the shadow and noise in images in a progressive way, compared with the recovered results of FLLRR and LatLRR.

Figure 3: Visualization of the representation coefficient matrices Z of our fDLRF-Net (a-d) and nDLRF-Net(e-h).

![Image](https://ijcai.org/proceedings/2020/fulltext/383/image_000002_62c8742cb262995464843fc43d200e164630568274d8582b51a44e981d2b30aa.png)

Figure 4: Recovered principal features on CMU PIE face database (Left: fDLRF-Net; Right: nDLRF-Net).

![Image](https://ijcai.org/proceedings/2020/fulltext/383/image_000003_030745914290b0cb3476a970c67874adc2e029472e56530aa703b74514edb6e6.png)

## 6.2 Quantitative Clustering Evaluations

We compare each model for clustering images. UMIST and Fashion MNIST are evaluated. UMIST has 1012 images from 20 different individuals. Fashion-MNIST has 10 classes and 70000 unique products. In this study on Fashion-MNIST, we choose 1000 samples per class, i.e., totally 10000 samples. To evaluate the performance, we follow the common procedures and use the coefficient matrix Z* of each method to construct the weights by   * * 2 T W Z Z   and then use the Normalized Cuts (NCut) [Shi et al., 2000] for clustering. For PLrSC and DLRF-Net, we use the coefficients from the final layer. For each number K of clusters, we choose K categories randomly and the results are averaged over 30 initializations.

The clustering accuracy (AC) [Cai et al., 2017] is used as the quantitative metric. The values of AC on evaluated databases are shown in Table 2. We see that: (1) the clustering accuracy of each method goes down as the number of categories increases, since clustering more data is difficult than clustering less; (2) Our fDLRF-Net and nDLRF-Net deliver higher ACs than the other competitors, especially for FLLRR and LatLRR, implying that DLRF-Net can learn more effective representations by mining deep information.

Table 2: Numerical clustering evaluation results on the UMIST and Fashion MNIST databases.

| Method               | Clustering Accuracy (%) on UMIST - K=2   | Clustering Accuracy (%) on UMIST - K=4   | Clustering Accuracy (%) on UMIST - K=6   | Clustering Accuracy (%) on UMIST - K=8   | Clustering Accuracy (%) on Fashion MNIST - K=2   | Clustering Accuracy (%) on Fashion MNIST - K=4   | Clustering Accuracy (%) on Fashion MNIST - K=6   | Clustering Accuracy (%) on Fashion MNIST - K=8   |
|----------------------|------------------------------------------|------------------------------------------|------------------------------------------|------------------------------------------|--------------------------------------------------|--------------------------------------------------|--------------------------------------------------|--------------------------------------------------|
| LRR                  | 78.57±15.67                              | 72.08±14.57                              | 66.67±4.97                               | 62.57±9.56                               | 85.21±4.37                                       | 62.27±9.11                                       | 51.83±1.99                                       | 45.75±3.24                                       |
| rLatLRR              | 86.25±15.06                              | 79.44±9.94                               | 69.67±7.40                               | 67.51±6.56                               | 89.55±13.83                                      | 65.50±9.01                                       | 63.33±8.47                                       | 56.75±4.83                                       |
| SA-LatLRR            | 84.57±14.30                              | 78.35±12.26                              | 72.08±7.40                               | 67.52±9.29                               | 91.07±11.74                                      | 68.74±4.28                                       | 63.54±11.91                                      | 56.12±6.41                                       |
| rLRR                 | 82.35±14.41                              | 78.52±6.48                               | 70.67±9.49                               | 66.57±7.52                               | 89.48±14.49                                      | 67.48±8.74                                       | 62.17±7.05                                       | 56.87±5.47                                       |
| PLrSC                | 87.89±8.74                               | 79.89±6.77                               | 71.11±10.27                              | 67.79±7.38                               | 91.42±5.39                                       | 72.06±6.47                                       | 64.43±8.27                                       | 57.61±6.75                                       |
| FLLRR                | 84.16±17.37                              | 79.25±8.98                               | 71.56±8.18                               | 66.29±7.70                               | 90.52±10.60                                      | 71.58±10.31                                      | 62.67±9.51                                       | 56.33±5.78                                       |
| fDLRF-Net (2 layers) | 88.17±16.58                              | 80.42±10.61                              | 73.83±8.20                               | 69.83±7.96                               | 91.67±10.69                                      | 73.50±11.06                                      | 64.58±9.92                                       | 58.38±4.87                                       |
| fDLRF-Net (3 layers) | 88.83±16.64                              | 81.08±10.66                              | 76.44±8.19                               | 70.88±8.62                               | 92.67±9.07                                       | 73.75±11.51                                      | 64.61±9.24                                       | 59.17±5.84                                       |
| fDLRF-Net (4 layers) | 90.16±16.49                              | 77.92±11.14                              | 71.50±8.70                               | 66.67±8.30                               | 93.17±8.76                                       | 72.75±11.51                                      | 64.67±10.03                                      | 57.87±5.61                                       |
| fDLRF-Net (5 layers) | 88.83±16.05                              | 75.0±13.47                               | 68.06±8.98                               | 65.08±7.40                               | 92.33±9.54                                       | 70.66±11.91                                      | 60.78±8.70                                       | 55.96±5.48                                       |
| LatLRR               | 81.50±9.73                               | 77.25±15.02                              | 69.17±10.77                              | 67.00±7.58                               | 90.17±13.03                                      | 70.25±12.91                                      | 65.66±8.39                                       | 59.83±7.68                                       |
| nDLRF-Net (2 layers) | 82.52±9.14                               | 79.52±12.41                              | 73.33±7.54                               | 67.38±9.98                               | 92.17±10.64                                      | 71.31±11.43                                      | 67.00±8.31                                       | 61.52±7.37                                       |
| nDLRF-Net (3 layers) | 85.05±12.35                              | 80.75±11.74                              | 76.59±12.70                              | 71.37±6.07                               | 93.33±8.54                                       | 73.03±11.58                                      | 67.60±9.16                                       | 61.56±6.68                                       |
| nDLRF-Net (4 layers) | 94.00±7.75                               | 76.25±12.20                              | 70.16±10.33                              | 65.00±3.46                               | 89.33±14.29                                      | 73.00±12.27                                      | 66.06±9.15                                       | 59.58±6.25                                       |
| nDLRF-Net (5 layers) | 90.20±7.09                               | 76.71±15.48                              | 68.83±14.03                              | 64.63±6.48                               | 87.33±15.52                                      | 72.11±12.39                                      | 63.50±8.18                                       | 57.13±5.79                                       |

Figure 5: Clustering performance vs. varying variance on the UMIST (a-b) and Fashion MNIST (d-e) database.

![Image](https://ijcai.org/proceedings/2020/fulltext/383/image_000004_c2970c3d6ce3a7bb009c2df7e89460e225134e4c3e313437f4b90adf08b01495.png)

## 6.3 Noisy Image Clustering Against Corruptions

We investigate the robustness properties against noisy cases that images are corrupted. UMIST and Fashion MNIST are used. Random Gaussian noise with different variance (100, 200, …, 500) is added to examine the robustness. For each setting, we average the result over 30 random initialization for NCut. We set the number K of clusters as 5. Some noisy images and clustering accuracy are shown in Figure 5, where the clustering results of five layers of our fDLRF-Net and nDLRF-Net are described. We see clearly that: (1) generally speaking the clustering accuracy of each method goes down with the increasing level of noise, as clustering data of high noise level is more difficult than clustering that of low noise level; (2) the best records are usually obtained in the 2-nd layer and the 3-rd layer, by comparing with the other cases.

## 6.4 Investigation of Parameters

UMIST database is used. DLRF-Net has one parameter  , so we can select the most important one by a linear search from   8 6 6 8 10 ,10 ,...,10 ,10   . We set the number K of clusters to 5 and show the analysis results of the first three layers in Figure 6. For each layer, we select the best parameter, and then we fix it to learn deeper features of next layer. We find that  usually becomes a lager one with increasing number of layers, which is easy to understand. Since DLRF-Net recovers the subspaces progressively and learnt susbapces become clean layer by layer. Note that similar observations can be found from the other datasets, but the results will be not presented.

Figure 6: Parameter sensitivity analysis on the UMIST face database.

![Image](https://ijcai.org/proceedings/2020/fulltext/383/image_000005_8037431a02157e6c68754a3801fda416bebd1836b551eb86e8c035b566ae804f.png)

## 7 Conclusion

We proposed a new progressive deep latent low-rank fusion network to uncover deep hidden features and deep clustering structures. DLRF-Net discovers the subspaces by refining the principal and salient features from previous layers progressively and then fusing the subspaces. Specifically, DLRF-Net recovers hierarchical features by congregating the projective subspace and subspaces in each layer. Most existing latent low-rank coding models can also be easily extended to multilayer scenario for learning deep features. In future, more effective deep low-rank fusion strategies will be explored.

## Acknowledgments

This work is supported by the NSFC (61672365, 61806035 and U1936217) and the Fundamental Research Funds for the Central Universities of China (JZ2019HGPA0102), and New Generation AI Major Project of Ministry of Science and Technology of China (2018AAA0100601). Zhao Zhang and Zheng Zhang are the corresponding authors of this paper.

## References

1. [Acharya et al., 2019] Anish Acharya, Rahul Goel, Angeliki Metallinou, Inderjit S Dhillon. Online Embedding Com- pression for Text Classification Using Low Rank Matrix Factorization. In: AAAI, Honolulu, Hawaii, 2019.
2. [Bao et al., 2012] Bingkun Bao, Guangcan Liu, Changsheng Xu, Shuicheng Yan. Inductive robust principal compo- nent analysis. IEEE TIP, 21(8):3794-3800, 2012.
3. [Cai et al., 2010] Jianfeng Cai, Emmanuel J. Cand`es, and Zuowei Shen. A singular value thresholding algorithm for matrix completion. SIAM Journal on optimization, 20(4): 1956-1982. 2010.
4. [Cai et al., 2017] Deng Cai, Xiaofei He, Jiawei Han. Docu- ment clustering using locality preserving indexing. IEEE TKDE, 17(12): 1624-1637, 2005.
5. [Chen et al., 2019] Xi Chen, Jie Li, Yun Song, Feng Li, Jianjun Chen, Kun Yang. Low-Rank Tensor Comple- tion for Image and Video Recovery via Capped Nuclear Norm. IEEE Access, 7:112142-112153, 2019.
6. [Chen et al., 2020] Yongyong Chen, Xiaolin Xiao, Yicong Zhou. Low-Rank Quaternion Approximation for Color Image Processing. IEEE TIP, 29:1426-1439, 2020.
7. [Ding et al., 2018] Zhengming Ding, Yun Fu. Deep Transfer Low-Rank Coding for Cross-Domain Learning. IEEE TNNLS, 30(6):1768-1779, 2018.
8. [Graham et al., 1998] Daniel B. Graham, Nigel M. Allinson. Characterizing virtual eigensignatures for general pur- pose face recognition. In: NATO ASI Series F, Computer and Systems Sciences, Berlin, Heidelberg, 1998.
9. [Kim et al., 2019] Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, Junmo Kim. Learning Not to Learn: Training Deep Neural Networks With Biased Data. In: IEEE CVPR, pp.9012-9020, 2019.
10. [Li et al., 2015] Jun Li, Heyou Chang, Jian Yang. Sparse deep stacking network for image classification. In: AAAI, Hyatt Regency in Austin, Texas, USA, pp.3804-3810, 2015.
11. [Li et al., 2017a] Zechao Li, Jinhui Tang. Weakly-supervised deep nonnegative low-rank model for social image tag refinement and assignment. In: AAAI, San Francisco, California, USA, pp.4154-4160, 2017.
12. [Li et al., 2017b] Jun Li, Hongfu Liu, Handong Zhao, Yun Fu. Projective low-rank subspace clustering via learning deep encoder. In: IJCAI, Melbourne, Australia, pp.2145- 2151, 2017.
13. [Lin et al., 2009] Zhouchen Lin, Minming Chen, Leqin Wu, Yi Ma. The augmented Lagrange multiplier method for exact recovery of corrupted low-rank matrices. Tech. Tep., 2009.
14. [Liu et al., 2011] Guangcan Liu and Shuicheng Yan. Latent low-rank representa-tion for subspace segmentation and feature extraction. In: ICCV, Barcelona, Spain, 2011.
15. [Liu et al., 2013] Guangcan Liu, Zhouchen Lin, Shuicheng Yan, Ju Sun, Yong Yu, Yi Ma. Robust Recovery of Subspace Structures by Low-Rank Representation. IEEE TPAMI, 35(1):171-627, 2013.
16. [Liu et al., 2019] Guangcan Liu, Zhao Zhang, Qingshan Liu, Hongkai Xiong. Robust Subspace Clustering with Compressed Data. IEEE TIP, 28(10): 5161-5170, 2019.
17. [Lu et al., 2019] Canyi Lu, Xi Peng and Yunchao Wei. Low-Rank Tensor Completion With a New Tensor Nu- clear Norm Induced by Invertible Linear Transforms. In: IEEE CVPR, Long Beach, USA, pp.5996-6004, 2019.
18. [Ren et al., 2019] Jiahuan Ren, Zhao Zhang, Sheng Li, Yang Wang, Guangcan Liu, Shuicheng Yan and Meng Wang. Learning Hybrid Representation by Robust Dictionary Learning in Factorized Compressed Space. IEEE TIP, 29(1): 3941-3956, 2020.
19. [Shi et al., 2000] Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE TPAMI, 22(8): 888- 905, 2000.
20. [Sim et al., 2003] Terence Sim, Simon Baker, Maan Bsat. The CMU pose, illuminlation, and expression database. IEEE TPAMI, 25(12): 1615-1618, 2003.
21. [Su et al., 2019] Fang Su, Haiyang Shang, Jingyan Wang. Low-Rank Deep Convolutional Neural Network for Multitask Learning. Computational Intelligence and Neuroscience, 7410701:1-7410701:10, 2019.
22. [Wang et al., 2018] Lei Wang, Zhao Zhang, Sheng Li, Guangcan Liu, Chenping Hou and Jie Qin. Similari- ty-Adaptive Latent Low-Rank Representation for Robust Data Representation. In: PRICAI, Nanjing, China, 2018.
23. [Xiao et al., 2017] Han Xiao, Kashif Rasul, Roland Vollgraf. Fashion-Mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747v2, 2017.
24. [Xie et al., 2017] Jianchun Xie, Jian Yang, Jianjun Qian, Ying Tai, Hengmin M Zhang. Robust Nuclear Norm- Based Matrix Regression With Applications to Robust Face Recognition. IEEE TIP, 26(5):2286-2295, 2017.
25. [Xue et al., 2019] Zhe Xue, Junping Du, Dawei Du, Siwei Lyu. Deep low-rank subspace ensemble for multi-view clustering. Information Sciences, 482: 210-227, 2019.
26. [Yu et al., 2018] Yu Song, Yiquan Wu. Subspace clustering based on latent low rank representation with Frobenius- norm. Neurocomputing, 275:2479-2489, 2018.
27. [Zhang et al., 2014a] Zhao Zhang. Shuicheng Yan, Mingbo Zhao. Similarity preserving low-rank representation for enhanced data representation and effective subspace learning. Neural Networks, 53: 81-94, 2014.
28. [Zhang et al., 2014b] Hongyang Zhang, Zhouchen Lin, Chao Zhang, Junbin Gao. Robust latent low rank representation for subspace clustering. Neurocomputing, 145:369-373, 2014.
29. [Zhang et al., 2016] Zhao Zhang, Fanzhang Li, Mingbo Zhao, Li Zhang, Shuicheng Yan. Joint Low-rank and Sparse Principal Feature Coding for Enhanced Robust Repre- sentation and Visual Classification. IEEE TIP, 25(6): 2429-2443, June 2016.
30. [Zhang et al., 2019] Zhao Zhang, Jiahuan Ren, Sheng Li, Richang Hong, Zhengjun Zha, Meng Wang. Robust subspace discovery by block-diagonal adaptive locali- ty-constrained representation. In: ACM MM, 2019.