Salient-Residual Decoupled Multi-View Learning for Clustering

1 Introduction

In the era of high informatization, the lack of label information brings difficulty in directly applying it to the supervised learning process, while the high cost of manual annotation has driven the rapid development of weakly labeled dependency learning methods [Zhang et al., 2025b; Zhang et al., 2025a]. By integrating information from different perspectives, multi-view clustering methods aim to improve clustering robustness and discrimination compared to single-view approaches [Fang et al., 2023; Ren et al., 2025; Chen et al., 2025a]. However, the redundant information brought by multi-view data makes this become the core challenge: how to maximize the effective information among views [Liu et al., 2019; Liu et al., 2020; Zhang et al., 2023]?

∗Corresponding author.

Mainstream approaches enhance the process of learning consensus representations by explicitly establishing consistency between different views, to capture shared clustering structures [Yang and Wang, 2018; Ren et al., 2026]. Despite their effectiveness, consensus-driven methods rely on an implicit assumption: representations that cannot be aligned across views or absorbed into the consensus space are either redundant or detrimental to clustering performance [Lyu et al., 2022]. Therefore, such structures are often suppressed or discarded during optimization. However, real-world data is frequently diverse, and the differences brought by which can also reflect the target information [Wang et al., 2016]. Ignoring these differences may lead to spurious consensus, degraded cluster discrimination, and representation degradation [Liang et al., 2019].

Generally speaking, consensus information and complementary information constitute all the features of a view, and it has been proved that correctly integrating is beneficial to the clustering process [Bai et al., 2024; Dong et al., 2023]. Complementary information typically refers to viewspecific cues that enrich cluster descriptions, thereby improving the clustering quality [Liu et al., 2024; Fu et al., 2024; Ren et al., 2024]. To ensure the distinction between consensus and complementary information, in practice, the explicitly irrelevant features are often separated and removed as view noise. Intuitively, such noise is assumed to be uninformative for clustering and should not be integrated into the feature space. While, although some of the complementary information does exist in an explicit way to assist clustering, which is the reason why complementary-based methods are functional [Chen et al., 2022], parts of view information hidden in a deeper level is hard to be utilized. When such information is ignored or discarded during optimization, it may lead to irreversible information loss and degraded clustering performance [Li et al., 2019]. We believe that although some view information that regarded as noise normally may appear to be in conflict with the clustering process, it can, under appropriate constraints, serve as crucial characteristic signals for utilization [Christoudias et al., 2012]. Based on this perspective, incompatible clustering structures should be explicitly modeled rather than implicitly suppressed.

To this end, we propose a new method named SRDMVC, leveraging residual subspaces under consensus constraints for multi-view clustering. We decompose the feature space into two interacting components: (i) the salient space that directly supports clustering, and (ii) the residual subspace that is less relevant to the current clustering process. The residual subspace contains both implicit view-level information and structures that conflict with the clustering objective. Specifically, SRDMVC adopts a novel decomposition-fusion iterative optimization, which first decouples the feature space and then effectively fuses the features using attention mechanisms. We propose salient-residual decoupled multi-view learning and cross-view salient-residual fusion. Salient-residual decoupled multi-view learning mainly achieves decoupling from the complete feature space to the salient feature space and the residual feature subspace. At this stage, separate networks are used to extract salient features and residual features which are initially decomposed through a new decomposition loss, where we design a residual position encoding to make the residual features have view correlation. Then, a novel salient-uniform constraint is proposed to reinforce the role of both features in clustering, and to ensure the positive contributions of residual features to clustering process. Inspired by the attention mechanism in existing clustering approaches [Ma et al., 2023; Zhang and Zhu, 2023; Diallo et al., 2023], cross-view salient-residual fusion uses a salient-residual cross-view attention mechanism to fuse features from different views, and achieves the reconstruction. The proposed method can avoid spurious consensus caused by overemphasizing consistency, enhance clustering discrimination, and alleviate representation degradation.

Our main contributions are as follows:

• We propose salient-residual decoupled multi-view learning, which divides the feature space into salient space and residual subspace, to avoid the information loss that occurred in previous methods when decomposing the common information and complementary information.
• We design cross-view salient-residual fusion, which leverages a salient-residual cross-view attention to fuse features, enabling the effective information in residual subspace to assist the clustering process.
• Extensive experiments on multiple benchmark datasets demonstrate the superior performance of our method, effectively alleviating spurious consensus and representation degradation.

2 RelatedWork

2.1 Multi-ViewClustering

Multi-view clustering (MVC) aims to exploit multiple feature representations of the same data to reveal robust underlying structures. Existing methods can be broadly categorized according to their modeling strategies. Representative works employ low-rank constraints for subspace clustering [Zhang et al., 2015], joint sparse representation with spectral clustering [Khan et al., 2023], or efficient anchor-based graph fusion [Liu et al., 2022]. Scalable and parameter-free fusion frameworks, such as GCFAgg [Yan et al., 2023], have also been widely adopted. A commonality among these methods is their treatment of information inconsistent with the learned consensus as noise to be suppressed. DSMVC [Tang and Liu, 2022] introduces a safety mechanism to mitigate performance degradation caused by increasing views. Contrastive learning, as a powerful method for multi-view clustering, has repeatedly optimized in terms of sample construction and learning approach, and still holds considerable potential [Wang et al., 2025; He et al., 2025]. For the core challenge of extracting consensus information between views in multi-view clustering, methods such as CoMVC [Trosten et al., 2021] and MFLVC [Xu et al., 2022] maximize feature agreement between different views. In order to better utilize the complementary information among views and alleviate the phenomenon of representation degradation, granular learning is introduced into clustering to obtain better feature representations [Su et al., 2025].

2.2 Consistency Constraint

Consistency constraint-based methods explicitly enforce agreement among different views at either the representation, prediction, or clustering assignment level, aiming to learn view-invariant yet discriminative structures. Early works such as Co-regularized Spectral Clustering [Kumar et al., 2011] encourage clustering results from different views to be mutually consistent through shared regularization terms. DCCAE [Wang et al., 2015] extends canonical correlation analysis to deep networks by maximizing cross-view correlation while preserving reconstruction consistency. DMVC [Zhao et al., 2017] introduces a deep framework that aligns latent representations across views using consistency losses to reduce view discrepancy. DAMC [Li et al., 2019] employs adversarial learning to enforce distribution-level consistency between view-specific embeddings and a shared latent space. More recently, the relationship between noise and consistency has received more attention, and more methods attempt to alleviate the noise interference in clustering. Methods such as RMCNC [Sun et al., 2024] multi-view contrastive learning method with noise tolerance, exploring the non-consistency issues in multi-view clustering, while MVCAN [Xu et al., 2024] designs a multi-view iterative optimization method to enhance the robustness of the training process. CANDY [Guo et al., 2024] integrates noise information into the context and utilizes the similarity between different views to discover misclassified samples, alleviating noise interference.

3 Method

3.1 Overview

SRDMVC constructs a decomposition-fusion iterative optimization target, decoupling the feature space into a salient space and a residual subspace. We first decouples the feature space and then effectively fuses the features using attention mechanisms, to ensure the effectiveness of residual features and the interactivity among features. The overall architecture is shown in Figure 1. Given a multi-view dataset {Xi }Vi=1 with N samples, and each sample has V different views. During salient-residual decoupled multi-view learning, for each view, samples are processed by a novel network to capture salient and residual features. The we design a correlationremoved loss to decompose different features, and propose a residual position encoding to make the residual features have view correlation. Moreover, we design a salient-uniform constraint to further optimize the correlation between different representations and clustering structures. In cross-view salient-residual fusion, the decoupled features are fused by a salient-residual cross-view attention mechanism, and then fed into decoders to achieve the reconstruction. SRDMVC is trained under decomposition-fusion iterative optimization.

3.2 Salient-Residual DecoupledMulti-View Learning

Model Architecture

For the feature space of a view, common approaches are to divide it into consensus features and complementary features, often accompanied by a simple separation of noise, which may ignore hidden deep information. Our model aims to explicitly decouple each view into a salient feature space and a residual feature subspace to fully extract the information. Specifically, given a multi-view dataset with V views, let {Xi }Vi=1 denote the input features of each view. For the i-th view, an independent encoder f (i) enc(·) is employed to extract initial features, and further obtain unified high-level representations. The process can be expressed as:

where Hi ∈ RB×DH , Zi ∈ RB×D , B represents the batch size of training, and DH is the feature dimension. During the decomposition process, instead of directly enforcing consensus on Zi , we transform it into two interacting spaces through parallel transformation modules with different structures. The salient space and residual subspace are extracted through two separate MLPs:

Si ∈ RB×D is designed to preserve compact and stable structures that can provide direct information for clustering. When considering the fusion of multiple views, consensus information, due to its stability, sharing, and directness, is more likely extracted into salient space. In contrast, Ri ∈ RB×D contains more structures that cannot be explicitly helpful for clustering, which are hidden within the high-level view information and can only function under specific constraints. Furthermore, we use a linear layer f (i) cat (·) to fuse the features within the view:

where [·||·] denotes feature concatenation, and Fi ∈ RB×D represents an intermediate embedding, integrating salient and residual information.

Salient-Residual Decomposition

Considering the salient features and residual features obtained from the model, we hope that each feature in Si and Ri should be as complementary as possible, in order to capture more comprehensive information. Furthermore, since the residual features should be weakly associated with the current clustering process, there exists a risk of degeneration into unstructured or view-agnostic representations. Therefore, we set an optimization objective LSRD to address the aforementioned issues, which can be defined as:

Corr(·) denotes the covariance calculation, and Rr (·) represents a regularization method for residual features. Specifically, to ensure that the residual information in each view is encoded into distinct subspaces, we propose the Residual Position Encoding approach, which ensures that the features of each view are only relevant to the current view.

Residual Position Encoding. The main idea is to assign a unique position label to each feature in the residual subspace, based on the specific view it comes from, enabling us to separate residual features from different views. For each view i, the residual features Ri are encoded with a corresponding position label. Given a multi-view dataset containing V views and the residual feature of the i-th view Ri ∈ RB×D , each residual feature will be assigned a one-hot encoded label yi ∈ RB×V as the label of view i, which can be defined as:

where 1 is at the i-th position and 0 elsewhere, 0 and 1 are vectors of length B. The position labels are assigned based on the index of each view, such that each feature in a given view has a view-unique label. Residual features Ri are passed through a residual view discriminator fdis(·) to classify the features according to their view. The process can be written as:

ˆ yi ∈ RB×V represents the discriminator's output logits of view i. To optimize the target, we use the cross-entropy loss as the regularization method to ensure that the residual features are correctly classified according to their corresponding view, which can be expressed as:

where ˆ yi is the predicted label and yi is the true one-hot label corresponding to its view. Overall, the residual subspace is encouraged to retain the distinguishable features across views by minimizing LSRD, thereby preventing it from degenerating into single or invalid representations. It is important to note that this enhancement mechanism does not guarantee that residual features are complementary. Instead, it ensures structural effectiveness and view relevance, which plays a crucial role in the following stage.

Salient-Uniform Constraint

In this stage, we do not enforce alignment between residual features and clustering assignments, but aim to prevent the residual features from distorting clustering process. Specifically, we introduce the Salient-Uniform Constraint to achieve three primary goals: (i) The salient features should directly reflect the clustering structure; (ii) The residual features should be uncorrelated with the clustering process; (iii) The fused features should have stronger clustering capabilities. And to establish a unified clustering reference, we introduce a shared learnable clustering head g(·), which maps inputs to a soft cluster assignment. For each view i, the clustering head is applied to salient features Si , residual features Ri , and fused features Fi to make consistent evaluation across spaces. The process can be expressed as:

Among them, Qis and Qif can directly reflect the clustering structures, while Qir should maintain a low correlation with the clustering structure. In other words, the salient and fused features are expected to be highly aligned with the clustering assignments, while the residual features are designed to be less directly associated with clustering structures. In practice, we propose a loss function to achieve these targets, which is formalized by the following Salient-Uniform Constraint loss:

where DKL(·) represents the KL divergence, Pis is the squared normalization of Qis , Rs (·) denotes a regularization method for salient features Si , and ¯qi = 1B ∑Bj=1 Qir,j . The first term DKL(Qis ||Pis ) ensures that the structures can be helpful, the second term- ∑Kk=1 ¯ qik log(¯ qik ) penalizes the residual features for deviating from a uniform distribution, which does not allow for residual features to have clustering preferences, the third term ∑Vi=1 Rs (Si ) forces the strongest clustering ability for fused features.

Salient Clustering Regularization. To guarantee that the interaction between salient and residual subspaces leads to positive results, we require the clustering structures reflected by Fi should not be inferior to that of the salient representation alone, which means that residual features should make positive contributions to clustering targets after initial feature fusion. So we design a regularization method for salient features:

where Pif is the squared normalization of Qif .

3.3 Cross-View Salient-Residual Fusion

Salient-Residual Cross-View Attention

In previous steps, the feature space of the view was decoupled and optimized through decomposition loss. Furthermore, in order to better facilitate the interaction and mutual promotion among multiple views, we first propose salient-residual cross-view attention to construct a unified and view-aware representation. Traditional fusion approaches may lose intrinsic connections between different features, which is not conducive capturing the relationship between salient space and residual subspace. Inspired by the multi-view attention mechanism, we utilize a variant to adaptively aggregate features. We consider three types of representations: H, R, and F, which are organized into a query-key-value triplet to facilitate structured information aggregation. These features are respectively obtained by concatenating all the corresponding view features Hi , Ri , and Fi . Specifically, H encodes the semantic context, R captures residual structures regulated by consensus constraints, and F provides the candidate representations to be fused. Finally, these representations are input into the attention mechanism to obtain fused features, which can be formulated as:

where attention is performed independently for each head and then aggregated through concatenation and linear projection. Unlike conventional multi-view fusion strategies, which rely on fixed weighting or direct concatenation, salientresidual cross-view attention explicitly conditions the aggregation process on residual cues and view-semantic representations, while allowing the fusion process to selectively emphasize views whose residual structures are informative under the current representation context, which provides a robust foundation for reconstruction.

View-Wise Reconstruction

To keep the effectiveness of the integration across views, we employ a view-wise reconstruction loss. The reconstruction process requires the utilization of Fi , which provides regulatory information for fusion. While during the attention fusing process, the salient feature may be weakened due to the characteristics of attention and the limitations of input structures. To alleviate this, the fused feature Xatt = Attn(H,R,F) is combined with the salient feature Si to obtain the final fused representation:

Xiatt is the attention-fused feature of view i fromXatt. Subsequently, we use a view-independent decoder f (i) dec (·) to decode the fusion feature:

where ˆXi denotes the decoded data of view v. By employing independent decoders, the model avoids forcing a single one to fit heterogeneous feature distributions, which maintains flexibility across views. For the formulation of reconstruction loss, we leverage the Mean Squared Error (MSE) to accomplish, which can be defined as:

Decomposition-Fusion Iterative Optimization

In SRDMVC, salient-residual decoupled multi-view learning and cross-view salient-residual fusion are two independent iterative processes, trained to achieve the decompositionfusion iterative optimization. During the Salient-Residual Decoupled Multi-View Learning stage, we first decouple the salient and residual features, ensuring that they are learned independently for each view. Once this stage is completed, the learned weights are inherited and continue the training process in the Cross-View Salient-Residual Fusion stage, where the features from multiple views are fused for clustering. Overall, the decomposition-fusion iterative optimization can be expressed as:

During the actual training process, we utilize some hyperparameters to adjust the weights of different terms in LiSUC to optimize the learning objective. Based on this, LiSUC can be written as:

α and β are hyperparameters used to adjust the regulating intensity. By minimizing Ldecom, SRDMVC can decouple the salient space and residual subspace while maintaining positive contributions of residual features for clustering. And the optimization of Lfusion makes the cross-view feature fusion more effective.

4 Experiments

4.1 Experimental Setup

Datasets

The datasets involved in our experiments are shown in Table 1. In particular, WebKB [Sun et al., 2007] is a multiview web page dataset, where each instance is extracted from various textual and structural sources. Prokaryotic [Brbi´c et al., 2016] describes prokaryotic organisms using multiple genomic feature representations, aiming to categorize samples based on functional or taxonomic similarities. MSRCv5 [Xia et al., 2023] is a widely used image segmentation dataset where each image is described by several visual features. ACM [Jin et al., 2021] is a citation network dataset, in which the documents are represented from different relational and content-based perspectives. Wikipedia [Rasiwasia et al., 2010] is a large multi-view document dataset constructed from Wikipedia pages, where each article is described by multiple sets of different modalities of features. Multi-COIL-20 [Nene et al., 1996] are multi-view object image datasets with each photo of an object taken from multiple perspectives is regarded as a different view.

Baselines

To verify the performance of proposed SRDMVC, we compare it with eight different multi-view methods for clustering, including: DSMVC [Tang and Liu, 2022], SEM [Xu et al., 2023], GCFAgg [Yan et al., 2023], SCMVC [Wu et al., 2024], DFL-NET [Chen et al., 2025b], MS2FACMVC[Zhang et al., 2025c], and two different backbones of GFSAF (GFSAF-AE, GFSAF-DAE).

Evaluation Metrics

We evaluate the clustering performance of all comparison methods by adopting four widely-used metrics: Clustering Accuracy (ACC), Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), and Purity (PUR).

Implementation Details

In terms of training parameters, we set the learning rate to 1e-4, and the batch size for all datasets to 256, and the Adam optimizer is used to train model. In Eq.(17), α and β are respectively set to 0.1 and 0.8. For feature dimension, the value of D is set to 64, and the value of DH is set to V ×

D. All experiments are conducted on a workstation equipped with an NVIDIA RTX 2080 Ti GPU (11 GB memory).

4.2 Comparison

Table 2 shows the performance comparison between SRDMVC and baseline methods. Our method achieves the optimal or near-optimal results on all datasets. Overall, SRDMVC has average improvements of on ACC, NMI, ARI, and PUR, respectively. On view-heterogeneous and weakly aligned datasets, which are more challenging, SRDMVC also achieves outstanding performance, which demonstrates the effectiveness of the modeling architecture we perform.

4.3 Ablation Study

Influence of Loss Modules

To verify the effectiveness of the loss components of SRDMVC, we conduct experiments using different combinations of loss terms, which are shown in Table 3. L′SRD represents the LSRD without the regulation method Rr . When L′SRD is employed, the result is the highest when only one loss module is activated. Experiments with combined loss terms reveal that introducing LSUC consistently leads to a substantial performance improvement, and further introduction of Rr provides additional gains for LSRD. These observations indicate that explicitly introducing the residual subspace into the view fusion process makes positive contributions to improving clustering performance. Finally, fusing all loss functions achieves the overall best performance of SRDMVC.

Effectiveness of Residual Subspace

To investigate the contribution of residual subspace in SRDMVC, we conducted a series of ablation experiments by removing or modifying the residual representation while keeping other components unchanged, as shown in Table 4. The residual subspace is defined as the feature sets that cannot explicitly contribute to the clustering process, which might be discarded as noises in traditional approaches. Therefore, in practice, we consider the following four variants: (1) w/o Residual, where the residual subspace is completely removed and only salient space is retained; (2) Random Residual, where the residual representation is initialized with randomly sampled noise following a uniform distribution, aiming to verify whether performance improves stem merely from increased feature dimensionality; (3) Gaussian Residual, where the residual subspace is replaced by Gaussian noise of zero mean and unit variance; (4) Full Model, where the residual subspace is explicitly learned and jointly optimized with salient space. The comparison between removing the residual subspace and using the full model demonstrates the positive contribution of residual features.

(a) WebKB

(b) Multi-COIL-20

Figure 3: Visualization of the fused feature on WebKB and MultiCOIL-20.

Influence of Loss Weights

The value of LSUC is affected by α and β. In this section, we used different combinations of alpha and beta to investigate the impact of loss weights on the model performance, as shown in Figure 2. Specifically, we evaluate four values for α, i.e., {0.01, 0.05, 0.1, 0.5}, and four values for β, i.e., {0.1, 0.3, 0.5, 0.8}, on WebKB and Prokaryotic. Experimental results show that when α is set too small, the salient features of the residual subspace might not be completely "erased". Conversely, overly large weights can also degrade performance, which may indicate that the residual subspace collapses toward uninformative noise. Rs encourages that the residual subspace should be conducive to feature fusion. As β increases, this effect will be enhanced, allowing the residual subspace to make more contributions to feature fusion. Overall, we select α = 0.1 and β = 0.8 as the best combination, which achieves the optimal performance of SRDMVC.

Visualization of Final Fusion Features

We visualize the fused features to show the clustering result, as illustrated in Figure 3, which shows the competitive clustering results.

5 Conclusion

In this paper, we propose a novel framework for multi-view clustering, called SRDMVC. Compared with traditional approaches, SRDMVC introduces a salient-residual decoupled method to enhance feature fusion, which alleviates the information loss that occurs during the feature decomposition process when using traditional methods. In particular, we adopt a novel decomposition-fusion iterative optimization. We first decouple the feature space to salient space and residual subspace, leverage proper constraints to reinforce the positive contributions of both features in clustering, and ensure the effectiveness of residual features. Further, we fuse the decoupled features through a salient-residual cross-view attention mechanism, and achieve the effective reconstruction for clustering structures. As shown in the experiments, SRDMVC shows outstanding performance, demonstrating the effectiveness of our method. Future work will continue to focus on the distribution of residual features, and develop more reasonable optimization.

Acknowledgments

This work was supported in part by National Key Research and Development Program of China (No. 2024YFC2310801) and National Natural Science Foundation of China (No. 62476052).