scGTN: Deep Siamese Graph Transformer Network for Single-cell RNA Sequencing Clustering

1 Introduction

Single-cell RNA sequencing (scRNA-seq) has emerged as a fundamental technology for profiling gene expression in bioinformatics, enabling high-resolution profiling of gene expression across individual cells. Based on the scRNA-seq data, cell clustering plays a central role by organizing cells into biologically meaningful groups based on their gene expression patterns, thereby revealing the intrinsic heterogeneity of cell populations and facilitating biological interpretation of their functions and interactions [Kiselev et al., 2019; Peng et al., 2020]. Accordingly, clustering scRNA-seq data is particularly valuable for a range of downstream tasks, including marker gene annotation [Dai et al., 2022] and cell type identification [Hu et al., 2020].

† Corresponding authors.

For effectively scRNA-seq cell clustering, numerous methods have been proposed to achieve this goal. Early approaches primarily relied on classical clustering algorithms, such as k-means [Hicks et al., 2021], hierarchical clustering [Johnson, 1967], and density-based methods [Petegrosso et al., 2020], which are computationally efficient but are inherently limited in modeling the complex nonlinear characteristics of scRNA-seq data. With the rapid advances in deep learning, recent studies have shifted toward learning-based clustering frameworks that leverage graph neural networks (GNNs) [Ju et al., 2025; Fan et al., 2026] to extract informative representations from scRNA-seq data. By representing cells as nodes and their interactions as edges in a graph, GNN-based approaches jointly exploit gene expression features and structure characteristics to learn context-aware cell embeddings, enabling a more comprehensive cell clustering [Wang et al., 2021; Gan et al., 2022; Xu et al., 2024; Xu et al., 2025; Zhang et al., 2026b].

Despite the effectiveness of these methods, we argue that two major challenges remain insufficiently addressed: ∂ Ignoring the Sparsity and Noise inherent within the scRNAseq Data. Arising from extensive dropout effects and measurement noise, scRNA-seq data present a high-dimensional expression matrix [Ou-Yang et al., 2022; Wu and Ma, 2022], which substantially complicates the reliable construction of cell relationships and frequently leads to graphs that are highly sparse and noisy. ∑ Limited Exploitation of Complex Intercellular Structural Information. While existing GNN-based approaches primarily focus on local neighborhood aggregation, they often overlook richer structural cues embedded within the cell graph. Features such as shortest paths and node distances are rarely leveraged, resulting in an insufficient utilization of the graph's inherent information. As shown in Figure 1, cell similarities derived solely from gene expression features tend to be highly homogeneous and difficult to distinguish, whereas incorporating structural information such as positional relationships and shortest-path distances enables a more discriminative characterization of intercellular relationships for cell clustering.

Having realized the above challenges with existing methods, in this paper, we propose a novel deep Siamese Graph Transformer Network for single-cell RNA sequencing (scGTN), which effectively exploits complementary graph information derived from gene expression data to learn discriminative and robust cell embeddings. Specifically, given the scRNA-seq data, we formulate it into a graph and construct two augmented graph views to serve as dual branches to capture and refine the intercellular information. Then, for each branch, we introduce a graph transformer network that incorporates shortest-path information and node-wise distances to capture the complementary graph structural dependencies, enabling a more expressive modeling of complex intercellular relationships. Based on the encoded information, we utilize an optimal transport-guided self-supervised clustering method to align cluster distributions for the clustering learning process.

To summarize, the contributions of this paper are:

• ∂ New Perspective: We study scRNA-seq data clustering by jointly considering gene expression profiles and cell graph structures, which effectively exploits complementary intercellular information beyond feature-level similarities.
• ∑ Novel Methodology: We propose a novel framework termed scGTN, which introduces a Siamese graph transformer network to incorporate richer graph structural information, including shortest paths and node-wise distances for the cell clustering task.
• ∏ Extensive Experiment: We conduct extensive experiments on multiple benchmark scRNA-seq datasets to evaluate the effectiveness of our scGTN. The results further show the superior clustering performance of our framework.

2 Methodology

2.1 ProblemDefinition

Let X = (xij) 2 RN⇥D denote the single-cell gene expression matrix, where each row xi 2 RD denotes the gene expression of the i-th cell, N and D denote the number of cells and gene respectively. Based on X, we construct an undirected cell graph G = (V↪ E), where each node represents an individual cell and edges are established via a k-nearest neighbor (kNN) strategy to capture intercellular similarities quantified by the Pearson correlation coefficient [Benesty et al., 2009]. The adjacency matrix of the constructed graph can be A = (aij) 2 RN⇥N , where entry aij = 1 if (vi↪ vj) 2 E for cell vi and vj , otherwise aij = 0. We associate the degree matrix of the graph as D = diag(d1↪ d2↪ ▷ ▷ ▷ ↪ dN) 2 RN⇥N , where di = PNj=1 aij is the degree of node vi. The goal of scRNA-seq clustering is to partition the cells within the graph G into C disjoint clusters, with each node vi belonging to a unique cluster according to its gene expression features and intercellular connectivity.

2.2 Framework Overview

In this work, we focus on scRNA-seq data clustering and propose a Siamese graph transformer network for the task. As shown in Figure 2, our scGTN consists of three modules: (1) Dual Augmentation Module for scRNA-seq Data, which jointly perturbs the gene expression features and the underlying cell graph structure to generate two complementary views, thereby facilitating robust representation learning; (2) Siamese Graph Transformer Network Fusion Module, which employs two weight-sharing Siamese encoders to process the augmented views while explicitly incorporating structural information of the cell graph; (3) Optimal Transport Clustering Module, which formulates clustering as a distribution alignment problem and employs optimal transport to guide the learning of cluster-consistent representations.

2.3 Dual Augmentation for scRNA-seq Data

To mitigate the effects of inherent sparsity and noise in scRNA-seq data, we introduce biologically reasonable augmentations by adding controlled noise to the gene expression profiles, generating two complementary views that improve the robustness of the learned representations.

Gene Expression Perturbation. Given the single-cell gene expression matrix X, we employ Gaussian noise to perturb the gene expression profiles, which can simulate the natural variability of the gene expression profile as augmentation for the clustering task. The perturbation can be formulated as:

where m denotes the random mask drawn from a Gaussian distribution. In practice, we set the distribution as N(1↪ 0▷1) following the previous work [Xu et al., 2025]. The two augmented gene matrices can be generated as ˜X1 and ˜X2 .

Intercellular Graph Structure Augmentation. In addition to perturbing the gene expression profiles, enhancing the intercellular structure is essential for improving the model's ability to capture the relationships between cells. Specifically, given the constructed cell graph, we remove spurious edges to refine the graph structure and preserve the most biologically relevant connections between cells as one view:

where M denotes the mask matrix based on pairwise cosine similarity. In practice, we identify and remove 10% of edges with the lowest similarity values using this mask. For another view, we enhance the intercellular structure by considering graph diffusion, which propagates information across the graph to strengthen meaningful cell connections:

where ⌘ denotes the diffusion coefficient to control the information propagation within the cell graph.

2.4 Siamese Graph Transformer Network Fusion

To further utilize the richer structural information from the intercellular graph, we introduce a Siamese graph transformer network [Wang et al., 2025] for the two augmented views and adaptively fuse the encoded feature to learn discriminative cell representations.

Gene Expression Embedding. Since cells with similar gene expression profiles are assumed to be connected in the graph, we leverage the gene expression profiles of connected cells to enhance the learning of discriminative gene expression embeddings. In particular, we employ two Siamese graph encoders to produce the cell embedding as:

where G(·↪ ·) denotes the Siamese graph encoder, which is shared across the two augmented views. The resulting encoded gene expression matrix Z⇤ 2 RN⇥d represents the learned feature embeddings for each cell, where d indicates the dimensionality of the feature space. Then, for each cell node with feature z⇤ i , we select the top-t neighbors with the most similar features and concatenate them into a sequence S⇤ i 2 R(t+1)⇥D . The sequence is then encoded as:

where Fge(·) denotes the gene expression encoding function.

Explicit Intercellular Structure Embedding. To explicitly capture the intercellular structure within scRNA-seq data, we incorporate both the position embedding and shortest path embedding of the central node and its neighbors. Specifically, we define the relative positions of the central node as Pos(z⇤ i ) = 0 and assign position values to its neighbors based on their intimacy to the central node, with nodes that are more similar to the central node receiving position values closer to 0. The position embedding is formulated as:

where Fpos(·) denotes the position mapping function. In addition to the intimacy between two nodes, the shortest path characterizes the connection structure between them and explicitly the crucial intercellular structure. Specifically, the shortest path between two nodes within the constructed cell graph can be defined as Sp(z⇤ i ↪ z⇤ j ). Based on this, we formulate the shortest path embedding as:

where Fsp(·) denote the shortest path mapping function.

Siamese Graph Transformer Refinement. We integrate the gene expression and intercellular structure embedding as:

Then we leverage the attention mechanism to encode the integrated embedding, which can be formulated as:

where WQ↪WK↪WV 2 Rd⇥d denote the query, key and value matrices respectively. We employ the multi-attention mechanism with feed-forward networks (FFN) and residual connections as a transformer layer. Through stacking Llayers, the output feature can be Y L↪⇤ i 2 R(t+1)⇥d . The final cell node embedding can be refined as:

We further introduce the correlation loss to prevent feature collapse and redundancy between two views, defined as:

The final cell embedding ˆZ can be fused from two views as:

2.5 Clustering with Optimal Transport

Given the refined cell embedding, we adopt a self-supervised strategy for the scRNA-seq data clustering. Specifically, we measure the similarity between cell embedding ˆ zi and the clustering center θc as the Student's t-distribution [Maaten and Hinton, 2008], formulated as:

where Q = [qic] 2 RN⇥C + denotes the cluster assignment of the cell and ✓ is the degrees of freedom in the Student's t-distribution. Following the previous work [Bo et al., 2020], we sharpen the cluster assignment as the target distribution P = [pic] 2 RN⇥C + , which can be calculated as:

where fc = Pi qic denotes the soft cluster frequency. We further introduce the optimal transport strategy to align the cluster distribution with the mixing proportions, which ensures the robust clustering and prevents the degenerate solution that all cells allocate to a single label:

where P and - logQ denote the transport plan and cost matrix, respectively. Here ⇡ in the constraints represents the proportion of cells assigned to each cluster, which can be derived from the intermediate clustering results. In practice, we utilize the Sinkhorn distance method [Sinkhorn, 1967] for the optimization by introducing entropy constraint H(·) with a Lagrange multiplier λ as:

Therefore, we can iteratively optimize the P as:

where we iteratively update u(t) and v as 1N◁(Qλv(t-1) ) and N⇡◁(Qλu(t) ) at iteration t with N⇡ representing the the total mass of the target distribution and the initial value setting as v(0) = 1N. During the training process, we fixed ˆ P and align Qwith the ˆ P as the clustering loss:

2.6 Overall Optimization

To preserve the intercellular structure information for cell clustering, we reconstruct the cell graph with the learned cell embedding and the Mean Squared Error (MSE) loss can be:

Since scRNA-seq data exhibits sparsity and overdispersion, we further incorporate the Zero-Inflated Negative Binomial (ZINB) loss [Eraslan et al., 2019; Xu et al., 2025] to model the excess zeros and variability. Finally, the overall objective of our scGTN can be defined as:

where ↵↪ β and γ denote the trade-off parameter to balance the contribution of each component.

3 Experiments

3.1 Experimental Setup

Benchmark Datasets. To evaluate the robustness of our model for clustering across species and diverse biological contexts, we conduct experiments on seven public single-cell benchmark datasets, covering both human and mouse samples and including pancreas, liver, and peripheral immune cell populations. In terms of data scale, the number of cells ranges from 1,064 to 8,617, the number of genes ranges from 2,000 to 20,125, and the expression matrices exhibit sparsity rates of approximately 73.02%-93.26%. To standardize the input dimensionality and focus on more informative features, we select either 1,500 or 2,000 highly variable genes (HVGs) for each dataset as the model input. Detailed statistics are summarized in the Table 5 of Appendix B.

Baselines. To highlight the advantages of scGTN, we compare it with ten representative clustering approaches. Specifically, pcaReduce [Žurauskien˙e and Yau, 2016] is included as a representative traditional clustering method. DEC [Xie et al., 2016], contrastive-sc [Ciortan and Defrance, 2021], scNAME [Wan et al., 2022], scDeepCluster [Tian et al., 2019] and AttentionAE-sc [Li et al., 2023] are deep learning-based clustering methods, which mainly learn representations via autoencoders or contrastive learning and then perform clustering. In addition, scGNN [Wang et al., 2021], scDSC [Gan et al., 2022], scCDCG [Xu et al., 2024] and scSiamese [Xu et al., 2025] are deep graph clustering methods that incorporate GNNs and structural information for representation learning and clustering. Detailed experimental settings and implementation details are provided in the Appendix B.

Evaluation Metrics. To evaluate the effectiveness of the proposed method, we assess clustering performance using three widely adopted metrics: clustering accuracy (ACC), normalized mutual information (NMI) [Strehl and Ghosh, 2002], and adjusted Rand index (ARI) [Vinh et al., 2009].

3.2 Overall Performance

Quantitative Analysis. Table 1 compares the clustering performance of scGTN with ten competitive baselines on seven benchmark datasets. From the results, we make the following observations. (1) scGTN achieves the best performance across all three metrics on all datasets, demonstrating its strong capability in handling high-dimensional and highly sparse scRNA-seq data. On average, compared with the second-best method, scGTN yields significant improvements of 3.00%, 2.42%, and 2.30% in terms of ACC, NMI, and ARI, respectively, validating the effectiveness and generalizability of our approach. (2) Compared with deep clustering methods that rely solely on gene expression features, our scGTN consistently delivers substantial gains, highlighting the benefit of jointly leveraging expression features and cell-cell structural information. (3) Although GNN-based methods like scGNN incorporate cell graphs, they predominantly rely on local neighborhood aggregation, which renders them vulnerable to over-smoothing and limits their ability to capture global dependencies. scGTN overcomes this by encoding complex structural information via the Siamese Graph Transformer, yielding more discriminative representations. Analysis of runtime is provided in Appendix F.

Visualization. To further investigate the model's ability to capture the underlying structure of cells in the embedding space, we apply UMAP [McInnes et al., 2020] to project the learned representations of the Human Liver dataset into two dimensions, as shown in Figure 3. We observe that our scGTN produces highly compact clusters with clear intercluster boundaries, faithfully reflecting the complex underlying tissue structure and effectively preserving the intrinsic heterogeneity among different cell types. In contrast, methods such as scDSC often yield projections with noticeable subtype mixing or diffuse distributions, making it difficult to separate rare cell populations from major groups.

3.3 Biological Analysis

Differential Gene Expression Analysis. Differentially expressed genes (DEGs) serve as a critical metric to validate the biological identity of learned clusters. Using Seurat's "FindAllMarkers" [Butler et al., 2018], we extract the top-100 DEGs for each cluster on the Muraro Human Pancreas dataset and quantitatively compute the overlap ratio with the corresponding ground-truth cell types. As shown in Figure 4, the high-overlap regions of scGTN are predominantly concentrated along the diagonal, consistently surpassing 90% overlap in most clusters. This indicates that our clusters can be stably mapped to specific cell types with minimal confusion. In stark contrast, baseline methods frequently exhibit lower overlap ratios due to ambiguity and inter-cluster confusion. To further investigate the biological fidelity, we compare the rankings of the top 5 DEGs between scGTN and the Gold Standard as illustrated in Figure 5. Beyond mere overlap, we observe remarkable consistency in the dominant markers. For instance, in the cluster corresponding to Pancreatic PP cells (Cluster 7), our scGTN identifies PPY and ETV1 as the toptwo markers, perfectly matching the Gold Standard. This precise alignment in both gene identity and ranking confidence demonstrates that our scGTN does not merely capture global similarities but accurately prioritizes the key marker genes essential for distinct cell identification.

KEGG Pathway Enrichment Analysis. To validate the biological interpretability of the clustering results, we perform KEGG pathway enrichment analysis on representative clusters. As shown in Figure 6, our scGTN successfully resolves distinct functional subpopulations in pancreatic tissue. Cluster 3 is significantly enriched in "Insulin secretion" and critical diabetes-related pathways including MODY, while also exhibiting neuroendocrine signatures such as the "Synaptic vesicle cycle", confirming its biological identity as pancreatic β cells. Cluster 4 shows strong enrichment in "Ribosome" and protein synthesis-related pathways, highlighting the high protein synthesis activity characteristic of acinar cells. Finally, Cluster 5 is enriched in pathways such as "Phagosome" and "Lysosome", indicating the presence of tissue-resident immune cells such as macrophages. These results demonstrate that our scGTN can accurately identify KEGG pathways and distinct functional phenotypes.

Furthermore, we provide trajectory inference analysis in Appendix D to validate preserved developmental manifolds.

3.4 Ablation Study

Effectiveness of Major Components. We evaluate the contribution of each loss term on the Muraro dataset. Table 2 indicates that the exclusion of any component leads to a sharp performance degradation of over 10% in ACC, highlighting the non-redundancy of each module. Specifically, the absence of the optimal transport clustering loss Lclu results in the most severe drop, underscoring its critical role in guiding partition assignments. Similarly, removing the correlation term Lcor or the reconstruction objectives LZINB and Lrec significantly impairs the capability of the model to align views and preserve topology. Ultimately, the full scGTN achieves superior performance, validating the synergy of jointly optimizing for structural, feature, and distribution constraints. Dualcomponent ablation analysis is provided in Appendix G.

Impact of Embedding Components. We further dissect the embedding module to assess the contributions of Gene Expression (GE), Position (Pos), and Shortest-Path (SP) embeddings in Table 3. Results indicate that while GE provides the foundational semantics, integrating it with structural priors yields massive gains. Notably, adding Pos alone boosts ACC by over 12% compared to the GE baseline, and the full combination achieves peak performance, confirming that structural information is essential for distinguishing subtle cell states.

3.5 Sensitivity Analysis

To evaluate the robustness of scGTN against hyperparameter variations, we conduct a sensitivity analysis on the Muraro Human Pancreas dataset. We examine neighborhood size K (5-60) and Transformer layers L (1-8). As illustrated in Figure 7, the model demonstrates high stability with respect to K, showing only marginal fluctuations across all metrics, indicating that our scGTN is robust to variations in the underlying graph construction. Conversely, performance depends more significantly on network depth L. Shallow networks where L = 1 yield suboptimal scores due to insufficient feature propagation and limited semantic capture. Increasing depth improves results until peaking at L = 6, where the model effectively balances local and global information. Further extending depth to 8 leads to a slight performance drop, likely due to over-smoothing. Further sensitivity analyses regarding other hyperparameters are detailed in Appendix E.

4 Conclusion

In this paper, we propose a novel deep Siamese Graph Transformer Network (scGTN), which effectively integrates gene expression profiles and intercellular structural information for scRNA-seq data clustering. Specifically, we formulate the scRNA-seq data as a graph and construct two augmented graphs as dual views to capture complementary intercellular relationships. Then, we introduce a Siamese graph transformer network which integrates richer graph structural information, i.e., shortest-path and node-wise distance, to explicitly capture intercellular relationships. To further refine the cell clustering, we incorporate an optimal transport mechanism in a self-supervised manner, enhancing representation learning while preserving biological relevance. Extensive experiments on scRNA-seq benchmark datasets verify the effectiveness and superiority of our proposed scGTN.

Contribution Statement

Jinke Wu and Yifan Wang contributed equally to this work.

Acknowledgments

This work is supported in part by the National Natural Science Foundation of China under Grant 62306014 and 12501344, Postdoctoral Fellowship Program (Grade A) of CPSF under Grant BX20250376 and BX20240239, China Postdoctoral Science Foundation under Grant 2024M762201, Sichuan Science and Technology Program under Grant 2025ZNSFSC1506 and 2025ZNSFSC0808, Sichuan University Interdisciplinary Innovation Fund.