Image-Enhanced Hybrid Encoding with Reinforced Contrastive Learning for Spatial Domain Identification in Spatial Transcriptomics

1 Introduction

Spatial transcriptomics has revolutionized biological research by integrating gene expression data with spatial information, offering a comprehensive view of cellular distribution and functional states within tissue microenvironments [Huang et al., 2024]. Unlike traditional single-cell transcriptomics, which lacks spatial context, this approach uncovers intricate spatial patterns of gene expression [Zhu et al., 2024a; Singhal et al., 2024]. Its applications span tumor microenvironment analysis, tissue development, and disease pathology [Nie et al., 2024; Zhong et al., 2024; Cao and Gao, 2022]. Recent advancements have improved resolution from spotlevel to single-cell, facilitating detailed investigations of cellular interactions and tissue organization [Zhou et al., 2025; Yuan, 2024; Yang et al., 2024]. However, the multimodal nature of spatial transcriptomics data-including gene expression, spatial coordinates, and tissue images-poses significant challenges for downstream analyses, particularly in spatial domain identification [Tang et al., 2023a; Jia et al., 2024; Jiang et al., 2024]. Integrating these diverse data modalities to enhance classification performance remains a critical hurdle, limiting the broader application of this powerful tool in biological research.

* Corresponding authors

Many strategies have been developed to analyze spatial transcriptomics data [Yuan et al., 2024; Ma and Zhou, 2022; Long et al., 2024]. Classical methods, such as Seurat and K-means, primarily cluster samples based on gene expression alone [Satija et al., 2015]. While effective in identifying transcriptomic patterns, these approaches often overlook spatial relationships between cells, limiting their ability to fully interpret tissue heterogeneity. To address this limitation, advanced methods like stCluster[Wang et al., 2024a], SpaCAE [Hu et al., 2024] and STAGUE [Nie et al., 2024] leverage graph neural networks (GNNs) to integrate gene expression and spatial information, enhancing performance by capturing spatial context . Image-based approaches, such as xSiGra, further incorporate tissue slice images to complement gene expression data, utilizing morphological and structural features [Budhkar et al., 2024]. However, many of these methods rely on straightforward concatenation for multimodal integration, which may fail to account for complex interdependencies between modalities. These challenges highlight the need for a unified framework capable of effectively modeling spatial relationships, integrating multimodal data, and overcoming issues like high dimensionality and data noise.

We are motivated by three key considerations to address the challenges in spatial domain identification. First, while existing methods incorporate spatial coordinates, they often fail to fully integrate these with gene expression data. To bridge this gap, we utilize GraphSAGE [Hamilton et al., 2017], a scalable and inductive graph neural network, to model spatial dependencies. By aggregating information from a fixed number of neighbors, GraphSAGE generates spatially informed cell representations, offering a robust foundation for identifying spatial domains. Second, tissue slice images contain rich morphological and structural features that complement gene expression data. Existing methods frequently rely on shallow feature extraction, overlooking deeper spatial and contextual information encoded in images. To address this, we employ a dual-encoding strategy: autoencoders capture nonspatial features, while GraphSAGE models spatial dependencies. This approach produces more discriminative and comprehensive representations of cellular environments. Third, traditional contrastive learning methods often encounter issues with false-negative samples during multimodal data fusion. To mitigate this, we introduce a reinforced contrastive learning strategy that incorporates high-order random walks to model transition probabilities between samples, reducing the impact of false negatives. Additionally, we leverage optimal transport to dynamically align clustering distributions with auxiliary distributions, further enhancing the robustness and consistency of the learned representations.

In this work, we propose Image-Enhanced Hybrid Encoding with Reinforced Contrastive Learning (IE-HERCL), a novel framework for spatial domain identification in spatial transcriptomics, as illustrated in Figure 1. IE-HERCL integrates autoencoders, graphSAGE, and attention mechanisms to extract spatial and non-spatial features from gene expression and image data. These features are fused through crossmodal attention mechanism to generate unified and consistent representation. To address challenges in multimodal data integration, we introduce a reinforced contrastive learning strategy that employs high-order random walks to mitigate the impact of false-negative samples. Additionally, optimal transport is leveraged to further enhance the robustness and reliability of the learned representations. Experimental evaluations across multiple spatial transcriptomics datasets demonstrate that IE-HERCL achieves outstanding performance in both spatial domain identification and cell type classification, consistently surpassing existing methods. This work offers an innovative and efficient framework for multimodal analysis of spatial transcriptomics data, paving the way for deeper insights into tissue organization and cellular heterogeneity.

2 RelatedWork

2.1 Self-Supervised Contrastive Learning

Self-supervised contrastive learning aims to bring positive pairs closer in the latent space while pushing negative pairs apart, thereby enhancing representation learning[Zeng et al., 2022; Wang et al., 2024b]. For example, SpaceFlow employed a spatially regularized deep graph network to encode gene expression data and spatial location information, optimizing embeddings through contrastive learning [Ren et al., 2022]. Similarly, GraphST utilized graph convolutional networks (GCNs) to jointly model gene expression and spatial coordinate information [Long et al., 2023]. It introduced self-supervised contrastive learning by perturbing gene expression vectors while maintaining topological structures to form negative pairs, improving latent representation learning. stDCL enhanced gene expression data to learn two latent embeddings and trained the network using spatial-aware contrastive learning and cluster-level contrastive learning to capture more discriminative representations [Yu et al., 2025]. However, these methods often face challenges in effectively handling negative samples, especially when jointly modeling spatial and gene data, which can reduce model stability and accuracy. In contrast, our method utilizes high-order random walks to learn transition probabilities between samples. By weighting negative samples with very low transition probabilities, we reduce their adverse effects, enabling the model to capture meaningful patterns more effectively and achieve superior performance in spatial domain identification tasks.

2.2 Graph Neural Networks for Spatial Domain Identification

Graph Neural Networks (GNNs) are widely applied in spatial domain identification by learning latent representations or reconstructing gene expression data [Sun et al., 2025; Tang et al., 2023b; Zhu et al., 2024b]. For example, SpaGCN constructed adjacency graphs based on spatial coordinates and used graph convolution to aggregate information from neighboring nodes [Hu et al., 2021]. The aggregated features were combined with gene expression matrices for unsupervised clustering to identify spatial domains. STAGATE employed a graph attention autoencoder to integrate spatial coordinate and gene expression data, enhancing spatial domain identification by jointly modeling spatial and transcriptomic relationships [Dong and Zhang, 2022]. STMGCN utilized a multi-graph convolutional network to encode spatial and gene expression information, employed attention mechanisms to fuse these modalities, and applied an unsupervised deep embedding clustering framework for spatial domain identification [Shi et al., 2023]. MAFN adopted an end-to-end GNN to encode coordinate and gene data, combining cross-view correlation reduction strategy and attention mechanisms to learn discriminative embeddings for spatial domain identification [Zhu et al., 2024c]. While these methods effectively capture relationships between spatial and transcriptomic data, they leave room for improvement in multimodal data integration, especially when combining gene expression with image data. Existing approaches often overlook deep correlations between modalities. To address this, we leverage GraphSAGE, which employ neighborhood sampling and feature aggregation strategies to generate flexible and robust node representations. Simultaneously, we incorporate image data as a complementary modality within the framework. By doing so, our method fully exploit the latent information in multimodal data, creating more discriminative embeddings for spatial domain identification.

3 Method

3.1 Preliminaries

The goal for image-enhanced spatial transcriptomics data integration is to incorporate spatial information, gene expression data, and image data to generate comprehensive latent representations. Given spatial coordinates of N points S = {(xi, yi)}Ni=1 , the corresponding gene expression data Xg ∈ RN∗d , and image data Ximg ∈ RN∗h∗w∗c , where N denotes the number of samples, d represents the dimension of gene features, and h, w, and c are the height, width, and channels of the image data, respectively. In our framework, we propose a novel approach that integrates three key innovations: first, a hybrid encoding strategy combines an autoencoder (AE) to capture non-spatial features with GraphSAGE to model spatial dependencies, effectively processing both gene and image data. Second, to mitigate the impact of false negatives in contrastive learning, we design a probability transition matrix using high-order random walks, which improves the detection of potential positive counterparts and enhances the quality of contrastive learning. Finally, we leverage optimal transport to align the auxiliary distribution of the fused consensus embedding from gene and image modalities, ensuring consistency in the representations and improving clustering accuracy. This approach enables robust multimodal data integration and significantly enhances spatial domain identification performance.

3.2 Multimodal Feature Representation Learning

In spatial transcriptomics analysis, gene expression data and image data exhibit high heterogeneity and dimensionality, making their integration challenging. To effectively extract and integrate features from these two modalities, we design a multimodal feature representation learning framework. This approach combines autoencoder (AE), GraphSAGE encoder, and cross-modal attention mechanism to extract high-quality latent representations and enhance semantic consistency between modalities.

Feature Extraction for Gene Expression Data. For gene expression data Xg ∈ RN∗d , an AE is employed for denoising and feature learning. The AE encoder maps the input data to a latent space Zg ∈ RN∗d′ , and the decoder reconstructs the original input. During this process, the AE removes noise, compresses redundant information, and extracts semantic features. The reconstruction loss function is defined as:

where f g θd and f g θe represent the encoder and decoder, respectively,∥·∥2F denotes the Frobenius norm to measure the difference between the input and reconstructed output. By minimizing this loss, the AE effectively extracts representative gene expression features.

Feature Extraction for Image Data. For image data Ximg ∈ RN∗h∗w∗c , high-level features Znet ∈ RN∗dnet are first extracted using a pre-trained ResNet50. Subsequently, an autoencoder further processes these features to learn a deeper latent representation Zimg ∈ RN∗d′ , removing redundancy and capturing meaningful patterns. The reconstruction loss is defined as:Limg

where Znet,i is the feature of the i-th image extracted by ResNet50, and f img θd and f img θe represent the encoder and decoder for image data, respectively. By minimizing this loss, the model extracts compact and representative image features.

Local Structure Learning. To capture local structural features in both gene expression and image data, we integrate a GraphSAGE encoder. The adjacency matrix A ∈ RN∗N is constructed based on spatial coordinates to define the relationships between nodes. The detailed construction process is provided in the data processing section. GraphSAGE updates node representations by aggregating information from neighboring nodes, using the following formula:

where Z v i represents the updated node representation, v denotes the data type (gene or image), N(i) is the set of neighboring nodes for node i, Aggregate(·) is an aggregation function (e.g., mean pooling), σ(·) is an activation function, and We is the learned weight matrix. GraphSAGE effectively captures local structural features and enhances spatial relationship modeling.

Cross-Modal Information Fusion. After extracting latent representations from gene expression and image data, intramodal attention is used to adaptively fuse features within each modality. The fused representations for genes and images are given by:

where αae and αgs are weights representing the contributions of AE and GraphSAGE encodings, respectively. Subsequently, cross-modal attention is applied to adaptively integrate the fused features from both modalities:

where βg and βimg are learnable attention weights that balance the contributions of gene and image features. This adaptive process ensures that the integrated latent representation captures consistent and robust information from both modalities.

Reconstruction of Gene ExpressionData. AGraphSAGE decoder is used to reconstruct gene expression data ˆXg from the integrated latent representation Z. The reconstruction process is defined as:

where Wd is the learned weight matrix. The reconstruction loss is given by:

The decoder reconstructs the gene expression data while preserving local structural information.

Overall Reconstruction Loss. To comprehensively consider the impact of each modality and cross-modal fusion, the total reconstruction loss is defined as:

By minimizing the total reconstruction loss, the model learns effective multimodal latent representations, ensuring the reconstruction of accurate and consistent data.

3.3 Negative Sample Mitigation Strategy in Contrastive Learning

In contrastive learning, the selection and treatment of negative samples critically impact model performance. Conventional approaches often treat all other samples as negative for a given anchor, which can introduce false negatives (FNs) in scenarios without clear class boundaries. These FNs-samples that are incorrectly assumed to be dissimilar-can hinder the ability of the model to learn meaningful representations. To address this issue, particularly in the context of gene expression data, we propose a random walkbased negative sample mitigation strategy. This approach is designed to adjust the influence of negative samples dynamically, leveraging structural relationships among data points.

Constructing the Transition Probability Matrix via RandomWalks. Given the high-dimensional and structured nature of gene expression data, we first construct an affinity graph where each sample is a node, and edge weights reflect pairwise similarities. Specifically, we use a heat kernel function to define the adjacency matrix A:

where xi and xj are the embeddings of samples i and j, and σ is a smoothing parameter that controls the decay rate of similarity. This affinity graph captures the inherent relationships among samples. To explore high-order neighborhood relationships, we perform a random walk on the graph. The random walk is represented by the transition matrix M:

where D is the degree matrix. By iterating this process, we compute a high-order transition probability matrix T :

where α is a balance parameter (default 0.5) controlling the contribution of direct and high-order neighbors, and t is the number of steps in the random walk. The resulting Tij encodes the probability that node j is a high-order neighbor of node i, effectively capturing distant relationships in the data. This transition matrix allows us to reduce the likelihood of incorrectly treating semantically similar samples as negatives.

Incorporating Transition Probabilities into Contrastive Learning. The transition probability matrix T is integrated into the contrastive learning framework to mitigate the impact of false negatives. In contrastive learning, views of the same sample are treated as positives, while other samples are treated as negatives. However, a naive random selection of negatives may introduce FNs in datasets without clear categorical separation. To address this, we use Tij to weight negative samples during the optimization process. For a consensus representation Zi and a view-specific representation Z v j , we define the contrastive loss as:

where cos(Zi, Z v i ) denotes the cosine similarity between Zi and Z v i , τ is a temperature parameter (default 0.5), and Tij represents the transition probability between samples i and j. Negative samples with lower transition probabilities are assigned reduced weights, ensuring that semantically distant samples have a minimal impact on the loss. This strategy effectively mitigates the influence of false negatives and improves the robustness of the learned representations.

3.4 Optimal Transport-Based Representation Optimization

To further enhance the quality of the unified embedding, particularly in terms of achieving superior clustering performance, we propose an optimal transport (OT)-based selfoptimization strategy. By leveraging a mutual supervision mechanism, this strategy dynamically aligns the clustering distribution with an auxiliary distribution, enabling adaptive clustering optimization.

Construction of Clustering and Auxiliary Distributions. We define two key distributions: the clustering distribution Q and the auxiliary distribution P. The clustering distribution Q represents the relationship between the latent embedding Z and the cluster centroids. The element qiu is calculated as:

where qiu indicates the probability that sample i belongs to the µu-th cluster. zi is the embedding of sample i, µu is the u-th cluster centroid computed using pseudo-labels, and k is the total number of clusters. This distribution quantifies the probability of each sample belonging to different clusters. To enhance the contribution of high-confidence samples in clustering optimization, we construct the auxiliary distribution P as:

This design amplifies the influence of high-confidence samples, allowing them to play a more prominent role during optimization and improving the robustness of the clustering process.

Optimal Transport Objective. To align the clustering distribution Q with the auxiliary distribution P, we frame the problem as an optimal transport task. The objective is to minimize the cost of transporting mass from Q to P , while ensuring smoothness through entropy regularization. The objective function is defined as:

where C is the cost matrix representing the Euclidean distance between points in Q and P, γi,j denotes the transport plan between the i-th element of Q and the j-th element of P, and ε controls the strength of entropy regularization. To measure the alignment between distributions efficiently, we

Algorithm 1 IE-HERCL

Input: Multi-modal data including RNA sequencing and image data, and spatial coordinates S.

Parameters: Total epochs E, number of random walk steps t, and weight parameters λ1, λ2, λ3.

• 1: Initialize the spatial coordinate graph G using distance nearest neighbors.
• 2: for epoch in 1, 2, . . . ,E do
• 3: Calculate Lgrec for the gene modality using Eq. (1).
• 4: Calculate Limg rec for the image modality using Eq. (2).
• 5: Obtain modality-specific encodings Z g f and Z img f .
• 6: Map all modality-specific encodes to a unified embedding Z using Eq. (6).
• 7: Update AE and GraphSAGE encoders and decoders using Eq. (9).
• 8: Calculate the contrastive learning loss with a negative sample mitigation strategy using Eq. (13).
• 9: Calculate the optimal transport loss Ls using Eq. (17).
• 10: Train IE-HERCL by minimizing L using Eq. (18).

11: end for

Output: Unified embedding Z and reconstructed gene expression data ˆX.

employ the Sinkhorn divergence [Yu et al., 2024], which is

In practice, the auxiliary distribution P is constructed based on the clustering distribution Q. By continuously optimizing the alignment between Q and P, high-confidence data points are expected to play a dominant role in the clustering process. This optimization continues iteratively until a predefined maximum number of iterations is reached, achieving deep refinement of the consistency representation and enhancing the performance of the model in clustering assignment tasks.

3.5 Overall Optimization Objective

The final loss function of IE-HERCL integrates three components: the reconstruction loss Lrec, the contrastive learning loss Lc, and the optimal transport loss Ls. The joint optimization objective is given as:

where λ1, λ2, and λ3 are the weight parameters for each loss term. By jointly optimizing these objectives, the proposed method effectively integrates spatial information, image data, and gene expression data to learn a more consensus embedding representation, significantly improving clustering performance. Algorithm 1 provides a detailed description of the entire model workflow.

4 Experiments

4.1 Experimental Setups

Datasets. We evaluate the proposed method on five widely used spatial transcriptomics datasets that include image information: (1) NanoString Lung 9-1, (2) 10x Visium DLPFC,

(3) 10x Visium human breast cancer data, (4) mouse brain anterior slice, and (5) mouse brain coronal slice. Details on the dataset processing can be found in the Supplementary Material.

Baseline Methods. To validate the superiority of our model, we compare it with recent state-of-the-art methods, including four single-modal spatial domain identification approaches, namely Scanpy [Wolf et al., 2018], STAGATE [Dong and Zhang, 2022], GraphST [Long et al., 2023] and stDCL [Yu et al., 2025], as well as two image-enhanced spatial domain identification methods, SiGra [Tang et al., 2023a] and xSiGra [Budhkar et al., 2024]. A detailed description of these methods can be found in the supplementary materials.

EvaluationMetrics. To comprehensively evaluate the performance of spatial domain identification, we adopt five widely used clustering metrics: Adjusted Rand Index (ARI), Normalized Mutual Information (NMI), Adjusted Mutual Information (AMI), Fowlkes-Mallows Index (FMI), and Homogeneity Score (HS) [Long et al., 2024]. These metrics collectively capture clustering quality from multiple perspectives, ensuring a thorough assessment of model performance.

Implementation Details. The entire model is implemented using PyTorch 1.13.0 and all experiments are conducted on an Ubuntu 20.04 server with an NVIDIA 3090 GPU. For the autoencoder in our model, we employ two layers (512dimensional to 64-dimensional), while the GraphSAGE layer is set to a single layer (64-dimensional). To optimize the model, we utilize the Adam optimizer with a weight decay of 0.0001 and an initial learning rate of 0.001. For the other comparison methods, we use the default parameters from the original papers.

4.2 Experimental Results

Results Comparison. To demonstrate the superiority of our model, we conduct both qualitative and quantitative evaluations on the datasets mentioned earlier. For the qualitative evaluation of IE-HERCL, we visualize the spatial domains of a lung cancer tissue FOV. As shown in Figure 2, our method exhibits a strong alignment with the true labels. In contrast, the comparison method, Scanpy, which does not incorporate spatial coordinate information or pathological image data, shows poor alignment between the spatial distribution and the true labels. The superior performance of IE-HERCL is primarily attributed to the incorporation of pathological image data, which enhances representation learning and enables the model to generate robust representations. In terms of quantitative metrics (Supplementary Figure 1), IE-HERCL achieves superior performance across ARI, NMI, and HS on all lung cancer and DLPFC tissue slices, with particularly notable improvements in lung cancer tissue where pathological image data is utilized. Furthermore, on other datasets, including human breast cancer, mouse brain anterior slices, and mouse brain coronal slices (as shown in Table 1), IE-HERCL outperforms all competing methods across all metrics. We also present spatial domain distributions, UMAP visualizations, and trajectory inferences for the DLPFC dataset; detailed results are provided in the supplementary materials. Notably, on the human breast cancer dataset, IE-HERCL surpasses the second-best method, stDCL, by 4.66% in ARI and 6.12% in FMI. This outstanding performance can be attributed to the reinforced contrastive learning strategy, which mitigates the influence of negative samples, and the optimal transport mechanism, which optimizes the learning of latent representations, resulting in highly competitive outcomes. For additional experimental results, please refer to the supplementary materials.

Parameter Analysis. To further validate the sensitivity of our model to parameter settings, we conduct a series of parameter sensitivity experiments. These experiments evaluate the effects of the weight factors λ1, λ2 and λ3, which are set to [1, 10], [0.0001, 0.001] and [0.1, 1], respectively, and the random walk step size t, which varies from 1 to 15 with an interval of 2. Fixing λ1, λ2 and λ3 at 10, 0.001 and 1, Figure 3 demonstrates that the optimal step size t for the random walk is 9 for the above dataset. Furthermore, fixing the random walk step size at t=5, Supplementary Table 1 shows that the model achieves the best performance on the human breast cancer datasets when λ1, λ2 and λ3 are set to 10, 0.001 and 1, respectively. These results suggest that high-order random walks effectively mitigate the influence of negative samples, thereby improving clustering performance.

4.3 Ablation Studies

In this section, we evaluate the contribution of each component in IE-HERCL using the human breast cancer dataset. To verify the effectiveness of contrastive learning, we remove the Lc module (denoted as "w/o Lc"). To assess the impact of optimal transport in aligning latent representations, we exclude the Ls module (denoted as "w/o Ls"). We also evaluate the effect of mitigating negative samples in contrastive learning by removing the high-order random walks (denoted as "w/o Random Walk"). Additionally, we analyze the importance of the image data modality by removing it from the model and using only gene expression data for representation learning and clustering (denoted as "w/o Image Data"). As shown in Table 2, each component contributes significantly to the model's performance. The removal of any single component results in a noticeable decline in clustering performance. The best results are achieved when all components are included and the image data modality is utilized. These findings highlight the robustness and effectiveness of the proposed framework, particularly the synergy between its components.

5 Conclusion

In this work, we present IE-HERCL, a novel framework for spatial domain identification in spatial transcriptomics that seamlessly integrates gene expression, spatial information, and histological image data. By employing a hybrid encoding strategy with autoencoders and GraphSAGE, IE-HERCL captures non-spatial features and spatial dependencies within and across multimodal data. Cross-modal attention mechanisms further fuse these features into unified representations, enabling a comprehensive understanding of tissue organization. Additionally, we introduce a reinforced contrastive learning strategy that combines high-order random walks and optimal transport to reduce the impact of false negatives on representation learning and improve clustering alignment. Extensive experiments on five diverse spatial transcriptomics datasets demonstrate the superior performance of IE-HERCL in spatial domain identification and cell type classification, consistently surpassing state-of-the-art methods.

Acknowledgments

This study was partially supported by the National Natural Science Foundation of China under Grant Nos. 62372279, 62322215, 62172296, and the Natural Science Foundation of Shandong Province under Grant No. ZR2023MF119. It was also partially supported by the high-performance computing server of Central South University.