PDAR-RSITR: A Progressive Decoupling-Aggregation-Refinement Framework for Remote Sensing Image-Text Retrieval

1 Introduction

The emergence of massive remote sensing data has made efficient analysis methods a core issue [Zhou et al., 2023; Li et al., 2025], and how to extract key information from large-scale datasets has attracted widespread attention [Zhang et al., 2023; Ji et al., 2023; Zhang et al., 2025; Yuan et al., 2022b]. Remote Sensing Image-Text Retrieval (RSITR) is dedicated to establishing bidirectional mapping relationships between images and texts, achieving cross-modal semantic association, and serves as one of the core technologies for remote sensing information extraction and analysis [FernandezBeltran et al., 2021; Mao et al., 2018; Chen et al., 2020].

Traditional methods [Zheng et al., 2023; Zhang et al., 2023] rely on CNNs and RNNs for feature encoding and project the features into a shared embedding space to compute similarity for retrieval. However, these traditional backbones have limited encoding capabilities [Pan et al., 2023; Tang et al., 2024], making it difficult to handle remote sensing data containing multi-dimensional cognitive attributes, resulting in poor performance. Recently, researchers utilized CLIP [Radford et al., 2021] to obtain better features to enhance performance. For instance, PE [Yuan et al., 2023] integrates adapters to enhance encoder performance, while RSCLIP [Cha et al., 2025] and RemoteCLIP [Cha et al., 2025] improve representation capability through enhanced training data quality and larger dataset scales. Nevertheless, these methods lack mechanisms to comprehensively capture multi-dimensional cognitive attributes, thus constraining performance gains in RSITR task.

∗Corresponding author.

Remote sensing data contains multi-dimensional cognitive attributes such as Geometric Spatial Relationships (GSR) and Object Counting and Quantity Recognition (OCQR), whose entanglement with objects severely impedes accurate retrieval. As shown in Figure 1, existing methods struggle to handle these multi-dimensional cognitive attributes simultaneously. For instance, given the query "A regular football field is integrated with surrounding high-rise buildings," existing methods may capture the OCQR attribute (i.e., football field and buildings) but overlook the GSR attribute (i.e., surrounding), resulting in candidates where only some buildings are present around the football field. Similarly, for "three bridges on a river with many buildings on two sides of it," methods focus on GSR (i.e., bridges and two sides) but miss OCQR (i.e., three), resulting in candidates with only two bridges. This demonstrates that existing methods lack mechanisms to comprehensively capture multi-dimensional cognitive attributes, limiting retrieval accuracy.

To overcome the above challenges, we propose a novel Progressive Decoupling-Aggregation-Refinement framework for Remote Sensing Image-Text Retrieval, termed PDARRSITR. Specifically, inspired by the recent success of CLIPMoE [Zhang et al., 2025] in general vision-language models, we adapt Multi-dimensional Cognitive Decoupling to the remote sensing domain to learn features of cognitive attributes across different dimensions via multi-process clustering, which learns each attribute separately through a progressive process for in-depth understanding. Subsequently, we leverage Salient Cognitive Aggregation to select and aggregate the most salient attributes via dynamic routing, which efficiently extracts clear feature representations. Furthermore, we propose Expert Collaborative Refinement to enhance critical cross-modal relationships via three complementary expert perspectives, thereby improving retrieval accuracy through better cross-modal relationships. We conducted extensive experiments on four widely-used datasets to validate the effectiveness of our method. The main contributions are summarized as follows:

• We propose PDAR-RSITR, a novel Progressive Decoupling-Aggregation-Refinement framework. This framework effectively addresses the challenge of handling multi-dimensional cognitive attributes in remote sensing data.
• We integrate three synergistic stages: (1) adapting a multi-dimensional cognitive decoupling stage to deeply understand each attribute, (2) leveraging a salient cognitive aggregation stage to aggregate the most salient attributes, and (3) proposing a novel expert collaborative refinement stage to enhance critical cross-modal relationships.
• Extensive experimental results show that the proposed method achieves significant improvements in overall retrieval performance, and attains state-of-the-art results on multiple mainstream performance metrics.

2 RelatedWork

In recent years, the RSITR field has continued to make progress. A variety of methods, such as LW-MCR [Yuan et al., 2021], AMFMN [Yuan et al., 2022a], MSITA [Chen et al., 2023], and MSA [Yang et al., 2024], employ traditional backbones and continuously advance model architectural designs. However, due to the limited capabilities of these backbone networks, breakthroughs in performance metrics have not been achieved despite advances in model innovation. Consequently, researchers have begun adopting CLIP or its variants for this task, achieving superior results. PE [Yuan et al., 2023] integrated adapters into pre-trained CLIP models to achieve parameter-efficient transfer learning, further enhancing performance. RSCLIP [Cha et al., 2025]

collected approximately 9.6 million vision-language pairs to train a better model, while RemoteCLIP [Liu et al., 2024] generated a pre-training dataset 12 times larger than all previously available datasets combined to further boost model performance.

Although previous works have achieved impressive performance in RSITR, existing methods lack mechanisms to comprehensively capture multi-dimensional cognitive attributes, effectively select and aggregate the most salient attributes, and strategically enhance critical cross-modal relationships to enable accurate retrieval.

3 Method

The overview of our PDAR-RSITR is illustrated in Figure 2. We first introduce the problem formulation and preliminaries in Sec. 3.1 and the overall framework in Sec. 3.2. Then, we detail our proposed method across three synergistic stages: the Multi-dimensional Cognitive Decoupling Stage (MCDS) in Sec. 3.3 (Figure 2①), the Salient Cognitive Aggregation Stage (SCAS) in Sec. 3.4 (Figure 2②), and the Expert Collaborative Refinement Stage (ECRS) in Sec. 3.5 (Figure 2③).

3.1 ProblemFormulation and Preliminaries

For RSITR, given a dataset D = (Ii, Ti)Ni=1 containing N image-text pairs, our goal is to learn clear feature representations for retrieval between remote sensing images and text descriptions. Each image Ii and text Ti = (w1,w2, . . . ,wL) are mapped into a shared embedding space, where matched pairs have high similarity scores. RemoteCLIP follows the CLIP architecture: The image encoder first generates patch tokens Ipatch through patch embedding, then prepends a learnable classification token Icls and adds positional embeddings Ipos to obtain X0 I = [Icls; Ipatch] + Ipos . The text encoder tokenizes captions using byte-pair encoding, adding special tokens [SOS] and [EOS] to generate Ttoken = ([SOS],w1,w2, . . . ,wL, [EOS]) , then adds positional embeddings Tpos to obtain X0 T = Ttoken + Tpos. Subsequently, X0 I andX0 T are processed through L transformer blocks, with the ℓ-th layer represented as:

where X represents image or text features. Finally, the CLS token and EOS token serve as global image feature v and text feature u, respectively, which are linearly projected to a Ddimensional shared embedding space and L2-normalized. In general, the base model is trained using standard contrastive learning:

where τ is the temperature parameter.

3.2 Overall Framework

Our PDAR-RSITR framework consists of three synergistic stages. First, we utilize the adapted MCDS and SCAS to train the image and text encoders. The encoders comprehensively capture multi-dimensional cognitive attributes, select and aggregate the most salient features, and efficiently extract clear feature representations. During the inference phase, the trained encoders are employed to extract features for images and texts, and compute their initial similarity scores in the shared embedding space. Then enhance critical cross-modal relationships through ECRS, improving retrieval accuracy.

3.3 Multi-dimensional Cognitive Decoupling Stage

Inspired by multi-stage contrastive learning [Zhang et al., 2024], we progressively perform an inference-clusteringtraining process. Through iterative inference and clustering, we gradually disentangle the entangled attribute space to generate joint attribute labels. Different from prior methods, to adapt to the remote sensing domain, we utilize RemoteCLIP [Liu et al., 2024] instead of generic CLIP as the base model. Meanwhile, by fine-tuning during training, we construct a complementary expert ensemble (i.e., a set of RemoteCLIP models), enabling each expert to focus on different dimensional cognitive attributes. Specifically, we perform fine-tuning over K processes on the FFN layers of K RemoteCLIP models, where K denotes the total number of progressive processes and k represents the current process index (k ∈ {1, 2, ...,K}). Similarly, our consideration of the K value follows the number of salient cognitive attributes in remote sensing, as analyzed in [Adejumo et al., 2025]. These FFN layers are complementary and are finally integrated into the MoE architecture as experts. The progressive process of MCDS is as follows:

Progressive process k (k = 1): In this process, we rely solely on the inference of the base RemoteCLIP model and obtain initial attribute labels through clustering. Specifically, we extract image features v(1) i and text features u(1) i , respectively, and obtain image attribute labels y(1,I) i and text attribute labels y(1,T) i via K-means clustering. The joint attribute label is denoted as ˆ y(1) i = (y(1,I) i , y(1,T ) i ).

Progressive process k (k > 1): In these processes, we fine-tune only the FFN layers of the base RemoteCLIP model. First, we employ the RemoteCLIP model fine-tuned in process k - 1 to extract image and text features v(k) i and u(k) i , respectively, and perform clustering to obtain new attribute labels ˆ y(k) i . To fully exploit the attribute labels of samples across different processes, we accumulate the joint attribute labels obtained at each process to form cumulative joint attribute labels ˜ y(k) i = [ˆ y(1) i , ˆ y(2) i , . . . , ˆ y(k) i ]. Next, during the fine-tuning of the RemoteCLIP model in process k, we select negative samples only from the set of samples whose cumulative joint attribute labels are completely identical. For example, when image Ii serves as the anchor, the negative sample set is defined as

that is, texts whose attribute labels in the previous k - 1 processes are all consistent with the anchor but do not match the anchor itself. This design encourages the model to focus on attributes that were not distinguished in the previous processes.

The training loss function for each progressive process is as follows:

where s(k) ij = v(k)⊤ i u(k) j /τ and τ is the temperature parameter.

3.4 Salient Cognitive Aggregation Stage

Following the upcycling strategy in CLIP-MoE [Zhang et al., 2025], after all stages fine-tuning, we obtain a set of RemoteCLIP models to construct a complementary expert ensemble, and integrate these specially fine-tuned FFN layers into a MoE architecture:

After obtaining these complementary experts, the key challenge is how to dynamically select the most relevant expert subsets, extracting salient attributes to efficiently extract clear feature representations. Therefore, we employ an adaptive routing mechanism to address this issue. For each transformer layer ℓ, given input h, we compute expert selection probabilities through a learnable router:

where W(ℓ) r are the router parameters. To control computational cost, we select top-r experts: S = TopR(g(ℓ) , r) and renormalize the routing weights:

The final output is obtained through weighted combination of selected experts' outputs:

To prevent expert collapse where only a few experts are utilized, we use load balancing constraints. The training objective combines standard contrastive loss with a load balancing term:

where λ = 0.01 is the balancing weight. The load balancing loss encourages uniform expert usage across batches:

where f (ℓ) j = 1B ∑Bb=1 I[argmax(g(ℓ) b ) = j] represents the selection frequency of expert j in layer ℓ, and p(ℓ) j = 1B ∑Bb=1 g(ℓ) b,j represents the average gate probability of expert j for batch size B. Finally, an end-to-end global fine-tuning is applied to ensure global consistency across the framework.

3.5 Expert Collaborative Refinement Stage

To further improve retrieval accuracy, we recognize that initial rankings may overlook important cross-modal relationships. Inspired by the expert collaboration mechanism in MoE architectures, we introduce an ECRS that optimizes matching results from multiple perspectives by leveraging the ranking confidence expert, symmetry verification expert, and saliency assessment expert working collaboratively, thereby significantly enhancing retrieval accuracy.

Specifically, given the initial similarity matrix S. The ranking confidence expert directly utilizes the initial similarity scores to preserve the original cross-modal matching information:

where w1 is the confidence weight. The symmetry verification expert identifies high-quality matches through bidirectional retrieval consistency. This expert computes the intersection of the image-to-text top-E mask M(E) i2t and the textto-image top-LmaskM(L) t2i :

where ⊙ denotes element-wise multiplication. This mask only retains matching pairs that rank highly in both bidirectional retrievals, ensuring bidirectional consistency of matches. The output of this expert is:

where w2 is the confidence weight. The saliency assessment expert provides additional saliency enhancement to matching pairs that pass bidirectional verification, reinforcing distinctive matches with high discriminability:

where w3 is the confidence weight. This expert provides additional confidence boost to matching pairs that pass symmetry verification, effectively avoiding ambiguous matching results. Finally, we integrate the outputs of the three experts through a weighted fusion strategy to obtain the refined similarity matrix:

where ξ is the smoothing factor. Through the collaborative work of the three experts, this mechanism can effectively improve the accuracy of retrieval results.

4 Experiments

4.1 Datasets and EvaluationMetrics

To evaluate the effectiveness of the proposed method, we select four widely-used datasets: RSICD [Lu et al., 2017], RSITMD [Yuan et al., 2022a] and UCM [Qu et al., 2016] for retrieval performance evaluation, and RSMMVP [Adejumo et al., 2025] for expert complementarity assessment. For retrieval performance evaluation, we adopt Recall at K (R@K, where K = 1, 5, 10) [Lu et al., 2017] as the evaluation metric, and calculate R@K for both Image-to-Text (I2T) and Text-to-Image (T2I) retrieval tasks to measure the model's performance. In addition, we report the mean Recall (mR) as an overall performance indicator. For expert complementarity assessment, we use Blind Pair Accuracy (BPA) [Adejumo et al., 2025] as the evaluation metric. A prediction is considered successful only if the model makes correct judgments on both pairs of images.

4.2 Implementation Details

All experiments are conducted on four NVIDIA A100 80GB GPUs. Following the protocol in [Yuan et al., 2022a], the RSICD, RSITMD and UCM datasets are divided into training (80%), validation (10%), and test (10%) sets. The RSMMVP dataset is used exclusively as the test set. To ensure statistical validity, we exclude categories with insufficient samples (i.e., CRC and SPC, which contain only 1-3 pairs). It is worth noting that the original RSMMVP dataset was designed for visual question answering tasks; we adapt it for image-text retrieval by concatenating the relevant information.

4.3 ComparisonMethods

We compare our method against 15 state-of-the-art methods, including traditional methods (LW-MCR [Yuan et al., 2021], AMFMN [Yuan et al., 2022a], MSITA [Chen et al., 2023], and MSA [Yang et al., 2024]), as well as recent CLIP-based methods such as CLIP [Radford et al., 2021], Linear Probing [Radford et al., 2021], CoOp [Zhou et al., 2022], VPT [Jia et al., 2022], AdaptFormer [Chen et al., 2022], RSCLIP [Cha et al., 2025], PE [Yuan et al., 2023], LAPS [Fu et al., 2024], CLIP-Adapter [Gao et al., 2024], RemoteCLIP [Liu et al.,

2024], CLGSA [Chen et al., 2025].

Traditional methods (LW-MCR, AMFMN, MSITA, MSA) primarily focus on multi-scale feature encoding and finegrained alignment for cross-modal retrieval. For CLIPbased methods, Linear Probing adds trainable classifiers; CoOp and VPT introduce learnable prompts; AdaptFormer, CLIP-Adapter and PE insert lightweight adaptation modules; RSCLIP and RemoteCLIP are vision-language foundation models specifically designed for remote sensing; LAPS and CLGSA introduce new modules to enhance cross-modal interaction mechanisms.

4.4 Main Results

As shown in Tables 1 and 2, we compare the performance of our proposed method with existing methods on I2T and T2I retrieval tasks across three benchmark datasets. The results demonstrate that PDAR-RSITR consistently outperforms all baseline methods, achieving state-of-the-art performance in both retrieval directions. Compared to traditional methods, pre-trained vision-language models inherently exhibit significant performance improvements. Among CLIP-based methods, PDAR-RSITR achieves better results than other methods. This advantage is attributed to our method's ability to comprehensively capture multi-dimensional cognitive attributes, select and aggregate the most salient features, and efficiently extract clear feature representations. Moreover, it can enhance critical cross-modal relationships, thereby improving retrieval accuracy.

4.5 Ablation Study

To validate each component's effectiveness, we conduct ablation studies on RSICD, with results shown in Table 3. Without all components, the model degrades to the RemoteCLIP baseline. Without MCDS, four randomly fine-tuned experts yield negligible gains. Without SCAS, we use uniform weighting to integrate these specially fine-tuned layers into a MoE architecture. Although this approach achieves certain improvements, the performance is far inferior to that with SCAS. ECRS further improves performance through ranking confidence assessment, symmetry verification, and saliency evaluation. The complete PDAR-RSITR model achieves optimal performance, demonstrating that MCDS, SCAS, and ECRS synergistically contribute to superior retrieval accuracy.

4.6 Analysis of Expert Complementarity

We evaluated each expert individually on RSMMVP, as illustrated in Figure 3. The results reveal that Expert 1 excels at ODC, Expert 2 at SPC, Expert 3 at GSR, and Expert 4 at OD. OCQR is mastered by Experts 2, 3, and 4. When these experts are combined within PDAR-RSITR, the model achieves the highest accuracy across all attributes.

To verify PDAR-RSITR's ability to comprehensively consider multi-dimensional cognitive attributes, we conducted experiments on the RSMMVP dataset, encompassing Object Counting and Quantity Recognition (OCQR), Object Detection and Classification (ODC), Geometric Spatial Relationships (GSR), and Orientation Detection (OD), as shown in Figure 4. We compared PDAR-RSITR with MSA, PE, LAPS, and RemoteCLIP on the challenging BPA task. The results show that PDAR-RSITR achieves improvements across all attributes, while other methods fail with some reaching 0% accuracy.

4.7 Clustering Strategy Analysis

To evaluate clustering methods for attribute label generation, we conducted experiments on RSICD with K-means, Spectral Clustering, and Birch using K ∈ {2, 3, 4}, as shown in Figure 5. The experimental results demonstrate that all methods achieve peak performance at K = 3. Specifically, K-means lacks sufficient discriminability at K = 2, reaches optimal granularity at K = 3, and suffers from performance degradation due to over-fragmentation at K = 4. Spectral Clustering exhibits higher sensitivity to the K value, showing significant performance deterioration at K = 4 due to instability in eigenvalue decomposition. While Birch demonstrates stable performance, it consistently underperforms, constrained by the limitations of its tree-based strategy in high-dimensional spaces. K-means achieves the best performance, benefiting from its effective convex space partitioning capability and computational efficiency of O(nKd), which is significantly superior to the O(n3 ) complexity of Spectral Clustering.

4.8 RoutingMechanismAnalysis

To evaluate whether PDAR-RSITR sufficiently utilizes all experts learned through MCDS, we analyzed the adaptive routing mechanism learned in SCAS. We used a PDAR-RSITR model with K = 4 experts and top-2 routing trained on RSICD, and calculated the proportion of tokens assigned to each expert, as shown in Figure 6. As can be seen, each expert from the MCDS stages (each column in the heatmap) is relatively balanced, indicating that these experts are heavily utilized, which demonstrates that all MCDS stages provide useful experts.

4.9 Training and Inference Speed Comparison

To evaluate computational efficiency, Table 4 compares the training time, inference time, and accuracy of different methods on RSICD. While PE and RemoteCLIP offer fast inference, PE suffers from lower accuracy, and RemoteCLIP requires excessive training time. Conversely, MSA and LAPS are inefficient in both speed and performance. PDAR-RSITR achieves the highest accuracy with moderate training time. Its slight inference overhead compared to RemoteCLIP is due to MoE routing, which is a worthwhile trade-off for the significant performance gain.

4.10 Sensitivity Analysis of ECRS Parameters

To investigate the impact of ECRS expert weights (w1, w2, w3), we conducted a parameter sensitivity analysis on the RSICD as shown in Figure 7. Results show that w1 (ranking confidence) has the most significant impact: performance steadily improves from 0.5 to 1.5 but declines at 2.0, indicating over-reliance on initial scores weakens multi-expert collaboration. The weights w2 (symmetry verification) and w3 (saliency assessment) show moderate and minimal effects respectively. The optimal configuration w1 : w2 : w3 = 1.5 : 1.0 : 1.0 balances all three experts and serves as a robust default for different scenarios.

5 Conclusion

This paper presents PDAR-RSITR, a novel framework that addresses the challenge of comprehensively capturing multidimensional cognitive attributes in RSITR. We integrate three synergistic stages to comprehensively capture multidimensional cognitive attributes, select and aggregate the most salient features, and efficiently extract clear feature representations. Moreover, our method can enhance critical cross-modal relationships, thereby improving retrieval accuracy. Extensive experiments on four widely-used benchmark datasets demonstrate that PDAR-RSITR achieves state-ofthe-art performance across multiple mainstream metrics.

Ethical Statement

There are no ethical issues.

Acknowledgements

This work was supported by the Beijing Municipal Science and Technology Program (Z251100003625011), the Beijing Natural Science Foundation (L252018), the NSFC through grant No.62402054, the China Postdoctoral Science Foundation through grant 2024M760279, and the Postdoctoral Fellowship Program and China Postdoctoral Science Foundation under grant BX20250390.