G-SalAlignMamba: Geometry-Aware Vision Mamba for Dual-Modal Salient Object Detection

1 Introduction

Dual-modal Salient Object Detection (SOD) aims to robustly identify visually attractive regions by integrating RGB images with auxiliary modalities (e.g., Depth or Thermal). This field has primarily relied on Convolutional Neural Networks (CNNs) [Fan et al., 2020a] or Transformer architectures [Liu et al., 2021]. Although these models have achieved significant progress, CNNs are limited by local receptive fields that restrict their ability to model complex global contexts. Transformers possess global receptive fields, but face computational costs that grow quadratically with resolution. To better balance computational efficiency and global modeling, Visual State Space Models (SSMs), represented by VMamba [Liu et al., 2024b], have recently garnered widespread attention for their ability to model long-range dependencies with linear computational complexity O(N) [Gu and Dao, 2024]. While promising, we argue that the direct application of VMamba to dual-modal SOD is constrained by three critical limitations rooted in current methodologies, as shown in Figure 1:

∗Corresponding authors.

1. Misalignment Vulnerability in Mamba Architecture. Most pioneering Mamba-based fusion frameworks follow a simple "concatenate-then-scan" paradigm [He et al., 2025], while leaving cross-modal geometric misalignment insufficiently addressed. Since Mamba relies on a continuous sequential scanning order to update hidden states, physical spatial shifts caused by parallax or calibration errors can disrupt token correspondence within the scanning sequence. This Misalignment Vulnerability makes simple fusion strategies sensitive to spatial mismatches between modalities, causing the SSMmodel to construct inaccurate dependencies and accumulate errors.

1. Signal Dilution Caused by General Scanning Mechanisms. General Vision Mamba backbones (e.g., ViM [Zhu et al., 2024], VMamba [Liu et al., 2024b]) are largely designed for image classification tasks, where feature distributions are assumed to be relatively uniform. However, in SOD tasks, salient objects typically occupy only a sparse fraction of the image. General raster scanning mechanisms treat all tokens indiscriminately, lacking a discriminative bias towards target regions. In the recursive propagation mechanism of SSMs, the current state depends on the accumulation of all historical information. This implies that sparse foreground signals must traverse extremely long spatial sequences. During this long-range propagation, effective information is prone to being "diluted" or overwhelmed by the vast majority of background noise. Without task-specific modulation, the Mamba encoder easily fails to maintain the prominence of salient features in deep layers.

3. Boundary Degradation During Sequence Reconstruction. Existing Vision Mamba variants also face a "coupling gap" between the SSMs' sequence representation and the integrity of the image's 2D physical structure during resolution recovery. Current decoders in general Vision Mamba models still adhere to traditional practices in vision tasks, employing spatially-agnostic implicit upsampling operators (e.g., bilinear interpolation) [Zhu et al., 2024; Liu et al., 2024b]. These operations lack awareness of Mamba's scanning characteristics and fail to utilize the geometric alignment cues captured during the encoding stage. This dimensional projection distortion during the "sequence-to-space" reconstruction not only triggers pixel-level Correspondence Confusion, but also leads to blurred edges and structural deterioration of salient targets, forming a structural bottleneck for accurate SOD [Qin et al., 2019; Zhao et al., 2019].

To address these interconnected issues, we propose GSalAlignMamba, a unified geometry-aware framework tailored for dual-modal SOD. Guided by an AlignmentGuidance-Preservation philosophy, we first introduce a Geometry-Aware Encoding scheme with coarse-to-fine explicit alignment to enforce physical synchronization, resolving the misalignment vulnerability. Second, a SemanticsInformed Refinement strategy is proposed to modulate the scanning process via a top-down feedback loop, preventing signal dilution. Finally, a Structure-Preserving Decoding scheme is presented for extending geometric constraints to the upsampling stage, ensuring sharp boundary recovery without explicit supervision. The code is available at https://github.com/PC1-99/G-SalAlignMamb.git.

Our main contributions are summarized as follows:

• We first identify the intrinsic Misalignment Vulnerability of Mamba's sequential modeling. To address this, we propose G-SalAlignMamba, pioneering a GeometryAware Encoding scheme that enforces physical syn-
• chronization to construct a robust foundation for multimodal SSMs.
• We propose a Semantics-Informed Refinement strategy to counter the Signal Dilution inherent in generic scanning. By constructing a top-down semantic feedback loop, we empower the encoder to prioritize sparse salient signals, preventing them from being overwhelmed by background noise.
• We design a Structure-Preserving Decoding scheme to bridge the geometric gap in conventional upsampling. This module enforces geometry-consistent reconstruction with an unsupervised boundary refinement mechanism, effectively solving Boundary Degradation without explicit supervision.
• Extensive experiments on RGB-D and RGB-T benchmarks demonstrate that G-SalAlignMamba achieves state-of-the-art performance, validating the necessity of geometric constraints in Mamba-based fusion.

2 RelatedWork

2.1 Visual Mamba

The success of SSMs has driven their extension to the visual domain, where Vision Mamba (ViM) [Zhu et al., 2024] and VMamba [Liu et al., 2024b] achieve efficient linearcomplexity modeling for fundamental recognition tasks. This advantage has rapidly sparked widespread adoption across dense prediction fields [Xing et al., 2024]. In the context of SOD, recent pioneers like Samba [He et al., 2025] have also validated the potential of Mamba for maintaining spatial continuity. However, current works primarily focus on single-modal representations or simple fusion strategies, usually overlooking the inherent cross-modal spatial and geometric disparities in dual-modal settings. This limitation inspires us to propose a geometry-aware framework that explicitly incorporates alignment mechanisms into the VMamba architecture to suppress misalignment noise.

2.2 RGB-Depth Salient Object Detection (RGB-D SOD)

To address RGB-only SOD limitations, many approaches incorporate depth information, named RGB-D SOD, which aims to enhance the model's geometric and 3D structural understanding. Depth helps detect salient objects when color/texture contrasts are weak. A representative work [Fu et al., 2021] used Siamese networks with joint learning and densely-cooperative fusion, improving robustness. Another method [Ji et al., 2021] tackled depth noise by using depth calibration followed by cross-modal fusion, boosting saliency detection. However, existing methods still struggle with lowquality depth maps and cross-modal spatial misalignment.

2.3 RGB-Thermal Salient Object Detection (RGB-T SOD)

RGB-T SOD fuses RGB and thermal imagery, particularly useful in low-light, nocturnal, or high-thermal-contrast environments. RGB-T methods utilize multi-stream networks, attention-based fusion, and multi-scale decoding to integrate RGB and thermal information [Pang et al., 2023]. For example, [Zhou et al., 2023] has shown that modeling pixelwise positional relations and using Transformer-based encoders and decoders leads to more accurate saliency masks in challenging thermal and visible scenarios. Thermal data significantly enhances performance when RGB data alone is insufficient. Nevertheless, they often fail to handle severe parallax and thermal noise interference in complex scenes [Tu et al., 2022a].

3 Method

To systematically resolve the issues of spatial misalignment and feature degradation in Mamba-based SOD, given paired RGB and auxiliary inputs IR, IA ∈ R3×H×W , we propose a geometry-aware dual-modal SOD framework to address cross-modal misalignment and signal dilution, as illustrated in Figure 2. The encoder employs a dual-branch Visual State-Space (VSS) backbone with lightweight Early Interaction to stabilize intermediate features, followed by a Coarse-to-Fine Explicit Alignment module that corrects spatial shifts via multi-scale displacement fields and a Complementary Gated Fusion block to produce a deeply fused representation Ffused. The refinement module applies SemanticsInformed Refinement to shallow features, modulating them via top-down Saliency Priors derived from Ffused to prioritize foreground signals. Finally, the Structure-Preserving Decoding scheme progressively reconstructs the saliency map by integrating Context-Aware Alignment Upsampling with unsupervised Multi-Scale Boundary Refinement, ensuring precise structural recovery and sharp edges essential for highperformance SOD.

3.1 Geometry-Aware Encoding: Cross-Modal Alignment and Fusion

Hierarchical Feature Extraction with Early Interaction

To extract robust representations from RGB image IR and auxiliary input IA, we employ a dual-branch encoder based on VSS architecture. Different from CNNs, VSS captures long-range dependencies with linear complexity O(N).

We share parameters θVSS between branches to learn modality-agnostic geometric representations and reduce parameter overhead as:

To prevent semantic divergence in deep layers, we introduce a lightweight Early Interaction mechanism. At stage i = 2, intermediate features are stabilized via window-based cross-attention as:

where gearly = σ(γearly) is a learnable gate with γearly (initialized to weakly inject cross-modal information to prevent early degradation). This ensures that the subsequent alignment module operates on features with compatible semantic levels.

Coarse-to-Fine Explicit Alignment

Implicit alignment in SOD often struggles with parallaxinduced displacements, leading to blurred boundaries. Hence, we propose a Coarse-to-Fine Explicit Alignment strategy that predicts a pixel-wise displacement field ∆p through multi-scale feature extraction and cross-attention fusion.

To capture misalignment at varying magnitudes, we construct a feature pyramid at three resolutions (original, 1/2, and 1/4) by projecting the stage-4 representations via scalespecific convolutions. For each scale s ∈ {0, 1, 2}, these multi-scale features are subsequently enhanced using crossattention to yield the fused representation F(s) fused . The final displacement field ∆p is an adaptive aggregation of multiscale predictions as:

where ws are learnable weights derived from Softmax. The auxiliary features are then explicitly aligned to the RGB coordinate system via differentiable warping W:

where ∆p is bounded by a learnable maximum offset αmax through tanh normalization. The explicit rectification eliminates geometric discrepancies, ensuring the spatial consistency critical for preserving sharp boundaries.

Complementary Gated Fusion

To address the failure of naive fusion in dual-modal SOD caused by indiscriminately treating noisy inputs, we propose a Complementary Gated Fusion block to balance shared and complementary cues:

1. Commonality Path: We integrate shared semantics using concatenation and Selective Scanning via the Concat Mamba Fusion Block:

This path concatenates RGB and aligned auxiliary features, then sequentially processes them through linear projection, reshape, depthwise convolution, Saliency-Guided SS2D (SGSS2D) [He et al., 2025], layer normalization (LN), and a final reshape to enhance feature representation.

1. Difference-Aware Path: To explicitly model complementary signals, we compute feature differences between the RGB and aligned auxiliary modalities across multiple scales. The difference maps are encoded, gated via adaptive thresholds, and processed by lightweight SS2D blocks, before being upsampled and fused into the final difference representation Fdiff. A learnable gating mechanism dynamically balances these paths as:

where g = σ(γgate) with γgate (initialized to bias towards the commonality path). The residual connection with FR ensures gradient flow and feature preservation. Consequently, this mechanism adaptively exploits complementarity with suppressing noise, which could ensure robust SOD performance even with low-quality inputs.

3.2 Semantics-Informed Refinement: Mitigating Signal Dilution

In general Vision Mamba architectures, the sequential scanning mechanism treats all spatial tokens equally. However, salient objects often occupy a small fraction of the image in SOD tasks. Hence, general methods may lead to signal dilution, where strong foreground signals are overwhelmed by background tokens during state transitions. We address this issue via a feedback-driven refinement scheme.

Deep Saliency Feedback Mechanism

We first introduce a top-down feedback loop inspired by cognitive attention. We utilize the deeply fused representation F(4) R (from the final encoder stage) to generate a coarse Saliency Prior S to guide shallow and high-resolution features.

Formally, we generate a raw prior Sraw by projecting and upsampling the deep semantic features, followed by spatial smoothing via a Holistic Attention (HA) module to ensure coherence, ultimately yielding the final coarse saliency prior S. The prior is then resized and transformed into a soft modulation mask via temperature scaling to guide the refinement:

where HA(·) denotes the Holistic Attention module that applies Gaussian smoothing for spatial coherence. This method leverages high-level semantic context to modulate low-level details, ensuring the enhancement module focuses computational resources on potential foreground regions before features propagate to the decoder.

Saliency-Weighted Feature Modulation

General state-space models' equal token treatment causes feature dilution in SOD tasks, where background often overwhelms sparse foregrounds. We propose a SaliencyWeighted Modulation strategy that utilizes a saliencyderived mask to modulate scanning, preserving foreground signals without altering sequence length.

For an input feature F(i) R at stage i ∈ {1, 2, 3}, the saliency prior S is resized to match the specific spatial resolution (Hi,Wi), yielding the mask M(i) . The mask is shaped using temperature scaling as:

where τ is the temperature parameter and γ is the power scaling factor. Guided by the soft mask M(i) soft , the input features F(i) R are processed by the Saliency Mamba Block-comprising linear projection, reshape, DWConv, SGSS2D [He et al., 2025], LN, and a final reshape to yield the refined representations.

The soft modulation ensures that the hidden state of the SSMs is primarily influenced by informative foreground signals, thereby preserving the integrity of salient features with suppressing background noise to yield the prior F(i) refined . The saliency-weighted modulation adds negligible computational overhead, maintaining inference efficiency comparable to general scanning approaches.

Additionally, to enhance channel specificity, we integrate a unified channel attention module Achannel (combining SE [Hu et al., 2018] and ECA mechanisms [Wang et al., 2020]) within the Saliency Mamba Block as:

where α is a learnable balancing parameter, enhancing the signal-to-noise ratio for precise and complete SOD results.

Adaptive Feature Integration

Finally, to regulate the information flow to the decoder, we employ an Adaptive Feature Integration mechanism. The refined features F(i) refined are modulated by a learnable gating factor σ(γi). This method allows the network to dynamically determine the importance of the refined features, selectively accepting informative signals with suppressing any residual artifacts or noise before decoding, ultimately producing the integrated feature representation F(i) out .

3.3 Structure-Preserving Decoding: Precise Upsampling

General decoders often suffer from feature blurring during upsampling, which is particularly detrimental to SOD, where precise boundary delineation is critical for high evaluation metrics. To address this issue, we propose a structurepreserving scheme that enforces explicit alignment. The decoder restores the resolution of the deepest encoder feature Ffused to the original scale.

Context-Aware Alignment Upsampling

In SOD tasks, spatial misalignment in skip connections often leads to boundary ghosting. The decoding process operates over three stages i ∈ {1, 2, 3}. At each stage, the module simultaneously processes the current decoder feature F(i) dec and the skip-connected feature F(i) skip from the encoder's shallow layers. Specifically, F(i) dec ∈ RB×Ci×Hi×Wi denotes the feature from the previous decoding stage (initialized as Ffused for i = 1). Correspondingly, F(i) skip ∈ RB×C(i) skip ×Hi×Wi represents the skip feature derived from encoder stage (4-i) (i.e., F(3) out , F(2) out , or F(1) out for i = 1, 2, 3 respectively).

To maintain feature correspondence with recovering details, we introduce a Context-Aware Alignment Upsampling mechanism. We first project the concatenated decoder and skip features into node embeddings N(i) and refine them via anisotropic graph propagation to capture boundary cues:

where β is a learnable diffusion strength, and D = {up, down, left, right} denotes the set of four cardinal neighbor directions used for anisotropic propagation. The refined nodes guide the prediction of a dense offset field ∆p(i) , which aligns the skip features to the decoder's coordinate system via differentiable warping W:

The explicit spatial shift ensures feature correspondence and preserves object structure during upsampling. Besides, this rectification guarantees the spatial precision required for high-quality SOD.

Contextual Sequence Reconstruction

In the decoding phase, the Contextual Sequence Reconstruction block (AlignedMambaBlock) plays a vital role in recovering spatial details. It takes the upsampled decoder feature F(i) dec and the explicitly aligned skip-connection feature F(i) aligned as inputs. Under the guidance of the resized saliency prior Psal, these features are fused to model global context and dependencies. The reconstructed sequence is then expanded and upsampled via a PixelShuffle operation to generate the high-resolution feature F(i) up for the subsequent stage, aiming to preserve the semantic completeness in SOD tasks.

Multi-Scale Boundary Refinement

Preserving high-frequency details is critical for distinguishing salient foregrounds in SOD tasks. To avoid boundary degradation, we employ an unsupervised Multi-Scale Boundary Refinement module that implicitly sharpens highfrequency details without requiring explicit edge supervision. The module operates by extracting local boundary cues from F(i) up using parallel convolutional branches at three distinct scales (k ∈ {1, 2, 4}). These multi-scale boundary features are subsequently upsampled, concatenated, and fused back into the main feature stream via a residual connection. This process yields the final refined output F(i) refined , characterized by sharper edges and clearer structural delineations.

Loss Functions

To stabilize training, we apply L2 regularization on the offset fields. The total loss combines the structure loss Lstructure for both main and auxiliary outputs with the offset regularization:

where no explicit boundary supervision is required. This composite loss drives structural accuracy, ensuring robust convergence for SOD tasks.

4 Experiment

4.1 Datasets andMetrics

For RGB-D SOD, we use five benchmark datasets: NJUD [Ju et al., 2014], NLPR [Peng et al., 2014], SIP [Fan et al.,

2020a], STERE [Niu et al., 2012] and DUTLF-D [Piao et al., 2019]. For RGB-T SOD, we adopt seven datasets: VT5000 [Tu et al., 2022b], VT1000 [Tu et al., 2019], VT821 [Wang et al., 2018], un-VT5000 [Tu et al., 2022a], un-VT1000 [Tu et al., 2022a], un-VT821 [Tu et al., 2022a] and UVT2000 [Wang et al., 2024]. For RGB-D and RGB-T SOD, we employ three standard metrics: E-measure [Fan et al., 2018] (Em) jointly assesses pixel-level accuracy and image-level consistency; S-measure [Fan et al., 2017] (Sm) evaluates structural similarity; and F-measure [Achanta et al., 2009] (Fm) balances precision and recall.

4.2 Implementation Details

Our method is implemented in PyTorch and trained on an NVIDIA GeForce RTX 4090 GPU. We train separate models for the RGB-D SOD and RGB-T SOD tasks, respectively. Following previous works, we arrange the training sets for each task as follows: the training sets of NJUD, NLPR, and DUTLF-D for RGB-D SOD; the training sets of VT5000 and UVT2000 for RGB-T SOD. For the specific hyperparameter settings, we initialize the learnable gating factors γearly and γgate to -2.0, while the adaptive integration factor γi is initialized to 2.0. The maximum offset for explicit alignment is bounded by αmax = 3.0. Both the temperature parameter

τ and the power scaling factor γ are set to 1.0. The weight λ in the loss function is set to 0.01. In the training process, we adopt the AdamW optimizer with an initial learning rate of 1 × 10-4 and a batch size of 2. All input images are uniformly resized to 448 × 448 for training and testing. The model converges after 30 training epochs.

4.3 Model Comparison

Quantitative Evaluation. To demonstrate the universality of our unified framework in dual-modal SOD, we compare it with 12 RGB-D and 11 RGB-T state-of-the-art methods, including the representative Mamba-based Samba baseline, in Table 1 and Table 2. Overall, G-SalAlignMamba delivers consistently competitive or superior performance across all three metrics.

For RGB-D (Table 1), our model gives the best Sm scores on all five datasets and remains highly competitive in Fm and Em; compared with Samba, it improves NJUD from [0.945, 0.938, 0.969] to [0.956, 0.948, 0.983] across (Sm, Fm,Em), showing the benefit of explicit crossmodal alignment over "concatenate-then-scan" fusion. For RGB-T (Table 2), it also achieves state-of-the-art results, improving VT5000 over Samba from [0.924, 0.904, 0.949] to [0.939, 0.913, 0.956] and yielding clear gains on misaligned UVT2000 (+7.3% Sm, +1.5% Fm, +2.5% Em). Benefiting from Mamba's linear complexity O(N), G-SalAlignMamba balances accuracy and cost with 83.80M parameters, about 61%-68% fewer than CPNet and CATNet, while maintaining real-time inference at 30.41 FPS and outperforming heavy Transformer baselines such as SwinNet-B (11.42 FPS) in speed.

Qualitative Evaluation. Figure 3 compares our method with state-of-the-art approaches on RGB-D (top) and RGBT (bottom) benchmarks. G-SalAlignMamba yields sharper boundaries and clearer object delineation under complex RGB-D topologies (e.g., 2nd row) or noisy depth (e.g., 3rd row), while suppressing artifacts and preserving fine-grained structures for thin instances (5th , 8th rows) and small objects (6th row) in RGB-T scenes with thermal degradation.

4.4 Ablation Study

We conduct ablations on NJUD, NLPR, VT5000, and UVT2000 to quantify the individual and combined contributions of all modules in G-SalAlignMamba.

Geometry-Aware Encoding. We use a strong Base with a pre-trained VMamba backbone, channel-concatenation fusion, and bilinear upsampling, excluding our geometry-aware and refinement modules. Table 3 then examines Early Interaction (EI), Coarse-to-Fine Explicit Alignment (CFEA), and Complementary Gated Fusion (CGF), testing them independently (E1-E3), in combinations, and finally as the full encoder (E5).

Semantics-Informed Refinement. For the refinement stage, we decouple Deep Saliency Feedback Mechanism (DSFM), Saliency-Weighted Modulation (SWM), and Adaptive Feature Integration (AFI), reporting their individual effects (H1-H3), feedback-modulation synergy (H4), and the complete refinement scheme (H5).

Structure-Preserving Decoding. Finally, we progressively add Context-Aware Alignment Upsampling (CAAU), Contextual Sequence Reconstruction (CSR), and Multi-Scale Boundary Refinement (MBR), assessing isolated components (D1D3) and stepwise integration (D4-D5). The full decoder achieves the best spatial alignment and boundary quality compared with general methods.

5 Conclusion

In this paper, we proposed G-SalAlignMamba, a geometryaware framework for dual-modal SOD that targets three key issues: Misalignment Vulnerability, Signal Dilution, and Boundary Degradation. Despite the promising performance on RGB-D and RGB-T benchmarks, our proposed G-SalAlignMamba may still encounter challenges in scenarios with extreme occlusion or highly cluttered backgrounds, where the target features are severely obstructed in both modalities. Future work will focus on incorporating stronger semantic priors to improve robustness in these edge cases.

Acknowledgments

This work was partially supported by the National Natural Science Foundation of China (Grant No. 62401226) and the Hong Kong Metropolitan University Research Grant (Project Reference No. RD/2025/2.6).

Contribution Statement

Haixiao Gao and Yimin Zheng contributed equally to this work. Haixiao Gao and Yimin Zheng developed the main methodology and conducted the experiments. Mengke Song and Zhi-Ri Tang supervised the research and revised the manuscript. Linyou Xiao and Tian-Tian Zhang contributed to the analysis and manuscript preparation.