VFM-Dynamo: Accurate Non-rigid Motion Identification via Vision Foundation Model for Self-supervised Monocular Depth Estimation

1 Introduction

Monocular depth estimation (MDE) plays a fundamental role in computer vision, with broad applications in autonomous navigation, augmented reality, and video analysis. Previous supervised methods requires ground-truth depth annotations for training, whose collection is expensive and timeconsuming. Consequently, self-supervised methods leverage synchronized stereo-pairs [Godard et al., 2017] or geometric constraints [Zhou et al., 2017] from monocular video as supervision signals, making monocular training more attractive.

* Corresponding author.

Current training pipeline of self-supervised MDE methods typically minimize geometry consistency loss to jointly estimate depth and camera pose [Godard et al., 2019; Chang and Wang, 2025; Zheng et al., 2025]. This pipeline relies on rigid geometric projection under static scene assumption. Despite overall high performance, the inconsistent geometric locations of moving objects lead to misaligned pixel matching, providing no longer valid supervision signals. To address this problem, a critical step is identifying dynamic objects from the rigid background. Several works leveraged semantic labels [Klingner et al., 2020; Lee et al., 2021; Dong et al., 2024], optical flow [Yin and Shi, 2018; Liu et al., 2019; Miao et al., 2023; Zhou et al., 2025], geometry consistency [Bian et al., 2019; Godard et al., 2019] or the discrepancy between single-frame and multi-frame depth [Watson et al., 2021; Woo et al., 2024] to discern dynamic objects. However, semantic labels involve needless static objects whereas geometry/discrepancy-induced methods fail to guarantee the accuracy in modeling dynamic objects. More fundamentally, motion identification is a complex and challenging problem that has been studied for decades, but there are still open questions [Goli et al., 2025].

Recent research in object identification has been transformative, with the vision foundation model (VFM) such as Grounding DINO [Liu et al., 2024b] and SAM [Kirillov et al., 2023] emerging as a significant breakthrough. However, both of them face identification challenges if objects are momentarily motionless, and in distinguishing foreground objects from background 'noise'. This naturally raises the question:"Can we leverage the power of VFM for non-rigid moving object identification and advance self-supervised MDE ?"

As shown in Figure 1, we propose VFM-Dynamo, a progressive motion identification pipeline inspired by Human Visual Perception (HVP) for self-supervised MDE, which simulates HVP process and effectively tailors VFM from global perception to fine-grained attention. Specifically, as in early HVP stages, by computing cross-modal correlations between visual features and text embeddings, we first drive Grounding DINO to derive reliable object boxes, achieving global perception of potential moving objects. Second, we design rigid consistency reasoning to derive coarse motion clues, acting as a selector of moving regions within a frame. This strategy decouples rigid flow in two different manners and penalizes the inconsistency in the predicted rigid flow between them, indirectly providing sufficient regularization for dynamic objects. We then introduce non-dynamical suppression, which leverages coarse motion identification to filter out static object boxes. Finally, the remaining dynamic object boxes serve as prompts for SAM to refine motion identification, further optimizing the joint training of depth and optical flow network. In addition, aim to alleviates the edge-blurring problem in the traditional upsampling method, we leverage the spatial coherence in neighbor pixels and propose a learnable neighbor affinity interpolation (LNAI) module to estimate the interpolation pixels from a lower resolution directly. The main contributions of our work are summarized as follows:

• To the best of our knowledge, our framework is the first work that tailors VFM for accurately non-rigid motion identification in self-supervised MDE. Inspired by HVP, our approach introduces a coarse to fine learning framework that leverages the advantage of VFM and flow to mitigate motion bias in training process.
• We propose an upsampling strategy that is compatible with existing self-supervised MDE pipelines and achieves significant performance improvements.
• We evaluate VFM-Dynamo framework on a serious of widely used datasets and establish a new state-of-the-art performance in dynamic areas.

2 RelatedWork

Self-Supervised Depth Estimation. Self-supervised methods mitigate the reliance on expensive 3D sensors and compute geometric consistency between stereo images [Godard et al., 2017] and monocular video frames [Zhou et al., 2017] for training. Since Monodepth2 [Godard et al., 2019] suggested a new benchmark with minimum reprojection loss, many advances adopt its methodology to improve depth predictions.

MonoMixer [Chang and Wang, 2025] compensated for the limited receptive field of convolutions with Transformer, improving performance while maintaining a lightweight architecture. DiffSQL [Zheng et al., 2025] leveraged the generative priors of stable diffusion for feature augmentation. Mono-ViFI [Liu et al., 2024a] incorporated a flow-based video frame interpolation model for temporal augmentation. Dynamic Scenes. To address the inaccurate training signals in dynamic scenes, identifying non-rigid motion from the rigid background is an essential step. MonoDepth2 [Godard et al., 2019] introduced an auto-masking for moving objects. SC-Depth [Bian et al., 2019] and SC-DepthV3 [Sun et al., 2023] derived motion segmentation from the proposed self-discovered mask. SGDepth [Klingner et al., 2020] and DynamicDepth [Feng et al., 2022] employed precomputed segmentation masks to identify objects. DFNet [Zou et al., 2018] and [Liu et al., 2019] achieved this through the forward-backward consistency and rigidity reasoning of optical flow. Dynamo-Depth [Sun and Hariharan, 2023] jointly learned depth, optical flow, and motion segmentation to disambiguate dynamic motion. ManyDepth [Watson et al., 2021] and ProDepth [Woo et al., 2024] exploited the inconsistency between single-frame and multi-frame depth. While these methods are insightful, the reliance on expensive semantic labels often includes irrelevant static objects, and other approaches struggle to accurately estimate the motion of all dynamic objects in the real world.

VFM-based Depth Estimation. VFM, such as visionlanguage models (VLMs) [Radford et al., 2021], Grounding DINO [Liu et al., 2024b] and SAM [Kirillov et al., 2023] are pre-trained on massive amounts of pairs of texts and images and has demonstrated powerful scene-reasoning abilities in computer vision tasks. DepthCLIP [Zhang et al., 2022] is a pioneering attempt that leveraged the pretrained knowledge of CLIP for zero-shot depth estimation. Furthermore, subsequent studies [Hu et al., 2024] and [Son and Lee, 2024] revealed that learnable prompts achieve better performance compared to manually designed templates. Hybriddepth [Zhang et al., 2025b] enhanced feature representation with aggregated multi-grained features from CLIP and DINO under contrastive language guidance. They are still limited to ambiguity introduced by the moving objects. Our work extends the robust VFM framework to non-rigid motion identification for self-supervised MDE by adapting it with coarse to fine framework and dynamic prompts.

3 Method

We focus on solving for the depth estimation in the dynamic scene observed from a moving camera. Figure 2 presents an overview of our method. In the following, we will describe specific components in details.

3.1 Motivation and Background

Unsupervised Depth and Optical Flow. Unsupervised depth and flow estimation both rely on photometric consistency between the synthesized image ˆ I and the original image I, formulated as:

where ρ is a measure of similarity calculated by L1 and SSIM, Θ is the model with learnable parameters.

Given a target image It and source image Is, s ∈ {t - 1, t + 1}, depth and optical flow synthesize image in fundamentally different ways. In terms of depth, the image ˆ I can be synthesized by predicted depth Dt and camera pose Tt→s:

where p is the pixel coordinates and K is camera intrinsics. In terms of optical flow, the image ˆ I can be synthesized by predicted flow field Ft→s:

In this case, optical flow reconstructs images via per-pixel motion warping, which naturally accommodates dynamic objects, whereas self-supervised depth estimation relies on rigid geometric projection/epipolar constraints, making it sensitive to violations caused by dynamic objects. Therefore, as optical flow is adept at tracking the movement of objects, this advantage motivates us to explore how such motion cues can be exploited during self-supervised depth training to provide effective regularization for dynamic regions.

Human Visual Perception. In visual cognition, humans tend to follow a global-to-local perception paradigm, in which a visual image is first perceived as an integrated whole and subsequently analyzed in terms of its constituent parts [Xia et al., 2024]. This observation motivates us to investigate whether we can mimic the staged process of HVP: global scene understanding to motion clues-driven region localization and refined non-rigid motion identification.

3.2 Coarse to Fine Motion Identification

To properly exploit the advantages of optical flow for dynamic objects, we reason rigid flow of static scene in two different manners and enforce consistency constraints between them. This process depends on accurate identification of moving objects, which is hindered by the complexity and time-varying nature of object motion. Inspired by HVP, we introduce VFM and design a coarse to fine motion identification strategy to effectively address this issue.

Overall Understanding. As in early HVP stages, we harness open-vocabulary 2D detector-Grounding DINO [Liu et al., 2024b] to access the overall scene understanding. Relevant text prompts, such as "car, people, motorcycle, bicycle,...," are defined and fed into trainable text encoder to output text embeddings T = [T1, ..., TN] ∈ RN×C . To ensure that the text embeddings are more similar to its corresponding visual features, text-visual similarity is computed as:

where El t ∈ RC×H×W is visual features from the last layer of DepthNet encoder. ⊗ is einstein summation over channel dimension. We normalize it along spatial dimension and aggregate visual features to derive text-guided visual features:

where E′t represents the aggregated visual features corresponding to each text token.

Finally, we adaptively update the text embeddings via a gated residual fusion:

where σ(·) denotes the sigmoid function. Then the vision conditioned text embeddings ˆ T are adopted to predict bounding boxes Bobj = {bi}Ni=1 for all referred objects.

Non-rigid Motion Awareness. We input It into the depth network D to estimate the depth Dsi t and concat It and Is as input to the PoseNet P, from which we estimate the relative camera pose Tt→s. Then the rigid flow that purely induced by the camera motion can be obtained:

where F rig1 t→s describes the motion of a rigid scene projected onto image plane. Afterwards, we cascade ResFlowNet R to learn non-rigid residual that capture object motion:

where ˆ I optr t denotes the synthesize image calculated by F rig1 t→s , The final full flow prediction F full1 t→s = F rig1 t→s +F non t→s . According to Equation 2 and 3, the synthetic image ˆ I d t and ˆ I opt t are reconstructed usingDsi t and F full1 t→s respectively, and then calculate the difference with the target image:

where α is set to 0.85. Instead of averaging differences over all source images, we use the minimum and predict coarse motion mask:

where [ ] is the Iverson bracket. In cases where binary mask Mc is 1 in regions considered to be dynamic, and 0 otherwise.

And then we predict the full flow F full2 t→s via concating and passing It and Is to a FullflowNet F. The rigid flow F rig2 t→s is computed in a manner gated by the motion mask Mc:

The consistency between the rigid optical flows generated from two distinct manners is enforced:

where ∥ ∥2 represents L2 norm.

In real-world scenes, the prevalence of low-texture regions and frequent occlusions makes it difficult to strictly satisfy the aforementioned criteria in Equation 10. As shown in the third row of Figure 3, Mc inevitably contains substantial noise, preventing consistency constraint in Equation 12. SAM [Kirillov et al., 2023] is exploited to refine coarse motion cues and obtain accurate motion masks.

Refined Identification. With coarse motion mask Mc, we measure overlap ratio and suppress non-dynamical objects:

where |Ω(bi)| is the area of box bi, τ is the threshold and we set 0.45. Each candidate box bi, specified by its four corner coordinates, is evaluated by computing its overlap ratio over the enclosed patches in Mc.

Finally, we harness the dynamic bounding box to prompt SAM with a trainable mask decoder:

where Mr is the refined motion identification, which substitutes Mc to facilitate rigid consistency reasoning on more reliable static regions.

3.3 Learnable Neighbor Affinity Interpolation

Most existing methods upsample low-resolution predictions to full resolution using fixed interpolation schemes (e.g., nearest-neighbor and bilinear) which often result in boundary artifacts. To alleviate this problem, we propose a Learnable Neighbor Affinity Interpolation (LNAI) module.

After general multi-frame depth estimation, the regressed depth Dmu t ∈ RH×W×1 could be obtained. We extracted contextual features Ψ ∈ RH×W×C from target frame, where C = 144. Our interpolation step can be defined as:

where U denotes LANI module. ˆDmu t is interpolated full resolution depth and Dmu t is multi-frame depth at 14 -resolution.

As shown in Figure 4, we first enriched Ψ with RoPE [Su et al., 2024] positional embeddings and derive matrices Y1 and Y2 through projection layer:

For Y1 ∈ RH×W×144 , we divide the channels into 16 groups, with 9 channels in each group and reshape it to the size of HW × 16 × 1 × 9. Then for Y2 ∈ RH×W×144 , we similarly group the features along the channel dimension and reshape it into size of 16×9×H×W. Afterwards, we divide Y2 into HW ×16×9 local windows with the size of N ×N, and reshape it to the size of HW×16×9×N 2 , the neighbor affinity A can be computed as:

where k ∈ {1, 2, . . . ,HW} is the index of contextual features, g ∈ {1, 2, . . . , 16} is the index of feature group. n ∈

{1, 2, . . . ,N 2 } is the index of neighbors, l ∈ {1, 2, . . . , 9} is the index of channel within each group. xk,g,l and xg,n,l are the points in the local window Ω ∈ N 2 . Then we execute softmax on A and reshape it into size of N 2 ×4×4×H×W.

To align with the N × N affinity A, we divide Dmu t into HW local windows with the size of N ×N and reshape it to the size of N 2 × 1 × 1 × H ×W. After that, we yield full resolution depth:

where ⊕ is combination operation, each target interpolated pixel is weighted by a window that is not shared by another pixel. Finally, we reshape the full resolution depth to the size of 4H × 4W × 1.

3.4 Loss Functions

Flow Photometric Loss. We adopt the same measurement in Equation 9 to compute the photometric loss of full flow F f ull1 t→s and F full2 t→s :Lopt

Depth Loss. For single-frame depth Dsi t and multi-frame depth ˆDmu t , we similarly compute photometric loss and use smoothness loss [Godard et al., 2017] Lsm to regularize:

where µ is auto-mask [Godard et al., 2019] and γ = 0.001. Final Loss. In summary, the final loss function becomes a weighted sum of the above terms:

where Ldsi p and Ldmu p is the loss of Dsi t and ˆDmu t . λ1, λ2 and λ3 are the loss weights and we set 0.001, 0.1, 0.1 respectively.

4 Experiments

We evaluate our method on three challenging datasets: KITTI [Geiger et al., 2013], DDAD [Guizilini et al., 2020] and Cityscapes [Cordts et al., 2016]. We use standard metrics to measure model performance. For dynamic regions, we follow the protocol from SC-DepthV3 [Sun et al., 2023].

4.1 Datasets

KITTI. We adopt Eigen split [Eigen et al., 2014] of KITTI, comprising 39,810 training images and 4,424 validation images. The test set includes 697 raw LiDAR frames and 652 sparsity-corrected ground truth images [Uhrig et al., 2017]. Depth is capped at 80m and images are resized to 640 × 192. DDAD. DDAD is a challenging driving dataset with more dynamic objects. It contains 3,950 test frames, with depth capped at 200m and image resolution set to 640 × 384.

Cityscapes. Cityscapes is a benchmark designed for urban scene understanding, including a substantial number of dynamic objects and containing 1,525 test images. The depth range is capped at 80m and images are resized to 512 × 192.

4.2 Implementation Details

We use two NVIDIA RTX 3090 GPUs and Pytorch with batch size 6 for all training experiments. We employ the AdamW optimizer to minimize the loss functions and train the network for 20 epochs. For our baseline model, the DepthNet D follows the same architecture in [Zhao et al., 2022]. The remaining network components adopt the design of [Watson et al., 2021]. We unfreeze PWCNet [Sun et al., 2018] as FullflowNet F in our framework. ResFlowNet R follows the same architecture in [Miao et al., 2023] and a pre-trained ResNet18 is used as our context encoder. Other experimental settings are similar with [Watson et al., 2021].

4.3 Results on KITTI

As shown in Table 1, for raw data, compared with DiffSQL [Zheng et al., 2025], our Abs Rel and RMSE error improve 8.5% and 4.1%. Compared with motion focused method DualDepth [Feng et al., 2025], our Abs Rel error improves 5.5%. When compared with Mono-ViFI [Liu et al., 2024a], our method performs best on the Abs Rel, RMSElog, δ1, and δ2. Since Mono-ViFI leverages a pretrained flow-based video frame interpolation model during inference to capture temporal information, it requires access to future frames that are not yet observed when making current predictions. For dynamic areas, existing motion focused methods fail to sustain consistent performance and show performance drop while our method achieves state-of-the-art results. Moreover, since raw LiDAR data is not well suited for reliably evaluating performance in dynamic areas, we further assess our method using the sparsity-corrected ground truth [Uhrig et al., 2017]. VFM-Dynamo outperforms other comparison methods on almost metrics for both full image and dynamic areas.

Qualitative results are shown in Figure 5, DS-Depth and Dynamo are two motion focused methods that utilize optical flow to introduce motion priors. The last two cases involve dynamic objects (yellow boxes), while comparison methods fail to estimate accurate depth. In the first case, our methods preserves noticeably sharper depth around object boundaries.

Figure 6 shows visual results for the two types of motion identification. MonoDepth2 [Godard et al., 2019] and ManyDepth [Watson et al., 2021] generate binary mask sometimes fails to (i) identify moving objects (e.g., missing car and cyclist in first two cases). ProDepth [Woo et al., 2024] produces soft mask but cannot (ii) distinguish between multiple objects (e.g., entangled car segmentation in the last case), while our method incorporates motion prior with dynamic prompts, resulting in the accurate identification of moving objects.

4.4 Results on DDAD and Cityscapes

The DDAD and Cityscapes datasets include a significant number of moving objects than KITTI because almost vehicles and pedestrian are moving on the road. As depicted in Table 2, VFM-Dynamo achieve superior performance compared with other methods, which demonstrates the excellent zero-shot generalization ability of our model.

4.5 Ablation Study

Our ablation study includes coarse to fine motion identification and LNAI module. All variants are trained on KITTI. The proposed components and their variants are integrated into a unified baseline model for fair comparison.

Effectiveness of coarse to fine. In Table 3, full flow F full1 t→s calculated from depth improves performance as it introduces additional geometric constraints (row 2). We then add Lrig con calculated from coarse mask (row 3), providing attention for dynamic areas indirectly. Furthermore, we adapt VFM for fine-grained motion mask. Removing dynamic prompts (DP) from coarse mask leads to performance drop (row 4) as the mask output by SAM inevitably contains static objects that hinder the rigid consistency constraint. Vision conditioned (VC) textual features integrate scene-aware visual features, yielding better performance than text features alone (row 5). Necessity of accurate non-rigid motion identification. As shown in Table 4, we apply the proposed method to refine the non-rigid motion identification of baseline approaches. For ProDepth [Woo et al., 2024], which uses a soft motion mask, we introduce a thresholding operation (threshold = 0.5) to adapt it to our framework. All methods show significant performance improvements, demonstrating that accurate motion identification effectively mitigates interference from erroneous geometric correspondences during training.

Universality of LNAI module. We adopt single-frame methods, including MonoDepth2 [Godard et al., 2019] and Litemono [Zhang et al., 2023], the multi-frame method Many- Depth [Watson et al., 2021] as baselines. For MonoDepth2 and ManyDepth, we remove the decoder of DepthNet from the last two scales, and for Lite-mono, we remove it from the last scale, retaining depth outputs only at 18 , 14 , and 14 resolution, respectively. Subsequently, we integrate the LNAI module to upsample the 14 -resolution depth maps directly to full resolution. As shown in Table 5, the LNAI module consistently improves nearly all metrics across all baseline methods. Moreover, by eliminating the final decoding stages, the introduction of LNAI incurs almost no additional inference overhead, highlighting its potential to enhance the efficiency of approaches that employ complex decoders.

Model complexity. In Table 6, our proposed model attains a good balance between model size and runtime speed. Notice that VFM-Dynamo outperforms the recent well-designed models ManyDepth2 [Zhou et al., 2025] and DualDepth [Feng et al., 2025] both in runtime and accuracy (Table 1).

4.6 Conclusion

In this paper, we present VFM-Dynamo, a novel selfsupervised MDE methods inspired by HVP and designed a coarse to fine motion identification strategy. Additionally, we facilitate the direct upsampling of low-resolution depth to full-resolution using neighbor affinity and proposed approach can be seamlessly integrated into existing self-supervised methods. In future work, we would like to utilize VFM to distinguish the dynamic objects in the end-to-end manner.

Acknowledgements

This work was supported by the National Key R&D Program of China under Grant (2025YFF0519403) and National Natural Science Foundation of China (U2468223, 51827813).