Diff-LMM: Diffusion Teacher-Guided Spatio-Temporal Perception for Video Large Multimodal Models

Dang, Jisheng; Chen, Ligen; Wu, Jingze; Lin, Ronghao; Wang, Bimei; Wang, Yun; Wang, Liting; Zhu, Nannan; Wang, Teng

doi:10.24963/ijcai.2025/98

Diff-LMM: Diffusion Teacher-Guided Spatio-Temporal Perception for Video Large Multimodal Models

Jisheng Dang, Ligen Chen, Jingze Wu, Ronghao Lin, Bimei Wang, Yun Wang, Liting Wang, Nannan Zhu, Teng Wang

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 873-881.

https://doi.org/10.24963/ijcai.2025/98

Keywords

Computer Vision: CV: Multimodal learning
Computer Vision: CV: Video analysis and understanding

Abstract

Dynamic spatio-temporal understanding is essential for video-based multimodal tasks, yet existing methods often struggle to capture fine-grained temporal and spatial relationships in long videos. Current approaches primarily rely on pre-trained CLIP encoders, which excel in semantic understanding but lack spatially-aware visual context. This leads to hallucinated results when interpreting fine-grained objects or scenes. To address these limitations, we propose a novel framework that integrates diffusion models into multimodal video models. By employing diffusion encoders at intermediate layers, we enhance visual representations through feature alignment and knowledge distillation losses, significantly improving the model's ability to capture spatial patterns over time. Additionally, we introduce a multi-level alignment strategy to learn robust feature correspondence from pre-trained diffusion models. Extensive experiments on benchmark datasets demonstrate our approach's state-of-the-art performance across multiple video understanding tasks. These results establish diffusion models as a powerful tool for enhancing multimodal video models in complex, dynamic scenarios.

Jisheng Dang, Ligen Chen, Jingze Wu, Ronghao Lin, Bimei Wang, Yun Wang, Liting Wang, Nannan Zhu, and Teng Wang. 2025. Diff-LMM: Diffusion Teacher-Guided Spatio-Temporal Perception for Video Large Multimodal Models. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. IJCAI, 873–881. https://doi.org/10.24963/ijcai.2025/98

@inproceedings{ijcai2025p98,
  title     = {Diff-LMM: Diffusion Teacher-Guided Spatio-Temporal Perception for Video Large Multimodal Models},
  author    = {Dang, Jisheng and Chen, Ligen and Wu, Jingze and Lin, Ronghao and Wang, Bimei and Wang, Yun and Wang, Liting and Zhu, Nannan and Wang, Teng},
  booktitle = {Proceedings of the Thirty-Fourth International Joint Conference on
               Artificial Intelligence, {IJCAI-25}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {James Kwok},
  pages     = {873--881},
  year      = {2025},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2025/98},
  url       = {https://doi.org/10.24963/ijcai.2025/98},
}

TY  - CPAPER
TI  - Diff-LMM: Diffusion Teacher-Guided Spatio-Temporal Perception for Video Large Multimodal Models
AU  - Dang, Jisheng
AU  - Chen, Ligen
AU  - Wu, Jingze
AU  - Lin, Ronghao
AU  - Wang, Bimei
AU  - Wang, Yun
AU  - Wang, Liting
AU  - Zhu, Nannan
AU  - Wang, Teng
T2  - Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2025
SP  - 873
EP  - 881
DO  - 10.24963/ijcai.2025/98
UR  - https://doi.org/10.24963/ijcai.2025/98
ER  -