External Memory Matters: Generalizable Object-Action Memory for Retrieval-Augmented Long-Term Video Understanding

Dang, Jisheng; Zheng, Huicheng; Wu, Xudong; Jiao, Jingmei; Wang, Bimei; Yang, Jun; Hu, Bin; Lai, Jianhuang; Chua, Tat-Seng

doi:10.24963/ijcai.2025/97

External Memory Matters: Generalizable Object-Action Memory for Retrieval-Augmented Long-Term Video Understanding

Jisheng Dang, Huicheng Zheng, Xudong Wu, Jingmei Jiao, Bimei Wang, Jun Yang, Bin Hu, Jianhuang Lai, Tat Seng Chua

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 864-872.

https://doi.org/10.24963/ijcai.2025/97

Keywords

Computer Vision: CV: Recognition (object detection, categorization)
Computer Vision: CV: Multimodal learning
Computer Vision: CV: Scene analysis and understanding
Computer Vision: CV: Video analysis and understanding

Abstract

Long video understanding with Large Language Models (LLMs) enables the description of objects that are not explicitly present in the training data. However, continuous changes in known objects and the emergence of new ones require up-to-date knowledge of objects and their dynamics for effective understanding of the open world. To alleviate this, we propose an efficient Retrieval-Enhanced Video Understanding method, dubbed REVU, which leverages external knowledge to enhance the performance of open-world learning. First, REVU introduces an extensible external text-object memory with minimal text-visual mapping, involving static and dynamic multimodal information to help LLMs-based models align text and vision features. Second, REVU retrieves object information from external databases and dynamically integrates frame-specific data from videos, enabling effective knowledge aggregation to comprehend the open world. We conducted experiments on multiple benchmark datasets, and our model demonstrates strong adaptability to out-of-domain data without requiring additional fine-tuning or re-training. Experiments on benchmark video understanding datasets reveal that our model achieves state-of-the-art performance and robust generalization.

Jisheng Dang, Huicheng Zheng, Xudong Wu, Jingmei Jiao, Bimei Wang, Jun Yang, Bin Hu, Jianhuang Lai, and Tat-Seng Chua. 2025. External Memory Matters: Generalizable Object-Action Memory for Retrieval-Augmented Long-Term Video Understanding. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. IJCAI, 864–872. https://doi.org/10.24963/ijcai.2025/97

@inproceedings{ijcai2025p97,
  title     = {External Memory Matters: Generalizable Object-Action Memory for Retrieval-Augmented Long-Term Video Understanding},
  author    = {Dang, Jisheng and Zheng, Huicheng and Wu, Xudong and Jiao, Jingmei and Wang, Bimei and Yang, Jun and Hu, Bin and Lai, Jianhuang and Chua, Tat-Seng},
  booktitle = {Proceedings of the Thirty-Fourth International Joint Conference on
               Artificial Intelligence, {IJCAI-25}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {James Kwok},
  pages     = {864--872},
  year      = {2025},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2025/97},
  url       = {https://doi.org/10.24963/ijcai.2025/97},
}

TY  - CPAPER
TI  - External Memory Matters: Generalizable Object-Action Memory for Retrieval-Augmented Long-Term Video Understanding
AU  - Dang, Jisheng
AU  - Zheng, Huicheng
AU  - Wu, Xudong
AU  - Jiao, Jingmei
AU  - Wang, Bimei
AU  - Yang, Jun
AU  - Hu, Bin
AU  - Lai, Jianhuang
AU  - Chua, Tat-Seng
T2  - Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2025
SP  - 864
EP  - 872
DO  - 10.24963/ijcai.2025/97
UR  - https://doi.org/10.24963/ijcai.2025/97
ER  -