Proceedings

Towards Vision-Spatiotemporal Fusion in Traffic Forecasting: A Survey on Cross-Modal Alignment

Download BibTeX Cite

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. Survey Track. Pages 8032-8040.

https://doi.org/10.24963/ijcai.2026/891

Abstract

Traffic forecasting is evolving, with world models emerging as a powerful framework applicable to tasks such as core state, trajectory, event, and demand forecasting. These tasks involve both visual and spatiotemporal data, yet most existing methods treat them separately, hindering a unified understanding of traffic scenes in both semantic meanings and spatiotemporal dynamics. The fusion of the two modalities is critical for building models that comprehend complex traffic scenarios. However, the fusion issue faces two fundamental misalignments: semantic, where pixels conflict with traffic concepts, and geometric, which requires spatial intelligence to map 2D inputs into 3D. This survey reframes vision-spatiotemporal fusion via the unique lens of cross-modal alignment, addressing semantic and geometric failures that limit forecasting reliability. First, we categorize existing methods into three paradigms: feature-level, semantic-level, and task-level. This reveals their progression from low-level feature manipulation to high-level architectural integration. Second, we synthesize representative techniques per paradigm, highlighting geometric challenges such as cross-view association and spatial mapping. Third, we examine current datasets and benchmarks, highlighting their deficiencies in evaluating alignment. Finally, we outline future directions, including spatiotemporal intelligence for robust perception and holistic traffic world models. The unified framework establishes a reference for robust and explainable forecasting systems.

  1. [Caesar et al., 2020] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. Nuscenes: A multimodal dataset for autonomous driving. In CVPR, pages 11621-11631, 2020.
  2. [Chen et al., 2025] Mouxiang Chen, Lefei Shen, Zhuo Li, Xiaoyun Joy Wang, Jianling Sun, and Chenghao Liu. Vi- sionts: Visual masked autoencoders are free-lunch zero- shot time series forecasters. In ICML, 2025.
  3. [Chen et al., 2026] Jinwen Chen, Hao Miao, Chenxi Liu, Yan Zhao, and Kai Zheng. Visionst: Coordinating cross- modal traffic prediction with interactive geo-image encod- ing. In WWW, pages 7307-7317, 2026.
  4. [Dai et al., 2025] Ruiting Dai, Chenxi Li, Yandong Yan, Lisi Mo, Ke Qin, and Tao He. Unbiased missing-modality mul- timodal learning. In ICCV, pages 24507-24517, 2025.
  5. [Ding et al., 2025] Jingtao Ding, Yunke Zhang, Yu Shang, Yuheng Zhang, Zefang Zong, Jie Feng, Yuan Yuan, Hongyuan Su, Nian Li, Nicholas Sukiennik, et al. Under- standing world or predicting future? A comprehensive sur- vey of world models. ACM Computing Surveys, 58(3):57, 2025.
  6. [Fang et al., 2026] Yuchen Fang, Hao Miao, Yuxuan Liang, Liwei Deng, Yue Cui, Ximu Zeng, Yuyang Xia, Yan Zhao, Torben Bach Pedersen, Christian S Jensen, et al. Unrav- eling spatio-temporal foundation models via the pipeline lens: A comprehensive review. TKDE, 38(3):2040-2063, 2026.
  7. [Ferreira et al., 2025] Gabriel O Ferreira, Fabrizio Dabbene, Francesco Malandrino, and Chiara Ravazzi. Unraveling urban mobility: A domain knowledge-free trajectory clas- sification using gramian angular fields. TITS, 2025.
  8. [Gong et al., 2022] Shenjian Gong, Shanshan Zhang, Jian Yang, Dengxin Dai, and Bernt Schiele. Bi-level alignment for cross-domain crowd counting. In ICCV, pages 7542- 7550, 2022.
  9. [He et al., 2025] Zhu He, Mingwei Lin, Xin Luo, and Zeshui Xu. Structure-preserved self-attention for fusion image in- formation in multiple color spaces. TNNLS, 36(7):13021- 13035, 2025.
  10. [Hua and Wang, 2024] Tongyan Hua and Lin Wang. Bench- marking implicit neural representation and geometric ren- dering in real-time rgb-d slam. In CVPR, pages 21346- 21356, 2024.
  11. [Huang et al., 2024] Brandon Huang, Chancharik Mitra, As- saf Arbelle, Leonid Karlinsky, Trevor Darrell, and Roei Herzig. Multimodal task vectors enable many-shot mul- timodal in-context learning. In NeurIPS, pages 22124- 22153, 2024.
  12. [Jain et al., 2025] Anubhooti Jain, Mayank Vatsa, and Richa Singh. Words over pixels? Rethinking vision in multi- modal large language models. In IJCAI, pages 10481- 10489, 2025. IJCAI
  13. [Jiang et al., 2024] Xiaohui Jiang, Shuailin Li, Yingfei Liu, Shihao Wang, Fan Jia, Tiancai Wang, Lijin Han, and Xi- angyu Zhang. Far3d: Expanding the horizon for surround- view 3d object detection. In AAAI, pages 2561-2569, 2024.
  14. [Kannan and Min, 2025] Shyam Sundar Kannan and Byung- Cheol Min. Zeroscd: Zero-shot street scene change detec- tion. In ICRA, pages 4665-4671, 2025.
  15. [Khan and Gauch, 2025] Reeshad Khan and John Gauch. Tinybev: Cross-modal knowledge distillation for efficient multi-task bird's-eye-view perception and planning. In ICCV, pages 4577-4585, 2025.
  16. [Kong et al., 2024] Quan Kong, Yuki Kawana, Rajat Saini, Ashutosh Kumar, Jingjing Pan, Ta Gu, Yohei Ozao, Bal- azs Opra, Yoichi Sato, and Norimasa Kobori. Wts: A pedestrian-centric traffic video dataset for fine-grained spatial-temporal understanding. In ECCV, pages 1-18, 2024.
  17. [Lee et al., 2022] Wei-Yu Lee, Ljubomir Jovanov, and Wil- fried Philips. Cross-modality attention and multimodal fu- sion transformer for pedestrian detection. In ECCV, pages 608-623, 2022.
  18. [Li et al., 2025a] Haoyang Li, Xin Wang, Ziwei Zhang, and Wenwu Zhu. Out-of-distribution generalization on graphs: A survey. TPAMI, 47(11):10490-10512, 2025.
  19. [Li et al., 2025b] Wentao Li, Bowen Yang, Witold Pedrycz, Chao Zhang, and Tao Zhan. Adaptive hyper-box gran- ulation with justifiable granularity for feature selection. TKDE, 37(12):6847-6862, 2025.
  20. [Li et al., 2025c] Xiaoyu Li, Yitian Zhang, Guodong Long, Yupeng Hu, Wenpeng Lu, Meng Chen, Chengqi Zhang, and Yongshun Gong. Adaptive traffic forecasting on daily basis: A spatio-temporal context learning approach. TKDE, 37(8):4446-4459, 2025.
  21. [Li et al., 2025d] Yujie Li, Xiangkun Wang, Xin Yang, Mar- cello Bonsangue, Junbo Zhang, and Tianrui Li. Improv- ing open-world continual learning under the constraints of scarce labeled data. In SIGKDD, pages 1647-1658, 2025.
  22. [Li et al., 2025e] Zhe Li, Xiangfei Qiu, Peng Chen, Yihang Wang, Hanyin Cheng, Yang Shu, Jilin Hu, Chenjuan Guo, Aoying Zhou, Christian S Jensen, et al. Tsfm-bench: A comprehensive and unified benchmark of foundation mod- els for time series forecasting. In SIGKDD, pages 5595- 5606, 2025.
  23. [Lin et al., 2025a] Jiawen Lin, Shiran Bian, Yihang Zhu, Wenbin Tan, Yachao Zhang, Yuan Xie, and Yanyun Qu. Seqvlm: Proposal-guided multi-view sequences reasoning via vlm for zero-shot 3d visual grounding. In SIGMM, pages 3094-3103, 2025.
  24. [Lin et al., 2025b] Mingwei Lin, Jiaqi Liu, Hong Chen, Xi- uqin Xu, Xin Luo, and Zeshui Xu. A 3d convolution- incorporated dimension preserved decomposition model for traffic data prediction. TITS, 26(1):673-690, 2025.
  25. [Liu et al., 2025] Chenxi Liu, Shaowen Zhou, Qianxiong Xu, Hao Miao, Cheng Long, Ziyue Li, and Rui Zhao. To- wards cross-modality modeling for time series analytics: A survey in the llm era. In IJCAI, pages 10564-10572, 2025. IJCAI
  26. [Lv et al., 2025] Qinzhi Lv, Lijuan Liu, Ruotong Yang, and Yan Wang. Multimodal urban traffic flow prediction based on multi-scale time series imaging. Pattern Recognition, 164:111499, 2025.
  27. [Malik et al., 2022] Sumbal Malik, Manzoor Ahmed Khan, and Hesham El-Sayed. Carla: Car learning to act-an inside out. Procedia Computer Science, 198:742-749, 2022.
  28. [Mirzaie and Rosenhahn, 2025] Mona Mirzaie and Bodo Rosenhahn. Interpretable decision-making for end-to-end autonomous driving. In ICCV, pages 794-804, 2025.
  29. [Ni et al., 2025] Jingchao Ni, Ziming Zhao, ChengAo Shen, Hanghang Tong, Dongjin Song, Wei Cheng, Dongsheng Luo, and Haifeng Chen. Harnessing vision models for time series analysis: A survey. In IJCAI, pages 10612- 10620, 2025. IJCAI
  30. [Nie et al., 2023] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In ICLR, 2023.
  31. [Palladin et al., 2025] Edoardo Palladin, Samuel Brucker, Filippo Ghilotti, Praveen Narayanan, Mario Bijelic, and Felix Heide. Self-supervised sparse sensor fusion for long range perception. In ICCV, pages 27498-27509, 2025.
  32. [Peddi et al., 2024] Rohith Peddi, Saksham Singh, Saurabh, Parag Singla, and Vibhav Gogate. Towards scene graph anticipation. In ECCV, pages 159-175, 2024.
  33. [Shaw et al., 2025] Ankit Kumar Shaw, Chandan Kumar Sah, Xiaoli Lian, Arsalan Shahid Baig, Tuopu Wen, Kun Jiang, Mengmeng Yang, Diange Yang, and Li Zhang. Saferoute: Enhancing traffic scene understanding via a unified deep learning and multimodal llm. In ICCV, pages 4547-4556, 2025.
  34. [Shrout et al., 2025] Oren Shrout, Ori Nizan, Yizhak Ben- Shabat, and Ayellet Tal. Patchcontrast: Self-supervised pre-training for 3d object detection. In CVPR, pages 2456- 2466, 2025.
  35. [Sun et al., 2020] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, pages 2446-2454, 2020.
  36. [Sun et al., 2024] Yubao Sun, Jihui Tang, Qingshan Liu, Zhendong Zhang, and Liang Huang. Highway visibil- ity level prediction using geometric and visual features driven dual-branch fusion network. TITS, 25(8):9992- 10004, 2024.
  37. [Tang et al., 2019] Zheng Tang, Milind Naphade, Ming-Yu Liu, Xiaodong Yang, Stan Birchfield, Shuo Wang, Rat- nesh Kumar, David Anastasiu, and Jenq-Neng Hwang. Cityflow: A city-scale benchmark for multi-target multi- camera vehicle tracking and re-identification. In CVPR, pages 8797-8806, 2019.
  38. [Tang et al., 2024] Qi Tang, Yao Zhao, Meiqin Liu, and Chao Yao. Seeclear: Semantic distillation enhances pixel condensation for video super-resolution. In NeurIPS, pages 134902-134926, 2024.
  39. [Vafa et al., 2024] Keyon Vafa, Justin Y Chen, Ashesh Ram- bachan, Jon Kleinberg, and Sendhil Mullainathan. Evalu- ating the world model implicit in a generative model. In NeurIPS, pages 26941-26975, 2024.
  40. [Wang et al., 2024] Xiaoding Wang, Jianmin Liu, Hui Lin, Sahil Garg, and Mubarak Alrashoud. A multi-modal spatial-temporal model for accurate motion forecasting with visual fusion. Information Fusion, 102:102046, 2024.
  41. [Wang et al., 2025] Liying Wang, Xiaoli Zhang, Chuanmin Jia, and Siwei Ma. Mafs: Masked autoencoder for infrared-visible image fusion and semantic segmentation. TIP, 34:6490-6505, 2025.
  42. [Wang et al., 2026] Bingjie Wang, Chao Zhang, Wentao Li, and Deyu Li. Mf3: Multimodal federated learning with dual-path mamba-transformer for metro flow prediction. In WWW, pages 5451-5462, 2026.
  43. [Wu et al., 2023] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In ICLR, 2023.
  44. [Wu et al., 2025] Hsiu-Fu Wu, Ya-Ting Yang, Yung-Ter Chen, and I-Fan Chou. Trafficinternvl: Understanding traffic scenarios with vision-language models. In ICCV, pages 5229-5236, 2025.
  45. [Yang et al., 2021] Weixiang Yang, Qi Li, Wenxi Liu, Yuan- long Yu, Yuexin Ma, Shengfeng He, and Jia Pan. Project- ing your view attentively: Monocular road scene layout estimation via cross-view transformation. In ICCV, pages 15536-15545, 2021.
  46. [Yi et al., 2024] Zhongchao Yi, Zhengyang Zhou, Qihe Huang, Yanjiang Chen, Liheng Yu, Xu Wang, and Yang Wang. Get rid of isolation: A continuous multi-task spatio- temporal learning framework. In NeurIPS, pages 136701- 136726, 2024.
  47. [Zeng et al., 2025] Tong Zeng, Longfeng Wu, Liang Shi, Dawei Zhou, and Feng Guo. Are vision llms road- ready? A comprehensive benchmark for safety-critical driving video understanding. In SIGKDD, pages 5972- 5983, 2025.
  48. [Zhang et al., 2023] Jiarui Zhang, Filip Ilievski, Kaixin Ma, Aravinda Kollaa, Jonathan Francis, and Alessandro Oltra- mari. A study of situational reasoning for traffic under- standing. In SIGKDD, pages 3262-3272, 2023.
  49. [Zhou et al., 2024] Siwei Zhou, Youngha Chang, Nobuhiko Mukai, Hiroaki Santo, Fumio Okura, Yasuyuki Mat- sushita, and Shuang Zhao. Path-space differentiable ren- dering of implicit surfaces. In SIGGRAPH, page 19, 2024.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards