{"type":"ScholarlyArticle","title":"Multimodal Transformer Networks for Pedestrian Trajectory Prediction","authors":[{"name":"Ziyi Yin","slug":"ziyi-yin","given":"Ziyi","family":"Yin","affiliation":""},{"name":"Ruijin Liu","slug":"ruijin-liu","given":"Ruijin","family":"Liu","affiliation":""},{"name":"Zhiliang Xiong","slug":"zhiliang-xiong","given":"Zhiliang","family":"Xiong","affiliation":""},{"name":"Zejian Yuan","slug":"zejian-yuan","given":"Zejian","family":"Yuan","affiliation":""}],"abstract":"We consider the problem of forecasting the future locations of pedestrians in an ego-centric view of a moving vehicle. Current CNNs or RNNs are flawed in capturing the high dynamics of motion between pedestrians and the ego-vehicle, and suffer from the massive parameter usages due to the inefficiency of learning long-term temporal dependencies. To address these issues, we propose an efficient multimodal transformer network that aggregates the trajectory and ego-vehicle speed variations at a coarse granularity and interacts with the optical flow in a fine-grained level to fill the vacancy of highly dynamic motion. Specifically, a coarse-grained fusion stage fuses the information between trajectory and ego-vehicle speed modalities to capture the general temporal consistency. Meanwhile, a fine-grained fusion stage merges the optical flow in the center area and pedestrian area, which compensates the highly dynamic motion of ego-vehicle and target pedestrian. Besides, the whole network is only attention-based that can efficiently model long-term sequences for better capturing the temporal variations. Our multimodal transformer is validated on the PIE and JAAD datasets and achieves state-of-the-art performance with the most light-weight model size. The codes are available at https://github.com/ericyinyzy/MTN_trajectory.","doi":"10.24963/ijcai.2021/174","url":"https://ijcai.org/proceedings/2021/174","pdf":"https://ijcai.org/proceedings/2021/0174.pdf","markdown":"https://ijcai.org/proceedings/2021/174.md","bibtex":"https://ijcai.org/proceedings/2021/bibtex/174","volume":{"slug":"2021","title":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2021","url":"https://ijcai.org/proceedings/2021/","year":2021,"location":"Montreal","dates":"19-27 August 2021","editors":"Prof. Zhi-Hua Zhou","isbn":"978-0-9992411-9-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2021"},"section":"Computer Vision II","track":"Main Track","number":174,"pages":{"first":1259,"last":1265},"keywords":["Computer Vision: Structural and Model-Based Approaches, Knowledge Representation and Reasoning","Computer Vision: 2D and 3D Computer Vision","Multidisciplinary Topics and Applications: Transportation"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}