{"type":"ScholarlyArticle","title":"Hypertron: Explicit Social-Temporal Hypergraph Framework for Multi-Agent Forecasting","authors":[{"name":"Yu Tian","slug":"yu-tian","given":"Yu","family":"Tian","affiliation":"Key Laboratory of Network Information System Technology , Aerospace Information Research Institute , Chinese Academy of Sciences; School of Electronic, Electrical and Communication Engineering , University of Chinese Academy of Sciences {tianyu181 ,"},{"name":"Xingliang Huang","slug":"xingliang-huang","given":"Xingliang","family":"Huang","affiliation":"Key Laboratory of Network Information System Technology , Aerospace Information Research Institute , Chinese Academy of Sciences; School of Electronic, Electrical and Communication Engineering , University of Chinese Academy of Sciences {tianyu181 ,"},{"name":"Ruigang Niu","slug":"ruigang-niu","given":"Ruigang","family":"Niu","affiliation":"Key Laboratory of Network Information System Technology , Aerospace Information Research Institute , Chinese Academy of Sciences; School of Electronic, Electrical and Communication Engineering , University of Chinese Academy of Sciences {tianyu181 ,"},{"name":"Hongfeng Yu","slug":"hongfeng-yu","given":"Hongfeng","family":"Yu","affiliation":"Key Laboratory of Network Information System Technology , Aerospace Information Research Institute , Chinese Academy of Sciences"},{"name":"Peijin Wang","slug":"peijin-wang","given":"Peijin","family":"Wang","affiliation":"Key Laboratory of Network Information System Technology , Aerospace Information Research Institute , Chinese Academy of Sciences"},{"name":"Xian Sun","slug":"xian-sun","given":"Xian","family":"Sun","affiliation":"Key Laboratory of Network Information System Technology , Aerospace Information Research Institute , Chinese Academy of Sciences; School of Electronic, Electrical and Communication Engineering , University of Chinese Academy of Sciences {tianyu181 ,"}],"abstract":"Forecasting the future trajectories of multiple agents is a core technology for human-robot interaction systems. To predict multi-agent trajectories more accurately, it is inevitable that models need to improve interpretability and reduce redundancy. However, many methods adopt implicit weight calculation or black-box networks to learn the semantic interaction of agents, which obviously lack enough interpretation. In addition, most of the existing works model the relation among all agents in a one-to-one manner, which might lead to irrational trajectory predictions due to its redundancy and noise. To address the above issues, we present Hypertron, a human-understandable and lightweight hypergraph-based multi-agent forecasting framework, to explicitly estimate the motions of multiple agents and generate reasonable trajectories. The framework explicitly interacts among multiple agents and learns their latent intentions by our coarse-to-fine hypergraph convolution interaction module. Our experiments on several challenging real-world trajectory forecasting datasets show that Hypertron outperforms a wide array of state-of-the-art methods while saving over 60% parameters and reducing 30% inference time.","doi":"10.24963/ijcai.2022/189","url":"https://ijcai.org/proceedings/2022/189","pdf":"https://ijcai.org/proceedings/2022/0189.pdf","markdown":"https://ijcai.org/proceedings/2022/189.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/189","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":189,"pages":{"first":1356,"last":1362},"keywords":["Computer Vision: Motion and Tracking"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38984766","length_sec":76,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38986623","length_sec":78,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Alahi et al., 2016] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Sil- vio Savarese. Social lstm: Human trajectory prediction in crowded spaces. In Proceedings of CVPR, pages 961-971, 2016.","title":"Social lstm: Human trajectory prediction in crowded spaces","authors":"Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, Silvio Savarese","year":"2016","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Feng et al., 2019] Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph neural net- works. In Proceedings of AAAI, volume 33, pages 3558- 3565, 2019.","title":"Hypergraph neural networks","authors":"Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, Yue Gao","year":"2019","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Giuliari et al., 2021] Francesco Giuliari, Irtiza Hasan, Marco Cristani, and Fabio Galasso. Transformer networks for trajectory forecasting. In Proceedings of ICPR, pages 10335-10342. IEEE, 2021.","title":"Transformer networks for trajectory forecasting","authors":"Francesco Giuliari, Irtiza Hasan, Marco Cristani, Fabio Galasso","year":"2021","venue":"Proceedings of ICPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gupta et al., 2018] Agrim Gupta, Justin Johnson, Li Fei- Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversar- ial networks. In Proceedings of CVPR, pages 2255-2264, 2018.","title":"Social gan: Socially acceptable trajectories with generative adversarial networks","authors":"Gupta","year":"2018","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ivanovic and Pavone, 2019] Boris Ivanovic and Marco Pavone. The trajectron: Probabilistic multi-agent trajec- tory modeling with dynamic spatiotemporal graphs. In Proceedings of ICCV, pages 2375-2384, 2019.","title":"The trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs","authors":"Pavone Ivanovic, Boris Ivanovic, Marco Pavone","year":"2019","venue":"Proceedings of ICCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ji et al., 2020] Shuyi Ji, Yifan Feng, Rongrong Ji, Xibin Zhao, Wanwan Tang, and Yue Gao. Dual channel hyper- graph collaborative filtering. In Proceedings of SIGKDD, pages 2020-2029, 2020.","title":"Dual channel hypergraph collaborative filtering","authors":"Shuyi Ji, Yifan Feng, Rongrong Ji, Xibin Zhao, Wanwan Tang, Yue Gao","year":"2020","venue":"Proceedings of SIGKDD","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jiang et al., 2019] Zhengkai Jiang, Peng Gao, Chaoxu Guo, Qian Zhang, Shiming Xiang, and Chunhong Pan. Video object detection with locally-weighted deformable neigh- bors. In Proceedings of AAAI, volume 33, pages 8529- 8536, 2019.","title":"Video object detection with locally-weighted deformable neighbors","authors":"Zhengkai Jiang, Peng Gao, Chaoxu Guo, Qian Zhang, Shiming Xiang, Chunhong Pan","year":"2019","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kajino, 2019] Hiroshi Kajino. Molecular hypergraph gram- mar with its application to molecular optimization. In Pro- ceedings of ICML, pages 3183-3191. PMLR, 2019.","title":"Molecular hypergraph grammar with its application to molecular optimization","authors":"Hiroshi Kajino, Kajino","year":"2019","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lee et al., 2017] Namhoon Lee, Wongun Choi, Paul Ver- naza, Christopher B Choy, Philip HS Torr, and Manmohan Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of CVPR, pages 336-345, 2017.","title":"Desire: Distant future prediction in dynamic scenes with interacting agents","authors":"Lee","year":"2017","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mangalam et al., 2020] Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Ji- tendra Malik, and Adrien Gaidon. It is not the journey but the destination: Endpoint conditioned trajectory predic- tion. In Proceedings of ECCV, pages 759-776. Springer, 2020.","title":"It is not the journey but the destination: Endpoint conditioned trajectory prediction","authors":"Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, Adrien Gaidon","year":"2020","venue":"Proceedings of ECCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Sadeghian et al., 2019] Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. Sophie: An attentive gan for predicting paths compliant to social and physical constraints. In Pro- ceedings of CVPR, pages 1349-1358, 2019.","title":"Sophie: An attentive gan for predicting paths compliant to social and physical constraints","authors":"Vineet Amir Sadeghian, Ali Kosaraju, Noriaki Sadeghian, Hamid Hirose, Silvio Rezatofighi, Savarese","year":"2019","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Salzmann et al., 2020] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasible trajectory forecasting with hetero- geneous data. In Proceedings of ECCV, pages 683-700. Springer, 2020.","title":"Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data","authors":"Salzmann","year":"2020","venue":"Proceedings of ECCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Sun et al., 2021] Jianhua Sun, Yuxuan Li, Hao-Shu Fang, and Cewu Lu. Three steps to multimodal trajectory predic- tion: Modality clustering, classification and synthesis. In Proceedings of ICCV, pages 13250-13259, October 2021.","title":"Three steps to multimodal trajectory prediction: Modality clustering, classification and synthesis","authors":"Jianhua Sun, Yuxuan Li, Hao-Shu Fang, Cewu Lu","year":"2021","venue":"Proceedings of ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of NeurIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Proceedings of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Wang et al., 2021] Chengxin Wang, Shaofeng Cai, and Gary Tan. Graphtcn: Spatio-temporal interaction mod- eling for human trajectory prediction. In Proceedings WCACV, pages 3450-3459, 2021.","title":"Graphtcn: Spatio-temporal interaction modeling for human trajectory prediction","authors":"Wang","year":"2021","venue":"Proceedings WCACV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Yi and Park, 2020] Jaehyuk Yi and Jinkyoo Park. Hyper- graph convolutional recurrent neural network. In Proceed- ings of SIGKDD, pages 3366-3376, 2020.","title":"Jaehyuk Yi and Jinkyoo Park. Hypergraph convolutional recurrent neural network","authors":"Park Yi","year":"2020","venue":"Proceedings of SIGKDD","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Yu et al., 2020] Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In Proceed- ings of ECCV, pages 507-523. Springer, 2020.","title":"Spatio-temporal graph transformer networks for pedestrian trajectory prediction","authors":"Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, Shuai Yi","year":"2020","venue":"Proceedings of ECCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Yuan and Kitani, 2020] Ye Yuan and Kris Kitani. Dlow: Di- versifying latent flows for diverse human motion predic- tion. In Proceedings of ECCV, pages 346-364. Springer, 2020.","title":"Dlow: Diversifying latent flows for diverse human motion prediction","authors":"Kitani ; ] Ye Yuan, Kris Yuan, Kitani","year":"2020","venue":"Proceedings of ECCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yuan et al., 2021] Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris Kitani. Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting. In Proceed- ings of ICCV, 2021.","title":"Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting","authors":"Yuan","year":"2021","venue":"Proceedings of ICCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhu et al., 2021] Yutao Zhu, Kun Zhou, Jian-Yun Nie, Shengchao Liu, and Zhicheng Dou. Neural sentence order- ing based on constraint graphs. In Proceedings of AAAI, pages 14656-14664. AAAI Press, 2021.","title":"Neural sentence ordering based on constraint graphs","authors":"Yutao Zhu, Kun Zhou, Jian-Yun Nie, Shengchao Liu, Zhicheng Dou","year":"2021","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}