{"type":"ScholarlyArticle","title":"Harnessing Fourier Isovists and Geodesic Interaction for Long-Term Crowd Flow Prediction","authors":[{"name":"Samuel S. Sohn","slug":"samuel-s-sohn","given":"Samuel S.","family":"Sohn","affiliation":"Rutgers University , USA"},{"name":"Seonghyeon Moon","slug":"seonghyeon-moon","given":"Seonghyeon","family":"Moon","affiliation":"Rutgers University , USA"},{"name":"Honglu Zhou","slug":"honglu-zhou","given":"Honglu","family":"Zhou","affiliation":"Rutgers University , USA"},{"name":"Mihee Lee","slug":"mihee-lee","given":"Mihee","family":"Lee","affiliation":"Rutgers University , USA"},{"name":"Sejong Yoon","slug":"sejong-yoon","given":"Sejong","family":"Yoon","affiliation":"The College of New Jersey , USA"},{"name":"Vladimir Pavlovic","slug":"vladimir-pavlovic","given":"Vladimir","family":"Pavlovic","affiliation":"Rutgers University , USA"},{"name":"Mubbasir Kapadia","slug":"mubbasir-kapadia","given":"Mubbasir","family":"Kapadia","affiliation":"Rutgers University , USA"}],"abstract":"With the rise in popularity of short-term Human Trajectory Prediction (HTP), Long-Term Crowd Flow Prediction (LTCFP) has been proposed to forecast crowd movement in large and complex environments. However, the input representations, models, and datasets for LTCFP are currently limited. To this end, we propose Fourier Isovists, a novel input representation based on egocentric visibility, which consistently improves all existing models. We also propose GeoInteractNet (GINet), which couples the layers between a multi-scale attention network (M-SCAN) and a convolutional encoder-decoder network (CED). M-SCAN approximates a super-resolution map of where humans are likely to interact on the way to their goals and produces multi-scale attention maps. The CED then uses these maps in either its encoder's inputs or its decoder's attention gates, which allows GINet to produce super-resolution predictions with substantially higher accuracy than existing models even with Fourier Isovists. In order to evaluate the scalability of models to large and complex environments, which the only existing LTCFP dataset is unsuitable for, a new synthetic crowd dataset with both real and synthetic environments has been generated. In its nascent state, LTCFP has much to gain from our key contributions. The Supplementary Materials, dataset, and code are available at sssohn.github.io/GeoInteractNet.","doi":"10.24963/ijcai.2022/185","url":"https://ijcai.org/proceedings/2022/185","pdf":"https://ijcai.org/proceedings/2022/0185.pdf","markdown":"https://ijcai.org/proceedings/2022/185.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/185","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":185,"pages":{"first":1328,"last":1334},"keywords":["Agent-based and Multi-agent Systems: Agent-Based Simulation and Emergence","Machine Learning: Applications","Computer Vision: Applications"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986953","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985096","length_sec":352,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Alahi et al., 2014] Alexandre Alahi, Vignesh Ramanathan, and Li Fei-Fei. Socially-aware large-scale crowd forecast- ing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2203-2210, 2014.","title":"Socially-aware large-scale crowd forecasting","authors":"Alexandre Alahi, Vignesh Ramanathan, Li Fei-Fei","year":"2014","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alahi et al., 2016] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Sil- vio Savarese. Social lstm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 961-971, 2016.","title":"Social lstm: Human trajectory prediction in crowded spaces","authors":"Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, Silvio Savarese","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Badrinarayanan et al., 2017] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolu- tional encoder-decoder architecture for image segmenta- tion. IEEE transactions on pattern analysis and machine intelligence, 39(12):2481-2495, 2017.","title":"Segnet: A deep convolutional encoder-decoder architecture for image segmentation","authors":"Vijay Badrinarayanan, Alex Kendall, Roberto Cipolla","year":"2017","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Benedikt, 1979] Michael L Benedikt. To take hold of space: isovists and isovist fields. Environment and Planning B: Planning and design, 6(1):47-65, 1979.","title":"To take hold of space: isovists and isovist fields","authors":"L Michael, Benedikt","year":"1979","venue":"Environment and Planning B: Planning and design","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bohannon, 2005] John Bohannon. Directing the herd: Crowds and the science of evacuation. Science, 310(5746):219-221, 2005.","title":"Directing the herd: Crowds and the science of evacuation","authors":"John Bohannon","year":"2005","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[DESA, 2018] UN DESA. 68% of the world population pro- jected to live in urban areas by 2050, says un. United Nafions Department of Economic and Social Affairs, 2018.","title":"UN DESA. 68% of the world population projected to live in urban areas by 2050, says un","authors":null,"year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gupta et al., 2018] Agrim Gupta, Justin Johnson, Li Fei- Fei, Silvio Savarese, and Alexandre Alahi. Social GAN: Socially Acceptable Trajectories with Generative Adver- sarial Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2255-2264, 2018.","title":"Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks","authors":"Gupta","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Helbing and Molnar, 1995] Dirk Helbing and Peter Molnar. Social force model for pedestrian dynamics. Physical re- view E, 51(5):4282, 1995.","title":"Social force model for pedestrian dynamics","authors":"Molnar Helbing, Dirk Helbing, Peter Molnar","year":"1995","venue":"Physical review E","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kuhl and Giardina, 1982] Frank P Kuhl and Charles R Gi- ardina. Elliptic fourier features of a closed contour. Computer graphics and image processing, 18(3):236-258, 1982.","title":"Elliptic fourier features of a closed contour","authors":"Giardina ; P Kuhl, Charles R Kuhl, Giardina","year":"1982","venue":"Computer graphics and image processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ma and Yarlagadda, 2015] Wenbo Ma and Prasad KDV Yarlagadda. Pedestrian dynamics in real and simulated world. Journal of Urban Planning and Development, 141(3):04014030, 2015.","title":"Pedestrian dynamics in real and simulated world","authors":"Wenbo Ma, Prasad Kdv Yarlagadda","year":"2015","venue":"Journal of Urban Planning and Development","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Oktay et al., 2018] Ozan Oktay, Jo Schlemper, Loic Le Fol- goc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al. Attention u-net: Learning where to look for the pancreas. arXiv preprint arXiv:1804.03999, 2018.","title":"Attention u-net: Learning where to look for the pancreas","authors":"Ozan Oktay, Jo Schlemper, Le Loic, Matthew Folgoc, Mattias Lee, Kazunari Heinrich, Kensaku Misawa, Steven Mori, Nils Y Mcdonagh, Bernhard Hammerla, Kainz","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pages 234-241. Springer, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"International Conference on Medical image computing and computerassisted intervention","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Salzmann et al., 2020] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasible trajectory forecasting with heteroge- neous data. arXiv preprint arXiv:2001.03093, 2020.","title":"Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data","authors":"Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, Marco Pavone","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sewall et al., 2010] Jason Sewall, David Wilkie, Paul Mer- rell, and Ming C Lin. Continuum traffic simulation. In Computer Graphics Forum, volume 29, pages 439-448. Wiley Online Library, 2010.","title":"Continuum traffic simulation","authors":"Jason Sewall, David Wilkie, Paul Merrell, Ming C Lin","year":"2010","venue":"Computer Graphics Forum","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sohn et al., 2020] Samuel S Sohn, Honglu Zhou, Seonghyeon Moon, Sejong Yoon, Vladimir Pavlovic, and Mubbasir Kapadia. Laying the foundations of deep long-term crowd flow prediction. In European Conference on Computer Vision, pages 711-728. Springer, 2020.","title":"Laying the foundations of deep long-term crowd flow prediction","authors":"Honglu Samuel S Sohn, Seonghyeon Zhou, Sejong Moon, Vladimir Yoon, Mubbasir Pavlovic, Kapadia","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sohn et al., 2021] Samuel S. Sohn, Mihee Lee, Seonghyeon Moon, Gang Qiao, Muhammad Usman, Sejong Yoon, Vladimir Pavlovic, and Mubbasir Kapadia. A2x: An agent and environment interaction benchmark for multi- modal human trajectory prediction. Motion, Interaction and Games, 2021.","title":"A2x: An agent and environment interaction benchmark for multimodal human trajectory prediction","authors":"S Samuel, Mihee Sohn, Seonghyeon Lee, Gang Moon, Muhammad Qiao, Sejong Usman, Vladimir Yoon, Mubbasir Pavlovic, Kapadia","year":"2021","venue":"Motion, Interaction and Games","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Turner and Penn, 2002] Alasdair Turner and Alan Penn. Encoding natural movement as an agent-based system: an investigation into human pedestrian behaviour in the built environment. Environment and planning B: Planning and Design, 29(4):473-490, 2002.","title":"Encoding natural movement as an agent-based system: an investigation into human pedestrian behaviour in the built environment","authors":"Penn ; Turner, Alasdair Turner, Alan Penn","year":"2002","venue":"Environment and planning B: Planning and Design","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Van Den Berg et al., 2011] Jur Van Den Berg, Stephen J Guy, Ming Lin, and Dinesh Manocha. Reciprocal n-Body Collision Avoidance. In Robotics Research, pages 3-19. Springer, 2011.","title":"Reciprocal n-Body Collision Avoidance","authors":"Van Den, Berg","year":"2011","venue":"Robotics Research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wiener and Franz, 2004] Jan M Wiener and Gerald Franz. Isovists as a means to predict spatial experience and be- havior. In International Conference on Spatial Cognition, pages 42-57. Springer, 2004.","title":"Isovists as a means to predict spatial experience and behavior","authors":"Jan M Wiener, Gerald Franz","year":"2004","venue":"International Conference on Spatial Cognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wiener et al., 2007] Jan M Wiener, Gerald Franz, Nicole Rossmanith, Andreas Reichelt, Hanspeter A Mallot, and Heinrich H Bülthoff. Isovist analysis captures properties of space relevant for locomotion and experience. Percep- tion, 36(7):1066-1083, 2007.","title":"Isovist analysis captures properties of space relevant for locomotion and experience","authors":"Jan M Wiener, Gerald Franz, Nicole Rossmanith, Andreas Reichelt, A Hanspeter, Heinrich H Mallot, Bülthoff","year":"2007","venue":"Perception","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2002] Dengsheng Zhang, Guojun Lu, et al. A comparative study of fourier descriptors for shape repre- sentation and retrieval. In Proc. 5th Asian Conference on Computer Vision, page 35. Citeseer, 2002.","title":"A comparative study of fourier descriptors for shape representation and retrieval","authors":"Zhang","year":"2002","venue":"Proc. 5th Asian Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhen et al., 2008] Wang Zhen, Liu Mao, and Zhao Yuan. Analysis of trample disaster and a case study-mihong bridge fatality in china in 2004. Safety Science, 46(8):1255-1270, 2008.","title":"Analysis of trample disaster and a case study-mihong bridge fatality in china in 2004","authors":"Wang Zhen, Liu Mao, Zhao Yuan","year":"2008","venue":"Safety Science","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}