{"type":"ScholarlyArticle","title":"CrowdFormer: An Overlap Patching Vision Transformer for Top-Down Crowd Counting","authors":[{"name":"Shaopeng Yang","slug":"shaopeng-yang","given":"Shaopeng","family":"Yang","affiliation":"Watrix Technology Co. LTD. , Beijing , China"},{"name":"Weiyu Guo","slug":"weiyu-guo","given":"Weiyu","family":"Guo","affiliation":"Information School , Central University of Finance and Economics , Beijing , China"},{"name":"Yuheng Ren","slug":"yuheng-ren","given":"Yuheng","family":"Ren","affiliation":"Watrix Technology Co. LTD. , Beijing , China"}],"abstract":"Crowd counting methods typically predict a density map as an intermediate representation of counting, and achieve good performance. However, due to the perspective phenomenon, there is a scale variation in real scenes, which causes the density map-based methods suffer from a severe scene generalization problem because only a limited number of scales are fitted in density map prediction and generation. To address this issue, we propose a novel vision transformer network, i.e., CrowdFormer, and a density kernels fusion framework for more accurate density map estimation and generation, respectively. Thereafter, we incorporate these two innovations into an adaptive learning system, which can take both the annotation dot map and original image as input, and jointly learns the density map estimator and generator within an end-to-end framework. The experimental results demonstrate that the proposed model achieves the state-of-the-art in the terms of MAE and MSE (e.g., it achieved a MAE of 67.1 and MSE of 301.6 on NWPU-Crowd dataset.), and confirm the effectiveness of the proposed two designs. The code is  https://github.com/special-yang/Top_Down-CrowdCounting.","doi":"10.24963/ijcai.2022/215","url":"https://ijcai.org/proceedings/2022/215","pdf":"https://ijcai.org/proceedings/2022/0215.pdf","markdown":"https://ijcai.org/proceedings/2022/215.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/215","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":215,"pages":{"first":1545,"last":1551},"keywords":["Computer Vision: Representation Learning","Computer Vision: Recognition (object detection, categorization)","Computer Vision: Machine Learning for Vision","Computer Vision: Video analysis and understanding","Computer Vision: Scene analysis and understanding"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986476","length_sec":76,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984619","length_sec":308,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Alexey et al., 2021] Dosovitskiy Alexey, Beyer Lucas, et al. An image is worth 16x16 words: Transformers for im- age recognition at scale. In International Conference on Learning Representations, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ashish et al., 2017] Vaswani Ashish, Shazeer Noam, et al. Attention is all you need. In Annual Conference on Neural Information Processing Systems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Shazeer Noam","year":"2017","venue":"Annual Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Boyu et al., 2020] Wang Boyu, Liu Huidong, et al. Distri- bution matching for crowd counting. In Annual Confer- ence on Neural Information Processing Systems, 2020.","title":"Distribution matching for crowd counting","authors":"Wang Boyu, Liu Huidong","year":"2020","venue":"Annual Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chan et al., 2008] Antoni B. Chan, Zhang-Sheng John Liang, and Nuno Vasconcelos. Privacy preserving crowd monitoring: Counting people without people models or tracking. In Computer Society Conference on Computer Vision and Pattern Recognition, pages 1-7, 2008.","title":"Privacy preserving crowd monitoring: Counting people without people models or tracking","authors":"Antoni B Chan, John Zhang-Sheng, Nuno Liang, Vasconcelos","year":"2008","venue":"Computer Society Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chattopadhyay et al., 2017] Prithvijit Chattopadhyay, Ra- makrishna Vedantam, et al. Counting everyday objects in everyday scenes. In Conference on Computer Vision and Pattern Recognition, pages 4428-4437, 2017.","title":"Prithvijit Chattopadhyay, Ramakrishna Vedantam, et al. Counting everyday objects in everyday scenes","authors":"Chattopadhyay","year":"2017","venue":"Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2021] Binghui Chen, Zhaoyi Yan, et al. Varia- tional attention: Propagating domain-specific knowledge for multi-domain learning in crowd counting. In Inter- national Conference on Computer Vision, pages 16065- 16075, 2021.","title":"Variational attention: Propagating domain-specific knowledge for multi-domain learning in crowd counting","authors":"Chen","year":"2021","venue":"International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dingkang et al., 2021] Liang Dingkang, Chen Xiwu, et al. Transcrowd: Weakly-supervised crowd counting with transformer. CoRR, abs/2104.09116, 2021.","title":"Transcrowd: Weakly-supervised crowd counting with transformer","authors":"Liang Dingkang, Chen Xiwu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Haroon et al., 2018] Idrees Haroon, Tayyab Muhmmad, Athrey Kishan, et al. Composition loss for counting, den- sity map estimation and localization in dense crowds. In ECCV, volume 11206, pages 544-559, 2018.","title":"Composition loss for counting, density map estimation and localization in dense crowds","authors":"Idrees Haroon, Tayyab Muhmmad, Athrey Kishan","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jia et al., 2020] Wan Jia, Antoni B. Chan, et al. Modeling noisy annotations for crowd counting. In Annual Confer- ence on Neural Information Processing Systems, 2020.","title":"Modeling noisy annotations for crowd counting","authors":"Wan Jia, Antoni B Chan","year":"2020","venue":"Annual Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jia et al., 2021] Wan Jia, Liu Ziquan, et al. A general- ized loss function for crowd counting and localization. In Conference on Computer Vision and Pattern Recognition, pages 1974-1983, 2021.","title":"A generalized loss function for crowd counting and localization","authors":"Wan Jia, Liu Ziquan","year":"2021","venue":"Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lempitsky et al., 2010] Victor S. Lempitsky, Andrew Zis- serman, et al. Learning to count objects in images. In Annual Conference on Neural Information Processing Sys- tems, pages 1324-1332, 2010.","title":"Learning to count objects in images","authors":"S Victor, Andrew Lempitsky, Zisserman","year":"2010","venue":"Annual Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin et al., 2021] Hui Lin, Xiaopeng Hong, et al. Direc- t measure matching for crowd counting. In Internation- al Joint Conference on Artificial Intelligence, pages 837- 844, 2021.","title":"Direct measure matching for crowd counting","authors":"Hui Lin, Xiaopeng Hong","year":"2021","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2021/116"},{"position":13,"raw":"[Liu et al., 2021] Hao Liu, Qiang Zhao, et al. Bipartite matching for crowd counting with point supervision. In International Joint Conference on Artificial Intelligence, pages 860-866, 2021.","title":"Bipartite matching for crowd counting with point supervision","authors":"Hao Liu, Qiang Zhao","year":"2021","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2021/119"},{"position":14,"raw":"[Min et al., 2008] Li Min, Zhang Zhaoxiang, Huang Kaiqi, and Tan Tieniu. Estimating the number of people in crowded scenes by mid based foreground segmentation and head-shoulder detection. In International Conference on Pattern Recognition, pages 1-4, 2008.","title":"Estimating the number of people in crowded scenes by mid based foreground segmentation and head-shoulder detection","authors":"Li Min, Zhang Zhaoxiang, Huang Kaiqi, Tan Tieniu","year":"2008","venue":"International Conference on Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nicolas et al., 2020] Carion Nicolas, Massa Francisco, et al. End-to-end object detection with transformers. In EC- CV, volume 12346 of Lecture Notes in Computer Science, pages 213-229, 2020.","title":"End-to-end object detection with transformers","authors":"Carion Nicolas, Massa Francisco","year":"2020","venue":"EC-CV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Qi et al., 2021] Wang Qi, Gao Junyu, et al. Nwpu-crowd: A large-scale benchmark for crowd counting and localiza- tion. Trans. Pattern Anal. Mach. Intell., 43(6):2141-2149, 2021.","title":"Nwpu-crowd: A large-scale benchmark for crowd counting and localization","authors":"Wang Qi, Gao Junyu","year":"2021","venue":"Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Qingyu et al., 2021] Song Qingyu, Wang Changan, et al. To choose or to fuse? scale selection for crowd counting. In AAAI Conference on Artificial Intelligence, volume 35, pages 2576-2583, 2021.","title":"To choose or to fuse? scale selection for crowd counting","authors":"Qingyu","year":"2021","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shuai et al., 2020] Bai Shuai, He Zhiqun, et al. Adaptive dilated network with self-correction supervision for count- ing. In Conference on Computer Vision and Pattern Recog- nition, pages 4593-4602, 2020.","title":"Adaptive dilated network with self-correction supervision for counting","authors":"Bai Shuai, He Zhiqun","year":"2020","venue":"Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Song et al., 2021] Qingyu Song, Changan Wang, et al. Re- thinking counting and localization in crowds: A purely point-based framework. In International Conference on Computer Vision, pages 3365-3374, 2021.","title":"Rethinking counting and localization in crowds: A purely point-based framework","authors":"Qingyu Song, Changan Wang","year":"2021","venue":"International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tsung-Yi et al., 2017] Lin Tsung-Yi, Dollár Piotr, et al. Feature pyramid networks for object detection. In Confer- ence on Computer Vision and Pattern Recognition, pages 936-944, 2017.","title":"Feature pyramid networks for object detection","authors":"Lin Tsung-Yi, Dollár Piotr","year":"2017","venue":"Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wan and Chan, 2019] Jia Wan and Antoni B. Chan. Adap- tive density map generation for crowd counting. In In- ternational Conference on Computer Vision, pages 1130- 1139, 2019.","title":"Adaptive density map generation for crowd counting","authors":"Jia Wan, Antoni B Chan","year":"2019","venue":"International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wan et al., 2020] Jia Wan, Qingzhong Wang, and Antoni B Chan. Kernel-based density map generation for dense ob- ject counting. Transactions on Pattern Analysis and Ma- chine Intelligence, 2020.","title":"Kernel-based density map generation for dense object counting","authors":"Jia Wan, Qingzhong Wang, Antoni B Chan","year":"2020","venue":"Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yifan et al., 2020] Yang Yifan, Li Guorong, et al. Weakly- supervised crowd counting learns from sorting rather than locations. In ECCV, pages 1-17, 2020.","title":"Weaklysupervised crowd counting learns from sorting rather than locations","authors":"Yang Yifan, Li Guorong","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yingying et al., 2016] Zhang Yingying, Zhou Desen, et al. Single-image crowd counting via multi-column convolu- tional neural network. In Conference on Computer Vision and Pattern Recognition, pages 589-597, 2016.","title":"Single-image crowd counting via multi-column convolutional neural network","authors":"Zhang Yingying, Zhou Desen","year":"2016","venue":"Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yuhong et al., 2018] Li Yuhong, Zhang Xiaofan, and Chen Deming. Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes. In Confer- ence on Computer Vision and Pattern Recognition, pages 1091-1100, 2018.","title":"Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes","authors":"Li Yuhong, Chen Zhang Xiaofan, Deming","year":"2018","venue":"Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ze et al., 2021] Liu Ze, Lin Yutong, et al. Swin trans- former: Hierarchical vision transformer using shifted win- dows. In International Conference on Computer Vision, pages 10012-1110022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Lin Liu Ze, Yutong","year":"2021","venue":"International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zheng et al., 2021] Sixiao Zheng, Jiachen Lu, et al. Re- thinking semantic segmentation from a sequence-to- sequence perspective with transformers. In Conference on Computer Vision and Pattern Recognition, pages 6881- 6890, 2021.","title":"Rethinking semantic segmentation from a sequence-tosequence perspective with transformers","authors":"Zheng","year":"2021","venue":"Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}