{"type":"ScholarlyArticle","title":"Crowd Counting using Deep Recurrent Spatial-Aware Network","authors":[{"name":"Lingbo Liu","slug":"lingbo-liu","given":"Lingbo","family":"Liu","affiliation":"School of Data and Computer Science , Sun Yat-sen University , Guangzhou , China"},{"name":"Hongjun Wang","slug":"hongjun-wang","given":"Hongjun","family":"Wang","affiliation":"School of Data and Computer Science , Sun Yat-sen University , Guangzhou , China"},{"name":"Guanbin Li","slug":"guanbin-li","given":"Guanbin","family":"Li","affiliation":"School of Data and Computer Science , Sun Yat-sen University , Guangzhou , China"},{"name":"Wanli Ouyang","slug":"wanli-ouyang","given":"Wanli","family":"Ouyang","affiliation":"School of Electrical and Information Engineering , The University of Sydney , Sydney , Australia"},{"name":"Liang Lin","slug":"liang-lin","given":"Liang","family":"Lin","affiliation":"School of Data and Computer Science , Sun Yat-sen University , Guangzhou , China"}],"abstract":"Crowd counting from unconstrained scene images is a crucial task in many real-world applications like urban surveillance and management, but it is greatly challenged by the camera’s perspective that causes huge appearance variations in people’s scales and rotations. Conventional methods address such challenges by resorting to fixed multi-scale architectures that are often unable to cover the largely varied scales while ignoring the rotation variations. In this paper, we propose a unified neural network framework, named Deep Recurrent Spatial-Aware Network, which adaptively addresses the two issues in a learnable spatial transform module with a region-wise refinement process. Specifically, our framework incorporates a Recurrent Spatial-Aware Refinement (RSAR) module iteratively conducting two components: i) a Spatial Transformer Network that dynamically locates an attentional region from the crowd density map and transforms it to the suitable scale and rotation for optimal crowd estimation; ii) a Local Refinement Network that refines the density map of the attended region with residual learning.  Extensive experiments on four challenging benchmarks show the effectiveness of our approach. Specifically, comparing with the existing best-performing methods, we achieve an improvement of 12\\% on the largest dataset WorldExpo’10 and 22.8\\% on the most challenging dataset UCF\\_CC\\_50","doi":"10.24963/ijcai.2018/118","url":"https://ijcai.org/proceedings/2018/118","pdf":"https://ijcai.org/proceedings/2018/0118.pdf","markdown":"https://ijcai.org/proceedings/2018/118.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/118","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":118,"pages":{"first":849,"last":855},"keywords":["Machine Learning: Deep Learning","Computer Vision: Computer Vision","Computer Vision: Video: Events, Activities and Surveillance"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arX- iv:1603.04467, 2016.","title":"Tensorflow: Large-scale machine learning on heterogeneous distributed systems","authors":"Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Boominathan et al., 2016] Lokesh Boominathan, Srini- vas SS Kruthiventi, and R Venkatesh Babu. Crowdnet: A deep convolutional network for dense crowd counting. In ACM MM, pages 640-644. ACM, 2016.","title":"Crowdnet: A deep convolutional network for dense crowd counting","authors":"Lokesh Boominathan, Srinivas Ss Kruthiventi, R Venkatesh, Babu","year":"2016","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chan et al., 2008] Antoni B Chan, Zhang-Sheng John Liang, and Nuno Vasconcelos. Privacy preserving crowd monitoring: Counting people without people models or tracking. In CVPR, pages 1-7. IEEE, 2008.","title":"Privacy preserving crowd monitoring: Counting people without people models or tracking","authors":"Zhang-Sheng John Antoni B Chan, Nuno Liang, Vasconcelos","year":"2008","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2012] Ke Chen, Chen Change Loy, Shaogang Gong, and Tony Xiang. Feature mining for localised crowd counting. In BMVC, volume 1, page 3, 2012.","title":"Feature mining for localised crowd counting","authors":"Ke Chen, Chen Change Loy, Shaogang Gong, Tony Xiang","year":"2012","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2013] Ke Chen, Shaogang Gong, Tao Xiang, and Chen Change Loy. Cumulative attribute space for age and crowd density estimation. In CVPR, pages 2467-2474, 2013.","title":"Cumulative attribute space for age and crowd density estimation","authors":"Ke Chen, Shaogang Gong, Tao Xiang, Chen Change Loy","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2016] Tianshui Chen, Liang Lin, Lingbo Liu, Xiaonan Luo, and Xuelong Li. Disc: Deep image saliency computing via progressive representation learning. IEEE TNNLS, 27(6):1135-1149, 2016.","title":"Disc: Deep image saliency computing via progressive representation learning","authors":"Chen","year":"2016","venue":"IEEE TNNLS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2017] Tianshui Chen, Zhouxia Wang, Guanbin Li, and Liang Lin. Recurrent attentional reinforcemen- t learning for multi-label image recognition. arXiv preprint arXiv:1712.07465, 2017.","title":"Recurrent attentional reinforcement learning for multi-label image recognition","authors":"Chen","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Idrees et al., 2013] Haroon Idrees, Imran Saleemi, Cody Seibert, and Mubarak Shah. Multi-source multi-scale counting in extremely dense crowd images. In CVPR, pages 2547-2554, 2013.","title":"Multi-source multi-scale counting in extremely dense crowd images","authors":"Haroon Idrees, Imran Saleemi, Cody Seibert, Mubarak Shah","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jaderberg et al., 2015] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In NIPS, pages 2017-2025, 2015.","title":"Spatial transformer networks","authors":"Jaderberg","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arX- iv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kuen et al., 2016] Jason Kuen, Zhenhua Wang, and Gang Wang. Recurrent attentional networks for saliency detec- tion. arXiv preprint arXiv:1604.03227, 2016.","title":"Recurrent attentional networks for saliency detection","authors":"Jason Kuen, Zhenhua Wang, Gang Wang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lempitsky and Zisserman, 2010] Victor Lempitsky and An- drew Zisserman. Learning to count objects in images. In NIPS, 2010.","title":"Victor Lempitsky and Andrew Zisserman. Learning to count objects in images","authors":"Zisserman Lempitsky","year":"2010","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2017] Guanbin Li, Yuan Xie, Liang Lin, and Y- izhou Yu. Instance-level salient object segmentation. In CVPR, pages 247-256. IEEE, 2017.","title":"Instance-level salient object segmentation","authors":"Li","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Onoro-Rubio and López-Sastre, 2016] Daniel Onoro- Rubio and Roberto J López-Sastre. Towards perspective- free object counting with deep learning. In ECCV, pages 615-629. Springer, 2016.","title":"Daniel Onoro-Rubio and Roberto J López-Sastre. Towards perspectivefree object counting with deep learning","authors":"Onoro-Rubio, López-Sastre","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pham et al., 2015] Viet-Quoc Pham, Tatsuo Kozakaya, Os- amu Yamaguchi, and Ryuzo Okada. Count forest: Co- voting uncertain number of targets using random forest for crowd density estimation. In ICCV, pages 3253-3261, 2015.","title":"Count forest: Covoting uncertain number of targets using random forest for crowd density estimation","authors":"Pham","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sam et al., 2017] Deepak Babu Sam, Shiv Surya, and R Venkatesh Babu. Switching convolutional neural net- work for crowd counting. In CVPR, volume 1, page 6, 2017.","title":"Switching convolutional neural network for crowd counting","authors":"Babu Deepak, Shiv Sam, R Surya, Babu Venkatesh","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shang et al., 2016] Chong Shang, Haizhou Ai, and Bo Bai. End-to-end crowd counting via joint learning local and global count. In ICIP, pages 1215-1219. IEEE, 2016.","title":"End-to-end crowd counting via joint learning local and global count","authors":"Chong Shang, Haizhou Ai, Bo Bai","year":"2016","venue":"ICIP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sindagi and Patel, 2017a] Vishwanath A Sindagi and Vishal M Patel. Cnn-based cascaded multi-task learning of high-level prior and density estimation for crowd counting. In AVSS, pages 1-6. IEEE, 2017.","title":"Cnn-based cascaded multi-task learning of high-level prior and density estimation for crowd counting","authors":"Patel ; Sindagi, A Vishwanath, Sindagi, M Vishal, Patel","year":"2017","venue":"AVSS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sindagi and Patel, 2017b] Vishwanath A Sindagi and Vishal M Patel. Generating high-quality crowd density maps using contextual pyramid cnns. In ICCV, pages 1879-1888. IEEE, 2017.","title":"Generating high-quality crowd density maps using contextual pyramid cnns","authors":"Patel ; Sindagi, A Vishwanath, Sindagi, M Vishal, Patel","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Walach and Wolf, 2016] Elad Walach and Lior Wolf. Learning to count with cnn boosting. In ECCV, pages 660-676. Springer, 2016.","title":"Learning to count with cnn boosting","authors":"Wolf ; Walach, Elad Walach, Lior Wolf","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2017] Zhouxia Wang, Tianshui Chen, Guanbin Li, Ruijia Xu, and Liang Lin. Multi-label image recog- nition by recurrently discovering attentional regions. In ICCV, 2017.","title":"Multi-label image recognition by recurrently discovering attentional regions","authors":"Wang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xiong et al., 2017] Feng Xiong, Xingjian Shi, and Dit-Yan Yeung. Spatiotemporal modeling for crowd counting in videos. In ICCV, pages 5161-5169. IEEE, 2017.","title":"Spatiotemporal modeling for crowd counting in videos","authors":"Feng Xiong, Xingjian Shi, Dit-Yan Yeung","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2015] Cong Zhang, Hongsheng Li, Xiaogang Wang, and Xiaokang Yang. Cross-scene crowd counting via deep convolutional neural networks. In CVPR, pages 833-841, 2015.","title":"Cross-scene crowd counting via deep convolutional neural networks","authors":"Cong Zhang, Hongsheng Li, Xiaogang Wang, Xiaokang Yang","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2016] Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, and Yi Ma. Single-image crowd counting via multi-column convolutional neural network. In CVPR, pages 589-597, 2016.","title":"Single-image crowd counting via multi-column convolutional neural network","authors":"Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, Yi Ma","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2017] Shanghang Zhang, Guanhang Wu, Joao P Costeira, and José MF Moura. Fcn-rlstm: Deep spatio-temporal neural networks for vehicle counting in city cameras. In ICCV, pages 3687-3696. IEEE, 2017.","title":"Fcn-rlstm: Deep spatio-temporal neural networks for vehicle counting in city cameras","authors":"Zhang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhu et al., 2017] Feng Zhu, Hongsheng Li, Wanli Ouyang, Nenghai Yu, and Xiaogang Wang. Learning spatial reg- ularization with image-level supervisions for multi-label image classification. In CVPR, pages 5513-5522, 2017.","title":"Learning spatial regularization with image-level supervisions for multi-label image classification","authors":"Feng Zhu, Hongsheng Li, Wanli Ouyang, Nenghai Yu, Xiaogang Wang","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}