Crowd Counting using Deep Recurrent Spatial-Aware Network

1 Introduction

Crowd counting, which aims at estimating the total number of people in unconstrained crowded scenes, has received increasing research interests in recent years due to its potential application in many real-world scenarios, such as video surveillance [Xiong et al., 2017] and traffic monitoring [Zhang et al., 2017]. Despite recent progress, crowd counting remains a challenging issue, especially in the face of extreme occlusions, changes in lighting and camera perspective.

∗Corresponding author is Guanbin Li.

In recent two years, deep convolutional neural networks have been widely used in crowd counting and have made substantial progress [Sindagi and Patel, 2017a; Onoro-Rubio and L´opez-Sastre, 2016; Sam et al., 2017; Xiong et al., 2017; Zhang et al., 2015]. The success of most existing deep models stems from the effective modeling of the scale variation of people/heads and the varying degrees of regional crowd density. However, all of these methods, without exception, obtain the density estimation of the whole image by merging the prediction results of a number of pre-designed fixed subnetwork structures (working on the whole image or image patches). Specifically, they are either proposed to fuses the features from multiple convolutional neural networks with different receptive fields to handle the scale variation of people groups [Zhang et al., 2016], or directly divide crowd scene into multiple non-overlapping patches and provide a pool of regression networks for each patch selection [Sam et al., 2017]. Although these strategies can, to a certain extent, improve the adaptability of the prediction to images of diverse density regions or people of various scales, the limited enumeration of a fixed number of tailor-designed networks or several sizes of receptive fields can not well cope with all the scenarios after all. Most importantly, none of these algorithms take into account the impact of different pose and photographic angles on crowd density estimation while crafting their network structures. As shown in Figure 1, the camera viewpoints in various scenes create different perspective effects and may result in large variation of scales, in-plane and out-plane rotation of people.

To address the aforementioned concerns, we propose a Deep Recurrent Spatial-Aware Network for crowd counting. The core of our network is a Recurrent Spatial-Aware Refinement (RSAR) module, which recurrently conducting refinement on an initial crowd density map through adaptive region selection and residual learning. Specifically, the RSAR module consists of two alternately performed components: i) a Spatial Transformer Network is incorporated in each LSTM step for simultaneously region cropping and warping, which allows the network to adaptively cope with the various degrees of congestion, people scale and rotation variation in the same scene; ii) a Local Refinement Network refines the density map of the selected region. In general, the main contributions of this work are three-fold.

• We provide an adaptive mode to simultaneously handle the effect of both scale and rotation variation by introducing a spatial transform module for crowd counting. To the best of our knowledge, we are the first to address the issue of the rotation variation on this task.
• We propose a novel deep recurrent spatial-aware network framework to recurrently select a region (with learnable scale and rotation parameters) from an initial density map for refinement, dependent on feature warping and residual learning.
• Extensive experiments and evaluations on several public benchmarks show that our proposed method achieves superior performance in comparison to other state-of-theart methods. Specifically, compared with the existing best-performing methods, we achieve an improvement of 12% on the WorldExpo'10 dataset and 22.8% on the most challenging UCF CC 50 dataset.

2 RelatedWork

Deep learning Methods for Crowd Counting: Inspired by the significant progress of deep learning on various computer vision tasks (such as image classification[Zhu et al., 2017; Chen et al., 2017] and salient object detection[Chen et al., 2016; Li et al., 2017]), many researchers also have attempted to adapt the deep neural network to the task of crowd counting and achieved great success. Most of the existing methods addressed the scale variation of people with multi-scale architectures. Boominathan et al. [Boominathan et al., 2016] proposed to tackle the issue of scale variation using a combination of shallow and deep networks along with an extensive data augmentation by sampling patches from multi-scale image representations. HydraCNN [Onoro-Rubio and L´opezSastre, 2016] proposed to learn a non-linear regression model which used a pyramid of image patches extracted at multiple scales to perform the final density prediction. A pioneering work was proposed by Zhang et al. [Zhang et al., 2016], in which they utilized multi-column convolutional neural networks with different receptive fields to learn scale-robust features. Sam et al. [Sam et al., 2017] proposed a SwitchingCNN to map a patch from the crowd scene to one of the three regression networks, each of which can handle the particular range of scale variation. CP-CNN [Sindagi and Patel, 2017b] proposed a Contextual Pyramid CNN to generate high-quality crowd density estimation by incorporating global and local contextual information into the multi-column networks. However, the above-mentioned methods have two significant limitations. First, as a neural network with the fixed static receptive field can only handle a limited enumeration of scale variation, these methods do not scale well to large scale change and cannot well cope with all level of scale variation in diverse scenarios. Second, they did not take the rotation variation of people into consideration, which limits the models' robustness towards camera perspective variations. To the best of our knowledge, we are the first work to simultaneously address the issue of large scale and rotation variations in an adaptively learnable mode on this task.

Deep learning Methods for Crowd Counting: Spatial transformer Network (STN) [Jaderberg et al., 2015] is a subdifferentiable sampling-based module, which is designed to spatially transform its input map to an output map that corresponds to a subregion of the input map and can be hence regarded as an effective region selection mechanism. It is convenient to incorporate a spatial transformer layer to the convolutional neural network and train it with the standard back-propagation algorithm. A parameter matrix is used to determine the location of the subregion, as well as its resize scale and the rotation angle. Recently, the spatial transformer has been applied to various computer vision tasks, e.g., multilabel image recognition [Wang et al., 2017] and saliency detection [Kuen et al., 2016]. To the best of our knowledge, our work is the first in successfully using multiple iterations of STN within a LSTM framework for crowd counting.

3 ProposedMethod

As illustrated in Figure 2, we propose a novel Deep Recurrent Spatial-Aware Network for crowd counting, which is composed of two modules, including a Global Feature Embedding (GFE) module and a Recurrent Spatial-Aware Refinement (RSAR) module. Specifically, the GFE module takes the whole image as input for global feature extraction, which is further used to estimate an initial crowd density map. And then the RSAR module is applied to iteratively locate image regions with a spatial transformer-based attention mechanism and refine the attended density map region with residual learning. For convenience in the following, we denote the crowd density map in the i-th iteration as Mi. Noted that M0 is the initial crowd density map.

3.1 Global Feature Embedding

The goal of the Global Feature Embedding module is to transform the input image into high-dimensional feature maps, which is further used to generate an initial crowd density map of the image. Inspired by previous works [Zhang et al., 2016; Sindagi and Patel, 2017b], we develop our GFM module in a multi-scale mode. As shown in Figure 3(a), the GFE module is composed of three columns of CNNs, each of which has seven convolutional layers with different kernel sizes and channel numbers as well as three max-pooling layers. Given an image I, we extract its global feature g by feeding it into GFM and concatenating the outputs of all the columns. After obtaining the global feature g, we generate the initial crowd density map M0 of image I using a convolutional layer with a kernel size of 1×1, which can be expressed as:

where Conv denotes the convolution operation. As shown in Figure 4, the initial crowd density is too rough to estimate the number of people in the image. To further improve the quality of crowd density map, we propose a Recurrent Attentive Refinement (RSAR) module to iteratively refine the density map, which will be described in the next subsection.

3.2 Recurrent Spatial-Aware Refinement

In this section, we propose a novel Recurrent Attentive Refinement (RSAR) module to iteratively refine the crowd density map. Our proposed RSAR consists of two alternately performed components: i) a Spatial Transformer Network dynamically locates an attentional region from the crowd density map; ii) a Local Refinement Network refines the density map of the selected region with residual learning. A highquality crowd density map with accurately estimated crowd number would be acquired after a refinement of n iterations.

Attentional Region Localization

In the i-th iteration, we first determine which image region to be refined with a spatial transformer network. We encode Mi-1 into a 512-dimensions feature with a fully-connected layer and put it into a Long Short-Term Memory (LSTM) [Hochreiter and Schmidhuber, 1997] layer, which can be formulated as:

where ci, hi are the memory cell and hidden state of current iteration and FC is the fully-connected layer. The LSTM is utilized for capturing the past information of the density maps and transformation actions. We define the transformation matrix Ti of the spatial transformer as:

which allows cropping, translation, scale and rotation to be applied to the input map. We take the hidden state hi to calculate the parameters of transformation matrix Ti with a fully-connected layers. Then we extract a region density map ri from the whole density map Mi-1 according to the transformation matrix Ti, which can be expressed as:

where ST denotes the spatial transformer. The region map ri will be resized to a given size w ∗ h by bilinear interpolation.

Region Density Map Refinement

Inspired by the previous works[Sindagi and Patel, 2017a; 2017b] which have validated the importance of global contextual modeling in crowd counting, we also take into consideration the global context when performing attended density region refinement. Intuitively, the global context contains the density level and density distribution of the given image. We encode the global feature g described in section 3.1 with two stacked fully-connected layers. The former one has 256 neurons and the latter one has w ∗ h neurons. We construct the global context map cg by reshaping the output of the last FC layer to a new tensor with a size of w ∗ h.

After obtaining the region map ri and global context map cg, we can perform crowd map refinement on the attentional region with a Local Refinement Network, which learns the residual between the region map ri and the ground truth of its corresponding region. As shown in Figure 3(b), the Local Refinement Network also consists of three columns of CNNs. It takes the concatenation of ri and cg as the input and calculates the residual map of the attentional region. Finally, we can get a new crowd density map Mi by adding the inverse transformed residual map to the Mi-1, which can be described as:

where IST is the inverse spatial transformer that can transform the residual map back to the attentional region according to the inverse of Ti, denoted as T -1 i .

3.3 Networks Optimization

For the existence of fully-connected layers in our model, we need to ensure that all the input images are of the same resolutions. Therefore, when optimizing the networks, we transform all training images or patches to the same resolutions and the details will be described in section 4. Suppose there are N training images after data augmentation, and each image Ii is annotated with a set of 2D points Li = {l1, ..., lC(i)}, where C(i) is the total number of the labeled pedestrian in the image. We generate ground truth density map Di of image Ii with the specific strategy introduced by Lempitsky et al. [Lempitsky and Zisserman, 2010], which ensures that the sum of the ground truth density map ∑p∈Ii Di(p) over the entire image Ii matches its crowd count with some negligible deviation.

Our networks are trained in an end-to-end manner and the overall loss function is defined as:

where M 0 i and M n i are the initial map and the refined density map for image Ii. We adopt the TensorFlow [Abadi et al., 2016] toolbox to implement our crowd counting network. The filter weights of all convolutional layers and fullyconnected layers are initialized by truncated normal distribution with a deviation equal to 0.01. The learning rate is set to

10-4 initially and multiplied by 0.98 every 1K training iterations. The batch size is set to 1. We optimize our networks parameters with Adam optimization [Kingma and Ba, 2014] by minimizing the loss function Eq.(8).

4 Experiments

In this section, we first compare our method with recent stateof-the-art methods of the crowd counting task on four public challenging datasets. We further conduct extensive ablation studies to demonstrate the effectiveness of each component of our model.

4.1 EvaluationMetric

Following the existing work [Zhang et al., 2016; Sindagi and Patel, 2017b], we adopt the mean absolute error (MAE) and mean squared error (MSE) as metrics to evaluate the accuracy of crowd counting estimation, which are defined as:

where N is the total number of testing images, pi and ˆ pi are the ground truth count and estimated count of the i-th image respectively. As discussed in section 3.3, ˆ pi can be calculated by summing over the estimated crowd density map.

4.2 Evaluations and Comparisons

ShanghaiTech [Zhang et al., 2016]. This dataset contains 1,198 images of unconstrained scenes with a total of 330,165 annotated people. And it is divided into two parts: Part A with 482 images crawled from the Internet, and Part B with 716 images taken from the busy shopping streets. For each part, we resize the images to its maximal resolution when training and testing.

We compare our method with five recently published stateof-the-art methods on this dataset, i.e., Zhang et al. [Zhang et al., 2015], MCNN [Zhang et al., 2016], CascadedMTL [Sindagi and Patel, 2017a], Switching-CNN [Sam et al., 2017], CP-CNN [Sindagi and Patel, 2017b]. As shown in Table 1, our proposed method outperforms other competing methods on both parts of the ShanghaiTech dataset. Specifically, our method achieves a significant improvement of 49.7% in MAE and 39.5% in MSE over the existing bestperforming algorithm CP-CNN on Part B.

UCF CC 50 [Idrees et al., 2013]. As an extremely challenging benchmark, this dataset contains 50 annotated images of diverse scenes collected from the Internet. Except for different resolutions, aspect ratios and perspective distortions, the images of this dataset also suffer from a wide range of person counts, varying from 94 to 4,543. Following the standard protocol discussed in [Idrees et al., 2013], we split the dataset into five subsets and perform a five-fold crossvalidation. When training, we randomly crop some regions with a range of [0.5, 0.9] from the original images and resize them to 1024 ∗ 768. The testing images are directly resized to the same resolution.

We perform comparison against several state-of-theart methods based on deep-learning, including Zhang et al. [Zhang et al., 2015], MCNN [Zhang et al., 2016], Walach et al. [Walach and Wolf, 2016], Cascaded-MTL [Sindagi and Patel, 2017a], Switching-CNN [Sam et al., 2017], CPCNN [Sindagi and Patel, 2017b] and ConvLSTM-nt [Xiong et al., 2017]. As shown in Table 2, our method achieves superior performance in comparison to other competing methods on UCF CC 50 dataset. Specifically, our method outperforms the existing best-performing method ConvLSTM-nt by 22.8% over the MAE metric.

MALL [Chen et al., 2012]. This dataset was captured by a publicly accessible surveillance camera in a shopping mall with more challenging lighting conditions and glass surface reflections. The video sequence consists of 2,000 frames with a resolution of 320 ∗ 240. Following the same setting as [Chen et al., 2012], we use the first 800 frames for training and the remaining 1,200 frames for evaluation. The images are kept the original resolution when training and testing.

We compare our method with Gaussian Process Regression [Chan et al., 2008], Ridge Regression [Chen et al., 2012], Cumulative Attribute Regression [Chen et al., 2013], Count forest [Pham et al., 2015] and ConvLSTM [Zhang et al., 2016]. The results of all methods are summarized in Table 3. Our still-image based method outperforms the existing best method ConvLSTM, which utilized extra temporal dependencies from surveillance video to estimate the crowd count.

WorldExpo'10 [Zhang et al., 2015]. This dataset contains 1,132 video sequences captured by 108 surveillance cameras during the Shanghai WorldExpo in 2010. The training set consists of 3,380 annotated frames from 103 scenes, while the testing images are extracted from other five different scenes with 120 frames per scene. The images are kept the original resolution for training and testing. When testing, the Region of Interest (ROI) maps for each scene are provided and we only consider the crowd count under the ROI.

We compare our method with seven recently published state-of-the-art methods, including Chen et al. [Chen et al., 2013], Zhang et al. [Zhang et al., 2015], MCNN [Zhang et al., 2016], Shang et al. [Shang et al., 2016], SwitchingCNN [Sam et al., 2017], CP-CNN [Sindagi and Patel, 2017b] and ConvLSTM [Xiong et al., 2017]. As demonstrated in Table 4, our method achieves state-of-the-art performance with respect to the average MAE of five scenes. Specifically, our model gets the lowest MAE in Scene 2 and Scene 5, which are the two most challenging scenes in the testing set.

4.3 Ablation Study

In this section, we perform extensive ablation studies on ShanghaiTech dataset and demonstrate the effects of several components in our framework.

Effectiveness of Spatial Transformation: The transformation matrix defined in Eq.(3) allows translation, rotation and scale to be applied to the input map. To validate the effectiveness of each of them, we train our model with three different constraints of spatial transformer: i) spatial transformation only with translation by directly setting θ11, θ22 to 1 and θ12, θ21 to 0, denoted as T ; ii) spatial transformation with translation and scale but without rotation by directly setting θ12, θ21 to 0, denoted as T +S; iii) spatial transformation with translation, rotation and scale, denoted as T +S +R. The performance of these variants are summarized in Table 7 and the performance of initial density map is denoted as "Basic". We can see that directly locating a region without scale and rotation operation for refinement can reduce the MAE from 83.1 to 75.9 on ShanghaiTech Part A. When dynamically resizing the local regions, the MAE on Part A can be reduced to 73.2. When simultaneously resizing and rotating the local regions, we can get the best performance on ShanghaiTech dataset. The experiment has demonstrated the effectiveness of the spatial transformation for crowd density map refinement.

Effectiveness of Global Context: In order to validate the effectiveness of the global context, we train a variant of our model that refines the local region map without the global context. As shown in Table 6, the performance would drop over all metrics when removing the global context. Actually, the global context contains the information of density level and density distribution, which is beneficial for the crowd counting inference.

Effectiveness of Recurrent Refinement: To validate the superiority of the recurrent refinement mechanism of our method, we train our model with different refinement iterations. As shown in Table 7, the performance gradually increases with more iterations and it slightly drops until 40 steps. Our method gets the best performance with 30 iterations, therefore we set the refinement iteration n to 30 in our model for all datasets. Figure 4 shows the visual comparison with the refined maps and the initial maps and we can see that our method can generate more accurate crowd density maps after multi-step refinements.

5 Conclusion

In this paper, we introduce a novel Deep Recurrent SpatialAware Network for crowd counting which simultaneously models the variations of crowd density as well as the pose changes in a unified learnable module. It can be regarded as a general framework for crowd map refinement. Extensive experiments on four challenging benchmarks show that our proposed method achieves superior performance in comparison to the existing state-of-the-art methods. In our future research, we plan to delve into the research of incorporating our model in other existing crowd flow prediction framework.

Acknowledgments

This work was supported by State Key Development Program under Grant 2016YFB1001004, National Natural Science Foundation of China under Grant 61622214 and Grant

61702565, and Guangdong Natural Science Foundation Project for Research Teams under Grant 2017A030312006. This work was also sponsored by CCF-Tencent Open Research Fund. Wanli Ouyang is supported by the SenseTime Group Limited.