High Resolution Feature Recovering for Accelerating Urban Scene Parsing

1 Introduction

Urban scene parsing is a significant and challenging task that benefits many applications, such as self-driving, driver assistance and traffic surveillance. The goal of urban scene parsing is to assign one of the semantic categories to each pixel within an urban scene image. Nowadays, approaches based on deep Convolutional Neural Networks (CNNs) [Krizhevsky et al., 2012] achieve remarkable success in urban scene parsing, such as Fully Convolutional Network (FCN) based frameworks [Shelhamer et al., 2017]. However, most of the existing scene parsing methods mainly focus on improving parsing accuracy through deeper networks [He et al., 2016] and higher resolution feature maps [Chen et al., 2016a] which result in lower inference speed. In practical applications, it is worth noting that both accuracy and speed are equally important in urban scene parsing.

∗Corresponding author: Sheng Tang. This work was supported by the National Key Research and Development Program of China (2017YFC0820605), and the National Natural Science Foundation of China (61525206, 61572472).

Recently, most of the network accelerating approaches are introduced to image classification task, such as parameter approximation [Yang et al., 2015; Wu et al., 2016a] or network mimicking [Hinton et al., 2015; Romero et al., 2015]. Nevertheless, there is a huge difference between classification and scene parsing. For image classification, the size of input images is small (e.g. 224×224) and the resolution of feature maps are low (e.g.1/32 of input size). On the contrary, scene parsing task usually has large-sized input images (e.g. 1024×2048 in Cityscapes dataset) and high resolution feature maps (e.g.1/8 of input size) to capture more details. Both of these two characteristics lead to low inference speed but are not encountered in classification tasks. To tackle this issue, many approaches down-sample the input images [Paszke et al., 2016; Badrinarayanan et al., 2017], which speeds up the inference process but sacrifices the parsing accuracy due to missing details during down-sampling.

Aiming at the problem of low inference speed due to largesized input and high resolution feature maps in scene parsing, in this paper, we propose a High Resolution Feature Recovering (HRFR) framework to accelerate the inference process of a given scene parsing network. It combines the advantages of the following two cases while trying to avoid both disadvantages: (1) fast parsing with down-sampled input at the cost of low parsing accuracy as shown in Figure 1 (a); (2) high accuracy with large original-sized input at the cost of low inference speed as shown in Figure 1 (b). Briefly speaking, the core idea of our HRFR is to recover features of large originalsized images from features of down-sampled input, as shown in Figure 1 (c). High efficiency can be achieved since learning features from the down-sampled input is much faster than that from the original-sized input. Moreover, recovering high resolution features of original-sized input can provide more information about details and reduce the dropping of parsing accuracy. This is feasible because the features learned from the above two cases are overall similar in spite of blur boundaries and missing details in feature maps of down-sampled input, as visualized in Figure 2.

Particularly, we apply a Super-Resolution Recovering (SRR) module to perform feature recovering. The SRR module consists of several convolutional/deconvolutional layers with a bottle-neck structure, and can be jointly learned with the whole framework end-to-end. Additionally, we employ auxiliary intermediate supervision and boundary region reweighting to enhance the optimization of the network.

Our proposed framework is oriented toward urban scene parsing by focusing on the problem of large-sized input and high resolution feature maps, wholly different with classification-based network accelerating approaches. The proposed method is orthogonal to most of the existing network accelerating methods and can be combined with them for further acceleration. Experimental results on the challenging Cityscapes dataset [Cordts et al., 2016] and CamVid dataset [Brostow et al., 2009] show the effectiveness of our proposed method, which can accelerate the scene parsing inference process by about 3.0× speedup from 1/2 downsampled input with negligible accuracy reduction.

2 RelatedWork

Scene parsing Compared with graph-based methods [Xie et al., 2014], effective and efficient approaches based on CNNs [Krizhevsky et al., 2012] achieve extraordinary success in scene parsing task, such as FCN based frameworks [Shelhamer et al., 2017; Chen et al., 2016a] and Deconvolutional Network based frameworks [Badrinarayanan et al., 2017]. Most of them aim at improving the parsing accuracy. Some approaches obtain higher resolution feature maps through atrous convolutions [Chen et al., 2016a] or fullresolution residual architecture [Pohlen et al., 2017]. Some approaches exploit features of multiple scales from intermediate layers [Mostajabi et al., 2015; Ghiasi and Fowlkes, 2016], spatial pyramid structures [Zhao et al., 2017; Chen et al., 2016a], multiple-sized input [Chen et al., 2016b] or adaptive learning [Zhang et al., 2017b]. Some approaches utilize contextual information and spatial dependencies through Recurrent Neural Networks (RNNs) [Shuai et al., 2016; Liang et al., 2016] or graphical models [Liu et al., 2015;

(d) Features of Down-sampled Input

Figure 2: Although features learned from the original-sized input and down-sampled input have similar responses, features learned from the original-sized input (c) have more details and more distinct boundaries than that from the down-sampled input (d).

Vemulapalli et al., 2016; Lin et al., 2016]. Other approaches focus on refinement to boost the accuracy of existing parsing predictions [Yu and Koltun, 2016; Zhang et al., 2017a]. However, most of these approaches only consider the accuracy but ignore the inference speed. To boost the parsing accuracy, they utilize complicated structures which are slow and maybe not suitable for self-driving and traffic surveillance.

Network Acceleration Network acceleration has received considerable attention in the recent years. Some approaches investigate approximation-based methods to reduce the redundancy in parameters and accelerate networks through lowrank decompositions of filters [Denton et al., 2014; Jaderberg et al., 2014; Yang et al., 2015], product quantization of network parameters [Han et al., 2016; Wu et al., 2016a] or approximate networks by binary weights [Soudry et al., 2014; Courbariaux et al., 2015]. Some researchers improve the efficiency of networks by distilling the knowledge from multiple cumbersome models [Hinton et al., 2015]. Network mimicking methods extend the distillation approach by training a student network with fewer parameters from mimicking a teacher network [Romero et al., 2015; Ba and Caruana, 2014; Iandola et al., 2016; Li et al., 2017], so as to reduce the computation cost while preserving the accuracy. However, most of these model accelerating approaches focus on speedup models of image classification. They consider the redundancy of network structure instead of high resolution of feature maps, which remains a problem leading to slow inference speed and high memory cost in scene parsing.

In this paper, we propose the HRFR framework to speed up a given scene parsing network. In contrast to the existing network accelerating methods which focus on network structures of image classification, the proposed method aims at solving the problem of large-sized input and high resolution feature maps in scene parsing.

3 High Resolution Feature Recovering

In this section, we first introduce the overall structure of the proposed HRFR framework, as shown in Figure 3, followed by the description of the important SRR module, auxiliary intermediate supervision and boundary region re-weighting. Finally, we provide the complexity analysis.

3.1 Overall Framework

Given an urban scene image x (with the original size), the goal of urban scene parsing is to output the pixel-level prediction yN through a feed-forward convolutional neural network N, formulated as

Most of the popular scene parsing frameworks are based on the FCN framework, which can be decomposed into two consecutive subnets, feature learning subnet and parsing subnet. The feature learning subnet often contains dozens or hundreds of convolutional layers transferred from classification networks, while the parsing subnet only contains several layers initialized by random values. Thus, we decompose N into the feature learning subnet Nfea and parsing subnet Npar, thus Equation (1) is decomposed to

where f represents the feature maps learned from the feature learning subnet. These feature maps encode rich semantic concepts and spatial location information from the urban scene image x. We name model N as the "teacher network", which represents the network trained on the original-sized images to learn feature maps of a high resolution.

The goal of the proposed HRFR method is to learn a "student network" M with down-sampled scene images to recover the high resolution f learned from the teacher network N. We set the student network to the same architecture with the teacher network for simplification, but the input sizes of them are different. The feature maps fd learned fromMhave a low resolution, formulated as

where xd denotes the image down-sampled from x, Mfea denotes the feature learning subnet of M. To recover the high resolution feature maps f learned from N, we design a SRR moduleMSRR, which recovers the low resolution fd to the high resolution fup guided by f, formulated as

Finally, the parsing prediction yM of M is obtained from the recovered high resolution feature maps fup:

In the training stage, we aim at optimizing the student network M (including Mfea, MSRR and Mpar) guided by the trained teacher network N. The recovered feature maps fup is expected to approximate the high resolution f learned from N. Therefore, we use the L2 distance LL2 to minimize the differences between fup and f:

To further optimize the parsing subnet Mpar, the original softmax loss Ls between parsing prediction yM and groundtruth should be preserved. Thus the overall objective function L is defined as

where λ1 and λ2 are the loss weights. The feature maps f learned from parsing networks are often high-dimensional and difficult to be regressed. To provide a good starting state for the optimization process, we initialize Mfea and Mpar in the student network with the parameters from Nfea and Npar in the teacher network instead of that from the classification network. This is because the teacher network can capture the semantic concepts of scenes better than image classification networks. What's more, feature maps generated from original-sized input and down-sampled input are similar, which can also reduce the difficulty of feature recovering. Additionally, parameters of MSRR are initialized with random values sampled from the Gaussian distribution. In the inference stage, we only use the student network and the down-sampled scene image to gain the parsing prediction. The teacher network is only exploited to optimize the student network during training.

3.2 Super-Resolution Recovering

The SRR module MSRR is designed for generating high resolution fup from the low resolution fd. For convenience, we suppose the input scene images are down-sampled by a factor of 1/2n , n = 1, 2, · · · . Thus the size of resultant low resolution feature maps fd is correspondingly reduced to 1/2n times of high resolution f. To recover the responses of f, we enlarge fd to the same size of f by exploiting n deconvolutional layers with kernel= 4, stride= 2, each of which can magnify the feature maps by a factor of 2. The structure of several deconvolutional layers is compatible with the whole network structure, so that all the components can be conveniently integrated for joint training in an end-to-end manner.

It is worth noting that the channel numbers of feature maps fup and f are usually very large, e.g. 2048 for ResNet. Thus each deconvolutional layer connects the two highdimensional ends and has a large number of parameters of 4×4×c×c, where c is the channel number and 4 is the kernel size. These huge number of parameters are redundant, causing slow inference speed. To solve this problem, we modify the structure of SRR module with a bottle-neck design inspired by [He et al., 2016]. We add 1×1 convolutional layers at the beginning and ending of SRR module for reducing and restoring dimensions respectively. Suppose 1×1 convolutional layers reduce the dimension from c to k(k < c), the parameter numbers of SRR module will decrease from n×42 × c2 to n×42 ×k2 +2× c ×k. For example, if we set k = c/4, the bottle-neck structure will remove more than 90% parameters compared with the original structure. Experiments also show that the bottle-neck design will speed up the inference process without damaging the parsing accuracy.

3.3 Auxiliary Intermediate Supervision

Deeper CNNs can bring better accuracy but lead to additional optimization difficulties, which can also be exhibited in the proposed HRFR framework. To conquer this problem, we exploit two auxiliary regression losses. Intuitively, if the feature maps from previous layers can be recovered, the feature maps of latter layers will be easier to be recovered. Therefore, we apply another two SRR modules after the 3rd and 4th blocks of the networks (e.g. the res3b3 and res4b22 of ResNet-101) to recover the intermediate feature maps and collect the regression losses, formulated as

where fb3 d , fb4 d are the intermediate feature maps learned from the 3rd and 4th blocks of the student network, fb3 up , fb4 up are the feature maps produced from the auxiliary SRR modules Maux1 SRR , Maux2 SRR . fb3 , fb4 are the feature maps learned from the 3rd and 4th blocks of the teacher network, which are employed as the target of the auxiliary regression losses Laux1 and Laux2. Thus the objective function in Equation (7) changes toL = λ1LL2 + λ2Ls + λ3(Laux1 + Laux2). (10)

We balance the auxiliary losses Laux1 + Laux2 with smaller weights λ3, so that the auxiliary losses can contribute to optimize the learning process while the loss of master regression branch LL2 and the softmax loss Ls take the most responsibility. With the help of auxiliary regression losses, the optimization of feature recovering is decomposed into three implicit stages, each is easier to optimize. Moreover, all the components of the network can be trained in balance instead of paying too much attention in the last subnet, which can also boost the parsing accuracy. During the inference stage, we simply abandon the two auxiliary branches and only recover the feature maps from the master branch.

3.4 Boundary Region Re-weighting

As visualized in Figure 2, the feature maps from the originalsized input resemble those from down-sampled input in overall. However, there are still some obvious distinctions such as blur details and inaccurate boundaries in down-sampled feature maps, which is the principal reason of accuracy degradation. We name the pixels or feature vectors near boundaries as boundary region. Thus, paying more attention to the optimization of the boundary regions will benefit feature recovering. We implement it by re-weighting the loss functions separately to increase the loss weights of predictions within boundary regions. Particularly, we obtain the boundary regions B through expanding the boundaries with a radius r. Then the objective functions are modified to

where L(p) represents the overall loss L defined in Equation (10) from the pixel p. We set γ1 > γ2 to pay more attention to optimizing the boundary regions.

3.5 Complexity Analysis

The computational cost ratio of the proposed method to the original method (i.e. the teacher network) can be formulated with the complexities of all the subnets

where O(·) measures the function complexity. Since the feature learning subnet often contains much more fully convolutional layers than those of the parsing subnet, we have O(Npar) ≪ O(Nfea) and O(Mpar) ≪ O(Mfea). Similarly, the SRR module only contains several convolutional/deconvolutional layers, so we also have O(MSRR) ≪ O(Mfea). Considering the teacher and student networks have the same architecture but different sizes of input, the complexity cost ratio is mostly determined by the downsampling factor. The speedup ratio is approximately in inverse proportion to the computational cost ratio and can be estimated with the FLOPs of the student and teacher networks.

4 Experiments

In this section, we perform experiments on the two challenging urban scene parsing benchmarks, including Cityscapes dataset and CamVid dataset.

4.1 Experimental Settings

Cityscapes Dataset The Cityscapes dataset is taken by carcarried cameras and collected in street scenes from 50 different cities. It contains 5000 images of 19 semantic classes, 2975 images for training, 500 images for validation and 1525

for testing. The Intersection over Union averaged over all the semantic categories (Mean IoU) is adopted for evaluation.

CamVidDataset The CamVid dataset is collectedwith images captured from driving videos at daytime and dusk. It contains 701 images with pixel-level annotations on 11 semantic classes. The performance is evaluated based on the average per-pixel accuracy over all the semantic categories (CA) and the overall pixel-wise accuracy (PA).

It is worth noting that images in these two datasets have high resolutions, 2048×1024 of Cityscapes and 960×720 of CamVid. Many scene parsing approaches down-sample images of the two datasets to speed up inference, which damages the parsing accuracy. Thus, it is suitable to evaluate the proposed method on these two datasets.

Implementation Details We perform the proposed method based on DeepLab-v2 [Chen et al., 2016a], which is the most popular framework used as the basis of many state-of-theart scene parsing approaches. The feature learning subnet of DeepLab-v2 is transferred from the ResNet-101 [He et al., 2016]. We add the master SRR module after the feature maps learned from the ResNet-101, i.e. the res5c layer. The two auxiliary SRR modules are applied after the 3rd and 4th blocks, i.e. the res3b3 and res4b22 layers. In the bottleneck structure, channel dimension is reduced to k=512. Postprocessing of dense CRF is not applied.

During training, the entire framework is trained end-to-end by the objective function in Equation (10), where we set the boundary radius r = 5, loss weights λ1=λ2=1, λ3=0.5, γ1=2, γ2=1 empirically. We adopt the standard stochastic gradient descent (SGD) with the mini-batch of 4 samples. The learning rate is maintained at 0.0005 for 60 epochs. We randomly crop samples of 500×500 from images, and apply horizontal flip and random resizing between 0.5 and 1.5. We evaluate the inference speed with one image per batch averaged on all images in the validation set. Our experiments are implemented based on the MXNet platform and performed on NVIDIA Tesla K40 GPUs.

4.2 Results on Cityscapes Dataset

Ablation Study of Performance We evaluate the accuracy and inference time of the proposed HRFR method for stepby-step results on Cityscapes validation set with DeepLabv2 framework. The down-sampling factor is set to 1/2. We also provide the results of two DeepLab-v2 models respectively trained on original-sized input (i.e.the teacher network) and 1/2 down-sampled input for comparison. As concluded from Table 1, we can infer that: (1) SRR with/without bottleneck: comparing the run 1/2-HRFR(Neck) with 1/2-HRFR (Non-neck), SRR with bottle-neck saves about 91 ms (relative 11.7%), with only 0.1% reduction of accuracy which can be ignored. It verifies that SRR with bottle-neck structure can obtain faster inference speed with only very slight degradation of accuracy. (2) Auxiliary SRR and boundary region re-weighting: From the run 1/2-HRFR(Neck+Aux) vs. 1/2HRFR(Neck), and 1/2-HRFR(Neck+Bound) vs. 1/2-HRFR (Neck), we can see that applying the auxiliary SRR modules and the boundary region re-weighting can bring 0.93% and 1.04% improvement of accuracy respectively, while employing them together can gain 1.75% improvement (1/2-HRFR (Neck+Aux+Bound) vs. 1/2-HRFR(Neck)). In summary, combination of the auxiliary SRR and boundary region reweighting can considerably improve parsing accuracy. Additionally, the introductions of both do not affect the inference speed due to both are only adopted during training. (3) Overall HRFR verification: The proposed HRFR method (1/2HRFR(Neck+Aux+Bound) with 1/2 down-sampled input can finally achieve 3.0× speedup compared with the teacher network with original-sized input (1-DeepLab-v2), with only 0.72% degradation of accuracy. Furthermore, compared with 1/2-DeepLab-v2 with 1/2 down-sampled input, our HRFR can gain obvious accuracy improvements of 3.79% with the same down-sampled input at merely cost of 125.7 ms mainly for SRR. Qualitative results are shown in Figure 4, which represent that boundaries in the result of the proposed HRFR method are more precise than that of 1/2-DeepLab-v2 with down-sampled input, and are comparable with that of the teacher network with original-sized input (1-DeepLab-v2). In summary, the above ablation experimental results show the effectiveness and advantages of our proposed HRFR method. Experiments of Down-sampling Factors Table 2 demonstrates the accuracy and inference time of the proposed HRFR method by different down-sampling factors, which are also evaluated on Cityscapes validation set based on DeepLabv2 framework. DeepLab-v2 with original-sized input provides the higher bound of accuracy (74.33%), while the 1/2, 1/4 and 1/8 down-sampled inputs for DeepLab-v2 give the higher bounds of inference speed to the corresponding downsampling factors. For DeepLab-v2, as the down-sampling

1/2-HRFR(Neck+Aux+Bound)

Table 3: Comparison with other scene parsing methods on Cityscapes test set, both accuracy and inference time are provided. Down: down-sample ratio of input images.

factor decreases (from 1/2 to 1/8), the inference speed increases exponentially (with inference time from 652.9 ms to 73.2 ms) but the accuracy decreases quickly (from 69.82% to 45.02%). By contrast, for our proposed HRFR method, accuracy decreases much slower (from 73.6% to 59.73%) as the down-sampling factor decreases, outperforming that of DeepLab-v2 by the same down-sampling factors. Based on the inference time of DeepLab-v2 with original-sized input, our proposed method can gain higher speedup rate (from 3× to 9.4×) by smaller down-sampling factors (from 1/2 to 1/8). However, as the down-sampling factor gets smaller, more layers are needed in SRR module to enlarge the resolution of feature maps. Thus the additional time for SRR module increases, which means the down-sampling factor should not be too small. In practice, the down-sampling factor of our proposed method can be determined by considering the trade-off between parsing accuracy and inference speed. In addition, the proposed method is orthogonal to the most of the existing network accelerating methods and can gain further acceleration through combining these techniques.

Comparison with previous methods We test some popular scene parsing methods on NVIDIA Tesla K40 GPUs to compare with our HRFR method on Cityscapes test set. As shown in Table 3, some methods with original-sized input and large networks can obtain a higher accuracy, but has a low inference speed, while the methods with down-sampled input and small networks have a higher speed but sacrifice the parsing accuracy. The proposed HRFR framework can make a trade-off between accuracy and speed. We believe our framework can be applied in these popular methods to speed up the methods with original-sized input or to improve the accuracy of methods with down-sampled input.

4.3 Results on CamVid Dataset

We also evaluate the proposed method on CamVid dataset with DeepLab-v2 framework and present the results in Table 4. We can get the similar conclusion with results on Cityscapes dataset. The bottle-neck structure can provide faster inference speed while maintaining the parsing accuracy, saving the inference time of about 44 ms (relative 13.6%) per image(1/2-HRFR(Neck) vs. 1/2-HRFR(Nonneck)). In the ablation results of step-by-step comparison, both auxiliary SRR modules and boundary region reweighting can improve the parsing accuracy by 0.93% (1/2HRFR(Neck+Aux) vs. 1/2-HRFR (Neck)) and 1.08% (1/2HRFR(Neck+Bound) vs. 1/2-HRFR (Neck)) in terms of average category accuracy, further combining them together can achieve the accuracy improvement of 1.97% in total (1/2HRFR(Neck+Aux+Bound) vs. 1/2-HRFR(Neck)). The proposed HRFR method can finally achieve 2.9× speedup with 1/2 down-sampled input with only 0.86% degradation of accuracy compared with the teacher network with originalsized input (1/2-HRFR(Neck+Aux +Bound) vs. 1-DeepLabv2). More 6.8× speedup can be obtained with 1/4 downsampled input, but has a large degradation of 4.37% in averaged category accuracy (1/4-HRFR(Neck+Aux+Bound) vs. 1-DeepLab-v2).

5 Conclusion

In this paper, we propose the HRFR framework to speed up a given urban scene parsing network. In our framework, we exploit the SRR module to up-sample feature maps learned from the student network with down-sampled input, aiming at recovering the high resolution features from the teacher network with original-sized input. Therefore, the proposed framework can achieve the fast parsing with down-sampled input while maintaining high accuracy with original largeinput network. We also present the auxiliary intermediate supervision and the boundary region re-weighting to facilitate the optimization of the network. Experimental results show the effectiveness of our proposed method.