COP: Customized Deep Model Compression via Regularized Correlation-Based Filter-Level Pruning

1 Introduction

The growing demands of deploying deep models to resourceconstrained devices such as mobile phones and FPGA have posed great challenges for us. Network pruning has become one of the most popular methods to compress the model without much loss in performance, and methods of network pruning could be divided into two categories: weight-level pruning [Han et al., 2015; Guo et al., 2016; Dong et al., 2017]

∗Corresponding author

and filter-level pruning [Li et al., 2016; Liu et al., 2017; He et al., 2018b].

The weight-level pruning methods try to find out unimportant weights and set them to zeros. In other words, weightlevel pruning can compress deep models because it induces sparsity in filters. However, it contributes little to accelerating them unless specialized libraries (such as cuSPARSE) are used. Unfortunately, the support for these libraries on mobile devices, especially on FPGA is very limited. Thus, filter-level pruning methods are proposed to address this problem. They locate and remove unimportant filters in convolutional layers or unimportant nodes in fully connected layers. In this way, both space and inference time cost can be saved a lot.

Undoubtedly, the key factor influencing a filter-level pruning method's performance is how it defines the importance of filters. We find the current state-of-the-art methods have at least two of the following problems.

HighRedundancy. Deepmodels with rich parameters may suffer from high redundancy[Denil et al., 2013] among filters, i.e., different filters may have similar expressibility. Some methods (e.g., [Li et al., 2016; Liu et al., 2017] only evaluate the importance based on the filter itself and neglect the correlations between filters.

Lack of Global Vision. Under the approaches definition of some previous methods (e.g., [Li et al., 2016; He et al., 2018b; He et al., 2018a; Luo et al., 2017]), the filters can be only compared within the same layer. Thus, it is difficult to specify how many filters should be pruned for each layer.

Sub-optimal Pruning Strategy. All the existing methods are prone to generate sub-optimal pruning strategies because they neglect that pruning the same number of parameters may not means reducing the same amount of computation cost. Take Figure 1 as an example. Pruning the same number of parameters in different layers of VGG16, we may reduce more computation cost by pruning the lower layers. Previous methods are insensitive to such differences.

To address the above limitations, we develop a correlationbased pruning algorithm (COP) to detect the redundancy among filters efficiently. We normalize the filter importance of different layers to the same scale to enable global comparison. In addition, we add parameter-quantity and computation-cost regularization terms to enable fine-grained filter pruning. The users can customize the compression simply through weight allocation according to their preference, i.e., whether they want to reduce more parameters or inference complexity.

It is worthwhile to highlight the advantages of COP:

• We propose a novel filter-level pruning algorithm for deep model compression, which can a) reduce the redundancy among filters significantly, b) learn proper pruning ratios for different layers automatically, and c) enable fine-grained pruning given users' preference.
• Extensive experiments on public datasets have demonstrated COP's advantages over the current state-of-theart methods.
• We also evaluate COP on specially designed compact neural network, MobileNets. COP still produces reasonably good compression ratio with little loss on the inference performance.

2 RelatedWorks

Network pruning has become one of the most popular methods to compress and accelerate deep CNNs. Network pruning includes two categories: weight-level pruning and filterlevel pruning. Weight-level pruning sets unimportant weights in filters to zero, which induces sparsity into filters. However, according to [Luo et al., 2017; Liu et al., 2017; Molchanov et al., 2016], it is difficult for weight-level pruning to accelerate deep CNNs without specialized libraries or hardware. Filter-level pruning solves this problem, it prunes unimportant filters in convolutional layers, which compresses and accelerates the deep CNNs simultaneously.

In recent years, many filter-level pruning methods have been proposed. [Li et al., 2016] evaluates the importance of filters through the sum of its absolute weights, i.e., l1 norm, and decide pruned ratio for each layer manually. [Luo et al., 2017] evaluates the importance of filters through reconstruction loss and use the same pruned ratio for all layers, i.e., large loss induced by pruning means high importance of the pruned filter. [He et al., 2017] evaluates the importance of filters based on LASSO regression. All the methods mentioned above only evaluate the local importance for filters, i.e., the importance could only be compared within the same layer, so they have to specify the pruned ratio for each layer manually. [Molchanov et al., 2016] evaluates the importance of filters through Taylor expansion, whose value could be compared globally. [Liu et al., 2017] also evaluates the global importance of filters based on the scale of batch normalization (BN) layer. However, as we say in Section 1, all these methods neglect the inequality between reducing parameters and reducing computational cost, the users cannot customize the pruned model for different purposes(smaller or faster).

PFA is a filter-level pruning method, which decides the pruned ratio for each layer by performing PCA on feature maps and evaluates the importance of filters by doing correlation analysis on feature maps. PFA is similar to our method because both PFA and COP use correlation to evaluating the importance of filters, however, there are three main differences between PFA and COP: 1) PFA is a data-driven method. It performs the correlation analysis on the activated feature maps, while COP performs correlation analysis on the filter weights. PFA needs all the training data when evaluating the importance of filters, which consumes more computing resources; however, COP only uses the trained model parameters when evaluating filters' importance. 2) PFA is a twostage pruning method, i.e., it needs to perform PCA first and then correlation analysis when pruning filters; COP only performs correlation analysis. 3) As we mentioned in Section 1, PFA is a sub-optimal pruning method; COP uses two regularization terms to generate fine-grained pruning plans, so users could customize their pruned model for different purposes.

3 Algorithm

3.1 Symbols and Annotations

Assuming there are L layers in a deep model, let P l be the lth layer in the deep model, whose input is Xl , output is Y l and weight is W l . We will omit superscript l for simplicity in the cases of no confusion.

When P l is a fully connected layer, W l is of shape M l × N l , where M l is the number of input nodes and N l is the number of output nodes; Xl is of shape M l × 1; Y l is of shape N l × 1. Let Xlm be the mth node of Xl , Y ln be the nth node of Y l . Let ⃗ωm be the mth row of W l , namely ⃗ωm = [Wm,1, · · · ,Wm,n].

When P l is a convolutional layer, W l is of shape Kl × Kl ×M l ×N l , where Kl is the kernel width (assumed to be symmetric), M l is the number of input channel, N l is the number of output channel; Xl is of shape I l × I l × M l ; Y l is of shape Ol × Ol × N l ; I l , Ol are the size of input and output feature maps respectively. Xlm is the mth input feature map in the lth layer. Let ⃗ωi,j,m be the vector [Wi,j,m,1,Wi,j,m,2, · · · ,Wi,j,m,n].

µ⃗ωm is the mean of ⃗ωm. σ⃗ωm is the standard deviation of ⃗ωm. Let E[·] be the expectation function.

Note that, in a convolutional layer, pruning feature maps is equivalent to pruning filters, and the importance of feature maps could also be seen as the importance of filters. Therefore, for simplicity, we will not differentiate pruning filters from pruning feature maps.

3.2 Overview

We propose three techniques to avoid the three limitations mentioned in Section 1. 1) Evaluate the correlation-based importance of filters to remove redundant filters. 2) Normalize the correlations from all layers to the same scale for global comparison. 3) Adding parameter-quantity and computational-cost regularization terms to the importance evaluation. The users could choose freely whether they want to reduce more parameters or more computational cost by adjusting the weights of two regularization terms.

3.3 Method

Normalized Correlation-Based Importance

We will first take fully connected layers as examples and introduce the calculation of correlation-based importance in detail, then generalize it to convolutional layers.

Figure 2 is an illustration of our idea. In a fully connected layer, Yn is calculated as Yn = ∑m Wm,nXm, we omit the bias term for simplicity. If ∃m1,m2 ∈ [1,M], α ∈ R such that ⃗ωm2 ≈ α⃗ωm1, then Yn could also be computed as Equation 1 for all n ∈ [1,N]. ϵ is the loss induced because ⃗ωm2 and ⃗ωm1 are not strictly linearly related. So, we could merge Xm1 and Xm2 through pruning and fine-tuning.

⃗ωm2 ≈ α⃗ωm1 implies that ⃗ωm1 and ⃗ωm2 are activated in similar patterns. On the other hand, it also implies Xm1 and Xm2 express similar information. We propose to measure the redundancy between ⃗ωm1 and ⃗ωm2 by using Pearson correlation, which also indicates the similarity between Xm1 and Xm2 (Equation 2).

The redundancy evaluation mentioned above could be generalized to convolutional layer with slight modifications because a fully connected layer could be seen as a convolutional layer whose weights are of shape 1×1×M ×N. For a convolutional layer whose filters are of shape K ×K ×M ×N, one filter can be seen as a group of K × K independent nodes. We regroup the filter tensors into K × K sets of nodes and calculate the pair-wise node-correlations accordingly. Finally, we calculate the filter-correlation by averaging the node-correlations on the respective K ×K nodes of the given filter (as shown in Equation 3).

For two correlated filters, we have to remove one of them. It is hard to tell which is more important if we only look at their correlation. Instead, if one filter is highly correlated with many other filters, we believe it can be removed because other filters can take over its job. Thus, among the correlation coefficients between the given filter and the others, we select the k highest ones and average them to get the importance of the filter.

However, the importance defined is still a local one because the correlations are calculated among filters within the same layer. Generally, the correlation distributions of different layers are quite different due to their different functions and scopes. Thus, we need to normalize the importance to enable cross-layer comparison. There are various methods to normalize the importance. [Molchanov et al., 2016] proposed a simple layer-wise l2-normalization which re-scales the importance distributions of different layers. Similarly, one can also normalize different importance distributions with l1normalization. However, we argue that they are not suitable for correlation-based importance because they will cause improper scaling of the importance. For instance, for layers with many filters, the importance of each filter tends to be very small because the denominator is large. Consequently, the procedure will always try to prune the layers with more filters. We propose to use max-normalization. Specifically, we normalize the correlation distribution of each layer by dividing them by the maximal importance to align the correlation distributions to [0, 1]. Formally, we define the normalized importance as Equation 4.

Regularizers

As we discussed above, reducing the same number of parameters from different positions in a neural network may reduce quite a different amount of computation cost. To make our approach aware of such differences, we add two regularization terms to enable fine-grained pruning plan generation. As a result, the users can customize the pruning plan through simple weight-allocation. Specifically, we add parameter-quantity and computational-cost regularization terms when evaluating the importance of filters. Pruning a filter of the lth layer influences the parameter quantity and computational cost of the lth layer and the (l+1)th layer, so the parameter quantity (Sl ) and computational cost (C l ) related with the lth layer are defined in Equation 5. Further, the regularizers are defined in Equation 6. Note that filters in the same layer share the same regularizers, and regularizers for fully connected layers could be computed in a similar way. Adding these two terms to the importance defined above will empower the procedure with the sensitivity of parameter-quantity and computation-cost. Thus, it can generate fine-grained pruning plans.

Finally, we define the regularized importance of Xlm as Equation 7.

The COP algorithm is a three-step pipeline : 1) Calculate the regularized importance for all filters and nodes; 2) Specify a global pruning ratio and remove the least important ones according to the ratio; 3) Fine-tune the pruned model with original data. Note that we only prune the network and finetune it once. In contrast, many existing methods follow an alternative-prune-and-fine-tune manner.

Pruning for Depth-Separable Convolutional Layer. A depth-separable convolutional layer contains a depth-wise convolutional layer and a point-wise convolutional layer. A point-wise convolutional layer is actually a convolutional layer whose filters are of shape 1 × 1 ×M ×N, so we prune the point-wise layers as normal convolutional layers. As for depth-wise convolutional layer, the number of input and output feature maps are always the same; the input feature maps of a point-wise layer are also the output feature maps of a depth-wise layer [Howard et al., 2017]. Therefore, pruning the point-wise layer will immediately prune the depth-wise layer; we do not need to prune the depth-wise layer again.

Pruning for Residual Block. A residual block contains more than one convolutional layer, and the number of input and output feature maps must be equal for a residual block unless there is an extra linear projection in the block [He et al., 2016]. Thus for all residual blocks which do not contain a linear projection, we compute the mean importance of input and output feature maps as overall importance and prune them simultaneously. Please refer to Figure 3 for details.

4 Experimental Settings

4.1 Datasets and Architecture

We perform the experiments on two well-known public datasets, CIFAR [Krizhevsky and Hinton, 2009] and ImageNet [Russakovsky et al., 2015]. Specifically, we use both CIFAR10 and CIFAR100 in the CIFAR collection.

We test the compression performance of different algorithms on several famous large CNN models, including VGGNet [Simonyan and Zisserman, 2014], ResNet [He et al., 2016], and MobileNet [Howard et al., 2017].

4.2 Evaluation Protocol

Following the previous works [Jiang et al., 2018; Liu et al., 2017; Yu et al., 2018], we record the parameter-reduction ratio (Prr) and FLOPs-reduction ratio (Frr) of each algorithm compared with the original model. A higher parameterreduction ratio means a smaller model size, and a higher FLOPs-reduction ratio means a faster inference.

4.3 Compared Algorithms

We select several recent state-of-the-art methods which are all filter-level pruning algorithms:

• NS [Liu et al., 2017] evaluates the importance of filters according to the BN's scaling factors 1 .
• NRE [Jiang et al., 2018] proposes to prune the model by minimizing the reconstruction error of nonlinear units.
• PFA [Suau et al., 2018] decides the pruned ratio for each layer by performing PCA on feature maps. They evaluate the importance of filters by doing correlation analysis on feature maps.
• SFP [He et al., 2018a] evaluates the importance of filters with l2-norm and prunes filters in a soft manner, i.e., the pruned filters may be retrieved after fine-tuning 2 .
• PFGM [He et al., 2018b] evaluates the importance of filters by analyzing the geometric correlation among the filters within the same layer.

We also compare our pruned MobileNets with thinner MobileNets[Howard et al., 2017]. MobileNets are compact deep neural networks which reduce the models' size and computational cost by replacing convolutional layers with depthseparable convolutional layers. [Howard et al., 2017] also proposes thinner MobileNets to balance accuracy and resources consumption.

4.4 Configuration

We set the batch-size of the stochastic gradient descent algorithm (SGD) to be 128, the momentum coefficient to be 0.9 for all models on CIFAR. For the VGG16 model on CIFAR, we use a weight decay rate of 0.0015, and we use a weight decay rate of 0.0006 and 0.0002 for the ResNet32 and MobileNet respectively on CIFAR.

1 https://github.com/Eric-mingjie/network-slimming

2 https://github.com/he-y/soft-filter-pruning

When training on ImageNet dataset, we use the same architecture and training policy with the original paper [Simonyan and Zisserman, 2014] for VGG11. We use the same architecture with the original paper [Howard et al., 2017] for MobileNets but the same training policy with VGG11 because it takes too much time to train MobileNets with the official training policy.

We use k = 3 for T opK when pruning the model.

5 Results and Analysis

5.1 Compression Efficiency

For the methods which do not report the results on given datasets and do not release their code either, we pass them in the experiments. The performance of different algorithms on the three datasets is given in Table 1, we experiment with several pruned ratio on COP and choose the maximal pruned ratio under the constraint of acceptable accuracy loss. On CIFAR, all the algorithms prune the VGG16 and ResNet32 models to get their compressed versions. On ImageNet, all the algorithms prune the VGG11 and ResNet18 models to get their compressed versions. As is shown in Table 1, we can draw some conclusions as follows:

1. COP achieves a higher compression ratio and speedup ratio with similar accuracy to other algorithms. Especially, COP gets the highest numbers on all three metrics on CIFAR100.
2. COP can prune much more parameters than most of the compared algorithms (e.g., NS, SFP, and PFGM) on different datasets and architecture because COP can detect the redundancy among filters better.
3. For the algorithms which can prune the similar number of parameters to COP (e.g., NRE and PFGM), COP gets more speed-up, which is the contribution of the computation-cost and parameter-size regularizers.

Though the MobileNet is a specially designed compact network, we can still prune it without much performance loss. The results are given in Table 2. MobileNet has its own way to compress itself [Howard et al., 2017]. We can see that COP can get better numbers in all the metrics.

5.2 Ablation Study

Redundancy Detector. We have tried dot-product similarity, cosine similarity, and Pearson correlation as the redundancy detectors, and also tried l1-, l2-, and maxnormalization methods as discussed above. The results are shown in Table 3. We can see that Pearson correlation and max-normalization are the best configurations because the Pearson correlation detect the redundancy better and the max-normalization is insensitive to the layer-width.

Regularizer Efficiency. We try different weights for two regularization terms, and the results are shown in Table 4. We can see that the regularizer do have significant impacts on the compression. Larger γ reduces more parameters while larger β reduces more computation cost, which is exactly consistent with our expectation.

5.3 Case Study

COP focuses on observing redundant filters, so our pruned models contain less redundancy than others. With less redundancy, feature maps in the model mainly express diversified information. Figure 4 is an example of visualized feature maps of "conv1 1" in VGG16. Every feature map pruned by COP expresses similar information to those preserved. We compare COP with SFP and NS. SFP and NS can prune many unimportant feature maps, but there are still some similar feature maps being preserved.

6 Conclusion

We propose a novel filter-level pruning method, COP, to address the limitations of previous works in the following aspects: removing redundancy among filters; enabling crosslayer importance comparison; generating fine-grained pruning strategies (sensitive to desired computation cost and model size). Extensive experiments have shown our significant advantages over other state-of-the-art methods. Moreover, COP can also prune specially designed compact networks such as MobileNet and get larger compression ratio and speedup ratio than its own compression method.