Channel Pruning via Automatic Structure Search

1 Introduction

The high demands in computing power and memory footprint of deep Convolutional Neural Networks (CNNs) prohibit their practical applications on edge computing devices such as smart phones or wearable gadgets. To address this problem, extensive studies have made on compressing CNNs. Prevalent techniques resort to quantization [Wang et al., 2019a], decomposition [Zhang et al., 2015], and pruning [Singh et al., 2019]. Among them, channel pruning has been recognized as one of the most effective tools for compressing CNNs [Luo et al., 2017; He et al., 2017; He et al., 2018b; He et al., 2018a; Liu et al., 2019a; Wang et al., 2019b].

∗Corresponding Author

Channel pruning targets at removing the entire channel in each layer, which is straightforward but challenging because removing channels in one layer might drastically change the input of the next layer. Most cutting-edge practice implements channel pruning by selecting channels (filters) based on rule-of-thumb designs. Existing works follow two mainstreams. The first pursues to identify the most important filter weights in a pre-trained model, which are then inherited by the pruned model as an initialization for the follow-up fine-tuning [Hu et al., 2016; Li et al., 2017; He et al., 2017; He et al., 2018a]. It usually performs layer-wise pruning and fine-tuning, or layer-wise weight reconstruction followed by a data-driven and/or iterative optimization to recover model accuracy, both of which however are time-cost. The second typically performs channel pruning based on handcrafted rules to regularize the retraining of a full model followed by pruning and fine-tuning [Liu et al., 2017; Huang and Wang, 2018; Lin et al., 2019]. It requires human experts to design and decide hyper-parameters like sparsity factor and pruning rate, which is not automatic and thus less practical in compressing various CNN models. Besides, rule-of-thumb designs usually produce the sub-optimal pruning [He et al., 2018b].

The motivation of our ABCPruner is two-fold. First, [Liu et al., 2019b] showed that the essence of channel pruning lies in finding optimal pruned structure, i.e., channel number in each layer, instead of selecting "important" channels. Second, [He et al., 2018b] proved the feasibility of applying automatic methods for controlling hyper-parameters to channel pruning, which requires less human interference.

However, exhaustively searching for the optimal pruning structure is of a great challenge. Given a CNN with L layers, the combinations of pruned structure could be ∏Lj=1 cj, where cj is channel number in the j-th layer. The combination overhead is extremely intensive for deep CNNs 1 , which is prohibitive for resource-limited scenarios.

In this paper, we introduce ABCPruner towards optimal channel pruning. To solve the above problem, two strategies are adopted in our ABCPruner. First, we propose to shrink the combinations by limiting the number of preserved channels to {0.1cj, 0.2cj, ..., αcj} where the value of α falls in {10%, 20%, ..., 100%}, which shows that there are 10α feasible solutions for each layer, and up to α% percentage of channels in each layer will be preserved. This operation is interpretable since channel pruning usually preserves a certain percentage of channels. By doing so, the combinations will be significantly reduced to (10α)L ≪ ∏Lj=1 cj, making the search more efficient. Second, we obtain the optimal pruned structure through the artificial bee colony (ABC) algorithm [Karaboga, 2005]. ABC is automatic-based rather than hand-crafted, which thus lessens the human interference in determining the channel number. As shown in Fig. 1, we first initialize a structure set, each element of which represents the preserved channel number in each layer. The filter weights of the full model are randomly selected and assigned to initialize each structure. We train it for a given number of epochs to measure its fitness, a.k.a, accuracy performance in this paper. Then, ABC is introduced to update the structure set. Similarly, filter assignment, training and fitness calculation for the updated structures are conducted. We continue the search for some cycles. Finally, the one with the best fitness is considered as the optimal pruned structure, and its trained weights are reserved as a warm-up for fine-tuning. Thus, our ABCPruner can effectively implement channel pruning via automatically searching for the optimal pruned structure.

1 For example, the combinations for VGGNet-16 are 2 104 , 2 448 for GoogLeNet, and 2 1182 for ResNet-152.

To our best knowledge, there exists only one prior work [Liu et al., 2019a] which considers pruned structure in an automatic manner. Our ABCPruner differs from it in two aspects: First, the pruning in [Liu et al., 2019a] is a twostage scheme where a large PruningNet is trained in advance to predict weights for the pruned model, and then the evolutionary algorithm is applied to generate a new pruned structure. While our ABCPruner is one-stage without particularly designed network, which thus simplifies the pruning process. Second, the combinations in ABCPruner are drastically reduced to (10α)L (α ∈ {10%, 20%, ..., 100%}), which provides more efficiency for the channel pruning.

2 RelatedWork

Network Pruning. Network pruning can be categorized into either weight pruning or channel pruning. Weight pruning removes individual neurons in the filters or connections across different layers [Han et al., 2015; Guo et al., 2016; Li et al., 2017; Aghasi et al., 2017; Zhu and Gupta, 2017; Frankle and Carbin, 2019]. After pruning, a significant portion of CNNs' weights are zero, and thus the memory cost can be reduced by arranging the model in a sparse format. However, such a pruning strategy usually leads to an irregular network structure and memory access, which requires customized hardware and software to support practical speedup.

In contrast, channel pruning is especially advantageous by removing the entire redundant filters directly, which can be well supported by general-purpose hardware and BLAS libraries. Existing methods pursue pruning by channel selection based on rule-of-thumb designs. To that effect, many works aim to inherit channels based on the importance estimation of filter weights, e.g., lp-norm [Li et al., 2017; He et al., 2018a] and sparsity of activation outputs [Hu et al., 2016]. [Luo et al., 2017; He et al., 2017] formulated channel pruning as an optimization problem, which selects the most representative filters to recover the accuracy of pruned network with minimal reconstruction error. However, it is still hard to implement in an end-to-end manner without iteratively pruning (optimization) and fine-tuning. Another group focuses on regularized-based pruning. For example, [Liu et al., 2017] imposed sparsity constraint on the scaling factor of batch normalization layer, while [Huang and Wang, 2018; Lin et al., 2019] proposed a sparsity-regularized mask for channel pruning, which is optimized through a data-driven selection or generative adversarial learning. However, these methods usually require another round of retraining and manual hyper-parameter analysis. Our work differs from traditional methods in that it is automatic and the corresponding fine-tuning is end-to-end, which has been demonstrated to be feasible by the recent work in [He et al., 2018b].

AutoML. Traditional pruning methods involve human expert in hyper-parameter analysis, which hinders their practical applications. Thus, automatic pruning has attracted increasing attention, which can be regarded as a specific AutoML task. Most prior AutoML based pruning methods are implemented in a bottom-up and layer-by-layer manner, which typically achieve the automations through Q-value [Lin et al., 2017], reinforcement learning [He et al., 2018b] or an automatic feedback loop [Yang et al., 2018]. Other lines include, but not limited to, constraint-aware optimization via an annealing strategy [Chen et al., 2018] and sparsityconstraint regularization via a joint training manner [Luo and Wu, 2018]. Different from most prior AutoML pruning, our work is inspired by the recent work in [Liu et al., 2019b] which reveals that the key of channel pruning lies in the pruned structure, i.e., channel number in each layer, instead of selecting "important" channels.

3 The Proposed ABCPruner

Given a CNN model N that contains L convolutional layers and its filters set W, we refer to C = (c1, c2, ..., cL) as the network structure of N , where cj is the channel number of the j-th layer. Channel pruning aims to remove a portion of filters in W while keeping a comparable or even better accuracy.

To that effect, traditional methods focus on selecting channels (filters) based on rule-of-thumb designs, which are usually sub-optimal [He et al., 2018b]. Instead, our ABCPruner differs from these methods by finding the optimal pruned structure, i.e., the channel number in each layer, and then solves the pruning in an automatic manner by integrating offthe-shelf ABC algorithm [Karaboga, 2005]. We illustrate our ABCPruner in Fig. 1.

3.1 Optimal Pruned Structure

Our channel pruning is inspired by the recent study in [Liu et al., 2019b] which reveals that the key step in channel pruning lies in finding the optimal pruned structure, i.e., channel number in each layer, rather than selecting "important" channels.

For any pruned model N ′ , we denote its structure as C ′ = (c′1 , c′2 , ..., c′L), where c′j ≤ cj is the channel number of the pruned model in the j-th layer. Given the training set Ttrain and test set Ttest, we aim to find the optimal combination of C ′ , such that the pruned model N ′ trained/fine-tuned on Ttrain obtains the best accuracy. To that effect, we formulate our channel pruning problem as

whereW′ is the weights of pruned model trained/fine-tuned on Ttrain, and acc(·) denotes the accuracy on Ttest for N ′ with structure C ′ . As seen, our implementation is one-stage where the pruned weights are updated directly on the pruned model, which differs from [Liu et al., 2019a] where an extra large PruningNet has to be trained to predict weights for N ′ , resulting in high complexity in the channel pruning.

However, the optimization of Eq. 1 is almost intractable. Detailedly, the potential combinations of network structure C ′ are extremely large. Exhaustively searching for Eq. 1 is infeasible. To solve this problem, we further propose to shrink the combinations in Sec. 3.2.

3.2 Combination Shrinkage

Given a network N, the combination of pruned structure could be ∏Li=1 ci, which are extremely large and computationally prohibitive for the resource-limited devices. Hence, we further propose to constrain Eq. 1 as:

where α = 10%, 20%, ..., 100% is a pre-given constant and its influence is analyzed in Sec. 4.5. It denotes that for the i-th layer, at most α percentage of the channels in the pre-trained network N are preserved in N ′ , i.e., c′i ≤ αci, and the value of c′i is limited in {0.1ci, 0.2ci, ..., αici} 2 . Note that α is shared across all layers, which greatly relieves the burden of human interference to involve in the parameter analysis.

2 When αci < 1, we set c′i = 1 to preserve one channel.

Our motivations of introducing α are two-fold: First, typically, a certain percentage of filters in each layer are preserved in the channel pruning, and α can play as the upbound of preserved filters. Second, the introduced α significantly decreases the combinations to (10α)L (α ≤ 1), which makes solving Eq. 1 more feasible and efficient.

Though the introduction of α significantly reduces the combinations of the pruned structure, it still suffers large computational cost to enumerate the remaining (10α)L combinations for deep neural networks. One possible solution is to rely on empirical settings, which however requires manual interference. To solve the problem, we further automate the structure search in Sec. 3.3.

3.3 Automatic Structure Search

Instead of resorting to manual structure design or enumeration of all potential structure (channel number in each layer), we propose to automatically search the optimal structure.

In particular, we initialize a set of n pruned structures {C ′j }nj=1 with the i-th element c′ji of C ′j randomly sampled from {0.1ci, 0.2ci, ..., αci}. Accordingly, we obtain a set of pruned model {N ′ j }nj=1 and a set of pruned weights {W′j }nj=1 . Each pruned structure C ′j represents a potential solution to the optimization problem of Eq. 2. Our goal is to progressively modify the structure set and finally pick up the best pruned structure. To that effect, we show that our optimal structure search for channel pruning can be optimized automatically by integrating off-the-shelf ABC algorithm [Karaboga, 2005]. We first outline our algorithm in Alg. 1. More details are elaborated below, which mainly contains three steps.

Employed Bee (Line 3 - Line 13). The employed bee generates a new structure candidate G′j for each pruned structure C ′j . The i-th element of G′j is defined as below:

where r is a random number within [-1,+1] and g ̸= j denotes the g-th pruned structure. ⌈·⌋ returns the value closest to the input in {0.1ci, 0.2ci, ..., αci}.

Then, employed bee will decide if the generated candidate G′j should replace C ′j according to their fitness, which is defined in our implementation as below:

If the fitness of G′j is better than C ′j , C ′j will be updated as C ′j = G′j . Otherwise, C ′j will keep unchanged.

Onlooker Bee (Line 14 - Line 28). The onlooker bee further updates {C ′j }nj=1 following the rule of Eq. 3. Differently, a pruned structure C ′j is chosen with a probability related to its fitness, defined as:

Algorithm 1: ABCPruner

Input: Cycles: T , Upbound: α, Number of pruned structures: n, Counter: {ti}ni=1 = 0, Max times: M. Output: Optimal pruned structure (C ′ )∗ . 1 Initialize the pruned structure set {C ′j }nj=1 ; 2 for h = 1 → T do 3 for j = 1 → n do 4 generate candidate G′j via Eq. 3; 5 calculate the fitness of C ′j and G′j via Eq. 4; 6 if fitC′j < fitG′j then 7 C ′j = G′j ; 8 fitC′j = fitG′j ; 9 tj = 0; 10 else 11 tj = tj + 1; 12 end 13 end 14 for j = 1 → n do 15 calculate the probability of Pj via Eq. 5; 16 generate a random number ϵj ∈ [0, 1]; 17 if ϵj <= Pj then 18 generate candidate G′j via Eq. 3; 19 calculate the fitness of G′j via Eq. 4; 20 if fitC′j < fitG′j then 21 C ′j = G′j ; 22 fitC′j = fitG′j ; 23 tj = 0; 24 else 25 tj = tj + 1; 26 end 27 end 28 end 29 for j = 1 → n do 30 if tj >Mthen 31 re-initialize C ′j ; 32 end 33 end 34 end 35 (C ′ )∗ = argmax C′j acc(N ′ j (C ′j , W′j ; Ttrain); Ltest ).

Therefore, the better fitness of C ′j is, the higher probability of C ′j would be selected, which then produces a new and better pruned structure. By this, ABCPruner will automatically result in the optimal pruned structure progressively.

Scout Bee (Line 29 - Line 33). If a pruned structure C ′j has not been updated more than M times, the scout bee will re-initialize it to further produce a new pruned structure.

However, to calculate the fitness in Eq. 4, it has to train N ′ j on Ttrain which is time-consuming and infeasible, especially when Ttrain is large-scale. To solve this problem, as shown in Fig. 1, given a potential combination of C ′ , we first randomly pick up c′j filters from the pre-trained model, which then serve as the initialization for the j-th layer of the pruned model N ′ . Then, we train N ′ for some small epochs to obtain its fitness. Lastly, we give more epochs to fine-tune the pruned model with the best structure (C ′ )∗ , a.k.a., best fitness. More details about the fine-tuning are discussed in Sec. 4.1.

4 Experiments

We conduct compression for representative networks, including VGGNet, GoogLeNet and ResNet-56/110 on CIFAR-10 [Krizhevsky et al., 2009], and ResNet-18/34/50/101/152 on ILSVRC-2012 [Russakovsky et al., 2015].

4.1 Implementation Details

Training Strategy. We use the StochasticGradient Descent algorithm (SGD) for fine-tuning with momentum 0.9 and the batch size is set to 256. On CIFAR-10, the weight decay is set to 5e-3 and we fine-tune the network for 150 epochs with a learning rate of 0.01, which is then divided by 10 every 50 training epochs. On ILSVRC-2012, the weight decay is set to 1e-4 and 90 epochs are given for fine-tuning. The learning rate is set as 0.1, and divided by 10 every 30 epochs.

Performance Metric. Channel number, FLOPs (floatingpoint operations), and parameters are used to measure the network compression. Besides, for CIFAR-10, top-1 accuracy of pruned models are provided. For ILSVRC-2012, both top-1 and top-5 accuracies are reported.

For each structure, we train the pruned model N ′ for two epochs to obtain its fitness. We empirically set T =2, n=3, and M=2 in the Alg. 1.

4.2 Results on CIFAR-10

We conduct our experiments on CIFAR-10 with three classic deep networks including VGGNet, GoogLeNet and ResNets. The experiment results are reported in Tab. 1.

VGGNet. For VGGNet, the 16-layer (13-Conv + 3FC) model is adopted for compression on CIFAR-10. We remove 61.20% channels, 73.68% FLOPs and 88.68% parameters while still keeping the accuracy at 93.08%, which is even slightly better than the baseline model. This greatly facilitates VGGNet model, a popular backbone for object detection and semantic segmentation, to be deployed on mobile devices.

GoogLeNet. For GoogLeNet, as can be seen from Tab. 1, we remove 22.19% inefficient channels with only 0.21% performance drop of accuracy. Besides, nearly 66.56% FLOPs computation is saved and 60.14% parameters are reduced. Therefore, ABCPruner can be well applied even on the compact-designed GoogLeNet.

ResNets. For ResNets, we choose two different depths, including ResNet-56 and ResNet-110. As seen from Tab. 1, the pruning rates with regard to channels, FLOPs and parameters increase from ResNet-56 to ResNet-110. Detailedly, the channels reduction raises from 27.07% to 33.28%; the FLOPs reduction raises from 54.13% to 65.05%; and the parameters reduction raises from 54.20% to 67.41%. To explain, ResNet110 is deeper and more over-parameterized. ABCPruner can automatically find out the redundancies and remove them.

Besides, it retains comparable accuracies against the baseline model, which verifies the efficacy of ABCPruner in compressing the residual-designed networks.

4.3 Results on ILSVRC-2012

We further perform our method on the large-scale ILSVRC-2012 for ResNets with different depths, including 18/34/50/101/152. We present two different pruning rates for each network and the results are reported in Tab. 2.

From Tab. 2, we have two observations. First observation is that the performance drops on ILSVRC-2012 are more than these on CIFAR-10. The explanations are two-fold: On one hand, the ResNet itself is a compact-designed network, there might exist less redundant parameters. On the other hand, ILSVRC-2012 is a large-scale dataset and contains 1,000 categories, which is much complex than the small-scale CIFAR10 with only 10 categories. Second observation comes that ABCPruner obtains higher pruning rates and less accuracy drops as the depth of network increases. To explain, compared with shallow ResNets, e.g., ResNet-18/34, the deeper ResNets, e.g., ResNet-50/191/152 contain relatively more redundancies, which means more pointless parameters are automatically found and removed by ABCPruner.

In-depth Analysis. Combining Tab. 1 and Tab. 2, we can see that ABCPruner can well compress popular CNNs while keeping a better or at least comparable accuracy performance against the full model. Its success lies in the automatically optimal pruned structure search. For deeper analysis, we display the layer-wise pruning results by ABCPruner-80% for VGGNet-16 in Fig. 2. ABCPruner-80% denotes that at most 80% percentage of the channels are preserved in each layer. As can be seen, over 20% channels are removed for most layers and the pruning rate differs across different layers. Hence, the ABCPruner can automatically obtain optimal pruned structure which then feeds back good performance.

4.4 Comparison with Other Methods

In Tab. 3, we compare ABCPruner with traditional methods including [Luo et al., 2017; He et al., 2017; Huang and Wang, 2018; Lin et al., 2019], and automatic method [Liu et al.,

2019a] 3 .

Effectiveness. The results in Tab. 3 show that ABCPruner obtains better FLOPs reduction and accuracy performance. Compared with importance-based pruning [Luo et al., 2017; He et al., 2017] and regularized-based pruning [Huang and Wang, 2018; Lin et al., 2019], ABCPruner also advances in its end-to-end fine-tuning and automatic structure search. It effectively verifies the finding in [Liu et al., 2019b] that optimal pruned structure is more important in channel pruning, rather than selecting "important" channels. In comparison with the automatic-based MetaPruning [Liu et al., 2019a], the superiority of ABCPruner can be attributed to the onestage design where the fitness of pruned structure is directly measured in the target network without any additionally introduced network as in [Liu et al., 2019a]. Hence, ABCPruner is more advantageous in finding the optimal pruned structure.

Efficiency. We also display the overall training epochs for all methods in Tab. 3. As seen, ABCPruner requires less training epochs of 102 compared with others. To analyze, 12 epochs are used to search for the optimal pruned structure and 90 epochs are adopted for fine-tuning the optimal pruned network. We note that the importance-based pruning methods [Luo et al., 2017; He et al., 2017] are extremely inefficient since they require layer-wise pruning or optimization (196 epochs). Besides, 48 epochs [Luo et al., 2017] and 10 epochs [He et al., 2017] are used to fine-tune the network. [Huang and Wang, 2018] consume similar training time with ABCPruner, but suffers more accuracy drops and less FLOPs reduction. [Lin et al., 2019] cost 90 epochs for retraining the network and additional 60 epochs are required for fine-tuning. Besides, ABCPruner also shows its better efficiency against the automatic-based counterpart [Liu et al., 2019a] which is a two-stage method where 32 epochs are first adopted to train the large PruningNet and 128 epochs are used for fine-tuning. Hence, ABCPruner is more efficient compared to the SOTAs.

3 We have carefully fixed the codes provided by the authors and re-run the experiments using our baseline of ResNet-50.

Figure 3: The influence of α for ResNet-56 on CIFAR-10. Generally, large α results in better accuracy but less pruning rate.

4.5 The Influence of α

In this section, we take ResNet-56 on CIFAR-10 as an example and analyze the influence of the introduced constant α, which represents the upbound of preserved channel percentage in each layer. It is intuitive that larger α leads less reductions of channel, parameters and FLOPs, but better accuracy performance. The experimental results in Fig. 3 demonstrate this assumption. To balance the accuracy performance and model complexity reduction, in this paper, we set α = 70% as shown in Tab. 1.

5 Conclusion

In this paper, we introduce a novel channel pruning method, termed as ABCPruner. ABCPruner proposes to find the optimal pruned structure, i.e., channel number in each layer via an automatic manner. We first propose to shrink the combinations of pruned structure, leading to efficient search of optimal pruned structure. Then, the artificial bee colony is integrated to achieve the optimal pruned structure search in an automatic manner. Extensive experiments on popular CNNs have demonstrated the efficacy of ABCPruner over traditional channel pruning methods and automatic-based counterpart.

Acknowledgements

This work is supported by the Nature Science Foundation of China (No.U1705262, No.61772443, No.61572410, No.61802324 and No.61702136), National Key R&D Program (No.2017YFC0113000, and No.2016YFB1001503), and Nature Science Foundation of Fujian Province, China (No. 2017J01125 and No. 2018J01106).