Unifying and Merging Well-trained Deep Neural Networks for Inference Stage

1 Introduction

The research on deep neural networks has gotten a rapid progress and achievement recently. It is successfully applied in a wide range of artificial intelligence (AI) applications, including computer vision, speech processing, natural language processing, bioinformatics, etc. To handle various tasks, we usually design different network models and train them with particular datasets separately, so they can behave well for specific purposes. However, in practical AI applications, it is common to handle multiple tasks simultaneously, leading to a high demand for the computation resource in both training and inference stages. Therefore, how to effectively integrate multiple network models in a system is a crucial problem towards successful AI applications.

This paper tackles the problems of merging multiple welltrained (known-weights) feed-forward networks and unifying them into a single but compact one. The original networks, whose architectures may not be identical, can be either single or multiple source input. After unification, the merged network should be capable of handling the original tasks but is more condensed than the whole original models. Our approach (NeuralMerger) contains two phases:

Alignment and encoding phase: First, we align the architectures of neural network models and encode the weights such that they are shared among the networks. The purpose is to unify and co-use the weights of different neural networks.

Fine-tuning phase: Second, we fine-tune the merged model with partial or all training data (calibration data). A method following the concept of distilling dark knowledge of neural networks in [Hinton et al., 2014] is employed in this phase.

Neural network models may have very different topologies. Currently, this study focuses on merging feed-forward networks, while merging networks with loops remains a future work. A modern feed-forward network consists of several kinds of layers, including convolution, pooling, and fullconnection, which is generally referred to as a convolutional network (CNN). When merging two CNNs, our approach aligns the same-type layers (convolution; full-connection) into pairs. The layers in a pair are merged into a single layer that shares a common weight codebook through the proposed encoding scheme. The codebooks in the merged single model can be further trained via back-propagation algorithm; it thus can be fine-tuned to seek for performance improvement.

1.1 Motivation of Our Study

Merging existing neural networks has a great potential for real-world applications. To tackle multiple recognition tasks in a single system based on either unique or various signal sources, a typical approach is to design a new model and train the model on the union datasets of these tasks, eg., [Kaiser et al., 2017; Aytar et al., 2017]. Such "learn-them-all" approaches train a single complex model to handle multiple tasks simultaneously. However, two issues may arise. First, it is hard to choose a suitable neural-net architecture for learning all the tasks well in advance; hence, a trial-and-error process is required to conduct suitable architectures. Second, learning from a random initial with large training data of different types could be demanding. To tackle these issues, the networks with bridging layers among the original models are conducted as a joint model for multi-task learning in [Ruder, 2017]. However, increased complexity and size of the joint model hinder their availability on resource-limited or edge devices in the inference stage.

As many models trained for various tasks are available now, a practical way to integrate different functionalities in a system would be leveraging on these individual-task models. In this paper, we introduce an approach that merges different neural networks by removing the co-redundancy among their filters or weights. The proposed NeuralMerger can take advantage of existing well-trained models. Our approach merges them via finding and sharing the representative codes of weights; the shared codes can still be refined by learning. To our knowledge, this is the first study on merging knownweights neural-nets into a more compact model. Because our approach compresses the networks for weight sharing and redundancy removal, it is useful for the deep-learning embedded system or edge computing in the inference stage.

1.2 Overview of Our Approach

When merging two different CNN models CA and CB, the output is a CNN model consisting of jointly encoded convolution (E-Conv) and fully-connected (E-FC) layers. An overview of our approach is illustrated in Fig. 1 and an example of merging three models via our approach is given in Fig. 2.

Contributions of this paper are summarized as follows:

(1) Given well-trained CNN models, the introduced NeuralMerger can merge them for multi-tasks even the models are different. The merging process preserves the general architectures of the well-trained networks and removes their redundancy. It avoids the cumbersome-design and trial-anderror process raised by the learn-them-all approaches.

(2) The proposed method produces a more compact model to handle the original tasks simultaneously. The compact model consumes less computational time and storage than the compound model of the original networks. It has a great potential to be fitted in low-end systems.

2 RelatedWork

To simultaneously achieve various tasks via a single neuralnet model, a typical way is to increase the output nodes (for multi-tasks) of a pre-chosen neural-net structure and train it from an initialization. In [Kaiser et al., 2017], MultiModel architecture is introduced to allow input data to be images, sound waves, and text of different dimensions, and then converts them into a unified representation. In [Aytar et al., 2017], a deep CNN leverages massive synchronized data (sound and sentences paired with images) to learn an aligned representation. Nevertheless, as mentioned earlier, applying the learn-them-all approaches has to pay cumbersome training effort and intensive inference computation.

Compressing a neural-net is an active direction to deploy the compact model on resource-limited embedded systems. To reduce the representation, binary weights and bit-wise operations are used in [Hubara et al., 2016] and [Rastegari et al., 2016]. Han et al. [Han et al., 2016] introduce a three-stage pipeline: pruning redundant network connections, quantizing weights with a codebook, and Huffman encoding weights and index, to reduce the storage required by CNN. Quantized CNN (Q-CNN) [Wu et al., 2016] is proposed to address both the speed and compression issues, which splits the input layer space and applies vector quantization to each subspace. Researchers also try to prune filters and feature maps to directly reduce the computational cost [Molchanov et al., 2016][Li et al., 2016][He et al., 2017]. Transferring the learned knowledge from a large network to a small network is another important direction towards effective network compression. In [Hinton et al., 2014], distilling the knowledge of an ensemble model into a smaller model is introduced.

Instead of compressing a single network, the goal of this study is to merge multiple networks simultaneously. Besides, our method can restore the performance of the jointly compressed models by fine-tuning it with the training samples.

3 DeepModel Integration

Assume that model A (or B) consists of cA (or cB) convolution (Conv) followed by fA (or fB) fully connected (FC) layers. Let cmin = min(cA, cB). In our approach, a correspondence (cA(i), cB(i)) is established between the Conv layers for the alignment of the two models, i ∈ {1 : cmin}; A(·) is a strictly increasing mapping from {1 : cmin} to {1 : cA}, and B(·) is a strictly increasing mappings from {1 : cmin} to {1 : cB}. Likewise, a correspondence (fA(i), fB(i)) is also established between the FC layers for i ∈ {1 : fmin}.

In our method, the merged layers have to be of the same type (Conv or FC). Given two layers, one in model A and the other in model B, the principle to merge them is finding a set of (fewer) exemplar codewords that represent the weights of the layers with small quantization errors. The layers are thus jointly compressed for redundancy removal. Below, we first consider unifying the Conv layers, and then the FC layers.

3.1 Merging Convolution Layers

Assume that some Conv layer in model A and some other in model B are to be merged. The layer of model A has the input volume size NA ×MA × dA, where NA ×MA is the spatial size and dA is the depth (number of channels). The input volume is convolved with pA convolution kernels, where the size of each kernel is nA×mA×dA. The output of the Conv layer in model A is thus a volume of NA×MA×pA (without loss of generality, assume that padded convolutions are used.)

Likewise, similar notations apply to the respective layer in model B. An input volume of the size NB ×MB × dB are convolved by pB convolution kernels of nB ×mB ×dB. The output volume of that layer is of the size NB ×MB × pB.

We aim to jointly encode the convolution coefficients. As there are pA (or pB) convolution kernels in the layers of A (or B), we hope to find a new set of fewer (than pA + pB) exemplars to express the original ones so that the models are fused and the redundancy between them is removed. To this end, a viable way is to perform vector quantization (such as k-means clustering) on the convolution kernels and find a smaller number of codewords (p < pA + pB) to jointly represent the kernels compactly. However, it is demanding to make this method practicable because the kernel dimensions could be inconsistent (i.e., nA ̸= mA or nB ̸= mB).

To address this issue, we unify the different convolution kernels by using spatially 1×1 convolutions, so that merging CNNs with convolution kernels of different sizes is attainable. In the following, we review the operations in a convolution layer at first and then show how to separate the dimensions so that different layers are unified and jointly encoded.

Operations in Convolution Layer

The operations in a Conv layer of CNNs are reviewed as follows. Suppose x ∈ RN×M×d is the input volume (a.k.a. 3D tensor) to a Conv layer and y ∈ RN×M×p is the output volume. Assume that p convolution kernels of size n ×m× d are applied to the layer, denoted as

Then, the t-th channel output is obtained as yt = x ⋆ g(t) , the volume convolution of x and g(t) , and the output y is the concatenation of yt,

Let xu ∈ RN×M and g(t) u ∈ Rn×m respectively be the u-th channel of x and g(t) . The volume convolution is formed by summing the 2D-convolution results of the d channels:

where ∗ denotes the 2D convolution operator.

In CNNs, various n and m (eg., n = m = 3, 5, 7 · · · ) are used in existing networks. Particularly, when n = m = 1, the volume convolution of size 1 × 1 × d is often referred to as a 1 × 1 convolution in CNNs for all d.

Kernel Decomposition in Spatial Directions

In the above, the volume convolution is computed as a spatially sliding operation (2D convolution) followed by a channel-wise summation along the depth direction. In this section, we show that, no matter what n and m are, it can be equivalently represented by 1×1 convolutions via decomposing the kernel along with the spatial directions as follows.

Given the kernel g(t) , let g(t) [i0,j0],u specify its entry at the spatial location (i0, j0) of the u-th channel. In particular, let g(t) [i0,j0] ∈ Rd stand for the 1 × 1 × d volume convolution at (i0, j0); e.g., a kernel of spatial size 5 × 5 consists of 25 kernels of spatial size 1×1, (i0, j0) ∈ {1 : 5}×{1 : 5}, ∀d.

Following the notation, we decompose an n×m×d volume convolution into multiple 1×1×d convolutions and combine them with shift operators: Without loss of generality, we assume that the spatial sizes n, m of the kernel are odd numbers and replace them with w = (n - 1)/2 and h = (m - 1)/2. The t-th channel output yt can be equivalently represented as

where g(t) [i0,j0] (-w ≤ i0 ≤ w, -h ≤ j0 ≤ h) are the 1×1×d convolutions depicted above, and S is the shift operator,

with i, j the spatial location and u (1 ≤ u ≤ d) the channel index. Hence, for all n,m, the t-th channel output of the volume convolution can be decomposed as the shifted sum of nm 1 × 1 convolutions via Eq. 4. Then, the output y is obtained via the concatenation in Eq. 2.

To address the issue caused by dimension mismatch in merging two convolution layers, we then propose to take the representation of 1×1 convolutions for both layers. Hence, a kernel in model A is decomposed into CA = nAmA convolutions of size 1×1×dA and that in model B is decomposed into CB = nBmB convolutions of size 1 × 1 × dB. The kernel is then unified into 1 × 1 in the spatial domain no matter whether nA (or mA) equals to nB (or mB).

Kernel Separation along Depth Direction

Then, we seek to jointly express the CAB 1 × 1 convolutions by a compact representation so that the two layers are cocompressed, where CAB = pACA + pBCB and pA, pB are the numbers of kernels of the layers in A and B, respectively. Though the subspace dimensions in the spatial domain are consistent (1×1) now, they are still inconsistent in the depth direction (dA vs. dB) and thus crucial to be jointly clustered. To address this problem, we simply separate the 1 × 1 × d kernel g(t) [i0,j0] into non-overlapping 1 × 1 × r kernels along the depth direction (r < d). As the convolution in CNNs are summation-based in the depth direction, we divide the kernel g(t) [i0,j0] ∈ Rd into ⌈d/r⌉ vectors of dimension r,

where g(t) [i0,j0];〈v〉 (of size 1×1×r) is the v-th segment of the original kernel. The output yt in Eq. 4 then becomes

where x〈v〉 ∈ RN×M×r is the v-th sub-volume of the input x for d = dA or dB. Specifically, a spatially 1 × 1 kernel is respectively segmented into ρA = ⌈dA/r⌉ (or ρB = ⌈dB/r⌉) kernels of dimension r in model A (or B), where the last segment is padded with zero if necessary.

Let ρ = min(ρA, ρB). There are then CAB kernels of size 1×1×r for the segment 1 ≤ v ≤ ρ. To jointly represent the kernels of both layers, we use C codewords (C < CAB) in the dim-r space to encode the convolution coefficients compactly. We run the k-means algorithm with various initials for the CAB vectors and then select the results yielding the least representation error to produce the C codewords (i.e., cluster centers of k-means), for v ∈ {1, · · · , ρ}. 1

E-Conv Layer and Weights Co-use

The merged convolution layer (called the E-Conv layer), is a newly-formed layer where the weights are co-used among the convolution kernels: Denote the C codewords in the v-th subspace to be {bc,v ∈ Rr |c = 1 : C}. We then replace each dim-r kernel at the spatial site (i0, j0) in the subspace v (namely, g(t) [i0,j0];〈v〉 ∈ Rr ) with bπ(i0,j0,v,t);v, the closest codeword in the dim-r space, where π(i0, j0, v, t) ∈ {1 : C}

1 For those remaining segments, ρ + 1 ≤ v ≤ max(ρA, ρB), we also use C codewords to encode the pACA (or pBCB) dim-r vectors in the respective subspaces if dA > dB (or dA < dB).

is the code-assignment mapping. Eq. 7 is then simplified as

Because the number of codewords C is fewer than that of the total kernel vectors CAB, Eq. 8 can be executed more efficiently via computing the 1 × 1 convolutions of the C codewords at first:

and then storing the results in a lookup table at run-time. The run-time operation of 1 × 1 convolution is thus replaced by table indexing. Hence, the convolution kernels of the two models A and B are representationally shared in a compact codebook, {bc,v|v = 1 : C}, and the computation time is saved. An illustration of the E-Conv layer is given in Fig. 3.

When choosing the codewords C fewer, the amount of convolution coefficients is reduced to C/CAB. The merged model is thus co-compressed as it consumes less storage than the total required for the two convolution layers. As for the computational speed, each xi,j;〈v〉 ∈ Rr is replaced with an index and there are NMd/r entries for indexing in x〈v〉, where i, j are the spatial location. Let τx and be the time unit for a table-indexing operation and τr be the time unit for a 1 × 1 × r convolution. The speedup ratio is then (Cτr + NMdτx/r)/(CABτr) in terms of the complexity. Hence, when the codewords C is fewer or the subspace dimension r is larger, the speedup is getting higher.

Derivatives of the Merged Layer

Besides condensing and unifying the convolution operations, the E-Conv layer is also differentiable and so end-to-end back-propagation learning still remains realizable. However, evaluating the derivatives would be hard based on the tablelookup structure as the indices are not continuous. Hence, the table is used for the inference stage in our approach. While for learning, we slightly change the form of Eq. 8 to conduct the derivatives of yi,j;t (the output at the spatial location i, j of channel t) to {bc,v} (the codewords). From Eq. 8,

where 〈·, ·〉 is the inner product. Let Φ = [bc,v] ∈ Rr×C be the matrix whose columns are the dim-r codewords of the vth segment. Let βi0,j0,v,t ∈ RC be the one-hot vector where the c-th entry in βi0,j0,v,t is 1 if c = π(i0, j0, v, t); otherwise the entry is 0. Then, the codeword mapping bπ(i0,j0,v,t);v in Eq. 8 can be replaced by bπ(i0,j0,v,t);v = Φβi0,j0,v,t. Hence, the derivative of the E-Conv layer is conducted as

As Φ is the matrix consisting of the codewords {bc,v}, they can then be fine-tuned via the gradients for learning.

3.2 Merging Fully-connected Layers

The volume input to an FC layer is re-shaped to a vector in general. Let xF ∈ RNI be the input vector of an FC layer and yF ∈ RNO be its output. Then yF = WxF , where W∈ RNO×NI are the weights of the FC layer.

Unlike Conv layers that have sliding operations, all the operations in FC are summation-based. Thus, given the two weight matrices of models A and B, namely, WA and WB, we simply divide them into length-r segments along the row direction. The dim-r weight vectors in the same segment are then clustered via k-means algorithm and C codewords are found. In this way an E-FC layer is built as well, and It is easy to show that the E-FC layer is also differentiable.

3.3 End-to-end Fine Tuning

As both E-Conv and E-FC layers are differentiable, once their codebooks are constructed, we can then fine-tune the entire model from partial or all training data through end-to-end back-propagation learning. The training data are referred to as calibration data and the fine-tuning process is called the calibration training in our work. Codebooks are also used for the weights quantization in the single-model compression approach [Wu et al., 2016]. However, unlike our work that the codebooks in all layers are tunable in an end-to-end manner, only the codebook in a single layer can be tuned per time (with the other layers fixed) in [Wu et al., 2016], making its learning process inefficient and demanding to seek better solutions. Besides, our approach merges and jointly compresses multiple models, instead of only a single model.

Two error terms are combined for the minimization in our calibration training. One is the classification (or regression) loss utilized in the original models A and B. The other is the layer-wise output mismatch error; when applying the input xI to the model A (or B), the output of every layer in the merged model should be close to the output of the associated layer in A (or B), and L1 norm is used to measure this error. We use a framework (TensorFlow [Abadi et al., 2016]) to implement the calibration training.

In the inference stage, to make the approach generalizable to edge devices that may not contain GPUs, we use CPU and OpenBLAS library [Zhang et al., 2012; Wang et al., 2013] to implement the merged model with the associated codebooks. To make fair comparisons, the Conv layers of the individual models compared with ours are realized via the unrolling convolution [Chellapilla et al., 2006; Anwar et al., 2017] that converts the volume convolution into a single matrix product, which is commonly used as an efficient implementation for the Conv layer. Our code is publicly available at GitHub 2 .

4 Experiments

In this section, we present the results of several experiments conducted to verify the effectiveness of our approach.

4.1 Merging Sound and Image Recognition CNNs

The first experiment is to merge two CNNs of heterogeneous signal sources: image and sound. Although the sources are different, the same network (LeNet [LeCun et al., 1998]) is used. It is applied to two datasets of two tasks: sound classification dataset, and dressing style classification dataset.

2 https://github.com/ivclab/NeuralMerger

Sound20 dataset

This dataset contains 20 classes of sounds recorded from animal and instrument with 16,636 training samples and 3,727 testing samples. It is constructed using Animal Sound Data [Keogh, 2012] and Instrument Data [Juliani, 2016]. The raw signals are converted to 2D spectrograms and then resized to 32×32 images for classification. It is public available at: https://github.com/ivclab/Sound20

Fashion-MNIST dataset

Fashion-MNIST [Xiao et al., 2017] contains a collection of 60,000 training and 10,000 testing images of size 28 × 28 in 10 classes of dressing styles. To make the image size compatible with LeNet models, we resize the input images to 32×32 in our experiments.

LeNet consists of 2 Conv layers with 32 and 64 kernels of size 5 × 5 × 1 and 5 × 5 × 32, respectively, each followed by a 2 × 2 max-pooling, and then an FC layer of 1024 units and a γ-way output (γ is the number of classes). Before merged, LeNet can achieve 78.08% and 91.57% accuracy on the above sound (Sound20) and image recognition (FashionMNIST) tasks, respectively. The models are simply aligned and merged layer by layer since they are identical.

There are two main parameters, r and C, in our method, where r is the length of the segment and C is the number of codewords. As the input layer's depth of LeNet is 1, we set r = 1 for the first Conv layer of both models and choose C = 64 for it. Then, we alter the parameters of r ∈ {8, 16, 32} and C ∈ {64, 128, 256} for the remaining Conv- and FC-layers (except for the classification layer) and find the parameters per layer according to the performance when merging this layer only. By doing so, we conducted two settings, namely 'ACCU' and 'LIGHT', as shown in Table. 1, where 'ACCU' enforces the accuracy and 'LIGHT' enforces the model size and inference speed, respectively. We test the inference speed of our model on the Intel Xeon CPU E5-2640 v4 (single thread). The C++ codes from [Wu et al., 2016], which adopts OpenBLAS library and realizes unrolling convolution based on Caffe's [Jia et al., 2014] CPU-mode implementation, is used and enhanced to realize the inference stage of our work.

The overall performance (after calibration training) and accuracy drop (denoted by ↓) are shown in Table 2. With the setting of 'ACCU', over 10 times of the compression ratio can be obtained on the joint model size, while only a negligible accuracy drop is imposed. As can be seen, for the image task, the accuracy of our merged model drops only 0.68%; for the sound task, the merged model even achieves a better accuracy than the original model (-0.06% accuracy drop). Under the 'LIGHT' setting, our approach can achieve over 15 times of joint model-size compression with less than 1.4% accuracy drop. Fig. 4 further shows the influence calibration data amount. It can be seen that no matter for the 'ACCU' or 'LIGHT' setting, more training data yield a lower accuracy drop in overall.

As our approach finds several codewords per layer and then performs end-to-end calibration training to refine the codewords, it is also applicable to an individual model. One may wonder how the individual models perform when they are compressed via our approach. For an even comparison, we use the same r/C settings to encode the single sound (or image) model and also fine-tune it using all training data. Hence, the compressed single model consumes the same resource as our merged one (i.e., they have exactly the same model size and execution time). As shown in Table 3, when the soundonly model is compressed, the accuracy is slightly dropped by 0.4%; that is, our merged model can achieve even better accuracy while an additional functionality (image recognition) is added. On the other hand, when the image-only model is compressed, the accuracy-drop only changes little (from 0.51% to 0.68%) but an extra functionality (sound recognition) is supplemented. We owe that the two models have coredundancy in between, and thus jointly encoding the models can take advantage of the additional inter-redundancy to achieve a better representation.

4.2 Merging Clothing and Gender CNNClassifiers

In the second experiment, we merge two image classifiers (one for clothing and the other for gender recognition) into a single model holding double functionalities. The clothing classifier adopted is VGG-avg [Yang et al., 2018] which substitutes the first FC layer in the VGG-16 model [Simonyan and Zisserman, 2014] with average-pooling; it has been shown that VGG-avg achieves similar performance to VGG-16 while consuming far less storage. The gender classifier adopted is ZF-Net [Zeiler and Fergus, 2014] that has fewer layers than VGG-avg. The overall structures and alignment between them can be found in Fig. 2.

The clothing and gender datasets employed are depicted as follows. The Adience Dataset [Eidinger et al., 2014] contains face pictures of different genders and ages, where 11,136 images (each resized to 227 × 227) are used for training and the other 3,000 are for testing. The Multi-View-Clothing dataset [Liu et al., 2016] contains clothing images of different views with over 250 attributes organized in a tree, where the frontal images (37,485 for training and 3,000 for testing) and the 13 attributes of the first two layers are used to construct a classifier of 17 classes. Before merged, ZF-Net and VGGavg achieve 83.43% and 89.80% accuracies on the gender and clothing classification tasks, respectively.

Because the kernel sizes in the two models are not always consistent, they are merged via the decomposition into 1×1×r convolutions as described in Section 3.1. Similar to the process in the first experiment, we choose the parameters r/C and conduct two settings 'ACCU' and 'LIGHT', where the former stresses accuracy and the latter emphasizes compression/speed. The detailed parameter settings are reported in Table 4, and the overall performance is reported in Table 5. As the model sizes of both VGG-avg and ZF-Net are larger than that of LeNet, there are more redundancies between the models. Hence, an even better performance is achieved than that of experiment 1. In the 'ACCU' setting, the compression ratio exceeds 12 times with even an increase of the accuracy in the clothing classification task (negative accuracy drop). The compression ratio is enhanced to more than 20 times with acceptable accuracy drops on the 'LIGHT' setting.

Besides, Table 6 shows the results when compressing the individual models by using our approach under the same setting. It can be seen that similar (or even slightly better) accuracies can be obtained, with an additional merit that the merged model holds an extra functionality than the individual models. The results reveal that our approach can exploit both the intra- the inter-models redundancy and demonstrate satisfactory performance on merging deep learning models.

4.3 MergingMore Models

Finally, we show that our method can merge more models incrementally. The clothing-gender model in the above is added with another one, LeNet, for sound recognition (Fig. 2). In Table 7, the accuracy-drop changes are negligibly small but one more usage is added, which further confirms that our method can employ the redundancy among the models to enforce an efficient representation.

5 Conclusions

In this paper, we present a method that can unify multiple feed-forward models into a single but more compact one. To our knowledge, this is the first study on merging well-trained deep models for the inference stage. In our method, overall architectures of the original models are preserved. The unified model is still differentiable and can be fine-tuned to restore or enhance the performance. Experimental results show that the merged model can be extensively compressed under very limited accuracy drops, which makes our method potentially useful for deep model inference on low-end devices.

Acknowledgements

This work was supported in part by the MOST under the grant MOST 107-2634-F-001-004.