Mixed Link Networks

1 Introduction

The exploration of connectivity patterns of deep neural networks has attracted extensive attention in the literature of Convolutional Neural Networks (CNNs). LeNet [LeCun et al., 1998] originally demonstrated its layer-wise feed-forward pipeline, and later GoogLeNet [Szegedy et al., 2015] introduced more effective multi-path topology. Recently, ResNet [He et al., 2016a; He et al., 2016b] successfully adopted skip connection which transferred early information through identity mapping by element-wisely adding input features to its block outputs. DenseNet [Huang et al., 2017] further proposed a seemingly "different" topology by using densely connected path to concatenate all the previous raw input features with the output ones.

∗Authors contributed equally

†Corresponding authors

2 Key Laboratory of Intelligent Perception and Systems for HighDimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, 210094, P.R. China & Jiangsu Key Laboratory of Image and Video Understanding for Social Security, School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, 210094, P.R. China

For the two recent ResNet and DenseNet, despite their externally large difference in path topology (skip connection vs. densely connected path), we discover and prove that both of them are essentially derived from the same "dense topology" (Fig. 1 (a)), where their only difference lies in the form of connection ("+" in Fig. 1 (b) vs. "∥" in Fig. 1 (c)). Here, "dense topology" is defined as a path topology in which each layer Hℓ is connected with all the previous layers H0,H1, ...,Hℓ-1 using the connection function C(·) . The great effectiveness of "dense topology" has been proved via the significant success of both ResNet and DenseNet, yet the form of connection in ResNet and DenseNet still has room for improvement. For example, too many additions on the same feature space may impede the information flow in ResNet [Huang et al., 2017], and there may be the same type of raw features from different layers, which leads to a certain redundancy in DenseNet [Chen et al., 2017]. Therefore, the question "does there exist a more efficient form of connection in the dense topology" still remains to be further explored.

To address the problem, in this paper, we propose a novel Mixed Link Network (MixNet) with an efficient form of connection (Fig. 1 (d)) in the "dense topology". That is, we mix the connections in ResNet and DenseNet, in order to combine both the advantages of them and avoid their possible limitations. In particular, the proposed MixNets are equipped with both inner link modules and outer link modules, where an inner link module refers to additive feature vectors (similar connection in ResNet), while an outer link module stands for concatenated ones (similar connection in DenseNet). More importantly, in the architectures of MixNets, these two types of link modules are flexible with their positions and sizes. As a result, ResNet, DenseNet and the recently proposed Dual Path Network (DPN) [Chen et al., 2017] can be regarded as a special case of MixNet, respectively (see the details in Fig. 4 and Table 1).

To show the efficiency and effectiveness of the proposed MixNets, we conduct extensive experiments on four competitive benchmark datasets, namely, CIFAR-10, CIFAR-100, SVHN and ImageNet. The proposed MixNets require fewer parameters than the existing state-of-the-art architectures whilst achieving better or at least comparable results. Notably, on CIFAR-10 and CIFAR-100 datasets, MixNet-250

surpasses ResNeXt-29 (16×64d) with 57% less parameters. On ImagNet dataset, the results of MixNet-141 are comparable to the ones of DPN-98 with 50% fewer parameters. The main contributions of this paper are as follows:

• ResNet and DenseNet are proved to have the same path topology - "dense topology" essentially, whilst their only difference lies in the form of connections.
• A Mixed Link Network (MixNet) is proposed, which has a more efficient connection - the blending of flexible inner link modules and outer link modules.
• The relation between MixNet and modern networks (ResNet, DenseNet and DPN) is discussed, and these networks are shown to be specific instances of MixNets.
• MixNet demonstrates its superior efficiency in parameter over the state-of-the-art architectures on many competitive benchmarks.

The remainder of the paper is organized as follows. We summarize related work in Section 2 and devote Section 3 to study the "dense topology". Section 4 contains the main technical design of Mixed Link Networks. Experimental results and comparisons are presented in Section 5. Finally, we conclude this paper in Section 6.

2 RelatedWork

Designing effective path topologies always pushes the frontier of the advanced neural network architecture. Following the initial layer-wise feed-forward pipeline [LeCun et al., 1998], AlexNet [LeCun et al., 1998] and VGG [Simonyan and Zisserman, 2015] showed that building deeper networks with tiny convolutional kernels is a promising way to increase the learning capacity of neural network. GoogLeNet [Szegedy et al., 2015] demonstrated that a multi-path topology (codenamed Inception) could easily outperform previous feed-forward baselines by blending various information flows. The effectiveness of multi-path topology was further validated in FractalNet [Larsson et al., 2016], Highway Networks [Srivastava et al., 2015], DFN [Wang et al., 2016], DFN-MR [Zhao et al., 2016], and IGC [Zhang et al., 2017]. Perhaps the most revolutionary topology - skip connection was successfully adopted by ResNet [He et al., 2016a;

He et al., 2016b], where micro-blocks were built sequentially and the skip connection bridged the micro-block's input features with its output ones via identity mappings. Since then, different works based on ResNet have arisen, aiming to find a more efficient transformer of that micro-block, such as WRN [Zagoruyko and Komodakis, 2016], Multi-ResNet [Abdi and Nahavandi, 2016] and ResNeXt [Xie et al., 2017]. Furthermore, DenseNet [Huang et al., 2017] achieved comparable accuracy with deep ResNet by proposing the densely connected topology, which connects each layer to its previous layers by concatenation. Recently, DPN [Chen et al., 2017] directly combines the two paths - ResNet path and DenseNet path together by a shared feature embedding in order to enjoy a mutual improvement.

3 Dense Topology

In this section, we first introduce and formulate the "dense topology". We then prove that both ResNet and DenseNet are intrinsically derived from the same "dense topology", and they only differ in the specific form of connection (addition vs. concatenation). Furthermore, we present analysis on strengths and weaknesses of these two network architectures, which motivates us to develop Mixed Link Networks.

Definitions of "dense topology". Let us consider a network that comprises L layers, each of which implements a non-linear transformation Hℓ(·), where ℓ indexes the layer. Hℓ(·) could be a composite function of several operations such as linear transformation, convolution, activation function, pooling [LeCun et al., 1998], batch normalization [Ioffe and Szegedy, 2015]. As illustrated in Fig. 2 (a), Xℓ refers to the immediate output of the transformation Hℓ(·) and Sℓ is the result of the connection function C(·) whose inputs come from all the previous feature-maps X (i.e., X0,X1, ...,Xℓ). Initially, S0 equals X0. As mentioned in Section 1, "dense topology" is defined as a path topology where each layer is connected with all the previous layers. Therefore, we can formulate the general form of "dense topology" simply as:

DenseNet is derived from "dense topology" obviously. For DenseNet [Huang et al., 2017], the input of ℓth layer is the concatenation of the outputs X0,X1, ...,Xℓ-1 from all the preceding layers. Therefore, we can write DenseNet as:

where "∥" refers to the concatenation. As shown in Eqn. (1) and Eqn. (2), DenseNet directly follows the formulation of "dense topology", whose connection function is the pure concatenation (Fig. 1 (c)).

ResNet is also derived from "dense topology". We then explain that ResNet also follows the "dense topology" whose connection is accomplished by addition. Given the standard definition from [He et al., 2016b], ResNet poses a skip connection that bypasses the non-linear transformations Hℓ(·) with an identity mapping as:

where R refers to the feature-maps directly after the skip connection (Fig. 2 (b)). Initially, R0 equals X0. Now we concentrate on Xℓ which is the output of Hℓ(·) as well:

By substituting Eqn. (3) into Eqn. (4) recursively, we can rewrite Eqn. (4) as:

As shown in Eqn. (5) clearly, Rℓ-1 in ResNet is deduced to be the element-wise addition result of all the previous layers - X0,X1, ...,Xℓ-1. It proves that ResNet is actually identical to a form of "dense topology", where the connection function C(·) is specified to the addition (Fig. 1 (b)).

The above analyses reveal that ResNet and DenseNet share the same "dense topology" in essence. Therefore, the "dense topology" is confirmed to be a fundamental and significant path topology that works practically, due to the extraordinary effectiveness of both ResNet and DenseNet in the recent progress. Meanwhile, from Eqn. (2) and Eqn. (5), the only difference between ResNet and DenseNet is the connection function C(·) ("+" vs. "∥") obviously.

Analysis of ResNet. The connection in ResNet is only the additive form ("+") that operates on the entire feature map. It combines the features from previous layers by element-wise addition, which makes the features more expressive and eases the gradient flow for optimization simultaneously. However, too many additions on the same feature space may impede the information flow in the network [Huang et al., 2017], which motivates us to develop a "shifted additions", by dislocating/shifting the additive positions in subsequent feature spaces along multiple layers (e.g., the black arrow in Fig. 4 (e)), to alleviate this problem.

Analysis of DenseNet. The connection in DenseNet is only the concatenative connection ("∥") which increases the feature dimension gradually along the depths. It concatenates the raw features from previous layers to form the input of the new layer. Concatenation allows the new layer to receive the raw features directly from previous layers and it also improves the flow of information between layers. However, there may be the same type of features from different layers, which leads to a certain redundancy [Chen et al., 2017]. This limitation also inspires us to introduce the "shifted additions" (e.g., the black arrow in Fig. 4 (e)) on these raw features in purpose of a modification to avoid that redundancy to some extent.

4 Mixed Link Networks

In this section, we first introduce and formulate the inner/outer link modules. Next, we present the generalized mixed link architecture with flexible inner/outer link modules and propose Mixed Link Network (MixNet), a representative form of the generalized mixed link architecture. At last, we describe the implementation details of MixNets.

4.1 Inner/Outer LinkModule

The inner link modules are based on the additive connections. Following the above preliminaries, we denote the output Sin ℓ which contains the inner link part as:

where Hin ℓ (·) refers to the function of producing featuremaps for inner linking - element-wisely adding new features Hin ℓ (Sin ℓ-1 ) inside the original ones Sin ℓ-1 .

The outer link modules are based on the concatenated connection. Similarly, we have Sout ℓ as:

where Hout ℓ (·) refers to the function of producing featuremaps for outer linking - appending new features Hout ℓ (Sout ℓ-1 ) outside the original ones Sout ℓ-1 .

4.2 Mixed Link Architecture

Due to the analyses in Section 3, we introduce the mixed link architecture which embraces both inner link modules and outer link modules (Fig. 3). The mixed link architecture can be formulated as Eqn. (8), a flexible combination of Eqn. (6) and Eqn. (7), to get a blending feature output Sℓ:

Definitions of parameters (k1, k2, fixed/unfixed) for mixed link architecture. Here we denote the channel number of feature-maps produced by Hin ℓ (·) and Hout ℓ (·) as k1 and k2, respectively. That is, k1 is the inner link size for inner link modules, and k2 controls the outer link size for outer link modules. As for the positional control for inner link modules, we simplify it into two choices - "fixed" or "unfixed". The "fixed" term is easy to understand - all the features are merged together by addition over the same fixed space, as in ResNet. Here is the explanation for "unfixed": there are extremely exponential combinations to pose the inner link modules' positions along multiple layers, and learning the variable position is currently unavailable since their arrangement is not derivable directly. Therefore, we make a compromise and choose one simple series of the unfixed-position version - the "shifted addition" (Fig. 4 (e)) as mentioned in our motivations in Section 3. Specifically, the position of inner link part exactly aligns with the growing boundary of entire feature embedding (see the black arrow in Arch-4) when the outer link parts increase the overall feature dimension. We denote this Arch-4 (Fig. 4 (e)) to be our proposed model exactly - Mixed LinkNetwork (MixNet). In summary, we have defined the above two simple options for controlling the positions of inner link modules as - "fixed" and "unfixed" .

Modern networks are special cases of MixNets. It can be seen from Fig. 4 that the mixed link architecture (Fig. 4 (a)) with different parametric configurations can reach four representative architectures (Fig. 4 (b)(c)(d)(e)). The configurations of these corresponding architectures are listed in Table 1. We show that MixNet is a more generalized form than other exsiting modern networks, under the perspective of mixed link architecture. Therefore, ResNet, DenseNet and DPN can be treated as a specific instance of MixNets, respectively.

4.3 Implementation Details of MixNets

The proposed network consists of multiple mixed link blocks. Each mixed link block has several layers, whose structure follows Arch-4 (Fig. 4 (e)). Motivated from the common practices [He et al., 2016a], we introduce bottleneck layers as unitary elements in MixNets. That is, we implement both Hin ℓ (·) and Hout ℓ (·) with such a bottleneck layer - BNReLU-Conv(1, 1)-BN-ReLU-Conv(3, 3). Here BN, ReLU, and Conv refer to batch normalization, rectified linear units and convolution, respectively.

On CIFAR-10, CIFAR-100 and SVHN datasets, the MixNets used in our experiments have three mixed link blocks with the same amount of layers. Before entering the first mixed link block, a convolution with max(k1, 2 × k2) output channels is performed on the input images. For convolutional layers with kernel size 3×3, each side of the inputs is zero-padded by one pixel to keep the feature-map size fixed. We use 1×1 convolution followed by 2×2 average pooling as transition layers between two contiguous blocks. At the end of the last block, a global average pooling is performed and then a softmax classifier is attached. The feature-map sizes in the three blocks are 32×32, 16×16, and 8×8, respectively. We survey the network structure with three configurations: {L = 100, k1 = 12, k2 = 12}, {L = 250, k1 = 24, k2 = 24} and {L = 190, k1 = 40, k2 = 40} in practice.

In our experiments on ImageNet dataset, we follow Arch-4 and use the network structure with four mixed link blocks on

224 × 224 input images. The initial convolution layer comprises max(k1, 2 × k2) filters whose size is 7 × 7 and stride is 2. The sizes of feature-maps in the following layers are determined by the settings of inner link parameter k1 and outer link parameter k2 (Table 2), consequently.

5 Experiment

In this section, we empirically demonstrate MixNets effectiveness and efficiency in parameter over the state-of-the-art architectures on many competitive benchmarks.

5.1 Datasets

CIFAR. The two CIFAR datasets [Krizhevsky and Hinton, 2009] consist of colored natural images with 32 × 32 pixels. The training and test sets contain 50K and 10K images, respectively. We follow the standard data augmentation scheme that is widely used for these two datasets [He et al., 2016a; Huang et al., 2016; Li et al., 2018]. For preprocessing, we normalize the data using the channel means and standard deviations. For the final run we use all 5K training images and report the final test error at the end of training.

SVHN. The Street View House Numbers (SVHN) dataset [Netzer et al., 2011] contains 32 × 32 colored digit images. There are 73,257 images in the training set, 26,032 images in the test set, and 531,131 images for extra training data. Following common practice [Huang et al., 2016; Lin et al., 2014], We use all the training data (training set and extra training data) without any data augmentation, and a validation set with 6,000 images is split from the training set. In addition, the pixel values in the dataset are divided by 255 and thus they are in the [0, 1] range. We select the model with the lowest validation error during training and report the test error.

ImageNet. The ILSVRC 2012 classification dataset [Deng et al., 2009] contains 1.2 million images for training, and 50K for validation, from 1K classes. We adopt the same data augmentation scheme for training images as in [He et al., 2016a;

He et al., 2016b], and apply a single-crop with size 224×224 at test time. Following [He et al., 2016a; He et al., 2016b], we report classification errors on the validation set.

5.2 Training

All the networks are trained by using stochastic gradient descent (SGD). On CIFAR and SVHN we train using batch size 64 for 300 epochs. The initial learning rate is set to 0.1, and is divided by 10 at 50% and 75% of the total number of training epochs. On ImageNet, we train models with a mini-batch size 150 (MixNet-121) and 100 (MixNet-141) due to GPU memory constraints. To compensate for the smaller batch size, the models are trained for 100 epochs, and the learning rate is lowered by 10 times at epoch 30, 60 and 90. Following [He et al., 2016a], we use a weight decay of 10-4 and a Nesterov momentum [Sutskever et al., 2013] of 0.9 without dampening. We adopt the weight initialization introduced by [He et al., 2015]. For the the dataset without data augmentation (i.e., SVHN), we follow the DenseNet setting [Huang et al., 2017] and add a dropout layer [Srivastava et al., 2014] after each convolutional layer (except the first one) by setting the dropout rate as 0.2.

5.3 Ablation Study for Mixed Link Architecture

Efficiency comparisons among the four architectures. We first evaluate the efficiency of the four representative architectures which are derived from the mixed link architecture. The comparisons are based on various amount of parameters (#params). Specifically, we increase the complexities of the four architectures in parallel and evaluate them on CIFAR100 dataset. The experimental results are reported in Fig. 5 (a), from which we can find that with various similar parameters, Arch-4 outperforms all other three architectures by a margin. It demonstrates the superior efficiency in parameter of Arch-4 which is exactly used in our proposed MixNets.

Fixed vs. unfixed for the inner link modules. Next we investigate "which is the more effective setting for the inner link modules - fixed or unfixed?". To ensure a fair comparison, we hold the outer link parameter k2 constant and train MixNets with different inner link parameter k1. In details, we set k2 to 12, and let k1 increase from 0 to 24. The models are also evaluated on CIFAR-100 dataset. Fig. 5 (c) shows the experimental results, from which we can find that with the growing of k1, the test error rate keeps dropping. Furthermore, with the same inner link parameter k1, the models with unfixed inner link modules (red curve) have much lower test errors than the models with the fixed ones (green curve), which suggests the superiority of unfixed inner link module.

Outer link size. We then study the effect of outer link size k2 by setting k1 = 12, under the configurations with the effective unfixed inner link modules on CIFAR-100 dataset. Fig. 5 (d) illustrates that the increasement of k2 reduces the test error rate consistently. However, the performance gain becomes tiny when k2 is relatively large. Therefore, we prefer to set k2 = k1 for a better space-time tradeoff in this study.

5.4 Experiments on CIFAR and SVHN

We train MixNets with different depths L, inner link parameters k1 and outer link parameters k2. The main results on CIFAR and SVHN are shown in Table 3.

As can be seen from the bottom rows of Table 3, MixNet190 outperforms many state-of-the-art architectures consistently on CIFAR datasets. Its error rates, 3.13% on CIFAR-10 and 16.96% on CIFAR-100, are significantly lower than the error rates achieved by DPN-29-10. Our results on SVHN are also remarkable. MixNet-100 achieves comparable test errors with DFN-MR (24.1M) and IGC-L32M26 (24.8M) whilst costing only 1.5M parameters.

5.5 Experiments on ImageNet

We evaluate MixNets with different depths and inner/outer link parameters on the ImageNet classification task, and compare it with the representative state-of-the-art architectures.

We report the single-crop validation errors of MixNets on ImageNet in Table 4. The single-crop top-1 validation errors of MixNets and different state-of-the-art architectures as a function of the number of parameters are shown in Fig. 5 (b). The results reveal that MixNets perform on par with the stateof-the-art architectures, whilst requiring significantly fewer parameters to achieve better or at least comparable performance. For example, MixNet-105 outperforms DenseNet169 and DPN-68 with only 11.16M parameters. MixNet121 (21.86M) yields better validation error than ResNeXt-50 (25M) and Densenet-264 (33.34M). Furthermore, the results of MixNet-141 are very close to the ones of DPN-98 with 50% fewer parameters.

6 Conclusion

In this paper, we first prove that ResNet and DenseNet are essentially derived from the same fundamental "dense topology", whilst their only difference lies in the specific form of connection. Next, by the analysis of superiority and insufficiency of their distinct connections, we propose a highly efficient form of it - the Mixed Link Networks (MixNets), making an effective bridge between ResNet and DenseNet. Extensive experimental results demonstrate that our proposed MixNet is efficient in parameter.

Acknowledgements

The authors would like to thank the editor and the anonymous reviewers for their critical and constructive comments and suggestions. The work was supported by the Natural Science Foundation of China under Grant No. 61672273, No. 61272218 and No. 61321491, the Science Foundation for Distinguished Young Scholars of Jiangsu under Grant No. BK20160021, Scientific Foundation of State Grid Corporation of China (Research on Ice-wind Disaster Feature Recognition and Prediction by Few-shot Machine Learning in Transmission Lines), the National Science Fund of China under Grant Nos. U1713208 and 61472187, the 973 Program No.2014CB349303, and Program for Changjiang Scholars.