{"type":"ScholarlyArticle","title":"Mixed Link Networks","authors":[{"name":"Wenhai Wang","slug":"wenhai-wang","given":"Wenhai","family":"Wang","affiliation":"National Key Lab for Novel Software Technology , Nanjing University; Jiangsu Key Laborato- ry of Image and Video Understanding for Social Security , School of Computer Science and Engineering , Nanjing University of Science and Technology , Nanjing , 210094 , P.R. China"},{"name":"Xiang Li","slug":"xiang-li","given":"Xiang","family":"Li","affiliation":"DeepInsight@PCALab , Nanjing University of Science and Technology; Key Laboratory of Intelligent Perception and Systems for High- Dimensional Information of Ministry of Education , School of Com- puter Science and Engineering , Nanjing University of Science and Technology , Nanjing , 210094 , P.R. China &"},{"name":"Tong Lu","slug":"tong-lu","given":"Tong","family":"Lu","affiliation":"National Key Lab for Novel Software Technology , Nanjing University"},{"name":"Jian Yang","slug":"jian-yang","given":"Jian","family":"Yang","affiliation":"DeepInsight@PCALab , Nanjing University of Science and Technology; Key Laboratory of Intelligent Perception and Systems for High- Dimensional Information of Ministry of Education , School of Com- puter Science and Engineering , Nanjing University of Science and Technology , Nanjing , 210094 , P.R. China &"}],"abstract":"On the basis of the analysis by revealing the equivalence of modern networks, we find that both ResNet and DenseNet are essentially derived from the same \"dense topology\", yet they only differ in the form of connection: addition (dubbed \"inner link\") vs. concatenation (dubbed \"outer link\"). However, both forms of connections have the superiority and insufficiency. To combine their advantages and avoid certain limitations on representation learning, we present a highly efficient and modularized Mixed Link Network (MixNet) which is equipped with flexible inner link and outer link modules. Consequently, ResNet, DenseNet and Dual Path Network (DPN) can be regarded as a special case of MixNet, respectively. Furthermore, we demonstrate that MixNets can achieve superior efficiency in parameter over the state-of-the-art architectures on many competitive datasets like CIFAR-10/100, SVHN and ImageNet.","doi":"10.24963/ijcai.2018/391","url":"https://ijcai.org/proceedings/2018/391","pdf":"https://ijcai.org/proceedings/2018/0391.pdf","markdown":"https://ijcai.org/proceedings/2018/391.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/391","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":391,"pages":{"first":2819,"last":2825},"keywords":["Machine Learning: Neural Networks","Computer Vision: Computer Vision","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Abdi and Nahavandi, 2016] Masoud Abdi and Saeid Naha- vandi. Multi-residual networks. CoRR, abs/1609.05672, 8, 2016.","title":"Multi-residual networks","authors":"Masoud Abdi, Saeid Nahavandi","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2017] Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin, Shuicheng Yan, and Jiashi Feng. Dual path networks. In NIPS, 2017.","title":"Dual path networks","authors":"Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin, Shuicheng Yan, Jiashi Feng","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In ICCV, 2015.","title":"Delving deep into rectifiers: Surpassing human-level performance on imagenet classification","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2016a] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"He","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016b] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016.","title":"Identity mappings in deep residual networks","authors":"He","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huang et al., 2016] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In ECCV, 2016.","title":"Deep networks with stochastic depth","authors":"Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, Kilian Q Weinberger","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. Densely con- nected convolutional networks. In CVPR, 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Kilian Q Weinberger, Laurens Van Der Maaten","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In ICML, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. Tech Report, 2009. Larsson et al., 2016] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neu- ral networks without residuals. arXiv preprint arX- iv:1605.07648, 2016.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":"Fractalnet: Ultra-deep neural networks without residuals","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2018] Xiang Li, Shuo Chen, Xiaolin Hu, and Jian Yang. Understanding the disharmony between dropout and batch normalization by variance shift. arXiv preprint arX- iv:1801.05134, 2018.","title":"Understanding the disharmony between dropout and batch normalization by variance shift","authors":"Xiang Li, Shuo Chen, Xiaolin Hu, Jian Yang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin et al., 2014] Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. In ICLR, 2014.","title":"Network in network","authors":"Lin","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Netzer et al., 2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learn- ing. In NIPS Workshop, 2011.","title":"Reading digits in natural images with unsupervised feature learning","authors":"Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, Andrew Y Ng","year":"2011","venue":"NIPS Workshop","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Re- search, 15(1):1929-1958, 2014.","title":"Dropout: A simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Srivastava et al., 2015] Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. Training very deep networks. In NIPS, 2015.","title":"Training very deep networks","authors":"K Rupesh, Klaus Srivastava, Jürgen Greff, Schmidhuber","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sutskever et al., 2013] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In ICML, 2013.","title":"On the importance of initialization and momentum in deep learning","authors":"Sutskever","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In CVPR, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2016] Jingdong Wang, Zhen Wei, Ting Zhang, and Wenjun Zeng. Deeply-fused nets. arXiv preprint arX- iv:1605.07716, 2016.","title":"Deeply-fused nets","authors":"Jingdong Wang, Zhen Wei, Ting Zhang, Wenjun Zeng","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xie et al., 2017] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In CVPR, 2017.","title":"Aggregated residual transformations for deep neural networks","authors":"Xie","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zagoruyko and Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.","title":"Wide residual networks","authors":"Sergey Zagoruyko, Nikos Komodakis","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2017] Ting Zhang, Guo-Jun Qi, Bin Xiao, and Jingdong Wang. Interleaved group convolutions. In CVPR, 2017.","title":"Interleaved group convolutions","authors":"Ting Zhang, Guo-Jun Qi, Bin Xiao, Jingdong Wang","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhao et al., 2016] Liming Zhao, Jingdong Wang, Xi Li, Zhuowen Tu, and Wenjun Zeng. On the connection of deep fusion to ensembling. arXiv preprint arXiv:1611.07718, 2016.","title":"On the connection of deep fusion to ensembling","authors":"Liming Zhao, Jingdong Wang, Xi Li, Zhuowen Tu, Wenjun Zeng","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}