{"type":"ScholarlyArticle","title":"Deep Convolutional Neural Networks with Merge-and-Run Mappings","authors":[{"name":"Liming Zhao","slug":"liming-zhao","given":"Liming","family":"Zhao","affiliation":"Zhejiang University; https://github.com/ zlmzju / fusenet"},{"name":"Mingjie Li","slug":"mingjie-li","given":"Mingjie","family":"Li","affiliation":"University of Science and Technology of China"},{"name":"Depu Meng","slug":"depu-meng","given":"Depu","family":"Meng","affiliation":"University of Science and Technology of China"},{"name":"Xi Li","slug":"xi-li","given":"Xi","family":"Li","affiliation":"Zhejiang University; https://github.com/ zlmzju / fusenet"},{"name":"Zhaoxiang Zhang","slug":"zhaoxiang-zhang","given":"Zhaoxiang","family":"Zhang","affiliation":"Institute of Automation , Chinese Academy of Sciences"},{"name":"Yueting Zhuang","slug":"yueting-zhuang","given":"Yueting","family":"Zhuang","affiliation":"Zhejiang University; https://github.com/ zlmzju / fusenet"},{"name":"Zhuowen Tu","slug":"zhuowen-tu","given":"Zhuowen","family":"Tu","affiliation":"UC San Diego"},{"name":"Jingdong Wang","slug":"jingdong-wang","given":"Jingdong","family":"Wang","affiliation":"Microsoft Research"}],"abstract":"A deep residual network, built by stacking a sequence of residual blocks, is easy to train, because identity mappings skip residual branches and thus improve information flow. To further reduce the training difficulty, we present a simple network architecture, deep merge-and-run neural networks. The novelty lies in a modularized building block, merge-and-run block, which assembles residual branches in parallel through a merge-and-run mapping: average the inputs of these residual branches (Merge), and add the average to the output of each residual branch as the input of the subsequent residual branch (Run), respectively. We show that the merge-and-run mapping is a linear idempotent function in which the transformation matrix is idempotent, and thus improves information flow, making training easy. In comparison with residual networks, our networks enjoy compelling advantages: they contain much shorter paths and the width, i.e., the number of channels, is increased, and the time complexity remains unchanged. We evaluate the performance on the standard recognition tasks. Our approach demonstrates consistent improvements over ResNets with the comparable setup, and achieves competitive results (e.g., 3.06% testing error on CIFAR-10, 17.55% on CIFAR-100, 1.51% on SVHN).","doi":"10.24963/ijcai.2018/440","url":"https://ijcai.org/proceedings/2018/440","pdf":"https://ijcai.org/proceedings/2018/0440.pdf","markdown":"https://ijcai.org/proceedings/2018/440.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/440","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":440,"pages":{"first":3170,"last":3176},"keywords":["Machine Learning: Deep Learning","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Abdi and Nahavandi, 2016] Masoud Abdi and Saeid Naha- vandi. Multi-residual networks. CoRR, abs/1609.05672, 2016.","title":"Multi-residual networks","authors":"Masoud Abdi, Saeid Nahavandi","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2015] Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. Mxnet: A flexible and efficient machine learning library for heterogeneous dis- tributed systems. CoRR, abs/1512.01274, 2015.","title":"Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems","authors":"Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, Zheng Zhang","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chollet, 2017] Francois Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, 2017.","title":"Xception: Deep learning with depthwise separable convolutions","authors":"Francois Chollet","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Girshick et al., 2014] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.","title":"Rich feature hierarchies for accurate object detection and semantic segmentation","authors":"Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Han et al., 2017] Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. In CVPR, 2017.","title":"Deep pyramidal residual networks","authors":"Han","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In ICCV, 2015.","title":"Delving deep into rectifiers: Surpassing human-level performance on imagenet classification","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016a] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"He","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016b] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016.","title":"Identity mappings in deep residual networks","authors":"He","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2016] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In ECCV, 2016.","title":"Deep networks with stochastic depth","authors":"Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, Kilian Q Weinberger","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely con- nected convolutional networks. In CVPR, 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky, 2009] Alex Krizhevsky. Learning multiple lay- ers of features from tiny images. Technical report, Univer- sity of Toronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Larsson et al., 2017] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. In ICLR, 2017.","title":"Fractalnet: Ultra-deep neural networks without residuals","authors":"Larsson","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lee et al., 2015] Chen-Yu Lee, Saining Xie, Patrick W. Gal- lagher, Zhengyou Zhang, and Zhuowen Tu. Deeply- supervised nets. In AISTATS, 2015.","title":"Deeplysupervised nets","authors":"Chen-Yu Lee, Saining Xie, Patrick W Gallagher, Zhengyou Zhang, Zhuowen Tu","year":"2015","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In CVPR, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Long","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Singh et al., 2016] Saurabh Singh, Derek Hoiem, and David A. Forsyth. Swapout: Learning an ensemble of deep architectures. In NIPS, 2016.","title":"Swapout: Learning an ensemble of deep architectures","authors":"Saurabh Singh, Derek Hoiem, David A Forsyth","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Srivastava et al., 2015] Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. Training very deep networks. In NIPS, 2015.","title":"Training very deep networks","authors":"K Rupesh, Klaus Srivastava, Jürgen Greff, Schmidhuber","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In CVPR, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Szegedy et al., 2017] Christian Szegedy, Sergey Ioffe, Vin- cent Vanhoucke, and Alexander A. Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In AAAI, 2017.","title":"Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A. Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning","authors":"Szegedy","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Veit et al., 2016] Andreas Veit, Michael J. Wilber, and Serge J. Belongie. Residual networks behave like ensem- bles of relatively shallow networks. In NIPS, 2016.","title":"Residual networks behave like ensembles of relatively shallow networks","authors":"Andreas Veit, Michael J Wilber, Serge J Belongie","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2016] Jingdong Wang, Zhen Wei, Ting Zhang, and Wenjun Zeng. Deeply-fused nets. CoRR, abs/1605.07716, 2016.","title":"Deeply-fused nets","authors":"Jingdong Wang, Zhen Wei, Ting Zhang, Wenjun Zeng","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu et al., 2016] Zifeng Wu, Chunhua Shen, and Anton van den Hengel. Wider or deeper: Revisiting the resnet model for visual recognition. CoRR, abs/1611.10080, 2016.","title":"Wider or deeper: Revisiting the resnet model for visual recognition","authors":"Wu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xie and Tu, 2015] Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In ICCV, 2015.","title":"Holistically-nested edge detection","authors":"Tu ; Xie, Saining Xie, Zhuowen Tu","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xie et al., 2017] Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In CVPR, 2017.","title":"Aggregated residual transformations for deep neural networks","authors":"Xie","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xie et al., 2018a] Guotian Xie, Jingdong Wang, Ting Zhang, Jianhuang Lai, Richang Hong, and Guo-Jun Qi. Igcv2: Interleaved structured sparse convolutional neural networks. In CVPR, 2018.","title":"Igcv2: Interleaved structured sparse convolutional neural networks","authors":"Xie","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xie et al., 2018b] Guotian Xie, Ting Zhang, Kuiyuan Yang, Jianhuang Lai, and Jingdong Wang. Decoupled convolu- tions for cnns. In AAAI, 2018.","title":"Decoupled convolutions for cnns","authors":"Xie","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zagoruyko and Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.","title":"Wide residual networks","authors":"Sergey Zagoruyko, Nikos Komodakis","year":"2016","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2017a] K. Zhang, M. Sun, X. Han, X. Yuan, L. Guo, and T. Liu. Residual networks of residual net- works: Multilevel residual networks. IEEE Trans. Cir. and Sys. for Video Technol., pages 1-1, 2017.","title":"Residual networks of residual networks: Multilevel residual networks","authors":"Zhang","year":"2017","venue":"IEEE Trans. Cir. and Sys. for Video Technol","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2017b] Ting Zhang, Guo-Jun Qi, Bin Xiao, and Jingdong Wang. Interleaved group convolutions. In ICCV, 2017.","title":"Interleaved group convolutions","authors":"Zhang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhao et al., 2016] Liming Zhao, Jingdong Wang, Xi Li, Zhuowen Tu, and Wenjun Zeng. On the connection of deep fusion to ensembling. CoRR, abs/1611.07718, 2016.","title":"On the connection of deep fusion to ensembling","authors":"Liming Zhao, Jingdong Wang, Xi Li, Zhuowen Tu, Wenjun Zeng","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}