{"type":"ScholarlyArticle","title":"Refine or Represent: Residual Networks with Explicit Channel-wise Configuration","authors":[{"name":"Yanyan Shen","slug":"yanyan-shen","given":"Yanyan","family":"Shen","affiliation":"Shanghai Jiao Tong University"},{"name":"Jinyang Gao","slug":"jinyang-gao","given":"Jinyang","family":"Gao","affiliation":"National University of Singapore"}],"abstract":"The successes of deep residual learning are mainly based on one key insight: instead of learning a completely new representation y = H(x), it is much easier to learn and optimize its residual mapping F(x)= H(x)-x, as F(x) could be generally closer to zero than the non-residual function H(x). In this paper, we further exploit this insight by explicitly configuring each feature channel with a fine-grained learning style. We define two types of channel-wise learning styles: Refine and Represent. A Refine channel is learnt via the residual function yi= Fi(x) + xi with a regularization term on the channel response ||Fi(x)||, aiming to refine the input feature channel xi of the layer. A Represent channel directly learns a new representation yi = Hi(x) without calculating the residual function with reference to xi. We apply random channel-wise configuration to each residual learning block. Experimental results on the CIFAR10, CIFAR100 and ImageNet datasets demonstrate that our proposed method can substantially improve the performance of conventional residual networks including ResNet, ResNeXt and SENet.","doi":"10.24963/ijcai.2018/372","url":"https://ijcai.org/proceedings/2018/372","pdf":"https://ijcai.org/proceedings/2018/0372.pdf","markdown":"https://ijcai.org/proceedings/2018/372.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/372","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":372,"pages":{"first":2682,"last":2688},"keywords":["Machine Learning: Neural Networks","Machine Learning: Deep Learning","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Baldi and Sadowski, 2013] Pierre Baldi and Peter J Sad- owski. Understanding dropout. In NIPS, 2013.","title":"Understanding dropout","authors":"Sadowski Baldi, Pierre Baldi, Peter J Sadowski","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2015] Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In ICML, 2015.","title":"Compressing neural networks with the hashing trick","authors":"Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, Yixin Chen","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2017] Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin, Shuicheng Yan, and Jiashi Feng. Dual path networks. In NIPS, 2017.","title":"Dual path networks","authors":"Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin, Shuicheng Yan, Jiashi Feng","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gastaldi, 2017] Xavier Gastaldi. Shake-shake regulariza- tion. arXiv preprint arXiv:1705.07485, 2017.","title":"Shake-shake regularization","authors":"Xavier Gastaldi","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Girshick et al., 2014] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.","title":"Rich feature hierarchies for accurate object detection and semantic segmentation","authors":"Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hariharan et al., 2014] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous detection and segmentation. In ECCV, 2014.","title":"Simultaneous detection and segmentation","authors":"Bharath Hariharan, Pablo Arbeláez, Ross Girshick, Jitendra Malik","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In ICCV, 2015.","title":"Delving deep into rectifiers: Surpassing human-level performance on imagenet classification","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016a] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"He","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016b] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016.","title":"Identity mappings in deep residual networks","authors":"He","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hu et al., 2017] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. arXiv preprint arXiv:1709.01507, 2017.","title":"Squeeze-and-excitation networks","authors":"Jie Hu, Li Shen, Gang Sun","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang et al., 2016a] Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. Densely connected convolutional networks. arXiv preprint arXiv:1608.06993, 2016.","title":"Densely connected convolutional networks","authors":"Huang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huang et al., 2016b] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In ECCV, 2016.","title":"Deep networks with stochastic depth","authors":"Huang","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang et al., 2017] Gao Huang, Shichen Liu, Laurens van der Maaten, and Kilian Q Weinberger. Condensenet: An efficient densenet using learned group convolutions. arXiv preprint arXiv:1711.09224, 2017.","title":"Condensenet: An efficient densenet using learned group convolutions","authors":"Gao Huang, Shichen Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In ICML, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Larsson et al., 2016] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648, 2016.","title":"Fractalnet: Ultra-deep neural networks without residuals","authors":"Larsson","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2013] Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv:1312.4400, 2013.","title":"Network in network","authors":"Lin","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Smith et al., 2016] Leslie N Smith, Emily M Hand, and Timothy Doster. Gradual dropin of layers to train very deep neural networks. In CVPR, 2016.","title":"Gradual dropin of layers to train very deep neural networks","authors":"Emily M Leslie N Smith, Timothy Hand, Doster","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey E Hin- ton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Srivastava et al., 2015] Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00387, 2015.","title":"Highway networks","authors":"Rupesh Kumar Srivastava, Klaus Greff, Jürgen Schmidhuber","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In CVPR, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Szegedy et al., 2017] Christian Szegedy, Sergey Ioffe, Vin- cent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In AAAI, 2017.","title":"Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning","authors":"Szegedy","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Veit et al., 2016] Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of rel- atively shallow networks. In NIPS, 2016.","title":"Residual networks behave like ensembles of relatively shallow networks","authors":"Andreas Veit, Michael J Wilber, Serge Belongie","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xie et al., 2017] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In CVPR, 2017.","title":"Aggregated residual transformations for deep neural networks","authors":"Xie","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zagoruyko and Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.","title":"Wide residual networks","authors":"Sergey Zagoruyko, Nikos Komodakis","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zeiler and Fergus, 2014] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.","title":"Visualizing and understanding convolutional networks","authors":"D Matthew, Rob Zeiler, Fergus","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2017] Xiangyu Zhang, Xinyu Zhou, Mengx- iao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. arXiv preprint arXiv:1707.01083, 2017.","title":"Shufflenet: An extremely efficient convolutional neural network for mobile devices","authors":"Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, Jian Sun","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}