{"type":"ScholarlyArticle","title":"Random Shifting for CNN: a Solution to Reduce Information Loss in Down-Sampling Layers","authors":[{"name":"Gangming Zhao","slug":"gangming-zhao","given":"Gangming","family":"Zhao","affiliation":"Research Center for Brain-inspired Intelligence , CASIA; University of Chinese Academy of Sciences"},{"name":"Jingdong Wang","slug":"jingdong-wang","given":"Jingdong","family":"Wang","affiliation":"Microsoft Research"},{"name":"Zhaoxiang Zhang","slug":"zhaoxiang-zhang","given":"Zhaoxiang","family":"Zhang","affiliation":"Research Center for Brain-inspired Intelligence , CASIA; National Laboratory of Pattern Recognition , CASIA; CAS Center for Excellence in Brain Science and Intelligence Technology; University of Chinese Academy of Sciences"}],"abstract":"Down-sampling is widely adopted in deep convolutional neural networks (DCNN) for reducing the number of network parameters while preserving the transformation invariance. However, it cannot utilize information effectively because it only adopts a fixed stride strategy, which may result in poor generalization ability and information loss. In this paper, we propose a novel random strategy to alleviate these problems by embedding random shifting in the down-sampling layers during the training process. Random shifting can be universally applied to diverse DCNN models to dynamically adjust receptive fields by shifting kernel centers on feature maps in different directions. Thus, it can generate more robust features in networks and further enhance the transformation invariance of down-sampling operators. In addition, random shifting cannot only be integrated in all down-sampling layers including strided convolutional layers and pooling layers, but also improve performance of DCNN with negligible additional computational cost. We evaluate our method in different tasks (e.g., image classification and segmentation) with various network architectures (i.e., AlexNet, FCN and DFN-MR). Experimental results demonstrate the effectiveness of our proposed method.","doi":"10.24963/ijcai.2017/486","url":"https://ijcai.org/proceedings/2017/486","pdf":"https://ijcai.org/proceedings/2017/0486.pdf","markdown":"https://ijcai.org/proceedings/2017/486.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/486","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":486,"pages":{"first":3476,"last":3482},"keywords":["Machine Learning: Machine Learning","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Berg et al., 2010] Alex Berg, Jia Deng, and L Fei-Fei. Large scale visual recognition challenge 2010, 2010.","title":"Large scale visual recognition challenge","authors":"Alex Berg, Jia Deng, L Fei-Fei","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2014] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Se- mantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint arXiv:1412.7062, 2014.","title":"Semantic image segmentation with deep convolutional nets and fully connected crfs","authors":"Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L Yuille","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2015] Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv:1511.05641, 2015.","title":"Net2net: Accelerating learning via knowledge transfer","authors":"Tianqi Chen, Ian Goodfellow, Jonathon Shlens","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Girshick et al., 2014] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580-587, 2014.","title":"Rich feature hierarchies for accurate object detection and semantic segmentation","authors":"Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik","year":"2014","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Glorot and Bengio, 2010] Xavier Glorot and Yoshua Ben- gio. Understanding the difficulty of training deep feed- forward neural networks. In Aistats, volume 9, pages 249- 256, 2010.","title":"Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks","authors":"Bengio Glorot","year":"2010","venue":"Aistats","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.","title":"Delving deep into rectifiers: Surpassing human-level performance on imagenet classification","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2016] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European Conference on Com- puter Vision, pages 646-661. Springer, 2016.","title":"Deep networks with stochastic depth","authors":"Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, Kilian Q Weinberger","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jia et al., 2014] Yangqing Jia, Evan Shelhamer, Jeff Don- ahue, Sergey Karayev, Jonathan Long, Ross Girshick, Ser- gio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia, pages 675-678. ACM, 2014.","title":"Caffe: Convolutional architecture for fast feature embedding","authors":"Jia","year":"2014","venue":"Proceedings of the 22nd ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural in- formation processing systems, pages 1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin et al., 2014] Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. Computer Science, 2014.","title":"Network in network","authors":"Lin","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2016] Ce Liu, Jenny Yuen, and Antonio Torralba. Sift flow: Dense correspondence across scenes and its ap- plications. In Dense Image Correspondences for Com- puter Vision, pages 15-49. Springer, 2016.","title":"Sift flow: Dense correspondence across scenes and its applications","authors":"Liu","year":"2016","venue":"Dense Image Correspondences for Computer Vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3431- 3440, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Jonathan Long, Evan Shelhamer, Trevor Darrell","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Oquab et al., 2014] Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid- level image representations using convolutional neural net- works. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 1717-1724, 2014.","title":"Learning and transferring midlevel image representations using convolutional neural networks","authors":"Maxime Oquab, Leon Bottou, Ivan Laptev, Josef Sivic","year":"2014","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Romero et al., 2014] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.","title":"Fitnets: Hints for thin deep nets","authors":"Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Springenberg et al., 2014] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Ried- miller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806, 2014.","title":"Striving for simplicity: The all convolutional net","authors":"Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, Martin Riedmiller","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey E Hin- ton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1-9, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wan et al., 2013] Li Wan, Matthew Zeiler, Sixin Zhang, Yann L Cun, and Rob Fergus. Regularization of neural net- works using dropconnect. In Proceedings of the 30th In- ternational Conference on Machine Learning (ICML-13), pages 1058-1066, 2013.","title":"Regularization of neural networks using dropconnect","authors":"Li Wan, Matthew Zeiler, Sixin Zhang, Rob Yann L Cun, Fergus","year":"2013","venue":"Proceedings of the 30th International Conference on Machine Learning (ICML-13)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xie and Tu, 2015] Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In Proceedings of the IEEE International Conference on Computer Vision, pages 1395-1403, 2015.","title":"Holistically-nested edge detection","authors":"Tu ; Xie, Saining Xie, Zhuowen Tu","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhao et al., 2016] Liming Zhao, Jingdong Wang, Xi Li, Zhuowen Tu, and Wenjun Zeng. On the connection of deep fusion to ensembling. 2016.","title":"On the connection of deep fusion to ensembling","authors":"Liming Zhao, Jingdong Wang, Xi Li, Zhuowen Tu, Wenjun Zeng","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}