{"type":"ScholarlyArticle","title":"Sharing Residual Units Through Collective Tensor Factorization To Improve Deep Neural Networks","authors":[{"name":"Yunpeng Chen","slug":"yunpeng-chen","given":"Yunpeng","family":"Chen","affiliation":"National University of Singapore"},{"name":"Xiaojie Jin","slug":"xiaojie-jin","given":"Xiaojie","family":"Jin","affiliation":"National University of Singapore"},{"name":"Bingyi Kang","slug":"bingyi-kang","given":"Bingyi","family":"Kang","affiliation":"National University of Singapore; AI Institute"},{"name":"Jiashi Feng","slug":"jiashi-feng","given":"Jiashi","family":"Feng","affiliation":"National University of Singapore"},{"name":"Shuicheng Yan","slug":"shuicheng-yan","given":"Shuicheng","family":"Yan","affiliation":"Qihoo 360; National University of Singapore"}],"abstract":"The residual unit and its variations are wildly used in building very deep neural networks for alleviating optimization difficulty. In this work, we revisit the standard residual function as well as its several successful variants and propose a unified framework based on tensor Block Term Decomposition (BTD) to explain these apparently different residual functions from the tensor decomposition view. With the BTD framework, we further propose a novel basic network architecture, named the Collective Residual Unit (CRU). CRU further enhances parameter efficiency of deep residual neural networks by sharing core factors derived from collective tensor factorization over the involved residual units. It enables efficient knowledge sharing across multiple residual units, reduces the number of model parameters, lowers the risk of over-fitting, and provides better generalization ability. Extensive experimental results show that our proposed CRU network brings outstanding parameter efficiency -- it achieves comparable classification performance with ResNet-200 while using a model size as small as ResNet-50 on the ImageNet-1k and Places365-Standard benchmark datasets.","doi":"10.24963/ijcai.2018/88","url":"https://ijcai.org/proceedings/2018/88","pdf":"https://ijcai.org/proceedings/2018/0088.pdf","markdown":"https://ijcai.org/proceedings/2018/88.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/88","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":88,"pages":{"first":635,"last":641},"keywords":["Computer Vision: 2D and 3D Computer Vision","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2015] Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. Mxnet: A flexible and efficient machine learn- ing library for heterogeneous distributed systems. arXiv preprint arXiv:1512.01274, 2015.","title":"Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems","authors":"Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, Zheng Zhang","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cohen et al., 2016] Nadav Cohen, Or Sharir, and Amnon Shashua. Deep simnets. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pages 4782-4791, 2016.","title":"Deep simnets","authors":"Nadav Cohen, Or Sharir, Amnon Shashua","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[De Lathauwer, 2008] Lieven De Lathauwer. Decompositions of a higher-order tensor in block terms-part ii: Definitions and uniqueness. SIAM Journal on Matrix Analysis and Applications, 30(3):1033-34, 2008.","title":"Decompositions of a higher-order tensor in block terms-part ii: Definitions and uniqueness","authors":"Lieven De, Lathauwer","year":"2008","venue":"SIAM Journal on Matrix Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Eigen et al., 2013] David Eigen, Jason Rolfe, Rob Fergus, and Yann LeCun. Understanding deep architectures using a recursive convolutional network. arXiv preprint arXiv:1312.1847, 2013.","title":"Understanding deep architectures using a recursive convolutional network","authors":"David Eigen, Jason Rolfe, Rob Fergus, Yann Lecun","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Garipov et al., 2016] Timur Garipov, Dmitry Podoprikhin, Alexander Novikov, and Dmitry Vetrov. Ultimate tensorization: compressing convolutional and fc layers alike. arXiv preprint arXiv:1611.03214, 2016.","title":"Ultimate tensorization: compressing convolutional and fc layers alike","authors":"Timur Garipov, Dmitry Podoprikhin, Alexander Novikov, Dmitry Vetrov","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016a] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"He","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016b] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Euro- pean Conference on Computer Vision, pages 630-645. Springer, 2016.","title":"Identity mappings in deep residual networks","authors":"He","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by re- ducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.","title":"Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jaderberg et al., 2014] Max Jaderberg, Andrea Vedaldi, and An- drew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866, 2014.","title":"Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions","authors":"Jaderberg","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kolda and Bader, 2009] Tamara G Kolda and Brett W Bader. Ten- sor decompositions and applications. SIAM review, 51(3):455- 500, 2009.","title":"Tensor decompositions and applications","authors":"G Tamara, Brett W Kolda, Bader","year":"2009","venue":"SIAM review","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Ge- offrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lebedev et al., 2014] Vadim Lebedev, Yaroslav Ganin, Mak- sim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. arXiv preprint arXiv:1412.6553, 2014.","title":"Speeding-up convolutional neural networks using fine-tuned cp-decomposition","authors":"Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, Victor Lempitsky","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2016] Yi Li, Kaiming He, Jian Sun, et al. R-fcn: Ob- ject detection via region-based fully convolutional networks. In Advances in Neural Information Processing Systems, pages 379- 387, 2016.","title":"R-fcn: Object detection via region-based fully convolutional networks","authors":"Yi Li, Kaiming He, Jian Sun","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liao and Poggio, 2016] Qianli Liao and Tomaso Poggio. Bridging the gaps between residual learning, recurrent neural networks and visual cortex. arXiv preprint arXiv:1604.03640, 2016.","title":"Bridging the gaps between residual learning, recurrent neural networks and visual cortex","authors":"Poggio Liao, Qianli Liao, Tomaso Poggio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Oseledets, 2011] Ivan V Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):2295-2317, 2011.","title":"Tensor-train decomposition","authors":"V Ivan, Oseledets","year":"2011","venue":"SIAM Journal on Scientific Computing","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91-99, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rigamonti et al., 2013] Roberto Rigamonti, Amos Sironi, Vincent Lepetit, and Pascal Fua. Learning separable filters. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, pages 2754-2761. IEEE, 2013.","title":"Learning separable filters","authors":"Roberto Rigamonti, Amos Sironi, Vincent Lepetit, Pascal Fua","year":"2013","venue":"Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexan- der C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vi- sion (IJCV), 115(3):211-252, 2015.","title":"ImageNet Large Scale Visual Recognition Challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, Li Fei-Fei","year":"2015","venue":"International Journal of Computer Vision (IJCV)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Singh and Gordon, 2008] Ajit P Singh and Geoffrey J Gordon. Relational learning via collective matrix factorization. In Pro- ceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 650-658. ACM, 2008.","title":"Relational learning via collective matrix factorization","authors":"P Ajit, Geoffrey J Singh, Gordon","year":"2008","venue":"Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tucker, 1966] Ledyard R Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3):279-311, 1966.","title":"Some mathematical notes on three-mode factor analysis","authors":"R Ledyard, Tucker","year":"1966","venue":"Psychometrika","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Veit et al., 2016] Andreas Veit, Michael J Wilber, and Serge Be- longie. Residual networks behave like ensembles of relatively shallow networks. In Advances in Neural Information Process- ing Systems, pages 550-558, 2016.","title":"Residual networks behave like ensembles of relatively shallow networks","authors":"Andreas Veit, Michael J Wilber, Serge Belongie","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang and Cheng, 2016] Peisong Wang and Jian Cheng. Acceler- ating convolutional neural networks for mobile applications. In Proceedings of the 2016 ACM on Multimedia Conference, pages 541-545. ACM, 2016.","title":"Accelerating convolutional neural networks for mobile applications","authors":"Wang, Cheng ; ] Peisong Wang, Jian Cheng","year":"2016","venue":"Proceedings of the 2016 ACM on Multimedia Conference","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2017] Qi Wang, Junyu Gao, and Yuan Yuan. A joint convolutional neural networks and context transfer for street scenes labeling. IEEE Transactions on Intelligent Transportation Systems, 2017.","title":"A joint convolutional neural networks and context transfer for street scenes labeling","authors":"Wang","year":"2017","venue":"IEEE Transactions on Intelligent Transportation Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2018] Qi Wang, Jia Wan, and Yuan Yuan. Locality constraint distance metric learning for traffic congestion detec- tion. Pattern Recognition, 75:272-281, 2018.","title":"Locality constraint distance metric learning for traffic congestion detection","authors":"Qi Wang, Jia Wan, Yuan Yuan","year":"2018","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wu et al., 2016] Zifeng Wu, Chunhua Shen, and Anton van den Hengel. Wider or deeper: Revisiting the resnet model for visual recognition. arXiv preprint arXiv:1611.10080, 2016.","title":"Wider or deeper: Revisiting the resnet model for visual recognition","authors":"Wu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xie et al., 2016] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. arXiv preprint arXiv:1611.05431, 2016.","title":"Aggregated residual transformations for deep neural networks","authors":"Xie","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zagoruyko and Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.","title":"Wide residual networks","authors":"Sergey Zagoruyko, Nikos Komodakis","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhao et al., 2016] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. arXiv preprint arXiv:1612.01105, 2016.","title":"Pyramid scene parsing network","authors":"Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, Jiaya Jia","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhou et al., 2016] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Antonio Torralba, and Aude Oliva. Places: An image database for deep scene understanding. arXiv preprint arXiv:1610.02055, 2016.","title":"Places: An image database for deep scene understanding","authors":"Zhou","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}