{"type":"ScholarlyArticle","title":"To Fold or Not to Fold: a Necessary and Sufficient Condition on Batch-Normalization Layers Folding","authors":[{"name":"Edouard Yvinec","slug":"edouard-yvinec","given":"Edouard","family":"Yvinec","affiliation":"Sorbonne Université , CNRS , Institut des Systèmes Intelligents et de Robotique , ISIR , Paris , France"},{"name":"Arnaud Dapogny","slug":"arnaud-dapogny","given":"Arnaud","family":"Dapogny","affiliation":"Sorbonne Université , CNRS , Institut des Systèmes Intelligents et de Robotique , ISIR , Paris , France; Datakalab , 114 Boulevard Malesherbes , 75017 Paris , France"},{"name":"Kevin Bailly","slug":"kevin-bailly","given":"Kevin","family":"Bailly","affiliation":"Sorbonne Université , CNRS , Institut des Systèmes Intelligents et de Robotique , ISIR , Paris , France"}],"abstract":"Batch-Normalization (BN) layers have become fundamental components in the evermore complex deep neural network architectures. Such models require acceleration processes for deployment on edge devices. However, BN layers add computation bottlenecks due to the sequential operation processing: thus, a key, yet often overlooked component of the acceleration process is BN layers folding. In this paper, we demonstrate that the current BN folding approaches are suboptimal in terms of how many layers can be removed. We therefore provide a necessary and sufficient condition for BN folding and a corresponding optimal algorithm. The proposed approach systematically outperforms existing baselines and allows to dramatically reduce the inference time of deep neural networks.","doi":"10.24963/ijcai.2022/223","url":"https://ijcai.org/proceedings/2022/223","pdf":"https://ijcai.org/proceedings/2022/0223.pdf","markdown":"https://ijcai.org/proceedings/2022/223.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/223","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":223,"pages":{"first":1601,"last":1607},"keywords":["Machine Learning: Other","Machine Learning: Learning Sparse Models","Computer Vision: Machine Learning for Vision"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986529","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984672","length_sec":357,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chen and others, 2017] Liang-Chieh Chen et al. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 2017.","title":"Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs","authors":"Liang-Chieh Chen","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cordts and others, 2016] Marius Cordts et al. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.","title":"The cityscapes dataset for semantic urban scene understanding","authors":"Marius Cordts","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2009] J. Deng, W. Dong, et al. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009.","title":"ImageNet: A Large-Scale Hierarchical Image Database","authors":"J Deng, W Dong","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Finkelstein et al., 2019] Alexander Finkelstein, Uri Almog, and Mark Grobman. Fighting quantization bias with bias. ECV workshop CVPR, 2019.","title":"Fighting quantization bias with bias","authors":"Alexander Finkelstein, Uri Almog, Mark Grobman","year":"2019","venue":"ECV workshop CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Frankle and Carbin, 2018] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. ICLR, 2018.","title":"The lottery ticket hypothesis: Finding sparse, trainable neural networks","authors":"Carbin Frankle, Jonathan Frankle, Michael Carbin","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, et al. Deep residual learning for image recognition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2017] Kaiming He, Georgia Gkioxari, et al. Mask r-cnn. In ICCV, 2017.","title":"Kaiming He, Georgia Gkioxari, et al. Mask r-cnn","authors":"He","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. ICML, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jacob and others, 2018] Benoit Jacob et al. Quantization and training of neural networks for efficient integer- arithmetic-only inference. In CVPR, 2018.","title":"Quantization and training of neural networks for efficient integerarithmetic-only inference","authors":"Jacob Benoit","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krishnamoorthi, 2018] Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv, 2018.","title":"Quantizing deep convolutional networks for efficient inference: A whitepaper","authors":"Raghuraman Krishnamoorthi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny im- ages. Master's thesis, Department of Computer Science, University of Toronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. NeurIPS, 25, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[LeCun and others, 1989] Yann LeCun et al. Backpropaga- tion applied to handwritten zip code recognition. Neural computation, 1(4), 1989.","title":"Backpropagation applied to handwritten zip code recognition","authors":"Yann Lecun","year":"1989","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin and others, 2014] Tsung-Yi Lin et al. Microsoft coco: Common objects in context. In ECCV. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin and others, 2017] Tsung-Yi Lin et al. Feature pyramid networks for object detection. In CVPR, 2017.","title":"Feature pyramid networks for object detection","authors":"Tsung-Yi Lin","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin and others, 2020] Mingbao Lin et al. Hrank: Filter pruning using high-rank feature map. In CVPR, 2020.","title":"Hrank: Filter pruning using high-rank feature map","authors":"Lin, Others ; Mingbao Lin","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu and others, 2017] Zhuang Liu et al. Learning efficient convolutional networks through network slimming. In ICCV, 2017.","title":"Learning efficient convolutional networks through network slimming","authors":"Zhuang Liu","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Luo and others, 2019] Ping Luo et al. Switchable nor- malization for learning-to-normalize deep representation. TPAMI, 2019.","title":"Switchable normalization for learning-to-normalize deep representation","authors":"Ping Luo, Luo","year":"2019","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Martin and Mahoney, 2019] Charles H Martin and Michael W Mahoney. Traditional and heavy-tailed self regularization in neural network models. ICML, 2019.","title":"Traditional and heavy-tailed self regularization in neural network models","authors":"Martin, Mahoney ; Charles, H Martin, Michael W Mahoney","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Meller et al., 2019] Eldad Meller, Alexander Finkelstein, Uri Almog, and Mark Grobman. Same, same but differ- ent: Recovering neural network quantization error through weight factorization. In ICML, 2019.","title":"Same, same but different: Recovering neural network quantization error through weight factorization","authors":"Eldad Meller, Alexander Finkelstein, Uri Almog, Mark Grobman","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nagel et al., 2019] Markus Nagel, Mart van Baalen, et al. Data-free quantization through weight equalization and bias correction. In ICCV, 2019.","title":"Data-free quantization through weight equalization and bias correction","authors":"Markus Nagel, Mart Van Baalen","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ren and others, 2015] Shaoqing Ren et al. Faster r-cnn: To- wards real-time object detection with region proposal net- works. NeurIPS, 28, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren, Shaoqing Ren","year":"2015","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Renda et al., 2020] Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. ICLR, 2020.","title":"Comparing rewinding and fine-tuning in neural network pruning","authors":"Alex Renda, Jonathan Frankle, Michael Carbin","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI. Springer, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sandler et al., 2018] Mark Sandler, Andrew Howard, et al. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.","title":"Mobilenetv2: Inverted residuals and linear bottlenecks","authors":"Mark Sandler, Andrew Howard","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. BMVC, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Srinivas and Babu, 2015] Suraj Srinivas and R Venkatesh Babu. Data-free parameter pruning for deep neural net- works. BMVC, 2015.","title":"Data-free parameter pruning for deep neural networks","authors":"Suraj Srinivas, R Venkatesh, Babu","year":"2015","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Szegedy and others, 2016] Christian Szegedy et al. Re- thinking the inception architecture for computer vision. In CVPR, 2016.","title":"Rethinking the inception architecture for computer vision","authors":"Christian Szegedy, Szegedy","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tan and Le, 2019] Mingxing Tan and Quoc V Le. Efficient- net: Rethinking model scaling for convolutional neural networks. ICML, 2019.","title":"Efficientnet: Rethinking model scaling for convolutional neural networks","authors":"Mingxing Tan, V Quoc, Le","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tan et al., 2020] Mingxing Tan, Ruoming Pang, and Quoc V Le. Efficientdet: Scalable and efficient object de- tection. In CVPR, 2020.","title":"Efficientdet: Scalable and efficient object detection","authors":"Mingxing Tan, Ruoming Pang, Quoc V Le","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wu and He, 2018] Yuxin Wu and Kaiming He. Group nor- malization. In ECCV, 2018.","title":"Group normalization","authors":"Yuxin Wu, Kaiming He","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Ye and others, 2018] Jianbo Ye et al. Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers. ICLR, 2018.","title":"Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers","authors":"Jianbo Ye","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yvinec et al., 2021] Edouard Yvinec, Arnaud Dapogny, Matthieu Cord, and Kevin Bailly. Red: Looking for re- dundancies for data-free structured compression of deep neural networks. NeurIPS, 2021.","title":"Red: Looking for redundancies for data-free structured compression of deep neural networks","authors":"Edouard Yvinec, Arnaud Dapogny, Matthieu Cord, Kevin Bailly","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhao and others, 2017] Hengshuang Zhao et al. Pyramid scene parsing network. In CVPR, 2017.","title":"Pyramid scene parsing network","authors":"Others ; Hengshuang Zhao, Zhao","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhao and others, 2019] Ritchie Zhao et al. Improving neu- ral network quantization without retraining using outlier channel splitting. In ICML, 2019.","title":"Improving neural network quantization without retraining using outlier channel splitting","authors":"Ritchie Zhao","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhou and others, 2019] Xingyi Zhou et al. Objects as points. ICCV, 2019.","title":"Objects as points","authors":"Zhou, Zhou","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}