{"type":"ScholarlyArticle","title":"Fully Nested Neural Network for Adaptive Compression and Quantization","authors":[{"name":"Yufei Cui","slug":"yufei-cui","given":"Yufei","family":"Cui","affiliation":"Department of Computer Science , City University of Hong Kong"},{"name":"Ziquan Liu","slug":"ziquan-liu","given":"Ziquan","family":"Liu","affiliation":"Department of Computer Science , City University of Hong Kong"},{"name":"Wuguannan Yao","slug":"wuguannan-yao","given":"Wuguannan","family":"Yao","affiliation":"Department of Mathematics , City University of Hong Kong"},{"name":"Qiao Li","slug":"qiao-li","given":"Qiao","family":"Li","affiliation":"Department of Computer Science , City University of Hong Kong"},{"name":"Antoni B. Chan","slug":"antoni-b-chan","given":"Antoni B.","family":"Chan","affiliation":"Department of Computer Science , City University of Hong Kong"},{"name":"Tei-wei Kuo","slug":"tei-wei-kuo","given":"Tei-Wei","family":"Kuo","affiliation":"Department of Computer Science , City University of Hong Kong; Department of Computer Science & Information Engineering , National Taiwan University {yufeicui3-c ,"},{"name":"Chun Jason Xue","slug":"chun-jason-xue","given":"Chun Jason","family":"Xue","affiliation":"Department of Computer Science , City University of Hong Kong"}],"abstract":"Neural network compression and quantization are important tasks for fitting state-of-the-art models into the computational, memory and power constraints of mobile devices and embedded hardware. Recent approaches to model compression/quantization are based on reinforcement learning or search methods to quantize the neural network for a specific hardware platform. However, these methods require multiple runs to compress/quantize the same base neural network to different hardware setups. In this work, we propose a fully nested neural network (FN3) that runs only once to build a nested set of compressed/quantized models, which is optimal for different resource constraints. Specifically, we exploit the additive characteristic in different levels of building blocks in neural network and propose an ordered dropout (ODO) operation that ranks the building blocks. Given a trained FN3, a fast heuristic search algorithm is run offline to find the optimal removal of components to maximize the accuracy under different constraints. Compared with the related works on adaptive neural network designed only for channels or bits, the proposed approach is applicable to different levels of building blocks (bits, neurons, channels, residual paths and layers). Empirical results validate strong practical performance of proposed approach.","doi":"10.24963/ijcai.2020/288","url":"https://ijcai.org/proceedings/2020/288","pdf":"https://ijcai.org/proceedings/2020/0288.pdf","markdown":"https://ijcai.org/proceedings/2020/288.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/288","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":288,"pages":{"first":2080,"last":2087},"keywords":["Machine Learning: Deep Learning","Machine Learning: Deep Learning: Convolutional networks"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24731/file_24731.m3u8","length_sec":308,"provider":"hls"}],"references":[{"position":1,"raw":"[Baram, 1988] Yoram Baram. Nested neural networks. NASA Technical Memorandum 101032, 1988.","title":"Nested neural networks","authors":"Yoram Baram, Baram","year":"1988","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai et al., 2018] Han Cai, Ligeng Zhu, and Song Han. Prox- ylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:1812.00332, 2018.","title":"Proxylessnas: Direct neural architecture search on target task and hardware","authors":"Han Cai, Ligeng Zhu, Song Han","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen and Guestrin, 2016] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowl- edge discovery and data mining, pages 785-794. ACM, 2016.","title":"Xgboost: A scalable tree boosting system","authors":"Tianqi Chen, Carlos Guestrin","year":"2016","venue":"Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Han et al., 2016] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In International Conference on Learning Representations, 2016.","title":"Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding","authors":"Han","year":"2016","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hastie et al., 2009] Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction. Springer Science & Business Media, 2009.","title":"The elements of statistical learning: data mining, inference, and prediction","authors":"Trevor Hastie, Robert Tibshirani, Jerome Friedman","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jiang et al., 2019] Weiwen Jiang, Xinyi Zhang, Edwin H- M Sha, Lei Yang, Qingfeng Zhuge, Yiyu Shi, and Jing- tong Hu. Accuracy vs. efficiency: Achieving both through fpga-implementation aware neural architecture search. In Proceedings of the 56th Annual Design Automation Con- ference 2019, page 5. ACM, 2019.","title":"Accuracy vs. efficiency: Achieving both through fpga-implementation aware neural architecture search","authors":"Weiwen Jiang, Xinyi Zhang, H-M Edwin, Lei Sha, Qingfeng Yang, Yiyu Zhuge, Jingtong Shi, Hu","year":"2019","venue":"Proceedings of the 56th Annual Design Automation Conference 2019","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jin et al., 2019] Qing Jin, Linjie Yang, and Zhenyu Liao. Adabits: Neural network quantization with adaptive bit- widths. arXiv preprint arXiv:1912.09666, 2019.","title":"Adabits: Neural network quantization with adaptive bitwidths","authors":"Qing Jin, Linjie Yang, Zhenyu Liao","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kim et al., 2018] Eunwoo Kim, Chanho Ahn, and Songh- wai Oh. Nestednet: Learning nested sparse structures in deep neural networks. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 8669-8678, 2018.","title":"Nestednet: Learning nested sparse structures in deep neural networks","authors":"Kim","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Ge- offrey Hinton. Deep learning. nature, 521(7553):436, 2015.","title":"Deep learning","authors":"Yann Lecun, Yoshua Bengio, Geoffrey Hinton","year":"2015","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Neklyudov et al., 2017] Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, and Dmitry P Vetrov. Structured bayesian pruning via log-normal multiplicative noise. In Advances in Neural Information Processing Systems, pages 6775-6784, 2017.","title":"Structured bayesian pruning via log-normal multiplicative noise","authors":"Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, Dmitry P Vetrov","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Rippel et al., 2014] Oren Rippel, Michael Gelbart, and Ryan Adams. Learning ordered representations with nested dropout. In International Conference on Machine Learning, pages 1746-1754, 2014.","title":"Learning ordered representations with nested dropout","authors":"Oren Rippel, Michael Gelbart, Ryan Adams","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Sandler et al., 2018] Mark Sandler, Andrew Howard, Men- glong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4510-4520, 2018.","title":"Mobilenetv2: Inverted residuals and linear bottlenecks","authors":"Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Veit et al., 2016] Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of rel- atively shallow networks. In Advances in neural informa- tion processing systems, pages 550-558, 2016.","title":"Residual networks behave like ensembles of relatively shallow networks","authors":"Andreas Veit, Michael J Wilber, Serge Belongie","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Wang et al., 2019] Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quantization with mixed precision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 8612-8620, 2019.","title":"Haq: Hardware-aware automated quantization with mixed precision","authors":"Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, Song Han","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Xie et al., 2017] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 1492-1500, 2017.","title":"Aggregated residual transformations for deep neural networks","authors":"Xie","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Yang et al., 2019] Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian-sheng Hua. Quantization networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7308-7316, 2019.","title":"Quantization networks","authors":"Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, Xian-Sheng Hua","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Yu and Huang, 2019] Jiahui Yu and Thomas S Huang. Uni- versally slimmable networks and improved training tech- niques. In Proceedings of the IEEE International Confer- ence on Computer Vision, pages 1803-1811, 2019.","title":"Universally slimmable networks and improved training techniques","authors":"Yu, Huang ; ] Jiahui Yu, Thomas S Huang","year":"2019","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Yu et al., 2019a] Haichao Yu, Haoxiang Li, Honghui Shi, Thomas S Huang, and Gang Hua. Any-precision deep neu- ral networks. arXiv preprint arXiv:1911.07346, 2019.","title":"Any-precision deep neural networks","authors":"Yu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yu et al., 2019b] Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In International Conference on Learning Representations, 2019.","title":"Slimmable neural networks","authors":"Yu","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhang et al., 2018] Xiangyu Zhang, Xinyu Zhou, Mengx- iao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6848-6856, 2018.","title":"Shufflenet: An extremely efficient convolutional neural network for mobile devices","authors":"Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, Jian Sun","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhou et al., 2016] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. CoRR, abs/1606.06160, 2016.","title":"Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients","authors":"Zhou","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhou, 2012] Zhi-Hua Zhou. Ensemble methods: founda- tions and algorithms. Chapman and Hall/CRC, 2012.","title":"Ensemble methods: foundations and algorithms","authors":"Zhi-Hua Zhou","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}