{"type":"ScholarlyArticle","title":"CP-NAS: Child-Parent Neural Architecture Search for 1-bit CNNs","authors":[{"name":"Li'an Zhuo","slug":"li-an-zhuo","given":"Li'an","family":"Zhuo","affiliation":"School of Automation Science and Electrical Engineering , Beihang University"},{"name":"Baochang Zhang","slug":"baochang-zhang","given":"Baochang","family":"Zhang","affiliation":"School of Automation Science and Electrical Engineering , Beihang University"},{"name":"Hanlin Chen","slug":"hanlin-chen","given":"Hanlin","family":"Chen","affiliation":"School of Automation Science and Electrical Engineering , Beihang University"},{"name":"Linlin Yang","slug":"linlin-yang","given":"Linlin","family":"Yang","affiliation":"University of Bonn"},{"name":"Chen Chen","slug":"chen-chen","given":"Chen","family":"Chen","affiliation":"University of North Carolina at Charlotte"},{"name":"Yanjun Zhu","slug":"yanjun-zhu","given":"Yanjun","family":"Zhu","affiliation":"University at Buffalo"},{"name":"David Doermann","slug":"david-doermann","given":"David","family":"Doermann","affiliation":"University at Buffalo"}],"abstract":"Neural architecture search (NAS) proves to be among the best approaches for many tasks by generating an application-adaptive neural architectures, which are still challenged by high computational cost and memory consumption. At the same time, 1-bit convolutional neural networks (CNNs) with binarized weights and activations show their potential for resource-limited embedded devices. One natural approach is to use 1-bit CNNs to reduce the computation and memory cost of NAS by taking advantage of the strengths of each in a unified framework. To this end, a Child-Parent model is introduced to a differentiable NAS to search the binarized architecture(Child) under the supervision of a full-precision model (Parent). In the search stage, the Child-Parent model uses an indicator generated by the parent and child model accuracy to evaluate the performance and abandon operations with less potential. In the training stage, a kernel level CP loss is introduced to optimize the binarized network. Extensive experiments demonstrate that the proposed CP-NAS achieves a comparable accuracy with traditional NAS on both the CIFAR and ImageNet databases. It achieves an accuracy of 95.27% on CIFAR-10, 64.3% on ImageNet with binarized weights and activations, and a 30% faster search than prior arts.","doi":"10.24963/ijcai.2020/144","url":"https://ijcai.org/proceedings/2020/144","pdf":"https://ijcai.org/proceedings/2020/0144.pdf","markdown":"https://ijcai.org/proceedings/2020/144.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/144","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":144,"pages":{"first":1033,"last":1039},"keywords":["Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Machine Learning: Deep Learning: Convolutional networks"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24793/file_24793.m3u8","length_sec":296,"provider":"hls"}],"references":[{"position":1,"raw":"[Cai et al., 2018a] Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In Proc. of AAAI, pages 2787- 2794, 2018.","title":"Efficient architecture search by network transformation","authors":"Cai","year":"2018","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai et al., 2018b] Han Cai, Jiacheng Yang, Weinan Zhang, Song Han, and Yong Yu. Path-level network transforma- tion for efficient architecture search. arXiv, 2018.","title":"Path-level network transformation for efficient architecture search","authors":"Cai","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cai et al., 2018c] Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv, 2018.","title":"Proxylessnas: Direct neural architecture search on target task and hardware","authors":"Cai","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2019a] Hanlin Chen, Li'an Zhuo, Baochang Zhang, Xiawu Zheng, Jianzhuang Liu, David Doermann, and Rongrong Ji. Binarized neural architecture search. arXiv, 2019.","title":"Binarized neural architecture search","authors":"Chen","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2019b] Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proc. of ICCV, October 2019.","title":"Progressive differentiable architecture search: Bridging the depth gap between search and evaluation","authors":"Chen","year":"2019","venue":"Proc. of ICCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Courbariaux et al., 2015] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propaga- tions. In Proc. of NIPS, pages 3123-3131, 2015.","title":"Binaryconnect: Training deep neural networks with binary weights during propagations","authors":"Matthieu Courbariaux, Yoshua Bengio, Jean-Pierre David","year":"2015","venue":"Proc. of NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Courbariaux et al., 2016] Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Ben- gio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv, 2016.","title":"Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1","authors":"Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[DeVries and Taylor, 2017] Terrance DeVries and Gra- ham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv, 2017.","title":"Improved regularization of convolutional neural networks with cutout","authors":"Terrance Devries, Graham W Taylor","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gu et al., 2019] Jiaxin Gu, Ce Li, Baochang Zhang, Jun- gong Han, Xianbin Cao, Jianzhuang Liu, and David Do- ermann. Projection convolutional neural networks for 1- bit cnns via discrete back propagation. In Proc. of AAAI, pages 8344-8351, 2019.","title":"Projection convolutional neural networks for 1bit cnns via discrete back propagation","authors":"Jiaxin Gu, Ce Li, Baochang Zhang, Jungong Han, Xianbin Cao, Jianzhuang Liu, David Doermann","year":"2019","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. Computer Science, 14(7):38-39, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":"Computer Science","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Juefei-Xu et al., 2017] Felix Juefei-Xu, Vishnu Naresh Boddeti, and Marios Savvides. Local binary convolutional neural networks. In Proc. of CVPR, pages 19-28, 2017.","title":"Local binary convolutional neural networks","authors":"Juefei-Xu","year":"2017","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky et al., 2014] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 dataset. online: http://www. cs. toronto. edu/kriz/cifar. html, 2014.","title":"The cifar-10 dataset","authors":"Alex Krizhevsky, Vinod Nair, Geoffrey Hinton","year":null,"venue":null,"doi":null,"url":"http://www.cs.toronto.edu/kriz/cifar.html,2014","internal_url":null},{"position":13,"raw":"[Liu et al., 2018a] Chenxi Liu, Barret Zoph, Maxim Neu- mann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progres- sive neural architecture search. In Proc. of ECCV, pages 19-34, 2018.","title":"Progressive neural architecture search","authors":"Liu","year":"2018","venue":"Proc. of ECCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2018b] Hanxiao Liu, Karen Simonyan, and Yim- ing Yang. Darts: Differentiable architecture search. arXiv, 2018.","title":"Darts: Differentiable architecture search","authors":"Liu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2018c] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with im- proved representational capability and advanced training algorithm. In Proc. of ECCV, pages 747-763. Springer, 2018.","title":"Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm","authors":"Liu","year":"2018","venue":"Proc. of ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Paszke et al., 2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De- Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In Proc. of NIPS, 2017.","title":"Automatic differentiation in pytorch","authors":"Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De-Vito, Zeming Lin, Alban Desmaison, Luca Antiga, Adam Lerer","year":"2017","venue":"Proc. of NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pham et al., 2018] Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. Efficient neural archi- tecture search via parameter sharing. arXiv, 2018.","title":"Efficient neural architecture search via parameter sharing","authors":"Hieu Pham, Melody Y Guan, Barret Zoph, Jeff Quoc V Le, Dean","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rastegari et al., 2016] Mohammad Rastegari, Vicente Or- donez, Joseph Redmon, and Ali Farhadi. Xnor-net: Ima- genet classification using binary convolutional neural net- works. In Proc. of ECCV, pages 525-542, 2016.","title":"Xnor-net: Imagenet classification using binary convolutional neural networks","authors":"Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, Ali Farhadi","year":"2016","venue":"Proc. of ECCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Real et al., 2019] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 4780-4789, 2019.","title":"Regularized evolution for image classifier architecture search","authors":"Esteban Real, Alok Aggarwal, Yanping Huang, Quoc V Le","year":"2019","venue":"Proceedings of the aaai conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recogni- tion challenge. International journal of computer vision, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein","year":"2015","venue":"International journal of computer vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tan and Le, 2019] Mingxing Tan and Quoc V Le. Efficient- net: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946, 2019.","title":"Efficientnet: Rethinking model scaling for convolutional neural networks","authors":"Mingxing Tan, V Quoc, Le","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu et al., 2019] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Partial channel connections for memory-efficient differen- tiable architecture search. arXiv, 2019.","title":"Partial channel connections for memory-efficient differentiable architecture search","authors":"Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, Hongkai Xiong","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zagoruyko and Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proc. of BMVC, 2016.","title":"Wide residual networks","authors":"Sergey Zagoruyko, Nikos Komodakis","year":"2016","venue":"Proc. of BMVC","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhao et al., 2019] Junhe Zhao, Jiaxin Gu, Xiaolong Jiang, Baochang Zhang, Liu Jianzhuang, Guodong Guo, and Rongrong Ji. Bayesian optimized 1-bit cnns. In Proc. of ICCV, 2019.","title":"Bayesian optimized 1-bit cnns","authors":"Junhe Zhao, Jiaxin Gu, Xiaolong Jiang, Baochang Zhang, Liu Jianzhuang, Guodong Guo, Rongrong Ji","year":"2019","venue":"Proc. of ICCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zheng et al., 2019] Xiawu Zheng, Rongrong Ji, Lang Tang, Baochang Zhang, Jianzhuang Liu, and Qi Tian. Multino- mial distribution learning for effective neural architecture search. In Proc. of ICCV, October 2019.","title":"Multinomial distribution learning for effective neural architecture search","authors":"Zheng","year":"2019","venue":"Proc. of ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zoph and Le, 2016] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv, 2016.","title":"Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning","authors":"Le Zoph","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zoph et al., 2018] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architec- tures for scalable image recognition. In Proc. of CVPR, pages 8697-8710, 2018.","title":"Learning transferable architectures for scalable image recognition","authors":"Barret Zoph, Vijay Vasudevan, Jonathon Shlens, V Quoc, Le","year":"2018","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}