{"type":"ScholarlyArticle","title":"DropNAS: Grouped Operation Dropout for Differentiable Architecture Search","authors":[{"name":"Weijun Hong","slug":"weijun-hong","given":"Weijun","family":"Hong","affiliation":"Shanghai Jiao Tong University , China"},{"name":"Guilin Li","slug":"guilin-li","given":"Guilin","family":"Li","affiliation":"Huawei Noah's Ark Lab , China"},{"name":"Weinan Zhang","slug":"weinan-zhang","given":"Weinan","family":"Zhang","affiliation":"Shanghai Jiao Tong University , China"},{"name":"Ruiming Tang","slug":"ruiming-tang","given":"Ruiming","family":"TANG","affiliation":"Huawei Noah's Ark Lab , China"},{"name":"Yunhe Wang","slug":"yunhe-wang","given":"Yunhe","family":"Wang","affiliation":"Huawei Noah's Ark Lab , China"},{"name":"Zhenguo Li","slug":"zhenguo-li","given":"Zhenguo","family":"Li","affiliation":"Huawei Noah's Ark Lab , China"},{"name":"Yong Yu","slug":"yong-yu","given":"Yong","family":"Yu","affiliation":"Shanghai Jiao Tong University , China"}],"abstract":"Neural architecture search (NAS) has shown encouraging results in automating the architecture design. Recently, DARTS relaxes the search process with a   differentiable formulation that leverages weight-sharing and SGD for cost reduction of NAS. In DARTS, all candidate operations are trained simultaneously during the network weight training step. Our empirical results show that this training procedure leads to the co-adaption problem and Matthew Effect: operations with fewer parameters would be trained maturely earlier.  This causes two problems: firstly, the operations with more parameters may never have the chance to express the desired function since those with less have already done the job; secondly, the system will punish those underperforming operations by lowering their architecture parameter and backward smaller loss gradients, this causes the Matthew Effect.   In this paper,  we systematically study these problems and propose a novel grouped operation dropout algorithm named DropNAS to fix the problems with DARTS.    Extensive experiments demonstrate that DropNAS solves the above issues and achieves promising performance. Specifically, DropNAS achieves 2.26% test error on CIFAR-10, 16.39% on CIFAR-100 and 23.4% on ImageNet (with the same training hyperparameters as DARTS for a fair comparison). It is also observed that  DropNAS is robust across variants of the DARTS search space. Code is available at https://github.com/huawei-noah.","doi":"10.24963/ijcai.2020/322","url":"https://ijcai.org/proceedings/2020/322","pdf":"https://ijcai.org/proceedings/2020/0322.pdf","markdown":"https://ijcai.org/proceedings/2020/322.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/322","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":322,"pages":{"first":2326,"last":2332},"keywords":["Machine Learning: Deep Learning","Computer Vision: Other"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25595/file_25595.m3u8","length_sec":278,"provider":"hls"}],"references":[{"position":1,"raw":"[Bender et al., 2018] Gabriel Bender, Pieter-Jan Kinder- mans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Un- derstanding and simplifying one-shot architecture search. In ICML, pages 549-558, 2018.","title":"Understanding and simplifying one-shot architecture search","authors":"Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, Quoc Le","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai et al., 2019] Han Cai, Ligeng Zhu, and Song Han. Prox- ylessNAS: Direct neural architecture search on target task and hardware. In ICLR, 2019.","title":"Prox-ylessNAS: Direct neural architecture search on target task and hardware","authors":"Han Cai, Ligeng Zhu, Song Han","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2019a] Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student net- works. In CVPR, pages 3514-3522, 2019.","title":"Data-free learning of student networks","authors":"Chen","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2019b] Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In ICCV, pages 1294-1303, 2019.","title":"Progressive differentiable architecture search: Bridging the depth gap between search and evaluation","authors":"Chen","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chu et al., 2019] Xiangxiang Chu, Bo Zhang, Jixiang Li, Qingyuan Li, and Ruijun Xu. Scarletnas: Bridging the gap between scalability and fairness in neural architecture search. arXiv preprint arXiv:1908.06022, 2019.","title":"Scarletnas: Bridging the gap between scalability and fairness in neural architecture search","authors":"Xiangxiang Chu, Bo Zhang, Jixiang Li, Qingyuan Li, Ruijun Xu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Guo et al., 2019] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uni- form sampling. arXiv preprint arXiv:1904.00420, 2019.","title":"Single path one-shot neural architecture search with uniform sampling","authors":"Guo","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2020] Chaoyang He, Haishan Ye, Li Shen, and Tong Zhang. Milenas: Efficient neural architecture search via mixed-level reformulation. In CVPR, 2020.","title":"Milenas: Efficient neural architecture search via mixed-level reformulation","authors":"He","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural in- formation processing systems, pages 1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2019] Guilin Li, Xing Zhang, Zitong Wang, Zhen- guo Li, and Tong Zhang. Stacnas: Towards stable and consistent optimization for differentiable neural architec- ture search. arXiv preprint arXiv:1909.11926, 2019.","title":"Stacnas: Towards stable and consistent optimization for differentiable neural architecture search","authors":"Guilin Li, Xing Zhang, Zitong Wang, Zhenguo Li, Tong Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liang et al., 2019] Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. Darts+: Improved differentiable archi- tecture search with early stopping. arXiv preprint arXiv:1909.06035, 2019.","title":"Darts+: Improved differentiable architecture search with early stopping","authors":"Liang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2019] Hanxiao Liu, Karen Simonyan, and Yim- ing Yang. DARTS: Differentiable architecture search. In ICLR, 2019.","title":"DARTS: Differentiable architecture search","authors":"Liu","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2020] Bin Liu, Chenxu Zhu, Guilin Li, Weinan Zhang, Jincai Lai, Ruiming Tang, Xiuqiang He, Zhen- guo Li, and Yong Yu. Autofis: Automatic feature inter- action selection in factorization models for click-through rate prediction. arXiv preprint arXiv:2003.11235, 2020.","title":"Autofis: Automatic feature interaction selection in factorization models for click-through rate prediction","authors":"Bin Liu, Chenxu Zhu, Guilin Li, Weinan Zhang, Jincai Lai, Ruiming Tang, Xiuqiang He, Zhenguo Li, Yong Yu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Nayman et al., 2019] Niv Nayman, Asaf Noy, Tal Ridnik, Itamar Friedman, Rong Jin, and Lihi Zelnik. Xnas: Neu- ral architecture search with expert advice. In Advances in Neural Information Processing Systems, pages 1975- 1985, 2019.","title":"Xnas: Neural architecture search with expert advice","authors":"Nayman","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Noy et al., 2019] Asaf Noy, Niv Nayman, Tal Ridnik, Na- dav Zamir, Sivan Doveh, Itamar Friedman, Raja Giryes, and Lihi Zelnik-Manor. Asap: Architecture search, anneal and prune. arXiv preprint arXiv:1904.04123, 2019.","title":"Asap: Architecture search, anneal and prune","authors":"Asaf Noy, Niv Nayman, Tal Ridnik, Nadav Zamir, Sivan Doveh, Itamar Friedman, Raja Giryes, Lihi Zelnik-Manor","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pham et al., 2018] Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In ICML, pages 4095- 4104, 2018.","title":"Efficient neural architecture search via parameters sharing","authors":"Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, Jeff Dean","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Real et al., 2019] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4780-4789, 2019.","title":"Regularized evolution for image classifier architecture search","authors":"Esteban Real, Alok Aggarwal, Yanping Huang, Quoc V Le","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017.","title":"Mastering the game of go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Stamoulis et al., 2019] Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Single-path nas: Designing hardware-efficient convnets in less than 4 hours. arXiv preprint arXiv:1904.02877, 2019.","title":"Single-path nas: Designing hardware-efficient convnets in less than 4 hours","authors":"Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, Diana Marculescu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tan and Le, 2019] Mingxing Tan and Quoc Le. Efficient- net: Rethinking model scaling for convolutional neural networks. In ICML, pages 6105-6114, 2019.","title":"Efficientnet: Rethinking model scaling for convolutional neural networks","authors":"Le ; Tan, Mingxing Tan, Quoc Le","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xie et al., 2019] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In International Conference on Learning Repre- sentations, 2019.","title":"SNAS: stochastic neural architecture search","authors":"Sirui Xie, Hehui Zheng, Chunxiao Liu, Liang Lin","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xu et al., 2020] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pc- darts: Partial channel connections for memory-efficient ar- chitecture search. In International Conference on Learn- ing Representations, 2020.","title":"Pcdarts: Partial channel connections for memory-efficient architecture search","authors":"Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, Hongkai Xiong","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yang et al., 2019] Zhaohui Yang, Yunhe Wang, Xinghao Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, and Chang Xu. Cars: Continuous evolution for efficient neu- ral architecture search. arXiv preprint arXiv:1909.04977, 2019.","title":"Cars: Continuous evolution for efficient neural architecture search","authors":"Zhaohui Yang, Yunhe Wang, Xinghao Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, Chang Xu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zela et al., 2020] Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hut- ter. Understanding and robustifying differentiable archi- tecture search. In ICLR, 2020.","title":"Understanding and robustifying differentiable architecture search","authors":"Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, Frank Hutter","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zoph and Le, 2017] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. In ICLR, 2017.","title":"Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning","authors":"Le Zoph","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zoph et al., 2018] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architec- tures for scalable image recognition. In CVPR, pages 8697-8710, 2018.","title":"Learning transferable architectures for scalable image recognition","authors":"Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V Le","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}