{"type":"ScholarlyArticle","title":"MergeNAS: Merge Operations into One for Differentiable Architecture Search","authors":[{"name":"Xiaoxing Wang","slug":"xiaoxing-wang","given":"Xiaoxing","family":"Wang","affiliation":"MoE Key Lab of Artificial Intelligence , AI Institute , Shanghai Jiao Tong University"},{"name":"Chao Xue","slug":"chao-xue","given":"Chao","family":"Xue","affiliation":"IBM Research - China"},{"name":"Junchi Yan","slug":"junchi-yan","given":"Junchi","family":"Yan","affiliation":"Department of Computer Science and Engineering , Shanghai Jiao Tong University; MoE Key Lab of Artificial Intelligence , AI Institute , Shanghai Jiao Tong University"},{"name":"Xiaokang Yang","slug":"xiaokang-yang","given":"Xiaokang","family":"Yang","affiliation":"MoE Key Lab of Artificial Intelligence , AI Institute , Shanghai Jiao Tong University"},{"name":"Yonggang Hu","slug":"yonggang-hu","given":"Yonggang","family":"Hu","affiliation":"IBM System"},{"name":"Kewei Sun","slug":"kewei-sun","given":"Kewei","family":"Sun","affiliation":"IBM Research - China"}],"abstract":"Differentiable architecture search (DARTS) has been a promising one-shot architecture search approach for its mathematical formulation and competitive results. However, besides its caused high memory utilization and a large computation requirement, many research works have shown that DARTS also often suffers notable over-fitting and thus does not work robustly for some new tasks. In this paper, we propose a one-shot neural architecture search method referred to as MergeNAS by merging different types of operations e.g. convolutions into one operation. This merge-based approach not only reduces the search cost (about half a GPU day), but also alleviates over-fitting by reducing the redundant parameters. Extensive experiments on different search space and various datasets have been conducted to verify our approach, showing that MergeNAS can converge to a stable architecture and achieve better performance with fewer parameters and search cost. For test accuracy and its stability, MergeNAS outperforms all NAS baseline methods implemented on NAS-Bench-201, including DARTS, ENAS, RS, BOHB, GDAS and hand-crafted ResNet.","doi":"10.24963/ijcai.2020/424","url":"https://ijcai.org/proceedings/2020/424","pdf":"https://ijcai.org/proceedings/2020/0424.pdf","markdown":"https://ijcai.org/proceedings/2020/424.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/424","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":424,"pages":{"first":3065,"last":3072},"keywords":["Machine Learning: Deep Learning","Machine Learning: Deep Learning: Convolutional networks"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24361/file_24361.m3u8","length_sec":286,"provider":"hls"}],"references":[{"position":1,"raw":"[Baker et al., 2017] Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network ar- chitectures using reinforcement learning. In International Conference on Learning Representations, 2017.","title":"Designing neural network architectures using reinforcement learning","authors":"Bowen Baker, Otkrist Gupta, Nikhil Naik, Ramesh Raskar","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bender et al., 2018] Gabriel Bender, Pieter-Jan Kinder- mans, Barret Zoph, Vijay Vasudevan, and Quoc V. Le. Un- derstanding and simplifying one-shot architecture search. In International Conference on Machine Learning, pages 549-558, 2018.","title":"Le. Understanding and simplifying one-shot architecture search","authors":"Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, V Quoc","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bi et al., 2019] Kaifeng Bi, Changping Hu, Lingxi Xie, Xin Chen, Longhui Wei, and Qi Tian. Stabilizing darts with amended gradient estimation on architectural parameters. arXiv preprint arXiv:1910.11831, 2019.","title":"Stabilizing darts with amended gradient estimation on architectural parameters","authors":"Bi","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cai et al., 2019] Han Cai, Ligeng Zhu, and Song Han. Prox- ylessnas: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations, 2019.","title":"Proxylessnas: Direct neural architecture search on target task and hardware","authors":"Han Cai, Ligeng Zhu, Song Han","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2019] Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE International Conference on Computer Vision, pages 1294-1303, 2019.","title":"Progressive differentiable architecture search: Bridging the depth gap between search and evaluation","authors":"Xin Chen, Lingxi Xie, Jun Wu, Qi Tian","year":"2019","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chu et al., 2019] Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li. Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search. arXiv preprint arXiv:1907.01845, 2019.","title":"Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search","authors":"Xiangxiang Chu, Bo Zhang, Ruijun Xu, Jixiang Li","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dong and Yang, 2020] Xuanyi Dong and Yi Yang. Nas- bench-201: Extending the scope of reproducible neural ar- chitecture search. In International Conference on Learn- ing Representations, 2020.","title":"Nasbench-201: Extending the scope of reproducible neural architecture search","authors":"Dong, Yang ;, Xuanyi Dong, Yi Yang","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Elsken et al., 2019] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objective neural archi- tecture search via lamarckian evolution. In International Conference on Learning Representations, 2019.","title":"Efficient multi-objective neural architecture search via lamarckian evolution","authors":"Thomas Elsken, Jan Hendrik Metzen, Frank Hutter","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pat- tern Recognition, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny im- ages. Technical report, Citeseer, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2018a] Chenxi Liu, Barret Zoph, Maxim Neu- mann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Pro- gressive neural architecture search. In ECCV, 2018.","title":"Progressive neural architecture search","authors":"Liu","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2018b] Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hierarchical representations for efficient architecture search. In International Conference on Learning Repre- sentations, 2018.","title":"Hierarchical representations for efficient architecture search","authors":"Liu","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2019] Hanxiao Liu, Karen Simonyan, and Yim- ing Yang. DARTS: differentiable architecture search. In International Conference on Learning Representations, 2019.","title":"DARTS: differentiable architecture search","authors":"Liu","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Miikkulainen et al., 2019] Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Daniel Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, et al. Evolving deep neural networks. In Artificial Intelligence in the Age of Neural Networks and Brain Computing, pages 293-312. Elsevier, 2019.","title":"Evolving deep neural networks","authors":"Jason Risto Miikkulainen, Elliot Liang, Aditya Meyerson, Daniel Rawal, Olivier Fink, Bala Francon, Hormoz Raju, Arshak Shahrzad, Nigel Navruzyan, Duffy","year":"2019","venue":"Artificial Intelligence in the Age of Neural Networks and Brain Computing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nayman et al., 2019] Niv Nayman, Asaf Noy, Tal Rid- nik, Itamar Friedman, Rong Jin, and Lihi Zelnik-Manor. XNAS: neural architecture search with expert advice. In NeurIPS, pages 1975-1985, 2019.","title":"XNAS: neural architecture search with expert advice","authors":"Nayman","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pham et al., 2018] Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural ar- chitecture search via parameter sharing. In International Conference on Machine Learning, 2018.","title":"Efficient neural architecture search via parameter sharing","authors":"Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, Jeff Dean","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Real et al., 2017] Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V. Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In International Conference on Machine Learning, pages 2902-2911, 2017.","title":"Large-scale evolution of image classifiers","authors":"Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, Alexey Kurakin","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Real et al., 2019] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. In AAAI, pages 4780-4789, 2019.","title":"Regularized evolution for image classifier architecture search","authors":"Esteban Real, Alok Aggarwal, Yanping Huang, Quoc V Le","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recogni- tion challenge. International journal of computer vision, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein","year":"2015","venue":"International journal of computer vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sandler et al., 2018] Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottle- necks. In IEEE Conference on Computer Vision and Pat- tern Recognition, pages 4510-4520, 2018.","title":"Mobilenetv2: Inverted residuals and linear bottlenecks","authors":"Mark Sandler, Andrew G Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Stamoulis et al., 2019] Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Single-path nas: Designing hardware-efficient convnets in less than 4 hours. arXiv preprint arXiv:1904.02877, 2019.","title":"Single-path nas: Designing hardware-efficient convnets in less than 4 hours","authors":"Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, Diana Marculescu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xie and Yuille, 2017] Lingxi Xie and Alan L. Yuille. Ge- netic CNN. In IEEE International Conference on Com- puter Vision, ICCV, pages 1388-1397, 2017.","title":"Genetic CNN","authors":"Yuille Xie, Lingxi Xie, Alan L Yuille","year":"2017","venue":"IEEE International Conference on Computer Vision, ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xie et al., 2019] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In International Conference on Learning Repre- sentations, 2019.","title":"SNAS: stochastic neural architecture search","authors":"Sirui Xie, Hehui Zheng, Chunxiao Liu, Liang Lin","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2020] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pc- darts: Partial channel connections for memory-efficient ar- chitecture search. In International Conference on Learn- ing Representations, 2020.","title":"Pcdarts: Partial channel connections for memory-efficient architecture search","authors":"Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, Hongkai Xiong","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zela et al., 2020] Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hut- ter. Understanding and robustifying differentiable archi- tecture search. In International Conference on Learning Representations, 2020.","title":"Understanding and robustifying differentiable architecture search","authors":"Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, Frank Hutter","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhong et al., 2018] Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Practical block-wise neural network architecture generation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2423- 2432, 2018.","title":"Practical block-wise neural network architecture generation","authors":"Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, Cheng-Lin Liu","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou et al., 2019] Hongpeng Zhou, Minghao Yang, Jun Wang, and Wei Pan. Bayesnas: A bayesian approach for neural architecture search. In International Conference on Machine Learning, pages 7603-7613, 2019.","title":"Bayesnas: A bayesian approach for neural architecture search","authors":"Zhou","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zoph and Le, 2017] Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In Inter- national Conference on Learning Representations, 2017.","title":"Neural architecture search with reinforcement learning","authors":"Le ; Zoph, Barret Zoph, Quoc V Le","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zoph et al., 2018] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architec- tures for scalable image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.","title":"Learning transferable architectures for scalable image recognition","authors":"Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V Le","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}