{"type":"ScholarlyArticle","title":"LISSNAS: Locality-based Iterative Search Space Shrinkage for Neural Architecture Search","authors":[{"name":"Bhavna Gopal","slug":"bhavna-gopal","given":"Bhavna","family":"Gopal","affiliation":"Duke University"},{"name":"Arjun Sridhar","slug":"arjun-sridhar","given":"Arjun","family":"Sridhar","affiliation":"Duke University"},{"name":"Tunhou Zhang","slug":"tunhou-zhang","given":"Tunhou","family":"Zhang","affiliation":"Duke University"},{"name":"Yiran Chen","slug":"yiran-chen","given":"Yiran","family":"Chen","affiliation":"Duke University"}],"abstract":"Search spaces hallmark the advancement of Neural Architecture Search (NAS). Large and complex search spaces with versatile building operators and structures provide more opportunities to brew promising architectures, yet pose severe challenges on efficient exploration and exploitation. Subsequently, several search space shrinkage methods optimize by selecting a single sub-region that contains some well-performing networks.  Small performance and efficiency gains are observed with these methods but such techniques leave room for significantly improved search performance and are ineffective at retaining architectural diversity. We propose LISSNAS, an automated algorithm that shrinks a large space into a diverse, small search space with SOTA search performance. Our approach leverages locality, the relationship between structural and performance similarity, to efficiently extract many pockets of well-performing networks. We showcase our method on an array of search spaces spanning various sizes and datasets. We accentuate the effectiveness of our shrunk spaces when used in one-shot search by achieving the best Top-1 accuracy in two different search spaces. Our method achieves a SOTA Top-1 accuracy of 77.6% in ImageNet under mobile constraints, best-in-class Kendal-Tau, architectural diversity, and search space size.","doi":"10.24963/ijcai.2023/86","url":"https://ijcai.org/proceedings/2023/86","pdf":"https://ijcai.org/proceedings/2023/0086.pdf","markdown":"https://ijcai.org/proceedings/2023/86.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/86","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":86,"pages":{"first":773,"last":781},"keywords":["Machine Learning: ML: Automated machine learning","Computer Vision: CV: Machine learning for vision"],"videos":[],"references":[{"position":1,"raw":"[Bergstra et al., 2013] James Bergstra, Daniel Yamins, and David Cox. Making a science of model search: Hyperpa- rameter optimization in hundreds of dimensions for vision architectures. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Ma- chine Learning Research, pages 115-123, Atlanta, Geor- gia, USA, 17-19 Jun 2013. PMLR.","title":"Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures","authors":"Bergstra","year":"2013","venue":"Proceedings of the 30th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brock et al., 2018] Andrew Brock, Theo Lim, J.M. Ritchie, and Nick Weston. SMASH: One-shot model architecture search through hypernetworks. In International Confer- ence on Learning Representations, 2018.","title":"SMASH: One-shot model architecture search through hypernetworks","authors":"Andrew Brock, Theo Lim, J M Ritchie, Nick Weston","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cai et al., 2019] Han Cai, Chuang Gan, and Song Han. Once for all: Train one network and specialize it for ef- ficient deployment. CoRR, abs/1908.09791, 2019.","title":"Once for all: Train one network and specialize it for efficient deployment","authors":"Han Cai, Chuang Gan, Song Han","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chu et al., 2019] Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li. Fairnas: Rethinking evaluation fair- ness of weight sharing neural architecture search. CoRR, abs/1907.01845, 2019.","title":"Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search","authors":"Xiangxiang Chu, Bo Zhang, Ruijun Xu, Jixiang Li","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ci et al., 2020] Yuanzheng Ci, Chen Lin, Ming Sun, Boyu Chen, Hongwen Zhang, and Wanli Ouyang. Evolv- ing search space for neural architecture search. CoRR, abs/2011.10904, 2020.","title":"Evolving search space for neural architecture search","authors":"Yuanzheng Ci, Chen Lin, Ming Sun, Boyu Chen, Hongwen Zhang, Wanli Ouyang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Guo et al., 2019] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uni- form sampling. CoRR, abs/1904.00420, 2019.","title":"Single path one-shot neural architecture search with uniform sampling","authors":"Guo","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Howard et al., 2019] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Wei- jun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasude- van, Quoc V. Le, and Hartwig Adam. Searching for mo- bilenetv3. CoRR, abs/1905.02244, 2019.","title":"Searching for mobilenetv3","authors":"Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V Le, Hartwig Adam","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2020] Yiming Hu, Yuding Liang, Zichao Guo, Ruosi Wan, Xiangyu Zhang, Yichen Wei, Qingyi Gu, and Jian Sun. Angle-based search space shrinking for neural architecture search. CoRR, abs/2004.13431, 2020.","title":"Angle-based search space shrinking for neural architecture search","authors":"Yiming Hu, Yuding Liang, Zichao Guo, Ruosi Wan, Xiangyu Zhang, Yichen Wei, Qingyi Gu, Jian Sun","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2021] Tao Huang, Shan You, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Greedynasv2: Greedier search with a greedy path filter. CoRR, abs/2111.12609, 2021.","title":"Greedynasv2: Greedier search with a greedy path filter","authors":"Tao Huang, Shan You, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, Chang Xu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2019] Hanxiao Liu, Karen Simonyan, and Yim- ing Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019.","title":"DARTS: Differentiable architecture search","authors":"Liu","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Luo et al., 2019a] Renqian Luo, Tao Qin, and Enhong Chen. Balanced one-shot neural architecture optimization. arXiv preprint arXiv:1909.10815, 2019.","title":"Balanced one-shot neural architecture optimization","authors":"Luo","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Luo et al., 2019b] Renqian Luo, Tao Qin, and Enhong Chen. Understanding and improving one-shot neural ar- chitecture optimization. CoRR, abs/1909.10815, 2019.","title":"Understanding and improving one-shot neural architecture optimization","authors":"Luo","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mendoza et al., 2016] Hector Mendoza, Aaron Klein, Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Towards automatically-tuned neural networks. In Workshop on automatic machine learning, pages 58-65. PMLR, 2016.","title":"Towards automatically-tuned neural networks","authors":"Hector Mendoza, Aaron Klein, Matthias Feurer, Jost Tobias Springenberg, Frank Hutter","year":"2016","venue":"Workshop on automatic machine learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pham et al., 2018] Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International conference on machine learning, pages 4095-4104. PMLR, 2018.","title":"Efficient neural architecture search via parameters sharing","authors":"Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, Jeff Dean","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Radosavovic et al., 2019] Ilija Radosavovic, Justin John- son, Saining Xie, Wan-Yen Lo, and Piotr Dollar. On net- work design spaces for visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.","title":"On network design spaces for visual recognition","authors":"Ilija Radosavovic, Justin Johnson, Saining Xie, Wan-Yen Lo, Piotr Dollar","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Radosavovic et al., 2020] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10428-10436, 2020.","title":"Designing network design spaces","authors":"Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, Piotr Dollár","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Real et al., 2017] Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In International Conference on Machine Learning, pages 2902-2911. PMLR, 2017.","title":"Large-scale evolution of image classifiers","authors":"Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Alexey Quoc V Le, Kurakin","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Real et al., 2019] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 4780-4789, 2019.","title":"Regularized evolution for image classifier architecture search","authors":"Esteban Real, Alok Aggarwal, Yanping Huang, Quoc V Le","year":"2019","venue":"Proceedings of the aaai conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sciuto et al., 2019] Christian Sciuto, Kaicheng Yu, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluat- ing the search phase of neural architecture search. CoRR, abs/1902.08142, 2019.","title":"Evaluating the search phase of neural architecture search","authors":"Sciuto","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Su et al., 2021] Xiu Su, Shan You, Mingkai Zheng, Fei Wang, Chen Qian, Changshui Zhang, and Chang Xu. K- shot NAS: learnable weight-sharing for NAS with k-shot supernets. CoRR, abs/2106.06442, 2021.","title":"Kshot NAS: learnable weight-sharing for NAS with k-shot supernets","authors":"Xiu Su, Shan You, Mingkai Zheng, Fei Wang, Chen Qian, Changshui Zhang, Chang Xu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tan et al., 2018] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V. Le. Mnasnet: Platform- aware neural architecture search for mobile. CoRR, abs/1807.11626, 2018.","title":"Mnasnet: Platformaware neural architecture search for mobile","authors":"Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, V Quoc, Le","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tyagi et al., 2022] Anjul Tyagi, Cong Xie, and Klaus Mueller. NAS-navigator: Visual steering for explainable one-shot deep neural network synthesis. IEEE Transac- tions on Visualization and Computer Graphics, pages 1- 11, 2022.","title":"NAS-navigator: Visual steering for explainable one-shot deep neural network synthesis","authors":"Anjul Tyagi, Cong Xie, Klaus Mueller","year":"2022","venue":"IEEE Transactions on Visualization and Computer Graphics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2020] Dilin Wang, Meng Li, Chengyue Gong, and Vikas Chandra. Attentivenas: Improving neural ar- chitecture search via attentive sampling. arXiv preprint arXiv:2011.09011, 2020.","title":"Attentivenas: Improving neural architecture search via attentive sampling","authors":"Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2021] Dilin Wang, Chengyue Gong, Meng Li, Qiang Liu, and Vikas Chandra. Alphanet: Improved training of supernets with alpha-divergence. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10760-10771. PMLR, 18-24 Jul 2021.","title":"Alphanet: Improved training of supernets with alpha-divergence","authors":"Wang","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wei et al., 2022] Chen Wei, Chuang Niu, Yiping Tang, Yue Wang, Haihong Hu, and Jimin Liang. Npenas: Neural predictor guided evolution for neural architecture search. IEEE Transactions on Neural Networks and Learning Sys- tems, 2022.","title":"Npenas: Neural predictor guided evolution for neural architecture search","authors":"Chen Wei, Chuang Niu, Yiping Tang, Yue Wang, Haihong Hu, Jimin Liang","year":"2022","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wen et al., 2020] Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Bender, and Pieter-Jan Kindermans. Neu- ral predictor for neural architecture search. In European Conference on Computer Vision, pages 660-676. Springer, 2020.","title":"Neural predictor for neural architecture search","authors":"Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Bender, Pieter-Jan Kindermans","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[White et al., 2021] Colin White, Arber Zela, Robin Ru, Yang Liu, and Frank Hutter. How powerful are perfor- mance predictors in neural architecture search? Advances in Neural Information Processing Systems, 34:28454- 28469, 2021.","title":"How powerful are performance predictors in neural architecture search","authors":"Colin White, Arber Zela, Robin Ru, Yang Liu, Frank Hutter","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wu et al., 2018] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. CoRR, abs/1812.03443, 2018.","title":"Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search","authors":"Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, Kurt Keutzer","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xia and Ding, 2020] Xin Xia and Wenrui Ding. HNAS: hi- erarchical neural architecture search on mobile devices. CoRR, abs/2005.07564, 2020.","title":"HNAS: hierarchical neural architecture search on mobile devices","authors":"Ding ; ] Xin Xia, Wenrui Xia, Ding","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xie et al., 2018] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. arXiv preprint arXiv:1812.09926, 2018.","title":"Snas: stochastic neural architecture search","authors":"Sirui Xie, Hehui Zheng, Chunxiao Liu, Liang Lin","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xue et al., 2021] Yu Xue, Pengcheng Jiang, Ferrante Neri, and Jiayu Liang. A multi-objective evolutionary approach based on graph-in-graph for neural architecture search of convolutional neural networks. International Journal of Neural Systems, 31(09):2150035, 2021.","title":"A multi-objective evolutionary approach based on graph-in-graph for neural architecture search of convolutional neural networks","authors":"Yu Xue, Pengcheng Jiang, Ferrante Neri, Jiayu Liang","year":"2021","venue":"International Journal of Neural Systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Ying et al., 2019] Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, and Frank Hutter. Nas-bench-101: Towards reproducible neural architecture search. CoRR, abs/1902.09635, 2019.","title":"Nas-bench-101: Towards reproducible neural architecture search","authors":"Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, Frank Hutter","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[You et al., 2020] Shan You, Tao Huang, Mingmin Yang, Fei Wang, Chen Qian, and Changshui Zhang. Greedynas: Towards fast one-shot nas with greedy supernet. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1996-2005, 2020.","title":"Greedynas: Towards fast one-shot nas with greedy supernet","authors":"Shan You, Tao Huang, Mingmin Yang, Fei Wang, Chen Qian, Changshui Zhang","year":"2020","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2018a] Xiangyu Zhang, Xinyu Zhou, Mengx- iao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 6848-6856, 2018.","title":"Shufflenet: An extremely efficient convolutional neural network for mobile devices","authors":"Zhang","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang et al., 2018b] Xinbang Zhang, Zehao Huang, and Naiyan Wang. You only search once: Single shot neural architecture search via direct sparse optimization. CoRR, abs/1811.01567, 2018.","title":"You only search once: Single shot neural architecture search via direct sparse optimization","authors":"Zhang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2020] Xinbang Zhang, Zehao Huang, Naiyan Wang, Shiming Xiang, and Chunhong Pan. You only search once: Single shot neural architecture search via direct sparse optimization. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 43(9):2891-2904, 2020.","title":"You only search once: Single shot neural architecture search via direct sparse optimization","authors":"Zhang","year":"2020","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zoph and Le, 2017] Barret Zoph and Quoc Le. Neural ar- chitecture search with reinforcement learning. In Interna- tional Conference on Learning Representations, 2017.","title":"Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning","authors":"Le Zoph","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zoph et al., 2018] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architec- tures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 8697-8710, 2018.","title":"Learning transferable architectures for scalable image recognition","authors":"Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V Le","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}