{"type":"ScholarlyArticle","title":"One-Shot Neural Architecture Search via Novelty Driven Sampling","authors":[{"name":"Miao Zhang","slug":"miao-zhang","given":"Miao","family":"Zhang","affiliation":"School of Information and Electronics , Beijing Institute of Technology; Faculty of Engineering and Information Technology , University of Technology Sydney"},{"name":"Huiqi Li","slug":"huiqi-li","given":"Huiqi","family":"Li","affiliation":"School of Information and Electronics , Beijing Institute of Technology"},{"name":"Shirui Pan","slug":"shirui-pan","given":"Shirui","family":"Pan","affiliation":"Faculty of Information Technology , Monash University"},{"name":"Taoping Liu","slug":"taoping-liu","given":"Taoping","family":"Liu","affiliation":"Faculty of Engineering and Information Technology , University of Technology Sydney"},{"name":"Steven Su","slug":"steven-su","given":"Steven","family":"Su","affiliation":"Faculty of Engineering and Information Technology , University of Technology Sydney"}],"abstract":"One-Shot Neural architecture search (NAS) has received wide attentions due to its computational efficiency. Most state-of-the-art One-Shot NAS methods use the validation accuracy based on inheriting weights from the supernet as the stepping stone to search for the best performing architecture, adopting a bilevel optimization pattern with assuming this validation accuracy approximates to the test accuracy after re-training. However, recent works have found that there is no positive correlation between the above validation accuracy and test accuracy for these One-Shot NAS methods, and this reward based sampling for supernet training also entails the rich-get-richer problem. To handle this deceptive problem, this paper presents a new approach, Efficient Novelty-driven Neural Architecture Search, to sample the most abnormal architecture to train the supernet. Specifically, a single-path supernet is adopted, and only the weights of a single architecture sampled by our novelty search are optimized in each step to reduce the memory demand greatly. Experiments demonstrate the effectiveness and efficiency of our novelty search based architecture sampling method.","doi":"10.24963/ijcai.2020/441","url":"https://ijcai.org/proceedings/2020/441","pdf":"https://ijcai.org/proceedings/2020/0441.pdf","markdown":"https://ijcai.org/proceedings/2020/441.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/441","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":441,"pages":{"first":3188,"last":3194},"keywords":["Machine Learning: Online Learning","Heuristic Search and Game Playing: Heuristic Search and Machine Learning","Heuristic Search and Game Playing: Meta-Reasoning and Meta-heuristics","Machine Learning: Deep Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24036/file_24036.m3u8","length_sec":295,"provider":"hls"}],"references":[{"position":1,"raw":"[Baker et al., 2018] Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. ICLR, 2018.","title":"Accelerating neural architecture search using performance prediction","authors":"Bowen Baker, Otkrist Gupta, Ramesh Raskar, Nikhil Naik","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bender et al., 2018] Gabriel Bender, Pieter-Jan Kinder- mans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Un- derstanding and simplifying one-shot architecture search. In ICML, pages 549-558, 2018.","title":"Understanding and simplifying one-shot architecture search","authors":"Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, Quoc Le","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Benyahia et al., 2019] Yassine Benyahia, Kaicheng Yu, Kamil Bennani Smires, Martin Jaggi, Anthony C Davi- son, Mathieu Salzmann, and Claudiu Musat. Overcoming multi-model forgetting. In ICML, pages 594-603, 2019.","title":"Overcoming multi-model forgetting","authors":"Yassine Benyahia, Kaicheng Yu, Kamil Bennani Smires, Martin Jaggi, Mathieu Anthony C Davison, Claudiu Salzmann, Musat","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cai et al., 2019] Han Cai, Ligeng Zhu, and Song Han. Prox- ylessnas: Direct neural architecture search on target task and hardware. ICLR, 2019.","title":"Proxylessnas: Direct neural architecture search on target task and hardware","authors":"Han Cai, Ligeng Zhu, Song Han","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Casale et al., 2019] Francesco Paolo Casale, Jonathan Gor- don, and Nicolo Fusi. Probabilistic neural architecture search. arXiv preprint arXiv:1902.05116, 2019.","title":"Probabilistic neural architecture search","authors":"Francesco Paolo Casale, Jonathan Gordon, Nicolo Fusi","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chu et al., 2019] Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li. Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search. arXiv preprint arXiv:1907.01845, 2019.","title":"Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search","authors":"Xiangxiang Chu, Bo Zhang, Ruijun Xu, Jixiang Li","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Conti et al., 2018] Edoardo Conti, Vashisht Madhavan, Fe- lipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty- seeking agents. In NeurIPS, pages 5027-5038, 2018.","title":"Improving exploration in evolution strategies for deep reinforcement learning via a population of noveltyseeking agents","authors":"Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, Jeff Clune","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dong and Yang, 2019] Xuanyi Dong and Yi Yang. Search- ing for a robust neural architecture in four gpu hours. In CVPR. IEEE Computer Society, 2019.","title":"Searching for a robust neural architecture in four gpu hours","authors":"Dong, Yang ;, Xuanyi Dong, Yi Yang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dong and Yang, 2020] Xuanyi Dong and Yi Yang. Nas- bench-201: Extending the scope of reproducible neural ar- chitecture search. ICLR, 2020.","title":"Xuanyi Dong and Yi Yang. Nasbench-201: Extending the scope of reproducible neural architecture search","authors":"Yang Dong","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Elsken et al., 2019] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. JMLR, 20(55):1-21, 2019.","title":"Neural architecture search: A survey","authors":"Thomas Elsken, Jan Hendrik Metzen, Frank Hutter","year":"2019","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Guo et al., 2019] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uni- form sampling, 2019.","title":"Single path one-shot neural architecture search with uniform sampling","authors":"Guo","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kendall, 1945] Maurice G Kendall. The treatment of ties in ranking problems. Biometrika, 33(3):239-251, 1945.","title":"The treatment of ties in ranking problems","authors":"Maurice G Kendall","year":"1945","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lehman and Stanley, 2011] Joel Lehman and Kenneth O Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation, 19(2):189-223, 2011.","title":"Abandoning objectives: Evolution through the search for novelty alone","authors":"Stanley ; Lehman, Joel Lehman, Kenneth O Stanley","year":"2011","venue":"Evolutionary computation","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li and Talwalkar, 2019] Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. arXiv preprint arXiv:1902.07638, 2019.","title":"Random search and reproducibility for neural architecture search","authors":"Talwalkar Li, Liam Li, Ameet Talwalkar","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2019] Xiang Li, Chen Lin, Chuming Li, Ming Sun, Wei Wu, Junjie Yan, and Wanli Ouyang. Improving one-shot nas by suppressing the posterior fading. arXiv preprint arXiv:1910.02543, 2019.","title":"Improving one-shot nas by suppressing the posterior fading","authors":"Xiang Li, Chen Lin, Chuming Li, Ming Sun, Wei Wu, Junjie Yan, Wanli Ouyang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2018] Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hierarchical representations for efficient architecture search. ICLR, 2018.","title":"Hierarchical representations for efficient architecture search","authors":"Liu","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2019] Hanxiao Liu, Karen Simonyan, and Yim- ing Yang. Darts: Differentiable architecture search. ICLR, 2019.","title":"Darts: Differentiable architecture search","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Luo et al., 2018] Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture optimization. In NeurIPS, pages 7816-7827, 2018.","title":"Neural architecture optimization","authors":"Luo","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pham et al., 2018] Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In ICML, pages 4092-4101, 2018.","title":"Efficient neural architecture search via parameter sharing","authors":"Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, Jeff Dean","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Real et al., 2019] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. AAAI, 2019.","title":"Regularized evolution for image classifier architecture search","authors":"Esteban Real, Alok Aggarwal, Yanping Huang, Quoc V Le","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sciuto et al., 2020] Christian Sciuto, Kaicheng Yu, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. In ICLR, 2020.","title":"Evaluating the search phase of neural architecture search","authors":"Sciuto","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Singh et al., 2019] Prabhant Singh, Tobias Jacobs, Se- bastien Nicolas, and Mischa Schmidt. A study of the learning progress in neural architecture search techniques. arXiv preprint arXiv:1906.07590, 2019.","title":"A study of the learning progress in neural architecture search techniques","authors":"Singh","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2020] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. A comprehensive survey on graph neural networks. TNNLS, 2020.","title":"A comprehensive survey on graph neural networks","authors":"Wu","year":"2020","venue":"TNNLS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xie et al., 2019] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. ICLR, 2019.","title":"Snas: stochastic neural architecture search","authors":"Sirui Xie, Hehui Zheng, Chunxiao Liu, Liang Lin","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ying et al., 2019] Chris Ying, Aaron Klein, Eric Chris- tiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas-bench-101: Towards reproducible neural architecture search. In ICML, pages 7105-7114, 2019.","title":"Nas-bench-101: Towards reproducible neural architecture search","authors":"Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, Frank Hutter","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2019] Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. ICLR, 2019.","title":"Graph hypernetworks for neural architecture search","authors":"Chris Zhang, Mengye Ren, Raquel Urtasun","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2020] Miao Zhang, Huiqi Li, Shirui Pan, Xi- aojun Chang, and Steven Su Su. Overcoming multi-model forgetting in one-shot nas with diversity maximization. In CVPR. IEEE Computer Society, 2020.","title":"Overcoming multi-model forgetting in one-shot nas with diversity maximization","authors":"Miao Zhang, Huiqi Li, Shirui Pan, Xiaojun Chang, Steven Su, Su","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zheng et al., 2019] Xiawu Zheng, Rongrong Ji, Lang Tang, Baochang Zhang, Jianzhuang Liu, and Qi Tian. Multino- mial distribution learning for effective neural architecture search. In ICCV, 2019.","title":"Multinomial distribution learning for effective neural architecture search","authors":"Zheng","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2019] Hongpeng Zhou, Minghao Yang, Jun Wang, and Wei Pan. Bayesnas: A bayesian approach for neural architecture search. In ICML, pages 7603-7613, 2019.","title":"Bayesnas: A bayesian approach for neural architecture search","authors":"Zhou","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zoph et al., 2018] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architec- tures for scalable image recognition. In CVPR, pages 8697-8710, 2018.","title":"Learning transferable architectures for scalable image recognition","authors":"Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V Le","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}