{"type":"ScholarlyArticle","title":"Graph Masked Autoencoder Enhanced Predictor for Neural Architecture Search","authors":[{"name":"Kun Jing","slug":"kun-jing","given":"Kun","family":"Jing","affiliation":"School of Computer Science and Technology , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Jungang Xu","slug":"jungang-xu","given":"Jungang","family":"Xu","affiliation":"School of Computer Science and Technology , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Pengfei Li","slug":"pengfei-li","given":"Pengfei","family":"Li","affiliation":"School of Computer Science and Technology , University of Chinese Academy of Sciences , Beijing , China"}],"abstract":"Performance estimation of neural architecture is a crucial component of neural architecture search (NAS). Meanwhile, neural predictor is a current mainstream performance estimation method. However, it is a challenging task to train the predictor with few architecture evaluations for efficient NAS. In this paper, we propose a graph masked autoencoder (GMAE) enhanced predictor, which can reduce the dependence on supervision data by self-supervised pre-training with untrained architectures. We compare our GMAE-enhanced predictor with existing predictors in different search spaces, and experimental results show that our predictor has high query utilization. Moreover, GMAE-enhanced predictor with different search strategies can discover competitive architectures in different search spaces. Code and supplementary materials are available at https://github.com/kunjing96/GMAENAS.git.","doi":"10.24963/ijcai.2022/432","url":"https://ijcai.org/proceedings/2022/432","pdf":"https://ijcai.org/proceedings/2022/0432.pdf","markdown":"https://ijcai.org/proceedings/2022/432.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/432","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":432,"pages":{"first":3114,"last":3120},"keywords":["Machine Learning: Self-supervised Learning","Machine Learning: Automated Machine Learning","Machine Learning: Learning Graphical Models","Computer Vision: Recognition (object detection, categorization)"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986459","length_sec":86,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984602","length_sec":351,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Baker et al., 2017] B. Baker, O. Gupta, N. Naik, et al. De- signing neural network architectures using reinforcement learning. In Proc. ICLR, 2017.","title":"Designing neural network architectures using reinforcement learning","authors":"B Baker, O Gupta, N Naik","year":"2017","venue":"Proc. ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2021a] X. Chen, L. Xie, J. Wu, et al. Fitting the search space of weight-sharing NAS with graph convolu- tional networks. In Proc. AAAI, 2021.","title":"Fitting the search space of weight-sharing NAS with graph convolutional networks","authors":"Chen","year":"2021","venue":"Proc. AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2021b] Y. Chen, Y. Guo, Q. Chen, et al. Con- trastive neural architecture search with neural architecture comparators. In Proc. CVPR, 2021.","title":"Contrastive neural architecture search with neural architecture comparators","authors":"Chen","year":"2021","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Devlin et al., 2019] J. Devlin, M. Chang, K. Lee, et al. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL-HLT, 2019.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"J Devlin, M Chang, K Lee","year":"2019","venue":"Proc. NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hassani and Ahmadi, 2020] K. Hassani and A. H. K. Ah- madi. Contrastive multi-view representation learning on graphs. In Proc. ICML, 2020.","title":"Contrastive multi-view representation learning on graphs","authors":"K Hassani, A H K Ahmadi","year":"2020","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2021] K. He, X. Chen, S. Xie, et al. Masked au- toencoders are scalable vision learners. arXiv preprint arXiv:2111.06377, 2021.","title":"Masked autoencoders are scalable vision learners","authors":"K He, X Chen, S Xie","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kandasamy et al., 2018] K. Kandasamy, W. Neiswanger, J. Schneider, et al. Neural architecture search with bayesian optimisation and optimal transport. In Proc. NeurIPS, 2018.","title":"Neural architecture search with bayesian optimisation and optimal transport","authors":"K Kandasamy, W Neiswanger, J Schneider","year":"2018","venue":"Proc. NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Liu et al., 2018] C. Liu, B. Zoph, M. Neumann, et al. Pro- gressive neural architecture search. In Proc. ECCV, 2018.","title":"Progressive neural architecture search","authors":"C Liu, B Zoph, M Neumann","year":"2018","venue":"Proc. ECCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Liu et al., 2019] H. Liu, K. Simonyan, and Y. Yang. DARTS: differentiable architecture search. In Proc. ICLR, 2019.","title":"DARTS: differentiable architecture search","authors":"H Liu, K Simonyan, Y Yang","year":"2019","venue":"Proc. ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Luo et al., 2020a] R. Luo, X. Tan, R. Wang, et al. Neu- ral architecture search with GBDT. arXiv preprint arXiv:2007.04785, 2020.","title":"Neural architecture search with GBDT","authors":"Luo","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Luo et al., 2020b] R. Luo, X. Tan, R. Wang, et al. Semi- supervised neural architecture search. In Proc. NeurIPS, 2020.","title":"Semisupervised neural architecture search","authors":"Luo","year":"2020","venue":"Proc. NeurIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ma et al., 2019] L. Ma, J. Cui, and B. Yang. Deep neural architecture search with deep graph bayesian optimization. In Proc. WI, 2019.","title":"Deep neural architecture search with deep graph bayesian optimization","authors":"L Ma, J Cui, B Yang","year":"2019","venue":"Proc. WI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Na et al., 2021] B. Na, J. Mok, H. Choe, et al. Accelerating neural architecture search via proxy data. In Proc. IJCAI, 2021.","title":"Accelerating neural architecture search via proxy data","authors":"B Na, J Mok, H Choe","year":"2021","venue":"Proc. IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/392"},{"position":14,"raw":"[Ning et al., 2020] X. Ning, Y. Zheng, T. Zhao, et al. A generic graph-based neural architecture encoding scheme for predictor-based NAS. In Proc. ECCV, 2020.","title":"A generic graph-based neural architecture encoding scheme for predictor-based NAS","authors":"X Ning, Y Zheng, T Zhao","year":"2020","venue":"Proc. ECCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"M. Park. Data proxy generation for fast and efficient neural architecture search. arXiv preprint arXiv:1911.09322, 2019.","title":"Data proxy generation for fast and efficient neural architecture search","authors":"M Park","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pham et al., 2018] H. Pham, M. Y. Guan, B. Zoph, et al. Ef- ficient neural architecture search via parameter sharing. In Proc. ICML, 2018.","title":"Efficient neural architecture search via parameter sharing","authors":"H Pham, M Y Guan, B Zoph","year":"2018","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Real et al., 2019] E. Real, A. Aggarwal, Y. Huang, et al. Regularized evolution for image classifier architecture search. In Proc. AAAI, 2019.","title":"Regularized evolution for image classifier architecture search","authors":"E Real, A Aggarwal, Y Huang","year":"2019","venue":"Proc. AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ru et al., 2020] B. X. Ru, X. Wan, X. Dong, et al. Neural architecture search using bayesian optimisa- tion with weisfeiler-lehman kernel. arXiv preprint arXiv:2006.07556, 2020.","title":"Neural architecture search using bayesian optimisation with weisfeiler-lehman kernel","authors":"B X Ru, X Wan, X Dong","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shi et al., 2020] H. Shi, R. Pi, H. Xu, et al. Bridging the gap between sample-based and one-shot neural architec- ture search with BONAS. In Proc. NeurIPS, 2020.","title":"Bridging the gap between sample-based and one-shot neural architecture search with BONAS","authors":"H Shi, R Pi, H Xu","year":"2020","venue":"Proc. NeurIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Siems et al., 2020] J. Siems, L. Zimmer, A. Zela, et al. Nas- bench-301 and the case for surrogate benchmarks for neu- ral architecture search. arXiv preprint arXiv:2008.09777, 2020.","title":"Nasbench-301 and the case for surrogate benchmarks for neural architecture search","authors":"J Siems, L Zimmer, A Zela","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Swersky et al., 2014] K. Swersky, D. Duvenaud, J. Snoek, et al. Raiders of the lost architecture: Kernels for bayesian optimization in conditional parameter spaces. arXiv preprint arXiv:1409.4011, 2014.","title":"Raiders of the lost architecture: Kernels for bayesian optimization in conditional parameter spaces","authors":"K Swersky, D Duvenaud, J Snoek","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tang et al., 2020] Y. Tang, Y. Wang, Y. Xu, et al. A semi-supervised assessor of neural architectures. In Proc. CVPR, 2020.","title":"A semi-supervised assessor of neural architectures","authors":"Y Tang, Y Wang, Y Xu","year":"2020","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2019] L. Wang, Y. Zhao, Y. Jinnai, et al. Al- phax: exploring neural architectures with deep neural networks and monte carlo tree search. arXiv preprint arXiv:1903.11059, 2019.","title":"Alphax: exploring neural architectures with deep neural networks and monte carlo tree search","authors":"L Wang, Y Zhao, Y Jinnai","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2021] R. Wang, X. Chen, M. Cheng, et al. RANK-NOSH: efficient predictor-based architecture search via non-uniform successive halving. arXiv preprint arXiv:2108.08019, 2021.","title":"RANK-NOSH: efficient predictor-based architecture search via non-uniform successive halving","authors":"R Wang, X Chen, M Cheng","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wen et al., 2020] W. Wen, H. Liu, Y. Chen, et al. Neural predictor for neural architecture search. In Proc. ECCV, 2020.","title":"Neural predictor for neural architecture search","authors":"W Wen, H Liu, Y Chen","year":"2020","venue":"Proc. ECCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[White et al., 2021a] C. White, W. Neiswanger, and Y. Sa- vani. BANANAS: bayesian optimization with neural ar- chitectures for neural architecture search. In Proc. AAAI, 2021.","title":"BANANAS: bayesian optimization with neural architectures for neural architecture search","authors":"White","year":"2021","venue":"Proc. AAAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[White et al., 2021b] C. White, A. Zela, B. Ru, et al. How powerful are performance predictors in neural architecture search? arXiv preprint arXiv:2104.01177, 2021.","title":"How powerful are performance predictors in neural architecture search","authors":"White","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wu et al., 2021] J. Wu, X. Dai, D. Chen, et al. Weak NAS predictors are all you need. arXiv preprint arXiv:2102.10490, 2021.","title":"Weak NAS predictors are all you need","authors":"J Wu, X Dai, D Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xu et al., 2021] Y. Xu, Y. Wang, K. Han, et al. Renas: Rel- ativistic evaluation of neural architecture search. In Proc. CVPR, 2021.","title":"Relativistic evaluation of neural architecture search","authors":"Y Xu, Y Wang, K Han","year":"2021","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yan et al., 2021] S. Yan, K. Song, F. Liu, et al. CATE: computation-aware neural architecture encoding with transformers. In Proc. ICML, 2021.","title":"CATE: computation-aware neural architecture encoding with transformers","authors":"S Yan, K Song, F Liu","year":"2021","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Ying et al., 2019] C. Ying, A. Klein, E. Christiansen, et al. Nas-bench-101: Towards reproducible neural architecture search. In Proc. ICML, 2019.","title":"Nas-bench-101: Towards reproducible neural architecture search","authors":"C Ying, A Klein, E Christiansen","year":"2019","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[You et al., 2020] Y. You, T. Chen, Y. Sui, et al. Graph con- trastive learning with augmentations. In Proc. NeurIPS, 2020.","title":"Graph contrastive learning with augmentations","authors":"Y You, T Chen, Y Sui","year":"2020","venue":"Proc. NeurIPS","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zoph and Le, 2017] B. Zoph and Q. V. Le. Neural architec- ture search with reinforcement learning. In Proc. ICLR, 2017.","title":"Neural architecture search with reinforcement learning","authors":"B Zoph, Q V Le","year":"2017","venue":"Proc. ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}