{"type":"ScholarlyArticle","title":"SI-VDNAS: Semi-Implicit Variational Dropout for Hierarchical One-shot Neural Architecture Search","authors":[{"name":"Yaoming Wang","slug":"yaoming-wang","given":"Yaoming","family":"Wang","affiliation":"Department of Electronic Engineering , Shanghai Jiao Tong University , China"},{"name":"Wenrui Dai","slug":"wenrui-dai","given":"Wenrui","family":"Dai","affiliation":"Department of Computer Science & Engineering , Shanghai Jiao Tong University , China"},{"name":"Chenglin Li","slug":"chenglin-li","given":"Chenglin","family":"Li","affiliation":"Department of Electronic Engineering , Shanghai Jiao Tong University , China"},{"name":"Junni Zou","slug":"junni-zou","given":"Junni","family":"Zou","affiliation":"Department of Computer Science & Engineering , Shanghai Jiao Tong University , China"},{"name":"Hongkai Xiong","slug":"hongkai-xiong","given":"Hongkai","family":"Xiong","affiliation":"Department of Electronic Engineering , Shanghai Jiao Tong University , China"}],"abstract":"Bayesian methods have improved the interpretability and stability of neural architecture search (NAS). In this paper, we propose a novel probabilistic approach, namely Semi-Implicit Variational Dropout one-shot Neural Architecture Search (SI-VDNAS), that leverages semi-implicit variational dropout to support architecture search with variable operations and edges. SI-VDNAS achieves stable training that would not be affected by the over-selection of skip-connect operation. Experimental results demonstrate that SI-VDNAS finds a convergent architecture with only 2.7 MB parameters within 0.8 GPU-days and can achieve 2.60% top-1 error rate on CIFAR-10. The convergent architecture can obtain a top-1 error rate of 16.20% and 25.6% when transferred to CIFAR-100 and ImageNet (mobile setting).","doi":"10.24963/ijcai.2020/289","url":"https://ijcai.org/proceedings/2020/289","pdf":"https://ijcai.org/proceedings/2020/0289.pdf","markdown":"https://ijcai.org/proceedings/2020/289.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/289","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":289,"pages":{"first":2088,"last":2095},"keywords":["Machine Learning: Probabilistic Machine Learning","Machine Learning: Deep Learning: Convolutional networks"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24801/file_24801.m3u8","length_sec":227,"provider":"hls"}],"references":[{"position":1,"raw":"[Bender et al., 2018] Gabriel Bender, Pieter-Jan Kinder- mans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Un- derstanding and simplifying one-shot architecture search. In Proceedings of the 35th International Conference on Machine Learning, pages 549-558, Stockholm, Sweden, July 2018.","title":"Understanding and simplifying one-shot architecture search","authors":"Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, Quoc Le","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bi et al., 2019] Kaifeng Bi, Changping Hu, Lingxi Xie, Xin Chen, Longhui Wei, and Qi Tian. Stabilizing DARTS with amended gradient estimation on architectural parameters. arXiv preprint arXiv:1910.11831, 2019.","title":"Stabilizing DARTS with amended gradient estimation on architectural parameters","authors":"Bi","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chang et al., 2019] Jianlong Chang, Xinbang Zhang, Yi- wen Guo, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Differentiable architecture search with ensemble gumbel-softmax. arXiv preprint arXiv:1905.01786, 2019.","title":"Differentiable architecture search with ensemble gumbel-softmax","authors":"Jianlong Chang, Xinbang Zhang, Yiwen Guo, Gaofeng Meng, Shiming Xiang, Chunhong Pan","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2019] Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1294-1303, Seoul, Korea, October 2019.","title":"Progressive differentiable architecture search: Bridging the depth gap between search and evaluation","authors":"Xin Chen, Lingxi Xie, Jun Wu, Qi Tian","year":"2019","venue":"IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chu et al., 2019] Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li. Fair DARTS: Eliminat- ing unfair advantages in differentiable architecture search. arXiv preprint arXiv:1911.12126, 2019.","title":"Fair DARTS: Eliminating unfair advantages in differentiable architecture search","authors":"Xiangxiang Chu, Tianbao Zhou, Bo Zhang, Jixiang Li","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Fei-Fei Li. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, Miami, FL, USA, June 2009.","title":"ImageNet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Fei-Fei Li","year":"2009","venue":"2009 IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Elsken et al., 2019] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objective neural archi- tecture search via Lamarckian evolution. In 7th Interna- tional Conference on Learning Representations, New Or- leans, LA, USA, May 2019.","title":"Efficient multi-objective neural architecture search via Lamarckian evolution","authors":"Thomas Elsken, Jan Hendrik Metzen, Frank Hutter","year":"2019","venue":"7th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Howard et al., 2017] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mo- bileNets: Efficient convolutional neural networks for mo- bile vision applications. arXiv preprint arXiv:1704.04861, 2017.","title":"Mo-bileNets: Efficient convolutional neural networks for mobile vision applications","authors":"Menglong Andrew G Howard, Bo Zhu, Dmitry Chen, Weijun Kalenichenko, Tobias Wang, Marco Weyand, Hartwig Andreetto, Adam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely con- nected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, pages 4700- 4708, Honolulu, HI, USA, July 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":"2017 IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liang et al., 2019] Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. Darts+: Improved differentiable archi- tecture search with early stopping. arXiv preprint arXiv:1909.06035, 2019.","title":"Darts+: Improved differentiable architecture search with early stopping","authors":"Liang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2018] Chenxi Liu, Barret Zoph, Maxim Neu- mann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Pro- gressive neural architecture search. In Proceedings of the European Conference on Computer Vision (ECCV), pages 19-34, Munich, Germany, September 2018.","title":"Progressive neural architecture search","authors":"Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, Kevin Murphy","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2019a] Hanxiao Liu, Karen Simonyan, and Yim- ing Yang. Darts: Differentiable architecture search. In 7th International Conference on Learning Representations, New Orleans, LA, USA, May 2019.","title":"Darts: Differentiable architecture search","authors":"Liu","year":"2019","venue":"7th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2019b] Yuhang Liu, Wenyong Dong, Lei Zhang, Dong Gong, and Qinfeng Shi. Variational Bayesian dropout with a hierarchical prior. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7124-7133, Long Beach, CA, USA, June 2019.","title":"Variational Bayesian dropout with a hierarchical prior","authors":"Liu","year":"2019","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Luo et al., 2018] Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture optimization. In Advances in Neural Information Processing Systems 31, pages 7816-7827, Montreal, QC, USA, December 2018.","title":"Neural architecture optimization","authors":"Luo","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ma et al., 2018] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. ShuffleNet V2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European Conference on Computer Vision (ECCV), pages 116-131, Munich, Germany, September 2018.","title":"ShuffleNet V2: Practical guidelines for efficient cnn architecture design","authors":"Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, Jian Sun","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Miikkulainen et al., 2019] Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Daniel Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, and Babak Hodjat. Evolving deep neural networks. In Artificial Intelligence in the Age of Neural Networks and Brain Computing, pages 293-312. Academic Press, 2019.","title":"Evolving deep neural networks","authors":"Jason Risto Miikkulainen, Elliot Liang, Aditya Meyerson, Daniel Rawal, Olivier Fink, Bala Francon, Hormoz Raju, Arshak Shahrzad, Nigel Navruzyan, Babak Duffy, Hodjat","year":"2019","venue":"Artificial Intelligence in the Age of Neural Networks and Brain Computing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Molchanov et al., 2017] Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout spar- sifies deep neural networks. In Proceedings of the 34th International Conference on Machine Learning, pages 2498-2507, Sydeny, NSW, Australia, August 2017.","title":"Variational dropout sparsifies deep neural networks","authors":"Dmitry Molchanov, Arsenii Ashukha, Dmitry Vetrov","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pham et al., 2018] Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural archi- tecture search via parameter sharing. In Proceedings of the 35th International Conference on Machine Learning, pages 4095-4104, Stockholm, Sweden, July 2018.","title":"Efficient neural architecture search via parameter sharing","authors":"Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, Jeff Dean","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Real et al., 2017] Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V. Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In Proceedings of the 34th International Conference on Machine Learning, pages 2902-2911, Syd- ney, NSW, Australia, August 2017.","title":"Large-scale evolution of image classifiers","authors":"Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, Alexey Kurakin","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Real et al., 2019] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, pages 4780- 4789, Honolulu, HI, USA, January 2019.","title":"Regularized evolution for image classifier architecture search","authors":"Esteban Real, Alok Aggarwal, Yanping Huang, Quoc V Le","year":"2019","venue":"Proceedings of the 33rd AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1958, 2014.","title":"Dropout: A simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-9, Boston, MA, USA, June 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xie et al., 2019] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In 7th International Conference on Learning Rep- resentations, New Orleans, LA, USA, May 2019.","title":"SNAS: stochastic neural architecture search","authors":"Sirui Xie, Hehui Zheng, Chunxiao Liu, Liang Lin","year":"2019","venue":"7th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2019] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pc- darts: Partial channel connections for memory-efficient ar- chitecture search. In International Conference on Learn- ing Representations, 2019.","title":"Pcdarts: Partial channel connections for memory-efficient architecture search","authors":"Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, Hongkai Xiong","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zheng et al., 2019] Xiawu Zheng, Rongrong Ji, Lang Tang, Baochang Zhang, Jianzhuang Liu, and Qi Tian. Multino- mial distribution learning for effective neural architecture search. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1304-1313, Seoul, Korea, October 2019.","title":"Multinomial distribution learning for effective neural architecture search","authors":"Zheng","year":"2019","venue":"IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhou et al., 2019] Hongpeng Zhou, Minghao Yang, Jun Wang, and Wei Pan. BayesNAS: A Bayesian approach for neural architecture search. In Proceedings of the 36th International Conference on Machine Learning, pages 7603-7613, Long Beach, CA, USA, June 2019.","title":"BayesNAS: A Bayesian approach for neural architecture search","authors":"Zhou","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zoph and Le, 2017] Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In 5th International Conference on Learning Representations, Toulon, France, April 2017.","title":"Neural architecture search with reinforcement learning","authors":"Le ; Zoph, Barret Zoph, Quoc V Le","year":"2017","venue":"5th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zoph et al., 2018] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architec- tures for scalable image recognition. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8697-8710, Salt Lake City, UT, USA, June 2018.","title":"Learning transferable architectures for scalable image recognition","authors":"Barret Zoph, Vijay Vasudevan, Jonathon Shlens, V Quoc, Le","year":"2018","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}