{"type":"ScholarlyArticle","title":"A Genetic Programming Approach to Designing Convolutional Neural Network Architectures","authors":[{"name":"Masanori Suganuma","slug":"masanori-suganuma","given":"Masanori","family":"Suganuma","affiliation":"RIKEN Center for AIP; Tohoku University"},{"name":"Shinichi Shirakawa","slug":"shinichi-shirakawa","given":"Shinichi","family":"Shirakawa","affiliation":"Yokohama National University"},{"name":"Tomoharu Nagao","slug":"tomoharu-nagao","given":"Tomoharu","family":"Nagao","affiliation":"Yokohama National University"}],"abstract":"We propose a method for designing convolutional neural network (CNN) architectures based on Cartesian genetic programming (CGP). In the proposed method, the architectures of CNNs are represented by directed acyclic graphs, in which each node represents highly-functional modules such as convolutional blocks and tensor operations, and each edge represents the connectivity of layers. The architecture is optimized to maximize the classification accuracy for a validation dataset by an evolutionary algorithm. We show that the proposed method can find competitive CNN architectures compared with state-of-the-art methods on the image classification task using CIFAR-10 and CIFAR-100 datasets.","doi":"10.24963/ijcai.2018/755","url":"https://ijcai.org/proceedings/2018/755","pdf":"https://ijcai.org/proceedings/2018/0755.pdf","markdown":"https://ijcai.org/proceedings/2018/755.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/755","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Sister Conferences Best Papers","track":"Best Sister Conferences","number":755,"pages":{"first":5369,"last":5373},"keywords":["Machine Learning: Neural Networks","Machine Learning: Deep Learning","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Baker et al., 2017] Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network ar- chitectures using reinforcement learning. In ICLR, 2017.","title":"Designing neural network architectures using reinforcement learning","authors":"Bowen Baker, Otkrist Gupta, Nikhil Naik, Ramesh Raskar","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai et al., 2018] Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In AAAI, pages 2787-2794, 2018.","title":"Efficient architecture search by network transformation","authors":"Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, Jun Wang","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In ICCV, pages 1026-1034, 2015.","title":"Delving deep into rectifiers: Surpassing human-level performance on imagenet classification","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In ICML, pages 448-456, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kingma and Ba, 2015] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Larsson et al., 2017] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. In ICLR, 2017.","title":"Fractalnet: Ultra-deep neural networks without residuals","authors":"Larsson","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, November 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lin et al., 2014] Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. In ICLR, 2014.","title":"Network in network","authors":"Lin","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2017] Chenxi Liu, Barret Zoph, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architec- ture search. arXiv:1712.00559, 2017.","title":"Progressive neural architecture search","authors":"Chenxi Liu, Barret Zoph, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, Kevin Murphy","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2018] Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hierarchical representations for efficient architecture search. In ICLR, 2018.","title":"Hierarchical representations for efficient architecture search","authors":"Liu","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Miikkulainen et al., 2017] Risto Miikkulainen, Jason Zhi Liang, Elliot Meyerson, Aditya Rawal, Dan Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, and Babak Hodjat. Evolving deep neural networks. In GECCO, 2017.","title":"Evolving deep neural networks","authors":"Jason Zhi Risto Miikkulainen, Elliot Liang, Aditya Meyerson, Dan Rawal, Olivier Fink, Bala Francon, Hormoz Raju, Arshak Shahrzad, Nigel Navruzyan, Babak Duffy, Hodjat","year":"2017","venue":"GECCO","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Miller and Smith, 2006] Julian F. Miller and Stephen L. Smith. Redundancy and computational efficiency in carte- sian genetic programming. IEEE Transactions on Evolu- tionary Computation, 10(2):167-174, April 2006.","title":"Redundancy and computational efficiency in cartesian genetic programming","authors":"Julian F Miller, Stephen L Smith","year":"2006","venue":"IEEE Transactions on Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Miller and Thomson, 2000] Julian F. Miller and Peter Thomson. Cartesian genetic programming. In EuroGP, pages 121-132, 2000.","title":"Cartesian genetic programming","authors":"Julian F Miller, Peter Thomson","year":"2000","venue":"EuroGP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nair and Hinton, 2010] Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann ma- chines. In ICML, pages 807-814, 2010.","title":"Rectified linear units improve restricted boltzmann machines","authors":"Vinod Nair, Geoffrey E Hinton","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pham et al., 2018] Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural ar- chitecture search via parameter sharing. arXiv preprint arXiv:1802.03268, 2018.","title":"Efficient neural architecture search via parameter sharing","authors":"Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, Jeff Dean","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Real et al., 2017] Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka L. Suematsu, Jie Tan, Quoc V. Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In ICML, pages 2902-2911, 2017.","title":"Large-scale evolution of image classifiers","authors":"Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka L Suematsu, Jie Tan, Quoc V Le, Alexey Kurakin","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Real et al., 2018] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. arXiv preprint arXiv:1802.01548, 2018.","title":"Regularized evolution for image classifier architecture search","authors":"Esteban Real, Alok Aggarwal, Yanping Huang, V Quoc, Le","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Stanley and Miikkulainen, 2002] Kenneth O. Stanley and Risto Miikkulainen. Evolving neural networks through augmenting topologies. Evolutionary computation, 10(2):99-127, 2002.","title":"Evolving neural networks through augmenting topologies","authors":"Kenneth O Stanley, Risto Miikkulainen","year":"2002","venue":"Evolutionary computation","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Suganuma et al., 2017a] Masanori Suganuma, Shinichi Shirakawa, and Tomoharu Nagao. A genetic program- ming approach to designing convolutional neural network architectures. In GECCO, pages 497-504, 2017.","title":"A genetic programming approach to designing convolutional neural network architectures","authors":"Suganuma","year":"2017","venue":"GECCO","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Suganuma et al., 2017b] Masanori Suganuma, Shinichi Shirakawa, and Tomoharu Nagao. A genetic program- ming approach to designing convolutional neural network architectures. arXiv preprint arXiv:1704.00764, 2017.","title":"A genetic programming approach to designing convolutional neural network architectures","authors":"Suganuma","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Suganuma et al., 2018] Masanori Suganuma, Mete Ozay, and Takayuki Okatani. Exploiting the potential of standard convolutional autoencoders for image restoration by evo- lutionary search. arXiv preprint arXiv:1803.00370, 2018.","title":"Exploiting the potential of standard convolutional autoencoders for image restoration by evolutionary search","authors":"Masanori Suganuma, Mete Ozay, Takayuki Okatani","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In CVPR, pages 1-9, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xie and Yuille, 2017] Lingxi Xie and Alan L. Yuille. Ge- netic CNN. In ICCV, pages 1379-1388, 2017.","title":"Genetic CNN","authors":"Yuille Xie, Lingxi Xie, Alan L Yuille","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zagoruyko and Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, pages 87.1-87.12, 2016.","title":"Sergey Zagoruyko and Nikos Komodakis. Wide residual networks","authors":"Komodakis Zagoruyko","year":"2016","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhong et al., 2017] Zhao Zhong, Junjie Yan, and Cheng- Lin Liu. Practical network blocks design with Q-Learning. In arXiv: 1708.05552, 2017.","title":"Practical network blocks design with Q-Learning","authors":"Zhao Zhong, Junjie Yan, Cheng-Lin Liu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zoph and Le, 2017] Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In ICLR, 2017.","title":"Neural architecture search with reinforcement learning","authors":"Le ; Zoph, Barret Zoph, Quoc V Le","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zoph et al., 2017] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architec- tures for scalable image recognition. arXiv:1707.07012, 2017.","title":"Learning transferable architectures for scalable image recognition","authors":"Barret Zoph, Vijay Vasudevan, Jonathon Shlens, V Quoc, Le","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}