{"type":"ScholarlyArticle","title":"Estimation and Comparison of Linear Regions for ReLU Networks","authors":[{"name":"Yuan Wang","slug":"yuan-wang","given":"Yuan","family":"Wang","affiliation":""}],"abstract":"We study the relationship between the arrangement of neurons and the complexity of the ReLU-activated neural networks measured by the number of linear regions. More specifically, we provide both theoretical and empirical evidence for the point of view that shallow networks tend to have higher complexity than deep ones when the total number of neurons is fixed. In the theoretical part, we prove that this is the case for networks whose neurons in the hidden layers are arranged in the forms of 1x2n, 2xn and nx2; in the empirical part, we implement an algorithm that precisely tracks (hence counts) all the linear regions, and run it on networks with various structures. Although the time complexity of the algorithm is quite high, we verify that the problem of calculating the number of linear regions of a ReLU network is itself NP-hard. So currently there is no surprisingly efficient way to solve it. Roughly speaking, in the algorithm we divide the linear regions into subregions called the \"activation regions\", which are convex and easy to propagate through the network. The relationship between the number of the linear regions and that of the activation regions is also discussed.","doi":"10.24963/ijcai.2022/492","url":"https://ijcai.org/proceedings/2022/492","pdf":"https://ijcai.org/proceedings/2022/0492.pdf","markdown":"https://ijcai.org/proceedings/2022/492.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/492","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":492,"pages":{"first":3544,"last":3550},"keywords":["Machine Learning: Theory of Deep Learning","Machine Learning: Learning Theory","Machine Learning: Optimisation"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986710","length_sec":88,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984853","length_sec":353,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Google, 2016] Google. Tensorflow playground. https:// playground.tensorflow.org/, 2016. Accessed: 2022-05-08.","title":"Google. Tensorflow playground","authors":null,"year":"2016","venue":null,"doi":null,"url":"https://playground.tensorflow.org/,2016","internal_url":null},{"position":2,"raw":"[Hanin and Rolnick, 2019a] Boris Hanin and David Rolnick. Complexity of linear regions in deep networks. Interna- tional Conference on Machine Learning (ICML), 2019.","title":"Complexity of linear regions in deep networks","authors":"Rolnick Hanin, Boris Hanin, David Rolnick","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Hanin and Rolnick, 2019b] Boris Hanin and David Rolnick. Deep relu networks has surprisingly few activation pat- terns. Neural Information Processing Systems (NIPS), 2019.","title":"Deep relu networks has surprisingly few activation patterns","authors":"Rolnick Hanin, Boris Hanin, David Rolnick","year":"2019","venue":"Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hunter et al., 2012] David Hunter, Hao Yu, Michael S. Puk- ish, Janusz Kolbusz, and Bogdan M. Wilamowski. Selec- tion of proper neural network sizes and architectures-a comparative study. IEEE Transactions on Industrial In- formatics, 8(2):228-240, 2012.","title":"Wilamowski. Selection of proper neural network sizes and architectures-a comparative study","authors":"David Hunter, Hao Yu, Michael S Pukish, Janusz Kolbusz, Bogdan M","year":"2012","venue":"IEEE Transactions on Industrial Informatics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[LeCun et al., 2010] Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2, 2010.","title":"MNIST handwritten digit database","authors":"Yann Lecun, Corinna Cortes, Burges","year":"2010","venue":null,"doi":null,"url":"http://yann.lecun.com/exdb/mnist,2","internal_url":null},{"position":6,"raw":"[Lei et al., 2020] Na Lei, Zhongxuan Luo, Shing-Tung Yau, and David Xianfeng Gu. Geometric understanding of deep learning. Engineering, 2020.","title":"Geometric understanding of deep learning","authors":"Na Lei, Zhongxuan Luo, Shing-Tung Yau, David Xianfeng Gu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Montúfar et al., 2014] Guido Montúfar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural network. Neural Information Processing Systems (NIPS), 2014.","title":"On the number of linear regions of deep neural network","authors":"Guido Montúfar, Razvan Pascanu, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":"Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Raghu et al., 2017] Maithra Raghu, Ben Poole, Jon Klen- berg, Surya Ganguli, and Jascha Sohl Dickstein. On the expressive power of deep neural networks. International Conference on Machine Learning (ICML), 2017.","title":"On the expressive power of deep neural networks","authors":"Raghu","year":"2017","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Serra et al., 2018] Thiago Serra, Christian Tjandraatmadja, and Srikumar Ramalingam. Bounding and counting linear regions of deep neural networks. International Conference on Machine Learning (ICML), 2018.","title":"Bounding and counting linear regions of deep neural networks","authors":"Thiago Serra, Christian Tjandraatmadja, Srikumar Ramalingam","year":"2018","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Sopena et al., 1999] Josep M. Sopena, Enrique Romero, and René Alquezar. Neural networks with periodic and monotonic activation functions: a comparative study in classification problems. Ninth International Conference on Artificial Neural Networks (ICANN 99), 1999.","title":"Neural networks with periodic and monotonic activation functions: a comparative study in classification problems","authors":"M Josep, Enrique Sopena, René Romero, Alquezar","year":"1999","venue":"Ninth International Conference on Artificial Neural Networks (ICANN 99)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Xiong et al., 2020] Huan Xiong, Lei Huang, Mengyang Yu, Li Liu, Fan Zhu, and Ling Shao. On the number of linear regions of convolutional neural networks. International Conference on Machine Learning (ICML), 2020.","title":"On the number of linear regions of convolutional neural networks","authors":"Huan Xiong, Lei Huang, Mengyang Yu, Li Liu, Fan Zhu, Ling Shao","year":"2020","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}