{"type":"ScholarlyArticle","title":"Tessellation-Filtering ReLU Neural Networks","authors":[{"name":"Bernhard A. Moser","slug":"bernhard-a-moser","given":"Bernhard A.","family":"Moser","affiliation":"Software Competence Center Hagenberg GmbH (SCCH) , Austria"},{"name":"Michal Lewandowski","slug":"michal-lewandowski","given":"Michal","family":"Lewandowski","affiliation":"Software Competence Center Hagenberg GmbH (SCCH) , Austria"},{"name":"Somayeh Kargaran","slug":"somayeh-kargaran","given":"Somayeh","family":"Kargaran","affiliation":"Software Competence Center Hagenberg GmbH (SCCH) , Austria"},{"name":"Werner Zellinger","slug":"werner-zellinger","given":"Werner","family":"Zellinger","affiliation":"Software Competence Center Hagenberg GmbH (SCCH) , Austria"},{"name":"Battista Biggio","slug":"battista-biggio","given":"Battista","family":"Biggio","affiliation":"Pattern Recognition and Application Lab (PRA Lab) at University of Cagliari , Italy"},{"name":"Christoph Koutschan","slug":"christoph-koutschan","given":"Christoph","family":"Koutschan","affiliation":"Radon Inst. for Computational and Applied Mathematics (RICAM) at Austrian Academy of Sciences"}],"abstract":"We identify tessellation-filtering  ReLU  neural networks that, when composed with another ReLU network, keep its non-redundant tessellation unchanged or reduce it.The additional network complexity modifies the shape  of the decision surface without increasing the number of linear regions. We provide a mathematical understanding of the related additional expressiveness by means of a novel measure of shape complexity by counting deviations from convexity which results in a Boolean algebraic characterization of this special class.  A local representation theorem gives rise to novel approaches for pruning and decision surface analysis.","doi":"10.24963/ijcai.2022/463","url":"https://ijcai.org/proceedings/2022/463","pdf":"https://ijcai.org/proceedings/2022/0463.pdf","markdown":"https://ijcai.org/proceedings/2022/463.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/463","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":463,"pages":{"first":3335,"last":3341},"keywords":["Machine Learning: Theory of Deep Learning","Machine Learning: Learning Theory"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986926","length_sec":84,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985069","length_sec":357,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Alfarra et al., 2020] Motasem Alfarra, Adel Bibi, Hasan Hammoud, Mohamed Gaafar, and Bernard Ghanem. On the decision boundaries of neural networks: A tropical ge- ometry perspective. arXiv, 2002.08838, 2020.","title":"On the decision boundaries of neural networks: A tropical geometry perspective","authors":"Alfarra","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arora et al., 2018] Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee. Understanding deep neu- ral networks with rectified linear units. In ICLR 2018, Van- couver, BC, Canada, April 30 -May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.","title":"Understanding deep neural networks with rectified linear units","authors":"Raman Arora, Amitabh Basu, Poorya Mianjy, Anirbit Mukherjee","year":"2018","venue":"Conference Track Proceedings. OpenReview.net","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bianchini and Scarselli, 2014] Monica Bianchini and Franco Scarselli. On the complexity of neural network classifiers: A comparison between shallow and deep architectures. IEEE Trans. Neural Networks Learn. Syst., 25(8):1553-1565, 2014.","title":"On the complexity of neural network classifiers: A comparison between shallow and deep architectures","authors":"Scarselli Bianchini, Monica Bianchini, Franco Scarselli","year":"2014","venue":"IEEE Trans. Neural Networks Learn. Syst","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Glorot et al., 2011] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Geoffrey G., D. Dunson, and M. Dudik, editors, Proc. 14th Int. Conf. on Artificial Intelligence and Statistics, vol- ume 15 of Proceedings of Machine Learning Research, pages 315-323, Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR.","title":"Deep sparse rectifier neural networks","authors":"Xavier Glorot, Antoine Bordes, Yoshua Bengio","year":"2011","venue":"Proc. 14th Int. Conf. on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hahnloser et al., 2000] Richard Hahnloser, R. Sarpeshkar, M. Mahowald, and Rodney Douglas. Digital selection and analog amplification co-exist in an electronic circuit in- spired by neocortex. Nature, 405:947-951, 01 2000.","title":"Digital selection and analog amplification co-exist in an electronic circuit inspired by neocortex","authors":"Richard Hahnloser, R Sarpeshkar, M Mahowald, Rodney Douglas","year":"2000","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hanin and Rolnick, 2019] Boris Hanin and David Rolnick. Complexity of linear regions in deep networks. In Ka- malika Chaudhuri and Ruslan Salakhutdinov, editors, Pro- ceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Cali- fornia, USA, volume 97 of Proceedings of Machine Learn- ing Research, pages 2596-2604. PMLR, 2019.","title":"Complexity of linear regions in deep networks","authors":"Rolnick Hanin, Boris Hanin, David Rolnick","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning, ICML 2019","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"X Kaiming He, Shaoqing Zhang, Jian Ren, Sun","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hein et al., 2018] Matthias Hein, Maksym An- driushchenko, and Julian Bitterwolf. Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem. arxiv, 1812.05720v2, 2018.","title":"Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem","authors":"Matthias Hein, Maksym Andriushchenko, Julian Bitterwolf","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kahn and Park, 2022] Jeff Kahn and Jinyoung Park. The number of maximal independent sets in the Hamming cube. Combinatorica, March 2022.","title":"The number of maximal independent sets in the Hamming cube","authors":"Jeff Kahn, Jinyoung Park","year":"2022","venue":"Combinatorica","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Montúfar et al., 2014] Guido F Montúfar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Wein- berger, editors, Advances in Neural Information Process- ing Systems, volume 27.","title":"On the number of linear regions of deep neural networks","authors":"Razvan Guido F Montúfar, Kyunghyun Pascanu, Yoshua Cho, Bengio","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Curran Associates, Inc., 2014.","title":null,"authors":"Curran Associates, Inc","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Montúfar et al., 2021] Guido Montúfar, Yue Ren, and Leon Zhang. Sharp bounds for the number of regions of max- out networks and vertices of Minkowski sums. CoRR, abs/2104.08135, 2021.","title":"Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sums","authors":"Guido Montúfar, Leon Yue Ren, Zhang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Peterson, 1969] F. P. Peterson. Review: Edwin H. Spanier, Algebraic topology. Bulletin of the American Mathemati- cal Society, 75(5):916 -917, 1969.","title":"Review: Edwin H. Spanier, Algebraic topology","authors":"F P Peterson","year":"1969","venue":"Bulletin of the American Mathematical Society","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Raghu et al., 2017] Maithra Raghu, Ben Poole, Jon Klein- berg, Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, vol- ume 70 of Proceedings of Machine Learning Research, pages 2847-2854. PMLR, 06-11 Aug 2017.","title":"On the expressive power of deep neural networks","authors":"Raghu","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Serra et al., 2018] Thiago Serra, Christian Tjandraatmadja, and Srikumar Ramalingam. Bounding and counting lin- ear regions of deep neural networks. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th Inter- national Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4558- 4566. PMLR, 10-15 Jul 2018.","title":"Bounding and counting linear regions of deep neural networks","authors":"Thiago Serra, Christian Tjandraatmadja, Srikumar Ramalingam","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shepeleva et al., 2020] Natalia Shepeleva, Werner Zellinger, Michal Lewandowski, and Bernhard Moser. ReLU code space: A basis for rating network quality besides accuracy. ICLR 2020 Workshop on Neu- ral Architecture Search (NAS 2020), arXiv preprint arXiv:2005.09903, 2020.","title":"ReLU code space: A basis for rating network quality besides accuracy","authors":"Natalia Shepeleva, Werner Zellinger, Michal Lewandowski, Bernhard Moser","year":"2020","venue":"ICLR 2020 Workshop on Neural Architecture Search","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, ICLR 2015, San Diego, CA, USA, May 7- 9, 2015, Conference Track Proceedings, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"Conference Track Proceedings","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Szandała, 2021] Tomasz Szandała. Review and compari- son of commonly used activation functions for deep neu- ral networks. In Akash Kumar Bhoi et al., editor, Bio- inspired Neurocomputing, pages 203-224. Springer Sin- gapore, 2021.","title":"Review and comparison of commonly used activation functions for deep neural networks","authors":"Tomasz Szandała","year":"2021","venue":"Bioinspired Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, and et al. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 1-9. IEEE Com- puter Society, 2015.","title":"Going deeper with convolutions","authors":"Szegedy","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Trimmel et al., 2021] Martin Trimmel, Henning Petzka, and Cristian Sminchisescu. Tropex: An algorithm for ex- tracting linear terms in deep neural networks. In Interna- tional Conference on Learning Representations, 2021.","title":"Tropex: An algorithm for extracting linear terms in deep neural networks","authors":"Martin Trimmel, Henning Petzka, Cristian Sminchisescu","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang et al., 2018] Liwen Zhang, Gregory Naitzat, and Lek-Heng Lim. Tropical geometry of deep neural net- works. In Jennifer Dy and Andreas Krause, editors, Pro- ceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 5824-5832. PMLR, 10-15 Jul 2018.","title":"Tropical geometry of deep neural networks","authors":"Liwen Zhang, Gregory Naitzat, Lek-Heng Lim","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}