{"type":"ScholarlyArticle","title":"Deep Neural Networks via Complex Network Theory: A Perspective","authors":[{"name":"Emanuele La Malfa","slug":"emanuele-la-malfa","given":"Emanuele","family":"La Malfa","affiliation":"University of Oxford"},{"name":"Gabriele La Malfa","slug":"gabriele-la-malfa","given":"Gabriele","family":"La Malfa","affiliation":"King's College London"},{"name":"Giuseppe Nicosia","slug":"giuseppe-nicosia","given":"Giuseppe","family":"Nicosia","affiliation":"University of Catania"},{"name":"Vito Latora","slug":"vito-latora","given":"Vito","family":"Latora","affiliation":"University of Catania; Queen Mary University of London"}],"abstract":"Deep Neural Networks (DNNs) can be represented as graphs whose links and vertices iteratively process data and solve tasks sub-optimally. Complex Network Theory (CNT), merging statistical physics with graph theory, provides a method for interpreting neural networks by analysing their weights and neuron structures. However, classic works adapt CNT metrics that only permit a topological analysis as they do not account for the effect of the input data. In addition, CNT metrics have been applied to a limited range of architectures, mainly including Fully Connected neural networks. \r\nIn this work, we extend the existing CNT metrics with measures that sample from the DNNs' training distribution, shifting from a purely topological analysis to one that connects with the interpretability of deep learning. For the novel metrics, in addition to the existing ones, we provide a mathematical formalisation for Fully Connected, AutoEncoder, Convolutional and Recurrent neural networks, of which we vary the activation functions and the number of hidden layers. We show that these metrics differentiate DNNs based on the architecture, the number of hidden layers, and the activation function. Our contribution provides a method rooted in physics for interpreting DNNs that offers insights beyond the traditional input-output relationship and the CNT topological analysis.","doi":"10.24963/ijcai.2024/482","url":"https://ijcai.org/proceedings/2024/482","pdf":"https://ijcai.org/proceedings/2024/0482.pdf","markdown":"https://ijcai.org/proceedings/2024/482.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/482","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":482,"pages":{"first":4361,"last":4369},"keywords":["Machine Learning: ML: Other","Machine Learning: ML: Applications","Machine Learning: ML: Explainable/Interpretable machine learning"],"videos":[],"references":[{"position":1,"raw":"[Bach et al., 2015] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130140, 2015.","title":"On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation","authors":"Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, Wojciech Samek","year":"2015","venue":"PloS one","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bishop, 1995] Christopher M. Bishop. Neural networks for pattern recognition. Oxford University Press, USA, 1995.","title":"Neural networks for pattern recognition","authors":"Christopher M Bishop","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Boccaletti et al., 2006] S. Boccaletti, V. Latora, Moreno Y, M. Chavez, and D. Hwang. Complex networks: Structure and dynamics. Physics Reports, 424(4-5):175-308, Febru- ary 2006.","title":"Complex networks: Structure and dynamics","authors":"S Boccaletti, V Latora, Y Moreno, M Chavez, D Hwang","year":"2006","venue":"Physics Reports","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chavez et al., 2010] Maisa Chavez, Miguel Valencia, Valy Navarro, Vito Latora, and J. Martinerie. Functional mod- ularity of background activities in normal and epileptic brain networks. Physical Review Letters, 104(11), March 2010.","title":"Functional modularity of background activities in normal and epileptic brain networks","authors":"Maisa Chavez, Miguel Valencia, Valy Navarro, Vito Latora, J Martinerie","year":"2010","venue":"Physical Review Letters","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Crucitti et al., 2004] Paolo Crucitti, Vito Latora, and Mas- simo Marchiori. A topological analysis of the italian elec- tric power grid. Physica A: Statistical Mechanics and its Applications, 338(1-2):92-97, July 2004.","title":"A topological analysis of the italian electric power grid","authors":"Paolo Crucitti, Vito Latora, Massimo Marchiori","year":"2004","venue":"Physica A: Statistical Mechanics and its Applications","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Erhan et al., 2009] Dumitru Erhan, Y. Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. Technical Report, Univeristé de Montréal, 01 2009.","title":"Visualizing higher-layer features of a deep network","authors":"Y Dumitru Erhan, Aaron Bengio, Pascal Courville, Vincent","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ghorbani et al., 2019] Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is frag- ile. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3681-3688, 2019.","title":"Interpretation of neural networks is fragile","authors":"Ghorbani","year":"2019","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org.","title":"Deep Learning","authors":"Ian Goodfellow, Yoshua Bengio, Aaron Courville","year":"2016","venue":null,"doi":null,"url":"http://www.deeplearningbook.org","internal_url":null},{"position":9,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Karpathy et al., 2015] Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and understanding recurrent networks, 2015.","title":"Visualizing and understanding recurrent networks","authors":"Andrej Karpathy, Justin Johnson, Li Fei-Fei","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kipf and Welling, 2017] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks, 2017.","title":"Semi-supervised classification with graph convolutional networks","authors":"Thomas N Kipf, Max Welling","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky et al., 2010] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). URL http://www. cs. toronto. edu/kriz/cifar. html, 5(4):1, 2010.","title":"Cifar-10 (canadian institute for advanced research","authors":"Alex Krizhevsky, Vinod Nair, Geoffrey Hinton","year":"2010","venue":null,"doi":null,"url":"http://www.cs.toronto.edu/kriz/cifar.html,5","internal_url":null},{"position":13,"raw":"[La Malfa et al., 2021] Emanuele La Malfa, Gabriele La Malfa, Giuseppe Nicosia, and Vito Latora. Charac- terizing learning dynamics of deep neural networks via complex networks. In 2021 IEEE 33rd International Conference on Tools with Artificial Intelligence (ICTAI), pages 344-351. IEEE, 2021.","title":"Characterizing learning dynamics of deep neural networks via complex networks","authors":"La Emanuele, Gabriele La Malfa, Giuseppe Malfa, Vito Nicosia, Latora","year":"2021","venue":"IEEE 33rd International Conference on Tools with Artificial Intelligence (ICTAI)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lecun and Bengio, 1995] Yann Lecun and Yoshua Bengio. Convolutional networks for images, speech, and time- series. MIT Press, 1995.","title":"Convolutional networks for images, speech, and timeseries","authors":"Bengio Lecun, Yann Lecun, Yoshua Bengio","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Montavon et al., 2018] Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Methods for inter- preting and understanding deep neural networks. Digital signal processing, 73:1-15, 2018.","title":"Methods for interpreting and understanding deep neural networks","authors":"Grégoire Montavon, Wojciech Samek, Klaus-Robert Müller","year":"2018","venue":"Digital signal processing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Montavon et al., 2019] Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojciech Samek, and Klaus-Robert Müller. Layer-Wise Relevance Propagation: An Overview, pages 193-209. Springer International Publishing, Cham, 2019.","title":"Layer-Wise Relevance Propagation: An Overview","authors":"Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojciech Samek, Klaus-Robert Müller","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Petri et al., 2021] Giovanni Petri, Sebastian Musslick, Biswadip Dey, Kayhan Özcimder, David Turner, Ahmed Nesreen K., Willke Theodore L., and Cohen Jonathan D. Topological limits to the parallel processing capability of network architectures. Nature Physics, 17(5):646-651, May 2021.","title":"Topological limits to the parallel processing capability of network architectures","authors":"Giovanni Petri, Sebastian Musslick, Biswadip Dey, Kayhan Özcimder, David Turner, Ahmed Nesreen, K, Willke Theodore, L, Cohen Jonathan, D","year":"2021","venue":"Nature Physics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Porta et al., 2006a] Sergio Porta, Paolo Crucitti, and Vito Latora. The network analysis of urban streets: A dual approach. Physica A: Statistical Mechanics and its Ap- plications, 369(2):853-866, September 2006.","title":"The network analysis of urban streets: A dual approach","authors":"Porta","year":"2006","venue":"Physica A: Statistical Mechanics and its Applications","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Porta et al., 2006b] Sergio Porta, Paolo Crucitti, and Vito Latora. The network analysis of urban streets: a primal approach. Environment and Planning B: planning and de- sign, 33(5):705-725, 2006.","title":"The network analysis of urban streets: a primal approach","authors":"Porta","year":"2006","venue":"Environment and Planning B: planning and design","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Saxe et al., 2022] Andrew Saxe, Shagun Sodhani, and Sam Jay Lewallen. The neural race reduction: Dynam- ics of abstraction in gated networks. In Kamalika Chaud- huri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 19287-19309. PMLR, 17-23 Jul 2022.","title":"The neural race reduction: Dynamics of abstraction in gated networks","authors":"Andrew Saxe, Shagun Sodhani, Sam Jay Lewallen","year":"2022","venue":"Proceedings of the 39th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Scabini et al., 2022] Leonardo Scabini, Bernard De Baets, and O. M. Bruno. Improving deep neural network random initialization through neuronal rewiring, 2022.","title":"Improving deep neural network random initialization through neuronal rewiring","authors":"Leonardo Scabini, Bernard De Baets, O M Bruno","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schmidhuber, 2015] Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural Networks, 61:85- 117, 2015.","title":"Deep learning in neural networks: An overview","authors":"Jürgen Schmidhuber, Schmidhuber","year":"2015","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Singh et al., 2018] Chandan Singh, W James Murdoch, and Bin Yu. Hierarchical interpretations for neural network predictions. arXiv preprint arXiv:1806.05337, 2018.","title":"Hierarchical interpretations for neural network predictions","authors":"Chandan Singh, James Murdoch, Bin Yu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Testolin et al., 2019] A. Testolin, M. Piccolini, and S. Suweis. Deep learning systems as complex networks. Journal of Complex Networks, June 2019.","title":"Deep learning systems as complex networks","authors":"A Testolin, M Piccolini, S Suweis","year":"2019","venue":"Journal of Complex Networks","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems 30: 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in Neural Information Processing Systems 30","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zambra et al., 2020] Matteo Zambra, Amos Maritan, and Alberto Testolin. Emergence of network motifs in deep neural networks. Entropy, 22(2):204, February 2020.","title":"Emergence of network motifs in deep neural networks","authors":"Matteo Zambra, Amos Maritan, Alberto Testolin","year":"2020","venue":"Entropy","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zeiler and Fergus, 2014] Matthew D. Zeiler and Rob Fer- gus. Visualizing and understanding convolutional net- works. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision -ECCV 2014, pages 818-833. Springer International Publishing, 2014.","title":"Visualizing and understanding convolutional networks","authors":"Matthew D Zeiler, Rob Fergus","year":"2014","venue":"Computer Vision -ECCV 2014","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}