{"type":"ScholarlyArticle","title":"Truth Table Net: Scalable, Compact & Verifiable Neural Networks with a Dual Convolutional Small Boolean Circuit Networks Form","authors":[{"name":"Adrien Benamira","slug":"adrien-benamira","given":"Adrien","family":"Benamira","affiliation":"Nanyang Technological University"},{"name":"Thomas Peyrin","slug":"thomas-peyrin","given":"Thomas","family":"Peyrin","affiliation":"Nanyang Technological University"},{"name":"Trevor Yap","slug":"trevor-yap","given":"Trevor","family":"Yap","affiliation":"Nanyang Technological University"},{"name":"Tristan Guérand","slug":"tristan-guerand","given":"Tristan","family":"Guérand","affiliation":"Nanyang Technological University"},{"name":"Bryan Hooi","slug":"bryan-hooi","given":"Bryan","family":"Hooi","affiliation":"National University of Singapore"}],"abstract":"We introduce \"Truth Table net\"' (TTnet), a novel Deep Neural Network (DNN) architecture designed to provide excellent scalability/compactness trade-offs among DNNs, allowing in turn to tackle the DNN challenge of fast formal verification. TTnet is constructed using Learning Truth Table (LTT) filters, analogous to how a Deep Convolutional Neural Network (DCNN) is built upon convolutional filters. The differentiable LTT filters are unique by their dual form: they are both a neural network-based function and a small-sized truth table that can be computed within a practical time frame. This characteristic guarantees, by design and independently of the overall architecture, the ability to practically extract an efficient (in terms of the number of logical gates) and functionally equivalent Conjunctive Normal Form (CNF) Boolean logic gate implementation. This CNF circuit is even optimal when the LTT truth table's input bit size n < 12. In particular, TTnet architecture is the first differentiable DNN with as dual form a compact logic gate representation that can scale to datasets larger than CIFAR-10: we achieve an accuracy of 41% on the ImageNet dataset while ensuring that each LTT filter truth table is fully computable within 2^{16} operations. We further compare the compactness and scalability performances of TTnet Boolean logic circuit representation to state-of-the-art differentiable logic DNNs across tabular, MNIST, and CIFAR-10 datasets. We emphasize that TTnet is the first solution to the open problem of designing differentiable convolutional neural networks with an exact dual logic gate circuit representation, bridging the gap between symbolic AI and trainable DCNNs. Finally, as improving DNNs compactness in Boolean logic circuit form reduces the complexity of their formal verification, we demonstrate TTnet effectiveness in exact sound and complete formal verification. Notably, our model achieves robustness verification in 10ms vs 100s for traditional state-of-the-art DNNs solvers.","doi":"10.24963/ijcai.2024/2","url":"https://ijcai.org/proceedings/2024/2","pdf":"https://ijcai.org/proceedings/2024/0002.pdf","markdown":"https://ijcai.org/proceedings/2024/2.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/2","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":2,"pages":{"first":13,"last":21},"keywords":["AI Ethics, Trust, Fairness: ETF: Trustworthy AI","Machine Learning: ML: Convolutional networks","Agent-based and Multi-agent Systems: MAS: Formal verification, validation and synthesis","Constraint Satisfaction and Optimization: CSO: Satisfiabilty"],"videos":[],"references":[{"position":1,"raw":"[Agarwal et al., 2021] Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoffrey E Hinton. Neural additive models: Interpretable machine learning with neural nets. Advances in Neural Information Processing Systems (NeurIPS), 2021.","title":"Neural additive models: Interpretable machine learning with neural nets","authors":"Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, Geoffrey E Hinton","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[AI, 2023] NIST AI. Artificial Intelligence Risk Management Framework (AI RMF 1.0). 2023.","title":"NIST AI. Artificial Intelligence Risk Management Framework","authors":"Ai","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Arora and Barak, 2009] Sanjeev Arora and Boaz Barak. Computational complexity: a modern approach. Cam- bridge University Press, 2009.","title":"Computational complexity: a modern approach","authors":"Sanjeev Arora, Boaz Barak","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bello et al., 2021] Irwan Bello, William Fedus, Xianzhi Du, Ekin Dogus Cubuk, Aravind Srinivas, Tsung-Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Im- proved training and scaling strategies. Advances in Neural Information Processing Systems (NeurIPS), 2021.","title":"Revisiting resnets: Improved training and scaling strategies","authors":"Irwan Bello, William Fedus, Xianzhi Du, Ekin Dogus Cubuk, Aravind Srinivas, Tsung-Yi Lin, Jonathon Shlens, Barret Zoph","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Blake, 1938] Archie Blake. Corrections to Canonical ex- pressions in Boolean algebra. Journal of Symbolic Logic, 1938.","title":"Corrections to Canonical expressions in Boolean algebra","authors":"Archie Blake, Blake","year":"1938","venue":"Journal of Symbolic Logic","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brendel and Bethge, 2018] Wieland Brendel and Matthias Bethge. Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet. In Inter- national Conference on Learning Representations (ICLR), 2018.","title":"Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet","authors":"Wieland Brendel, Matthias Bethge","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Brix et al., 2023] Christopher Brix, Mark Niklas Müller, Stanley Bak, Taylor T Johnson, and Changliu Liu. First three years of the international verification of neural net- works competition (VNN-COMP). International Journal on Software Tools for Technology Transfer, 2023.","title":"First three years of the international verification of neural networks competition","authors":"Christopher Brix, Mark Niklas Müller, Stanley Bak, Taylor T Johnson, Changliu Liu","year":"2023","venue":"VNN-COMP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chatterjee, 2018] Satrajit Chatterjee. Learning and memo- rization. In International conference on machine learning (ICML), 2018.","title":"Learning and memorization","authors":"Satrajit Chatterjee","year":"2018","venue":"International conference on machine learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Daemen and Rijmen, 2002] Joan Daemen and Vincent Rij- men. The Design of Rijndael: AES -The Advanced Encryp- tion Standard. Springer, 2002.","title":"The Design of Rijndael: AES -The Advanced Encryption Standard","authors":"Rijmen Daemen","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Driscoll, 2020] Michael Driscoll. System and method for adapting a neural network model on a hardware platform, 2020. US Patent App. 16/728,884.","title":"System and method for adapting a neural network model on a hardware platform","authors":"Michael Driscoll","year":"2020","venue":"US Patent App","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT Press, 2016.","title":"Deep learning","authors":"Ian Goodfellow, Yoshua Bengio, Aaron Courville","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Han et al., 2016] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In International Conference on Learning Representations (ICLR), 2016.","title":"Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding","authors":"Han","year":"2016","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hirtzlin et al., 2019] Tifenn Hirtzlin, Bogdan Penkovsky, Marc Bocquet, Jacques-Olivier Klein, Jean-Michel Portal, and Damien Querlioz. Stochastic computing for hardware implementation of binarized neural networks. IEEE Access, 2019.","title":"Stochastic computing for hardware implementation of binarized neural networks","authors":"Hirtzlin","year":"2019","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hubara et al., 2016] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Bi- narized neural networks. Advances in Neural Information Processing Systems (NeurIPS), 2016.","title":"Binarized neural networks","authors":"Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ignatiev et al., 2018] Alexey Ignatiev, Antonio Morgado, and Joao Marques-Silva. PySAT: A Python Toolkit for Prototyping with SAT Oracles. In International Conference on Theory and Applications of Satisfiability Testing (SAT), 2018.","title":"PySAT: A Python Toolkit for Prototyping with SAT Oracles","authors":"Alexey Ignatiev, Antonio Morgado, Joao Marques-Silva","year":"2018","venue":"International Conference on Theory and Applications of Satisfiability Testing (SAT)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jia and Rinard, 2020] Kai Jia and Martin Rinard. Efficient exact verification of binarized neural networks. Advances in Neural Information Processing Systems (NeurIPS), 2020.","title":"Efficient exact verification of binarized neural networks","authors":"Kai Jia, Martin Rinard","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jia and Rinard, 2021] Kai Jia and Martin Rinard. Exploiting verified neural networks via floating point numerical error. In International Static Analysis Symposium, 2021.","title":"Exploiting verified neural networks via floating point numerical error","authors":"Rinard ; Jia, Kai Jia, Martin Rinard","year":"2021","venue":"International Static Analysis Symposium","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Klir et al., 1997] George J Klir, Ute St. Clair, and Bo Yuan. Fuzzy set theory: foundations and applications. Prentice- Hall, Inc, 1997.","title":"Fuzzy set theory: foundations and applications","authors":"Ute St George J Klir, Bo Clair, Yuan","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liffiton and Maglalang, 2012] Mark H Liffiton and Jor- dyn C Maglalang. A cardinality solver: More expressive constraints for free. In International Conference on Theory and Applications of Satisfiability Testing (SAT), 2012.","title":"A cardinality solver: More expressive constraints for free","authors":"H Mark, Jordyn C Liffiton, Maglalang","year":"2012","venue":"International Conference on Theory and Applications of Satisfiability Testing (SAT)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2018] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. Proceedings of the European Conference on Computer Vi- sion (ECCV), 2018.","title":"Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm","authors":"Liu","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mocanu et al., 2018] Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 2018.","title":"Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science","authors":"Mocanu","year":"2018","venue":"Nature communications","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Molchanov et al., 2017] Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In International conference on machine learning (ICML), 2017.","title":"Variational dropout sparsifies deep neural networks","authors":"Dmitry Molchanov, Arsenii Ashukha, Dmitry Vetrov","year":"2017","venue":"International conference on machine learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Müller et al., 2022] Mark Niklas Müller, Gleb Makarchuk, Gagandeep Singh, Markus Püschel, and Martin Vechev. PRIMA: general and precise neural network certification via scalable convex hull approximations. Proceedings of the ACM on Programming Languages, 2022.","title":"PRIMA: general and precise neural network certification via scalable convex hull approximations","authors":"Mark Niklas Müller, Gleb Makarchuk, Gagandeep Singh, Markus Püschel, Martin Vechev","year":"2022","venue":"Proceedings of the ACM on Programming Languages","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Narodytska et al., 2019] Nina Narodytska, Hongce Zhang, Aarti Gupta, and Toby Walsh. In search for a SAT-friendly binarized neural network architecture. In International Conference on Learning Representations (ICLR), 2019.","title":"In search for a SAT-friendly binarized neural network architecture","authors":"Nina Narodytska, Hongce Zhang, Aarti Gupta, Toby Walsh","year":"2019","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Paszke et al., 2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in Neural Information Processing Systems (NeurIPS), 2019.","title":"Pytorch: An imperative style, highperformance deep learning library","authors":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Petersen et al., 2022] Felix Petersen, Christian Borgelt, Hilde Kuehne, and Oliver Deussen. Deep differentiable logic gate networks. Advances in Neural Information Pro- cessing Systems (NeurIPS), 2022.","title":"Deep differentiable logic gate networks","authors":"Felix Petersen, Christian Borgelt, Hilde Kuehne, Oliver Deussen","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Rastegari et al., 2016] Mohammad Rastegari, Vicente Or- donez, Joseph Redmon, and Ali Farhadi. Xnor-net: Ima- genet classification using binary convolutional neural net- works. Proceedings of the European Conference on Com- puter Vision (ECCV), 2016.","title":"Xnor-net: Imagenet classification using binary convolutional neural networks","authors":"Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, Ali Farhadi","year":"2016","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Roussel and Manquinho, 2009] Olivier Roussel and Vasco Manquinho. Pseudo-Boolean and cardinality constraints. In Handbook of satisfiability. IOS Press, 2009.","title":"Pseudo-Boolean and cardinality constraints","authors":"Manquinho Roussel, Olivier Roussel, Vasco Manquinho","year":"2009","venue":"Handbook of satisfiability","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Rumelhart et al., 1986] David E Rumelhart, Geoffrey E Hin- ton, and Ronald J Williams. Learning representations by back-propagating errors. nature, 1986. [Udovenko, 2023] Aleksei Udovenko. DenseQMC: an ef- ficient bit-slice implementation of the Quine-McCluskey algorithm. arXiv, 2023.","title":"Learning representations by back-propagating errors","authors":"Geoffrey E David E Rumelhart, Ronald J Hinton, Williams","year":"1986","venue":"Aleksei Udovenko. DenseQMC: an efficient bit-slice implementation of the Quine-McCluskey algorithm","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Umuroglu et al., 2017] Yaman Umuroglu, Nicholas J Fraser, Giulio Gambardella, Michaela Blott, Philip Leong, Magnus Jahre, and Kees Vissers. Finn: A framework for fast, scal- able binarized neural network inference. In Proceedings of the ACM/SIGDA International Symposium On Field- Programmable Gate Arrays, 2017.","title":"Finn: A framework for fast, scalable binarized neural network inference","authors":"Yaman Umuroglu, Nicholas J Fraser, Giulio Gambardella, Michaela Blott, Philip Leong, Magnus Jahre, Kees Vissers","year":"2017","venue":"Proceedings of the ACM/SIGDA International Symposium On Field-Programmable Gate Arrays","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Van Der Walt et al., 2011] Stefan Van Der Walt, S Chris Col- bert, and Gael Varoquaux. The NumPy array: a structure for efficient numerical computation. Computing in Science & Engineering, 2011.","title":"The NumPy array: a structure for efficient numerical computation","authors":"Walt Van Der","year":"2011","venue":"Computing in Science & Engineering","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2019] Erwei Wang, James J Davis, Peter YK Cheung, and George A Constantinides. LUTNet: Rethink- ing inference in FPGA soft logic. In Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), 2019.","title":"LUTNet: Rethinking inference in FPGA soft logic","authors":"Wang","year":"2019","venue":"Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wang et al., 2021] Shiqi Wang, Huan Zhang, Kaidi Xu, Xue Lin, Suman Jana, Cho-Jui Hsieh, and J Zico Kolter. Beta-crown: Efficient bound propagation with per-neuron split constraints for neural network robustness verifica- tion. Advances in Neural Information Processing Systems (NeurIPS), 2021.","title":"Beta-crown: Efficient bound propagation with per-neuron split constraints for neural network robustness verification","authors":"Shiqi Wang, Huan Zhang, Kaidi Xu, Xue Lin, Suman Jana, Cho-Jui Hsieh, J Zico Kolter","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Xu et al., 2020] Kaidi Xu, Huan Zhang, Shiqi Wang, Yihan Wang, Suman Jana, Xue Lin, and Cho-Jui Hsieh. Fast and Complete: Enabling Complete Neural Network Verification with Rapid and Massively Parallel Incomplete Verifiers. In International Conference on Learning Representations (ICLR), 2020.","title":"Fast and Complete: Enabling Complete Neural Network Verification with Rapid and Massively Parallel Incomplete Verifiers","authors":"Kaidi Xu, Huan Zhang, Shiqi Wang, Yihan Wang, Suman Jana, Xue Lin, Cho-Jui Hsieh","year":"2020","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zeng and Urtasun, 2018] Wenyuan Zeng and Raquel Urta- sun. Mlprune: Multi-layer pruning for automated neural network compression. openreview, 2018.","title":"Mlprune: Multi-layer pruning for automated neural network compression","authors":"Urtasun ; Zeng, Raquel Zeng, Urtasun","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhou et al., 2021] Xiao Zhou, Weizhong Zhang, Hang Xu, and Tong Zhang. Effective sparsification of neural networks with global sparsity constraint. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.","title":"Effective sparsification of neural networks with global sparsity constraint","authors":"Xiao Zhou, Weizhong Zhang, Hang Xu, Tong Zhang","year":"2021","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}