{"type":"ScholarlyArticle","title":"The #DNN-Verification Problem: Counting Unsafe Inputs for Deep Neural Networks","authors":[{"name":"Luca Marzari","slug":"luca-marzari","given":"Luca","family":"Marzari","affiliation":"Department of Computer Science , University of Verona , Verona , Italy"},{"name":"Davide Corsi","slug":"davide-corsi","given":"Davide","family":"Corsi","affiliation":"Department of Computer Science , University of Verona , Verona , Italy"},{"name":"Ferdinando Cicalese","slug":"ferdinando-cicalese","given":"Ferdinando","family":"Cicalese","affiliation":"Department of Computer Science , University of Verona , Verona , Italy"},{"name":"Alessandro Farinelli","slug":"alessandro-farinelli","given":"Alessandro","family":"Farinelli","affiliation":"Department of Computer Science , University of Verona , Verona , Italy"}],"abstract":"Deep Neural Networks are increasingly adopted in critical tasks that require a high level of safety, e.g., autonomous driving.\r\nWhile state-of-the-art verifiers can be employed to check whether a DNN is unsafe w.r.t. some given property (i.e., whether there is at least one unsafe input configuration), their yes/no output is not informative enough for other purposes, such as shielding, model selection, or training improvements.\r\nIn this paper, we introduce the #DNN-Verification problem, which involves counting the number of input configurations of a DNN that result in a violation of a particular safety property. We analyze the complexity of this problem and propose a novel approach that returns the exact count of violations. Due to the #P-completeness of the problem, we also propose a randomized, approximate method that provides a provable probabilistic bound of the correct count while significantly reducing computational requirements. \r\nWe present experimental results on a set of safety-critical benchmarks that demonstrate the effectiveness of our approximate method and evaluate the tightness of the bound.","doi":"10.24963/ijcai.2023/25","url":"https://ijcai.org/proceedings/2023/25","pdf":"https://ijcai.org/proceedings/2023/0025.pdf","markdown":"https://ijcai.org/proceedings/2023/25.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/25","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":25,"pages":{"first":217,"last":224},"keywords":["AI Ethics, Trust, Fairness: ETF: Safety and robustness","Agent-based and Multi-agent Systems: MAS: Formal verification, validation and synthesis"],"videos":[],"references":[{"position":1,"raw":"[Amir et al., 2023] Guy Amir, Davide Corsi, Raz Yerushalmi, Luca Marzari, David Harel, Alessandro Farinelli, and Guy Katz. Verifying learning-based robotic navigation systems. In 29th International Conference, TACAS 2023, pages 607-627. Springer, 2023.","title":"Verifying learning-based robotic navigation systems","authors":"Guy Amir, Davide Corsi, Raz Yerushalmi, Luca Marzari, David Harel, Alessandro Farinelli, Guy Katz","year":"2023","venue":"29th International Conference, TACAS 2023","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baluta et al., 2019] Teodora Baluta, Shiqi Shen, Shweta Shinde, Kuldeep S. Meel, and Prateek Saxena. Quanti- tative verification of neural networks and its security ap- plications. In Proceedings of the 2019 ACM SIGSAC Con- ference on Computer and Communications Security, 2019.","title":"Quantitative verification of neural networks and its security applications","authors":"Teodora Baluta, Shiqi Shen, Shweta Shinde, Kuldeep S Meel, Prateek Saxena","year":"2019","venue":"Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bunel et al., 2018] Rudy R Bunel, Ilker Turkaslan, Philip Torr, Pushmeet Kohli, and Pawan K Mudigonda. A uni- fied view of piecewise linear neural network verification. Advances in Neural Information Processing Systems, 31, 2018.","title":"A unified view of piecewise linear neural network verification","authors":"Rudy R Bunel, Ilker Turkaslan, Philip Torr, Pushmeet Kohli, Pawan K Mudigonda","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Corsi et al., 2021] Davide Corsi, Enrico Marchesini, and Alessandro Farinelli. Formal verification of neural net- works for safety-critical tasks in deep reinforcement learn- ing. In Uncertainty in Artificial Intelligence, pages 333- 343. PMLR, 2021.","title":"Formal verification of neural networks for safety-critical tasks in deep reinforcement learning","authors":"Davide Corsi, Enrico Marchesini, Alessandro Farinelli","year":"2021","venue":"Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ghosh et al., 2021] Bishwamittra Ghosh, Debabrota Basu, and Kuldeep S Meel. Justicia: A stochastic sat approach to formally verify fairness. In Proceedings of the AAAI Con- ference on Artificial Intelligence, pages 7554-7563, 2021.","title":"Justicia: A stochastic sat approach to formally verify fairness","authors":"Ghosh","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gomes et al., 2007] Carla P Gomes, Jörg Hoffmann, Ashish Sabharwal, and Bart Selman. From sampling to model counting. In IJCAI, volume 2007, pages 2293-2299, 2007.","title":"From sampling to model counting","authors":"Carla P Gomes, Jörg Hoffmann, Ashish Sabharwal, Bart Selman","year":"2007","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2007/369"},{"position":7,"raw":"[Gomes et al., 2021] Carla P Gomes, Ashish Sabharwal, and Bart Selman. Model counting. In Handbook of satisfiabil- ity, pages 993-1014. IOS press, 2021.","title":"Model counting","authors":"Carla P Gomes, Ashish Sabharwal, Bart Selman","year":"2021","venue":"Handbook of satisfiability","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Katz et al., 2017] Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. Reluplex: An efficient smt solver for verifying deep neural networks. In International conference on computer aided verification, pages 97-117. Springer, 2017.","title":"Reluplex: An efficient smt solver for verifying deep neural networks","authors":"Guy Katz, Clark Barrett, David L Dill, Kyle Julian, Mykel, Kochenderfer","year":"2017","venue":"International conference on computer aided verification","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Katz et al., 2019] Guy Katz, Derek A Huang, Duligur Ibel- ing, Kyle Julian, Christopher Lazarus, Rachel Lim, Parth Shah, Shantanu Thakoor, Haoze Wu, Aleksandar Zeljić, et al. The marabou framework for verification and analy- sis of deep neural networks. In International Conference on Computer Aided Verification, 2019.","title":"The marabou framework for verification and analysis of deep neural networks","authors":"Guy Katz, Derek A Huang, Duligur Ibeling, Kyle Julian, Christopher Lazarus, Rachel Lim, Parth Shah, Shantanu Thakoor, Haoze Wu, Aleksandar Zeljić","year":"2019","venue":"International Conference on Computer Aided Verification","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Katz et al., 2021] Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. Reluplex: a cal- culus for reasoning about deep neural networks. Formal Methods in System Design, pages 1-30, 2021.","title":"Reluplex: a calculus for reasoning about deep neural networks","authors":"Guy Katz, Clark Barrett, David L Dill, Kyle Julian, Mykel, Kochenderfer","year":"2021","venue":"Formal Methods in System Design","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2021] Changliu Liu, Tomer Arnon, Christopher Lazarus, Christopher Strong, Clark Barrett, Mykel J Kochenderfer, et al. Algorithms for verifying deep neu- ral networks. Foundations and Trends® in Optimization, 4(3-4):244-404, 2021.","title":"Algorithms for verifying deep neural networks","authors":"Liu","year":"2021","venue":"Foundations and Trends® in Optimization","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lomuscio and Maganti, 2017] Alessio Lomuscio and Lalit Maganti. An approach to reachability analysis for feed-forward relu neural networks. arXiv preprint arXiv:1706.07351, 2017.","title":"An approach to reachability analysis for feed-forward relu neural networks","authors":"Maganti Lomuscio, Alessio Lomuscio, Lalit Maganti","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Marchesini et al., 2021] Enrico Marchesini, Davide Corsi, and Alessandro Farinelli. Benchmarking safe deep re- inforcement learning in aquatic navigation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5590-5595, 2021.","title":"Benchmarking safe deep reinforcement learning in aquatic navigation","authors":"Enrico Marchesini, Davide Corsi, Alessandro Farinelli","year":"2021","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Marchesini et al., 2022] Enrico Marchesini, Davide Corsi, and Alessandro Farinelli. Exploring safer behaviors for deep reinforcement learning. In Association for the Ad- vancement of Artificial Intelligence (AAAI), 2022.","title":"Exploring safer behaviors for deep reinforcement learning","authors":"Enrico Marchesini, Davide Corsi, Alessandro Farinelli","year":"2022","venue":"Association for the Advancement of Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Marzari et al., 2022] Luca Marzari, Davide Corsi, Enrico Marchesini, and Alessandro Farinelli. Curriculum learn- ing for safe mapless navigation. In Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing, pages 766-769, 2022.","title":"Curriculum learning for safe mapless navigation","authors":"Luca Marzari, Davide Corsi, Enrico Marchesini, Alessandro Farinelli","year":"2022","venue":"Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Marzari et al., 2023] Luca Marzari, Enrico Marchesini, and Alessandro Farinelli. Online safety property collection and refinement for safe deep reinforcement learning in map- less navigation. In 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.","title":"Online safety property collection and refinement for safe deep reinforcement learning in mapless navigation","authors":"Luca Marzari, Enrico Marchesini, Alessandro Farinelli","year":"2023","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Moore et al., 2009] Ramon E Moore, R Baker Kearfott, and Michael J Cloud. Introduction to interval analysis. SIAM, 2009.","title":"Introduction to interval analysis","authors":"R Ramon E Moore, Michael J Baker Kearfott, Cloud","year":"2009","venue":"SIAM","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Szegedy et al., 2013] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.","title":"Intriguing properties of neural networks","authors":"Szegedy","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tjeng et al., 2018] Vincent Tjeng, Kai Y Xiao, and Russ Tedrake. Evaluating robustness of neural networks with mixed integer programming. In International Conference on Learning Representations, 2018.","title":"Evaluating robustness of neural networks with mixed integer programming","authors":"Vincent Tjeng, Kai Y Xiao, Russ Tedrake","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Valiant, 1979] Leslie G Valiant. The complexity of enumer- ation and reliability problems. SIAM Journal on Comput- ing, 8(3):410-421, 1979.","title":"The complexity of enumeration and reliability problems","authors":"G Leslie, Valiant","year":"1979","venue":"SIAM Journal on Computing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2018] Shiqi Wang, Kexin Pei, Justin White- house, Junfeng Yang, and Suman Jana. Efficient formal safety analysis of neural networks. Advances in Neural Information Processing Systems, 31, 2018.","title":"Efficient formal safety analysis of neural networks","authors":"Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, Suman Jana","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2021] Shiqi Wang, Huan Zhang, Kaidi Xu, Xue Lin, Suman Jana, Cho-Jui Hsieh, and J Zico Kolter. Beta-crown: Efficient bound propagation with per-neuron split constraints for neural network robustness verifica- tion. Advances in Neural Information Processing Systems, 34:29909-29921, 2021.","title":"Beta-crown: Efficient bound propagation with per-neuron split constraints for neural network robustness verification","authors":"Shiqi Wang, Huan Zhang, Kaidi Xu, Xue Lin, Suman Jana, Cho-Jui Hsieh, J Zico Kolter","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2021] Yedi Zhang, Zhe Zhao, Guangke Chen, Fu Song, and Taolue Chen. Bdd4bnn: a bdd-based quan- titative analysis framework for binarized neural networks. In Computer Aided Verification: 33rd International Con- ference, CAV 2021, Virtual Event, July 20-23, 2021, Pro- ceedings, Part I 33, pages 175-200. Springer, 2021.","title":"Bdd4bnn: a bdd-based quantitative analysis framework for binarized neural networks","authors":"Zhang","year":"2021","venue":"Computer Aided Verification: 33rd International Conference, CAV 2021, Virtual Event","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}