{"type":"ScholarlyArticle","title":"Global Robustness Evaluation of Deep Neural Networks with Provable Guarantees for the Hamming Distance","authors":[{"name":"Wenjie Ruan","slug":"wenjie-ruan","given":"Wenjie","family":"Ruan","affiliation":"Department of Computer Science , University of Oxford"},{"name":"Min Wu","slug":"min-wu","given":"Min","family":"Wu","affiliation":"Department of Computer Science , University of Oxford"},{"name":"Youcheng Sun","slug":"youcheng-sun","given":"Youcheng","family":"Sun","affiliation":"Department of Computer Science , University of Oxford"},{"name":"Xiaowei Huang","slug":"xiaowei-huang","given":"Xiaowei","family":"Huang","affiliation":"Department of Computer Science , University of Liverpool"},{"name":"Daniel Kroening","slug":"daniel-kroening","given":"Daniel","family":"Kroening","affiliation":"Department of Computer Science , University of Oxford"},{"name":"Marta Kwiatkowska","slug":"marta-kwiatkowska","given":"Marta","family":"Kwiatkowska","affiliation":"Department of Computer Science , University of Oxford"}],"abstract":"Deployment of deep neural networks (DNNs) in safety-critical systems requires provable guarantees for their correct behaviours. We compute the maximal radius of a safe norm ball around a given input, within which there are no adversarial examples for a trained DNN. We define global robustness as an expectation of the maximal safe radius over a test dataset, and develop an algorithm to approximate the global robustness measure by iteratively computing its lower and upper bounds. Our algorithm is the first efficient method for the Hamming (L0) distance, and we hypothesise that this norm is a good proxy for a certain class of physical attacks. The algorithm is anytime, i.e., it returns intermediate bounds and robustness estimates that are gradually, but strictly, improved as the computation proceeds; tensor-based, i.e., the computation is conducted over a set of inputs simultaneously to enable efficient GPU computation; and has provable guarantees, i.e., both the bounds and the robustness estimates can converge to their optimal values. Finally, we demonstrate the utility of our approach by applying the algorithm to a set of challenging problems.","doi":"10.24963/ijcai.2019/824","url":"https://ijcai.org/proceedings/2019/824","pdf":"https://ijcai.org/proceedings/2019/0824.pdf","markdown":"https://ijcai.org/proceedings/2019/824.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/824","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"AI for Improving Human Well-being","track":"AI for Improving Human Well-being","number":824,"pages":{"first":5944,"last":5952},"keywords":["Special Track on AI for Improving Human-Well Being: AI certification (Special Track on AI and Human Wellbeing)","Special Track on AI for Improving Human-Well Being: AI safety (Special Track on AI and Human Wellbeing)"],"videos":[],"references":[{"position":1,"raw":"[Biggio et al., 2013] Battista Biggio, Giorgio Fumera, and Fabio Roli. Security evaluation of pattern classifiers un- der attack. IEEE Transactions on Knowledge and Data Engineering, 26(4):984-996, 2013.","title":"Security evaluation of pattern classifiers under attack","authors":"Battista Biggio, Giorgio Fumera, Fabio Roli","year":"2013","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Carlini and Wagner, 2017] Nicholas Carlini and David Wag- ner. Towards evaluating the robustness of neural networks. In Security and Privacy (SP), 2017 IEEE Symposium on, pages 39-57. IEEE, 2017.","title":"Towards evaluating the robustness of neural networks","authors":"Nicholas Carlini, David Wagner","year":"2017","venue":"Security and Privacy (SP)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carlini et al., 2017] Nicholas Carlini, Guy Katz, Clark Bar- rett, and David L Dill. Ground-truth adversarial examples. arXiv preprint arXiv:1709.10207, 2017.","title":"Ground-truth adversarial examples","authors":"Nicholas Carlini, Guy Katz, Clark Barrett, David L Dill","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dutta et al., 2017] Souradeep Dutta, Susmit Jha, Sriram Sanakaranarayanan, and Ashish Tiwari. Output range analysis for deep neural networks. arXiv preprint arXiv:1709.09130, 2017.","title":"Output range analysis for deep neural networks","authors":"Dutta","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gehr et al., 2018] Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, and Martin T. Vechev. AI 2 : Safety and robustness certification of neural networks with abstract interpretation. In 2018 IEEE Symposium on Security and Privacy (SP), 2018.","title":"Vechev. AI 2 : Safety and robustness certification of neural networks with abstract interpretation","authors":"Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, Martin T","year":"2018","venue":"IEEE Symposium on Security and Privacy (SP)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goodfellow et al., 2015] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR), 2015.","title":"Explaining and harnessing adversarial examples","authors":"Ian J Goodfellow, Jonathon Shlens, Christian Szegedy","year":"2015","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gopinath et al., 2018] Divya Gopinath, Guy Katz, Corina S. Pasareanu, and Clark Barrett. DeepSafe: A data-driven approach for checking adversarial robustness of neural net- works. In Automated Technology for Verification and Anal- ysis (ATVA), pages 3-19. Springer, 2018.","title":"DeepSafe: A data-driven approach for checking adversarial robustness of neural networks","authors":"Divya Gopinath, Guy Katz, Corina S Pasareanu, Clark Barrett","year":"2018","venue":"Automated Technology for Verification and Analysis (ATVA)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2017] Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu. Safety verification of deep neural networks. In CAV, pages 3-29. Springer, 2017.","title":"Safety verification of deep neural networks","authors":"Xiaowei Huang, Marta Kwiatkowska, Sen Wang, Min Wu","year":"2017","venue":"CAV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jacobsen et al., 2019] Jörn-Henrik Jacobsen, Jens Behr- mann, Nicholas Carlini, Florian Tramèr, and Nicolas Papernot. Exploiting excessive invariance caused by norm-bounded adversarial robustness. arXiv preprint arXiv:1903.10484, 2019.","title":"Exploiting excessive invariance caused by norm-bounded adversarial robustness","authors":"Jacobsen","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Katz et al., 2017] Guy Katz, Clark Barrett, David Dill, Kyle Julian, and Mykel Kochenderfer. Reluplex: An efficient SMT solver for verifying deep neural networks. In CAV. Springer, 2017.","title":"Reluplex: An efficient SMT solver for verifying deep neural networks","authors":"Guy Katz, Clark Barrett, David Dill, Kyle Julian, Mykel Kochenderfer","year":"2017","venue":"CAV. Springer","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kurakin et al., 2016] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533, 2016.","title":"Adversarial examples in the physical world","authors":"Alexey Kurakin, Ian Goodfellow, Samy Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lomuscio and Maganti, 2017] Alessio Lomuscio and Lalit Maganti. An approach to reachability analysis for feed- forward ReLU neural networks. CoRR, abs/1706.07351, 2017.","title":"An approach to reachability analysis for feedforward ReLU neural networks","authors":"Maganti Lomuscio, Alessio Lomuscio, Lalit Maganti","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mirman et al., 2018] Matthew Mirman, Timon Gehr, and Martin Vechev. Differentiable abstract interpretation for provably robust neural networks. In International Confer- ence on Machine Learning, pages 3575-3583, 2018.","title":"Differentiable abstract interpretation for provably robust neural networks","authors":"Matthew Mirman, Timon Gehr, Martin Vechev","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Narodytska et al., 2017] Nina Narodytska, Shiva Prasad Ka- siviswanathan, Leonid Ryzhyk, Mooly Sagiv, and Toby Walsh. Verifying properties of binarized deep neural net- works. CoRR, abs/1709.06662, 2017.","title":"Nina Narodytska, Shiva Prasad Kasiviswanathan","authors":"Narodytska","year":"2017","venue":"Verifying properties of binarized deep neural networks","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Papernot et al., 2016] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Anan- thram Swami. The limitations of deep learning in adversar- ial settings. In EuroS&P 2016, 2016.","title":"The limitations of deep learning in adversarial settings","authors":"Nicolas Papernot, Patrick Mcdaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, Ananthram Swami","year":"2016","venue":"EuroS&P 2016","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Peck et al., 2017] Jonathan Peck, Joris Roels, Bart Goossens, and Yvan Saeys. Lower bounds on the robustness to ad- versarial perturbations. In Advances in Neural Information Processing Systems, pages 804-813, 2017.","title":"Lower bounds on the robustness to adversarial perturbations","authors":"Jonathan Peck, Joris Roels, Bart Goossens, Yvan Saeys","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pei et al., 2017] Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. DeepXplore: Automated whitebox testing of deep learning systems. In SOSP, pages 1-18. ACM, 2017.","title":"DeepXplore: Automated whitebox testing of deep learning systems","authors":"Kexin Pei, Yinzhi Cao, Junfeng Yang, Suman Jana","year":"2017","venue":"SOSP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pulina and Tacchella, 2010] Luca Pulina and Armando Tac- chella. An abstraction-refinement approach to verifica- tion of artificial neural networks. In CAV, pages 243-257. Springer, 2010.","title":"Luca Pulina and Armando Tacchella. An abstraction-refinement approach to verification of artificial neural networks","authors":"Tacchella Pulina","year":"2010","venue":"CAV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ruan et al., 2018a] Wenjie Ruan, Xiaowei Huang, and Marta Kwiatkowska. Reachability analysis of deep neural networks with provable guarantees. In The 27th Interna- tional Joint Conference on Artificial Intelligence (IJCAI), pages 2651-2659, 2018.","title":"Reachability analysis of deep neural networks with provable guarantees","authors":"Ruan","year":"2018","venue":"The 27th International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2018/368"},{"position":20,"raw":"[Ruan et al., 2018b] Wenjie Ruan, Min Wu, Youcheng Sun, Xiaowei Huang, Daniel Kroening, and Marta Kwiatkowska. Global robustness evaluation of deep neural networks with provable guarantees for L 0 norm. arXiv preprint arXiv:1804.05805, 2018.","title":"Global robustness evaluation of deep neural networks with provable guarantees for L 0 norm","authors":"Ruan","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sun et al., 2018] Youcheng Sun, Min Wu, Wenjie Ruan, Xi- aowei Huang, Marta Kwiatkowska, and Daniel Kroening. Concolic testing for deep neural networks. In Automated Software Engineering (ASE), pages 109-119. ACM, 2018.","title":"Concolic testing for deep neural networks","authors":"Sun","year":"2018","venue":"Automated Software Engineering (ASE)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Szegedy et al., 2014] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.","title":"Intriguing properties of neural networks","authors":"Szegedy","year":"2014","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tsipras et al., 2018] Dimitris Tsipras, Shibani Santurkar, Lo- gan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. arXiv preprint arXiv:1805.12152, May 2018.","title":"Robustness may be at odds with accuracy","authors":"Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, Aleksander Madry","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wicker et al., 2018] Matthew Wicker, Xiaowei Huang, and Marta Kwiatkowska. Feature-guided black-box safety test- ing of deep neural networks. In TACAS, LNCS, pages 408-426. Springer, 2018.","title":"Feature-guided black-box safety testing of deep neural networks","authors":"Matthew Wicker, Xiaowei Huang, Marta Kwiatkowska","year":"2018","venue":"TACAS, LNCS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu et al., 2018] Min Wu, Matthew Wicker, Wenjie Ruan, Xiaowei Huang, and Marta Kwiatkowska. A game-based approximate verification of deep neural networks with prov- able guarantees. To appear in Theoretical Computer Sci- ence. CoRR, abs/1807.03571, 2018.","title":"A game-based approximate verification of deep neural networks with provable guarantees","authors":"Wu","year":"2018","venue":"Theoretical Computer Science","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}