{"type":"ScholarlyArticle","title":"Reachability Analysis of Deep Neural Networks with Provable Guarantees","authors":[{"name":"Wenjie Ruan","slug":"wenjie-ruan","given":"Wenjie","family":"Ruan","affiliation":"University of Oxford , Oxford , UK"},{"name":"Xiaowei Huang","slug":"xiaowei-huang","given":"Xiaowei","family":"Huang","affiliation":"University of Liverpool , Liverpool , UK"},{"name":"Marta Kwiatkowska","slug":"marta-kwiatkowska","given":"Marta","family":"Kwiatkowska","affiliation":"University of Oxford , Oxford , UK"}],"abstract":"Verifying correctness for deep neural networks (DNNs) is challenging. We study a generic reachability problem for feed-forward DNNs which, for a given set of inputs to the network and a Lipschitz-continuous function over its outputs computes the lower and upper bound on the function values. Because the network and the function are Lipschitz continuous, all values in the interval between the lower and upper bound are reachable. We show how to obtain the safety verification problem, the output range analysis problem and a robustness measure by instantiating the reachability problem. We present a novel algorithm based on adaptive nested optimisation to solve the reachability problem. The technique has been implemented and evaluated on a range of DNNs, demonstrating its efficiency, scalability and ability to handle a broader class of networks than state-of-the-art verification approaches.","doi":"10.24963/ijcai.2018/368","url":"https://ijcai.org/proceedings/2018/368","pdf":"https://ijcai.org/proceedings/2018/0368.pdf","markdown":"https://ijcai.org/proceedings/2018/368.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/368","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":368,"pages":{"first":2651,"last":2659},"keywords":["Machine Learning: Neural Networks","Machine Learning: Deep Learning","Computer Vision: Computer Vision","Agent-based and Multi-agent Systems: Formal Verification, Validation and Synthesis"],"videos":[],"references":[{"position":1,"raw":"[Amodei et al., 2016] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016.","title":"Concrete problems in ai safety","authors":"Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bunel et al., 2017] Rudy Bunel, Ilker Turkaslan, Philip HS Torr, Pushmeet Kohli, and M Pawan Kumar. Piecewise linear neural network verification: A comparative study. arXiv preprint arXiv:1711.00455, 2017.","title":"Piecewise linear neural network verification: A comparative study","authors":"Rudy Bunel, Ilker Turkaslan, Pushmeet Philip Hs Torr, M Pawan Kohli, Kumar","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carlini and Wagner, 2016] Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural net- works. CoRR, abs/1608.04644, 2016.","title":"Towards evaluating the robustness of neural networks","authors":"Nicholas Carlini, David A Wagner","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cheng et al., 2017] Chih-Hong Cheng, Georg Nührenberg, and Harald Ruess. Maximum resilience of artificial neu- ral networks. In Deepak D'Souza and K. Narayan Kumar, editors, Automated Technology for Verification and Anal- ysis, pages 251-268, Cham, 2017. Springer International Publishing.","title":"Maximum resilience of artificial neural networks","authors":"Cheng","year":"2017","venue":"Automated Technology for Verification and Analysis","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dutta et al., 2017] Souradeep Dutta, Susmit Jha, Sriram Sanakaranarayanan, and Ashish Tiwari. Output range analysis for deep neural networks. arXiv preprint arXiv:1709.09130, 2017.","title":"Output range analysis for deep neural networks","authors":"Dutta","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ehlers, 2017] Ruediger Ehlers. Formal verification of piece- wise linear feed-forward neural networks. In International Symposium on Automated Technology for Verification and Analysis, pages 269-286. Springer, 2017. Figure","title":"Formal verification of piecewise linear feed-forward neural networks","authors":"Ruediger Ehlers, Ehlers","year":"2017","venue":"International Symposium on Automated Technology for Verification and Analysis","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"A high-level comparison with state-of-the-art methods: SHERLOCK [Dutta et al., 2017], Reluplex [Katz et al., 2017], Planet [","title":"A high-level comparison with state-of-the-art methods: SHERLOCK","authors":null,"year":"2017","venue":"Planet","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"Ehlers, 2017], MIP [","title":"MIP","authors":"Ehlers","year":"2017","venue":"MIP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Cheng et al., 2017;","title":null,"authors":"Cheng","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lomuscio and Maganti, 2017] and BaB [Bunel et al., 2017].","title":null,"authors":"Maganti ; Lomuscio, Bunel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gergel et al., 2016] Victor Gergel, Vladimir Grishagin, and Alexander Gergel. Adaptive nested optimization scheme for multidimensional global search. Journal of Global Op- timization, 66(1):35-51, 2016.","title":"Adaptive nested optimization scheme for multidimensional global search","authors":"Gergel","year":"2016","venue":"Journal of Global Optimization","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Goodfellow et al., 2014] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and Harnessing Adversarial Ex- amples. ArXiv e-prints, December 2014.","title":"Explaining and Harnessing Adversarial Examples","authors":"Goodfellow","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Grishagin et al., 2018] Vladimir Grishagin, Ruslan Is- rafilov, and Yaroslav Sergeyev. Convergence conditions and numerical comparison of global optimization methods based on dimensionality reduction schemes. Applied Mathematics and Computation, 318:270-280, 2018.","title":"Vladimir Grishagin, Ruslan Israfilov, and Yaroslav Sergeyev. Convergence conditions and numerical comparison of global optimization methods based on dimensionality reduction schemes","authors":"Grishagin","year":"2018","venue":"Applied Mathematics and Computation","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Huang et al., 2017] Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu. Safety verification of deep neu- ral networks. In Computer Aided Verification, pages 3-29. Springer Berlin Heidelberg, 2017.","title":"Safety verification of deep neural networks","authors":"Xiaowei Huang, Marta Kwiatkowska, Sen Wang, Min Wu","year":"2017","venue":"Computer Aided Verification","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Katz et al., 2017] Guy Katz, Clark Barrett, David Dill, Kyle Julian, and Mykel Kochenderfer. Reluplex: An efficient smt solver for verifying deep neural networks. arXiv preprint arXiv:1702.01135, 2017.","title":"Reluplex: An efficient smt solver for verifying deep neural networks","authors":"Katz","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kolter and Wong, 2017] J Zico Kolter and Eric Wong. Provable defenses against adversarial examples via the convex outer adversarial polytope. arXiv preprint arXiv:1711.00851, 2017.","title":"Provable defenses against adversarial examples via the convex outer adversarial polytope","authors":"Wong ; ] J Zico Kolter, Eric Kolter, Wong","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lomuscio and Maganti, 2017] Alessio Lomuscio and Lalit Maganti. An approach to reachability analysis for feed- forward relu neural networks. CoRR, abs/1706.07351, 2017.","title":"An approach to reachability analysis for feedforward relu neural networks","authors":"Maganti Lomuscio, Alessio Lomuscio, Lalit Maganti","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Moosavi-Dezfooli et al., 2016] Seyed-Mohsen Moosavi- Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. CoRR, abs/1610.08401, 2016.","title":"Universal adversarial perturbations","authors":"Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, Pascal Frossard","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Narodytska et al., 2017] Nina Narodytska, Shiva Prasad Kasiviswanathan, Leonid Ryzhyk, Mooly Sagiv, and Toby Walsh. Verifying properties of binarized deep neural net- works. CoRR, abs/1709.06662, 2017.","title":"Verifying properties of binarized deep neural networks","authors":"Nina Narodytska, Shiva Prasad Kasiviswanathan, Leonid Ryzhyk, Mooly Sagiv, Toby Walsh","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nguyen et al., 2014] Anh Mai Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. CoRR, abs/1412.1897, 2014.","title":"Deep neural networks are easily fooled: High confidence predictions for unrecognizable images","authors":"Anh Mai Nguyen, Jason Yosinski, Jeff Clune","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Papernot et al., 2015] Nicolas Papernot, Patrick D. Mc- Daniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. CoRR, abs/1511.07528, 2015.","title":"The limitations of deep learning in adversarial settings","authors":"Nicolas Papernot, D Patrick, Somesh Mc-Daniel, Matt Jha, Z Berkay Fredrikson, Ananthram Celik, Swami","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Piyavskii, 1972] SA Piyavskii. An algorithm for finding the absolute extremum of a function. USSR Computa- tional Mathematics and Mathematical Physics, 12(4):57- 67, 1972.","title":"SA Piyavskii. An algorithm for finding the absolute extremum of a function","authors":"Piyavskii","year":"1972","venue":"USSR Computational Mathematics and Mathematical Physics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pulina and Tacchella, 2010] Luca Pulina and Armando Tac- chella. An abstraction-refinement approach to verification of artificial neural networks. In Computer Aided Verifica- tion, pages 243-257. Springer Berlin Heidelberg, 2010.","title":"Luca Pulina and Armando Tacchella. An abstraction-refinement approach to verification of artificial neural networks","authors":"Tacchella Pulina","year":"2010","venue":"Computer Aided Verification","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ruan et al., 2018a] Wenjie Ruan, Xiaowei Huang, and Marta Kwiatkowska. Reachability analysis of deep neu- ral networks with provable guarantees. arXiv preprint arXiv:1805.02242, 2018.","title":"Reachability analysis of deep neural networks with provable guarantees","authors":"Ruan","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ruan et al., 2018b] Wenjie Ruan, Min Wu, Youcheng Sun, Xiaowei Huang, Daniel Kroening, and Marta Kwiatkowska. Global robustness evaluation of deep neu- ral networks with provable guarantees for L0 norm. arXiv preprint arXiv:1804.05805, 2018.","title":"Global robustness evaluation of deep neural networks with provable guarantees for L0 norm","authors":"Ruan","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sohrab, 2003] Houshang H Sohrab. Basic real analysis, volume 231. Springer, 2003.","title":"Basic real analysis","authors":"H Houshang, Sohrab","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sun et al., 2018] Youcheng Sun, Min Wu, Wenjie Ruan, Xi- aowei Huang, Marta Kwiatkowska, and Daniel Kroening. Concolic testing for deep neural networks. arXiv preprint arXiv:1805.00089v1, 2018.","title":"Concolic testing for deep neural networks","authors":"Sun","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Szegedy et al., 2013] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv:1312.6199v4, 2013.","title":"Intriguing properties of neural networks","authors":"Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, Rob Fergus","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Torn and Zilinskas, 1989] Aimo Torn and Antanas Zilin- skas. Global Optimization. Springer-Verlag New York, Inc., New York, NY, USA, 1989.","title":"Aimo Torn and Antanas Zilinskas","authors":"Zilinskas Torn","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wicker et al., 2018] Matthew Wicker, Xiaowei Huang, and Marta Kwiatkowska. Feature-guided black-box safety testing of deep neural networks. In Proc. 24th Interna- tional Conference on Tools and Algorithms for the Con- struction and Analysis of Systems (TACAS'18), pages 408- 426, 2018.","title":"Feature-guided black-box safety testing of deep neural networks","authors":"Matthew Wicker, Xiaowei Huang, Marta Kwiatkowska","year":"2018","venue":"Proc. 24th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS'18)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xiang et al., 2017] Weiming Xiang, Hoang-Dung Tran, and Taylor T Johnson. Output reachable set estimation and verification for multi-layer neural networks. arXiv preprint arXiv:1708.03322, 2017.","title":"Output reachable set estimation and verification for multi-layer neural networks","authors":"Weiming Xiang, Hoang-Dung Tran, Taylor T Johnson","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}