{"type":"ScholarlyArticle","title":"Formal Analysis of Deep Binarized Neural Networks","authors":[{"name":"Nina Narodytska","slug":"nina-narodytska","given":"Nina","family":"Narodytska","affiliation":"VMware Research , USA"}],"abstract":"Understanding properties of deep neural networks is an important challenge in deep learning.  Deep learning networks are among the most successful artificial intelligence technologies that is making impact in a variety of practical applications.  However, many concerns  were raised about `magical' power of these networks.  It is disturbing that we are really lacking  of understanding of the decision making process behind this technology. Therefore, a natural question is whether we can trust decisions that neural networks make.  One way to address this issue is to define properties that we want a neural network to satisfy. Verifying whether a neural network fulfills these properties sheds light on the properties of the function that it represents. In this work, we take the verification approach. Our goal is to design a framework for analysis of properties of neural networks. We start by defining a set of interesting properties to analyze.  Then we focus on Binarized Neural Networks that can be represented and analyzed using  well-developed means of Boolean Satisfiability and Integer Linear Programming. One of our main results is an exact representation of a binarized neural network as a Boolean formula. We also discuss how  we can take advantage of the structure of neural networks in the search procedure.","doi":"10.24963/ijcai.2018/811","url":"https://ijcai.org/proceedings/2018/811","pdf":"https://ijcai.org/proceedings/2018/0811.pdf","markdown":"https://ijcai.org/proceedings/2018/811.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/811","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Early Career","track":"Early Career","number":811,"pages":{"first":5692,"last":5696},"keywords":["Constraints and SAT: SAT","Constraints and SAT: SAT: Applications","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Bastani et al., 2016] Osbert Bastani, Yani Ioannou, Leonidas Lampropoulos, Dimitrios Vytiniotis, Aditya Nori, and An- tonio Criminisi. Measuring Neural Net Robustness with Constraints. In NIPS'16, 2016.","title":"Measuring Neural Net Robustness with Constraints","authors":"Osbert Bastani, Yani Ioannou, Leonidas Lampropoulos, Dimitrios Vytiniotis, Aditya Nori, Antonio Criminisi","year":"2016","venue":"NIPS'16","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bau et al., 2017] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. CoRR, abs/1704.05796, 2017.","title":"Network dissection: Quantifying interpretability of deep visual representations","authors":"David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, Antonio Torralba","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bunel et al., 2017] Rudy Bunel, Ilker Turkaslan, Philip Torr, Pushmeet Kohli, and Pawan Kumar. Piecewise linear neu- ral network verification: A comparative study. CoRR, abs/1711.00455, 2017.","title":"Piecewise linear neural network verification: A comparative study","authors":"Rudy Bunel, Ilker Turkaslan, Philip Torr, Pushmeet Kohli, Pawan Kumar","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cheng et al., 2017a] Chih-Hong Cheng, Georg Nührenberg, and Harald Ruess. Maximum resilience of artificial neural networks. In Automated Technology for Verification and Analysis, volume 10482, pages 251-268, 2017.","title":"Maximum resilience of artificial neural networks","authors":"Cheng","year":"2017","venue":"Automated Technology for Verification and Analysis","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cheng et al., 2017b] Chih-Hong Cheng, Georg Nührenberg, and Harald Ruess. Verification of binarized neural networks. CoRR, abs/1710.03107, 2017.","title":"Verification of binarized neural networks","authors":"Cheng","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Clarke et al., 2000] Edmund Clarke, Orna Grumberg, Somesh Jha, Yuan Lu, and Helmut Veith. Counterexample- guided abstraction refinement. In CAV'00, pages 154-169. Springer, 2000.","title":"Counterexampleguided abstraction refinement","authors":"Edmund Clarke, Orna Grumberg, Somesh Jha, Yuan Lu, Helmut Veith","year":"2000","venue":"CAV'00","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dutta et al., 2017] Souradeep Dutta, Susmit Jha, Sriram Sanakaranarayanan, and Ashish Tiwari. Output range anal- ysis for deep neural networks. CoRR, abs/1709.09130, 2017.","title":"Output range analysis for deep neural networks","authors":"Dutta","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ehlers, 2017] Rudiger Ehlers. Formal verification of piece- wise linear feed-forward neural networks. In ATVA, pages 269-286, 2017.","title":"Formal verification of piecewise linear feed-forward neural networks","authors":"Rudiger Ehlers","year":"2017","venue":"ATVA","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fischetti and Jo, 2017] Matteo Fischetti and Jason Jo. Deep neural networks as 0-1 mixed integer linear programs: A feasibility study. CoRR, abs/1712.06174, 2017.","title":"Deep neural networks as 0-1 mixed integer linear programs: A feasibility study","authors":"Matteo Fischetti, Jason Jo","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Frosst and Hinton, 2017] Nicholas Frosst and Geoffrey E. Hinton. Distilling a neural network into a soft decision tree. In Tarek R. Besold and Oliver Kutz, editors, Proceedings of the First International Workshop on Comprehensibility and Explanation in AI and ML 2017, Bari, Italy, November 16th and 17th, 2017., volume 2071. CEUR-WS.org, 2017.","title":"Distilling a neural network into a soft decision tree","authors":"Nicholas Frosst, Geoffrey E Hinton","year":"2017","venue":"Proceedings of the First International Workshop on Comprehensibility and Explanation in AI and ML 2017","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Goodfellow et al., 2014] Ian J. Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sher- jil Ozair, Aaron C. Courville, and Yoshua Bengio. Genera- tive adversarial nets. In Annual Conference on Neural In- formation Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C Courville, Yoshua Bengio","year":"2014","venue":"Annual Conference on Neural Information Processing Systems 2014","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Goodfellow et al., 2015] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. In ICLR'15, 2015.","title":"Explaining and harnessing adversarial examples","authors":"Ian J Goodfellow, Jonathon Shlens, Christian Szegedy","year":"2015","venue":"ICLR'15","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hornung, 1997] Ulrich Hornung, editor. Homogenization and Porous Media. Springer-Verlag New York, Inc., New York, NY, USA, 1997.","title":"Homogenization and Porous Media","authors":null,"year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Huang et al., 2017] Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu. Safety Verification of Deep Neural Networks. In CAV'17, Lecture Notes in Computer Science, pages 3-29. Springer, 2017.","title":"Safety Verification of Deep Neural Networks","authors":"Xiaowei Huang, Marta Kwiatkowska, Sen Wang, Min Wu","year":"2017","venue":"CAV'17","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hubara et al., 2016] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Bina- rized Neural Networks. In Advances in Neural Information Processing Systems 29, pages 4107-4115.","title":"Binarized Neural Networks","authors":"Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In ICML'157, pages 448-456, 2015.","title":"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"ICML'157","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Katz et al., 2017] Guy Katz, Clark Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer. Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks. In CAV'17, pages 97-117, 2017.","title":"Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks","authors":"Guy Katz, Clark Barrett, David L Dill, Kyle Julian, Mykel J Kochenderfer","year":"2017","venue":"CAV'17","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Koh and Liang, 2017] Pang Wei Koh and Percy Liang. Un- derstanding black-box predictions via influence functions. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1885-1894. PMLR, 2017.","title":"Understanding black-box predictions via influence functions","authors":"Pang Wei, Koh, Percy Liang","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning, ICML 2017","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Korneev et al., 2018] Svyatoslav Korneev, Nina Narodytska, Luca Pulina, Armando Tacchella, Nikolaj Bjorner, and Mooly Sagiv. Constrained image generation using bina- rized neural networks with decision procedures. CoRR, abs/1802.08795, 2018.","title":"Constrained image generation using binarized neural networks with decision procedures","authors":"Svyatoslav Korneev, Nina Narodytska, Luca Pulina, Armando Tacchella, Nikolaj Bjorner, Mooly Sagiv","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kung et al., 2017] Jaeha Kung, David Zhang, Gooitzen Van der Wal, Sek Chai, and Saibal Mukhopadhyay. Ef- ficient Object Detection Using Embedded Binarized Neural Networks. Journal of Signal Processing Systems, pages 1-14, 2017.","title":"Efficient Object Detection Using Embedded Binarized Neural Networks","authors":"Kung","year":"2017","venue":"Journal of Signal Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Leofante et al., 2018] Francesco Leofante, Nina Narodytska, Luca Pulina, and Armando Tacchella. Automated verifica- tion of neural networks: Advances, challenges and perspec- tives. CoRR, abs/1805.09938, 2018.","title":"Automated verification of neural networks: Advances, challenges and perspectives","authors":"Francesco Leofante, Nina Narodytska, Luca Pulina, Armando Tacchella","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[McDanel et al., 2017] Bradley McDanel, Surat Teerapit- tayanon, and H. T. Kung. Embedded binarized neural networks. In EWSN, pages 168-173. Junction Publishing, Canada / ACM, 2017.","title":"Embedded binarized neural networks","authors":"Mcdanel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[McMillan, 2003] Kenneth McMillan. Interpolation and SAT- based model checking. In CAV'03, volume 3, pages 1-13. Springer, 2003.","title":"Interpolation and SATbased model checking","authors":"Kenneth Mcmillan","year":"2003","venue":"CAV'03","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[McMillan, 2005] Kenneth L. McMillan. Applications of craig interpolants in model checking. In TACAS, volume 3440 of Lecture Notes in Computer Science, pages 1-12. Springer, 2005.","title":"Applications of craig interpolants in model checking","authors":"L Kenneth, Mcmillan","year":"2005","venue":"TACAS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Moosavi-Dezfooli et al., 2016] Seyed-Mohsen Moosavi- Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. arXiv preprint arXiv:1610.08401, 2016.","title":"Universal adversarial perturbations","authors":"Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, Pascal Frossard","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Narodytska et al., 2017] Nina Narodytska, Shiva Prasad Ka- siviswanathan, Leonid Ryzhyk, Mooly Sagiv, and Toby Walsh. Verifying properties of binarized deep neural net- works. CoRR, abs/1709.06662, 2017.","title":"Verifying properties of binarized deep neural networks","authors":"Nina Narodytska, Shiva Prasad Kasiviswanathan, Leonid Ryzhyk, Mooly Sagiv, Toby Walsh","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Pulina and Tacchella, 2010] Luca Pulina and Armando Tac- chella. An abstraction-refinement approach to verification of artificial neural networks. In CAV, pages 243-257, 2010.","title":"Luca Pulina and Armando Tacchella. An abstraction-refinement approach to verification of artificial neural networks","authors":"Tacchella Pulina","year":"2010","venue":"CAV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Pulina and Tacchella, 2012] Luca Pulina and Armando Tac- chella. Challenging SMT solvers to verify neural networks. AI Commun., 25(2):117-135, 2012.","title":"Luca Pulina and Armando Tacchella. Challenging SMT solvers to verify neural networks","authors":"Tacchella Pulina","year":"2012","venue":"AI Commun","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Reagen et al., 2017] Brandon Reagen, Robert Adolf, Paul N. Whatmough, Gu-Yeon Wei, and David M. Brooks. Deep Learning for Computer Architects. Synthesis Lectures on Computer Architecture. Morgan & Claypool Publishers, 2017.","title":"Deep Learning for Computer Architects","authors":"Brandon Reagen, Robert Adolf, Paul N Whatmough, Gu-Yeon, David M Wei, Brooks","year":"2017","venue":"Synthesis Lectures on Computer Architecture","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Ribeiro et al., 2016] Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. \"why should I trust you?\": Explain- ing the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowl- edge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 1135-1144. ACM, 2016.","title":"why should I trust you?\": Explaining the predictions of any classifier","authors":"Marco Túlio Ribeiro, Sameer Singh, Carlos Guestrin","year":"2016","venue":"Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Simonyan et al., 2013] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional net- works: Visualising image classification models and saliency maps. CoRR, abs/1312.6034, 2013.","title":"Deep inside convolutional networks: Visualising image classification models and saliency maps","authors":"Karen Simonyan, Andrea Vedaldi, Andrew Zisserman","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Sinz, 2005] Carsten Sinz. Towards an optimal CNF encod- ing of boolean cardinality constraints. Technical report, University of Tubingen, 2005.","title":"Towards an optimal CNF encoding of boolean cardinality constraints","authors":"Carsten Sinz","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Szegedy et al., 2014] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.","title":"Intriguing properties of neural networks","authors":"Szegedy","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Tjeng and Tedrake, 2017] Vincent Tjeng and Russ Tedrake. Verifying neural networks with mixed integer programming. CoRR, abs/1711.07356, 2017.","title":"Verifying neural networks with mixed integer programming","authors":"Tedrake Tjeng, Vincent Tjeng, Russ Tedrake","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}