{"type":"ScholarlyArticle","title":"DeepInspect: A Black-box Trojan Detection and Mitigation Framework for Deep Neural Networks","authors":[{"name":"Huili Chen","slug":"huili-chen","given":"Huili","family":"Chen","affiliation":"University of California , San Diego"},{"name":"Cheng Fu","slug":"cheng-fu","given":"Cheng","family":"Fu","affiliation":"University of California , San Diego"},{"name":"Jishen Zhao","slug":"jishen-zhao","given":"Jishen","family":"Zhao","affiliation":"University of California , San Diego"},{"name":"Farinaz Koushanfar","slug":"farinaz-koushanfar","given":"Farinaz","family":"Koushanfar","affiliation":"University of California , San Diego"}],"abstract":"Deep Neural Networks (DNNs) are vulnerable to Neural Trojan (NT) attacks where the adversary injects malicious behaviors during DNN training. This type of ‘backdoor’ attack is activated when the input is stamped with the trigger pattern specified by the attacker, resulting in an incorrect prediction of the model. Due to the wide application of DNNs in various critical fields, it is indispensable to inspect whether the pre-trained DNN has been trojaned before employing a model. Our goal in this paper is to address the security concern on unknown DNN to NT attacks and ensure safe model deployment. We propose DeepInspect, the first black-box Trojan detection solution with minimal prior knowledge of the model. DeepInspect learns the probability distribution of potential triggers from the queried model using a conditional generative model, thus retrieves the footprint of backdoor insertion. In addition to NT detection, we show that DeepInspect’s trigger generator enables effective Trojan mitigation by model patching. We corroborate the effectiveness, efficiency, and scalability of DeepInspect against the state-of-the-art NT attacks across various benchmarks. Extensive experiments show that DeepInspect offers superior detection performance and lower runtime overhead than the prior work.","doi":"10.24963/ijcai.2019/647","url":"https://ijcai.org/proceedings/2019/647","pdf":"https://ijcai.org/proceedings/2019/0647.pdf","markdown":"https://ijcai.org/proceedings/2019/647.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/647","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Multidisciplinary Topics and Applications","track":"Main track","number":647,"pages":{"first":4658,"last":4664},"keywords":["Machine Learning: Adversarial Machine Learning","Multidisciplinary Topics and Applications: Security and Privacy"],"videos":[],"references":[{"position":1,"raw":"[Biggio et al., 2012] Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. arXiv preprint arXiv:1206.6389, 2012.","title":"Poisoning attacks against support vector machines","authors":"Battista Biggio, Blaine Nelson, Pavel Laskov","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2018] Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. arXiv preprint arXiv:1811.03728, 2018.","title":"Detecting backdoor attacks on deep neural networks by activation clustering","authors":"Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, Biplav Srivastava","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chou et al., 2018] Edward Chou, Florian Tramèr, Giancarlo Pellegrino, and Dan Boneh. Sentinet: Detecting physi- cal attacks against deep learning systems. arXiv preprint arXiv:1812.00292, 2018.","title":"Sentinet: Detecting physical attacks against deep learning systems","authors":"Edward Chou, Florian Tramèr, Giancarlo Pellegrino, Dan Boneh","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Esteva et al., 2017] Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542(7639):115, 2017.","title":"Dermatologist-level classification of skin cancer with deep neural networks","authors":"Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, Sebastian Thrun","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fredrikson et al., 2015] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Pro- ceedings of the 22nd ACM SIGSAC Conference on Com- puter and Communications Security, pages 1322-1333. ACM, 2015.","title":"Model inversion attacks that exploit confidence information and basic countermeasures","authors":"Matt Fredrikson, Somesh Jha, Thomas Ristenpart","year":"2015","venue":"Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gu et al., 2017] Tianyu Gu, Brendan Dolan-Gavitt, and Sid- dharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733, 2017.","title":"Badnets: Identifying vulnerabilities in the machine learning model supply chain","authors":"Tianyu Gu, Brendan Dolan-Gavitt, Siddharth Garg","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Isola et al., 2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with con- ditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125-1134, 2017.","title":"Image-to-image translation with conditional adversarial networks","authors":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A Efros","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kang, 2017] Hyeonwoo Kang. Pytorch implementation of conditional generative adversarial networks (cgan) and conditional deep convolutional generative adver- sarial networks (cdcgan). https://github.com/znxlwm/ pytorch-MNIST-CelebA-cGAN-cDCGAN, 2017.","title":"Pytorch implementation of conditional generative adversarial networks (cgan) and conditional deep convolutional generative adversarial networks (cdcgan)","authors":"Hyeonwoo Kang","year":"2017","venue":null,"doi":null,"url":"https://github.com/znxlwm/pytorch-MNIST-CelebA-cGAN-cDCGAN","internal_url":null},{"position":10,"raw":"[Liu et al., 2018] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In 25nd An- nual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California, USA, February 18-221, 2018. The Internet Society, 2018.","title":"Trojaning attack on neural networks","authors":"Liu","year":"2018","venue":"25nd Annual Network and Distributed System Security Symposium, NDSS 2018","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Madry et al., 2017] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.","title":"Towards deep learning models resistant to adversarial attacks","authors":"Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, Adrian Vladu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Parkhi et al., 2015] Omkar M. Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. In BMVC, 2015.","title":"Deep face recognition","authors":"M Omkar, Andrea Parkhi, Andrew Vedaldi, Zisserman","year":"2015","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Redmon et al., 2016] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Uni- fied, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779-788, 2016.","title":"You only look once: Unified, real-time object detection","authors":"Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rosenmai, 2013] Peter Rosenmai. Using the median abso- lute deviation to find outliers. https://eurekastatistics.com/ using-the-median-absolute-deviation-to-find-outliers/, 2013.","title":"Using the median absolute deviation to find outliers","authors":"Peter Rosenmai","year":null,"venue":null,"doi":null,"url":"https://eurekastatistics.com/using-the-median-absolute-deviation-to-find-outliers/,2013","internal_url":null},{"position":15,"raw":"[Rouhani et al., 2018] Bita Darvish Rouhani, Mohammad Samragh, Mojan Javaheripi, Tara Javidi, and Farinaz Koushanfar. Deepfense: Online accelerated defense against adversarial deep learning. In 2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pages 1-8. IEEE, 2018.","title":"Deepfense: Online accelerated defense against adversarial deep learning","authors":"Darvish Bita, Mohammad Rouhani, Mojan Samragh, Tara Javaheripi, Farinaz Javidi, Koushanfar","year":"2018","venue":"IEEE/ACM International Conference on Computer-Aided Design (ICCAD)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rubinstein et al., 2009] Benjamin IP Rubinstein, Blaine Nel- son, Ling Huang, Anthony D Joseph, Shing-hon Lau, Satish Rao, Nina Taft, and JD Tygar. Stealthy poisoning attacks on pca-based anomaly detectors. ACM SIGMETRICS Per- formance Evaluation Review, 37(2):73-74, 2009.","title":"Stealthy poisoning attacks on pca-based anomaly detectors","authors":"Blaine Benjamin Ip Rubinstein, Ling Nelson, Anthony D Huang, Shing-Hon Joseph, Satish Lau, Nina Rao, Taft, Tygar","year":"2009","venue":"ACM SIGMETRICS Performance Evaluation Review","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shafahi et al., 2018] Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poi- soning attacks on neural networks. In Advances in Neural Information Processing Systems, pages 6103-6113, 2018.","title":"Poison frogs! targeted clean-label poisoning attacks on neural networks","authors":"Ali Shafahi, Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, Tom Goldstein","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2019] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Net- works, page 0. IEEE, 2019.","title":"Neural cleanse: Identifying and mitigating backdoor attacks in neural networks","authors":"Wang","year":"2019","venue":"Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}