{"type":"ScholarlyArticle","title":"Orion: Online Backdoor Sample Detection via Evolution Deviance","authors":[{"name":"Huayang Huang","slug":"huayang-huang","given":"Huayang","family":"Huang","affiliation":"Key Laboratory of Aerospace Information Security and Trusted Computing , Ministry of Education ,"},{"name":"Qian Wang","slug":"qian-wang","given":"Qian","family":"Wang","affiliation":"Key Laboratory of Aerospace Information Security and Trusted Computing , Ministry of Education ,"},{"name":"Xueluan Gong","slug":"xueluan-gong","given":"Xueluan","family":"Gong","affiliation":"School of Computer Science , Wuhan University , Hubei , China; School of Cyber Science and Engineering , Wuhan University , Hubei , China"},{"name":"Tao Wang","slug":"tao-wang","given":"Tao","family":"Wang","affiliation":"Key Laboratory of Aerospace Information Security and Trusted Computing , Ministry of Education ,"}],"abstract":"Widely-used DNN models are vulnerable to backdoor attacks, where the backdoored model is only triggered by specific inputs but can maintain a high prediction accuracy on benign samples. Existing backdoor input detection strategies rely on the assumption that benign and poisoned samples are separable in the feature representation of the model. However, such an assumption can be broken by advanced feature-hidden backdoor attacks. In this paper, we propose a novel detection framework, dubbed Orion (online backdoor sample detection via evolution deviance). Specifically, we analyze how predictions evolve during a forward pass and find deviations between the shallow and deep outputs of the backdoor inputs. By introducing side nets to track such evolution divergence, Orion eliminates the need for the assumption of latent separability. Additionally, we put forward a scheme to restore the original label of backdoor samples, enabling more robust predictions. Extensive experiments on six attacks, three datasets, and two architectures verify the effectiveness of Orion. It is shown that Orion outperforms state-of-the-art defenses and can identify feature-hidden attacks with an F1-score of 90%, compared to 40% for other detection schemes. Orion can also achieve 80% label recovery accuracy on basic backdoor attacks.","doi":"10.24963/ijcai.2023/96","url":"https://ijcai.org/proceedings/2023/96","pdf":"https://ijcai.org/proceedings/2023/0096.pdf","markdown":"https://ijcai.org/proceedings/2023/96.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/96","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":96,"pages":{"first":864,"last":874},"keywords":["Computer Vision: CV: Adversarial learning, adversarial attack and defense methods","AI Ethics, Trust, Fairness: ETF: Safety and robustness","Machine Learning: ML: Adversarial machine learning"],"videos":[],"references":[{"position":1,"raw":"[Cai et al., 2022] Ruisi Cai, Zhenyu Zhang, Tianlong Chen, Xiaohan Chen, and Zhangyang Wang. Randomized chan- nel shuffling: Minimal-overhead backdoor attack detec- tion without clean datasets. In Advances in Neural Infor- mation Processing Systems, 2022.","title":"Randomized channel shuffling: Minimal-overhead backdoor attack detec-tion without clean datasets","authors":"Ruisi Cai, Zhenyu Zhang, Tianlong Chen, Xiaohan Chen, Zhangyang Wang","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2017] Xinyun Chen, Chang Liu, Bo Li, Kim- berly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526, 2017.","title":"Targeted backdoor attacks on deep learning systems using data poisoning","authors":"Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, Dawn Song","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2019a] Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting back- door attacks on deep neural networks by activation cluster- ing. In Workshop on Artificial Intelligence Safety, 2019.","title":"Detecting backdoor attacks on deep neural networks by activation clustering","authors":"Chen","year":"2019","venue":"Workshop on Artificial Intelligence Safety","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2019b] Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian M. Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. In Workshop on Artificial Intelligence Safety co-located with AAAI, volume 2301, 2019.","title":"Detecting backdoor attacks on deep neural networks by activation clustering","authors":"Chen","year":"2019","venue":"Workshop on Artificial Intelligence Safety co-located with AAAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2020] Xinshi Chen, Hanjun Dai, Yu Li, Xin Gao, and Le Song. Learning to stop while learning to pre- dict. In International Conference on Machine Learning, pages 1520-1530. PMLR, 2020.","title":"Learning to stop while learning to predict","authors":"Xinshi Chen, Hanjun Dai, Yu Li, Xin Gao, Le Song","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chou et al., 2020] Edward Chou, Florian Tramer, and Gi- ancarlo Pellegrino. Sentinet: Detecting localized universal attacks against deep learning systems. In IEEE Security and Privacy Workshops (SPW), pages 48-54. IEEE, 2020.","title":"Sentinet: Detecting localized universal attacks against deep learning systems","authors":"Edward Chou, Florian Tramer, Giancarlo Pellegrino","year":"2020","venue":"IEEE Security and Privacy Workshops (SPW)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gao et al., 2019] Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural net- works. In Annual Computer Security Applications Confer- ence, pages 113-125, 2019.","title":"Strip: A defence against trojan attacks on deep neural networks","authors":"Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Surya Damith C Ranasinghe, Nepal","year":"2019","venue":"Annual Computer Security Applications Conference","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gu et al., 2019] Tianyu Gu, Kang Liu, Brendan Dolan- Gavitt, and Siddharth Garg. Badnets: Evaluating back- dooring attacks on deep neural networks. IEEE Access, 7:47230-47244, 2019.","title":"Badnets: Evaluating backdooring attacks on deep neural networks","authors":"Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, Siddharth Garg","year":"2019","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hayase et al., 2021] Jonathan Hayase, Weihao Kong, Raghav Somani, and Sewoong Oh. Spectre: defending against backdoor attacks using robust statistics. In International Conference on Machine Learning, pages 4129-4139, 2021.","title":"Spectre: defending against backdoor attacks using robust statistics","authors":"Jonathan Hayase, Weihao Kong, Raghav Somani, Sewoong Oh","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pat- tern Recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hu et al., 2020] Ting-Kuei Hu, Tianlong Chen, Haotao Wang, and Zhangyang Wang. Triple wins: Boosting accu- racy, robustness and efficiency together by enabling input- adaptive inference. In International Conference on Learn- ing Representations, 2020.","title":"Triple wins: Boosting accuracy, robustness and efficiency together by enabling inputadaptive inference","authors":"Ting-Kuei Hu, Tianlong Chen, Haotao Wang, Zhangyang Wang","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kaya et al., 2019] Yigitcan Kaya, Sanghyun Hong, and Tu- dor Dumitras. Shallow-deep networks: Understanding and mitigating network overthinking. In International Con- ference on Machine Learning, pages 3301-3310. PMLR, 2019.","title":"Shallow-deep networks: Understanding and mitigating network overthinking","authors":"Kaya","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kolouri et al., 2020] Soheil Kolouri, Aniruddha Saha, Hamed Pirsiavash, and Heiko Hoffmann. Universal litmus patterns: Revealing backdoor attacks in cnns. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 301-310, 2020.","title":"Universal litmus patterns: Revealing backdoor attacks in cnns","authors":"Soheil Kolouri, Aniruddha Saha, Hamed Pirsiavash, Heiko Hoffmann","year":"2020","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Hinton Krizhevsky","year":"2009","venue":"Alex Krizhevsky and Geoffrey Hinton","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kumar et al., 2020] Ram Shankar Siva Kumar, Magnus Nyström, John Lambert, Andrew Marshall, Mario Go- ertzel, Andi Comissoneru, Matt Swann, and Sharon Xia. Adversarial machine learning-industry perspectives. In IEEE Security and Privacy Workshops, pages 69-75. IEEE, 2020.","title":"Adversarial machine learning-industry perspectives","authors":"Ram Shankar, Siva Kumar, Magnus Nyström, John Lambert, Andrew Marshall, Mario Goertzel, Andi Comissoneru, Matt Swann, Sharon Xia","year":"2020","venue":"IEEE Security and Privacy Workshops","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Le and Yang, 2015] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.","title":"Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge","authors":"Yang Le","year":"2015","venue":"CS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lee et al., 2016] Chen-Yu Lee, Patrick W Gallagher, and Zhuowen Tu. Generalizing pooling functions in convo- lutional neural networks: Mixed, gated, and tree. In Ar- tificial Intelligence and Statistics, pages 464-472. PMLR, 2016.","title":"Generalizing pooling functions in convolutional neural networks: Mixed, gated, and tree","authors":"Chen-Yu Lee, Patrick W Gallagher, Zhuowen Tu","year":"2016","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2019] Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In SIGSAC Conference on Computer and Communications Security, pages 1265-1282. ACM, 2019.","title":"Abs: Scanning neural networks for back-doors by artificial brain stimulation","authors":"Liu","year":"2019","venue":"SIGSAC Conference on Computer and Communications Security","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2022] Yingqi Liu, Guangyu Shen, Guanhong Tao, Zhenting Wang, Shiqing Ma, and Xiangyu Zhang. Complex backdoor detection by symmetric feature differ- encing. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15003-15013, 2022.","title":"Complex backdoor detection by symmetric feature differencing","authors":"Liu","year":"2022","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Luo et al., 2016] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. In Advances in Neural Information Processing Systems, volume 29, 2016.","title":"Understanding the effective receptive field in deep convolutional neural networks","authors":"Wenjie Luo, Yujia Li, Raquel Urtasun, Richard Zemel","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ma et al., 2023] Wanlun Ma, Derui Wang, Ruoxi Sun, Min- hui Xue, Sheng Wen, and Yang Xiang. The \"beatrix\" res- urrections: Robust backdoor detection via gram matrices. In Annual Network and Distributed System Security Sym- posium, 2023.","title":"The \"beatrix\" resurrections: Robust backdoor detection via gram matrices","authors":"Ma","year":"2023","venue":"Annual Network and Distributed System Security Symposium","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nguyen and Tran, 2020] Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. In Advances in Neural Information Processing Systems, volume 33, pages 3454-3464, 2020.","title":"Input-aware dynamic backdoor attack","authors":"Anh Tuan, Anh Nguyen, Tran","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Nguyen and Tran, 2021] Tuan Anh Nguyen and Anh Tuan Tran. Wanet -imperceptible warping-based backdoor at- tack. In International Conference on Learning Represen- tations, 2021.","title":"Wanet -imperceptible warping-based backdoor attack","authors":"Anh Tuan, Anh Nguyen, Tran Tuan","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Parkhi et al., 2015] Omkar M Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. British Ma- chine Vision Conference 2015, 2015.","title":"Deep face recognition","authors":"Andrea Omkar M Parkhi, Andrew Vedaldi, Zisserman","year":"2015","venue":"British Machine Vision Conference","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rajkomar et al., 2018] Alvin Rajkomar, Eyal Oren, Kai Chen, Andrew M Dai, Nissan Hajaj, Michaela Hardt, Pe- ter J Liu, Xiaobing Liu, Jake Marcus, Mimi Sun, et al. Scalable and accurate deep learning with electronic health records. NPJ digital medicine, 1(1):1-10, 2018.","title":"Scalable and accurate deep learning with electronic health records","authors":"Alvin Rajkomar, Eyal Oren, Kai Chen, Andrew M Dai, Nissan Hajaj, Michaela Hardt, Peter J Liu, Xiaobing Liu, Jake Marcus, Mimi Sun","year":"2018","venue":"NPJ digital medicine","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Redmon et al., 2016] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Uni- fied, real-time object detection. In IEEE Conference on Computer Vision and Pattern Recognition, pages 779-788, 2016.","title":"You only look once: Unified, real-time object detection","authors":"Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Selvaraju et al., 2017] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In IEEE/CVF International Conference on Computer Vision, pages 618-626, 2017.","title":"Grad-cam: Visual explanations from deep networks via gradient-based localization","authors":"Michael Ramprasaath R Selvaraju, Abhishek Cogswell, Ramakrishna Das, Devi Vedantam, Dhruv Parikh, Batra","year":"2017","venue":"IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Stallkamp et al., 2012] Johannes Stallkamp, Marc Schlips- ing, Jan Salmen, and Christian Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks, 32:323-332, 2012.","title":"Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition","authors":"Johannes Stallkamp, Marc Schlipsing, Jan Salmen, Christian Igel","year":"2012","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sun et al., 2022] Yi Sun, Jian Li, and Xin Xu. Meta-gf: Training dynamic-depth neural networks harmoniously. In European Conference on Computer Vision, pages 691- 708. Springer, 2022.","title":"Meta-gf: Training dynamic-depth neural networks harmoniously","authors":"Yi Sun, Jian Li, Xin Xu","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tang et al., 2021] Di Tang, XiaoFeng Wang, Haixu Tang, and Kehuan Zhang. Demon in the variant: Statistical analysis of dnns for robust backdoor contamination detec- tion. In USENIX Security Symposium, pages 1541-1558. USENIX Association, 2021.","title":"Demon in the variant: Statistical analysis of dnns for robust backdoor contamination detection","authors":"Di Tang, Xiaofeng Wang, Haixu Tang, Kehuan Zhang","year":"2021","venue":"USENIX Security Symposium","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Tran et al., 2018] Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. In Ad- vances in Neural Information Processing Systems, vol- ume 31, 2018.","title":"Spectral signatures in backdoor attacks","authors":"Brandon Tran, Jerry Li, Aleksander Madry","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2019] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In IEEE Symposium on Security and Privacy (SP), pages 707-723. IEEE, 2019.","title":"Neural cleanse: Identifying and mitigating backdoor attacks in neural networks","authors":"Wang","year":"2019","venue":"IEEE Symposium on Security and Privacy (SP)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wang et al., 2022] Zhenting Wang, Kai Mei, Hailun Ding, Juan Zhai, and Shiqing Ma. Rethinking the reverse- engineering of trojan triggers. In Advances in Neural In- formation Processing Systems, 2022.","title":"Rethinking the reverseengineering of trojan triggers","authors":"Wang","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Xu et al., 2021] Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A Gunter, and Bo Li. Detecting ai trojans us- ing meta neural analysis. In IEEE Symposium on Security and Privacy, pages 103-120, 2021.","title":"Detecting ai trojans using meta neural analysis","authors":"Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A Gunter, Bo Li","year":"2021","venue":"IEEE Symposium on Security and Privacy","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhong et al., 2022] Nan Zhong, Zhenxing Qian, and Xin- peng Zhang. Imperceptible backdoor attack: From input space to feature representation. In International Joint Con- ference on Artificial Intelligence, pages 1736-1742, 2022.","title":"Imperceptible backdoor attack: From input space to feature representation","authors":"Nan Zhong, Zhenxing Qian, Xinpeng Zhang","year":"2022","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2022/242"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}