{"type":"ScholarlyArticle","title":"Understanding and Mitigating Data Contamination in Deep Anomaly Detection: A Kernel-based Approach","authors":[{"name":"Shuang Wu","slug":"shuang-wu","given":"Shuang","family":"Wu","affiliation":"Huawei Noah's Ark Lab"},{"name":"Jingyu Zhao","slug":"jingyu-zhao","given":"Jingyu","family":"Zhao","affiliation":"The University of Hong Kong"},{"name":"Guangjian Tian","slug":"guangjian-tian","given":"Guangjian","family":"Tian","affiliation":"Huawei Noah's Ark Lab"}],"abstract":"Deep anomaly detection has become popular for its capability of handling complex data. However, training a deep detector is fragile to data contamination due to overfitting. In this work, we study the performance of the anomaly detectors under data contamination and construct a data-efficient countermeasure against data contamination. We show that training a deep anomaly detector induces an implicit kernel machine. We then derive an information-theoretic bound of performance degradation with respect to the data contamination ratio. To mitigate the degradation, we propose a contradicting training approach. Apart from learning normality on the contaminated dataset, our approach discourages learning an additional small auxiliary dataset of labeled anomalies. Our approach is much more affordable than constructing a completely clean training dataset. Experiments on public datasets show that our approach significantly improves anomaly detection in the presence of contamination and outperforms some recently proposed detectors.","doi":"10.24963/ijcai.2022/322","url":"https://ijcai.org/proceedings/2022/322","pdf":"https://ijcai.org/proceedings/2022/0322.pdf","markdown":"https://ijcai.org/proceedings/2022/322.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/322","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Data Mining","track":"Main Track","number":322,"pages":{"first":2319,"last":2325},"keywords":["Data Mining: Anomaly/Outlier Detection","Machine Learning: Kernel Methods"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986577","length_sec":81,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984720","length_sec":333,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Breunig et al., 2000] Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. LOF: iden- tifying density-based local outliers. In SIGMOD, pages 93-104, 2000.","title":"LOF: identifying density-based local outliers","authors":"Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, Jörg Sander","year":"2000","venue":"SIGMOD","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chandola et al., 2009] Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM Computing Surveys, 41(3):1-58, 2009.","title":"Anomaly detection: A survey","authors":"Varun Chandola, Arindam Banerjee, Vipin Kumar","year":"2009","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Diakonikolas et al., 2019] Ilias Diakonikolas, Themis Gouleakis, and Christos Tzamos. Distribution- independent PAC learning of halfspaces with Massart noise. In NeurIPS, 2019.","title":"Distributionindependent PAC learning of halfspaces with Massart noise","authors":"Ilias Diakonikolas, Themis Gouleakis, Christos Tzamos","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Domingos, 2020] Pedro Domingos. Every model learned by gradient descent is approximately a kernel machine. arXiv preprint arXiv:2012.00152, 2020.","title":"Every model learned by gradient descent is approximately a kernel machine","authors":"Pedro Domingos","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dong et al., 2020] Jiahua Dong, Yang Cong, Gan Sun, Bi- neng Zhong, and Xiaowei Xu. What can be transferred: Unsupervised domain adaptation for endoscopic lesions segmentation. In CVPR, pages 4023-4032, 2020.","title":"What can be transferred: Unsupervised domain adaptation for endoscopic lesions segmentation","authors":"Jiahua Dong, Yang Cong, Gan Sun, Bineng Zhong, Xiaowei Xu","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Han et al., 2018] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS, 2018.","title":"Co-teaching: Robust training of deep neural networks with extremely noisy labels","authors":"Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, Masashi Sugiyama","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hendrycks et al., 2018] Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. In NeurIPS, 2018.","title":"Using trusted data to train deep networks on labels corrupted by severe noise","authors":"Dan Hendrycks, Mantas Mazeika, Duncan Wilson, Kevin Gimpel","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jacot et al., 2018] Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In NeurIPS, 2018.","title":"Neural tangent kernel: Convergence and generalization in neural networks","authors":"Arthur Jacot, Clément Hongler, Franck Gabriel","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jehl et al., 2019] Laura Jehl, Carolin Lawrence, and Stefan Riezler. Learning neural sequence-to-sequence models from weak feedback with bipolar ramp loss. TACL, 7:233- 248, 2019.","title":"Learning neural sequence-to-sequence models from weak feedback with bipolar ramp loss","authors":"Laura Jehl, Carolin Lawrence, Stefan Riezler","year":"2019","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2020] Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neu- ral networks. In AISTATS, pages 4313-4324, 2020.","title":"Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks","authors":"Li","year":"2020","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2012] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation-based anomaly detection. TKDD, 6(1):1- 39, 2012.","title":"Isolation-based anomaly detection","authors":"Tony Fei, Kai Liu, Ming Ting, Zhi-Hua Zhou","year":"2012","venue":"TKDD","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mei et al., 2020] Jincheng Mei, Chenjun Xiao, Bo Dai, Li- hong Li, Csaba Szepesvári, and Dale Schuurmans. Escap- ing the gravitational pull of softmax. In NeurIPS, 2020.","title":"Escaping the gravitational pull of softmax","authors":"Mei","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Northcutt et al., 2021] Curtis G Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. arXiv preprint arXiv:2103.14749, 2021.","title":"Pervasive label errors in test sets destabilize machine learning benchmarks","authors":"Anish Curtis G Northcutt, Jonas Athalye, Mueller","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pang et al., 2019] Guansong Pang, Chunhua Shen, and An- ton van den Hengel. Deep anomaly detection with devia- tion networks. In KDD, page 353-362, 2019.","title":"and Anton van den Hengel. Deep anomaly detection with deviation networks","authors":"Pang","year":"2019","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pang et al., 2021a] Guansong Pang, Chunhua Shen, Long- bing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM Computing Surveys, 54(2):1-38, 2021.","title":"Deep learning for anomaly detection: A review","authors":"Pang","year":"2021","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pang et al., 2021b] Guansong Pang, Anton van den Hengel, Chunhua Shen, and Longbing Cao. Toward deep super- vised anomaly detection: Reinforcement learning from partially labeled anomaly data. In KDD, pages 1298-1308, 2021.","title":"Toward deep supervised anomaly detection: Reinforcement learning from partially labeled anomaly data","authors":"Pang","year":"2021","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rolnick et al., 2017] David Rolnick, Andreas Veit, Serge Belongie, and Nir Shavit. Deep learning is robust to mas- sive label noise. arXiv preprint arXiv:1705.10694, 2017.","title":"Deep learning is robust to massive label noise","authors":"David Rolnick, Andreas Veit, Serge Belongie, Nir Shavit","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ruff et al., 2018] Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexan- der Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In ICML, pages 4393-4402, 2018.","title":"Deep one-class classification","authors":"Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Ahmed Shoaib, Alexander Siddiqui, Emmanuel Binder, Marius Müller, Kloft","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ruff et al., 2020] Lukas Ruff, Robert A Vandermeulen, Nico Görnitz, Alexander Binder, Emmanuel Müller, Klaus-Robert Müller, and Marius Kloft. Deep semi- supervised anomaly detection. In ICLR, 2020.","title":"Deep semisupervised anomaly detection","authors":"Lukas Ruff, Robert A Vandermeulen, Nico Görnitz, Alexander Binder, Emmanuel Müller, Klaus-Robert Müller, Marius Kloft","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schlegl et al., 2017] Thomas Schlegl, Philipp Seeböck, Se- bastian M Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In Int. Conf. Info. Proc. Medical Imaging, pages 146-157, 2017.","title":"Unsupervised anomaly detection with generative adversarial networks to guide marker discovery","authors":"Thomas Schlegl, Philipp Seeböck, Ursula Sebastian M Waldstein, Georg Schmidt-Erfurth, Langs","year":"2017","venue":"Int. Conf. Info. Proc. Medical Imaging","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schölkopf et al., 2001] Bernhard Schölkopf, John C Platt, John Shawe-Taylor, Alex J Smola, and Williamson Robert C. Estimating the support of a high-dimensional distribu- tion. Neural Computation, 13(7):1443-1471, 2001.","title":"Estimating the support of a high-dimensional distribution","authors":"Bernhard Schölkopf, John C Platt, John Shawe-Taylor, Alex J Smola, Williamson Robert, C","year":"2001","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shen and Sanghavi, 2019] Yanyao Shen and Sujay Sang- havi. Learning with bad training data via iterative trimmed loss minimization. In ICML, pages 5739-5748, 2019.","title":"Learning with bad training data via iterative trimmed loss minimization","authors":"Sanghavi ; Sujay Shen, Sanghavi","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2021] Shuang Wu, Zhen Qin, Zhen Wang, Siwei Rao, Qiang Ye, Xingyue Quan, and Guangjian Tian. Flex- ible o&m for telecom networks at huawei: A language model-based approach. In CIKM, pages 4224-4233, 2021.","title":"Flexible o&m for telecom networks at huawei: A language model-based approach","authors":"Shuang Wu, Zhen Qin, Zhen Wang, Siwei Rao, Qiang Ye, Xingyue Quan, Guangjian Tian","year":"2021","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ye et al., 2021] Ziyu Ye, Yuxin Chen, and Haitao Zheng. Understanding the effect of bias in deep anomaly detec- tion. In IJCAI, 2021.","title":"Understanding the effect of bias in deep anomaly detection","authors":"Ziyu Ye, Yuxin Chen, Haitao Zheng","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/456"},{"position":25,"raw":"[Zhang et al., 2017] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.","title":"Understanding deep learning requires rethinking generalization","authors":"Zhang","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2020] Zizhao Zhang, Han Zhang, Sercan O Arik, Honglak Lee, and Tomas Pfister. Distilling effec- tive supervision from severe label noise. In ICCV, pages 9294-9303, 2020.","title":"Distilling effective supervision from severe label noise","authors":"Zhang","year":"2020","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2021] Huayi Zhang, Lei Cao, Peter VanNos- trand, Samuel Madden, and Elke A Rundensteiner. Elite: Robust deep anomaly detection with meta gradient. In KDD, pages 2174-2182, 2021.","title":"Elite: Robust deep anomaly detection with meta gradient","authors":"Huayi Zhang, Lei Cao, Peter Vannostrand, Samuel Madden, Elke A Rundensteiner","year":"2021","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou and Paffenroth, 2017] Chong Zhou and Randy Paf- fenroth. Anomaly detection with robust deep autoen- coders. In KDD, page 665-674. ACM, 2017.","title":"Anomaly detection with robust deep autoencoders","authors":"Paffenroth Zhou","year":"2017","venue":"KDD","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}