{"type":"ScholarlyArticle","title":"CADET: Calibrated Anomaly Detection for Mitigating Hardness Bias","authors":[{"name":"Ailin Deng","slug":"ailin-deng","given":"Ailin","family":"Deng","affiliation":"National University of Singapore"},{"name":"Adam Goodge","slug":"adam-goodge","given":"Adam","family":"Goodge","affiliation":"National University of Singapore"},{"name":"Lang Yi Ang","slug":"lang-yi-ang","given":"Lang Yi","family":"Ang","affiliation":"A * STAR , Singapore"},{"name":"Bryan Hooi","slug":"bryan-hooi","given":"Bryan","family":"Hooi","affiliation":"National University of Singapore"}],"abstract":"The detection of anomalous samples in large, high-dimensional datasets is a challenging task with numerous practical applications. Recently, state-of-the-art performance is achieved with deep learning methods: for example, using the reconstruction error from an autoencoder as anomaly scores. However, the scores are uncalibrated: that is, they follow an unknown distribution and lack a clear interpretation. Furthermore, the reconstruction error is highly influenced by the `hardness' of a given sample, which leads to false negative and false positive errors. In this paper, we empirically show the significance of this hardness bias present in a range of recent deep anomaly detection methods. To mitigate this, we propose an efficient and plug-and-play error calibration method which mitigates this hardness bias in the anomaly scoring without the need to retrain the model. We verify the effectiveness of our method on a range of image, time-series, and tabular datasets and against several baseline methods.","doi":"10.24963/ijcai.2022/278","url":"https://ijcai.org/proceedings/2022/278","pdf":"https://ijcai.org/proceedings/2022/0278.pdf","markdown":"https://ijcai.org/proceedings/2022/278.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/278","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Data Mining","track":"Main Track","number":278,"pages":{"first":2002,"last":2008},"keywords":["AI Ethics, Trust, Fairness: Trustworthy AI","Data Mining: Anomaly/Outlier Detection"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986556","length_sec":86,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984699","length_sec":360,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Aggarwal, 2015] Charu C Aggarwal. Outlier analysis. In Data mining, pages 237-263. Springer, 2015.","title":"Outlier analysis","authors":"C Charu, Aggarwal","year":"2015","venue":"Data mining","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[An, 2015] Jinwon An. Variational Autoencoder based Anomaly Detection using Reconstruction Probability. In SNU Data Mining Center 2015-2 Special Lecture on IE, 2015.","title":"Jinwon An. Variational Autoencoder based Anomaly Detection using Reconstruction Probability","authors":"An","year":"2015","venue":"SNU Data Mining Center 2015-2 Special Lecture on IE","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bartels et al., 1995] Richard H Bartels, John C Beatty, and Brian A Barsky. An introduction to splines for use in com- puter graphics and geometric modeling. Morgan Kauf- mann, 1995.","title":"An introduction to splines for use in computer graphics and geometric modeling","authors":"H Richard, John C Bartels, Brian A Beatty, Barsky","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2018] Zhaomin Chen, Chai Kiat Yeo, Bu Sung Lee, and Chiew Tong Lau. Autoencoder-based network anomaly detection. In 2018 Wireless Telecommunications Symposium (WTS), pages 1-5, 2018.","title":"Autoencoder-based network anomaly detection","authors":"Chen","year":"2018","venue":"Wireless Telecommunications Symposium (WTS)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cortés-Ciriano and Bender, 2019] Isidro Cortés-Ciriano and Andreas Bender. Concepts and applications of conformal prediction in computational drug discovery. arXiv preprint arXiv:1908.03569, 2019.","title":"Concepts and applications of conformal prediction in computational drug discovery","authors":"Cortés-Ciriano And Bender","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Deng and Hooi, 2021] Ailin Deng and Bryan Hooi. Graph neural network-based anomaly detection in multivariate time series. In Proceedings of the AAAI Conference on Ar- tificial Intelligence, volume 35, pages 4027-4035, 2021.","title":"Graph neural network-based anomaly detection in multivariate time series","authors":"Hooi ; ] Ailin Deng, Bryan Deng, Hooi","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dinh et al., 2016] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803, 2016.","title":"Density estimation using real nvp","authors":"Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Eklund et al., 2015] Martin Eklund, Ulf Norinder, Scott Boyer, and Lars Carlsson. The application of conformal prediction to the drug discovery process. Annals of Math- ematics and Artificial Intelligence, 74(1):117-132, 2015.","title":"The application of conformal prediction to the drug discovery process","authors":"Eklund","year":"2015","venue":"Annals of Mathematics and Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goodge and Hooi, 2022] Adam Goodge and Bryan Hooi. Lunar: Unifying local outlier detection methods via graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 2022.","title":"Lunar: Unifying local outlier detection methods via graph neural networks","authors":"Hooi ; ] Goodge, Goodge Adam, Bryan Hooi","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goodge et al., 2020] Adam Goodge, Bryan Hooi, See- Kiong Ng, and Wee Siong Ng. Robustness of autoencoders for anomaly detection under adversarial impact. In IJCAI, pages 1244-1250, 2020.","title":"Robustness of autoencoders for anomaly detection under adversarial impact","authors":"Adam Goodge, Bryan Hooi, See-Kiong Ng, Wee Siong Ng","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2020/173"},{"position":11,"raw":"[Guo et al., 2017] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural net- works. In International Conference on Machine Learning, pages 1321-1330. PMLR, 2017.","title":"On calibration of modern neural networks","authors":"Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q Weinberger","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lei et al., 2018] Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the Ameri- can Statistical Association, 113(523):1094-1111, 2018.","title":"Distribution-free predictive inference for regression","authors":"Jing Lei, G' Max, Alessandro Sell, Ryan J Rinaldo, Larry Tibshirani, Wasserman","year":"2018","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Menon and Williamson, 2018] Aditya Krishna Menon and Robert C Williamson. A loss framework for calibrated anomaly detection. In Proceedings of the 32nd Interna- tional Conference on Neural Information Processing Sys- tems, pages 1494-1504, 2018.","title":"A loss framework for calibrated anomaly detection","authors":"Aditya, Krishna Menon, Robert C Williamson","year":"2018","venue":"Proceedings of the 32nd International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Naeini et al., 2015] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.","title":"Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning","authors":"Naeini","year":"2015","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pang et al., 2021] Guansong Pang, Chunhua Shen, Long- bing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM Computing Surveys (CSUR), 54(2):1-38, 2021.","title":"Deep learning for anomaly detection: A review","authors":"Pang","year":"2021","venue":"ACM Computing Surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Romano et al., 2019] Yaniv Romano, Evan Patterson, and Emmanuel J Candès. Conformalized quantile regression. arXiv preprint arXiv:1905.03222, 2019.","title":"Conformalized quantile regression","authors":"Yaniv Romano, Evan Patterson, Emmanuel J Candès","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Romano et al., 2020] Yaniv Romano, Matteo Sesia, and Emmanuel J Candès. Classification with valid and adap- tive coverage. arXiv preprint arXiv:2006.02544, 2020.","title":"Classification with valid and adaptive coverage","authors":"Yaniv Romano, Matteo Sesia, Emmanuel J Candès","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vovk et al., 2005] Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic learning in a random world. Springer Science & Business Media, 2005.","title":"Algorithmic learning in a random world","authors":"Vladimir Vovk, Alex Gammerman, Glenn Shafer","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vovk et al., 2020] Vladimir Vovk, Ivan Petej, Paolo Toc- caceli, Alexander Gammerman, Ernst Ahlberg, and Lars Carlsson. Conformal calibrators. In Conformal and Probabilistic Prediction and Applications, pages 84-99. PMLR, 2020.","title":"Conformal calibrators","authors":"Vladimir Vovk, Ivan Petej, Paolo Toccaceli, Alexander Gammerman, Ernst Ahlberg, Lars Carlsson","year":"2020","venue":"Conformal and Probabilistic Prediction and Applications","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xiao et al., 2017] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for bench- marking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.","title":"Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms","authors":"Han Xiao, Kashif Rasul, Roland Vollgraf","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yao et al., 2019] Rong Yao, Chongdang Liu, Linxuan Zhang, and Peng Peng. Unsupervised anomaly detection using variational auto-encoder based feature extraction. In 2019 IEEE International Conference on Prognostics and Health Management (ICPHM), pages 1-7. IEEE, 2019.","title":"Unsupervised anomaly detection using variational auto-encoder based feature extraction","authors":"Rong Yao, Chongdang Liu, Linxuan Zhang, Peng Peng","year":"2019","venue":"IEEE International Conference on Prognostics and Health Management (ICPHM)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yu et al., 2011] Dong Yu, Jinyu Li, and Li Deng. Calibra- tion of confidence measures in speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 19(8):2461-2473, 2011.","title":"Calibration of confidence measures in speech recognition","authors":"Dong Yu, Jinyu Li, Li Deng","year":"2011","venue":"IEEE Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhao et al., 2017] Chunhui Zhao, Xueyuan Li, and Haifeng Zhu. Hyperspectral anomaly detection based on stacked denoising autoencoders. Journal of Applied Remote Sens- ing, 11(4):042605, 2017.","title":"Hyperspectral anomaly detection based on stacked denoising autoencoders","authors":"Chunhui Zhao, Xueyuan Li, Haifeng Zhu","year":"2017","venue":"Journal of Applied Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhou and Paffenroth, 2017] Chong Zhou and Randy C Paf- fenroth. Anomaly detection with robust deep autoen- coders. In Proceedings of the 23rd ACM SIGKDD in- ternational conference on knowledge discovery and data mining, pages 665-674, 2017.","title":"Anomaly detection with robust deep autoencoders","authors":"Chong Zhou, Randy C Paffenroth","year":"2017","venue":"Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}