{"type":"ScholarlyArticle","title":"Experimental Comparison and Survey of Twelve Time Series Anomaly Detection Algorithms (Extended Abstract)","authors":[{"name":"Cynthia Freeman","slug":"cynthia-freeman","given":"Cynthia","family":"Freeman","affiliation":"Verint Systems Inc"},{"name":"Jonathan Merriman","slug":"jonathan-merriman","given":"Jonathan","family":"Merriman","affiliation":"Verint Systems Inc"},{"name":"Ian Beaver","slug":"ian-beaver","given":"Ian","family":"Beaver","affiliation":"Verint Systems Inc"},{"name":"Abdullah Mueen","slug":"abdullah-mueen","given":"Abdullah","family":"Mueen","affiliation":"University of New Mexico"}],"abstract":"The existence of an anomaly detection method that is optimal for all domains is a myth. Thus, there exists a plethora of anomaly detection methods which increases every year for a wide variety of domains. But a strength can also be a weakness; given this massive library of methods, how can one select the best method for their application? Current literature is focused on creating new anomaly detection methods or large frameworks for experimenting with multiple methods at the same time. However, and especially as the literature continues to expand, an extensive evaluation of every anomaly detection method is simply not feasible. To reduce this evaluation burden, we present guidelines to intelligently choose the optimal anomaly detection methods based on the characteristics the time series displays such as seasonality, trend, level change concept drift, and missing time steps. We provide a comprehensive experimental validation and survey of twelve anomaly detection methods over different time series characteristics to form guidelines based on several metrics: the AUC (Area Under the Curve), windowed F-score, and Numenta Anomaly Benchmark (NAB) scoring model. Applying our methodologies can save time and effort by surfacing the most promising anomaly detection methods instead of experimenting extensively with a rapidly expanding library of anomaly detection methods, especially in an online setting.","doi":"10.24963/ijcai.2022/801","url":"https://ijcai.org/proceedings/2022/801","pdf":"https://ijcai.org/proceedings/2022/0801.pdf","markdown":"https://ijcai.org/proceedings/2022/801.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/801","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Journal Track","track":"Journal Track","number":801,"pages":{"first":5737,"last":5741},"keywords":["Data Mining: Anomaly/Outlier Detection","Machine Learning: Time-series; Data Streams","Machine Learning: Applications"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987244","length_sec":82,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985387","length_sec":568,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Campos and others, 2016] Guilherme O Campos et al. On the evaluation of unsupervised outlier detection: measures, datasets, and an empirical study. Data Mining and Knowl- edge Discovery, 30(4):891-927, 2016.","title":"On the evaluation of unsupervised outlier detection: measures, datasets, and an empirical study","authors":"O Guilherme, Campos","year":"2016","venue":"Data Mining and Knowledge Discovery","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chandola and others, 2009] Varun Chandola et al. Anomaly detection: A survey. ACM computing surveys (CSUR), 41(3):15, 2009.","title":"Anomaly detection: A survey","authors":"Varun Chandola","year":"2009","venue":"ACM computing surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cleveland and others, 1990] Robert B Cleveland et al. Stl: A seasonal-trend decomposition. Journal of Official Statis- tics, 6(1):3-73, 1990.","title":"Stl: A seasonal-trend decomposition","authors":"Cleveland Robert","year":"1990","venue":"Journal of Official Statistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cook and others, 2019] Andrew Cook et al. Anomaly de- tection for iot time-series data: A survey. IEEE Internet of Things Journal, 2019.","title":"Anomaly detection for iot time-series data: A survey","authors":"Andrew Cook","year":"2019","venue":"IEEE Internet of Things Journal","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Demšar, 2006] Janez Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research, 7(Jan):1-30, 2006.","title":"Janez Demšar. Statistical comparisons of classifiers over multiple data sets","authors":"Demšar","year":"2006","venue":"Journal of Machine learning research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Emmott and others, 2015] Andrew Emmott et al. A meta- analysis of the anomaly detection problem. arXiv preprint 1503.01158, 2015.","title":"A metaanalysis of the anomaly detection problem","authors":"Andrew Emmott","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Feremans and others, 2019] Len Feremans et al. Pattern- based anomaly detection in mixed-type time series. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 240-256. Springer, 2019.","title":"Patternbased anomaly detection in mixed-type time series","authors":"Len Feremans, Feremans","year":"2019","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Freeman et al., 2021] Cynthia Freeman, Jonathan Merri- man, Ian Beaver, and Abdullah Mueen. Experimental comparison and survey of twelve time series anomaly de- tection algorithms. Journal of Artificial Intelligence Re- search, 72:849-899, 2021.","title":"Experimental comparison and survey of twelve time series anomaly detection algorithms","authors":"Cynthia Freeman, Jonathan Merriman, Ian Beaver, Abdullah Mueen","year":"2021","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gupta and others, 2014] Manish Gupta et al. Outlier de- tection for temporal data: A survey. IEEE Transactions on Knowledge and Data Engineering, 26(9):2250-2267, 2014.","title":"Outlier detection for temporal data: A survey","authors":"Manish Gupta","year":"2014","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hawkins and others, 2010] Jeff Hawkins et al. Hierarchi- cal temporal memory including htm cortical learning al- gorithms. White Paper, Numenta, Inc, Palto Alto, 2010.","title":"Hierarchical temporal memory including htm cortical learning algorithms","authors":"Jeff Hawkins","year":"2010","venue":"White Paper","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Haykin, 2002] Simon Haykin. Adaptive Filter Theory (Fourth Edition).","title":"Adaptive Filter Theory","authors":"Simon Haykin","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hochenbaum and others, 2017] Jordan Hochenbaum et al. Automatic anomaly detection in the cloud via statistical learning. arXiv preprint 1704.07706, 2017.","title":"Automatic anomaly detection in the cloud via statistical learning","authors":"Jordan Hochenbaum, Hochenbaum","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hodge and Austin, 2004] Victoria Hodge and Jim Austin. A survey of outlier detection methodologies. Artificial intel- ligence review, 22(2):85-126, 2004.","title":"Victoria Hodge and Jim Austin. A survey of outlier detection methodologies","authors":"Austin Hodge","year":"2004","venue":"Artificial intelligence review","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hyndman and others, 2015] Rob J Hyndman et al. Large- scale unusual time series detection. In Data Mining Work- shop (ICDMW), 2015 IEEE International Conference on, pages 1616-1619. IEEE, 2015.","title":"Largescale unusual time series detection","authors":"Rob J Hyndman","year":"2015","venue":"IEEE International Conference on","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hyndman, 2018] Rob J Hyndman. Anomalous. https: //github.com/robjhyndman/anomalous, 2018. Accessed: 2021-11-01.","title":"Anomalous","authors":"Rob J Hyndman","year":"2018","venue":null,"doi":null,"url":"https://github.com/robjhyndman/anomalous","internal_url":null},{"position":16,"raw":"[Keogh, 2006] Eamonn Keogh. A decade of progress in in- dexing and mining large time series databases. In Proceed- ings of the 32nd international conference on Very large data bases, pages 1268-1268. VLDB Endowment, 2006.","title":"A decade of progress in indexing and mining large time series databases","authors":"Eamonn Keogh","year":"2006","venue":"Proceedings of the 32nd international conference on Very large data bases","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Laptev and others, 2015] Nikolay Laptev et al. Generic and scalable framework for automated time-series anomaly de- tection. In Proceedings of the 21th ACM SIGKDD Inter- national Conference on Knowledge Discovery and Data Mining, pages 1939-1947. ACM, 2015.","title":"Generic and scalable framework for automated time-series anomaly detection","authors":"Others ; Nikolay Laptev, Laptev","year":"2015","venue":"Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Makridakis and others, 2018] Spyros Makridakis et al. Sta- tistical and machine learning forecasting methods con- cerns and ways forward. PloS one, 13(3), 2018.","title":"Statistical and machine learning forecasting methods concerns and ways forward","authors":"Spyros Makridakis","year":"2018","venue":"PloS one","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Numenta, 2017] Numenta. Anomaly labeling instructions. https://drive.google.com/file/d/ 0B1 XUjaAXeV3YlgwRXdsb3Voa1k/view, 2017. Accessed: 2021-11-21.","title":"Numenta. Anomaly labeling instructions","authors":"Numenta","year":"2017","venue":null,"doi":null,"url":"https://drive.google.com/file/d/0B1XUjaAXeV3YlgwRXdsb3Voa1k/view","internal_url":null},{"position":20,"raw":"[Numenta, 2018] Numenta. The numenta anomaly bench- mark. https://github.com/numenta/NAB, 2018. Accessed: 2021-11-21.","title":"Numenta. The numenta anomaly benchmark","authors":"Numenta","year":"2018","venue":null,"doi":null,"url":"https://github.com/numenta/NAB","internal_url":null},{"position":21,"raw":"[Saurav and others, 2018] Sakti Saurav et al. Online anomaly detection with concept drift adaptation using re- current neural networks. In Proceedings of the ACM India Joint International Conference on Data Science and Man- agement of Data, pages 78-87. ACM, 2018.","title":"Online anomaly detection with concept drift adaptation using recurrent neural networks","authors":"Others ; Sakti Saurav, Saurav","year":"2018","venue":"Proceedings of the ACM India Joint International Conference on Data Science and Management of Data","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tan and others, 2011] Swee Chuan Tan et al. Fast anomaly detection for streaming data. In Twenty-Second Interna- tional Joint Conference on Artificial Intelligence, 2011.","title":"Fast anomaly detection for streaming data","authors":"Tan Swee Chuan","year":"2011","venue":"Twenty-Second International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2011/254"},{"position":23,"raw":"[Taylor and Letham, 2018] Sean J Taylor and Benjamin Letham. Forecasting at scale. The American Statistician, 72(1):37-45, 2018. [Twitter, 2015] Twitter. Anomalydetection. https://github. com/twitter/AnomalyDetectionc, 2015. Accessed: 2021- 11-01.","title":"Forecasting at scale","authors":"J Sean, Benjamin Taylor, Letham","year":"2018","venue":"The American Statistician","doi":null,"url":"https://github.com/twitter/AnomalyDetectionc,2015","internal_url":null},{"position":24,"raw":"[Wu, 2016] Hu-Sheng Wu. A survey of research on anomaly detection for time series. In 2016 13th International Com- puter Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP), pages 426- 431. IEEE, 2016.","title":"A survey of research on anomaly detection for time series","authors":"Hu-Sheng Wu","year":"2016","venue":"13th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu and others, 2018] Haowen Xu et al. Unsupervised anomaly detection via variational auto-encoder for sea- sonal kpis in web applications. In Proceedings of the 2018 World Wide Web Conference on World Wide Web, pages 187-196. International World Wide Web Confer- ences Steering Committee, 2018.","title":"Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications","authors":"Others ; Haowen Xu, Xu","year":"2018","venue":"Proceedings of the 2018 World Wide Web Conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yeh and others, 2018] Chin-Chia Michael Yeh et al. Time series joins, motifs, discords and shapelets: a unifying view that exploits the matrix profile. Data Mining and Knowledge Discovery, 32(1):83-123, 2018.","title":"Time series joins, motifs, discords and shapelets: a unifying view that exploits the matrix profile","authors":"Chin-Chia Michael Yeh","year":"2018","venue":"Data Mining and Knowledge Discovery","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}