{"type":"ScholarlyArticle","title":"OptIForest: Optimal Isolation Forest for Anomaly Detection","authors":[{"name":"Haolong Xiang","slug":"haolong-xiang","given":"Haolong","family":"Xiang","affiliation":"Macquarie University"},{"name":"Xuyun Zhang","slug":"xuyun-zhang","given":"Xuyun","family":"Zhang","affiliation":"Macquarie University"},{"name":"Hongsheng Hu","slug":"hongsheng-hu","given":"Hongsheng","family":"Hu","affiliation":"CSIRO's Data61"},{"name":"Lianyong Qi","slug":"lianyong-qi","given":"Lianyong","family":"Qi","affiliation":"Qufu Normal University"},{"name":"Wanchun Dou","slug":"wanchun-dou","given":"Wanchun","family":"Dou","affiliation":"Nanjing University"},{"name":"Mark Dras","slug":"mark-dras","given":"Mark","family":"Dras","affiliation":"Macquarie University"},{"name":"Amin Beheshti","slug":"amin-beheshti","given":"Amin","family":"Beheshti","affiliation":"Macquarie University"},{"name":"Xiaolong Xu","slug":"xiaolong-xu","given":"Xiaolong","family":"Xu","affiliation":"Nanjing University of Information Science and Technology"}],"abstract":"Anomaly detection plays an increasingly important role in various fields for critical tasks such as intrusion detection in cybersecurity, financial risk detection, and human health monitoring. A variety of anomaly detection methods have been proposed, and a category based on the isolation forest mechanism stands out due to its simplicity, effectiveness, and efficiency, e.g., iForest is often employed as a state-of-the-art detector for real deployment. While the majority of isolation forests use the binary structure, a framework LSHiForest has demonstrated that the multi-fork isolation tree structure can lead to better detection performance. However, there is no theoretical work answering the fundamentally and practically important question on the optimal tree structure for an isolation forest with respect to the branching factor. In this paper, we establish a theory on isolation efficiency to answer the question and determine the optimal branching factor for an isolation tree. Based on the theoretical underpinning, we design a practical optimal isolation forest OptIForest incorporating clustering based learning to hash which enables more information to be learned from data for better isolation quality. The rationale of our approach relies on a better bias-variance trade-off achieved by bias reduction in OptIForest. Extensive experiments on a series of benchmarking datasets for comparative and ablation studies demonstrate that our approach can efficiently and robustly achieve better detection performance in general than the state-of-the-arts including the deep learning based methods.","doi":"10.24963/ijcai.2023/264","url":"https://ijcai.org/proceedings/2023/264","pdf":"https://ijcai.org/proceedings/2023/0264.pdf","markdown":"https://ijcai.org/proceedings/2023/264.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/264","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Data Mining","track":"Main Track","number":264,"pages":{"first":2379,"last":2387},"keywords":["Machine Learning: ML: Ensemble methods","Data Mining: DM: Anomaly/outlier detection"],"videos":[],"references":[{"position":1,"raw":"[Aggarwal and Sathe, 2015] Charu C Aggarwal and Saket Sathe. Theoretical foundations and algorithms for outlier ensembles. Acm sigkdd explorations newsletter, 17(1):24- 47, 2015.","title":"Theoretical foundations and algorithms for outlier ensembles","authors":"C Charu, Saket Aggarwal, Sathe","year":"2015","venue":"Acm sigkdd explorations newsletter","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ahmed et al., 2016] Mohiuddin Ahmed, Abdun Naser Mahmood, and Jiankun Hu. A survey of network anomaly detection techniques. Journal of Network and Computer Applications, 60:19-31, 2016.","title":"A survey of network anomaly detection techniques","authors":"Mohiuddin Ahmed, Abdun Naser Mahmood, Jiankun Hu","year":"2016","venue":"Journal of Network and Computer Applications","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bandaragoda et al., 2014] Tharindu R Bandaragoda, Kai Ming Ting, David Albrecht, Fei Tony Liu, and Jonathan R Wells. Efficient anomaly detection by isolation using nearest neighbour ensemble. In IEEE International conference on data mining workshop (ICDM workshop), pages 698-705. IEEE, 2014.","title":"Efficient anomaly detection by isolation using nearest neighbour ensemble","authors":"Kai Tharindu R Bandaragoda, Ming Ting, David Albrecht, Fei, Tony Liu, Jonathan R Wells","year":"2014","venue":"IEEE International conference on data mining workshop (ICDM workshop)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bawa et al., 2005] Mayank Bawa, Tyson Condie, and Prasanna Ganesan. Lsh forest: self-tuning indexes for sim- ilarity search. In Proceedings of the 14th international conference on World Wide Web (WWW), pages 651-660, 2005.","title":"Lsh forest: self-tuning indexes for similarity search","authors":"Mayank Bawa, Tyson Condie, Prasanna Ganesan","year":"2005","venue":"Proceedings of the 14th international conference on World Wide Web (WWW)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chandola et al., 2009] Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM computing surveys, 41(3):1-58, 2009.","title":"Anomaly detection: A survey","authors":"Varun Chandola, Arindam Banerjee, Vipin Kumar","year":"2009","venue":"ACM computing surveys","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen and Tsourakakis, 2022] Tianyi Chen and Charalam- pos Tsourakakis. Antibenford subgraphs: Unsupervised anomaly detection in financial networks. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Dis- covery and Data Mining (SIGKDD), pages 2762-2770, 2022.","title":"Antibenford subgraphs: Unsupervised anomaly detection in financial networks","authors":"Chen, Tsourakakis, Chen, Charalampos Tsourakakis","year":"2022","venue":"Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (SIGKDD)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2017] Jinghui Chen, Saket Sathe, Charu Aggar- wal, and Deepak Turaga. Outlier detection with autoen- coder ensembles. In Proceedings of the 2017 SIAM inter- national conference on data mining (SDM), pages 90-98. SIAM, 2017.","title":"Outlier detection with autoencoder ensembles","authors":"Jinghui Chen, Saket Sathe, Charu Aggarwal, Deepak Turaga","year":"2017","venue":"Proceedings of the 2017 SIAM international conference on data mining (SDM)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fernando et al., 2021] Tharindu Fernando, Harshala Gam- mulle, Simon Denman, Sridha Sridharan, and Clinton Fookes. Deep learning for medical anomaly detection-a survey. ACM Computing Surveys, 54(7):1-37, 2021.","title":"Deep learning for medical anomaly detection-a survey","authors":"Tharindu Fernando, Harshala Gammulle, Simon Denman, Sridha Sridharan, Clinton Fookes","year":"2021","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Han et al., 2022] Songqiao Han, Xiyang Hu, Hailiang Huang, Minqi Jiang, and Yue Zhao. Adbench: Anomaly detection benchmark. Advances in Neural Information Processing Systems (NeurIPS), 2022.","title":"Adbench: Anomaly detection benchmark","authors":"Han","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hariri et al., 2019] Sahand Hariri, Matias Carrasco Kind, and Robert J Brunner. Extended isolation forest. IEEE Transactions on Knowledge and Data Engineering, 33(4):1479-1489, 2019.","title":"Extended isolation forest","authors":"Hariri","year":"2019","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kurt et al., 2020] Mehmet Necip Kurt, Yasin Yilmaz, and Xiaodong Wang. Real-time nonparametric anomaly de- tection in high-dimensional settings. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.","title":"Real-time nonparametric anomaly de-tection in high-dimensional settings","authors":"Mehmet Necip, Kurt, Yasin Yilmaz, Xiaodong Wang","year":"2020","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2022] Zheng Li, Yue Zhao, Xiyang Hu, Nicola Botta, Cezar Ionescu, and George Chen. Ecod: Unsuper- vised outlier detection using empirical cumulative distri- bution functions. IEEE Transactions on Knowledge and Data Engineering, 2022.","title":"Ecod: Unsupervised outlier detection using empirical cumulative distribution functions","authors":"Zheng Li, Yue Zhao, Xiyang Hu, Nicola Botta, Cezar Ionescu, George Chen","year":"2022","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2008] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In IEEE international conference on data mining (ICDM), pages 413-422, 2008.","title":"Isolation forest","authors":"Tony Fei, Kai Liu, Ming Ting, Zhi-Hua Zhou","year":"2008","venue":"IEEE international conference on data mining (ICDM)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2010] Fei Tony Liu, Kai Ming Ting, and Zhi Hua Zhou. On detecting clustered anomalies using scifor- est. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML-PKDD), pages 274-290. Springer, 2010.","title":"On detecting clustered anomalies using sciforest","authors":"Tony Fei, Kai Liu, Ming Ting, Zhi Hua Zhou","year":"2010","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML-PKDD)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2012] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation-based anomaly detection. ACM Trans- actions on Knowledge Discovery from Data, 6(1):1-39, 2012.","title":"Isolation-based anomaly detection","authors":"Tony Fei, Kai Liu, Ming Ting, Zhi-Hua Zhou","year":"2012","venue":"ACM Transactions on Knowledge Discovery from Data","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2019] Yezheng Liu, Zhe Li, Chong Zhou, Yuanchun Jiang, Jianshan Sun, Meng Wang, and Xiang- nan He. Generative adversarial active learning for unsu- pervised outlier detection. IEEE Transactions on Knowl- edge and Data Engineering, 32(8):1517-1528, 2019.","title":"Generative adversarial active learning for unsupervised outlier detection","authors":"Yezheng Liu, Zhe Li, Chong Zhou, Yuanchun Jiang, Jianshan Sun, Meng Wang, Xiangnan He","year":"2019","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pang et al., 2018] Guansong Pang, Longbing Cao, Ling Chen, and Huan Liu. Learning representations of ultrahigh-dimensional data for random distance-based out- lier detection. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining (SIGKDD), pages 2041-2050, 2018.","title":"Learning representations of ultrahigh-dimensional data for random distance-based outlier detection","authors":"Pang","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining (SIGKDD)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pang et al., 2019] Guansong Pang, Chunhua Shen, and An- ton van den Hengel. Deep anomaly detection with devia- tion networks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining (SIGKDD), pages 353-362, 2019.","title":"and Anton van den Hengel. Deep anomaly detection with deviation networks","authors":"Pang","year":"2019","venue":"Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining (SIGKDD)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pang et al., 2021a] Guansong Pang, Chunhua Shen, Long- bing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM Computing Surveys, 54(2):1-38, 2021.","title":"Deep learning for anomaly detection: A review","authors":"Pang","year":"2021","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pang et al., 2021b] Guansong Pang, Anton van den Hengel, Chunhua Shen, and Longbing Cao. Toward deep super- vised anomaly detection: Reinforcement learning from partially labeled anomaly data. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining (SIGKDD), pages 1298-1308, 2021.","title":"Toward deep supervised anomaly detection: Reinforcement learning from partially labeled anomaly data","authors":"Pang","year":"2021","venue":"Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining (SIGKDD)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ruff et al., 2021] Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Woj- ciech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 109(5):756-795, 2021.","title":"A unifying review of deep and shallow anomaly detection","authors":"Lukas Ruff, Robert A Jacob R Kauffmann, Grégoire Vandermeulen, Wojciech Montavon, Marius Samek, Thomas G Kloft, Klaus-Robert Dietterich, Müller","year":"2021","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Russell, 1991] KG Russell. Estimating the value of e by simulation. The American Statistician, 45(1):66-68, 1991.","title":"Estimating the value of e by simulation","authors":"; Kg Russell, Russell","year":"1991","venue":"The American Statistician","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2017] Jingdong Wang, Ting Zhang, Nicu Sebe, Heng Tao Shen, et al. A survey on learning to hash. IEEE transactions on pattern analysis and machine intelligence, 40(4):769-790, 2017.","title":"A survey on learning to hash","authors":"Jingdong Wang, Ting Zhang, Nicu Sebe, Heng Tao Shen","year":"2017","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2021] Hu Wang, Guansong Pang, Chunhua Shen, and Congbo Ma. Unsupervised representation learn- ing by predicting random distances. In International Joint Conference on Artificial Intelligence (IJCAI), pages 2950- 2956, 2021.","title":"Unsupervised representation learning by predicting random distances","authors":"Hu Wang, Guansong Pang, Chunhua Shen, Congbo Ma","year":"2021","venue":"International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2020/408"},{"position":25,"raw":"[Xu et al., 2022] Hongzuo Xu, Guansong Pang, Yijie Wang, and Yongjun Wang. Deep isolation forest for anomaly de- tection. arXiv preprint arXiv:2206.06602, 2022.","title":"Deep isolation forest for anomaly detection","authors":"Xu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zavrtanik et al., 2021] Vitjan Zavrtanik, Matej Kristan, and Danijel Skočaj. Reconstruction by inpainting for visual anomaly detection. Pattern Recognition, 112:107706, 2021.","title":"Reconstruction by inpainting for visual anomaly detection","authors":"Zavrtanik","year":"2021","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zha et al., 2020] Daochen Zha, Kwei-Herng Lai, Mingyang Wan, and Xia Hu. Meta-aad: Active anomaly detection with deep reinforcement learning. In IEEE International conference on data mining (ICDM), pages 771-780. IEEE, 2020.","title":"Meta-aad: Active anomaly detection with deep reinforcement learning","authors":"Zha","year":"2020","venue":"IEEE International conference on data mining (ICDM)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2017] Xuyun Zhang, Wanchun Dou, Qiang He, Rui Zhou, Christopher Leckie, Ramamohanarao Ko- tagiri, and Zoran Salcic. Lshiforest: A generic frame- work for fast tree isolation based ensemble anomaly analy- sis. In IEEE international conference on data engineering (ICDE), pages 983-994, 2017.","title":"Lshiforest: A generic framework for fast tree isolation based ensemble anomaly analysis","authors":"Zhang","year":"2017","venue":"IEEE international conference on data engineering (ICDE)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2021] Huayi Zhang, Lei Cao, Peter VanNos- trand, Samuel Madden, and Elke A Rundensteiner. Elite: Robust deep anomaly detection with meta gradient. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (SIGKDD), pages 2174-2182, 2021.","title":"Elite: Robust deep anomaly detection with meta gradient","authors":"Huayi Zhang, Lei Cao, Peter Vannostrand, Samuel Madden, Elke A Rundensteiner","year":"2021","venue":"Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (SIGKDD)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zimek et al., 2013] Arthur Zimek, Matthew Gaudet, Ri- cardo JGB Campello, and Jörg Sander. Subsampling for efficient and effective unsupervised outlier detection en- sembles. In Proceedings of the 19th ACM SIGKDD in- ternational conference on Knowledge discovery and data mining (SIGKDD), pages 428-436, 2013.","title":"Subsampling for efficient and effective unsupervised outlier detection ensembles","authors":"Arthur Zimek, Matthew Gaudet, Ricardo Jgb Campello, Jörg Sander","year":"2013","venue":"Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining (SIGKDD)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zong et al., 2018] Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In Inter- national conference on learning representations (ICLR), 2018.","title":"Deep autoencoding gaussian mixture model for unsupervised anomaly detection","authors":"Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, Haifeng Chen","year":"2018","venue":"International conference on learning representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}