{"type":"ScholarlyArticle","title":"Shaken, and Stirred: Long-Range Dependencies Enable Robust Outlier Detection with PixelCNN++","authors":[{"name":"Barath Mohan Umapathi","slug":"barath-mohan-umapathi","given":"Barath Mohan","family":"Umapathi","affiliation":"Department of Physics , Indian Institute of Science"},{"name":"Kushal Chauhan","slug":"kushal-chauhan","given":"Kushal","family":"Chauhan","affiliation":"Google Research"},{"name":"Pradeep Shenoy","slug":"pradeep-shenoy","given":"Pradeep","family":"Shenoy","affiliation":"Google Research"},{"name":"Devarajan Sridharan","slug":"devarajan-sridharan","given":"Devarajan","family":"Sridharan","affiliation":"Center for Neuroscience , Indian Institute of Science; Computer Science and Automation , Indian Institute of Science"}],"abstract":"Reliable outlier detection is critical for real-world deployment of deep learning models. Although extensively studied, likelihoods produced by deep generative models have been largely dismissed as being impractical for outlier detection. First, deep generative model likelihoods are readily biased by low-level input statistics. Second, many recent solutions for correcting these biases are computationally expensive, or do not generalize well to complex, natural datasets. Here, we explore outlier detection with a state-of-the-art deep autoregressive model: PixelCNN++. We show that biases in PixelCNN++ likelihoods arise primarily from predictions based on local dependencies. We propose two families of bijective transformations -- ``stirring'' and ``shaking'' -- which ameliorate low-level biases and isolate the contribution of long-range dependencies to PixelCNN++ likelihoods. These transformations are inexpensive and readily computed at evaluation time. We test our approaches extensively with five grayscale and six natural image datasets and show that they achieve or exceed state-of-the-art outlier detection, particularly on datasets with complex, natural images. We also show that our solutions work well with other types of generative models (generative flows and variational autoencoders) and that their efficacy is governed by each model's reliance on local dependencies. In sum, lightweight remedies suffice to achieve robust outlier detection on image data with deep generative models.","doi":"10.24963/ijcai.2023/160","url":"https://ijcai.org/proceedings/2023/160","pdf":"https://ijcai.org/proceedings/2023/0160.pdf","markdown":"https://ijcai.org/proceedings/2023/160.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/160","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":160,"pages":{"first":1440,"last":1450},"keywords":["Data Mining: DM: Anomaly/outlier detection","Computer Vision: CV: Neural generative models, auto encoders, GANs","Machine Learning: ML: Robustness"],"videos":[],"references":[{"position":1,"raw":"[Andrews et al., 2016] Jerone Andrews, Edward Morton, and Lewis Griffin. Detecting anomalous data using auto- encoders. International Journal of Machine Learning and Computing, 6:21, 01 2016.","title":"Detecting anomalous data using autoencoders","authors":"Jerone Andrews, Edward Morton, Lewis Griffin","year":null,"venue":"International Journal of Machine Learning and Computing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bergman and Hoshen, 2020] Liron Bergman and Yedid Hoshen. Classification-based anomaly detection for gen- eral data. In International Conference on Learning Repre- sentations, 2020.","title":"Classification-based anomaly detection for general data","authors":"Hoshen Bergman, Liron Bergman, Yedid Hoshen","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cao et al., 2021] Chenjie Cao, Yuxin Hong, Xiang Li, Chengrong Wang, Chengming Xu, Yanwei Fu, and Xi- angyang Xue. The image local autoregressive transformer. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural In- formation Processing Systems, volume 34, pages 18433- 18445.","title":"The image local autoregressive transformer","authors":"Cao","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"Curran Associates, Inc., 2021.","title":null,"authors":"Curran Associates, Inc","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chauhan et al., 2022] Kushal Chauhan, Barath Mohan U, Pradeep Shenoy, Manish Gupta, and Devarajan Sridha- ran. Robust outlier detection by de-biasing vae likeli- hoods. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9881-9890, June 2022.","title":"Robust outlier detection by de-biasing vae likeli-hoods","authors":"Kushal Chauhan, Barath Mohan, U, Pradeep Shenoy, Manish Gupta, Devarajan Sridharan","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Choi et al., 2018] Hyunsun Choi, Eric Jang, and Alexan- der A. Alemi. WAIC, but Why? Generative Ensem- bles for Robust Anomaly Detection. arXiv e-prints, page arXiv:1810.01392, October 2018.","title":"WAIC, but Why? Generative Ensembles for Robust Anomaly Detection","authors":"Choi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cohen et al., 2017] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. Emnist: an exten- sion of mnist to handwritten letters. arXiv preprint arXiv:1702.05373, 2017.","title":"Emnist: an extension of mnist to handwritten letters","authors":"Gregory Cohen, Saeed Afshar, Jonathan Tapson, André Van Schaik","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Deng, 2012] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-142, 2012.","title":"The mnist database of handwritten digit images for machine learning research","authors":"Li Deng","year":"2012","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hendrycks and Gimpel, 2017] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of- distribution examples in neural networks. In International Conference on Learning Representations, 2017.","title":"A baseline for detecting misclassified and out-ofdistribution examples in neural networks","authors":"Gimpel Hendrycks, Dan Hendrycks, Kevin Gimpel","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Johnson et al., 2017] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compo- sitional language and elementary visual reasoning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2901-2910, 2017.","title":"Clevr: A diagnostic dataset for compositional language and elementary visual reasoning","authors":"Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, Ross Girshick","year":"2017","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Dhariwal, 2018] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, vol- ume 31.","title":"Glow: Generative flow with invertible 1x1 convolutions","authors":"P Durk, Prafulla Kingma, Dhariwal","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Curran Associates, Inc., 2018.","title":null,"authors":"Curran Associates, Inc","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Welling, 2014] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd Inter- national Conference on Learning Representations, ICLR 2014, Conference Track Proceedings, 2014.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"2nd International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kirichenko et al., 2020] Polina Kirichenko, Pavel Izmailov, and Andrew G Wilson. Why normalizing flows fail to detect out-of-distribution data. In Advances in neural in- formation processing systems, volume 33, pages 20578- 20589, 2020.","title":"Why normalizing flows fail to detect out-of-distribution data","authors":"Polina Kirichenko, Pavel Izmailov, Andrew G Wilson","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krizhevsky, 2009] Alex Krizhevsky. Cifar-10. https://www. cs.toronto.edu/ ∼ kriz/cifar.html, 2009. Accessed: 2023-05- 29.","title":"Cifar-10","authors":"Alex Krizhevsky","year":"2009","venue":null,"doi":null,"url":"https://www.cs.toronto.edu/∼kriz/cifar.html","internal_url":null},{"position":16,"raw":"[Lakshminarayanan et al., 2017] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scal- able predictive uncertainty estimation using deep ensem- bles. In Advances in Neural Information Processing Sys- tems, volume 30, 2017.","title":"Simple and scalable predictive uncertainty estimation using deep ensembles","authors":"Alexander Balaji Lakshminarayanan, Charles Pritzel, Blundell","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liang et al., 2018] Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image de- tection in neural networks. In 6th International Confer- ence on Learning Representations, ICLR 2018, Confer- ence Track Proceedings, 2018.","title":"Enhancing the reliability of out-of-distribution image detection in neural networks","authors":"Shiyu Liang, Yixuan Li, R Srikant","year":"2018","venue":"6th International Conference on Learning Representations, ICLR 2018, Conference Track Proceedings","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, pages 3730-3738, 2015.","title":"Deep learning face attributes in the wild","authors":"Ziwei Liu, Ping Luo, Xiaogang Wang, Xiaoou Tang","year":"2015","venue":"IEEE International Conference on Computer Vision, ICCV 2015","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nalisnick et al., 2019a] Eric T. Nalisnick, Akihiro Mat- sukawa, Yee Whye Teh, Dilan Görür, and Balaji Laksh- minarayanan. Do deep generative models know what they don't know? In 7th International Conference on Learning Representations, ICLR 2019, 2019.","title":"Do deep generative models know what they don't know?","authors":"Nalisnick","year":"2019","venue":"7th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nalisnick et al., 2019b] Eric T. Nalisnick, Akihiro Mat- sukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detecting out-of-distribution inputs to deep generative models using a test for typicality. arXiv preprint arXiv:1906.02994, 2019.","title":"Detecting out-of-distribution inputs to deep generative models using a test for typicality","authors":"Nalisnick","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Netzer et al., 2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learn- ing. In NIPS Workshop on Deep Learning and Unsuper- vised Feature Learning 2011, 2011.","title":"Reading digits in natural images with unsupervised feature learning","authors":"Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, Andrew Y Ng","year":"2011","venue":"NIPS Workshop on Deep Learning and Unsupervised Feature Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ren et al., 2019] Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of- distribution detection. In Advances in Neural Information Processing Systems, volume 32, 2019.","title":"Likelihood ratios for out-ofdistribution detection","authors":"Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, Balaji Lakshminarayanan","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ruff et al., 2018] Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexan- der Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceed- ings of Machine Learning Research, pages 4393-4402, 10-15 Jul 2018.","title":"Deep one-class classification","authors":"Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Ahmed Shoaib, Alexander Siddiqui, Emmanuel Binder, Marius Müller, Kloft","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Salimans et al., 2017] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelcnn++: Improv- ing the pixelcnn with discretized logistic mixture likeli- hood and other modifications. In 5th International Con- ference on Learning Representations, ICLR 2017, Confer- ence Track Proceedings, 2017.","title":"Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications","authors":"Tim Salimans, Andrej Karpathy, Xi Chen, Diederik P Kingma","year":"2017","venue":"5th International Conference on Learning Representations, ICLR 2017, Conference Track Proceedings","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Serrà et al., 2020] Joan Serrà, David Álvarez, Vicenc ¸ Gómez, Olga Slizovskaia, José F. Núñez, and Jordi Luque. Input complexity and out-of-distribution detection with likelihood-based generative models. In 8th International Conference on Learning Representations, ICLR 2020, Conference Track Proceedings, 2020. [Sign Language MNIST, 2017] Sign Language MNIST. Sign language mnist. https://www.kaggle.com/datasets/ datamunge/sign-language-mnist, 2017. Accessed: 2023-05-29.","title":"Input complexity and out-of-distribution detection with likelihood-based generative models","authors":"Joan Serrà, David Álvarez, ¸ Vicenc, Olga Gómez, José F Slizovskaia, Jordi Núñez, Luque","year":"2017","venue":"8th International Conference on Learning Representations, ICLR 2020, Conference Track Proceedings","doi":null,"url":"https://www.kaggle.com/datasets/datamunge/sign-language-mnist,2017","internal_url":null},{"position":26,"raw":"[Song et al., 2021] Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Sys- tems, volume 34, pages 1415-1428.","title":"Maximum likelihood training of score-based diffusion models","authors":"Yang Song, Conor Durkan, Iain Murray, Stefano Ermon","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"Curran Associates, Inc., 2021.","title":null,"authors":"Curran Associates, Inc","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Stallkamp et al., 2011] Johannes Stallkamp, Marc Schlips- ing, Jan Salmen, and Christian Igel. The German Traffic Sign Recognition Benchmark: A multi-class classification competition. In IEEE International Joint Conference on Neural Networks, pages 1453-1460, 2011.","title":"The German Traffic Sign Recognition Benchmark: A multi-class classification competition","authors":"Johannes Stallkamp, Marc Schlipsing, Jan Salmen, Christian Igel","year":"2011","venue":"IEEE International Joint Conference on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Van Den Oord et al., 2016] Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proceedings of the 33rd Interna- tional Conference on International Conference on Ma- chine Learning -Volume 48, ICML'16, page 1747-1756, 2016.","title":"Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks","authors":"Oord Van Den","year":"2016","venue":"Proceedings of the 33rd International Conference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xiao et al., 2017] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for bench- marking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.","title":"Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms","authors":"Han Xiao, Kashif Rasul, Roland Vollgraf","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xiao et al., 2020] Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An out-of-distribution detection score for variational auto-encoder. In Advances in Neural In- formation Processing Systems, volume 33, pages 20685- 20696, 2020.","title":"Likelihood regret: An out-of-distribution detection score for variational auto-encoder","authors":"Zhisheng Xiao, Qing Yan, Yali Amit","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2015] Linjie Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. A large-scale car dataset for fine- grained categorization and verification. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3973-3981, 2015.","title":"A large-scale car dataset for finegrained categorization and verification","authors":"Yang","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yu et al., 2015] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of a large- scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.","title":"Lsun: Construction of a largescale image dataset using deep learning with humans in the loop","authors":"Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, Jianxiong Xiao","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yun et al., 2019] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Oc- tober 2019.","title":"Cutmix: Regularization strategy to train strong classifiers with localizable features","authors":"Yun","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}