{"type":"ScholarlyArticle","title":"On Attacking Out-Domain Uncertainty Estimation in Deep Neural Networks","authors":[{"name":"Huimin Zeng","slug":"huimin-zeng","given":"Huimin","family":"Zeng","affiliation":"Unversity of Illinois at Urbana-Champaign"},{"name":"Zhenrui Yue","slug":"zhenrui-yue","given":"Zhenrui","family":"Yue","affiliation":"Unversity of Illinois at Urbana-Champaign"},{"name":"Yang Zhang","slug":"yang-zhang","given":"Yang","family":"Zhang","affiliation":"University of Notre Dame {huiminz3 ,"},{"name":"Ziyi Kou","slug":"ziyi-kou","given":"Ziyi","family":"Kou","affiliation":"Unversity of Illinois at Urbana-Champaign"},{"name":"Lanyu Shang","slug":"lanyu-shang","given":"Lanyu","family":"Shang","affiliation":"Unversity of Illinois at Urbana-Champaign"},{"name":"Dong Wang","slug":"dong-wang","given":"Dong","family":"Wang","affiliation":"Unversity of Illinois at Urbana-Champaign"}],"abstract":"In many applications with real-world consequences, it is crucial to develop reliable uncertainty estimation for the predictions made by the AI decision systems. Targeting at the goal of estimating uncertainty, various deep neural network (DNN) based uncertainty estimation algorithms have been proposed. However, the robustness of the uncertainty returned by these algorithms has not been systematically explored. In this work, to raise the awareness of the research community on robust uncertainty estimation, we show that state-of-the-art uncertainty estimation algorithms could fail catastrophically under our proposed adversarial attack despite their impressive performance on uncertainty estimation. In particular, we aim at attacking out-domain uncertainty estimation: under our attack, the uncertainty model would be fooled to make high-confident predictions for the out-domain data, which they originally would have rejected. Extensive experimental results on various benchmark image datasets show that the uncertainty estimated by state-of-the-art methods could be easily corrupted by our attack.","doi":"10.24963/ijcai.2022/678","url":"https://ijcai.org/proceedings/2022/678","pdf":"https://ijcai.org/proceedings/2022/0678.pdf","markdown":"https://ijcai.org/proceedings/2022/678.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/678","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Uncertainty in AI","track":"Main Track","number":678,"pages":{"first":4893,"last":4899},"keywords":["Uncertainty in AI: Uncertainty Representations","Machine Learning: Adversarial Machine Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986757","length_sec":77,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984900","length_sec":352,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Alarab and Prakoonwit, 2021] Ismail Alarab and Simant Prakoonwit. Adversarial attack for uncertainty estima- tion: Identifying critical regions in neural networks. arXiv preprint arXiv:2107.07618, 2021.","title":"Adversarial attack for uncertainty estimation: Identifying critical regions in neural networks","authors":"Prakoonwit Alarab","year":"2021","venue":"Ismail Alarab and Simant Prakoonwit","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bradshaw et al., 2017] John Bradshaw, Alexander G de G Matthews, and Zoubin Ghahramani. Adversarial ex- amples, uncertainty, and transfer testing robustness in gaussian process hybrid deep networks. arXiv preprint arXiv:1707.02476, 2017.","title":"Adversarial examples, uncertainty, and transfer testing robustness in gaussian process hybrid deep networks","authors":"John Bradshaw, Zoubin Alexander G De G Matthews, Ghahramani","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carlini and Wagner, 2017] Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methodsd. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 3-14, 2017.","title":"Adversarial examples are not easily detected: Bypassing ten detection methodsd","authors":"Wagner Carlini, Nicholas Carlini, David Wagner","year":"2017","venue":"Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Feng et al., 2018] Di Feng, Lars Rosenbaum, and Klaus Di- etmayer. Towards safe autonomous driving: Capture un- certainty in the deep neural network for lidar 3d vehicle detection. In 2018 21st International Conference on Intel- ligent Transportation Systems (ITSC), pages 3266-3273. IEEE, 2018.","title":"Towards safe autonomous driving: Capture uncertainty in the deep neural network for lidar 3d vehicle detection","authors":"Di Feng, Lars Rosenbaum, Klaus Dietmayer","year":"2018","venue":"21st International Conference on Intelligent Transportation Systems (ITSC)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gal and Ghahramani, 2016] Yarin Gal and Zoubin Ghahra- mani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international con- ference on machine learning, pages 1050-1059. PMLR, 2016.","title":"Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning","authors":"Ghahramani Gal","year":"2016","venue":"international conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Galil and El-Yaniv, 2021] Ido Galil and Ran El-Yaniv. Dis- rupting deep uncertainty estimation without harming ac- curacy. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.","title":"Disrupting deep uncertainty estimation without harming accuracy","authors":"El-Yaniv ; Galil, Ido Galil, Ran El-Yaniv","year":"2021","venue":"Thirty-Fifth Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goodfellow et al., 2014] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.","title":"Explaining and harnessing adversarial examples","authors":"Ian J Goodfellow, Jonathon Shlens, Christian Szegedy","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Han et al., 2021] Lei Han, Xiao Dong, and Gianluca De- martini. Iterative human-in-the-loop discovery of un- known unknowns in image datasets. In Proceedings of the AAAI Conference on Human Computation and Crowd- sourcing, volume 9, pages 72-83, 2021.","title":"Iterative human-in-the-loop discovery of unknown unknowns in image datasets","authors":"Lei Han, Xiao Dong, Gianluca Demartini","year":"2021","venue":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kannan et al., 2018] Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing. arXiv preprint arXiv:1803.06373, 2018.","title":"Adversarial logit pairing","authors":"Alexey Harini Kannan, Ian Kurakin, Goodfellow","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lakshminarayanan et al., 2016] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scal- able predictive uncertainty estimation using deep ensem- bles. arXiv preprint arXiv:1612.01474, 2016.","title":"Simple and scalable predictive uncertainty estimation using deep ensembles","authors":"Alexander Balaji Lakshminarayanan, Charles Pritzel, Blundell","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2020] Jeremiah Zhe Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax-Weiss, and Balaji Lakshmi- narayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. arXiv preprint arXiv:2006.10108, 2020.","title":"Simple and principled uncertainty estimation with deterministic deep learning via distance awareness","authors":"Liu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Netzer et al., 2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learn- ing. NeurIPS Workshop on Deep Learning and Unsuper- vised Feature Learning, 2011.","title":"Reading digits in natural images with unsupervised feature learning","authors":"Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, Andrew Y Ng","year":"2011","venue":"NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Reiter, 1981] Raymond Reiter. On closed world data bases. In Readings in artificial intelligence, pages 119-140.","title":"On closed world data bases","authors":"Raymond Reiter","year":null,"venue":"Readings in artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Szegedy et al., 2013] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.","title":"Intriguing properties of neural networks","authors":"Szegedy","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Van Amersfoort et al., 2020] Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty esti- mation using a single deep deterministic neural network. In International Conference on Machine Learning, pages 9690-9700. PMLR, 2020.","title":"Uncertainty estimation using a single deep deterministic neural network","authors":"Van Amersfoort","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[van Amersfoort et al., 2021] Joost van Amersfoort, Lewis Smith, Andrew Jesson, Oscar Key, and Yarin Gal. On fea- ture collapse and deep kernel learning for single forward pass uncertainty. arXiv preprint arXiv:2102.11409, 2021.","title":"On feature collapse and deep kernel learning for single forward pass uncertainty","authors":"Van Amersfoort","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}