{"type":"ScholarlyArticle","title":"On the Effectiveness of Low Frequency Perturbations","authors":[{"name":"Yash Sharma","slug":"yash-sharma","given":"Yash","family":"Sharma","affiliation":""},{"name":"Gavin Weiguang Ding","slug":"gavin-weiguang-ding","given":"Gavin Weiguang","family":"Ding","affiliation":""},{"name":"Marcus A. Brubaker","slug":"marcus-a-brubaker","given":"Marcus A.","family":"Brubaker","affiliation":""}],"abstract":"Carefully crafted, often imperceptible, adversarial perturbations have been shown to cause state-of-the-art models to yield extremely inaccurate outputs, rendering them unsuitable for safety-critical application domains. In addition, recent work has shown that constraining the attack space to a low frequency regime is particularly effective. Yet, it remains unclear whether this is due to generally constraining the attack search space or specifically removing high frequency components from consideration. By systematically controlling the frequency components of the perturbation, evaluating against the top-placing defense submissions in the NeurIPS 2017 competition, we empirically show that performance improvements in both the white-box and black-box transfer settings are yielded only when low frequency components are preserved. In fact, the defended models based on adversarial training are roughly as vulnerable to low frequency perturbations as undefended models, suggesting that the purported robustness of state-of-the-art ImageNet defenses is reliant upon adversarial perturbations being high frequency in nature. We do find that under L-inf-norm constraint 16/255, the competition distortion bound, low frequency perturbations are indeed perceptible. This questions the use of the L-inf-norm, in particular, as a distortion metric, and, in turn, suggests that explicitly considering the frequency space is promising for learning robust models which better align with human perception.","doi":"10.24963/ijcai.2019/470","url":"https://ijcai.org/proceedings/2019/470","pdf":"https://ijcai.org/proceedings/2019/0470.pdf","markdown":"https://ijcai.org/proceedings/2019/470.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/470","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":470,"pages":{"first":3389,"last":3396},"keywords":["Computer Vision: Perception","Machine Learning: Adversarial Machine Learning","Computer Vision: Statistical Methods and Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Alzantot et al., 2018a] Moustafa Alzantot, Yash Sharma, Supriyo Chakraborty, and Mani Srivastava. Genattack: Practical black-box attacks with gradient-free optimiza- tion. arXiv preprint arXiv:1805.11090, 2018.","title":"Genattack: Practical black-box attacks with gradient-free optimization","authors":"Alzantot","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alzantot et al., 2018b] Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai- Wei Chang. Generating natural language adversarial ex- amples. arXiv preprint arXiv:1804.07998, 2018.","title":"Generating natural language adversarial examples","authors":"Alzantot","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Athalye et al., 2018] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv:1802.00420, 2018.","title":"Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples","authors":"Anish Athalye, Nicholas Carlini, David Wagner","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brendel et al., 2017] Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-based adversarial attacks: Re- liable attacks against black-box machine learning models. arXiv preprint arXiv:1712.04248, 2017.","title":"Decision-based adversarial attacks: Reliable attacks against black-box machine learning models","authors":"Wieland Brendel, Jonas Rauber, Matthias Bethge","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Carlini and Wagner, 2017a] Nicholas Carlini and David Wagner. Adversarial examples are not easily de- tected: Bypassing ten detection methods. arXiv preprint arXiv:1705.07263, 2017.","title":"Adversarial examples are not easily detected: Bypassing ten detection methods","authors":"Wagner Carlini, Nicholas Carlini, David Wagner","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Carlini and Wagner, 2017b] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural net- works. arXiv preprint arXiv:1608.04644, 2017.","title":"Towards evaluating the robustness of neural networks","authors":"Wagner Carlini, Nicholas Carlini, David Wagner","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2017a] Pin-Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsieh. Ead: Elastic-net attacks to deep neural networks via adversarial examples. arXiv preprint arXiv:1709.0414, 2017.","title":"Ead: Elastic-net attacks to deep neural networks via adversarial examples","authors":"Chen","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2017b] Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural net- works without training substitute models. In AISec '17, pages 15-26. ACM, 2017.","title":"Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models","authors":"Chen","year":"2017","venue":"AISec '17","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pat- tern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248-255. IEEE, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ding et al., 2018] Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Max-margin ad- versarial (mma) training: Direct input space margin max- imization through adversarial training. arXiv preprint arXiv:1812.02637, 2018.","title":"Max-margin adversarial (mma) training: Direct input space margin maximization through adversarial training","authors":"Gavin Weiguang, Ding, Yash Sharma, Kry Yik, Chau Lui, Ruitong Huang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dong et al., 2017] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. arXiv preprint arXiv:1710.06081, 2017.","title":"Boosting adversarial attacks with momentum","authors":"Dong","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Goodfellow et al., 2014] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.","title":"Explaining and harnessing adversarial examples","authors":"Ian J Goodfellow, Jonathon Shlens, Christian Szegedy","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Guo et al., 2018] Chuan Guo, Jared S. Frank, and Kilian Q. Weinberger. Low frequency adversarial perturbation. arXiv preprint arXiv:1809.08758, 2018.","title":"Low frequency adversarial perturbation","authors":"Chuan Guo, Jared S Frank, Kilian Q Weinberger","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kurakin et al., 2016] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533, 2016.","title":"Adversarial examples in the physical world","authors":"Alexey Kurakin, Ian Goodfellow, Samy Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kurakin et al., 2018] Alexey Kurakin, Ian Goodfellow, Samy Bengio, Yinpeng Dong, Fangzhou Liao, Ming Liang, Tianyu Pang, Jun Zhu, Xiaolin Hu, Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, Alan Yuille, Sangxia Huang, Yao Zhao, Yuzhe Zhao, Zhonglin Han, Junjiajia Long, Yerkebulan Berdibekov, Takuya Akiba, Seiya Tokui, and Motoki Abe. Adversarial attacks and defences competition. arXiv preprint arXiv:1804.00097, 2018.","title":"Adversarial attacks and defences competition","authors":"Alexey Kurakin, Ian Goodfellow, Samy Bengio, Yinpeng Dong, Fangzhou Liao, Ming Liang, Tianyu Pang, Jun Zhu, Xiaolin Hu, Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, Alan Yuille, Sangxia Huang, Yao Zhao, Yuzhe Zhao, Zhonglin Han, Junjiajia Long, Yerkebulan Berdibekov, Takuya Akiba, Seiya Tokui, Motoki Abe","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lei et al., 2018] Qi Lei, Lingfei Wu, Pin-Yu Chen, Alexan- dros G. Dimakis, Inderjit S. Dhillon, and Michael Wit- brock. Discrete attacks and submodular optimization with applications to text classification. arXiv preprint arXiv:1812.00151, 2018.","title":"Discrete attacks and submodular optimization with applications to text classification","authors":"Lei","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2018] Yingzhen Li, John Bradshaw, and Yash Sharma. Are generative classifiers more robust to adver- sarial attacks? arXiv preprint arXiv:1802.06552, 2018.","title":"Are generative classifiers more robust to adversarial attacks","authors":"Li","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liao et al., 2017] Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser. arXiv preprint arXiv:1712.02976, 2017.","title":"Defense against adversarial attacks using high-level representation guided denoiser","authors":"Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, Jun Zhu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Madry et al., 2017] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.","title":"Towards deep learning models resistant to adversarial attacks","authors":"Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, Adrian Vladu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Papernot and McDaniel, 2018] Nicolas Papernot and Patrick McDaniel. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv:1803.04765, 2018.","title":"Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning","authors":"Nicolas Papernot, Patrick Mcdaniel","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rao and Yip, 2014] K Ramamohan Rao and Ping Yip. Dis- crete cosine transform: algorithms, advantages, applica- tions. Academic press, 2014.","title":"Discrete cosine transform: algorithms, advantages, applications","authors":"Yip ; K Ramamohan Rao, Ping Rao, Yip","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schott et al., 2018] Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adversari- ally robust neural network model on mnist. arXiv preprint arXiv:1805.09190, 2018.","title":"Towards the first adversari-ally robust neural network model on mnist","authors":"Lukas Schott, Jonas Rauber, Matthias Bethge, Wieland Brendel","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sharma and Chen, 2017] Yash Sharma and Pin-Yu Chen. Attacking the madry defense model with l1-based adver- sarial examples. arXiv preprint arXiv:1710.10733, 2017.","title":"Attacking the madry defense model with l1-based adversarial examples","authors":"Chen ; Sharma, Yash Sharma, Pin-Yu Chen","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sharma and Chen, 2018] Yash Sharma and Pin-Yu Chen. Bypassing feature squeezing by increasing adversary strength. arXiv preprint arXiv:1803.09868, 2018.","title":"Bypassing feature squeezing by increasing adversary strength","authors":"Chen ; Sharma, Yash Sharma, Pin-Yu Chen","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sharma et al., 2018] Yash Sharma, Tien-Dung Le, and Moustafa Alzantot. Caad 2018: Generating transferable adversarial examples. arXiv preprint arXiv:1810.01268, 2018.","title":"Caad 2018: Generating transferable adversarial examples","authors":"Yash Sharma, Tien-Dung Le, Moustafa Alzantot","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Smith and Gal, 2018] Lewis Smith and Yarin Gal. Under- standing measures of uncertainty for adversarial example detection. arXiv preprint arXiv:1803.08533, 2018.","title":"Understanding measures of uncertainty for adversarial example detection","authors":"Lewis Smith, Yarin Gal","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Szegedy et al., 2013] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.","title":"Intriguing properties of neural networks","authors":"Szegedy","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Thomas and Elibol, 2017] Anil Thomas and Oguz Elibol. Defense against adversarial attack: 3rd place. https:// github.com/anlthms/nips-2017, 2017.","title":"Defense against adversarial attack: 3rd place","authors":"Anil Thomas, Oguz Elibol","year":null,"venue":null,"doi":null,"url":"https://github.com/anlthms/nips-2017,2017","internal_url":null},{"position":29,"raw":"[Tramèr et al., 2017] Florian Tramèr, Alexey Kurakin, Nico- las Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and de- fenses. arXiv preprint arXiv:1705.07204, 2017.","title":"Ensemble adversarial training: Attacks and defenses","authors":"Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, Patrick Mc-Daniel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xie et al., 2017] Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. arXiv preprint arXiv:1711.01991, 2017.","title":"Mitigating adversarial effects through randomization","authors":"Cihang Xie, Jianyu Wang, Zhishuai Zhang, Alan Zhou Ren, Yuille","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xie et al., 2018] Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan Yuille, and Kaiming He. Feature denois- ing for improving adversarial robustness. arXiv preprint arXiv:1812.03411, 2018.","title":"Feature denoising for improving adversarial robustness","authors":"Xie","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2018] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. arXiv preprint arXiv:1801.03924, 2018.","title":"The unreasonable effectiveness of deep features as a perceptual metric","authors":"Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, Oliver Wang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhou et al., 2018] Wen Zhou, Xin Hou, Yongjun Chen, Mengyun Tang, Xiangqi Huang, Xiang Gan, and Yong Yang;. Transferable adversarial perturbations. In ECCV (14), pages 471-486. Springer, 2018.","title":"Transferable adversarial perturbations","authors":"Wen Zhou, Xin Hou, Yongjun Chen, Mengyun Tang, Xiangqi Huang, Xiang Gan, Yong Yang","year":"2018","venue":"ECCV (14)","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zoph et al., 2017] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable archi- tectures for scalable image recognition. arXiv preprint arXiv:1707.07012, 2017.","title":"Learning transferable architectures for scalable image recognition","authors":"Barret Zoph, Vijay Vasudevan, Jonathon Shlens, V Quoc, Le","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}