{"type":"ScholarlyArticle","title":"Tactics of Adversarial Attack on Deep Reinforcement Learning Agents","authors":[{"name":"Yen-Chen Lin","slug":"yen-chen-lin","given":"Yen-Chen","family":"Lin","affiliation":"National Tsing Hua University , Taiwan"},{"name":"Zhang-Wei Hong","slug":"zhang-wei-hong","given":"Zhang-Wei","family":"Hong","affiliation":"National Tsing Hua University , Taiwan"},{"name":"Yuan-Hong Liao","slug":"yuan-hong-liao","given":"Yuan-Hong","family":"Liao","affiliation":"National Tsing Hua University , Taiwan"},{"name":"Meng-Li Shih","slug":"meng-li-shih","given":"Meng-Li","family":"Shih","affiliation":"National Tsing Hua University , Taiwan"},{"name":"Ming-Yu Liu","slug":"ming-yu-liu","given":"Ming-Yu","family":"Liu","affiliation":"NVIDIA , Santa Clara , California , USA"},{"name":"Min Sun","slug":"min-sun","given":"Min","family":"Sun","affiliation":"National Tsing Hua University , Taiwan"}],"abstract":"We introduce two tactics, namely the strategically-timed attack and the enchanting attack, to attack reinforcement learning agents trained by deep reinforcement learning algorithms using adversarial examples. In the strategically-timed attack, the adversary aims at minimizing the agent's reward by only attacking the agent at a small subset of time steps in an episode. Limiting the attack activity to this subset helps prevent detection of the attack by the agent. We propose a novel method to determine when an adversarial example should be crafted and applied. In the enchanting attack, the adversary aims at luring the agent to a designated target state. This is achieved by combining a generative model and a planning algorithm: while the generative model predicts the future states, the planning algorithm generates a preferred sequence of actions for luring the agent. A sequence of adversarial examples is then crafted to lure the agent to take the preferred sequence of actions. We apply the proposed tactics to the agents trained by the state-of-the-art deep reinforcement learning algorithm including DQN and A3C. In 5 Atari games, our strategically-timed attack reduces as much reward as the uniform attack (i.e., attacking at every time step) does by attacking the agent 4 times less often. Our enchanting attack lures the agent toward designated target states with a more than 70% success rate. Example videos are available at http://yclin.me/adversarial_attack_RL/.","doi":"10.24963/ijcai.2017/525","url":"https://ijcai.org/proceedings/2017/525","pdf":"https://ijcai.org/proceedings/2017/0525.pdf","markdown":"https://ijcai.org/proceedings/2017/525.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/525","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Multidisciplinary Topics and Applications","track":"Main track","number":525,"pages":{"first":3756,"last":3762},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning: Deep Learning","Machine Learning: New Problems","Multidisciplinary Topics and Applications: AI&Security and Privacy"],"videos":[],"references":[{"position":1,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Carlini and Wagner, 2016] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. https://arxiv.org/abs/1608.04644, 2016.","title":"Towards evaluating the robustness of neural networks","authors":"Wagner Carlini, Nicholas Carlini, David Wagner","year":"2016","venue":null,"doi":null,"url":"https://arxiv.org/abs/1608.04644","internal_url":null},{"position":3,"raw":"[Goodfellow et al., 2015] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. In ICLR, 2015.","title":"Explaining and harnessing adversarial examples","authors":"Ian Goodfellow, Jonathon Shlens, Christian Szegedy","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hannun et al., 2014] Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deep speech: Scaling up end-to-end speech recognition. https://arxiv.org/abs/1412.5567, 2014.","title":"Deep speech: Scaling up end-to-end speech recognition","authors":"Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates","year":null,"venue":null,"doi":null,"url":"https://arxiv.org/abs/1412.5567,2014","internal_url":null},{"position":5,"raw":"[Huang et al., 2017] Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. https://arxiv.org/abs/1702.02284, 2017.","title":"Adversarial attacks on neural network policies","authors":"Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, Pieter Abbeel","year":null,"venue":null,"doi":null,"url":"https://arxiv.org/abs/1702.02284,2017","internal_url":null},{"position":6,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kurakin et al., 2016] Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. CoRR, abs/1607.02533, 2016.","title":"Adversarial examples in the physical world","authors":"Alexey Kurakin, Ian J Goodfellow, Samy Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[massoud Farahmand, 2011] Amir massoud Farahmand. Action-gap phenomenon in reinforcement learning. In J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger, editors, Advances in Neural Infor- mation Processing Systems 24, pages 172-180.","title":"Action-gap phenomenon in reinforcement learning","authors":"Farahmand Massoud","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Curran Associates, Inc., 2011.","title":null,"authors":"Curran Associates, Inc","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, and Mehdi Mirza. Asynchronous methods for deep reinforcement learning. In ICML, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Mnih","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Moosavi-Dezfooli et al., 2016] Seyed-Mohsen Moosavi- Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deep- fool: A simple and accurate method to fool deep neural networks. In CVPR, June 2016.","title":"Deepfool: A simple and accurate method to fool deep neural networks","authors":"Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Pascal Frossard","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Oh et al., 2015] Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. In NIPS, 2015.","title":"Action-conditional video prediction using deep networks in atari games","authors":"Oh","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Papernot et al., 2016a] Nicolas Papernot, Patrick Mc- Daniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In Proceedings of the 1st IEEE European Symposium on Security and Privacy, 2016.","title":"The limitations of deep learning in adversarial settings","authors":"Papernot","year":"2016","venue":"Proceedings of the 1st IEEE European Symposium on Security and Privacy","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Papernot et al., 2016b] Nicolas Papernot, Patrick Mc- Daniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In Proceedings of the 37th IEEE Symposium on Security and Privacy, 2016.","title":"Distillation as a defense to adversarial perturbations against deep neural networks","authors":"Papernot","year":"2016","venue":"Proceedings of the 37th IEEE Symposium on Security and Privacy","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rozsa et al., 2016] Andras Rozsa, Manuel Günther, and Terrance E. Boult. Towards robust deep neural networks with BANG. CoRR, abs/1612.00138, 2016.","title":"Towards robust deep neural networks with BANG","authors":"Andras Rozsa, Manuel Günther, Terrance E Boult","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rubinstein and Kroese, 2013] Reuven Y Rubinstein and Dirk P Kroese. The cross-entropy method: a unified ap- proach to combinatorial optimization, Monte-Carlo simu- lation and machine learning. Springer Science & Business Media, 2013.","title":"The cross-entropy method: a unified approach to combinatorial optimization, Monte-Carlo simulation and machine learning","authors":"Y Reuven, Dirk P Rubinstein, Kroese","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NIPS, 2014.","title":"Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Szegedy et al., 2014] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.","title":"Intriguing properties of neural networks","authors":"Szegedy","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zheng et al., 2016] Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. Improving the robustness of deep neural networks via stability training. In CVPR'2016, 2016.","title":"Improving the robustness of deep neural networks via stability training","authors":"Stephan Zheng, Yang Song, Thomas Leung, Ian Goodfellow","year":"2016","venue":"CVPR'2016","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}