{"type":"ScholarlyArticle","title":"Poisoning the Well: Can We Simultaneously Attack a Group of Learning Agents?","authors":[{"name":"Ridhima Bector","slug":"ridhima-bector","given":"Ridhima","family":"Bector","affiliation":"Nanyang Technological University"},{"name":"Hang Xu","slug":"hang-xu","given":"Hang","family":"Xu","affiliation":"Nanyang Technological University"},{"name":"Abhay Aradhya","slug":"abhay-aradhya","given":"Abhay","family":"Aradhya","affiliation":"Nanyang Technological University"},{"name":"Chai Quek","slug":"chai-quek","given":"Chai","family":"Quek","affiliation":"Nanyang Technological University"},{"name":"Zinovi Rabinovich","slug":"zinovi-rabinovich","given":"Zinovi","family":"Rabinovich","affiliation":"Nanyang Technological University"}],"abstract":"Reinforcement Learning's (RL) ubiquity has instigated research on potential threats to its training and deployment. Many works study single-learner training-time attacks that \"pre-programme\" behavioral triggers into a strategy. However, attacks on collections of learning agents remain largely overlooked. We remedy the situation by developing a constructive training-time attack on a population of learning agents and additionally make the attack agnostic to the population's size. The attack constitutes a sequence of environment (re)parameterizations (poisonings), generated to overcome individual differences between agents and lead the entire population to the same target behavior while minimizing effective environment modulation. Our method is demonstrated on populations of independent learners in \"ghost\" environments (learners do not interact or perceive each other) as well as environments with mutual awareness, with or without individual learning. From the attack perspective, we pursue an ultra-blackbox setting, i.e., the attacker's training utilizes only across-policy traces of the victim learners for both attack conditioning and evaluation. The resulting uncertainty in population behavior is managed via a novel Wasserstein distance-based Gaussian embedding of behaviors detected within the victim population. To align with prior works on environment poisoning, our experiments are based on a 3D Grid World domain and show:  a) feasibility, i.e., despite the uncertainty, the attack forces a population-wide adoption of target behavior; b) efficacy, i.e., the attack is size-agnostic and transferable. Code and Appendices are available at \"bit.ly/github-rb-cep\".","doi":"10.24963/ijcai.2023/386","url":"https://ijcai.org/proceedings/2023/386","pdf":"https://ijcai.org/proceedings/2023/0386.pdf","markdown":"https://ijcai.org/proceedings/2023/386.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/386","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":386,"pages":{"first":3470,"last":3478},"keywords":["Machine Learning: ML: Adversarial machine learning","Agent-based and Multi-agent Systems: MAS: Multi-agent learning","Machine Learning: ML: Deep reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Álvarez-Esteban et al., 2016] Pedro C Álvarez-Esteban, E Del Barrio, JA Cuesta-Albertos, and C Matrán. A fixed-point approach to barycenters in wasserstein space. Journal of Mathematical Analysis and Applications, 441(2):744-762, 2016.","title":"A fixed-point approach to barycenters in wasserstein space","authors":"Pedro C Álvarez-Esteban, E Del Barrio, J A Cuesta-Albertos, C Matrán","year":"2016","venue":"Journal of Mathematical Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Behzadan and Munir, 2017] Vahid Behzadan and Arslan Munir. Vulnerability of deep reinforcement learning to policy induction attacks. In International Conference on Machine Learning and Data Mining in Pattern Recogni- tion (MLDM), pages 262-275, 2017.","title":"Vulnerability of deep reinforcement learning to policy induction attacks","authors":"Vahid Behzadan, Arslan Munir","year":"2017","venue":"International Conference on Machine Learning and Data Mining in Pattern Recognition (MLDM)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chelarescu, 2021] Paul Chelarescu. Deception in social learning: A multi-agent reinforcement learning perspec- tive. arXiv preprint arXiv:2106.05402, 2021.","title":"Deception in social learning: A multi-agent reinforcement learning perspective","authors":"Paul Chelarescu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2019] Tong Chen, Jiqiang Liu, Yingxiao Xiang, Wenjia Niu, Endong Tong, and Zhen Han. Adversarial at- tack and defense in reinforcement learning-from ai secu- rity view. Cybersecurity, 2(1):1-22, 2019.","title":"Adversarial attack and defense in reinforcement learning-from ai security view","authors":"Tong Chen, Jiqiang Liu, Yingxiao Xiang, Wenjia Niu, Endong Tong, Zhen Han","year":"2019","venue":"Cybersecurity","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2022] Yanjiao Chen, Zhicong Zheng, and Xueluan Gong. Marnet: Backdoor attacks against coop- erative multi-agent reinforcement learning. IEEE Trans- actions on Dependable and Secure Computing, 2022.","title":"Marnet: Backdoor attacks against cooperative multi-agent reinforcement learning","authors":"Chen","year":"2022","venue":"IEEE Transactions on Dependable and Secure Computing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dimakopoulou et al., 2018] Maria Dimakopoulou, Ian Os- band, and Benjamin Van Roy. Scalable coordinated explo- ration in concurrent reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2018.","title":"Scalable coordinated exploration in concurrent reinforcement learning","authors":"Maria Dimakopoulou, Ian Osband, Benjamin Van, Roy","year":"2018","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ilahi et al., 2021] Inaam Ilahi, Muhammad Usama, Ju- naid Qadir, Muhammad Umar Janjua, Ala Al-Fuqaha, Dinh Thai Hoang, and Dusit Niyato. Challenges and coun- termeasures for adversarial attacks on deep reinforcement learning. IEEE Transactions on Artificial Intelligence, 3(2):90-109, 2021.","title":"Challenges and countermeasures for adversarial attacks on deep reinforcement learning","authors":"Inaam Ilahi, Muhammad Usama, Junaid Qadir, Muhammad Umar Janjua, Ala Al-Fuqaha, Dinh Thai Hoang, Dusit Niyato","year":"2021","venue":"IEEE Transactions on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lample and Chaplot, 2017] Guillaume Lample and Deven- dra Singh Chaplot. Playing fps games with deep rein- forcement learning. In Thirty-First Association for the Advancement of Artificial Intelligence Conference (AAAI), 2017.","title":"Guillaume Lample and Devendra Singh Chaplot. Playing fps games with deep reinforcement learning","authors":"Chaplot Lample","year":"2017","venue":"Thirty-First Association for the Advancement of Artificial Intelligence Conference (AAAI)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lin et al., 2020] Jieyu Lin, Kristina Dzeparoska, Sai Qian Zhang, Alberto Leon-Garcia, and Nicolas Papernot. On the robustness of cooperative multi-agent reinforcement learning. In 2020 IEEE Security and Privacy Workshops (SPW), pages 62-68, 2020.","title":"On the robustness of cooperative multi-agent reinforcement learning","authors":"Lin","year":"2020","venue":"2020 IEEE Security and Privacy Workshops (SPW)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2023] Tongtong Liu, Joe McCalmon, Md Asi- fur Rahman, Cameron Lischke, Talal Halabi, and Sarra Alqahtani. Weaponizing actions in multi-agent reinforce- ment learning: Theoretical and empirical study on security and robustness. In International Conference on Principles and Practice of Multi-Agent Systems (PRIMA), pages 347- 363, 2023.","title":"Weaponizing actions in multi-agent reinforcement learning: Theoretical and empirical study on security and robustness","authors":"Liu","year":"2023","venue":"International Conference on Principles and Practice of Multi-Agent Systems (PRIMA)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lupu and Precup, 2020] Andrei Lupu and Doina Precup. Gifting in multi-agent reinforcement learning. In Proceed- ings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), pages 789-797, 2020.","title":"Gifting in multi-agent reinforcement learning","authors":"Precup Lupu, Andrei Lupu, Doina Precup","year":"2020","venue":"Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Marthi et al., 2005] Bhaskara Marthi, Stuart Russell, David Latham, and Carlos Guestrin. Concurrent hierarchical re- inforcement learning. In Proceedings of the Nineteenth In- ternational Joint Conference on Artificial Intelligence (IJ- CAI), pages 779-785, 2005.","title":"Concurrent hierarchical reinforcement learning","authors":"Bhaskara Marthi, Stuart Russell, David Latham, Carlos Guestrin","year":"2005","venue":"Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence (IJ-CAI)","doi":null,"url":null,"internal_url":"/proceedings/2005/205"},{"position":13,"raw":"[Mialon et al., 2020] Grégoire Mialon, Dexiong Chen, Alexandre d'Aspremont, and Julien Mairal. A trainable optimal transport embedding for feature aggregation. In International Conference on Learning Representations (ICLR), 2020.","title":"A trainable optimal transport embedding for feature aggregation","authors":"Grégoire Mialon, Dexiong Chen, Alexandre Aspremont, Julien Mairal","year":"2020","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Has- sabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Parisotto et al., 2019] Emilio Parisotto, Soham Ghosh, Sai Bhargav Yalamanchi, Varsha Chinnaobireddy, Yuhuai Wu, and Ruslan Salakhutdinov. Concurrent meta rein- forcement learning. arXiv preprint arXiv:1903.02710, 2019.","title":"Concurrent meta reinforcement learning","authors":"Emilio Parisotto, Soham Ghosh, Bhargav Sai, Varsha Yalamanchi, Yuhuai Chinnaobireddy, Ruslan Wu, Salakhutdinov","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pham et al., 2022] Nhan Pham, Lam Nguyen, Jie Chen, Lam Thanh Hoang, Subhro Das, and Lily Weng. c-mba: Adversarial attack for cooperative marl using learned dy- namics model. In Machine Learning Safety Workshop at NeurIPS, 2022.","title":"c-mba: Adversarial attack for cooperative marl using learned dynamics model","authors":"Nhan Pham, Lam Nguyen, Jie Chen, Thanh Lam, Subhro Hoang, Lily Das, Weng","year":"2022","venue":"Machine Learning Safety Workshop at NeurIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Qi et al., 2017] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 652-660, 2017.","title":"Pointnet: Deep learning on point sets for 3d classification and segmentation","authors":"Hao Charles R Qi, Kaichun Su, Leonidas J Mo, Guibas","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rabinovich et al., 2010] Zinovi Rabinovich, Lachlan Dufton, Kate Larson, and Nicholas R. Jennings. Cultivating desired behaviour: Policy teaching via environment-dynamics tweaks. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 1097-1104, 2010.","title":"Cultivating desired behaviour: Policy teaching via environment-dynamics tweaks","authors":"Rabinovich","year":"2010","venue":"Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems (AAMAS)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rakhsha et al., 2020] Amin Rakhsha, Goran Radanovic, Rati Devidze, Xiaojin Zhu, and Adish Singla. Policy teaching via environment poisoning: Training-time adver- sarial attacks against reinforcement learning. In Interna- tional Conference on Machine Learning (ICML), pages 7974-7984. PMLR, 2020.","title":"Policy teaching via environment poisoning: Training-time adversarial attacks against reinforcement learning","authors":"Amin Rakhsha, Goran Radanovic, Rati Devidze, Xiaojin Zhu, Adish Singla","year":"2020","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Skianis et al., 2020] Konstantinos Skianis, Giannis Niko- lentzos, Stratis Limnios, and Michalis Vazirgiannis. Rep the set: Neural networks for learning set representations. In International conference on artificial intelligence and statistics (AISTATS), pages 1410-1420, 2020.","title":"Rep the set: Neural networks for learning set representations","authors":"Konstantinos Skianis, Giannis Nikolentzos, Stratis Limnios, Michalis Vazirgiannis","year":"2020","venue":"International conference on artificial intelligence and statistics (AISTATS)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vaserstein, 1969] Leonid Nisonovich Vaserstein. Markov processes over denumerable products of spaces, describ- ing large systems of automata. Problemy Peredachi Infor- matsii, 5(3):64-72, 1969.","title":"Markov processes over denumerable products of spaces, describing large systems of automata","authors":"Leonid Nisonovich, Vaserstein","year":"1969","venue":"Problemy Peredachi Informatsii","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu et al., 2021] Hang Xu, Rundong Wang, Lev Raizman, and Zinovi Rabinovich. Transferable environment poi- soning: Training-time attack on reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), pages 1398-1406, 2021.","title":"Transferable environment poisoning: Training-time attack on reinforcement learning","authors":"Xu","year":"2021","venue":"Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2022] Hang Xu, Xinghua Qu, and Zinovi Rabi- novich. Spiking pitch black: Poisoning an unknown en- vironment to attack unknown reinforcement learners. In Proceedings of the 21st International Conference on Au- tonomous Agents and Multiagent Systems (AAMAS), pages 1409-1417, 2022.","title":"Spiking pitch black: Poisoning an unknown environment to attack unknown reinforcement learners","authors":"Xu","year":"2022","venue":"Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems (AAMAS)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2020] Jiachen Yang, Ang Li, Mehrdad Fara- jtabar, Peter Sunehag, Edward Hughes, and Hongyuan Zha. Learning to incentivize other learning agents. In Advances in Neural Information Processing Systems (NeurIPS), pages 15208-15219, 2020.","title":"Learning to incentivize other learning agents","authors":"Yang","year":"2020","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zaheer et al., 2017] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in Neural Information Processing Systems (NeurIPS), 2017.","title":"Deep sets","authors":"Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, Alexander J Smola","year":"2017","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}