{"type":"ScholarlyArticle","title":"Decentralized Anomaly Detection in Cooperative Multi-Agent Reinforcement Learning","authors":[{"name":"Kiarash Kazari","slug":"kiarash-kazari","given":"Kiarash","family":"Kazari","affiliation":"Division of Network and Systems Engineering , School of Electrical Engineering and Computer Science KTH Royal Institute of Technology , Stockholm , Sweden"},{"name":"Ezzeldin Shereen","slug":"ezzeldin-shereen","given":"Ezzeldin","family":"Shereen","affiliation":"Division of Network and Systems Engineering , School of Electrical Engineering and Computer Science KTH Royal Institute of Technology , Stockholm , Sweden"},{"name":"Gyorgy Dan","slug":"gyorgy-dan","given":"Gyorgy","family":"Dan","affiliation":"Division of Network and Systems Engineering , School of Electrical Engineering and Computer Science KTH Royal Institute of Technology , Stockholm , Sweden"}],"abstract":"We consider the problem of detecting adversarial attacks against cooperative multi-agent reinforcement learning. We propose a decentralized scheme that allows agents to detect the abnormal behavior of one compromised agent. Our approach is based on a recurrent neural network (RNN) trained during cooperative learning to predict the action distribution of other agents based on local observations. The predicted distribution is used for computing a normality score for the agents, which allows the detection of the misbehavior of other agents. To explore the robustness of the proposed detection scheme, we formulate the worst-case attack against our scheme as a constrained reinforcement learning problem. We propose to compute an attack policy by optimizing the corresponding dual function using reinforcement learning. Extensive simulations on various multi-agent benchmarks show the effectiveness of the proposed detection scheme in detecting state-of-the-art attacks and in limiting the impact of undetectable attacks.","doi":"10.24963/ijcai.2023/19","url":"https://ijcai.org/proceedings/2023/19","pdf":"https://ijcai.org/proceedings/2023/0019.pdf","markdown":"https://ijcai.org/proceedings/2023/19.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/19","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":19,"pages":{"first":162,"last":170},"keywords":["AI Ethics, Trust, Fairness: ETF: Trustworthy AI","Agent-based and Multi-agent Systems: MAS: Multi-agent learning"],"videos":[],"references":[{"position":1,"raw":"[Abusnaina et al., 2021] Ahmed Abusnaina, Yuhang Wu, Sunpreet Arora, Yizhen Wang, Fei Wang, Hao Yang, and David Mohaisen. Adversarial example detection using latent neighborhood graph. In 2021 IEEE/CVF Inter- national Conference on Computer Vision (ICCV), pages 7667-7676, 2021.","title":"Adversarial example detection using latent neighborhood graph","authors":"Ahmed Abusnaina, Yuhang Wu, Sunpreet Arora, Yizhen Wang, Fei Wang, Hao Yang, David Mohaisen","year":"2021","venue":"IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Behzadan and Munir, 2017] Vahid Behzadan and Arslan Munir. Vulnerability of deep reinforcement learning to policy induction attacks. In Proceedings of International Conference on Machine Learning and Data Mining in Pat- tern Recognition, pages 262-275. Springer, 2017.","title":"Vulnerability of deep reinforcement learning to policy induction attacks","authors":"Vahid Behzadan, Arslan Munir","year":"2017","venue":"Proceedings of International Conference on Machine Learning and Data Mining in Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Canese et al., 2021] Lorenzo Canese, Gian Carlo Cardar- illi, Luca Di Nunzio, Rocco Fazzolari, Daniele Giardino, Marco Re, and Sergio Spanò. Multi-agent reinforcement learning: A review of challenges and applications. Applied Sciences, 11(11), 2021.","title":"Multi-agent reinforcement learning: A review of challenges and applications","authors":"Lorenzo Canese, Gian, Carlo Cardarilli, Luca Di Nunzio, Rocco Fazzolari, Daniele Giardino, Marco Re, Sergio Spanò","year":"2021","venue":"Applied Sciences","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gleave et al., 2019] Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Ad- versarial policies: Attacking deep reinforcement learning. arXiv preprint arXiv:1905.10615, 2019.","title":"Adversarial policies: Attacking deep reinforcement learning","authors":"Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, Stuart Russell","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goodfellow et al., 2014] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.","title":"Explaining and harnessing adversarial examples","authors":"Ian J Goodfellow, Jonathon Shlens, Christian Szegedy","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Guo et al., 2022] Jun Guo, Yonghong Chen, Yihang Hao, Zixin Yin, Yin Yu, and Simin Li. Towards comprehen- sive testing on the robustness of cooperative multi-agent reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 115-122, 2022.","title":"Towards comprehensive testing on the robustness of cooperative multi-agent reinforcement learning","authors":"Jun Guo, Yonghong Chen, Yihang Hao, Zixin Yin, Yin Yu, Simin Li","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hausknecht and Stone, 2015] Matthew Hausknecht and Pe- ter Stone. Deep recurrent q-learning for partially observ- able mdps. In AAAI fall symposium series, 2015.","title":"Deep recurrent q-learning for partially observable mdps","authors":"Matthew Hausknecht, Peter Stone","year":"2015","venue":"AAAI fall symposium series","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2017] Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversar- ial attacks on neural network policies. arXiv preprint arXiv:1702.02284, 2017.","title":"Adversarial attacks on neural network policies","authors":"Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, Pieter Abbeel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lehmann et al., 2005] Erich Leo Lehmann, Joseph P Ro- mano, and George Casella. Testing statistical hypotheses, volume 3. Springer, 2005.","title":"Testing statistical hypotheses","authors":"Erich Leo Lehmann, Joseph P Romano, George Casella","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2022] Tianxu Li, Kun Zhu, Nguyen Cong Luong, Dusit Tao Niyato, Qi hui Wu, Yang Zhang, and Bing Chen. Applications of multi-agent reinforcement learning in fu- ture internet: A comprehensive survey. IEEE Communi- cations Surveys & Tutorials, 24:1240-1279, 2022.","title":"Applications of multi-agent reinforcement learning in future internet: A comprehensive survey","authors":"Li","year":"2022","venue":"IEEE Communications Surveys & Tutorials","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 2020] Jieyu Lin, Kristina Dzeparoska, Sai Qian Zhang, Alberto Leon-Garcia, and Nicolas Papernot. On the robustness of cooperative multi-agent reinforcement learning. In Proceedings of IEEE Security and Privacy Workshops (SPW), pages 62-68, 2020.","title":"On the robustness of cooperative multi-agent reinforcement learning","authors":"Lin","year":"2020","venue":"Proceedings of IEEE Security and Privacy Workshops (SPW)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Malhotra et al., 2016] Pankaj Malhotra, Anusha Ramakr- ishnan, Gaurangi Anand, Lovekesh Vig, Puneet Agar- wal, and Gautam Shroff. Lstm-based encoder-decoder for multi-sensor anomaly detection. arXiv preprint arXiv:1607.00148, 2016.","title":"Lstm-based encoder-decoder for multi-sensor anomaly detection","authors":"Pankaj Malhotra, Anusha Ramakrishnan, Gaurangi Anand, Lovekesh Vig, Puneet Agarwal, Gautam Shroff","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Maschler et al., 2020] Michael Maschler, Shmuel Zamir, and Eilon Solan. Game theory. Cambridge University Press, 2020.","title":"Game theory","authors":"Michael Maschler, Shmuel Zamir, Eilon Solan","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mordatch and Abbeel, 2017] Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional lan- guage in multi-agent populations. arXiv preprint arXiv:1703.04908, 2017.","title":"Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations","authors":"Abbeel Mordatch","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Müller et al., 2022] Robert Müller, Steffen Illium, Thomy Phan, Tom Haider, and Claudia Linnhoff-Popien. Towards anomaly detection in reinforcement learning. In Proceed- ings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pages 1799-1803, 2022.","title":"Towards anomaly detection in reinforcement learning","authors":"Robert Müller, Steffen Illium, Thomy Phan, Tom Haider, Claudia Linnhoff-Popien","year":"2022","venue":"Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Oh and Iyengar, 2019] Min-hwan Oh and Garud Iyengar. Sequential anomaly detection using inverse reinforcement learning. In Proceedings of the 25th ACM SIGKDD In- ternational Conference on Knowledge Discovery & data mining, pages 1480-1490, 2019.","title":"Min-hwan Oh and Garud Iyengar. Sequential anomaly detection using inverse reinforcement learning","authors":"Iyengar Oh","year":"2019","venue":"Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & data mining","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Papernot et al., 2016] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Anan- thram Swami. The limitations of deep learning in adver- sarial settings. In Proceedings of IEEE European Sympo- sium on Security and Privacy (EuroS&P), pages 372-387, 2016.","title":"The limitations of deep learning in adversarial settings","authors":"Nicolas Papernot, Patrick Mcdaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, Ananthram Swami","year":"2016","venue":"Proceedings of IEEE European Symposium on Security and Privacy (EuroS&P)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Papoudakis et al., 2021] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Proceedings of NeurIPS, 2021.","title":"Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks","authors":"Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, Stefano V Albrecht","year":"2021","venue":"Proceedings of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Paternain et al., 2019] Santiago Paternain, Luiz Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Con- strained reinforcement learning has zero duality gap. In Proceedings of NeurIPS, 2019.","title":"Constrained reinforcement learning has zero duality gap","authors":"Paternain","year":"2019","venue":"Proceedings of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pattanaik et al., 2017] Anay Pattanaik, Zhenyi Tang, Shui- jing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial at- tacks. arXiv preprint arXiv:1712.03632, 2017.","title":"Robust deep reinforcement learning with adversarial attacks","authors":"Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, Girish Chowdhary","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rashid et al., 2018] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In Proceedings of ICML, pages 4295-4304, 2018.","title":"Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning","authors":"Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, Shimon Whiteson","year":"2018","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Russo and Proutiere, 2021] Alessio Russo and Alexandre Proutiere. Towards optimal attacks on reinforcement learning policies. In Proceedings of American Control Conference (ACC), pages 4561-4567, 2021.","title":"Towards optimal attacks on reinforcement learning policies","authors":"Alessio Russo, Alexandre Proutiere","year":"2021","venue":"Proceedings of American Control Conference (ACC)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Samvelyan et al., 2019] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon White- son. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.04043, 2019.","title":"The StarCraft Multi-Agent Challenge","authors":"Tabish Mikayel Samvelyan, Christian Rashid, Gregory Schroeder De Witt, Nantas Farquhar, Nardelli, G J Tim, Chia-Man Rudner, Hung, H S Philiph, Jakob Torr, Shimon Foerster, Whiteson","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sedlmeier et al., 2020] Andreas Sedlmeier, Robert Müller, Steffen Illium, and Claudia Linnhoff-Popien. Policy en- tropy for out-of-distribution classification. In Proceedings of International Conference on Artificial Neural Networks, pages 420-431, 2020.","title":"Policy entropy for out-of-distribution classification","authors":"Andreas Sedlmeier, Robert Müller, Steffen Illium, Claudia Linnhoff-Popien","year":"2020","venue":"Proceedings of International Conference on Artificial Neural Networks","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shames et al., 2011] Iman Shames, André M.H. Teixeira, Henrik Sandberg, and Karl H. Johansson. Distributed fault detection for interconnected second-order systems. Auto- matica, 47(12):2757-2764, 2011.","title":"Distributed fault detection for interconnected second-order systems","authors":"Iman Shames, M H André, Henrik Teixeira, Karl H Sandberg, Johansson","year":"2011","venue":"Automatica","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sunehag et al., 2017] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296, 2017.","title":"Value-decomposition networks for cooperative multi-agent learning","authors":"Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech, Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tampuu et al., 2017] Ardi Tampuu, Tambet Matiisen, Do- rian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. PloS one, 12(4):e0172395, 2017.","title":"Multiagent cooperation and competition with deep reinforcement learning","authors":"Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, Raul Vicente","year":"2017","venue":"PloS one","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2021] Zhiwei Wang, Zhengzhang Chen, Jingchao Ni, Hui Liu, Haifeng Chen, and Jiliang Tang. Multi-scale one-class recurrent neural networks for dis- crete event sequence anomaly detection. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 3726-3734, 2021.","title":"Multi-scale one-class recurrent neural networks for discrete event sequence anomaly detection","authors":"Zhiwei Wang, Zhengzhang Chen, Jingchao Ni, Hui Liu, Haifeng Chen, Jiliang Tang","year":"2021","venue":"Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Ye et al., 2019] Dan Ye, Meng-Meng Chen, and Hai-Jiao Yang. Distributed adaptive event-triggered fault-tolerant consensus of multiagent systems with general linear dy- namics. IEEE Trans. on Cybernetics, 49(3):757-767, 2019.","title":"Distributed adaptive event-triggered fault-tolerant consensus of multiagent systems with general linear dynamics","authors":"Dan Ye, Meng-Meng Chen, Hai-Jiao Yang","year":"2019","venue":"IEEE Trans. on Cybernetics","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2021] Hongming Zhang, Ke Sun, Bo Xu, Lin- glong Kong, and Martin Müller. A simple unified frame- work for anomaly detection in deep reinforcement learn- ing. arXiv preprint arXiv:2109.09889, 2021.","title":"A simple unified framework for anomaly detection in deep reinforcement learning","authors":"Hongming Zhang, Ke Sun, Bo Xu, Linglong Kong, Martin Müller","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}