{"type":"ScholarlyArticle","title":"Exploring the Vulnerability of Deep Reinforcement Learning-based Emergency Control for Low Carbon Power Systems","authors":[{"name":"Xu Wan","slug":"xu-wan","given":"Xu","family":"Wan","affiliation":"Zhejiang University"},{"name":"Lanting Zeng","slug":"lanting-zeng","given":"Lanting","family":"Zeng","affiliation":"Zhejiang University"},{"name":"Mingyang Sun","slug":"mingyang-sun","given":"Mingyang","family":"Sun","affiliation":"Zhejiang University"}],"abstract":"Decarbonization of global power systems significantly increases the operational uncertainty and modeling complexity that drive the necessity of widely exploiting cutting-edge Deep Reinforcement Learning (DRL) technologies to realize adaptive and real-time emergency control, which is the last resort for system stability and resiliency. The vulnerability of the DRL-based emergency control scheme may lead to severe real-world security issues if it can not be fully explored before implementing it practically. To this end, this is the first work that comprehensively investigates adversarial attacks and defense mechanisms for DRL-based power system emergency control. In particular, recovery-targeted (RT) adversarial attacks are designed for gradient-based approaches, aiming to dramatically degrade the effectiveness of the conducted emergency control actions to prevent the system from restoring to a stable state. Furthermore, the corresponding robust defense (RD) mechanisms are proposed to actively modify the observations based on the distances of sequential states. Experiments are conducted based on the standard IEEE reliability test system, and the results show that security risks indeed exist in the state-of-the-art DRL-based power system emergency control models. The effectiveness, stealthiness, instantaneity, and transferability of the proposed attacks and defense mechanisms are demonstrated with both white-box and black-box settings.","doi":"10.24963/ijcai.2022/549","url":"https://ijcai.org/proceedings/2022/549","pdf":"https://ijcai.org/proceedings/2022/0549.pdf","markdown":"https://ijcai.org/proceedings/2022/549.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/549","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":549,"pages":{"first":3954,"last":3961},"keywords":["Multidisciplinary Topics and Applications: Smart Cities","Multidisciplinary Topics and Applications: Security and Privacy","Machine Learning: Reinforcement Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986787","length_sec":64,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984930","length_sec":306,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Athay et al., 1979] T Athay, Robin Podmore, and Sudhir Virmani. A practical method for the direct analysis of tran- sient stability. IEEE Transactions on Power Apparatus and Systems, (2):573-584, 1979.","title":"A practical method for the direct analysis of transient stability","authors":"Robin Athay, Sudhir Podmore, Virmani","year":"1979","venue":"IEEE Transactions on Power Apparatus and Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Banijamali and Amraee, 2018] Seyed Sohrab Banijamali and Turaj Amraee. Semi-adaptive setting of under fre- quency load shedding relays considering credible gener- ation outage scenarios. IEEE Transactions on Power De- livery, 34(3):1098-1108, 2018.","title":"Semi-adaptive setting of under frequency load shedding relays considering credible generation outage scenarios","authors":"Seyed Sohrab, Banijamali, Turaj Amraee","year":"2018","venue":"IEEE Transactions on Power Delivery","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018] Yize Chen, Yushi Tan, and Deepjyoti Deka. Is machine learning in power systems vulnera- ble? In 2018 IEEE International Conference on Commu- nications, Control, and Computing Technologies for Smart Grids (SmartGridComm), pages 1-6, 2018.","title":"Is machine learning in power systems vulnerable?","authors":"Chen","year":"2018","venue":"IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2020] Chunyu Chen, Mingjian Cui, Fangxing Li, Shengfei Yin, and Xinan Wang. Model-free emergency frequency control based on reinforcement learning. IEEE Transactions on Industrial Informatics, 17(4):2336-2346, 2020.","title":"Model-free emergency frequency control based on reinforcement learning","authors":"Chunyu Chen, Mingjian Cui, Fangxing Li, Shengfei Yin, Xinan Wang","year":"2020","venue":"IEEE Transactions on Industrial Informatics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dong et al., 2018] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In Proceed- ings of the IEEE conference on computer vision and pat- tern recognition, pages 9185-9193, 2018.","title":"Boosting adversarial attacks with momentum","authors":"Dong","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goodfellow et al., 2014] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.","title":"Explaining and harnessing adversarial examples","authors":"Ian J Goodfellow, Jonathon Shlens, Christian Szegedy","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gottlob et al., 2002] Georg Gottlob, Nicola Leone, and Francesco Scarcello. Hypertree decompositions and tractable queries. Journal of Computer and System Sci- ences, 64(3):579-627, May 2002.","title":"Hypertree decompositions and tractable queries","authors":"Gottlob","year":"2002","venue":"Journal of Computer and System Sciences","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gottlob, 1992] Georg Gottlob. Complexity results for non- monotonic logics. Journal of Logic and Computation, 2(3):397-425, June 1992.","title":"Complexity results for nonmonotonic logics","authors":"Georg Gottlob, Gottlob","year":"1992","venue":"Journal of Logic and Computation","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2017] Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversar- ial attacks on neural network policies. arXiv preprint arXiv:1702.02284, 2017.","title":"Adversarial attacks on neural network policies","authors":"Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, Pieter Abbeel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2019] Qiuhua Huang, Renke Huang, Weituo Hao, Jie Tan, Rui Fan, and Zhenyu Huang. Adaptive power system emergency control using deep reinforcement learn- ing. IEEE Transactions on Smart Grid, 11(2):1171-1182, 2019.","title":"Adaptive power system emergency control using deep reinforcement learning","authors":"Qiuhua Huang, Renke Huang, Weituo Hao, Jie Tan, Rui Fan, Zhenyu Huang","year":"2019","venue":"IEEE Transactions on Smart Grid","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kundur et al., 2000] P Kundur, GK Morison, and L Wang. Techniques for on-line transient stability assessment and control. In 2000 IEEE Power Engineering Society Winter Meeting. Conference Proceedings (Cat. No. 00CH37077), volume 1, pages 46-51. IEEE, 2000.","title":"Techniques for on-line transient stability assessment and control","authors":"Kundur, L Gk Morison, Wang","year":"2000","venue":"IEEE Power Engineering Society Winter Meeting. Conference Proceedings (Cat. No. 00CH37077)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kurakin et al., 2016] Alexey Kurakin, Ian Goodfellow, Samy Bengio, et al. Adversarial examples in the physi- cal world, 2016.","title":"Adversarial examples in the physical world","authors":"Alexey Kurakin, Ian Goodfellow, Samy Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Levesque, 1984a] Hector J. Levesque. Foundations of a functional approach to knowledge representation. Artifi- cial Intelligence, 23(2):155-212, July 1984.","title":"Foundations of a functional approach to knowledge representation","authors":"Hector J Levesque, Levesque","year":"1984","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Levesque, 1984b] Hector J. Levesque. A logic of implicit and explicit belief. In Proceedings of the Fourth Na- tional Conference on Artificial Intelligence, pages 198- 202, Austin, Texas, August 1984. American Association for Artificial Intelligence.","title":"A logic of implicit and explicit belief","authors":"Hector J Levesque, Levesque","year":"1984","venue":"Proceedings of the Fourth National Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2021] Jiangnan Li, Yingyuan Yang, Jinyuan Stella Sun, Kevin Tomsovic, and Hairong Qi. Conaml: Con- strained adversarial machine learning for cyber-physical systems. In Proceedings of the 2021 ACM Asia Confer- ence on Computer and Communications Security, pages 52-66, 2021.","title":"Conaml: Constrained adversarial machine learning for cyber-physical systems","authors":"Li","year":"2021","venue":"Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2017] Yen-Chen Lin, Zhang-Wei Hong, Yuan- Hong Liao, Meng-Li Shih, Ming-Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learn- ing agents. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 3756-3762, 2017.","title":"Tactics of adversarial attack on deep reinforcement learning agents","authors":"Yen-Chen Lin, Zhang-Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming-Yu Liu, Min Sun","year":"2017","venue":"Proceedings of the 26th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2017/525"},{"position":17,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nebel, 2000] Bernhard Nebel. On the compilability and expressive power of propositional planning formalisms. Journal of Artificial Intelligence Research, 12:271-315, 2000.","title":"On the compilability and expressive power of propositional planning formalisms","authors":"Bernhard Nebel","year":"2000","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nikoloska and Simeone, 2021] Ivana Nikoloska and Os- valdo Simeone. Black-box and modular meta-learning for power control via random edge graph neural networks. arXiv preprint arXiv:2108.13178, 2021.","title":"Ivana Nikoloska and Osvaldo Simeone. Black-box and modular meta-learning for power control via random edge graph neural networks","authors":"Simeone Nikoloska","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Papernot et al., 2016] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Anan- thram Swami. The limitations of deep learning in adversar- ial settings. In 2016 IEEE European symposium on secu- rity and privacy (EuroS&P), pages 372-387. IEEE, 2016.","title":"The limitations of deep learning in adversarial settings","authors":"Nicolas Papernot, Patrick Mcdaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, Ananthram Swami","year":"2016","venue":"IEEE European symposium on security and privacy (EuroS&P)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang and He, 2021] Xiaosen Wang and Kun He. Enhanc- ing the transferability of adversarial attacks through vari- ance tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1924- 1933, 2021.","title":"Enhancing the transferability of adversarial attacks through variance tuning","authors":"Wang, He ;, Xiaosen Wang, Kun He","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yan and Xu, 2020] Ziming Yan and Yan Xu. A multi-agent deep reinforcement learning method for cooperative load frequency control of a multi-area power system. IEEE Transactions on Power Systems, 35(6):4599-4608, 2020.","title":"A multi-agent deep reinforcement learning method for cooperative load frequency control of a multi-area power system","authors":"Ziming Yan, Yan Xu","year":"2020","venue":"IEEE Transactions on Power Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2018] Jingyi Zhang, Chao Lu, Chen Fang, Xi- ang Ling, and Yong Zhang. Load shedding scheme with deep reinforcement learning to improve short-term voltage stability. In 2018 IEEE Innovative Smart Grid Technologies-Asia (ISGT Asia), pages 13-18. IEEE, 2018.","title":"Load shedding scheme with deep reinforcement learning to improve short-term voltage stability","authors":"Jingyi Zhang, Chao Lu, Chen Fang, Xiang Ling, Yong Zhang","year":"2018","venue":"IEEE Innovative Smart Grid Technologies-Asia (ISGT Asia)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}