{"type":"ScholarlyArticle","title":"Safety Constrained Multi-Agent Reinforcement Learning for Active Voltage Control","authors":[{"name":"Yang Qu","slug":"yang-qu","given":"Yang","family":"Qu","affiliation":"School of Computer Science and Technology , University of Science and Technology of China , Hefei , China"},{"name":"Jinming Ma","slug":"jinming-ma","given":"Jinming","family":"Ma","affiliation":"School of Computer Science and Technology , University of Science and Technology of China , Hefei , China"},{"name":"Feng Wu","slug":"feng-wu","given":"Feng","family":"Wu","affiliation":"School of Computer Science and Technology , University of Science and Technology of China , Hefei , China"}],"abstract":"Active voltage control presents a promising avenue for relieving power congestion and enhancing voltage quality, taking advantage of the distributed controllable generators in the power network, such as roof-top photovoltaics. While Multi-Agent Reinforcement Learning (MARL) has emerged as a compelling approach to address this challenge, existing MARL approaches tend to overlook the constrained optimization nature of this problem, failing in guaranteeing safety constraints. In this paper, we formalize the active voltage control problem as a constrained Markov game and propose a safety-constrained MARL algorithm. We expand the primal-dual optimization RL method to multi-agent settings, and augment it with a novel approach of double safety estimation to learn the policy and to update the Lagrange-multiplier. In addition, we proposed different cost functions and investigated their influences on the behavior of our constrained MARL method. We evaluate our approach in the power distribution network simulation environment with real-world scale scenarios. Experimental results demonstrate the effectiveness of the proposed method compared with the state-of-the-art MARL methods.","doi":"10.24963/ijcai.2024/21","url":"https://ijcai.org/proceedings/2024/21","pdf":"https://ijcai.org/proceedings/2024/0021.pdf","markdown":"https://ijcai.org/proceedings/2024/21.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/21","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":21,"pages":{"first":184,"last":192},"keywords":["Machine Learning: ML: Multiagent Reinforcement Learning","Agent-based and Multi-agent Systems: MAS: Multi-agent learning","Agent-based and Multi-agent Systems: MAS: Applications","Agent-based and Multi-agent Systems: MAS: Coordination and cooperation"],"videos":[],"references":[{"position":1,"raw":"[Achiam et al., 2017] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimiza- tion. In Doina Precup and Yee Whye Teh, editors, Pro- ceedings of the 34th International Conference on Ma- chine Learning (ICML), volume 70 of Proceedings of Ma- chine Learning Research, pages 22-31. PMLR, 06-11 Aug 2017.","title":"Constrained policy optimization","authors":"Achiam","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ackermann et al., 2019] Johannes Ackermann, Volker Gabler, Takayuki Osa, and Masashi Sugiyama. Reducing overestimation bias in multi-agent domains using double centralized critics, 2019.","title":"Reducing overestimation bias in multi-agent domains using double centralized critics","authors":"Johannes Ackermann, Volker Gabler, Takayuki Osa, Masashi Sugiyama","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Altman and Shwartz, 1998] Eitan Altman and Adam Shwartz. Constrained markov games: Nash equilibria. Annals of Dynamic Games, 5, 02 1998.","title":"Constrained markov games: Nash equilibria","authors":"Shwartz Altman, Eitan Altman, Adam Shwartz","year":"1998","venue":"Annals of Dynamic Games","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bertsekas, 2014] Dimitri P Bertsekas. Constrained opti- mization and Lagrange multiplier methods. Academic Press, 2014.","title":"Constrained optimization and Lagrange multiplier methods","authors":"Dimitri P Bertsekas","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Canese et al., 2021] Lorenzo Canese, Gian Carlo Cardar- illi, Luca Di Nunzio, Rocco Fazzolari, Daniele Giardino, Marco Re, and Sergio Spanò. Multi-agent reinforcement learning: A review of challenges and applications. Applied Sciences, 11(11), 2021.","title":"Multi-agent reinforcement learning: A review of challenges and applications","authors":"Lorenzo Canese, Gian, Carlo Cardarilli, Luca Di Nunzio, Rocco Fazzolari, Daniele Giardino, Marco Re, Sergio Spanò","year":"2021","venue":"Applied Sciences","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cao et al., 2020a] Di Cao, Weihao Hu, Junbo Zhao, Qi Huang, Zhe Chen, and Frede Blaabjerg. A multi-agent deep reinforcement learning based voltage regulation us- ing coordinated pv inverters. IEEE Transactions on Power Systems, 35(5):4120-4123, 2020.","title":"A multi-agent deep reinforcement learning based voltage regulation using coordinated pv inverters","authors":"Cao","year":"2020","venue":"IEEE Transactions on Power Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cao et al., 2020b] Di Cao, Junbo Zhao, Weihao Hu, Fei Ding, Qi Huang, and Zhe Chen. Distributed voltage reg- ulation of active distribution system based on enhanced multi-agent deep reinforcement learning, 2020.","title":"Distributed voltage regulation of active distribution system based on enhanced multi-agent deep reinforcement learning","authors":"Cao","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cao et al., 2021] Di Cao, Junbo Zhao, Weihao Hu, Fei Ding, Qi Huang, Zhe Chen, and Frede Blaabjerg. Data- driven multi-agent deep reinforcement learning for dis- tribution system decentralized voltage control with high penetration of pvs. IEEE Transactions on Smart Grid, 12(5):4137-4150, 2021.","title":"Datadriven multi-agent deep reinforcement learning for distribution system decentralized voltage control with high penetration of pvs","authors":"Di Cao, Junbo Zhao, Weihao Hu, Fei Ding, Qi Huang, Zhe Chen, Frede Blaabjerg","year":"2021","venue":"IEEE Transactions on Smart Grid","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2022] Xin Chen, Guannan Qu, Yujie Tang, Steven Low, and Na Li. Reinforcement learning for selec- tive key applications in power systems: Recent advances and future challenges. IEEE Transactions on Smart Grid, 13(4):2935-2958, 2022.","title":"Reinforcement learning for selective key applications in power systems: Recent advances and future challenges","authors":"Xin Chen, Guannan Qu, Yujie Tang, Steven Low, Na Li","year":"2022","venue":"IEEE Transactions on Smart Grid","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Farivar et al., 2013] Masoud Farivar, Lijun Chen, and Steven Low. Equilibrium and dynamics of local voltage control in distribution systems. In Proceedings of the 52nd IEEE Conference on Decision and Control (CDC), pages 4329-4334, 2013.","title":"Equilibrium and dynamics of local voltage control in distribution systems","authors":"Masoud Farivar, Lijun Chen, Steven Low","year":"2013","venue":"Proceedings of the 52nd IEEE Conference on Decision and Control (CDC)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Foerster et al., 2017] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. Proceedings of the AAAI Conference on Artificial Intelli- gence, 32, 05 2017.","title":"Counterfactual multi-agent policy gradients","authors":"Foerster","year":"2017","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gan et al., 2013] Lingwen Gan, Na Li, Ufuk Topcu, and Steven H. Low. Optimal power flow in tree networks. In Proceedings of the 52nd IEEE Conference on Decision and Control (CDC), pages 2313-2318, 2013.","title":"Low. Optimal power flow in tree networks","authors":"Gan","year":"2013","venue":"Proceedings of the 52nd IEEE Conference on Decision and Control (CDC)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gao et al., 2021] Yuanqi Gao, Wei Wang, and Nanpeng Yu. Consensus multi-agent reinforcement learning for volt-var control in power distribution networks. IEEE Transactions on Smart Grid, 12(4):3594-3604, 2021.","title":"Consensus multi-agent reinforcement learning for volt-var control in power distribution networks","authors":"Gao","year":"2021","venue":"IEEE Transactions on Smart Grid","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gu et al., 2022] Shangding Gu, Longyu Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. ArXiv, abs/2205.10330, 2022.","title":"A review of safe reinforcement learning: Methods, theory and applications","authors":"Gu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ha et al., 2020] Sehoon Ha, Peng Xu, Zhenyu Tan, Sergey Levine, and Jie Tan. Learning to walk in the real world with minimal human effort, 2020.","title":"Learning to walk in the real world with minimal human effort","authors":"Sehoon Ha, Peng Xu, Zhenyu Tan, Sergey Levine, Jie Tan","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hu et al., 2022] Daner Hu, Zhenhui Ye, Yuanqi Gao, Zuzhao Ye, Yonggang Peng, and Nanpeng Yu. Multi-agent deep reinforcement learning for voltage control with co- ordinated active and reactive power optimization. IEEE Transactions on Smart Grid, 13(6):4873-4886, 2022.","title":"Multi-agent deep reinforcement learning for voltage control with coordinated active and reactive power optimization","authors":"Daner Hu, Zhenhui Ye, Yuanqi Gao, Zuzhao Ye, Yonggang Peng, Nanpeng Yu","year":"2022","venue":"IEEE Transactions on Smart Grid","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jahangiri and Aliprantis, 2013] Pedram Jahangiri and Dionysios C. Aliprantis. Distributed volt/var control by pv inverters. IEEE Transactions on Power Systems, 28(3):3429-3439, 2013.","title":"Distributed volt/var control by pv inverters","authors":"Pedram Jahangiri, Dionysios C Aliprantis","year":"2013","venue":"IEEE Transactions on Power Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liang et al., 2018] Qingkai Liang, Fanyu Que, and Eytan Modiano. Accelerated primal-dual policy optimization for safe reinforcement learning, 2018.","title":"Accelerated primal-dual policy optimization for safe reinforcement learning","authors":"Liang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu and Wu, 2021] Haotian Liu and Wenchuan Wu. On- line multi-agent reinforcement learning for decentralized inverter-based volt-var control. IEEE Transactions on Smart Grid, 12(4):2980-2990, 2021.","title":"Online multi-agent reinforcement learning for decentralized inverter-based volt-var control","authors":"Wu Liu, Haotian Liu, Wenchuan Wu","year":"2021","venue":"IEEE Transactions on Smart Grid","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2023] Zemin Eitan Liu, Quan Zhou, Yanfei Li, Shijin Shuai, and Hongming Xu. Safe deep reinforcement learning-based constrained optimal control scheme for hev energy management. IEEE Transactions on Transporta- tion Electrification, 9(3):4278-4293, 2023.","title":"Safe deep reinforcement learning-based constrained optimal control scheme for hev energy management","authors":"Quan Zemin Eitan Liu, Yanfei Zhou, Shijin Li, Hongming Shuai, Xu","year":"2023","venue":"IEEE Transactions on Transportation Electrification","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lowe et al., 2017] Ryan Lowe, YI WU, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competitive en- vironments. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems, volume 30.","title":"Multiagent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Y I Wu, Aviv Tamar, Jean Harb, Pieter Openai, Igor Abbeel, Mordatch","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"Curran Associates, Inc., 2017.","title":null,"authors":"Curran Associates, Inc","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ma and Wu, 2023] Jinming Ma and Feng Wu. Learning to coordinate traffic signals with adaptive network partition. IEEE Transactions on Intelligent Transportation Systems (IEEE T-ITS), pages 1-12, 2023.","title":"Learning to coordinate traffic signals with adaptive network partition","authors":"Ma, Wu ; ] Jinming Ma, Feng Wu","year":"2023","venue":"IEEE Transactions on Intelligent Transportation Systems (IEEE T-ITS)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pu et al., 2021] Yuan Pu, Shaochen Wang, Rui Yang, Xin Yao, and Bin Li. Decomposed soft actor-critic method for cooperative multi-agent reinforcement learning, 2021.","title":"Decomposed soft actor-critic method for cooperative multi-agent reinforcement learning","authors":"Yuan Pu, Shaochen Wang, Rui Yang, Xin Yao, Bin Li","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ray et al., 2019] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep rein- forcement learning. arXiv preprint arXiv:1910.01708, 7(1):2, 2019.","title":"Benchmarking safe exploration in deep reinforcement learning","authors":"Alex Ray, Joshua Achiam, Dario Amodei","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Schardt and te Heesen, 2021] Johannes Schardt and Henrik te Heesen. Performance of roof-top pv systems in se- lected european countries from 2012 to 2019. Solar En- ergy, 217:235-244, 2021.","title":"Performance of roof-top pv systems in selected european countries from 2012 to 2019","authors":"Te Schardt, Johannes Heesen, Henrik Schardt, Te Heesen","year":"2021","venue":"Solar Energy","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Shi et al., 2022] Yuanyuan Shi, Guannan Qu, Steven Low, Anima Anandkumar, and Adam Wierman. Stability con- strained reinforcement learning for real-time voltage con- trol. In 2022 American Control Conference (ACC), pages 2715-2721, 2022.","title":"Stability constrained reinforcement learning for real-time voltage control","authors":"Yuanyuan Shi, Guannan Qu, Steven Low, Anima Anandkumar, Adam Wierman","year":"2022","venue":"2022 American Control Conference (ACC)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Shi et al., 2023] Yufeng Shi, Mingxiao Feng, Minrui Wang, Wengang Zhou, and Houqiang Li. Multi-agent reinforce- ment learning with safety layer for active voltage con- trol. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems (AAMAS), AAMAS '23, page 1533-1541, Richland, SC, 2023. Inter- national Foundation for Autonomous Agents and Multia- gent Systems.","title":"Multi-agent reinforcement learning with safety layer for active voltage control","authors":"Yufeng Shi, Mingxiao Feng, Minrui Wang, Wengang Zhou, Houqiang Li","year":"2023","venue":"Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems (AAMAS), AAMAS '23","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sun and Qiu, 2021] Xianzhuo Sun and Jing Qiu. Two-stage volt/var control in active distribution networks with multi- agent deep reinforcement learning method. IEEE Trans- actions on Smart Grid, 12(4):2903-2912, 2021.","title":"Xianzhuo Sun and Jing Qiu. Two-stage volt/var control in active distribution networks with multiagent deep reinforcement learning method","authors":"Qiu Sun","year":"2021","venue":"IEEE Transactions on Smart Grid","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Till et al., 2020] Jeremy Till, Shutang You, Yilu Liu, and Pengwei Du. Impact of high pv penetration on voltage sta- bility. In 2020 IEEE/PES Transmission and Distribution Conference and Exposition, pages 1-5, 2020.","title":"Impact of high pv penetration on voltage stability","authors":"Jeremy Till, Shutang You, Yilu Liu, Pengwei Du","year":"2020","venue":"IEEE/PES Transmission and Distribution Conference and Exposition","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang and Wu, 2020] Yixiang Wang and Feng Wu. Policy adaptive multi-agent deep deterministic policy gradient. In Proceedings of the 23rd International Conference on Principles and Practice of Multi-Agent Systems (PRIMA), Nagoya, Japan, Novermber 2020.","title":"Policy adaptive multi-agent deep deterministic policy gradient","authors":"Yixiang Wang, Feng Wu","year":"2020","venue":"Proceedings of the 23rd International Conference on Principles and Practice of Multi-Agent Systems (PRIMA)","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2019] Wei Wang, Nanpeng Yu, Jie Shi, and Yuanqi Gao. Volt-var control in power distribution systems with deep reinforcement learning. In 2019 IEEE Interna- tional Conference on Communications, Control, and Com- puting Technologies for Smart Grids (SmartGridComm), pages 1-7, 2019.","title":"Volt-var control in power distribution systems with deep reinforcement learning","authors":"Wei Wang, Nanpeng Yu, Jie Shi, Yuanqi Gao","year":"2019","venue":"IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wang et al., 2020] Shengyi Wang, Jiajun Duan, Di Shi, Chunlei Xu, Haifeng Li, Ruisheng Diao, and Zhiwei Wang. A data-driven multi-agent autonomous voltage con- trol framework using deep reinforcement learning. IEEE Transactions on Power Systems, 35(6):4644-4654, 2020.","title":"A data-driven multi-agent autonomous voltage control framework using deep reinforcement learning","authors":"Shengyi Wang, Jiajun Duan, Di Shi, Chunlei Xu, Haifeng Li, Ruisheng Diao, Zhiwei Wang","year":"2020","venue":"IEEE Transactions on Power Systems","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wang et al., 2021] Jianhong Wang, Wangkun Xu, Yunjie Gu, Wenbin Song, and Tim C Green. Multi-agent re- inforcement learning for active voltage control on power distribution networks. In Advances in Neural Information Processing Systems, volume 34, pages 3271-3284.","title":"Multi-agent reinforcement learning for active voltage control on power distribution networks","authors":"Jianhong Wang, Wangkun Xu, Yunjie Gu, Wenbin Song, Tim C Green","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"Curran Associates, Inc., 2021.","title":null,"authors":"Curran Associates, Inc","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wang et al., 2022] Minrui Wang, Mingxiao Feng, Wengang Zhou, and Houqiang Li. Stabilizing voltage in power distribution networks via multi-agent reinforcement learn- ing with transformer. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), KDD '22, page 1899-1909, New York, NY, USA, 2022. Association for Computing Machinery.","title":"Stabilizing voltage in power distribution networks via multi-agent reinforcement learning with transformer","authors":"Wang","year":"2022","venue":"Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), KDD '22","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zheng et al., 2016] Weiye Zheng, Wenchuan Wu, Boming Zhang, Hongbin Sun, and Yibing Liu. A fully distributed reactive power optimization and control method for active distribution networks. IEEE Transactions on Smart Grid, 7(2):1021-1033, 2016.","title":"A fully distributed reactive power optimization and control method for active distribution networks","authors":"Weiye Zheng, Wenchuan Wu, Boming Zhang, Hongbin Sun, Yibing Liu","year":"2016","venue":"IEEE Transactions on Smart Grid","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}