{"type":"ScholarlyArticle","title":"Model-Free Real-Time Autonomous Energy Management for a Residential Multi-Carrier Energy System: A Deep Reinforcement Learning Approach","authors":[{"name":"Yujian Ye","slug":"yujian-ye","given":"Yujian","family":"Ye","affiliation":"Department of Electrical and Electronic Engineering , Imperial College London , London , U.K.; Fetch.ai , Cambridge , U.K."},{"name":"Dawei Qiu","slug":"dawei-qiu","given":"Dawei","family":"Qiu","affiliation":"Department of Electrical and Electronic Engineering , Imperial College London , London , U.K."},{"name":"Jonathan Ward","slug":"jonathan-ward","given":"Jonathan","family":"Ward","affiliation":"Fetch.ai , Cambridge , U.K."},{"name":"Marcin Abram","slug":"marcin-abram","given":"Marcin","family":"Abram","affiliation":"Fetch.ai , Cambridge , U.K."}],"abstract":"The problem of real-time autonomous energy management is an application area that is receiving unprecedented attention from consumers, governments, academia, and industry. This paper showcases the first application of deep reinforcement learning (DRL) to real-time autonomous energy management for a multi-carrier energy system. The proposed approach is tailored to align with the nature of the energy management problem by posing it in multi-dimensional continuous state and action spaces, in order to coordinate power flows between different energy devices, and to adequately capture the synergistic effect of couplings between different energy carriers. This fundamental contribution is a significant step forward from earlier approaches that only sought to control the power output of a single device and neglected the demand-supply coupling of different energy carriers. Case studies on a real-world scenario demonstrate that the proposed method significantly outperforms existing DRL methods as well as model-based control approaches in achieving the lowest energy cost and yielding a representation of energy management policies that adapt to system uncertainties.","doi":"10.24963/ijcai.2020/48","url":"https://ijcai.org/proceedings/2020/48","pdf":"https://ijcai.org/proceedings/2020/0048.pdf","markdown":"https://ijcai.org/proceedings/2020/48.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/48","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Agent-based and Multi-agent Systems","track":"Main track","number":48,"pages":{"first":339,"last":346},"keywords":["Agent-based and Multi-agent Systems: Agent-Based Simulation and Emergence","Multidisciplinary Topics and Applications: Real-Time Systems","Machine Learning Applications: Applications of Reinforcement Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25149/file_25149.m3u8","length_sec":299,"provider":"hls"}],"references":[{"position":1,"raw":"[Basit et al., 2017] Abdul Basit, Guftaar A. S. Sidhu, Anzar Mah- mood, and Feifei Gao. Efficient and Autonomous Energy Man- agement Techniques for the Future Smart Homes. IEEE Trans. on Smart Grid, 8(2):917-926, 2017.","title":"Efficient and Autonomous Energy Management Techniques for the Future Smart Homes","authors":"Abdul Basit, A S Guftaar, Anzar Sidhu, Feifei Mahmood, Gao","year":"2017","venue":"IEEE Trans. on Smart Grid","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Berlink et al., 2015] Heider Berlink, Nelson Kagan, and Anna H. R. Costa. Intelligent Decision-Making for Smart Home En- ergy Management. J. Intell. Robot. Syst., 80(1):331-354, 2015.","title":"Intelligent Decision-Making for Smart Home Energy Management","authors":"Heider Berlink, Nelson Kagan, Anna H R Costa","year":"2015","venue":"J. Intell. Robot. Syst","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bozchalui et al., 2012] Mohammad C. Bozchalui, Syed A. Hashmi, Hussin Hassen, Claudio A. Canizares, and Kankar Bhattacharya. Optimal Operation of Residential Energy Hubs in Smart Grids. IEEE Trans. on Smart Grid, 3(4):1755-1766, 2012.","title":"Optimal Operation of Residential Energy Hubs in Smart Grids","authors":"Mohammad C Bozchalui, A Syed, Hussin Hashmi, Claudio A Hassen, Kankar Canizares, Bhattacharya","year":"2012","venue":"IEEE Trans. on Smart Grid","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen and Su, 2018] Tao Chen and Wencong Su. Local Energy Trading Behavior Modeling with Deep Reinforcement Learning. IEEE Access, 6:62806-62814, 2018.","title":"Local Energy Trading Behavior Modeling with Deep Reinforcement Learning","authors":"Chen, Su ;, Tao Chen, Wencong Su","year":"2018","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Conejo et al., 2010] Antonio J Conejo, Miguel Carrión, Juan M Morales, et al. Decision Making Under Uncertainty in Electricity Markets, volume 1. Springer, 2010.","title":"Decision Making Under Uncertainty in Electricity Markets","authors":"Miguel Antonio J Conejo, Juan M Carrión, Morales","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[E.ON, 2019] E.ON. Fix Online v27 Gas Tariff, 2019.","title":"Fix Online v27 Gas Tariff","authors":"E On","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hydro, 2019] London Hydro. Smart Meters and Time-of-Use Rates, 2019.","title":"London Hydro. Smart Meters and Time-of-Use Rates","authors":null,"year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kim and Lim, 2018] Sunyong Kim and Hyuk Lim. Reinforcement Learning Based Energy Management Algorithm for Smart En- ergy Buildings. Energies, 11(8):1-19, 2018.","title":"Reinforcement Learning Based Energy Management Algorithm for Smart Energy Buildings","authors":"Lim Kim, Sunyong Kim, Hyuk Lim","year":"2018","venue":"Energies","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Konda and Tsitsiklis, 2000] Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms. In Proc. NIPS, pages 1008-1014, CO, USA, 2000.","title":"Actor-critic algorithms","authors":"R Vijay, John N Konda, Tsitsiklis","year":"2000","venue":"Proc. NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kou et al., 2015] Peng Kou, Deliang Liang, Lin Gao, and Feng Gao. Stochastic coordination of plug-in electric vehicles and wind turbines in microgrid: A model predictive control approach. IEEE Trans. on Smart Grid, 7(3):1537-1551, 2015.","title":"Stochastic coordination of plug-in electric vehicles and wind turbines in microgrid: A model predictive control approach","authors":"Peng Kou, Deliang Liang, Lin Gao, Feng Gao","year":"2015","venue":"IEEE Trans. on Smart Grid","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2016] Yu-Shuai Li, Hua-Guang Zhang, Bo-Nan Huang, and Fei Teng. Distributed Optimal Economic Dispatch Based on Multi-Agent System Framework in Combined Heat and Power Systems. Appl. Sci., 6(10):308, 2016.","title":"Distributed Optimal Economic Dispatch Based on Multi-Agent System Framework in Combined Heat and Power Systems","authors":"Yu-Shuai Li, Hua-Guang Zhang, Bo-Nan Huang, Fei Teng","year":"2016","venue":"Appl. Sci","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liang et al., 2013] Yong Liang, Long He, Xinyu Cao, and Zuo- Jun Shen. Stochastic Control for Smart Grid Users with Flexible Demand. IEEE Trans. on Smart Grid, 4(4):2296-2308, 2013.","title":"Stochastic Control for Smart Grid Users with Flexible Demand","authors":"Yong Liang, Long He, Xinyu Cao, Zuo-Jun Shen","year":"2013","venue":"IEEE Trans. on Smart Grid","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lillicrap and et al., 2016] Timothy P. Lillicrap and et al. Continu- ous Control with Deep Reinforcement Learning. In Proc. ICLR, pages 1-14, San Juan, US, 2016.","title":"Continuous Control with Deep Reinforcement Learning","authors":"Timothy P Lillicrap","year":"2016","venue":"Proc. ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mancarella, 2014] Pierluigi Mancarella. MES (Multi-Energy Sys- tems): An Overview of Concepts and Evaluation Models. En- ergy, 65:1-17, 2014.","title":"MES (Multi-Energy Systems): An Overview of Concepts and Evaluation Models","authors":"Pierluigi Mancarella","year":"2014","venue":"Energy","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mason and Grijalva, 2019] Karl Mason and Santiago Grijalva. A Review of Reinforcement Learning for Autonomous Building Energy Management. Comput. Electr. Eng., 78:300-312, 2019.","title":"A Review of Reinforcement Learning for Autonomous Building Energy Management","authors":"Grijalva ; Mason, Karl Mason, Santiago Grijalva","year":"2019","venue":"Comput. Electr. Eng","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mnih and et al., 2015] Volodymyr Mnih and et al. Human- Level Control Through Deep Reinforcement Learning. Nature, 518(7540):529-533, 2015.","title":"Volodymyr Mnih and et al. Human-Level Control Through Deep Reinforcement Learning","authors":"Mnih","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mocanu et al., 2019] Elena Mocanu, Decebal C. Mocanu, Phuong H. Nguyen, Antonio Liotta, Michael E. Webber, Madeleine Gibescu, and Johannes G. Slootweg. On-Line Build- ing Energy Optimization Using Deep Reinforcement Learning. IEEE Trans. on Smart Grid, 10(4):3698-3708, 2019.","title":"On-Line Building Energy Optimization Using Deep Reinforcement Learning","authors":"Elena Mocanu, C Decebal, Phuong H Mocanu, Antonio Nguyen, Michael E Liotta, Madeleine Webber, Johannes G Gibescu, Slootweg","year":"2019","venue":"IEEE Trans. on Smart Grid","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Moghaddam et al., 2016] Iman G. Moghaddam, Mohsen Saniei, and Elaheh Mashhour. A Comprehensive Model for Self- Scheduling an Energy Hub to Supply Cooling, Heating and Elec- trical Demands of a Building. Energy, 94:157-170, 2016.","title":"A Comprehensive Model for Self-Scheduling an Energy Hub to Supply Cooling, Heating and Electrical Demands of a Building","authors":"G Iman, Mohsen Moghaddam, Elaheh Saniei, Mashhour","year":"2016","venue":"Energy","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ofgem, 2019] Ofgem. Feed-In Tariff (FIT) Rates, 2019.","title":"Ofgem. Feed-In Tariff (FIT) Rates","authors":null,"year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pazouki et al., 2014] Samaneh Pazouki, Mahmoud-Reza Haghi- fam, and Albert Moser. Uncertainty Modeling in Optimal Oper- ation of Energy Hub in Presence of Wind, Storage and Demand Response. Int. J. Elec. Power., 61:335-345, 2014.","title":"Samaneh Pazouki, Mahmoud-Reza Haghifam, and Albert Moser. Uncertainty Modeling in Optimal Operation of Energy Hub in Presence of Wind, Storage and Demand Response","authors":"Pazouki","year":"2014","venue":"Int. J. Elec. Power","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pudjianto et al., 2013] Danny Pudjianto, Marko Aunedi, Predrag Djapic, and Goran Strbac. Whole-Systems Assessment of the Value of Energy Storage in Low-Carbon Electricity Systems. IEEE Trans. on Smart Grid, 5(2):1098-1109, 2013.","title":"Whole-Systems Assessment of the Value of Energy Storage in Low-Carbon Electricity Systems","authors":"Danny Pudjianto, Marko Aunedi, Predrag Djapic, Goran Strbac","year":"2013","venue":"IEEE Trans. on Smart Grid","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rastegar and Fotuhi-Firuzabad, 2015] Mohammad Rastegar and Mahmud Fotuhi-Firuzabad. Load Management in a Residential Energy Hub with Renewable Distributed Energy Resources. En- ergy and Buildings, 107:234-242, 2015.","title":"Mohammad Rastegar and Mahmud Fotuhi-Firuzabad. Load Management in a Residential Energy Hub with Renewable Distributed Energy Resources","authors":"Fotuhi-Firuzabad Rastegar","year":"2015","venue":"Energy and Buildings","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schaul et al., 2016] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized Experience Replay. In Pro. ICLR, pages 1-21, San Juan, US, 2016.","title":"Prioritized Experience Replay","authors":"Tom Schaul, John Quan, Ioannis Antonoglou, David Silver","year":"2016","venue":"Pro. ICLR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sedighizadeh et al., 2018] Mostafa Sedighizadeh, Masoud Es- maili, and Nahid Mohammadkhani. Stochastic Multi-Objective Energy Management in Residential Microgrids with Combined Cooling, Heating, and Power Units Considering Battery Energy Storage Systems and Plug-in Hybrid Electric Vehicles. J. Clean. Prod., 195:301-317, 2018.","title":"Stochastic Multi-Objective Energy Management in Residential Microgrids with Combined Cooling, Heating, and Power Units Considering Battery Energy Storage Systems and Plug-in Hybrid Electric Vehicles","authors":"Mostafa Sedighizadeh, Masoud Esmaili, Nahid Mohammadkhani","year":"2018","venue":"J. Clean. Prod","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Silver et al., 2014] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deter- ministic Policy Gradient Algorithms. In Proc. ICML, pages 1-9, Beijing, China, 2014.","title":"Deterministic Policy Gradient Algorithms","authors":"David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, Martin Riedmiller","year":"2014","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vahid-Pakdel et al., 2017] MJ Vahid-Pakdel, Sayyad Nojavan, B Mohammadi-Ivatloo, and Kazem Zare. Stochastic Optimiza- tion of Energy Hub Operation with Consideration of Thermal En- ergy Market and Demand Response. Energy Convers. Manag., 145:117-128, 2017.","title":"Stochastic Optimization of Energy Hub Operation with Consideration of Thermal Energy Market and Demand Response","authors":"Vahid-Pakdel","year":"2017","venue":"Energy Convers. Manag","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wan et al., 2019] Zhiqiang Wan, Hepeng Li, Haibo He, and Danil Prokhorov. Model-Free Real-Time EV Charging Scheduling Based on Deep Reinforcement Learning. IEEE Trans. on Smart Grid, 10(5):5246-5257, 2019.","title":"Model-Free Real-Time EV Charging Scheduling Based on Deep Reinforcement Learning","authors":"Zhiqiang Wan, Hepeng Li, Haibo He, Danil Prokhorov","year":"2019","venue":"IEEE Trans. on Smart Grid","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wei et al., 2017] Tianshu Wei, Yanzhi Wang, and Qi Zhu. Deep Reinforcement Learning for Building HVAC Control. In Proc. DAC, pages 1-7, Austin, USA, 2017.","title":"Deep Reinforcement Learning for Building HVAC Control","authors":"Tianshu Wei, Yanzhi Wang, Qi Zhu","year":"2017","venue":"Proc. DAC","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wen et al., 2015] Zheng Wen, Daniel O'Neill, and Hamid Maei. Optimal Demand Response Using Device-Based Reinforcement Learning. IEEE Trans. on Smart Grid, 6(5):2312-2324, 2015.","title":"Optimal Demand Response Using Device-Based Reinforcement Learning","authors":"Zheng Wen, O' Daniel, Hamid Neill, Maei","year":"2015","venue":"IEEE Trans. on Smart Grid","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wu et al., 2018] Di Wu, Guillaume Rabusseau, Vincent Francois- lavet, Doina Precup, and Benoit Boulet. Optimizing Home En- ergy Management and Electric Vehicle Charging with Reinforce- ment Learning. In Proc. ALA, pages 1-8, Stockholm, Sweden, 2018.","title":"Optimizing Home Energy Management and Electric Vehicle Charging with Reinforcement Learning","authors":"Di Wu, Guillaume Rabusseau, Vincent Francoislavet, Doina Precup, Benoit Boulet","year":"2018","venue":"Proc. ALA","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Ye et al., 2014] Yujian Ye, Dimitrios Papadaskalopoulos, and Goran Strbac. Factoring flexible demand non-convexities in elec- tricity markets. IEEE Trans. on Power Syst., 30(4):2090-2099, 2014.","title":"Yujian Ye, Dimitrios Papadaskalopoulos, and Goran Strbac. Factoring flexible demand non-convexities in electricity markets","authors":"Ye","year":"2014","venue":"IEEE Trans. on Power Syst","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Ye et al., 2019] Yujian Ye, Dawei Qiu, Jing Li, and Goran Str- bac. Multi-period and multi-spatial equilibrium analysis in im- perfect electricity markets: A novel multi-agent deep reinforce- ment learning approach. IEEE Access, 7:130515-130529, 2019.","title":"Multi-period and multi-spatial equilibrium analysis in imperfect electricity markets: A novel multi-agent deep reinforcement learning approach","authors":"Ye","year":"2019","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Ye et al., 2020] Yujian Ye, Dawei Qiu, Mingyang Sun, Dimitrios Papadaskalopoulos, and Goran Strbac. Deep reinforcement learning for strategic bidding in electricity markets. IEEE Trans. on Smart Grid, 11(2):1343-1355, 2020.","title":"Deep reinforcement learning for strategic bidding in electricity markets","authors":"Ye","year":"2020","venue":"IEEE Trans. on Smart Grid","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2017] Huaguang Zhang, Yushuai Li, David Wen- zhong Gao, and Jianguo Zhou. Distributed optimal energy management for energy internet. IEEE Trans. Ind. Informat., 13(6):3081-3097, 2017.","title":"Distributed optimal energy management for energy internet","authors":"Huaguang Zhang, Yushuai Li, David Wenzhong Gao, Jianguo Zhou","year":"2017","venue":"IEEE Trans. Ind. Informat","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}