{"type":"ScholarlyArticle","title":"Large-Scale Home Energy Management Using Entropy-Based Collective Multiagent Deep Reinforcement Learning Framework","authors":[{"name":"Yaodong Yang","slug":"yaodong-yang","given":"Yaodong","family":"Yang","affiliation":"College of Intelligence and Computing , Tianjin University , Tianjin , China"},{"name":"Jianye Hao","slug":"jianye-hao","given":"Jianye","family":"Hao","affiliation":"College of Intelligence and Computing , Tianjin University , Tianjin , China"},{"name":"Yan Zheng","slug":"yan-zheng","given":"Yan","family":"Zheng","affiliation":"College of Intelligence and Computing , Tianjin University , Tianjin , China"},{"name":"Chao Yu","slug":"chao-yu","given":"Chao","family":"Yu","affiliation":"School of Data and Computer Science , Sun Yat-Sen University , Guangzhou , China"}],"abstract":"Smart grids are contributing to the demand-side management by integrating electronic equipment, distributed energy generation and storage and advanced meters and controllers. With the increasing adoption of electric vehicles and distributed energy generation and storage systems, residential energy management is drawing more and more attention, which is regarded as being critical to demand-supply balancing and peak load reduction. In this paper, we focus on a microgrid scenario in which modern homes interact together under a large-scale setting to better optimize their electricity cost. We first make households form a group with an economic stimulus. Then we formulate the energy expense optimization problem of the household community as a multi-agent coordination problem and present an Entropy-Based Collective Multiagent Deep Reinforcement Learning (EB-C-MADRL) framework to address it. Experiments with various real-world data demonstrate that EB-C-MADRL can reduce both the long-term group power consumption cost and daily peak demand effectively compared with existing approaches.","doi":"10.24963/ijcai.2019/89","url":"https://ijcai.org/proceedings/2019/89","pdf":"https://ijcai.org/proceedings/2019/0089.pdf","markdown":"https://ijcai.org/proceedings/2019/89.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/89","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Agent-based and Multi-agent Systems","track":"Main track","number":89,"pages":{"first":630,"last":636},"keywords":["Agent-based and Multi-agent Systems: Agent-Based Simulation and Emergence","Agent-based and Multi-agent Systems: Coordination and Cooperation","Machine Learning Applications: Applications of Reinforcement Learning","Agent-based and Multi-agent Systems: Multi-agent Learning"],"videos":[],"references":[{"position":1,"raw":"[Atzeni et al., 2013] Italo Atzeni, Luis Garcia Ordóñez, Gesualdo Scutari, Daniel Pérez Palomar, and Javier Rodríguez Fonollosa. Demand-side management via distributed energy generation and storage optimiza- tion. IEEE Transactions on Smart Grid, 4:866-876, 2013.","title":"Demand-side management via distributed energy generation and storage optimization","authors":"Atzeni","year":"2013","venue":"IEEE Transactions on Smart Grid","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Berlink and Costa, 2015] Heider Berlink and Anna He- lena Reali Costa. Batch reinforcement learning for smart home energy management. In Proceedings of the 24th International Conference on Artificial Intelligence, pages 2561-2567, 2015.","title":"Batch reinforcement learning for smart home energy management","authors":"Costa Berlink","year":"2015","venue":"Proceedings of the 24th International Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2015/363"},{"position":3,"raw":"[Cailliere et al., 2016] Romain Franc ¸ois Cailliere, Samir Ak- nine, and Antoine Nongaillard. Multi-agent mechanism for efficient cooperative use of energy. In Proceedings of the 15th International Conference on Autonomous Agents & Multiagent Systems, pages 1365-1366, 2016.","title":"Multi-agent mechanism for efficient cooperative use of energy","authors":"Romain Franc ¸ois Cailliere, Samir Aknine, Antoine Nongaillard","year":"2016","venue":"Proceedings of the 15th International Conference on Autonomous Agents & Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dusparic et al., 2013] Ivana Dusparic, Colin Harris, Andrei Marinescu, Vinny Cahill, and Siobhan Clarke. Multi-agent residential demand response based on load forecasting. In Proceedings of the 1st IEEE Conference on Technologies for Sustainability (SusTech), pages 90-96, 2013.","title":"Multi-agent residential demand response based on load forecasting","authors":"Ivana Dusparic, Colin Harris, Andrei Marinescu, Vinny Cahill, Siobhan Clarke","year":"2013","venue":"Proceedings of the 1st IEEE Conference on Technologies for Sustainability (SusTech)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Lin et al., 2012] Zhenhong Lin, Jing Dong, Changzheng Liu, and David Greene. Estimation of energy use by plug- in hybrid electric vehicles: Validating gamma distribution for representing random daily driving distance. Trans- portation Research Record: Journal of the Transportation Research Board, 2287(1):37-43, 2012.","title":"Estimation of energy use by plugin hybrid electric vehicles: Validating gamma distribution for representing random daily driving distance","authors":"Lin","year":"2012","venue":"Transportation Research Record: Journal of the Transportation Research Board","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Littman, 1994] Michael L. Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine Learning Proceedings, pages 157-163. Elsevier, 1994.","title":"Markov games as a framework for multi-agent reinforcement learning","authors":"L Michael, Littman","year":"1994","venue":"Machine Learning Proceedings","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Logenthiran et al., 2016] Thillainathan Logenthiran, W. Li, and W. L. Woo. Intelligent multi-agent system for smart home energy management. In Proceedings of the IEEE In- novative Smart Grid Technologies -Asia, pages 1-6, 2016.","title":"Intelligent multi-agent system for smart home energy management","authors":"Logenthiran","year":"2016","venue":"Proceedings of the IEEE Innovative Smart Grid Technologies -Asia","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lowe et al., 2017] Ryan Lowe, YI WU, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competitive envi- ronments. In Proceedings of the 31th Advances in Neural Information Processing Systems, pages 6379-6390, 2017.","title":"Multiagent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Y I Wu, Aviv Tamar, Jean Harb, Pieter Openai, Igor Abbeel, Mordatch","year":"2017","venue":"Proceedings of the 31th Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Mohsenian-Rad et al., 2010] Amir-Hamed Mohsenian- Rad, Vincent W. S. Wong, Juri Jatskevich, Robert Schober, and Alberto Leon-Garcia. Autonomous demand- side management based on game-theoretic energy consumption scheduling for the future smart grid. IEEE Transactions on Smart Grid, 1(3):320-331, 2010.","title":"Autonomous demandside management based on game-theoretic energy consumption scheduling for the future smart grid","authors":"Amir-Hamed Mohsenian-Rad, W S Vincent, Juri Wong, Robert Jatskevich, Alberto Schober, Leon-Garcia","year":"2010","venue":"IEEE Transactions on Smart Grid","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Muratori, 2018] Matteo Muratori. Impact of uncoordinated plug-in electric vehicle charging on residential power de- mand. Nature Energy, 3(3):193-201, 2018.","title":"Impact of uncoordinated plug-in electric vehicle charging on residential power demand","authors":"Matteo Muratori","year":"2018","venue":"Nature Energy","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[of Queensland, 2018] University of Queensland. Uq solar photovoltaic data. http://solar.uq.edu.au/user/reportPower. php, 2018.","title":"Uq solar photovoltaic data","authors":null,"year":"2018","venue":null,"doi":null,"url":"http://solar.uq.edu.au/user/reportPower.php,2018","internal_url":null},{"position":12,"raw":"[O'Neill et al., 2010] Daniel O'Neill, Marco Levorato, An- drea Goldsmith, and Urbashi Mitra. Residential demand response using reinforcement learning. In Proceedings of the 1st IEEE International Conference on Smart Grid Communications, pages 409-414, 2010.","title":"Andrea Goldsmith, and Urbashi Mitra. Residential demand response using reinforcement learning","authors":"O'neill","year":"2010","venue":"Proceedings of the 1st IEEE International Conference on Smart Grid Communications","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Palensky and Dietrich, 2011] Peter Palensky and Dietmar Dietrich. Demand side management: Demand response, intelligent energy systems, and smart loads. IEEE Trans- actions on Industrial Informatics, 7(3):381-388, 2011.","title":"Demand side management: Demand response, intelligent energy systems, and smart loads","authors":"Dietrich ; Palensky, Peter Palensky, Dietmar Dietrich","year":"2011","venue":"IEEE Transactions on Industrial Informatics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Plötz et al., 2017] Patrick Plötz, Niklas Jakobsson, and Frances Sprei. On the distribution of individual daily driv- ing distances. Transportation Research Part B: Method- ological, 101:213-227, 2017.","title":"On the distribution of individual daily driving distances","authors":"Patrick Plötz, Niklas Jakobsson, Frances Sprei","year":"2017","venue":"Transportation Research Part B: Methodological","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Robu et al., 2013] Valentin Robu, Enrico Gerding, Sebas- tian Stein, David C. Parkes, Alex Rogers, and Nicholas R. Jennings. An online mechanism for multi-unit demand and its application to plug-in hybrid electric vehicle charging. Journal of Artificial Intelligence Research, 48:175-230, 2013.","title":"An online mechanism for multi-unit demand and its application to plug-in hybrid electric vehicle charging","authors":"Valentin Robu, Enrico Gerding, Sebastian Stein, David C Parkes, Alex Rogers, Nicholas R Jennings","year":"2013","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shweta Jain et al., 2014] Shweta Jain, Balakrishnan Narayanaswamy, and Y. Narahari. A multiarmed bandit incentive mechanism for crowdsourcing demand re- sponse in smart grids. In Proceedings of the 28th AAAI Conference on Artificial Intelligence, pages 721-727, 2014.","title":"A multiarmed bandit incentive mechanism for crowdsourcing demand response in smart grids","authors":"Shweta Jain, Balakrishnan Narayanaswamy, Y Narahari","year":"2014","venue":"Proceedings of the 28th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Strbac, 2008] Goran Strbac. Demand side management: Benefits and challenges. Energy Policy, 36(12):4419- 4426, 2008.","title":"Goran Strbac. Demand side management: Benefits and challenges","authors":"Strbac","year":"2008","venue":"Energy Policy","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Verma et al., 2018] Tanvi Verma, Pradeep Varakantham, and Hoong Chuin Lau. Entropy controlled non- stationarity for improving performance of independent learners in anonymous MARL settings. arXiv preprint, abs/1803.09928, 2018.","title":"Entropy controlled nonstationarity for improving performance of independent learners in anonymous MARL settings","authors":"Tanvi Verma, Pradeep Varakantham, Hoong Chuin Lau","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wu et al., 2018] Di Wu, Rabusseau Guillaume, Franc ¸ois lavet Vincent, Precup Doina, and Boulet Benoit. Optimiz- ing home energy management and electric vehicle charg- ing with reinforcement learning. In Proceedings of the 16th Adaptive Learning Agents, 2018.","title":"Optimizing home energy management and electric vehicle charging with reinforcement learning","authors":"Di Wu, Rabusseau Guillaume, Franc Vincent, Precup Doina, Boulet Benoit","year":"2018","venue":"Proceedings of the 16th Adaptive Learning Agents","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yang et al., 2018] Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi- agent reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, vol- ume 80, pages 5567-5576. PMLR, 2018.","title":"Mean field multiagent reinforcement learning","authors":"Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, Jun Wang","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yuan-Yih Hsu and Chung-Ching Su, 1991] Yuan-Yih Hsu and Chung-Ching Su. Dispatch of direct load control us- ing dynamic programming. IEEE Transactions on Power Systems, 6(3):1056-1061, 1991.","title":"Dispatch of direct load control using dynamic programming","authors":"Yuan-Yih Hsu, Chung-Ching Su, ; ] Yuan-Yih Hsu, Chung-Ching Su","year":"1991","venue":"IEEE Transactions on Power Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2015] Baosen Zhang, Ramesh Johari, and Ram Rajagopal. Competition and coalition formation of renewable power producers. IEEE Transactions on Power Systems, 30(3):1624-1632, 2015.","title":"Competition and coalition formation of renewable power producers","authors":"Zhang","year":"2015","venue":"IEEE Transactions on Power Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}