Large-Scale Home Energy Management Using Entropy-Based Collective Multiagent Deep Reinforcement Learning Framework

1 Introduction

Meeting the growing energy demand due to the presence of more volatile types of loads raises a major challenge for the power grid [Robu et al., 2013]. In order to satisfy demand that varies sharply, companies usually have to install additional generation capacity to meet the peak demand with a heavy price. The requirement of massive investment into peaking power generation, in turn, results in higher costs for end-users. At the same time, distributed renewable generation such as wind and solar energy is gaining prominence and is perceived as vital to achieving cost and carbon reduction [Shweta Jain et al., 2014]. However, renewable generation has the unreliable nature that it is quite intermittent and sensitive to weather changes. As a result, the increase in electricity supply from renewable sources leads to larger fluctuations which makes the power grid hard to maintain the demandsupply balance. With the increasing number of active electricity customers and the advent of decentralized power generation technologies, the peak load and supply-demand imbalance have received more and more attention by energy generation and distribution companies [Zhang et al., 2015].

∗Corresponding author: Jianye Hao.

To handle the above problems, the demand-side management (DSM) [Strbac, 2008] has been proposed, which aims to adjust the consumer's energy activities such as shifting customers' consumption from peak hours to off-peak hours. From the perspective of energy providers, lots of DSM technologies have been developed to reschedule consumption such as Time-of-Use (TOU) tariffs [Shweta Jain et al., 2014]. On the other hand, there are also many DSM techniques focusing on the home energy management, such as dynamic programming [Yuan-Yih Hsu and Chung-Ching Su, 1991], game theory [Mohsenian-Rad et al., 2010] and reinforcement learning (RL) [O'Neill et al., 2010]. However, these works only consider different subsets of the home power systems instead of the complete home energy system. Another disadvantage of these solutions is the rigid schedule for end users' appliances usage. Recently, smart homes combined with the distributed energy generation (DG) and distributed energy storage (DS) show the great possibility for the revolution of the power grid [Palensky and Dietrich, 2011; Logenthiran et al., 2016]. It provides us with opportunities of unfreezing the rigid schedule for users with the emerging DG and DS. RL based DSM techniques for the smart home was first investigated in [Berlink and Costa, 2015] and then extended in [Wu et al., 2018] with electric vehicles (EV). Compared with traditional methods, RL could learn flexible energy management strategies and well suits the promising smart home scenario without manual adjustments.

However, these smart home DSMworks focus on optimizing the energy activities for a single household. They do not consider the aggregate effect of all customers shifting the demand to low price/low demand periods, which would result in a sudden increase in demand and overloads on the transformer [Dusparic et al., 2013]. On the contrary, one exception is that Dusparic et al. [Dusparic et al., 2013] proposed a multiagent approach to manage the energy demand of a small house group. Their strategy, calledW-learning, consists of integrating independent Q-learning agents that one agent schedules on the appliance usage for one home. However, the role of households considered in their approach is the traditional power consumer and the rigid appliance usage schedules violate user habits and bring inconvenience. To the best of our knowledge, no work has investigated DSM for the concernful large-scale smart-home energy management problem.

To this end, in this paper, we research on the user-friendly DSM techniques for a smart home community. We first entice the community with an incentive mechanism to form a trading group and consider this scenario as partially observable Markov Games. Then we propose an entropy-based collective multiagent reinforcement learning (MARL) framework to address the large-scale energy cost optimization problem. Our framework optimizes the energy consumption by abstracting the market dynamics to mitigate the non-stationary problem and avoid the action space explosion. Besides, it utilizes the concept of entropy to reduce peak load. Experiments simulated from real-world data exhibit excellent performance of our framework compared with previous approaches.

The remainder of this paper is organized as follows. We introduce the Markov Games, the smart home and deep reinforcement learning algorithms in Section 2. Then we describe the microgrid electricity market in Section 3. And in Section 4 we explain our EB-C-MADRL framework in details. Finally, we demonstrate the effectiveness of our framework in the microgrid simulated from real-world data in Section 5. Conclusions and the future work are provided in Section 6.

2 Background

2.1 Markov Games

In this work, we consider a multiagent extension of the Markov decision process (MDP) called partially observable Markov games [Littman, 1994]. A Markov game for n agents identified by i ∈ I ≡ {1,2, ...,n} is defined by a set of states S describing the global state, a set of actions A1, ...,An and a set of observations O1, ...,On for each agent. Each agent i chooses actions according to its policy pi : Oi ×Ai → [0,1], which produces the next state according to the transition function T : S×A1×...×An →S. After the transition, each agent i obtains its reward ri(s,a1, ...,an) : S×A1×...×An×I →R, and receives a private observation correlated with the state oi : S→Oi. The initial states are determined by distribution r : S→[0,1]. Each agent i aims to maximize its own total expected return Ji = Ea i∼p1,...,a N ∼pN,s∼T � T t=0 g t ri,t (s, a1, ..., an) where g is a discount factor and T is the time horizon.

2.2 Smart Home

We follow the smart home concept in [Wu et al., 2018] as shown in Figure 1. The main components can be divided into:

• base load electricity consumption, consumed from conventional household appliances;
• EV charging power consumption;
• micro-generation, such as Photovoltaics (PV);
• energy storage, the home battery for storing power;
• and home energy management system (EMS) for DSM.

Advanced meters are installed at home, enabling bidirectional communication. A bi-directional flow guarantees smart homes acquire or insert energy into the power grid.

2.3 Deep Reinforcement Learning

Here we introduce Deep Q-Networks (DQN) and Advantage Actor-Critic (A2C). For maximizing the accumulated expected return J, Q-learning uses an action-value function for policy p as Q p (s,a) = E[J|st = s,at = a] and updates its Qvalues based on each experience given by (s, a, s′ , r):

where a is the learning rate and g is the discount factor. The e-greedy is used to trade off exploration-exploitation.

Deep Q-Networks (DQN)

DQN represents the action-value function with a deep neural network parameterized by q. The Q-function can be recursively rewritten as Q p (st ,at) = Est+1 [rt+1 + gEat+1∼p[Q p (st+1,at+1)]] for updating. DQNs use a replay buffer to store the transition (s,a,s′ ,r). DQN is trained with experience replay by minimizing the squared TD-error:

where y DQN k = rt +gmax at+1 Q(st+1,at+1;q -). q - are the target network parameters.

Advantage Actor-Critic (A2C)

Policy gradient techniques aim to estimate the gradient of expected returns with respect to the parameters of its policy:

where J(pq ) is the accumulated expected return. � ¥ t ′ =t g t ′ -t r t ′ term leads to high variance, as the return can vary drastically between episodes. Actor-Critic methods aim to ameliorate this issue by using a function approximation of the expected return V p (s) = Ea∼p(s)[J|st = s, at = a] and replacing the original return in the policy gradient estimator with this function. A2C employs two deep networks, a policy network to learn policy, p(a|s;q) parameterized by q and a value network to learn value function, V p (s;j) parameterized by j.

The policy network is updated according to the policy loss:

3 Microgrid ElectricityMarket

3.1 Microgrid Dynamics

We model the smart home in Section 2.2 as the basic unit, which mainly follows settings in [Wu et al., 2018]. The tariff scheme in the microgrid is TOU, which gives different prices according to the time slot. At the beginning of each time slot, given the price signal, all households need to schedule the power trading and EV charging. To avoid the rigid schedule for the appliances usage, the base load from traditional appliances activities remains unchanged. The microgrid market can be modeled as partially observable Markov Games (G) and below we present each component of G.

States The state for each home is composed of the following local observations: electricity price pt at current time slot t, home based battery state of charge Hb,t , home based PV energy generation amount Hp,t , base load power consumption Hl,t , EV charging availability Ea,t , time to EV departure Ed,t and current EV battery state of charge Eb,t . Hb,t and Eb,t are 100% when the battery is fully charged and 0% when fully discharged. Ea,t is the charging availability of EV (set as 1 when available and 0 otherwise). Ed,t shows how many hours remains before EV departure. Formally, household i's status o i t ∈ O is defined as o i t = (pt ,H i b,t , H i p,t , H i l,t , E i a,t , E i d,t , E i b,t ).

Actions At the beginning of each time slot t, the home EMS needs to decide two actions: Pc,t for power trading amount and Ce,t for the EV charging rate. Homes can sell energy to the power grid and purchase energy for consumption and storage. Both the EV and home battery can be charged or discharged with continuous values (from zero to the maximal allowed charging rate). The home battery is responsible for redundancy caused by Pc,t and Ce,t with satisfying Equation 5.

where Cb,t is the charging rate for the home battery. Following [Wu et al., 2018], the EV charging actionCe,t is designed as five discrete rates: 100% charging rate, 50% charging rate, 0, 50% discharging rate and 100% discharging rate. Each household's original action space for power trading contains a continuous value domain, which ranges from the sum of its current stored and generation energy to the maximum energy the household can consume during the current time slot. To reduce the power trading action space, we set two subaction sets for trading power. For individuality, each household's action values refer to its yesterday's power consumption and generation statistics. The home EMS purchases power at different rates based on the average hourly gross power consumption s of previous day. Home EMS can also sell power at different rates based on the average hourly gross power generation b of previous day. Thus, the action set for Pc,t is {s,75%s,50%s,25%s,0,-25%b,-50%b,-75%b,-b}. Action 0 means no trading or no charging. Let Be denotes the EV battery capacity and Bh the home battery capacity. When charging or discharging, the efficiency h is set at 0.9. At each time slot, we check the availability of EV to decide whether applying a constraint to set Ce,t as 0. After determiningCe,t and Pc,t , the power surplus or shortage is calculated as Ct =Hp,t +Hb,t ·Bh-Hl,t -Ce,t . The home battery is updated passively for redundant operation if Ct ̸= 0 after performing Ce,t and Pc,t . The battery charging rate and discharging rate are no more than the maximal charging power rate while deciding the allowed actions.

Rewards The electricity cost is calculated as follows:

where pb and ps are the current electricity buying price and selling price separately. ps is smaller than pb because of specific utility programs such as long-distance transmission fees.

Transitions At each time slot, after all households deciding actions, the microgrid processes these energy decisions to deterministically step into the next state. All the decisions determine electricity prices as shown in the next Section 3.2. After energy actions done, each household updates its state based on its own energy actions and the TOU price signal.

3.2 Group Incentive Mechanism

In the local microgrid, each household can interact with power grid directly. But it is better for these households to coordinate into a group as in [Cailliere et al., 2016] to locally match the production and the consumption of the group, which not only helps balance the demand and supply for the power grid but also saves costs for households. We design an incentive-driven market mechanism to attract users to join in a group for coordination. The microgrid market mechanism has two trading processes: the internal trading process and the external trading process. Households trade inside the group first to satisfy the demand of each other. If the internal trading cannot fully meet the group, then the external smart grid will deal with the unsatisfied demand. To encourage users to actively participate in such a microgrid, we set the internal power price and external power prices as follows:

where pos,t and pob,t are the current power selling and buying prices for customers, which are different due to utility program fees. pin,t is the current internal power trading price. With such a constraint price, households are willing to trade inside first because of the better price and the same usage guarantee as directly interacting with the outer grid. On the other hand, smart grid may benefit from such a market mechanism because of possible peak load reduction by the group coordination. It is common to see extra aggregate demand or supply after internal trading. Thus, the final cleaning price for electricity integrated with external trading is:

where ps,t and pb,t are the power selling price and buying price at time t. ys,t and yb,t are the total power selling and buying amount at t. We can observe that ps,t is always in the range [pos,t , pin,t ] and pb,t is always in [pin,t , pob,t ], which indicates customers are willing to trading inside for better prices.

For reducing energy cost, households need capture overall market dynamics to decide when to buy and sell power. Through the incentive mechanism, we turn the smart home community a multiagent system, where each agent's reward is determined by trading prices affected through the aggregation of the group behavior. Promoting the group coordination is needed for better cost optimization and such a problem is inherently multiagent and can be solved by MARL approaches.

4 Entropy-Based Collective Multiagent Reinforcement Learning Framework

4.1 EB-C-MARL Algorithm

Previous works of the home energy management mainly focus on optimizing a single household's or a small-scale customers' electricity operating cost [Atzeni et al., 2013; Berlink and Costa, 2015; Wu et al., 2018]. As they are never designed for large-scale multiagent systems, applying existing approaches directly for the household community may bring the potential harm such as raising a new peak load and even jeopardizes the infrastructure of power grids. Thus, it is necessary to investigate the home EMS algorithm from the perspective of a household community as presented in this paper. Algorithm 1 describes our EB-C-MADRL framework.

Algorithm 1 Entropy-based collective MARL in microgrid

Input: the episode number M; each episode's step number T; the household number N.

Output: constructed DQN or A2C.

• 1: Initialize the deep network with randomweights;
• 2: Set the initial collective group behavior approximations a′s,t = 0, a′b,t = 0, ⃗C′e,t =⃗0, t = 1, 2, .., 24;
• 3: for episode =1,2, ...,M do
• 4: Initialize state s1 and all households' state (o 1 1 , o 2 1 , ..., o N 1 );

5:

6:

for t = 1,2, ...,T do

for i = 1, 2, ..., N do

• 7: Household i chooses P i c,t and C i e,t based on its state o i t and the approximations of market dynamics a′s,t , a′b,t and ⃗C′e,t ;

8:

end for

9:

Process actions, collect market information as,t , ab,t and

10:

⃗Ce,t , update a′s,t%24 = as,t , a′b,t%24 = ab,t and ⃗Ce,t%24 = ⃗Ce,t ;

for i = 1, 2, ..., N do

11: Compute individual entropy h i t by Equation 15 and electricity operating cost r i t by Equation 17;

12:

end for

13:

14:

Observe st+1 and all households' state (o 1

t ,o 2

t , ...,o N

t );

Update neural networks with Equation 2 or Equation 4;

• 15: end for

16: end for

The descriptions of the process of EB-C-MADRL are shown in Algorithm 1. Line 1 initializes network parameters. Line 2 initializes approximations of the collective group behavior for 24 hours. Line 4 initializes the starting state and households' observations. Line 6-8 chooses actions for each household based on its observation and the approximations of collective group behavior detailed in Section 4.2. Line 9 executes actions and abstracts market dynamics. Line 10-13 shows that once actions are performed, each household's reward and entropy are calculated as in Section 4.3 and the next state is observed. Line 14 updates the neural network.

4.2 Collective Group Behavior

All households are frequently interacting in the local microgrid market as each household determines its action choices at each time slot. Such a massive dynamic property raises huge challenges for the home EMS algorithms particularly RL based ones. One primary problem is that each agent's policy is changing as training progresses, and the environment becomes non-stationary from the perspective of any individual agent [Lowe et al., 2017]. Even if we could obtain actions from other agents without the privacy conservation to mitigate the non-stationary issue, in the large-scale multiagent systems, the representation of the joint action becomes another prominent problem. The joint action space of the agents grows exponentially with the number of agents, which makes the value function learning extremely hard [Yang et al., 2018]. The mean field approach in [Yang et al., 2018] is not applicable in microgrid since there is no "neighbor" for agents to interact with and each agent instead interact with the microgrid market only. However, in market settings where agents are influenced from their collective action effect, we could represent such collective influence by the market dynamics abstraction to avoid the action space explosion and finally mitigate the non-stationary problem. To approximate the collective influence of agents, we also resort to the daily periodicity in the power grid. Next we describe how the market dynamics abstraction is integrated with DQN and A2C.

Collective DQN

As Section 3.2 shows, households can be divided into buyer and seller groups dynamically to determine the electricity buying and selling prices. From each agent i's perspective, it is coordinating with the mircogird market instead of directly interacting with any individual agent. Thus, we could abstract market macro-actions to replace other agents' joint action to simplify the multiagent Q-function significantly. The jointaction Q-function can be simplified as follows:

where the abstraction of market dynamics a market includes the seller group collective action as (total supply amount), the buyer group collective action ab (total demand amount) and group EV charging distribution ⃗Ce. One additional privacy benefit is that each household only need to access to its own states and abstract market dynamics and cannot access to any other household. Then we obtain Equation 10:

The abstractions of current market dynamics cannot be exactly obtained as all households make decisions at the same time. Instead we propose using group collective actions at the same time slot in the previous day to approximate current market dynamics. As inhabitants exhibit similar living habits in daily life, it is reasonable to assume their energy activities have similar daily periodicity especially for group statistics:

where a′s, a′b and ⃗C′e are group action statistics at one day ago. The loss function for collective DQN is defined as:

Collective A2C

Similarly, the collective market actions also benefit each agent's home EMS policy when applying A2C by providing abstract information of other agents' current policies.

With ˜o i = (o i ,a′s,a′b ,⃗C′e) for simple presentation, the loss of policy network of collective A2C is given by:

V p i q ( ˜o i t ;j) =�a i p i q (a i | ˜o i t )Q i ( ˜o i t ,a i ) denotes the expected return.

4.3 Reward Shaping with Individual Entropy

For reducing the daily peak load, we use a mechanism called individual entropy to diversify household EV charging to different time slots. As [Muratori, 2018] shows, with more market share, the uncoordinated EV charging would result in higher peak loads as EV charging usually happens when people arrive home after work. Even learning with RL algorithms for each household, the uncoordinated learning will result in high peak load. It is because that EVwould charge in the low electricity price period coincidentally and there is no explicit factor to diversify the EV charging in training. Inspired by [Verma et al., 2018] which maximizes agent density entropy to make taxi drivers well-proportioned in different regions, we utilize the entropy to diversify the EV charging behavior.

Unlike using the system's total entropy for each agent in [Verma et al., 2018], we use more accurate individual entropy h i t to assign credits of contributing to the system's entropy for each household. Intuitively, if one household chooses a low-frequency action different from others, a higher bonus would be assigned to the household as it contributes more to the system's entropy Ht . If user i chooses an action a i t from action set A at t, then h i t for user i is calculated as follows:

where pat is the frequency of action at in all actions performed at t. h i t gives the incentive to choose a different action from current high-frequency actions. Therefore, it helps reduce the peak load by mitigating the phenomenon that households charge EV concurrently. The individual entropy is accurate credit assignment of the system's entropy which represents the distribution degree of EV charging behavior:

Also, the individual entropy is hoped to only affect the EV charging action which often happens in low-price periods. Therefore, we add an adjustment term to lower the weight of entropy when the current TOU price pb,t is high. The final reward for household i with individual entropy is given as:

where pb,t and ps,t are the power buying and selling prices at t. P i c,t and P i c,t are household i's current power purchasing and selling amount. h i t is the entropy bonus term for household i in Equation 15 while coe f is the entropy coefficient. Each agent receives the reward signal including h i t from themarket.

5 Experiments and Analysis

5.1 Experiment Settings

We use various real-world data to establish our microgrid simulation environment. TOU electricity price from [Wu et al., 2018] and residential power consumption data in 2013 from [Muratori, 2018] are used. The power consumption data contains 200 households' consumption patterns in US. The home battery and EV configurations are the same as in [Wu et al., 2018]. The daily driving distance data used to calculate the EV status when arriving home obeys gamma distribution [Lin et al., 2012] with shape at 1.6 and scale at 20. The parameters are highly frequent in daily vehicle driving distance datasets [Lin et al., 2012] and the average daily driving distance is close to [Pl¨otz et al., 2017] in the US. We use irradiance data in 2013 of Building 8167, Gatton Campus [of Queensland, 2018] for PV generation. For user diversity, we vary each household's installed power capacity from 1.1kW p to 4.4kWp at uniform distribution. For simplification, we set the electricity selling price in the external trading process pos at half of current TOU price, which is used as external electricity purchase price pob for households. The internal trading price is set as the average between current pos and pob. Next, we validate EB-C-MADRL in the simulated microgrid.

5.2 Validating the Collective Group Behavior

We first validate the collective group behavior abstraction. At each time t, agents i in total 200 agents receives (pt , H i b,t , H i p,t , H i l,t , E i a,t , E i d,t , E i b,t ,a′s,t ,a′b,t ,⃗C′e,t ). We adopt DQN and A2C as control algorithms for each household. For speeding up the training process in the large-scale settings, we consider parameter sharing in the home EMS strategies as households can be viewed as homogeneous agents and the learned policy is a generalized strategy for households.

Such a training paradigm is widely adopted in multiagent reinforcement learning algorithms and shows excellent performance [Yang et al., 2018]. We use the first four weeks' data in 2013 for training and the remaining data for evaluation. The training phase contains 125 episodes and each episode lasts for 28 days with interaction interval being an hour.

We compare the performance of the proposed control algorithms with two baselines: rule-based and DQN, which performs best in the previous single household case [Wu et al., 2018]. The rule-based control algorithm is called Naivegreedy policy described in [Berlink and Costa, 2015], which charges the EV when arriving home and sell the energy when there is a power surplus. Such a policy results in high electricity operation costs as EVs are charged at high-price periods. Then we augment both DQN and A2C with market dynamics approximations to validate the collective group behavior abstraction. Table 1 shows the electricity operating results. And Daily Peak Load is calculated by adding all daily peak loads up in testing, which is not optimized temporarily.

All results are averaged over the running of 20 random seeds. Naive-Greedy policy performs poorly as it charges EV in high-price periods and never utilizes the home battery to store energy for the later use. With the collective group behavior approximations, collective DQN performs better than DQN by 16.24% in terms of the total operating cost while collective A2C's cost is less than A2C by 3.58%. The collective A2C achieves the least operating cost by more internal trading and shifting the EV charging to off-peak hours.

5.3 Validating the Individual EntropyMechanism

Despite achieving the least cost, collective A2C still has high peak loads caused by the uncoordinated EV charging aggregation. To be specific, RL agents would independently learn to optimize cost by discharging EV when arriving home to make profits and charging EV in low-price periods, which centrally results in high peak loads. To mitigate the new peaks, we enhance collective DQN and collective A2C with individual entropy in Equation 17 to encourage agents to diversify EV charging. Table 2 gives the results of related methods and the EB-C-MADRL framework with the best settings. Compared with DQN, entropy-based collective A2C (EB-CA2C) achieves 24.69% cost reduction and 5.15% peak load reduction. Note that, the operation cost is counted based on the real electricity cost and does not include the entropy term.

Next we investigate EB-C-A2C and entropy-based collective DQN (EB-C-DQN) using different coefficients as Figure 2. EB-C-A2C achieves the best results for both the operating cost and peak load. EB-C-DQN seems not sensitive to the individual entropy because of its deterministic policy. On the other hand, collective A2C's stochastic policy can change its action more easily with the slight adjustment of action probabilities to get the higher entropy bonus. With the entropy coefficient being 1, the group daily peak load could achieve the least amount while the electricity operating cost reduces a lot dramatically. Such a cost reduction mainly results from two reasons. One reason is that the individual entropy implicitly divides action space into two sub-spaces by directly affecting one action sub-space only, leading to easier separate learning of power trading and EV charging actions. Another reason is that the combination of the approximation of current EV charging distribution and the individual entropy reflects microgrid dynamics and other households' policy tendency, which further alleviates the non-stationary problem. Too large entropy coefficient would increase both the group peak load and electricity operating cost. This is because that households learn to discharge and charge EV repeatedly to get more entropy bonus which makes energy cost ignored.

6 Conclusion and FutureWork

In this paper, we focus on a large-scale smart home EMS problem. First, we model the group energy optimization as a multiagent coordination problem in an incentive market. Then we propose EB-C-MADRL to learn home EMS control policies. Experiments exhibit superior performance of our method for operating cost saving and peak load reduction.

As future work, auction mechanisms could be considered to make microgrid market more liberal. Also, using EBC-MARL integrated with more advanced DRL algorithms to explore other large-scale multiagent markets such as ecommerce markets or water allocation markets is interesting.

Acknowledgements

The work is supported by the National Natural Science Foundation of China (Grant Nos.: 61702362, U1836214), Special Program of Artificial Intelligence and Special Program of Artificial Intelligence of Tianjin Municipal Science and Technology Commission (No.: 569 17ZXRGGX00150).