Batch Reinforcement Learning for Smart Home Energy Management

1 Introduction

Smart grids are power grids that use information and communication technology to gather data about the behavior of suppliers and consumers, so as to improve the efficiency, reliability, and sustainability of electricity production and distribution [Hammoudeh et al., 2013; Hashmi et al., 2011; Uluski, 2010]. In this scenario, consumers can insert energy into the power grid and participate in the energy supply management [Palensky and Dietrich, 2011]. Such new functionalities leads to a new house concept that is integrated with smart grids: the smart home, depicted in Figure 3.

Fundamental elements of smart homes are: Microgeneration, providing power generation through alternative sources such as wind and solar; Energy Storage, storing energy through devices such as rechargeable batteries or electric vehicles; Demand Control, regulating the use of home appliances, so as to reduce the energy bill; Bidirectional Power Flow, so that the smart home can acquire or insert energy into the power grid; and Differentiated Tariff, meaning different electricity prices for each time of the day, thus inducing the consumer to avoid the peak periods.

Because the price of energy and power generation through alternative sources are variable during the day and because the user can store energy using storage devices, the optimization of the house energy balance faces a very complex dynamics. Thus, it is necessary to develop decision-making tools that operate as Energy Management Systems (EMS), to minimize the user electricity bill or to maximize the user profit in a given period.

Indeed, in the power systems community many optimization algorithms have been developed for EMSs. Depending on the optimization objectives and constraints, appropriate techniques for modeling and for policy generation can be applied. Dusparic et al. [2013] propose a multiagent approach that uses predicted energy consumption data to manage the energy demand of a group of houses. Their strategy, called W-learning, consists of integrating independent Q-learning agents, one for each house. In contrast, O'Neill et al. [2010] present an algorithm called CAES, based on a single reinforcement learning agent that controls the home appliances to smoothen the energy usage. Their objective is to operate the appliances when the price is low and with minimum delay, reducing energy costs up to 40%. Many proposals combine different optimization techniques to provide a fast and robust solution. In [Chen et al., 2013], linear programming (LP) and Monte Carlo techniques are combined in an EMS that has as output the usage schedule of a set of appliances. Their approach has as its main advantage the fast solution provided by the LP solver. Kim and Poor [2011] and Chen et al. [2012] also propose a solution that aims at operating the home appliances, which are categorized under a deadline constraint that is defined considering the particular operation of each appliance.

One of the disadvantages of these solutions is the rigid schedule for the appliances usage; this is clearly uncomfortable for the end users. Scheduling is better handled by Truong et al. [2013], who propose a system that learns the users preferences, minimizing the impact on their daily habits while reducing the energy bill up to 47%. In [Mohsenian-Rad et al., 2010], a distributed demand-side EMS that uses game theory (GT) is used with several users, by formulating an energy consumption scheduling game. GT is also used by Atzeni et al. [2013], which consider a day-ahead optimization process regulated by an independent central unit. Shann and Seuken [2013] promote a demand response while learning the user's preferences over time and automatically adjust home temperature in real-time as prices change.

In short, these previous proposals consider different ways of dealing with the decision-making problem. Each framework considers subsets of the subsystems depicted in Figure 3 to provide demand response, and it is difficult to pick one as benchmark.

In this paper, microgeneration and storage are controlled to supply energy, while power consumption is considered only as additional information in our model. Consequently, we avoid rigid schedules for appliance usage. We adopt a control strategy that coordinates energy generation and storage in response to prices, promoting an efficient response to demand while ensuring the comfort of the user in deciding when to use appliances.

Our main contribution is a new and flexible EMS, the Reinforcement Learning-based Energy Management System (RLbEMS), that learns an operation policy for selling or storing energy surplus in smart homes. We propose a new approach for both modeling and solving the EMS problem. The modeling of the decision-making problem as a Markov Decision Problem (MDP) offers a novel way to take energy prices into account. The solution by Batch Reinforcement Learning (BRL) is effective because of smartly uses the available historical data. RLbEMS was tested with real operational data from two different places. As these data have a strong dependence on location, we demonstrate that our proposed system can be used in situations characterized by distinct levels of uncertainty.

The remainder of this paper is structured as follows. In Section 2, we briefly describe our mathematical framework. Our proposal, the RLbEMS system, is described in Section 3. In Section 4 we experimentally evaluate and analyze the RLbEMS system and, in Section 5, we conclude and discuss future steps.

2 Theoretical Framework

An EMS is a sequential decision-making system whose aims is to maximize the smart home long-term gain. We use Markov Decision Processes as modeling tool, and produce policies using a Batch Reinforcement Learning algorithm, called Fitted Q-iteration [Ernst et al., 2005a]. Our model and solution pay attention to historical data on energy prices, the current season, and energy generation, storage, and demand. The required theoretical background is given in this section.

2.1 Markov Decision Process

Sequential decision systems evolve probabilistically according to a finite and discrete set of states. At each time step the system first observes the state of the process, then chooses and executes an action that leads it to another state, and receives a reward (as shown in Figure 2). A formalism widely used for this is the Markov Decision Process (MDP) [Puterman, 1994]. Its key concept is the Markov property: every state encodes all the information needed to make the optimal decision in that state.

Formally, an MDP is a tuple < S, A, T, R >, where [Puterman, 1994]: S is a finite set of states; A is a finite set of possible actions; T : S ×A× S → [0, 1] is a state transition probability function, which defines the transition probability from a state s(t) ∈ S to a state s(t + 1) ∈ S when an action a(t) ∈ A is applied in s(t); R : S × A → ℜ is the reward function. So, for each state transition r(t) = R(s(t), a(t)). We define As as the admissible set of actions for state s ∈ S. So, if i ∈ S is the current state, the transition from i ∈ S to j ∈ S in response to the application of the action a ∈ Ai will occur with probability T (i, a, j) and a reward R(i, a) will be received. Solving an MDP means finding a policy π that specifies which action must be executed in each state, considering the maximization of the discounted cumulative rewards received during an infinite time horizon.

2.2 Batch Reinforcement Learning

Reinforcement Learning (RL) [Sutton and Barto, 1998] solves MDPs in problems where the agent has little or no information about the controlled system. Batch Reinforcement Learning (BRL) is a subfield of RL whose main purpose is the efficient use of collected data. Moreover, BRL tends to yield better policies, faster than classical RL algorithms such as the popular model-free on-line Q-learning algorithm [Watkins, 1989]. BRL is used in RL problems where the complete amount of learning experiences is fixed and given a priori [Ernst et al., 2005a]. By taking the learning experience as a set of transitions samples from the system, the learning mechanism seeks the best possible solution out of this given batch of samples. The major benefits of this approach come from the way it uses the amount of available data and the stability of the learning process. The fast convergence and the use of a predefined set of given samples are crucial for implementing this technique in real problems, because we usually need a good solution quickly and with the lowest possible process interaction.

BRL has three specific and independent phases: Exploration, Training and Execution. In the Exploration phase, the system acquires a batch of transition samples from the environment; in the Training phase, the system learns the best policy from this batch; and in the Execution phase the system just applies the learned policy.

The Fitted Q-iteration (FQI) algorithm [Ernst et al., 2005a] is one of the most popular algorithms in BRL due to its simple implementation and its excellent results. FQI converts the "learning from interaction" scheme to a series of supervised learning problems. The FQI Training algorithm is shown in Algorithm 1. Let F = {(s(t), a(t), r(t), s(t + 1))|t = 1, ..., n} be a batch of transition samples from the environment. At each iteration, for each transition (s(t), a(t), r(t), s(t + 1)), the value ¯ qh s,a is updated. A training set TSh is defined, with ((s, a), ¯ qh s,a ) ∈ TSh , in which (s, a) are the inputs and ¯ qh s,a is the target of the supervised learning problem. At each iteration, a new training set is defined as the last one united to the updated values, ¯ qh s,a . After that, a supervised learning algorithm is used to train a function approximator on the pattern set TSh . At the end, the resulting function, ¯QH , is an approximation of the Q-function after H steps of dynamic programming. Thus, we can use a greedy policy in FQI to define the policy π. The obtained policy is the best one with respect to the fixed batch of transition samples acquired from the exploration process.

Algorithm 1 Fitted Q-Iteration Algorithm Require: Load F = {(s(t), a(t), r(t), s(t + 1)) | t = 1, ..., n}; Require: Define ¯Q0 (s, a) = 0, ∀(s, a) ∈ F, and ¯ qh s,a ∈ ¯Qh (s, a); Require: Define H as the Horizon to be performed. Require: Define TS0 as an initially empty training set; h = 1; while h ≤ H do for all (s(t), a(t), r(t), s(t + 1)) ∈ F do ¯ qh s,a = r(t) + γ maxa∈As(t+1) ¯Qh-1 (s(t + 1), a); if ((s(t), a(t)), .) ∈ TSh-1 then TSh-1 ← TSh-1 - {((s(t), a(t)), .)}; end if TSh ← TSh-1 ⋃{((s(t), a(t)), ¯ qh s,a )}; end for %% Supervised Learning %% Use supervised learning to train a function approximator ¯Qh (s, a) on the training set TSh ; h ← h + 1; end while %% Obtaining Policy %% for ∀s ∈ S do π(s) = argmaxa∈As ¯QH (s, a), ∀s ∈ S; end for

3 The RLbEMS System for Smart Homes

Here we describe the RLbEMS system, and describe how RLbEMS learns a selling policy through the FQI algorithm. Finally, we describe how RLbEMS applies the learned policy when new data from the environment are available.

3.1 RLbEMS as anMDP

We consider a single residence that has its own energy generation by a Solar PV System, that receives the pricing signal in real-time, and that has batteries to store energy whenever it is convenient, with BMAX as storage capacity. RLbEMS must not sell more than the total available amount of energy (stored and surplus energy) and must respect the battery maximum capacity of storage and the battery maximum charge and discharge rate.

States

We define the MDP states using completely observable and relevant information about the problem: the energy storage level, the amount of energy generated, the energy consumed by the smart home, and data on price trends. Information on price trends consists of two indexes that represent the evolution of prices and average prices in a given time window. Considering a time window with three instants, and having prices in instants k, k-1, and k-2, we define Price interval ∆p(k) = [p(k - 2), p(k - 1), p(k)]. For each ∆p(k), the price average value, ∆p(k), is defined. We also define ∆2 = p(k) - p(k - 1), and ∆1 = p(k - 1) - p(k - 2). These variables are mapped into the indexes --→ p(k)IND and ∆p(k)IND as indicated in Tables 1 and 2. The MDP state, s(k) ∈ S, is defined as an array composed of five variables: the power stored in the battery, B(k); the power generated by the Solar PV System, G(k); the power consumed by the smart home, D(k); and the indexes of price trend, --→ p(k)IND∈ {1, 2, ..., 5}, and price average level, ∆p(k)IND ∈ {1, 2, ..., 8}:

Actions

Actions indicate when and how much of the surplus energy to sell given the system state and the current season. A null action means that no energy should be sold at a given time, i.e., RLbEMS must store the surplus in the battery or, if there is no surplus, RLbEMS must not act. A null surplus means that the energy demand is supplied by the microgeneration and by the battery, or that the EMS must acquire energy from the power grid to attend its demand. The discrete set of actions is defined as:

where GMAX is the maximum amount of energy generated, and SMAX is the maximum amount of energy that can be charged/discharged from the battery in each discrete instant. We define the amount of available energy Cu(k) as:

A subset of admissible actions is defined for each state, taking into account the value of Cu. In our model, the user consumption changes the control action to be taken, also indicating when we might sell energy to the grid. Thus, we have three cases, all complying the limit of charge/discharge of the battery, SMAX :

1. If Cu(k) ≤ 0, then the EMS should buy Cu(k) energy or do nothing (if Cu(k) = 0), and As = {0};
2. If Cu(k) > 0 andCu(k) ≤ BMAX, then there is surplus and we can even store all of it; so As = {0, ..., Cu(k)};
3. If Cu(k) > 0 and Cu(k) > BMAX, then there is surplus and we can not store all of it; so As = {Cu(k) - BMAX , ..., Cu(k)}.

Reward Function

We use a reward function that reflects whether energy is sold for a price that is above or below the average historical energy price for the current season. That is, the reward function conveys the average price of all available prices for that season. We adopt:

where p corresponds to the average historical price, p(k) is the price at the decision time and a ∈ As is the amount of energy sold in state s ∈ S.

3.2 FQI in RLbEMS

BRL has three independent phases: Exploration, Training, and Execution. In the Exploration phase, RLbEMS captures historical data of energy generation, energy consumption, and energy price for each season of the year. Data are acquired synchronously, i.e., there is a measurement of each variable at each discrete instant, k. The battery energy level is initiated with a random value and, using the first measure of the historical data, the initial state, s(0), is defined. For each state s, the set of admissible actions, As is defined, and an action a ∈ As is chosen randomly and applied. After applying a(k) in s(k), the next state s(k + 1) is reached and the immediate reward r(k) is calculated (Equation 4); then this experience

(s(k), a(k), r(k), s(k + 1)) is inserted in F as a new sample from the environment. This process runs until all historical data are used for each season. The main result of this step is the batch of samples, F, that is used in the Training phase to obtain the energy selling policy.

In the Training phase, RLbEMS runs the Algorithm 1 given in Section 2.2. This FQI algorithm aims at finding a function that approximates the value-action function Q(s, a), ∀s ∈ S, ∀a ∈ A. Thus, the main result of this step is to produce an approximate surface QS : S × A → Q, that maps each pair state-action to its corresponding Q(s, a) value. By analyzing experimental results from a benchmarking test, we chose a Radial Basis Neural Network [Park and Sandberg, 1991] to approximate the function Q(s, a). Here the Algorithm 1 runs until a convergence criterion dQ is met or until a fixed value of the horizon H is reached. We define dQ = 5%, which means that the convergence criterion is met if the estimated Q-value for each sample of F varies less than 5 % from one iteration to the next. We use H = 200.

After training, RLbEMS enters the Execution phase, where the current data determine the state of the system. Having the current state and the energy-selling policy learned in the training phase for the current season, an action is defined and the user profit is calculated, as Algorithm 2 describes. If the user has purchased energy from the power grid, the energy bill is acquired directly from the smart meter. Thus, the financial gain is calculated for each season by subtracting the energy bill from the total profit acquired with Algorithm 2, T otalPR.

Algorithm 2 The RLbEMS Execution Mode

Require: Observe the current season of the year;

Require: Load the learned policy πSEASON;

Require: Define TotalPR = 0.

while TRUE do

Observe the current state s(k);

a(k) ← πSEASON (s(k))

Apply a(k);

Calculate the current profit: PR(k) = a(k) × p(k) T otalP R ← T otalP R + P R(k);

end while

4 Experiments and Results

Two case studies were performed to evaluate the RLbEMS performance. Given the influence of climate characteristics and the energy price model, we conducted these case studies in two different places, the USA and Brazil.

All tests compared RLbEMS-generated policies with a Na¨ıve-greedy policy, which at each instant sells all energy surplus, Sr(k), by the current price:

If Sr(k) ≤ 0, the policy applies a null action, which means that the generation meets the demand or that EMS must buy energy from the grid. We also compared the RLbEMSgenerated policies with a Simple Storage Policy (SSP), which sells the energy surplus to the grid when the price is above average, and stores surplus when the price is bellow average.

To test RLbEMS, we have gathered real historical data of energy generation and price for Brazil and the USA. The energy consumption data were generated from real statistical data on the usage of home appliances, available in [PROCEL, 2014]. The energy microgeneration, the energy demand and the energy price are external variables that vary their profiles in a long period, following a typical pattern that is related to the season. This can be viewed in Figure 3 for the energy price in the USA. The same behavior is observed for the energy generation and consumption, and these data are always synchronized by day time, as stated earlier in Section 3.2.

Due to these facts, the annual historical data were divided into four sets of data, each one corresponding to one season of the year. After this, each database was divided into two subsets of data: Training Data and Validation Data. For each season, the Training Data were used in Algorithm 1 to obtain the energy selling policy πSEASON, and the Validation Data were used in Algoritm 2, when we evaluated the real gains of applying the proposed algorithms. To perform our tests, we simulated the RLbEMS on-line execution using the Validation Data for Brazil and the USA. As we will see, the first one was a simplified version of the second one and was used as a preliminary test, to confirm the benefits of the proposed approach. Both smart homes, in Brazil and in the USA, generated their own energy with a Solar PV System and stored energy in a rechargeable battery. The data used in this article consider an output of a real Solar PV System. The rechargeable battery, in both cases, uses the model proposed by Atzeni et al. [2013], that considers as main parameters the charge and discharge efficiency, the maximum charging and discharging rate, rate of energy loss over time, and maximum storage capacity.

As a comparative index, we calculate the percentage increase of the financial gain over each season, given by the ratio of the financial gain obtained by using RLbEMS and SSP in comparison to the Na¨ıve-greedy policy. We also calculate the average peak reduction in each case study. The grid demand has its peak reduced by RLbEMS. To calculate this, we make a daily analysis of the energy request curve of the house from the power grid perspective: for each day of operation, we compare the peak demand when using the RLbEMS or not. We record the peak reduction for each day of operation, and calculate the average reduction for each season.

4.1 Case Study in Brazil

This case study considered a smart home located in S˜ao Paulo, Brazil. The differentiated tariff implemented in Brazil is a Time-of-Use (TOU) tariff. In this case, there are three values of tariff during the day [Bueno et al., 2013]. This pricing signal remains fixed during all days of the year and concentrate major values of energy price in times when there is a high demand of energy. The energy generation data were acquired from a real solar power plant in S˜ao Paulo, Brazil [Berlink et al., 2014]. The solar power plant consists of ten modules, connected as a serial array. Each module generates 255W on the peak of generation, resulting in a total peak of 2.55kW. We used hourly energy generation data from August 2013 to January 2014. Given that the amount of available data is small, we chose to implement a solution with a single policy. This decision did not affect the final result, because in this case study the pricing signal was fixed and the energy generation did not vary too much during the seasons.

In this case study, we achieved a financial-gain growth of 20.78% and an average peak demand reduction of 39.78%. The results achieved show that RLbEMS is feasible for the proposed problem. In this case, the obtained policy is basically to store the surplus energy during the day when the price is low, until the maximum capacity of the battery is reached; when the battery is full, it sells the surplus at the current price. In the evening, when energy is more expensive and there is no generation, the system either uses or sells energy, with greater profitability for the user.

4.2 Case Study in the USA

This case study was performed with the real pricing signal for the District of Columbia, USA. These data correspond to the Local Marginal Price (LMP), which is a Real-Time Pricing (RTP) tariff and reflects the value of energy at a specific location at the time that it is delivered [PJM, 2015]. This price was also used to calculate the amount of money paid to the consumer when he/she sells energy to the power grid. For this test, we used the hourly pricing data of five consecutive years (2008-2013). For the simulation, we used the Solar PV System given by Chen et al. [2013]. We considered a system composed of four KD200-54 P modules from the Kyocera Solar Incorporation [Kyocera, 2015] that has 220Wp as the peak energy generation per module. The same storage device proposed by Atzeni et al. [2013] was used, and the energy consumption was generated by the same statistical data mentioned before. Results per season are shown in Table 3.

The application of the RLbEMS policy increased the financial gain by the end of the period, resulting in an annual gain of 14.51%. The reduction of the financial gain per season when compared to the brazilian case was expected given that here we used the RTP tariff and that the algorithm had to handle a step level of uncertainty. However, it is worth noting that the use of the RLbEMS system resulted in a reduction of the average peak demand of 29.38%.

In Figure 4, we can see the RLbEMS execution for two consecutive days in summer. Note that when the system is able to identify the price trend correctly, the policy obtained is similar to the one achieved in the first case study: RLbEMS identifies periods of low price and stores energy to use when there is a peak of price. However, if this peak of price is not high enough when compared to the low period, the policy loses efficiency. This happens because, when the RLbEMS stores energy, it stops selling (and profits nothing). If the price difference between these two periods is small, the sale when the price is (only slightly) higher does not justify the storage of energy surplus. In this case, always selling the energy surplus at the time it is generated (as the Na¨ıve-greedy policy does), would give greater financial gain. Hence, the application of the RLbEMS policy could lead to a result that is worse than the application of the Na¨ıve-Greedy policy, as in spring, for example. Comparing the obtained results with the SSP policy we can see that the RLbEMS policy reachs a higher financial gain, which means that the algorithm actually identifies a better time to sell/store the energy, mainly because it considers not only the price in the decision, but also the dynamics of generation and consumption in the house. A fact to note occurs in spring, when the application of the SSP policy resulted in a greater gain for the user.

5 Conclusion

In this paper we have described our efforts to develop effective decision-making tools for energy management in smart homes. The specific problem is whether to sell or store surplus energy. To solve this problem, a new energy management system was proposed, RLbEMS, which employs MDPs and BRL to maximize user financial gain. In particular, RLbEMS learns autonomously a selling policy for each season, based on batches of historical data about energy price curves, microgeneration, and storage energy. We also demonstrated, through evaluation with real world data taken from two different places, Brazil and the USA, that RLbEMS outperforms Na¨ıve-greedy policy by up to 20.78% and 10.64%, respectively. Another important result achieved by our system was the reduction of the peak demand. In most cases, RLbEMS decreases peak demand by more than 30%, an exceptional result. By using RLbEMS the user does not suffer from restrictions regarding the use of appliances. We believe this degree of freedom is essential so that a solution can be accepted into the routine of end users, leading to the popularization of EMS systems.

Our future work will focus on the improvement of RLbEMS, by implementing an architecture that integrates off-line and on-line approaches through transfer learning. In this case, knowledge would be acquired by the application of BRL algorithms, and it would be transferred to an adaptive module that runs an on-line classical RL approach. Thus, the policy obtained by the BRL algorithm would be updated in real-time and the system would be completely adaptive.

Acknowledgments

We would like to thank CNPq (131239/2013-9 and 311608/2014-0) for supporting this research. We also thank the Smart Grid and Power Quality Laboratory ENERQ-CT at Universidade de S˜ao Paulo for providing the solar photovoltaic generation data used in this work.