Multi-Agent Reinforcement Learning for Automated Peer-to-Peer Energy Trading in Double-Side Auction Market

1 Introduction

Power systems are undergoing a significant transition from the fossil fuel resources to the decarbonlization of renewable energy resources (RES), providing the environmental energy challenges [Haller et al., 2012]. However, the less controllable and predictable output of RES introduce the new challenges to the power system planning and operation. In this respect, there has been a significant increase in developing small-scale distributed energy resources (DER) that are connected to the end consumers and provide flexibility toward to a more reliable system. The key categories of DER include distributed generation, energy storage (ES) and demand response [Jiayi et al., 2008]. In this context, traditional consumers evolve to prosumers, who can manage their energy consumption, production and storage of electricity through a home energy management system (HEMS) targeted to minimize their cost of energy bills and provide the system flexibility [Parag and Sovacool, 2016].

∗Corresponding author.

With the change of system structure, the electricity market will have to accommodate a large number of small-scale renewable energy prosumers. Feed-in Tariff (FiT) scheme has been introduced as the most common production subsidies across the world in which prosumers receive payments for the energy exported to the grid at FiT offered by the upstream utility company [Qiu et al., 2020]. This scheme allows prosumers to make use of the flexibility of their self-generated PV electricity and feed the surplus into the grid. Nevertheless, this subsidy may not recover the installation and operation cost of flexible DER. On the other hand, consumers who want to import energy from the grid may complain about the expensive Time-of-Use (ToU) price offered by the upstream utility company [Qiu et al., 2020]. Furthermore, with the traditional electricity market in Figure 1 (a), prosumers and consumers under such an independent and uncoordinated fashion only benefit from their own DER flexibility to ensure the selftemporal balancing of generation and consumption, but do not benefit from their sufficient energy flexibility interacted with each other to satisfy the overall system energy balance [Morstyn et al., 2018].

To this end, we introduce a peer-to-peer (P2P) energy trading paradigm that enables prosumers and consumers to trade locally independent of the upstream utility company [Morstyn et al., 2018], as illustrated in Figure 1 (b). In order to coordinate this energy trading activity, double-side auction (DA) is set up as an incentive-driven market mechanism to attract prosumers and consumers to cooperatively participate in local trading with the application of information and communication technologies (ICTs), and is validated as a highly efficient market mechanism [Friedman, 2018]. InDAmarket, buyers (consumers) and sellers (prosumers) strategically bid and offer their willing price and corresponding energy quantity, respectively. The market dynamics are expected to maximize the market clearing efficiency by shifting demand to the periods of PV generation and/or low prices as well as enabling sellers with PV and/or ES to sell their self-generated energy to buyers. We use multi-agent reinforcement learning (MARL) to learn individual trading strategies and energy schedules of flexible DER to minimize the energy bills. The experiments on a real-world scenario demonstrate that 1) the above effects in P2P energy trading can potentially result in economic benefits for all the market participants as well as higher balance of local demand and generation; and 2) the proposed DA-MADDPG obtains the superior performance over the Zero Intelligence (ZI) strategy [Friedman, 2018] and the other MARL algorithms (e.g., IQL [Watkins and Dayan, 1992], IDDPG [Lillicrap et al., 2016], IPPO [Schulman et al., 2017] and MADDPG [Lowe et al., 2017]).

2 RelatedWork

An appropriate market mechanism is required to facilitate the P2P energy trading among prosumers and consumers. [Alam et al., 2019] proposed a central coordinator that directly manages the energy schedules of all participants by maximizing the overall economic benefits. In this setting, the central coordinator has to require all participants' economic and technical parameters, violating their security and privacy issues of energy preferences and usage behaviors [Aitzhan and Svetinovic, 2018]. In this contrast, the introduced DA market under a decentralized manner has shown its market efficiency [Guerrero et al., 2019], and only a small amount of information is delivered to participants at relatively low computational costs.

In DA market, buyers and sellers are faced with a complex quotation decision process. Thus, an appropriate trading strategy is challenging to select in such a complicated market environment. Zero Intelligence (ZI) is a fundamental and popular trading strategy adopted by traders in DA market [Friedman, 2018]. ZI traders set their order price as a random surplus offset from its valuation, based on a uniform distribution from a specified range (e.g., FiT and ToU) without considering market transactions. In particular, the extant ZI strategy has been typically developed assuming that the market is static and all participants' energy availability is preoptimized by solving a day-ahead energy planning with perfect information, which means there is no change in demand and generation at the beginning of each trading day [Guerrero et al., 2019]. However, the real market is typically very dynamic considering that all participants' strategies are adjusted in real time as well as the information of price signals, PV generation and energy usage behaviors are stochastic.

To address this challenge, reinforcement learning (RL) is a framework to study sequential decision-making problems of agents (prosumers and consumers) gradually learning the optimal trading strategies by utilizing experiences acquired from its repeated interactions with the environment (P2P energy trading in DA market) [Sutton and Barto, 2018], which has been applied in many smart grid applications [Zhang et al., 2018]. Specifically, for DA market, [Nicolaisen et al., 2001] applied a modified Roth-Erev RL algorithm to help energy traders determine their price-quantity strategies in each auction round. Authors in [Sun et al., 2015] presented a general RL bidding strategy for controlling and coordinating HVAC (heat, ventilating, and air conditioning) systems in a DA market. Authors in [Pedasingu et al., 2020] applied DQN to adjust the price-quantity strategies in a day-ahead DA market. However, the majority of them only consider discrete state and/or action spaces by simply discretizing the original continuous spaces. Thus, the na¨ıve discretization throws away valuable information regarding the structure of the state and action domain, which may be essential to achieve optimal trading strategies. To this end, this paper has employed the deep MARL algorithm based on DDPG [Lillicrap et al., 2016] method to obtain multi-dimensional and continuous state and action spaces.

3 Energy Trading in DAMarket

In this section, we define the energy trading rules in DA market and then formulate them into Markov Games.

3.1 DAMarket Mechanism

The DA market matches multiple buyers (consumers) and sellers (prosumers) who are interested in (energy) trading, and is deemed as highly efficient mechanism. They are widely used in the trading of various types of commodities, such as stocks and electricity. A DA market lasts a fixed period of time, known as the auction period (e.g., hourly resolution in electricity market). It allows traders to submit their bids/offers at the beginning of an auction period, then the auctioneer (e.g., market operator) clears the market and publishes the public market outcomes (e.g., trading prices and quantities) at the end of each auction period [Friedman, 2018]. More specifically, a DA market comprises:

• A set of buyers B, where each i ∈ B defines its trading price pbi and the amount of energy to buy qb i , which means the buyer i would like to buy the qb i amount of energy at the price pbi .
• A set of sellers S, where each j ∈ S defines its trading price psj and the amount of energy to sell qs j , which means the seller j would like to sell the qs j amount of energy at the price psj .
• A public order book managed by an auctioneer, where the accepted bids and offers are listed, respectively. Buy orders queue in order book kb (i, pbi , qb i ) and are sorted by decreasing submitted buy prices, while sell orders queue in order book ks (j, psj , qs j ) and are sorted by increasing submitted sell prices.

The pseudo-code of the matching process in DA market is given in Algorithm 1. Once an auction period begins, traders submit their order information with a trading price and a corresponding energy quantity to the market. All submitted or-

Algorithm 1 DA market clearing algorithm

1: Allocate order book kb t and ks t at auction period t 2: Initialize i = j = 1 3: while pbi,t ≥ psj,t do 4: match the trading energy: qlt =min(qb i,t , qs j,t ) 5: calculate the trading price: plt = (pbi,t +psj,t )/2 6: update buy order book qb i,t ← qb i,t - qlt 7: if qb i,t = 0 then 8: i ← i + 1 9: update sell order book qs j,t ← qs j,t - qlt 10: if qs j,t = 0 then 11: j ← j +1 12: break if 13: i > length of kb t or j > length of ks t 14: end while 15: Balance the unmatched quantity at ToU λbt and FiT λst

ders are allocated in the order book (step 1). The matching algorithm iterates down the order books and attempts to match each buy order with sell order (steps 4-11) until the buy price is less than the sell price or no unmatched sell/buy order exists anymore (steps 12-13). Specifically, when two orders get matched, the auctioneer calculates the market clearing price between the matched buy price and sell price, using the traditional mid-pricing method [Friedman, 2018] (step 5), while the transaction quantity is equal to the minimum quantity between the matched orders (step 4). Due to the sorting principle and clearing algorithm, the clearing results promise the social welfare maximization [Friedman, 2018]. Finally, at the end of the auction period, the remaining quantity of energy and the unmatched orders are balanced by the auctioneer with the utility company at the grid prices of ToU and FiT. It should be noted that the pricing strategies of all traders are bounded between FiT and ToU to ensure the economic benefits.

3.2 Numerical Example of DAMarket Mechanism

To better illustrate the DA market clearing algorithm, we take the scenario in Figure 2 as an example.

Order Books

There are six agents participating into DA market. They are divided into three buyers and three sellers corresponding to the sign of their submitted quantities (positive for buyer, negative for seller). Then, the auctioneer allocates the order books according to the principle of price first (i.e., the quotes of buyers are in high-to-low order, whereas the quotes of sellers are in low-to-high), as presented in the left table.

Market Transactions

Transaction 1: the first transaction occurs when the first bid price ($0.12/kWh) is higher than the first ask price ($0.06/kWh), the matched quantity is the minimum quantity of buyer i1 and seller j1 (i.e., 2kWh) and the transaction price is the average of $0.12/kWh and $0.06/kWh (i.e., $0.09/kWh). In this transaction, buyer i1 is completed matched and should be removed, and buyer i2 is updated on the top of buy order book. On the other hand, the quotes of sellers do not change but the quantity of seller j1 is reduced to 2kWh.

Transaction 2: the second transaction occurs when the first bid price ($0.08/kWh) is still higher than the first ask price ($0.06/kWh), the matched quantity is the minimum quantity of buyer i2 and unmatched seller j1 (i.e., 2kWh) and the transaction price is the average of $0.08/kWh and $0.06/kWh (i.e., $0.07/kWh). In this transaction, seller j1 is completely matched and should be removed, while seller j2 is updated on the top of sell order book. The quantity of buyer i2 is reduced to 1kWh accordingly.

Transaction ends: this matching process stops when the first bid price ($0.08/kWh) is lower than the first ask price ($0.10/kWh). Finally, the unmatched quantities in the order book (right table) are balanced via ToU ($0.13/kWh) for buyer i2 at 1kWh and buyer i3 at 2kWh while FiT ($0.04/kWh) for seller j2 at 2kWh and seller j3 at 1kWh.

Market Outcomes

After all transactions occur in the DA market, the auctioneer publishes the market clearing outcomes that comprises: 1) the local trading price ($0.09/kWh for buyer i1 & seller j1, $0.07/kWh for buyer i2 & seller j1); 2) the cleared quantity in DA market for each agent; 3) the remaining/unmatched quantity traded with the utility company for each agent; and 4) the updated order books for all agents, as presented in the right table.

3.3 P2P Energy Trading as Markov Games

The DA market clearing algorithm outlined above can be formulated as a multi-agent coordination problem in the form of a finite Partially Observable Markov Game (POMG) [Shoham and Leyton-Brown, 2008] with discrete time steps. The POMG is defined by N agents with a set of state S describing global state, a collection of private observations

{O1:N}, a collection of action sets {A1:N}, a collection of reward functions {R1:N} and a state transition function T . The time interval between two consecutive steps is one auction period (∆t = 1 hour). At time step t, each agent n chooses an action an,t according to its policy µn(on,t) based on its private observation on,t. The environment then moves into the next state according to the state transition function conditioned on the actions of all agents. Each agent n obtains a reward rn,t and a private observation for next step on,t+1. Each agent n aims to maximize its cumulative discounted reward Rn = ∑Tt=0 γ t rn,t, where γ ∈ [0, 1) is the discount factor and T is the time horizon of the coordination problem (24 hours). The components of the POMG are detailed as:

Observation. Agent n at time step t has its observation on,t = [P inf n,t ,Ees n,t , λbt , λst ], which comprises: 1) the local information of its inflexible load P inf n,t 1 , ES battery energy content Ees n,t ; and 2) the grid information of ToU λbt and FiT λst .

Action. Agent n at time step t controls its action an,t = [apn,t , aqn,t ], which comprises: 1) the price decision apn,t ∈ [0, 1] representing the magnitude of willing price submitted to DA market as a ratio of FiT and ToU price differentials (pn,t = λst +apn,t (λbt -λst )); and 2) the energy decision aqn,t ∈ [-1, 1] representing the magnitude of charging (positive) and discharging (negative) power of ES as a ratio of its power capacity P es n .

State Transition. The state transition from time step t to t + 1 is governed by a function: st+1 = T (st, a1:N,t, ωt). It can be observed that the transition is influenced partly by all agents' actions a1:N,t and partly by the environment stochasticity ωt. In the examined problem, this corresponds to the exogenous state features [P d , P pv , λb , λs ], which are decoupled from agent's actions and are characterized by inherent variability and uncertainty. In this context, it presents significant challenges to identify suitable probabilistic models which can fully capture such randomness since it is influenced by many exogenous factors, such as energy usage behaviors, solar radiation, and utility pricing mechanism. RL remedies this problem in a data-driven approach thta does not rely on accurate mathematical models of the underlying uncertainties.

By contrast, the state transition for the endogenous state features Ees n,t is determined by the action aqn,t adopted at step t. Let C es n,t and Des n,t denote the charging and discharging power of ES, respectively. The mutually exclusive quantities C es n,t andDes n,t (as the charging and discharging activity of ES cannot occur simultaneously at a given step) are managed by action aqn,t , and are also limited by Ees n,t and the ES operating parameters: 1) minimum and maximum energy levels Ees n ,Ees n ; 2) charging and discharging efficiencies ηesc n , ηesd n .

1 for consumers: inflexible load equals to inflexible demand P inf n,t = P d n,t; for prosumers: inflexible load equals to the difference between inflexible demand and PV generation P inf n,t =P d n,t- P pv n,t .

Based on C es n,t and Des n,t , the state transition of Ees n,t can be expressed as:

Consequently, the quantity submitted to DA market qn,t of agent n at step t can be expressed as the summation of inflexible load and ES charging / discharging power, where the positive value represents the net demand to buy while the negative value represents the net generation to sell in DA market:

After collecting the price-quantity strategies (pn,t, qn,t) of all agents, the auctioneer allocates the order book kb t and ks t , clears the market (Algorithm 1) and publishes the market clearing outcomes [λln,t , qda n,t , qgrid n,t , kb t , ks t ], which comprises: the local trading price λln,t , the cleared quantity in DA market qda n,t , the remaining/unmatched quantity traded with the utility company qgrid n,t , and the updated public order books kb t , ks t .

Reward Function. Agent n at step t obtains its reward rn,t as the negative cost of energy bills developing from the DA market clearing outcomes. Specifically, for these agents who are successfully cleared in DA market will receive the local price λln,t and its cleared quantity qda n,t , then each agent n can calculate its corresponding cost in DA market and the remaining/unmatched quantity qgrid n,t will be bought or sold through the utility company at ToU λbt or FiT λst . For these agents who are unsuccessfully cleared in DA market, their quantity qgrid n,t = qn,t will be directly traded at ToU λbt or FiT λst .

4 MARLMethod

Directly applying single-agent RL methods to the multi-agent setting by treating other agents as part of the environment is problematic as the environment appears non-stationary from the view of any one agent, violating Markov assumptions required for convergence [Shoham and Leyton-Brown, 2008]. Specifically, this non-stationary issue becomes more severe in the case of deep RL with neural networks as function approximators. To this end, we propose an extension of MADDPG [Lowe et al., 2017], namely DA-MADDPG that learns a centralized Q-function for each agent (to alleviate the nonstationary problem and stabilize training) by abstracting the other agents' observations and actions through the DA market information (to promise the scalability and protect the private information). The general architecture of DA-MADDPG is illustrated in Figure 3.

Concretely, we consider a game of N agents with policies parameterized by θµ θµ θµ = {θµ 1:N }, and let µ = {µ1:N} be the set of all agents' policies. Then the gradient of the expected return for agent n with policy µn(on|θµ n ), Jn = Eµn [Rn] is written as follows:

here Qn(x, a1:N|θQ n ) parameterized by θQ n is a centralized action-value function that takes as input all agents' actions a1:N, in addition to some state information x (e.g., all agents' observations o1:N) and outputs the Q-value for agent n.

The conventional approach in MADDPG involves training all agents using a centralized critic estimator which takes as input the observations and actions of all agents. However, it is evident that the input dimension of such centralized critic grows exponentially with the number of agents, quickly rendering the problem intractable. Furthermore, driven by prosumers/consumers' privacy concerns, they are not willing to exchange their local observations (inflexible demand and/or PV generation) and actions (ES energy behaviors) with each other. Motivated by [Yang et al., 2019], this paper assumes the DA auctioneer as a trusted third party supplying agents with the public and dynamic market information of order books that epitomize the collective behavior of the market (thereby substituting the high-dimensional vector with all agents' information) in the centralized training process. This substantially improves the scalability of the proposed MARL method and also protects the privacy of prosumers/consumers. To this effect, we approximate the joint Q-value function as:

where ˆ kda n = {ˆ kbn , ˆ ksn } = {kbn-, ksn-, ∀n- ∈ N \ {n}} denotes the combination of buy and sell order books other than agent n in DA market. It can be observed that ˆ kda n is an embedded function that not only abstracts all other agents' observations (e.g., P inf n- ) as well as actions of the price strategies pn- and the quantity strategies qn- resulting from their ES energy schedules C es n- and Des n-, but also displays the DA market dynamics of P2P energy trading. As a result, this function provides a good approximation of agents' observations and actions as well as the DA market dynamics. Incorporating ˆ kda n in the critic estimation, each agent can make acquainted decisions on the basis of the impact of the actions of other agents in DA market, albeit not knowing their specific energy portfolios and usage activities. This averts the explosion of action and observation spaces and alleviates the environmental non-stationarity.

Given the collective Q-value function, the experience replay buffer Dn contains the tuples (on, an, rn, o′n , ˆ kda n ), recording experiences for each agent n. The centralized action-value function Qn is updated as:

where µ′ = {µ′1:N } is the set of target policies with soft updated parameters θµ′ n , ˆ kda′ is evaluated byµ′ µ′ µ′ given all agents' next observations. Note that the centralized Q function is only used during training. During decentralized execution, each policy µn(on|θµ n ) only takes its own observation on to produce the action.

Finally, it has to be discussed for this particular DA market applied problem that the DA market efficiency is completely represented by the public information of order books rather than the local observations and actions of all agents, since the market clearing outcomes (trading prices and clearing quantities) are directly calculated based on the order information. As a result, each agent in DA-MADDPG method receiving the market dynamics (i.e., order books) can make more acquainted decisions than the conventional MADDPG method.

5 Experiments and Analysis

5.1 Experiment Setup

Environment. The proposed MARL method is evaluated on a real-world open-source dataset recorded by the Australian distribution utility Ausgrid [Ratnam et al., 2017]. We collect the corresponding electricity load and PV generation data of 8 residential households, and form them into 4 prosumers (with PV generation) and 4 consumers (electricity demand only). For each of prosumers and consumers, an ES is installed to provide the system flexibility potentials, where its operating parameters are derived from [Papadaskalopoulos and Strbac, 2016] and presented in Table 1. The grid prices are offered by Ausgrid, including the ToU tariff as the grid buy price varying for the time presented in Table 2 and the FiT as the grid sell price fixed at $0.04/kWh during the whole day.

Baselines. We compare the proposed DA-MADDPG with the conventional ZI strategy [Friedman, 2018], four RL algorithms with three independent methods, e.g., Independent Q-learning (IQL) [Watkins and Dayan, 1992], Independent DDPG (IDDPG) [Lillicrap et al., 2016] and Independent PPO (IPPO) [Schulman et al., 2017], and one state-of-theart MARL method with centralised critics, e.g., MADDPG [Lowe et al., 2017]. To further evaluate the benefit of P2P energy trading, we benchmark the performance against to the scenario that agents trade independently with the utility company using DDPG method without P2P energy trading (Grid).

5.2 Training Performance

The convergence curves of mean episodic total energy bills of 8 agents evaluated by different MARL methods are illustrated in Figure 4. As for baselines, we first show the performance of three independent learning algorithms (IQL, IPPO, IDDPG) with decentralized critics. They fail to learn an optimal policy, where IPPO and IDDPG exhibit high variances which may be due to the non-stationarity issue. Conversely, MADDPG can alleviate the non-stationarity and learn the more effective coordination policies via the centralized critic incorporating with all agents' local observations and actions, but its capability could be limited due to the lack of the DA market information. Instead, we can see that the proposed DA-MADDPG method converges to the lowest energy bills owing to its more sufficient representation of the DA market collective dynamics for the critic.

5.3 Trading Analysis

Having demonstrated the superiority of DA-MADDPG method over the state-of-the-art MARL algorithms, this section aims at analysing: 1) the ES flexibility potentials in reducing peak demands; 2) the energy exchanges in local trading via DA market; and 3) the trading strategies under the dynamic DA-MADDPG method w.r.t. the statistic ZI method.

ES Flexibility. Although Grid, ZI and DA-MADDPG differ with regard to the P2P energy trading (under the latter twos) or not (under the first one), they exhibit some common trends. Compared with the inflexible load (w/o ES) in Figure 5, the net generation during mid-day hours 9-16 and the net demand during peak hours 17-20 of 4 prosumers are both reduced, since the abundant (free) PV generation during midday hours is locally absorbed by flexible ES, and the demand at peak hours is shaved through discharging of ES (Figure 6).

On the other hand, 4 consumers cannot make use of free PV generation locally, but still perform the same demand reduction effect during peak hours 17-20, either through charging ES from grid at cheap ToU during off-peak hours 1-8 (Grid, ZI) or buy energy locally from 4 prosumers in DA market during mid-day hours 9-16 (ZI, DA-MADDPG) in Figure 6. In this context, we can observe that agents under all methods are incentived to use their ES flexibility potentials to discharge to supply demand during peak hours through charging free PV generation during mid-day hours or cheap energy during offpeak hours.

Grid vs. DA. When P2P energy trading is allowed in DA market, prosumers/consumers with energy surplus/deficiency are incentived to trade locally among themselves. Thereby, we can observe that compared with the Grid case without P2P energy trading, the generation of 4 prosumers and the demand of 4 consumers during mid-day hours (when both energy surplus and deficiency exist) in Figure 5 are both reduced under ZI and DA-MADDPG methods, since a amount of energy is balanced locally in DA market.

ZI vs. DA-MADDPG. The final comparison is regarding statistic ZI and dynamic DA-MADDPG methods in the same DA market environment. On the one hand, we can observe in Figure 6 that consumers under ZI method charge ES from grid during off-peak hours 1-8 at low ToU and discharge ES to supply demand during peak hours 17-20 when ToU is high. However, consumers under ZI method do not exhibit any charging behavior during mid-day hours when prosumers' PV generation is abundant, this is because the energy schedules of ES are pre-optimized given the grid buy and sell prices in day-ahead planning without considering P2P energy trading. In other words, the quantity availability traded in DA market is the net demand/generation that assumes to trade with the grid in day-ahead (i.e., Grid case). On the other hand, we can observe in Figure 6 that consumers under DA-MADDPG method stop charging ES from grid during off-peak hours 1-8 but start charging ES during mid-day hours 9-16 with PV generation, this is driven by the availability of DA market allowing consumers to buy energy directly from prosumers with PV generation in real-time. Under such method, consumers learn the dynamics of DA market to save energy during off-peak hours and buy energy in DA market during mid-day hours, while prosumers learn to sell energy in DA market instead of selling their energy surplus (PV generation) back to the grid at the unfavorable FiT.

5.4 Economic Benefits

The final section aims at 1) evaluating the agents' economic benefits from DA market and 2) comparing the proposed DAMADDPG with the benchmark ZI strategy for DA market.

It can be observed in Figure 7 that all 8 agents receive the highest energy bills under Grid case without P2P energy trading. After participating in DA market, the energy bills lower down in ZI method. Furthermore, with DA-MADDPG modeling the dynamics of DA market, the energy bills achieve the lowest. The above economic trends can be also validated in Table 3: a) there is no internal trading under Grid case, so the net demand and generation (441.70kWh in total) are bought at high ToU or sold at low FiT with the utility company; b) ZI achieves $2.85 total energy bills saving by 31.74kWh internal trading within DA market; c) DA-MADDPG achieves the least total energy bills by making the highest internal trading at 65.42kWh. In relative terms, DA-MADDPG achieves approximately 17.81% and 29.62% lower external trading with the utility company as well as 13.22% and 16.69% lower total energy bills over ZI and Grid, respectively.

6 Conclusion and FutureWork

We introduce a P2P energy trading problem among energy prosumers and consumers evaluated on a highly efficient DA market mechanism. For this specific problem, we propose a novel MARL algorithm namely DA-MADDPG. Based on MADDPG, we construct a representative Q-value function for each agent by abstracting the other agents' observations and actions through the DA market public information. The proposed DA-MADDPG is evaluated on a realworld dataset and compared with five baselines (e.g., ZI, IQL, IDDPG, IPPO, MADDPG). The experimental results demonstrate the benefit of more internal energy exchange among agents through DA market. Consequently, it leads to higher economic benefits in reducing the cost of energy bills.

The future works are in two directions. The first one lies in modelling a more realistic P2P energy trading problem by increasing the population of agents and forms them into a large-scale multi-agent system and further demonstrate the scalability of the proposed DA-MADDPG method under this setting. Secondly, the economic model in this paper neglects the physical operations of electricity systems. To this end, future work will adopt the introduced DA market to an environment with network constrained microgrid.

Acknowledgments

This work was supported by the UK EPSRC project "Integrated Development of Low-Carbon Energy Systems (IDLES): A Whole-System Paradigm for Creating a National Strategy" (project code: EP/R045518/1).