--- title: "VidyutVanika21: An Autonomous Intelligent Broker for Smart-grids" authors: - "Sanjay Chandlekar" - "Bala Suraj Pedasingu" - "Easwar Subramanian" - "Sanjay Bhat" - "Praveen Paruchuri" - "Sujit Gujar" doi: "10.24963/ijcai.2022/23" url: "https://ijcai.org/proceedings/2022/23" pdf: "https://ijcai.org/proceedings/2022/0023.pdf" venue: "Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence" year: 2022 pages: "158-164" track: "Main Track" keywords: - "Agent-based and Multi-agent Systems: Economic Paradigms, Auctions and Market-Based Systems" - "Agent-based and Multi-agent Systems: Applications" --- # VidyutVanika21: An Autonomous Intelligent Broker for Smart-grids Sanjay Chandlekar, Bala Suraj Pedasingu, Easwar Subramanian, Sanjay Bhat, Praveen Paruchuri, Sujit Gujar *Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence*, pages 158-164. DOI: [10.24963/ijcai.2022/23](https://doi.org/10.24963/ijcai.2022/23) ## Abstract An autonomous broker that liaises between retail customers and power-generating companies (GenCos) is essential for the smart grid ecosystem. The efficiency brought in by such brokers to the smart grid setup can be studied through a well-developed simulation environment. In this paper, we describe the design of one such energy broker called VidyutVanika21 (VV21) and analyze its performance using a simulation platform called PowerTAC (PowerTrading Agent Competition). Specifically, we discuss the retail (VV21–RM) and wholesale market (VV21–WM) modules of VV21 that help the broker achieve high net profits in a competitive setup. Supported by game-theoretic analysis, the VV21–RM designs tariff contracts that a) maintain a balanced portfolio of different types of customers; b) sustain an appropriate level of market share, and c) introduce surcharges on customers to reduce energy usage during peak demand times. The VV21–WM aims to reduce the cost of procurement by following the supply curve of the GenCo to identify its lowest ask for a particular auction which is then used to generate suitable bids. We further demonstrate the efficacy of the retail and wholesale strategies of VV21 in PowerTAC 2021 finals and through several controlled experiments. ## Full text ## 1 Introduction A smart grid is an electricity network used to supply energy to consumers via two-way digital communication. It allows for monitoring, analysis, control and communication between participants to improve efficiency, transparency and reliability of the energy supply chain [Techopedia.com, 2021]. A smart grid environment consists of a retail market involving various types of customers, a wholesale market involving power generating companies, distribution utility and energy brokers who interact between retail and wholesale markets. There are potential opportunities to improve smart grid operations, especially when renewable energy generation is getting bigger by the day, to efficiently manage fluctuating supplydemand scenarios and grid imbalance situations. ∗Authors contributed equally to this work To investigate potential solutions to improve smart grids, the Power Trading Agent Competition (PowerTAC) [Ketter et al., 2020] provides an efficient simulation of real-world smart grids, with energy brokers playing a pivotal role. In PowerTAC, a broker serves its customer base in the retail market by purchasing the required energy from the wholesale market. The broker handles supply-demand imbalances via balancing market. The primary goal of the broker is to maximize its profit by operating in these three markets. Hence, PowerTAC organizes an annual tournament where multiple teams design autonomous broker agents to compete against each other across several games in multiple player configurations. Brokers get revenue from subscriptions in the retail market and incur costs for procuring energy in the wholesale market. Often, brokers also pay penalties for contribution to grid imbalance and peak time usages. In PowerTAC, peak time usage charges are relatively high and are called as capacity transaction charges. The brokers aim to develop retail strategies to maximize revenue and lessen penalties; moreover, to design wholesale strategies to reduce procurement costs. Brokers in previous PowerTAC tournaments used gradientbased optimization strategies [Urieli and Stone, 2016], MDP based strategies [Ghosh et al., 2020], genetic algorithm based approaches [ ¨Ozdemir and Unland, 2018] to offer tariffs in the retail market. The seemingly optimal class of strategies of monopolizing the retail market have been found to suffer from high capacity transactions charges. To overcome this, broker agent TUC TAC proposed an interesting strategy aimed at acquiring half the retail market share [Orfanoudakis et al., 2021] in PowerTAC 2020 tournament. To the best of our knowledge, most retail strategies proposed thus far have been generic and are not effectively specialized for different player configurations and therefore fail to maintain performance across different player configurations. Thus, a careful design of the broker's retail strategy is needed. Similarly, in the wholesale market, past brokers used MDP-based approaches [Urieli and Stone, 2014; Ghosh et al., 2020] or heuristics [Chowdhury et al., 2018] to minimize energy procurement costs without paying much attention to the bidding behavior of sellers. This often results in higher costs to brokers, and sub-optimal bidding patterns by brokers will push energy purchases to the balancing market where the costs are typically high. Hence, a bidding strategy that considers the bidding behavior of sellers is warranted. Our Approach and Contributions: To address the shortcomings discussed above, we present our broker VidyutVanika21 (VV21), which is the winner of the PowerTAC 2021 tournament. The retail strategy of VV21 selects the market share of its retail portfolio by ingeniously following a gametheoretic analysis specialized to player configuration. In addition, tariffs contain surcharges for peak hours, resulting in customers shifting load to off-peak hours apart from providing a cushion to handle peak demand charges. The wholesale strategy aims to place bids by identifying the lowest ask of the largest seller by following its supply curve. This ask price is used as an upper bound to procure as much power from other sellers present in the wholesale market at cheaper prices. Further, to reduce imbalance costs, VV21 attempts to procure all the required energy from the wholesale market, thus enabling effective energy procurement. To summarise: - We present the design of an autonomous energy broker, VV21, the winner of the PowerTAC 2021 tournament. - We describe the design of VV21-RM, the retail module of VV21, which uses game-theoretic analysis to offer optimal tariffs with the aim of maximizing net revenue. - We detail the working of VV21-WM, the wholesale module of VV21, that recognizes the bidding pattern of sellers and uses it for procuring power at cheaper prices. - We analyze the performance of VV21 in the PowerTAC 2021 tournament and showcase its efficacy through several controlled offline experiments. ## 2 RelatedWork Research work on smart grids has focused on various subproblems in energy management with approaches backed by theory and methods driven by data analysis. For example, the work of [Chung et al., 2020] considers the problem of load scheduling by modeling it as a stochastic game and uses deep RL to find the Nash equilibrium (NE) of the game. In [Jain and Gujar, 2020], the authors proposed bandit-based mechanisms to achieve demand response in the smart grids. However, such strategies are impractical in PowerTAC as brokers need to respond in real-time. Among the strategies deployed in past tournaments, MDPbased strategies are popular and have been used both in retail and wholesale market operations. For example, Reddy & Veloso, COLD Energy, and VidyutVanika18 modeled the decision process in the retail market as an MDP to generate tariff contracts [Reddy and Veloso, 2011; Serrano et al., 2017; Ghosh et al., 2019]. Whereas TacTex'13 used a gradientbased optimization method to generate tariffs and AgentUDE17 implemented a genetic algorithm-based tariff strategy [Urieli and Stone, 2016; ¨Ozdemir and Unland, 2018]. In the wholesale market, TacTex'13, VidyutVanika18, and AstonTAC employed an MDP-based strategy [Urieli and Stone, 2014; Ghosh et al., 2020; Kuate et al., 2013]. AgentUDE17 employed adaptive Q-learning, while SPOT and TUC TAC adopted MCTS to design their bidding strategies [Chowdhury et al., 2018; Orfanoudakis et al., 2021]. Moreover, a DDPG-based bidding strategy is employed by [Chandlekar et al., 2022], which works in continuous state and action spaces. However, none of the previous works present an equilibriumbased strategy for the retail market; furthermore, none of the past brokers have attempted to learn seller's behavior in the wholesale market. Our novelty lies in designing a gametheory-inspired strategy in the retail market and adopting a bidding strategy based on the supply curve of the prominent seller in the wholesale market. ## 3 The PowerTAC Environment PowerTAC is a simulation platform that replicates crucial elements of a smart-grid ecosystem like retail, wholesale, and balancing markets along with a distribution utility (DU). The retail market contains state-of-the-art customer models that replicate a smart grid's real-world users like consumers, producers, and storage users. Consumers vary from households, offices, and villages to hospitals. Producers are modeled to generate energy from renewable sources like solar or wind. Storage customers use batteries or electric vehicles to store energy which can be supplied to the grid on a need basis. The wholesale market consists of a major power generating company (GenCo) which sells energy to buyers using day-ahead periodic double auctions (PDAs). The balancing market manages the real-time balance of supply and demand using economic incentives. The distribution utility in PowerTAC owns and manages the distribution infrastructure. A detailed description of the simulator is provided in [Ketter et al., 2020]. A crucial feature of PowerTAC is that the simulator provides a way for several autonomous energy brokers to compete to make profits by operating in retail, wholesale, and balancing markets. An annual PowerTAC tournament is organized wherein multiple teams deploy autonomous energy brokers that compete with each other in all three markets. The tournament consists of several games organized between brokers in different player configurations and varying weather conditions. The duration of a game is around 60 simulation days, and the simulation time is discretized into time slots corresponding to every hour of the day. During the game, a broker agent aims to develop a subscriber base in the retail market by offering competitive tariffs, which could be fixed price (FPT), tiered, or time-of-use (ToU). Brokers satisfy the energy requirement of their subscriber base by buying power in the wholesale market through day-ahead PDAs. A broker can participate in 24 double auctions corresponding to consumption slots that could be one to twenty-four hours away at any simulation time. Through subscriptions to storage customers, brokers can also manage grid imbalances. The simulation environment broadcasts various information, including wholesale market clearing prices, anonymized uncleared bids and asks, retail market briefs consisting of new and revoked tariffs and weather information to help brokers in decision-making. Additionally, before starting a game, a twoweek bootstrap simulation exercise is arranged in which the DU acts as the default broker. The output of this boot period, which includes identities and consumption data of all retail customers, average clearing prices in the wholesale market, weather and calendar information, is made known to the brokers. Equipped with all this data, a broker agent acts in various markets to attain a healthy cash position in each game compared to its peers. The final cash position of all brokers is aggregated across games to arrive at a normalized score which is used to determine the winner of the tournament. Figure 1: Architecture of VV21 ![Image](https://ijcai.org/proceedings/2022/fulltext/23/image_000000_f951cc6ab1329ee0a2a8789de7940a7f16bf61ea8a491e4fccb1ad181c3a5568.png) ## 4 Design and Overview of VV21 The system design of VV21 as shown in Fig. 1 consists of two main modules, namely, the retail (or tariff) module called VV21-RM and the wholesale module denoted as VV21WM. VV21-RM handles tariff publication and revocation in the retail market, while VV21-WM is responsible for energy procurement in the wholesale market. Additionally, VV21 maintains repositories to store data received from the game server about the weather, wholesale, balancing and retail market, which is then used in the decision-making process. The strategy used by VV21-RM in the retail market is inspired by game theory and aims to maintain an appropriate level of market share in the retail market. Furthermore, the retail module also targets to achieve a balanced portfolio consisting of customers of different types (consumption, production and storage). Tariffs usually contain surcharges during peak hours to further mitigate the effect of capacity transaction charges. For effective bidding in the wholesale market, VV21-WM follows the supply curve of the GenCo to recognize its lowest ask for a particular auction. It uses this lowest ask as an upper bound on its limit price and tries to procure as much power from other sellers present in the wholesale market at cheaper prices than what is demanded by GenCo. Finally, our design of VV21-RM and VV21WM helps in handling balancing market activities as well. Precisely, VV21-WM procures a large share of energy from the wholesale market, thus reducing purchases from the balancing market, which results in low balancing costs. During high-demand scenarios, when GenCo cannot fulfill all of the market demand, VV21-RM supplies energy via producers and storage customers in its portfolio to make up for the market deficit and, in the process, generates revenue. ## 4.1 Retail Module of VV21 (VV21-RM) As shown in Fig. 1, VV21-RM has three sub-modules, namely, Tariff Enhancer (TE), Tariff Designer (TD) and Tariff Health Checker (THC). The TE sub-module calculates mean tariff rates to maintain retail market share within predefined bounds. The TD sub-module generates weekly ToU tariffs based on the mean rates suggested by TE with surcharges during peak hours. The THC sub-module periodically checks for active tariffs' profitability and subscription rates and removes loss-making and under-subscribed tariffs. Algorithm 1 shows the working of VV21-RM. Below We detail TE and TD. Tariff Enhancer (TE): TE is in charge of maintaining an appropriate level of market share in a game by improving VV21's active tariffs by considering all active tariffs present in the market. Recall that aiming to monopolize the retail market will result in high penalties for peak time usage, a fact that is recognized by brokers like TUC TAC [Orfanoudakis et al., 2021] in past tournaments. Therefore, a broker needs to carefully choose market share bounds and publish tariffs to maintain market share within the chosen bounds. More precisely, we maintain three hyperparameters HB, LB and MB which correspond to higher, lower and middle bounds. TE aims to maintain retail market share between MB and HB, which helps it attain higher profits through sufficient retail market revenue without being penalized heavily for peak time usage. Based on the game-theoretical analysis, we choose values for these hyperparameters for each player configuration. We include the details of these calculations in Sec. 5. Algorithm 2 shows the working of TE, which suggests average tariff rates for all customer types (referred to as powerType). First, TE computes the cumulative market share of all its active tariffs using the EstimateMarketShare() method for each customer type. The EstimateMarketShare() method relies on the tariff repository shown in Fig. 1 to calculate market share of each of its active tariff. Thereafter, if VV21's current market share is higher than HB, to reduce market share, TE removes VV21's cheapest tariff and suggests costlier tariff rates by calling the method GenerateCostlierTariffs(). Similarly, if the market share is lower than LB, TE calls GenerateCheaperTariffs() that suggests cheaper tariff rates to help increase the market share. In the case when the market share is between LB and MB, TE calls FineTuneTariffs() to finetune the VV21's cheapest tariff by making it slightly cheaper to achieve the desired market share bounds. TE checks the viability of new tariff rates to ensure that the mean tariff rates are higher than the average energy procurement costs before calling the TD sub-module to generate weekly ToU tariffs. Tariff Designer (TD): TD is responsible for designing a weekly ToU tariff based on the average input price (avgPrice) received from TE as shown in Algorithm 3. First, TD calls DefineWeeklyTariffPattern() method that generates a binary weekly tariff pattern indicating peak and non-peak hours. This pattern is designed by analyzing the historical net market demand values retrieved from the logs of past PowerTAC tournaments. For example, we found that morning and evening hours of a day have high demand, and these hours are marked as peak hours along with several other peak hours identified from the analysis. Then TD calls DefineSurplusMultipliers() method to define surplus multipliers si for each of the 168 hours in a week. These multipliers are greater than 1 for peak hours and are set to 1 for non-peak hours. Specifically, for peak hours, the value of si would depend on the magnitude of the peak as observed from market demand data. ## Algorithm 1 VV21-RM - 1: activeTariffs ←GenerateInitialTariffs() - 2: if assesmentInterval then - 3: revokedTariffs ← TariffHealthChecker(activeTariffs) - 4: RevokeTariffsFromMarket(revokedTariffs) - 5: activeTariffs ←activeTariffs \ revokedTariffs - 6: newTariffs ←TariffEnhancer(activeTariffs) - 7: PublishTariffs(newTariffs) - 8: activeTariffs ←activeTariffs ∪ newTariffs - 9: end if ## Algorithm 2 TariffEnhancer(activeTariffs) ``` 1: for powerType in offeredPowerTypes do 2: tariffs ← getTariffs(activeTariffs, powerType) 3: marketShare←EstimateMarketShare(tariffs) 4: if marketShare ≥ HB then 5: RevokeCheapestTariff() 6: newTariffs ← GenerateCostlierTariffs(tariffs) 7: else if marketShare ≤ LB then 8: newTariffs ← GenerateCheaperTariffs(tariffs) 9: else if marketShare ≤MB then 10: newTariffs ← FineTuneTariffs(tariffs) 11: end if 12: newToUTariffs = [] 13: for tariff in newTariffs do 14: if isTariffViable(tariff) then 15: ToUTariff ← TariffDesigner(tariff, powerType) 16: newToUTariffs ← newToUTariffs ∪ ToUTariff 17: end if 18: end for 19: end for 20: return newToUTariffs ``` ## Algorithm 3 TariffDesigner(avgPrice, powerType) - 1: pattern←DefineWeeklyTariffPattern(). - 2: s[] ←DefineSurplusMultipliers(pattern) - 3: find normRate : ∑168 i=1 si∗normRate 168 = avgPrice - 4: rate[i] ← si ∗ normRate, for i ∈ {1, 2, ..., 168} - 5: ToUTariff ←CreateTariff(rate, powerType) - 6: return ToUTariff We observed that such tariffs help in mitigating the effect of capacity transaction penalties, and this is shown in Sec. 7. ## 4.2 Wholesale Module of VV21 (VV21-WM) The wholesale module of VV21 (VV21-WM), as explained in Algorithm 4, aims to procure the energy needed for its subscriber base at the lowest possible cost. To this end, first, VV21-WM uses the demand predictor (DP) sub-module to estimate the expected demand (QT ) of its subscriber base for a delivery time slot T. After that, the bid generator (BG) sub-module participates in periodic double auctions (PDA) by placing bids of the form (p, q) where q is the amount of energy sought to be procured at no more than price p. Recall that in the PowerTAC setting, for a delivery time slot T, a broker can start procuring energy from time slot T - 24 at hourly intervals. Hence, BG has 24 opportunities to buy the required energy. This also implies that at any bidding time slot t, BG participates simultaneously in 24 auctions corresponding to 24 future (delivery) time slots denoted by (t + 1, t + 2, · · · , t + 24). BG can submit multiple bids for a single auction. The supply curve follower (SCF) sub-module aids BG in determining suitable limit prices for bids that are to be placed in an auction. Any shortfall or excess in energy procurement by BG after exhausting all 24 opportunities is adjusted in the balancing market, which is typically expensive for the broker. We describe SCF and BG sub-modules here and use the PowerTAC sample broker's predictor as a DP sub-module. Note that, although the wholesale module is described from a buyer's perspective, it is straightforward to design a similar behavior for brokers who wish to sell energy using production customers in their portfolio. Supply Curve Follower (SCF): The SCF sub-module helps in determining suitable limit prices for generating bids. Recall that, in the PowerTAC wholesale market, a GenCo acts as the leading supplier. The asks placed by this GenCo follow a quadratic price curve. The core idea behind SCF is to follow this quadratic price curve through uncleared asks made available to the broker, which helps in identifying the least ask that the GenCo could exhort for a particular auction. Since the wholesale auctions are PDAs, each auction can be marked using the tuple (t, T) where t ∈ {T -24, · · · , T -1} are the bidding times, and T is the delivery time. VV21-WM does not place any bids at the first opportunity (i.e., at t = T -24). After that, at any bidding time t = T-k for k ∈ {23, · · · , 1}, the least price that the GenCo could ask is identified using the uncleared asks of the previous auction corresponding to bidding time t = T - k - 1. This least ask price would also depend on the outstanding energy requirement of VV21 and its competitors for the delivery time T. When the uncleared asks are empty, the least ask price is taken as the maximum of the market-clearing prices of past auctions where the difference between the delivery time and bidding time is T - t. Algorithm 5 outlines the details of SCF. Bid Generator (BG): The BG sub-module is responsible for generating multiple bids (M bids in total) on behalf of VV21-WM using the outputs of DP and SCF. Based on empirical observation, BG recognizes that it is possible to procure energy at a cheaper price than the least ask of the GenCo as there are other smaller sellers (like opponent brokers) who sell their excess procurement in the wholesale market. However, it is difficult to predict the exact time and ask prices of such sales. Hence, when the bidding time t is far from the delivery time T , BG generates bids with limit prices less than the predicted least ask of the GenCo (using hyper parameters αf , βf ), hoping to capitalize on asks from smaller sellers. Closer to the delivery slot, BG generates bids with limit prices closer or greater than the least ask of the GenCo (using hyper parameters αc, βc). This ensures procurement of the outstanding energy from GenCo and thereby avoids going to the balancing market where energy costs are typically high. Algorithm 6 describes the working of BG. ## 5 Optimizing VV21-RMParameters In this section, we present a game-theoretical analysis to determine the optimal market share bounds determined by parameters HB, MB and LB used in Algorithm 2 specific to player configuration. We show the analysis for the 5-player configuration and follow the same methodology for other configurations as well. We model the PowerTAC game as ## Algorithm 4 VV21-WM(currentTime) | 1: | mktUsage[] = NetDemandPredictor(currentTime) | |------|----------------------------------------------------------| | 2: | askPrices[] = SupplyCurveFollower(currentTime, mktUsage) | | 3: | bids[][] = BidGenerator(currentTime, askPrices) | | 4: | for hour in [1,2...23]) do | | 5: | futureTime = currentTime + hour | | 6: | FetchM bids from bids[futureTime] | | 7: | Submit M bids to Auction(currentTime,futureTime) | | 8: | end for | ## Algorithm 5 SupplyCurveFollower(currentTime, netDemand) ``` 1: for hour in [1 . . . 23] do 2: futureTime = currentTime + hour 3: Get UnclearedAsks of Auction(currentTime-1,futureTime) 4: if UnclearedAsks is empty then 5: Get ClearedPrices of Auction(t,t+hour) ∀ t