A market framework for grid balancing support through imbalances trading

To correct grid imbalances and avoid grid failures, the transmission system operator (TSO) deploys balancing reserves and settles these imbalances by penalizing the market actors that caused them. In several countries, it is forbidden to influence the grid imbalances in order to let the TSO retain full control of grid regulation. In this paper, we argue that this approach is not optimal as market actors that trade imbalances under the supervision of the TSO can help balancing the grid more efficiently. For instance, some systems such as solar farms cannot participate in the standard balancing market but do have economic incentives to help regulate the grid by trading with imbalances. Based on this argument, we propose a new market framework where any market actor is allowed to trade with imbalances. We show that, using the new market mechanism, the TSO can keep full control of the grid balance while decreasing the balancing cost. This is of primary importance as: 1) novel approaches to reduce grid imbalances are needed as, while renewable sources are generally not used for grid balancing, the increasing integration of renewable energy sources creates higher imbalances. 2) While long-term storage of energy is key in the energy transition, it needs to become an attractive investment to ensure its widespread use; as we show, the proposed market can guarantee that. Based on a real case study, we show that the new market can provide 10 – 20% of the total balancing energy needed and reduce the balancing costs.


Introduction
As one of the many actions to reduce carbon emissions and mitigate the effects of climate change, the so-called energy transition [1] aims at dramatically increasing the share of renewable energy sources (RESs) in the energy mix. However, before achieving the desirable goal of nearly 100% RES generation, there are several problems that need to be addressed [2]. In particular, electricity still cannot be stored efficiently and economically over long periods of time and electricity networks require constant balancing between generation and consumption. However, due to the uncertainty in the supply and demand, a perfect balance between generation and consumption is hardly possible and grid imbalances are unavoidable. To prevent grid instabilities, these imbalances have to be corrected in real time by the transmission system operator (TSO). In this context, due to the weather dependence of RES generation, electricity generation becomes more uncertain and grid imbalances become larger as the integration of RESs increases. Consequently, as we approach the 100% RES generation target, the grid becomes harder to balance and control [3].
To mitigate this issue, some RES systems could potentially contribute to grid balancing but they are not being used for this purpose at the moment largely due to the current rules applied to system balancing. Examples of such systems include solar photovoltaic installations, storage systems such as seasonal storage, or even-in some countries-wind farms. In this paper, we argue that allowing them to trade in the imbalance settlement mechanism, i.e. the mechanism used by the TSO to financially settle imbalances with the participants that caused them, is a potential solution for those systems to assist the TSO in reducing grid imbalances.

Electricity markets
As previously stated, generation and consumption of electricity must be equal at all times in order to preserve a balanced grid [4,5]. Particularly, grid imbalances affect the grid frequency [6] and large deviations from the stable frequency can lead to grid instabilities and rolling blackouts [4].
To help obtain a balanced grid, wholesale electricity markets have a very specific structure. In particular, two specific features of the European electricity markets, self-dispatch and balancing responsibility, are relevant for grid balancing. Self-dispatch refers to the fact that market participants make their own decisions regarding the dispatch of their generators 1 but are obliged to submit their projected generation and consumption schedules ahead of time. Balancing responsibility refers to the fact that all market parties carry a balancing responsibility: they are financially responsible for deviations from their schedules as these deviations create grid imbalances. In order to avoid deviations from their notified schedules, market participants can trade in different markets that are mainly distinguished by the time of execution. Particularly, as market participants obtain more accurate information about their actual generation and consumption, they can adjust their schedules by trading in markets with execution times closer to real time. In this context, besides bilateral trading, European actors have several organized marketplaces at their disposal: • Forward market: electricity is traded weeks or months in advance.
• Day-ahead market: electricity is traded up to one day in advance.
• Intraday market: electricity is traded one day ahead of delivery to 1 h or some minutes before delivery time. In addition, electricity is traded continuously, in hourly or quarterly auctions, or a mix thereof [7].
For these three markets, contracts between buyers and sellers are established in a market exchange and supervised by a market operator. Moreover, the last two markets are also sometimes referred to as spot markets.
In theory, by having all these markets with different gate closure times, market participants are provided with several opportunities to correct their imbalances. Particularly, due to generation and consumption uncertainty, it is nearly impossible for a market actor to know in advance how much electricity the given actor should trade, e.g. the electricity traded in a forward market (months in advance) is rarely the electricity that the agents would like to trade in real time. By having these different markets, actors can minimize their economic risks by trading the bulk of their energy in more stable markets (the ones with earlier execution times), and then continuously adjusting their trades to make sure that the sum of the traded electricity in all the markets matches their submitted schedule.
Despite this market structure, due to unplanned unit outages and the uncertainty in the supply and demand of electricity, imbalances still occur as market agents rarely consume or generate what they have traded [6]. To avoid frequency deviations and grid failures, the TSO corrects the imbalances via the balancing market [5,8]. In this market, participating actors offer their balancing capacity (for potential activation) to the TSO months to days ahead. Then, in real time, the TSO activates the required reserves to correct positive (generation exceeding consumption) and negative (consumption exceeding generation) grid imbalances. As a final step, the costs of balancing are covered in the imbalance settlement by financially penalizing the actors that caused the imbalances. 2 In this settlement, market participants are charged for the imbalance they produced within a defined time interval, which is known as the imbalance settlement period (ISP) and in most European markets equals to 15 min. The unit price paid for having an imbalanced position is called the imbalance price.
It is important to note that the participation in the balancing market is currently fairly restricted. This is to a large extent explained by the main distinguishing feature between the balancing market and the other markets: in order to participate, potential balancing providers are subject to a prequalification procedure. In this procedure, depending on the balancing product, i.e. up or down-regulation and primary, secondary, or tertiary reserve, balancing providers are required to satisfy certain technical requirements involving the speed and duration of activation, frequency of activation within a contracting period, or ramp-up and ramp-down rates (among others). As the prequalification requirements are fairly restrictive, only a handful of technologies are able to fulfill these criteria [8].
A schematic representation of the electricity markets is displayed in Fig. 1. The figure represents the time frame for the decision making process in each of the markets. In particular, the figure includes a nonlinearly scaled timeline that spans from months ahead to real time. Over the timeline, the different markets are represented by grey boxes and their gate opening and closure times defined by the position of the vertical borders of the boxes.
In some countries, e.g. Germany or France [9], it is discouraged or even forbidden to actively influence and trade with grid imbalances. Instead, market agents are expected to trade honestly 3 in the markets available before delivery time and they are expected to only generate unexpected imbalances. This rule, despite granting the TSO full control of the grid balance, is economically suboptimal as the economic incentives for imbalance trading of some market agents are in fact aligned with the balancing duties of the TSO. In particular, as during periods of positive imbalances imbalance prices are low, some market agents might be willing to buy cheap electricity during those periods; by doing so, they would indirectly help in reducing the imbalance [9]. Similarly, as during periods of negative imbalances prices are high, some market agents could be willing to reduce their consumption or increase their generation in order to increase their profit. In both cases, not only would the market agents improve their profits, but the imbalances would be reduced and the imbalance price would decrease as the TSO would no longer have to activate more expensive balancing reserves.
Based on this argument, although some countries forbid causing imbalances for trading purposes, some others, e.g. The Netherlands [10], Belgium, or the UK [9], allow participation in this type of trading. Nevertheless, despite this consent, the TSOs in those countries have no mechanism in place to ensure that the imbalances created during trading do not harm the grid stability, e.g. they still face the risk of market actors potentially creating an imbalance that would aggravate the grid stability.
For the remainder of this paper, we will refer to the imbalance settlement mechanism as the imbalance market. This is done because, since we will propose a market framework for trading with imbalances, we assume that the imbalance settlement mechanism is simply a type of market. It is important to note that this term refers to the imbalance settlement and not to the balancing market.

Seasonal storage systems
The availability of a reliable and profitable long-term energy storage is crucial for ensuring the success of the energy transition. In particular, since the penetration of solar and wind energy is expected to reach very high levels by 2030 (70-80% in some countries) [11], the uncertainty in energy supply is expected to increase. Similarly, as the generation of renewable sources is season dependent [11], e.g. the production of solar power is larger in summer than in winter, the generation of electricity is expected to be characterized by very strong seasonal fluctuations [12]. In this context, seasonal storage solutions [11], which can store energy across several months, are crucial to reduce the uncertainty and seasonal fluctuations [12].
While there are two seasonal storage technologies, i.e. hydrogen storage and synthetic natural gas storage [11], with capabilities to store 1 As opposed to systems with central dispatch (e.g. in the U.S.) where the system operator makes dispatch decisions. 2 The exact calculation of the imbalance price differs across the EU countries. 3 By honest trading we refer to trading based on their forecast and electricity needs, instead of trading to intentionally create an imbalance for their own benefit.
electricity and feed it back into the grid, in their current state they are economically non-viable. First, both technologies [13] are expensive technologies and in early stages of development [13][14][15]. In addition, synthetic fuels have a very low energy efficiency due to conversion losses [13]. An arguably better technology to store energy over seasons is seasonal thermal energy storage systems (STESSs) [16]. Although these systems store electricity as heat and cannot transform it back to electricity, they have the advantage of being less expensive than electrical energy storage [12] and they represent a more reliable and mature technology. Moreover, as 45% of the commercial and domestic energy usage corresponds to cooling and heating demand [17], STESSs have the potential to become a key element in the energy transition as they have a large market to commercialize the stored heat.

Motivation
One of the main pillars of the energy transition is the large integration of RESs, a change of paradigm that, as mentioned before, is expected to increase grid imbalances [18]. In this context, as traditional power plants are taken off the grid, it becomes clear that RES systems need to contribute to grid balance if the grid is to remain stable. However, in the current balancing market, many RES technologies are not allowed to participate due stringent prequalification requirements and/or due to the procurement time frames. The prequalification requirements, which include aspects such as the speed and duration of activation, are largely based on the technical capabilities of traditional large-scale generators and are very hard or nearly impossible to fulfill for RESs. 4 In addition, since balancing services are often procured days or weeks in advance, RESs cannot properly quantify their balancing potential due to their inherent generation uncertainty.
A second issue involving the energy transition and the balancing market is the need for new mechanisms to ensure sufficient economic incentives for RESs. Particularly, as the RES support policies are being phased out, it is necessary to identify new value streams to sustain investment in RESs and to advance the energy transition. In this context, allowing RES systems to participate in the balancing services, not only might ensure a more reliable grid operation, but also would provide new sources of revenues for RESs.
These two issues are also relevant for STESSs. These systems, despite their potential for grid balancing support, cannot fully help to reduce grid imbalances due to the current electricity market rules. In particular, as STESSs cannot transform the stored heat back to electricity, they either cannot participate in the balancing market or they are limited to down-regulation 5 only. As a result, despite their potential to correct grid imbalances and adapt to the seasonal fluctuations of RESs, their capability to help is limited. Besides this problem, their economic profit is also limited by the current market rules. Particularly, as their business model relies on trading in markets with large price volatility, i.e. using their flexibility in order to buy electricity during the periods of very low prices, they would ideally trade in the balancing market [19].
Besides the described problems, the current balancing markets allow very high prices. Particularly, the high technical requirements make the number of balancing service providers very limited, i.e. they usually range between a few to a few dozen depending on the country and balancing product (see e.g. Refs. [20]). In this context, it becomes easier for actors to exert market power and to bid strategically, leading to situations where balancing prices may reach thousands of euros. As an example, if we consider the balancing market in Germany in 2018, the average balancing price was approximately 43 €/MWh, but the prices oscillated between 2013 and -1873 €/MWh. Similarly, if we look at the same balancing market but in 2017, the deviations are even larger: while the average price was 33 €/MWh, the prices oscillated between 24455 and -2558 €/MWh. In short, due to the structure of the balancing market and the existing incentives for RESs, we can identify three research areas where significant improvements can be made to help the energy transition: • The existing structure in the balancing market prevents RESs from participating on it. This is a problem as RESs create imbalances and more RESs are integrated due to the energy transition. • The lack of RESs and STESSs participation in the balancing market reduces the much needed economic incentives for RESs and STESSs. • The limited number of agents in the current balancing market leads to very high spikes in balancing prices.

Contributions of the paper
To tackle the described problems, we propose a new market framework for providing balancing services through the imbalance settlement mechanism, a.k.a. imbalance market. The goal of the new framework is to incorporate systems that cannot participate in the traditional balancing market into the portfolio of balancing providers. The core idea of the new framework is to allow trading in the imbalance market under direct control of the TSO so that the grid stability is never compromised. In detail, market actors send their bids to the TSO stating their availability to deviate from their schedule. Then, the TSO automatically activates the available units in real time using a process that is similar to the one employed for standard balancing products. As the deviations are controlled by the TSO, participants cannot worsen the frequency regulation.
Besides increasing the number of balancing services, the proposed framework also has the advantage of ensuring competitive trading practices and avoiding the gaming practices of the balancing market. Particularly, as actors do not bid prices, their ability to behave strategically and to drive market prices is very limited; similarly, as any actor can participate in the new market, the market power of a single actor  4 For a comprehensive discussion of the barriers to RESs in the balancing market please see Ref. [8]. 5 Down-regulation might not even be profitable for STESSs if they have to bid too much in advance as markets with execution times closer to real time might offer more volatile prices.
becomes smaller.
To study the proposed market framework, we consider a real STESS as a case study. This selection is done because, in addition to not being able to participate in the traditional balancing market, STESSs are arguably one of the key elements to obtain a smooth and reliable grid operation [12]. Using the proposed framework and optimally controlling a STESS we show that: 1) the proposed market framework allows actors such as STESSs to efficiently assist the TSO in stabilizing the grid; 2) STESSs can help reduce grid imbalances while increasing their profits; 3) the TSO can reduce the balancing cost without losing control over the grid regulation. Despite using STESSs as a case study, it is important to note that the proposed market framework is very general and is valid for any type of technology with the same property: not being able to participate on the balancing market, but having economic incentives to trade in the imbalance settlement to reduce grid imbalances.
In summary, the contribution of this paper is twofold: 1. We propose a new market framework to provide balancing services using systems that cannot participate in the traditional balancing market. The new framework has the advantages of lowering the balancing costs and procure more systems for balancing the grid. The new market grants further incentives for RESs and STESSs by allowing them participation in grid balancing. 2. We demonstrate that the novel framework can help reduce balancing costs (and thus balancing price spikes) without compromising grid stability. Despite the case study focus on STESSs, the proposed approach and market modification is universally applicable and valid for any type of technology.
The paper is organized as follows: Section 3 introduces the proposed market framework. Section 4 defines different case studies and presents the obtained results. Finally, Section 5 discusses and analyzes the results and Section 6 concludes the paper.

Literature survey
In the literature, the discussion about using energy markets for integrating RESs and new sources of flexibility, e.g. energy storage systems, has been centered on three different topics: 1. The adjustment of the existing balancing markets to facilitate the integration of a broader scope of technologies and participants. 2. The creation of new marketplaces to reveal the economic value of flexibility for different electricity stakeholders. 3. The generation of appropriate economic incentives for RESs in the current market conditions.
In general, the first two points are concerned with the global market perspective, i.e. how can we modify the market frameworks to integrate more RESs. By contrast, the last point is concerned with the participant perspective, i.e. what can we offer to market actors to make their participation more enticing.

Adjustment of existing balancing markets
As the EU Member States are in the process of progressively integrate balancing energy markets, optimal design of balancing markets for RES integration has attracted a lot of attention. In particular, the prequalification requirements in the balancing market (see Section 1.1) differ considerably from TSO to TSO and are generally rather restrictive; consequently, only a handful of technologies and providers can fulfill them [8]. Besides prequalification requirements, balancing markets have other barriers that prevent participation of multiple technologies: long procurement timeframes [21], limitations for aggregation [22], or pay-as-bid pricing [23]. As a result, studying and investigating modifications to the existing balancing markets has become a wide area of research.
In [24], researchers study the effect of the settlement rule (marginal or pay-as-bid) on the balancing market; they conclude that the settlement rule is the key factor for efficient market design. In Ref. [8], a framework for analyzing balancing market design is proposed and the main European balancing market guidelines are analyzed. Based on this framework, it is shown that the independence of the balancing energy market and the balancing capacity market must be addressed first to ensure effective integration of RESs; furthermore, it is also shown that the settlement rule needs to be adjusted last in order to prevent the exploitation of market power. Further evidence of the the value of the standalone balancing energy market has been demonstrated in Ref. [25]. Similarly, further evidence of the importance of strategic bidding has also been analyzed using the example of the German balancing market [23,26]. Besides analyzing market rules and policies, the value of market actors to grid balancing through short-term trading has also been advocated in Ref. [18].

New marketplaces
A different line of research argues for the creation of new marketplaces so that different technologies, e.g. RESs and storage systems, can provide their flexibility. Many of the existing proposals focus on socalled flexibility markets where local, often aggregated small-scale, providers deliver flexibility to distribution and/or transmission system operators [27]. Distribution system operators (DSO) are usually seen as the main beneficiaries of such local markets, using them to avoid grid reinforcement and ensure grid stability. A comprehensive review of local flexibility markets and related challenges is presented in Ref. [28]. Similarly, an overview of emerging initiative and pilot projects on flexibility markets is provided in Refs. [29].
Besides being mostly DSO-centered, another disadvantage of these approaches is that they often require a substantial change to the existing market design and have thus far not been clearly addressed in the national or the EU regulatory frameworks [30].

Generating economic incentives
Besides market design, generating incentives for market agents is key to ensure a wide integration of RESs. In particular, for RESs and flexible energy systems to be profitable, it is necessary to ensure several value streams such as optimization of own consumption, short-term market participation, or balancing service provision [31]. Hence, analyzing and developing new incentives for RESs and flexible systems has been widely researched [32][33][34]. An important result to these studies is that, although aggregators are often seen as key enablers [35], they face numerous country-specific barriers [36].

Filling the gap
As described in detail in Sections 1.3 and 1.4, in this paper we combine the participant (in this case a STESS operator) perspective and the market perspectives to provide a solution that both ensures a sufficient incentive for participation in system support and formulates a market design proposal that addresses the market efficiency issues studied in previous research. In particular, to the best of our knowledge, our approach is the first to provide an efficient marketplace that goes beyond the local flexibility markets. Moreover, the proposed framework not only provides new incentives to RESs and flexible systems, i.e. addresses the participant perspective, but also ensures that grid regulation is improved.

Method
In this section, we introduce the proposed market framework for trading in the imbalance market.

Market framework
The core idea of the new market framework is to allow trading in the imbalance market, coupled with a communication channel with the TSO to facilitate coordination and to prevent situations in which actors would negatively affect grid stability. In detail, the market framework can be divided in three sequential steps: 1. Before the beginning of each ISP, the actors communicates to the TSO how much power they are willing to provide for upward and/or downward regulation. Particularly, they send a bid to the TSO indicating the availability of each of their units to create a positive or negative imbalance together with the maximum volume of that imbalance. It is important to note that these bids are not traditional bids as they do not include a price. 2. Then, at any time during the ISP, the TSO activates any unit whose bid helps to regulate the grid. To keep full control, the activation of the units is automatically done as with traditional balancing products: the TSO sends a direct signal to the unit and the unit automatically creates the imbalance. 3. At the end, each actor pays or receives the imbalance price multiplied by the net imbalance volume created, or an economic penalty if they failed to provide the requested regulation.
It is important to note that the timeline to submit bids is not restricted by the market framework. Instead, it is a decision variable that the TSO has to define given two requirements: 1) it should be possible to submit bids after the intraday gate closure time so actors have an updated schedule; 2) all bids must be submitted before the beginning of the ISP so that the TSO knows the available balancing energy in the imbalance market. The timeline of the different electricity markets including the proposed imbalance market is displayed in Fig. 2. As can be seen, bidding in the new market can be done after the intraday market closes, but all the bids are submitted before the beginning of the ISP. Then, in real time, the TSO activates the participant actors similar to how it does for the regular balancing service providers.
A simplified schematic view of the framework is depicted in Fig. 3, which represents a possible interaction between the TSO and an actor with three units. As can be seen, before the ISP starts, the actor sends a bid. Then, during the ISP, the TSO automatically activates units 1 and 2 (unit 1 is activated before unit 2) but deactivates unit 2 before the end of the ISP. Finally, the actor gets paid the imbalance price.
The advantages of this framework are fourfold: • As the TSO automatically activates the actors' units to create imbalances, it does not lose control of the grid balance. • For market actors that cannot participate in the balancing market, e.
g. seasonal storage systems or solar photovoltaic farms, this framework allows them to contribute to grid regulation and to increase their profit. • The cost of balancing can only decrease as the actors only get paid the imbalance price. In particular, if they are activated, in the worst case the price does not change; in the best case, the price decreases as their activation prevented a more expensive balancing technology from being used. • The TSO can count on more balancing flexibility without losing control over the grid and use that flexibility to reduce balancing costs.

The added value to the existing markets
While some arguments could be made against the proposed market, the new framework complements and adds specific value to the existing markets. In this section, we discuss the added value of the proposed market as well as the possible arguments against it.

The added value to the balancing market
A potential argument against the proposed framework is that if there already exists a liquid and working balancing market, the proposed market framework might be unnecessary. However, in the current balancing market, there are two issues that prevent some actors to balance the grid and that arguably makes the procurement of balancing actors economically inefficient [8,23]: • Prequalification: As explained in Section 1.3, prequalification requirements leave out of the balancing market actors such as thermal seasonal storage systems, solar photovoltaic farms, or in some countries even wind turbines. Considering the increasing integration of these systems into the electricity grid, it is important to find a way to also integrate them into the balancing portfolio of the TSO. • Time frame: In most balancing markets, participants are expected to send their bids hours or days in advance. Due to the uncertainty of RES generation, this deadline constrains the participation of RES systems as they are unable to accurately quantify their regulatory power in advance. Moreover, because of the same deadline, systems that cannot generate power are limited to down-regulation. As a result, it can be argued that the current use of some systems might be economically inefficient.
With the proposed market framework, these issues would be solved. As prequalification would not be needed, more systems could take part on balancing the grid. This is of primary importance as, while the integration of RES systems is increasing, these systems are not being used for balancing the grid despite their correlation with larger imbalances. Similarly, as bids are submitted closer to real time, the uncertainty of RES system decreases and they are able to provide balancing services. Moreover, since actors balance the grid by deviating from their schedule, they are not limited to down-regulate the grid as they can purchase energy in other markets, e.g. the day-ahead market, and use that energy to provide up-regulation.
Despite these benefits, it could be argued that the time frame problem will be solved in the near future as European countries must adjust their balancing markets in the next few years to incorporate free bids, i. e. bids submitted by actors whose capacity was not reserved ahead of time, and bids close to real time [37]. However, there are two issues with this argument: • Bidders are still required to pass the prequalification process, which limits the use of important RES systems for grid balancing. By contrast, the proposed imbalance market framework allows integrating many participants into the balancing portfolio of the TSO and has the potential of lowering balancing costs. • The proposed market framework has an advantage over the balancing market with free bids: it avoids gaming practices as market actors do not know the imbalance price. In detail, with free bids, market actors can submit strategic bids and drive the balancing price if they have market power. By contrast, in the proposed market framework, market actors only bid the imbalance volume but not the price; instead, they have to accept the unknown imbalance price and cannot drive the balancing cost. As a result, the proposed market framework has the potential to lower the balancing cost and ensure competitive trading practices.

The added value to the intraday market
Another possible argument against an imbalance market could be that STESSs and other technologies could use the intraday market instead of the imbalance market for reducing imbalances. Particularly, as actors usually participate in the intraday market as their last resort to correct imbalances, RES technologies could already trade on the intraday market and help other actors to reduce their imbalances. However, this approach is not sufficient if it is intended to provide the TSO with a greater flexibility in managing grid imbalances: 1. The energy traded by a STESS in the intraday market does not ensure that the grid imbalances are reduced. Particularly, a given actor might have a positive imbalance and sell some energy to the STESS in the intraday market to correct his imbalance, but then the grid imbalance might be negative. 6 In that scenario, it is clear that it would be more beneficial if the STESS could instead create a individual positive imbalance (i.e. sell electricity in the imbalance market), instead of buying electricity in the intraday market. 2. In terms of the economic profits of RESs, it could more beneficial to trade with imbalances. If we compare the intraday market to the imbalance market, there is a price spread that favors imbalance trading. For example, in Germany, the imbalance price during upregulation is on average 20 euros higher that the price in the intraday market [18]. In those scenarios, it is clear that for RESs it is more beneficial to sell energy in the imbalance market rather than in the intraday market. Similarly in Germany, the imbalance price during down-regulation is on average 37 euros cheaper that the price in the intraday market [18]. In those situations, storage systems like STESSs would buy energy in the imbalance market rather than in the intraday market.
In short, the proposed market framework can be seen as an additional tool in the balancing portfolio of the TSO that complements the existing balancing market. This new tool can lower the balancing costs, allow larger participation of market actors in balancing services, and ensure competitive trading practices.

Control algorithm
The proposed market framework is based on the idea that while there are systems that cannot participate in the traditional balancing market, these systems may have economic incentives that are aligned with the balancing responsibilities of the TSO. Therefore, any system that participates in the proposed market framework needs a control algorithm that optimizes its profits, i.e. that exploits its economic incentives. In particular, the system requires an algorithm that can decide, before the beginning of each ISP, whether up or down-regulation is economically beneficial for the system during that ISP.
In this paper, as the framework is analyzed in the context of STESSs, we consider the optimal control algorithm for STESSs proposed in Ref. [19]. Further details of this algorithm are provided in the next section.

Case study
As a case study, we consider a real SSTES from the company Ecovat [38] trading in the proposed market. Based on experimental results we show that, while the STESS increases its profits by trading in the new market, the proposed market improves grid stability as it reduces grid imbalances.

Goal
The aim of the study is to quantify how much balancing energy the TSO can save by using the new market and a single actor, i.e. a STESS. To perform the study, we have built a simulation environment of the STESS and of the different markets. For the markets, we have built a simulator that replicates the day-ahead market, the imbalance market, and the balancing actions of the TSO. For the STESS, we have considered its   3. Representation of the proposed market framework for a single ISP and a single actor with 3 units. The arrows cross the timeline at the time intervals during which the action described takes place. In this example, the TSO activates unit 1 until the end of the ISP, and unit 2 for a short period of time. 6 The grid imbalance is the cumulative imbalance of all the actors. Thus, it is possible for the imbalance of a given actor to have a different direction than the cumulative grid imbalance. dynamic model as described in Refs. [39].
As a secondary goal, we also study how profitable it is for the STESS to participate in the proposed market. In particular, we compare the profits of the STESS when trading only in the day-ahead market to its profits when trading in both the day-ahead and the imbalance market.

Real STESS
The considered STESS is a large subterranean thermal stratified storage vessel with the ability to store heat for seasonal periods and to supply heat demand to a cluster of buildings. The system is divided into different segments or heat buffers that can be charged and discharged separately. The size of the system and the number of heat buffers depends on the heat demand and the use case of each particular STESS. In this case study, we consider a STESS that is being built in Arnhem (The Netherlands) to cover the heat demand of 500 houses. The system contains 20000 m 3 of water, can store 1.3 GWh of energy, can supply a yearly heat demand of 2.8 GWh, and has a maximum electrical power of 1 MW. Fig. 4 provides a schematic representation of one of these vessels and illustrates one of them during the construction phase. For further details on these systems we refer to Refs. [19,39,40].

Control algorithm
As indicated in Section 3.3, any system that participates in the proposed market framework needs a control algorithm that can decide, before the beginning of each ISP, whether up or down-regulation is economically beneficial for the system during that ISP. For this case study, we consider a modified version of the optimal control algorithm for STESSs proposed in Refs. [19].
The original algorithm consists of two collaborative reinforcement learning (RL) [41] agents that trade in the day-ahead and in the imbalance market. In detail, given a stochastic heat demand that the STESS needs to satisfy, the first agent buys electricity in the day-ahead market. Then, in the imbalance market, the second agent chooses between selling the energy purchased in the day-ahead market or buying more energy. If the second agent sells the day-ahead energy and the overall grid imbalance is negative, the STESS helps the TSO to up-regulate the grid. Similarly, if the STESS buys energy and the grid imbalance is positive, the STESS helps the TSO to down-regulate the grid. The goal of the algorithm is to maximize the profits of the STESS while satisfying the heat demand. A scheme of the control algorithm is depicted in Fig. 5. It is important to note that in this scheme the RL agent simply takes a decision, i.e. creates an imbalance, without knowing the actual imbalance price or imbalance volume. As such, it does not know if its decision helps regulate the grid.
In this paper, in order to fit the control algorithm to the new market framework, the control algorithm has two modifications: • Instead of directly taking the desired action in the imbalance market (desired imbalance position), the agent sends a bid to the TSO before the beginning of the ISP. Then, the TSO choses to activate that actor if needed. • Instead of taking the desired action for an entire ISP, the agent is activated by the TSO only for the time within the ISP when the action is really needed (the TSO might down and up-regulate within the same ISP).
The new control algorithm is depicted in Fig. 6. For the sake of simplicity, further details about the control algorithm are not provided as the focus of the paper is the proposed market framework and the control algorithm is one of many trading strategies to interact with the new market. However, the details of the algorithm can be found in Refs. [19].
As a final remark it is important to note that, to fully maximize the profits, it could be argued that trading in the intraday and forward markets should also be considered. However, the proposed framework would work exactly the same if the STESS would trade in the forward, day-ahead, and intraday markets. The only difference in that case would be that the STESS would up-regulate using the net power purchased in the three markets. As the goal of the paper is to show the potential of the new market framework for grid regulation, a simpler control strategy is thus employed.

Experimental setup
In this section, we describe the specific experimental setup used to analyze and study the proposed imbalance market. In particular, we explain how the the proposed market is simulated and which data and software is used.

Data
To build a simulator of the new market framework, real data regarding day-ahead market prices, imbalance prices, the activation of balancing products by the TSO, and the heat-demand that the STESS satisfies are required. For the prices, we consider the day-ahead and imbalance prices in The Netherlands (as the STESS is located there). For the behavior of the TSO, we consider the activated volume of secondary balancing energy reserve, a.k.a. automatic frequency restoration reserve, in The Netherlands in 1-min intervals. The prices are collected using the ENTSO-E transparency platform [42] and the activated balancing products using the TenneT transparency platform [43]. To simulate the behavior of the STESS, we consider the real heat demand that the STESS supplies: the heat demand of a cluster of 500 buildings (with a yearly-average energy consumption of 2.8 GWh) 7 using the same time resolution as the imbalance market, i.e. 1 min.
Regarding the periods considered, the data are collected for the years 2015-2017. Then, the data of 2015 and 2016 are used as training data for the control algorithm, and the data of 2017 are used as out-of-sample data to evaluate the performance of the proposed market framework.

Market framework simulator
To simulate the market framework, we assume that the STESS is a price taker. Then, we allow the STESS to purchase electricity in the dayahead market through regular market bidding. In particular, the RL agent of the day-ahead market builds bidding curves and the day-ahead market is cleared assuming historical prices; then, the STESS receives an energy allocation based on its bidding curve. An example of such bidding curve together with the associated market price is depicted in Fig. 7. As can be seen, considering the bidding curve and the historical market price, the STESS receives 1 MWh.
For the imbalance market, we allow the STESS to submit imbalance bids up to 3 min before the start of the ISP. Then, we simulate the clearing of the imbalance market as follows: for each of the 1-min intervals, we activate the STESS if the submitted bid reduces the original activated volume. Thereby, this simulation provides the yearly balancing energy that the new market and a small sized actor, i.e. a STESS, can potentially save the TSO. A schematic representation of this bidding process is depicted in Fig. 8. As can be seen, the STESS bids 3 min before starting the ISP; then, for each of the 1-min intervals within the ISP the TSO activates the STESS if needed, i.e. if the STESS bid reduces the volume that the TSO has to activate.
This discretization in 1-min intervals obviously has an impact on the ancillary services provided. In particular, it could be argued that using 1min intervals limits the activation of the STESS as the STESS can only be activated at the beginning of each of these 1-min intervals. Ideally, one would instead consider the exact time when individually products are activated and replace these products with the balancing energy provided by the STESS. However, the data regarding individual activated 7 Obtained from one of our research partners.
products are not available, and the data with the smallest time resolution that are available are the cumulatively activated products in 1-min intervals. Nonetheless, this limitation only underestimates the full potential of the new market framework as the STESS cannot be activated as often as it would be in real life.

Variability in the experimental setup
Due to the nature of the STESS and its controller, the obtained results are not deterministic: • The performance of the RL agent depends on the initial conditions of the training algorithm. Thus, the results will vary depending on how the RL control algorithm is estimated. • The maximum balancing energy that the STESS can provide depends on the initial energy in the STESS.
To account for this variability, the experiment is repeated 100 times by taking into account these two sources of uncertainty. In particular, the experiment is repeated for 10 different RL agents that are trained using 10 different random initializations, and for 10 STESSs that have 10 different initial states.

Market uncertainty
It is important to note that, although the study is based on historical data, we perform a stochastic study based on 100 different simulations where different sources of uncertainty are considered: • The study is performed for different control algorithms, i.e. RL agents, to reflect the fact that market actors can be controlled with different strategies. • The initial state of the STESS is not fixed but it is initialized with different values to model the uncertainty surrounding the operational conditions of market actors, i.e. every market actor, although based on the same technology, might consider a different working regime.   5. Representation of the original optimal control algorithm for trading in the day-ahead and balancing markets proposed in Ref. [19].
• At every time step, the STESS takes two decisions, i.e. the bid of the day-ahead market bid and the consecutive bid in the imbalance market. When the first decisions is taken, the agent has uncertainty information: at the time of the decision, the imbalances and prices are unknown. This fact combined with different initial states and different control algorithms makes every simulation run completely different. • Finally, we also consider uncertainty in the market prices and the grid imbalances. In particular, these values are uncertain as the control algorithm never uses them for estimation.
As described in Section 4.4.3, to have a good understanding on these stochastic variables, the simulation is perform 100 times and the results, i.e. costs, profits, balancing energy, are provided not only in terms of the mean but also the standard deviation.

Overview representation
To provide a better overview of the experimental setup, Fig. 9 provides a flowchart of the experiment and the associated analysis. As can be seen, the core of the experiment is a daily simulation of the two markets, i.e. the day-ahead market and the proposed imbalance market. This simulation is repeated for 365 days, i.e. a year, and each year simulation is repeated 100 times for different initializations of the RL agent and the STESS state. Finally, the performance statistics of the experiments are extracted.
For the sake of completion, we also include in Table 1 a summary of the key parameters in the case study:

Software
The entire market simulator is developed in python and the model of the STESS is implemented using Casadi [44]. As in Ref. [19], the optimal control algorithm is implemented as a RL agent using the fitted Q-iteration algorithm [45] and the Xgboost [46] library.

Results
The obtained results are listed in Tables 2-5. In detail, Table 2 shows   Table 3 lists the amount of time that the STESS is used for balancing the grid. From these two tables, the following observations can be made: • The STESS is able to provide 674 MWh of balancing energy for upregulation and 1426 MWh for down-regulation. • While the energy for up-regulation and down-regulation can vary by 24 % depending on the control algorithm and the initial state, the total amount of balancing energy only varies by 18 %. A possible reason for this is that, depending on the initial state and/or the RL agent, either up-regulation or down-regulation could be more profitable. • The STESS is activated to balance the grid 2099 h, which is equivalent of being activated 24 % of the year.
Besides analyzing the total amount of balancing energy provided by the STESS, we also study the relative contribution of that energy to the total energy activated by the TSO. This metric is provided in Table 4 for up-regulation and down-regulation. From this table, it can be observed how the STESS can supply the TSO between 0.6 % and 0.25 % of the total balancing energy needed.
Finally, it is also important to analyze how profitable it is for the STESS to participate in the new market framework. For that, Table 5 lists the economic costs of the STESS when it trades optimally on the dayahead market, 8 i.e. the standard case for these type of systems, and when it does so but in both the day-ahead and the imbalance markets. It can be observed how, by trading in the new market framework, the STESS can reduce its costs by 40 % and the variability on these savings is minimal, i.e. only 0.5 %.

Market performance for different market sizes
To further show that the algorithm works for different market conditions, we also consider the case where the imbalance market is halved and doubled. For that, we make the assumption that a market of double size will have twice the size of the imbalances, and a market of half size will have the same imbalances but halved. Then, we repeat the same experiments, i.e. training a control algorithm and evaluating the market framework, but with the imbalance data from the Netherlands doubled and halved.
The results when doubling the market are listed in Table 6. As it could be expected, the relative contribution of a single STESS decreases as the overall balancing energy need increases, i.e. while the balancing energy needed doubles the maximum power the STESS can trade does       not change. However, despite its lower contribution in relative terms, the STESS is able to provide a similar amount of balancing energy. In particular, while the market size doubles, the size of the STESS remains constant and this the amount of energy it can provide does not vary. A similar analysis is provided in Table 7 for the case of halving the market. Unlike before, the relative contribution increases as the balancing energy needed is halved but the maximum power the STESS can trade does not change. Nonetheless, despite the larger relative contribution, the STESS is not able to provide as much balancing energy as before. In particular, the absolute energy contribution of the STESS decreases as the STESS finds less opportunities for balancing the grid due to the smaller imbalances.
In both cases, despite the differences in the relative and absolute contributions to balancing the grid, it is clear that the proposed market structure works as expected as the STESS is able to contribute to grid balancing while optimizing its profits.

Market performance for different simulation parameters
To validate that the market framework works independently of the selected simulation parameters and that the uncertainty study is correct, we repeat the experiments but considering different simulation parameters. In particular, we upscale and downscale the three main simulation parameters: the number of initial states in the STESS, the number of different control algorithms, and the market size. As the results of varying the market size have been already presented in Section 4.6, we simply present here the results of varying the other two.
As with the market size, we first consider the case of upscaling the parameters. For that, we perform the same experiment but doubling the number of initial states and the number of control algorithms, i.e. we consider 20 initial states and 20 control algorithms. The results of this experiment are listed in Table 8. Comparing these results with the results listed in Table 2, it is clear that upscaling the simulation parameters have little effect on the final performance of the market, i.e. the market framework works as expected independently of the simulation parameters.
A similar analysis is provided in Table 9 for the case of downscaling the simulation parameters. For the sake of simplicity, we apply the same scaling factor of 2, i.e. we halve the simulating parameters. Similarly to the upscaling case, if we compare these results with the results listed in Table 2, it is clear that downscaling the simulation parameters have little effect on the final performance, i.e. the market framework behaves independently of the simulation parameters.

Discussion
In this section, to discuss the obtained results and analyze the benefits of the proposed market framework, we first discuss the results in the context of the market, and then in the context of the energy transition.

Potential of the imbalance market
The TSO clearly benefits from the new market framework as it obtains extra balancing energy that can only reduce the overall balancing costs. In particular, even from a single relatively small STESS, the TSO is able to extract 2000 MWh of balancing energy in a given year. While this amount represents only 0.35 % of the total balancing energy needed, the considered STESS is a relatively small system. In particular, the current STESS only serves the heat demand of a small neighborhood, i.e. 500 houses. If we consider 25-50 of these systems (something very reasonable in a country like The Netherlands), STESSs alone could potentially reduce the current balancing demand by 10-20 %.
Based on these results, it becomes clear that the proposed market framework has the potential to provide a large share of the energy required to keep the grid balanced. Particularly, besides STESSs, the proposed market framework is designed to integrate more systems into this new portfolio of balancing resources. In this context, while accurately estimating the potential contribution of the new framework is nearly impossible (it would require a simulation including all possible systems that would participate in this market), we can consider the relative contributions of STESSs and make some qualitative inferences. In detail, STESSs are not very large systems, i.e. the considered STESS had a maximum power of 1 MW. Therefore, if STESSs can already provide 10-20 % of the total balancing energy needed, it is clear that the market framework has the potential to revolutionize the balancing market, provide a large share of the balancing energy needed, and reduce the total balancing cost.
It is important to note that all these benefits come without any additional operational costs or risks: even though the new market is based on imbalance trading, the TSO has full control over the grid stability. Moreover, as the market actors do not bid prices but only volumes, the new market can only drive the balancing costs down.

Benefits for the energy transition
Besides benefiting the TSO and its balancing duties, the new market framework promotes the use and expansion of long-term energy storage solutions. In particular, by participating in this new market, STESSs can reduce their operational costs by 40%. As this represents saving nearly half of the operational cost, the importance of the new market framework in promoting a widespread use of STESS is evident. This is of great importance since, as mentioned in the introduction, the availability of reliable and profitable long-term energy storage is crucial for ensuring the success of the energy transition. As the new market framework encourages the use of these type of storage systems, it is a potentially very valuable tool to further advance the energy transition.   Table 9 Balancing energy used by the TSO during 2017 and percent contribution of the STESS to that amount when downscaling by a factor of 2 the main simulation parameters.

Conclusions
In this paper, a new market framework to aid the transmission system operator (TSO) to balance the grid has been proposed. The framework is based on an adaptation of the current imbalance settlement mechanism to explicitly allow trading with imbalances under the TSO supervision. The goal of the new framework is to allow renewable energy sources (RESs) and other systems, e.g. seasonal storage systems, to proactively contribute to grid regulation. In detail, driven by the goals set by the EU to achieve the energy transition, the integration of RESs into the energy mix is increasing. In addition, an economically viable longterm energy storage solution is necessary to provide the flexibility needed to ensure the success of the energy transition. Yet, the technical prequalification requirements and design characteristics of the traditional balancing market prevent RESs and seasonal storage systems from participating in it. This limitation poses two problems for the energy transition: 1) as RESs are a common source of grid imbalances, they cannot be effectively integrated into the energy mix without allowing them to contribute to system stability; 2) as seasonal storage systems are flexible devices whose business case is to exploit price differences in volatile markets, their lack of access to the balancing market limits their profitability and their economic viability.
The proposed market framework solves these issues by integrating RESs and seasonal storage systems into the portfolio of balancing resources. Besides permitting these technologies to balance the grid, the proposed imbalance market has three main advantages: 1) it increases the number of balancing resources available to the TSO at no additional operational cost; 2) it ensures competitive trading practices and avoids the gaming practices of the balancing market; 3) it allows the actions of market actors to be controlled by the TSO so that grid stability is never compromised.
To demonstrate and quantify the benefits of the proposed market, a real seasonal thermal energy storage system (STESS) trading in the new market was considered as a case study. Based on the obtained results we show that: • The proposed market framework can provide a large share of the balancing energy needed and reduce the total balancing cost of the TSO. Particularly, by using 25-50 small-scale STESSs, the new market can provide 10-20 % of the total balancing energy needed. • Even though the new market is based on trading with imbalances, the grid stability does not worsen. • STESSs can reduce their operational costs by 40% by participating in the imbalance market. This is key to ensure their economic viability and, in turn, to guarantee the widespread use of seasonal storage solutions needed in the energy transition.
As future research, we will investigate the performance of the new market framework for the scenario where day-ahead pre-dispatch and forecasting have significant errors. This analysis will be important to understand whether the new market framework can help mitigate the effect of large day-ahead forecasting errors. In addition, we will evaluate the performance of the new market using other RES systems. Moreover, to address the empirical limitation of this study, in future work we will also study the mathematical properties of the market using a game theoretic approach in order to derive a set of immutable claims regarding the market properties. In addition, we will further assess the proposed market using the data from other European countries and we will consider more complex trading strategies to quantify the maximum potential of the proposed market.

Declaration of competing interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.