Predicting Electricity Imbalance Prices and Volumes: Capabilities and Opportunities

Abstract

Electricity imbalance pricing provides the ultimate incentive for generators and suppliers to contract with one another ahead of time and deliver against their obligations. As delivery time approaches, traders must judge whether to trade-out a position or settle it in the balancing market at the as-yet-unknown imbalance price. Forecasting the imbalance price (and related volumes) is therefore a necessity in short-term markets. However, this topic has received surprisingly little attention in the academic literature despite clear need by practitioners. Furthermore, the emergence of algorithmic trading demands automated forecasting and decision-making, with those best able to extract predictive information from available data gaining a competitive advantage. Here we present the case for developing imbalance price forecasting methods and provide motivating examples from the Great Britain’s balancing market, demonstrating forecast skill and value.

1. Introduction

Electricity is a special commodity. Supply must meet demand continuously, and delivery is constrained by the physics of complex transmission and distribution networks. Liberalisation of the energy sector has given rise to a range of market places where electricity can be bought and sold ahead of time, and to settle differences between what was contracted and actually delivered. The settlement process involves resolving participants’ so-called imbalance volumes at a price that reflects the cost incurred by the system operator when correcting these imbalances. Imbalance pricing is designed to incentivise participants to minimise their imbalance volumes, i.e., to generate or consume what they have contracted.

Market participants continuously manage their portfolio’s potential imbalance volume (e.g., due to uncertain renewable energy production or customer demand) and therefore exposure to the imbalance price. Traders must take a view on what the imbalance price may be—hence the need for imbalance price forecasts. It is therefore remarkable that the academic literature on this problem is sparse, especially when compared to the extensive literature on forecasting day-ahead market prices, reviewed in [1,2].

Furthermore, what little literature considers imbalance price forecasting does so alongside the development of trading strategies that require a view to be taken on future imbalance prices, but this is treated as secondary to trading: Approaches to modelling future imbalance prices include assuming that participants anticipate the annual and quarterly average imbalance prices [3], applying the average relationship to day-ahead price [4], and prediction using time series models [5]. The need for imbalance price forecasts may even be removed if it is assumed that trading parties balance their portfolio, as in [6], though this results in sub-optimal trading decisions. In general, these studies have focused on producing forecasts ‘good enough’ to demonstrate the efficacy of trading strategies and not on producing high-quality forecasts. Few examples of more sophisticated approaches exist, which include Bottieau et al. [7] who showed the benefit of machine learning methods over standard benchmarks with a feature set including forward prices plus recent and predicted generation and load. However, Bottieau et al. only considered one-step-ahead prediction of imbalance volume and not price.

Indeed, imbalance volume forecasting has received comparatively more attention than price, with early works including [8], which identifies imbalance volume as a particularly challenging but important quantity to forecast, and proposed a Neural Network for this task. Very-short-term imbalance volume forecasting to support power system operators is considered in [9,10], the latter having a particular focus on probabilistic forecasting and interpretability. Both consider lagged imbalance volumes, calendar/temporal features, wholesale prices, as well as forecasts/schedules and lagged observations of renewables, conventional generation, and interconnector flows as explanatory variables. However, potentially valuable information is not included, such as intraday market order books and executed trades, system operator actions, and the generation stack, for example.

Others have exploited regional characteristics, combining imbalance volume forecasts with reserve costs [11]. Studies that focus on day-ahead forecasting and trading strategies have concluded that climatological or simple models perform well and are very difficult to improve on, whereas within-day it is possible to produce more accurate forecasts. However, to the best of our knowledge, there are no example of imbalance price or volume forecasting in the academic literature that leverage even close to the full range of relevant data now readily available to market participants.

The transition to renewables is central to the decarbonisation of society, and as such, integrating them into electricity markets, including balancing markets, is critical [12]. The stochastic nature of wind and solar power in particular continue to motivate research to improve forecasting of power production and associated quantities, including electricity prices. This short communication presents the key features of imbalance price forecasting, and current challenges faced by forecasters in this area, which has received surprisingly little attention in the literature to date. We demonstrate the value of both day-ahead and intraday imbalance price forecasts in a simple trading case study based on a small conventional generator participating in the GB balancing market with a flexible asset.

2. Imbalance Price Forecasting

In this section, imbalance price forecasts are examined on two important time scales. First, a forecast made in advance of gate closure for day-ahead markets, and second in advance of gate closure for intraday markets (i.e., up until just before the start of the delivery period).

2.1. Day-Ahead

At the day-ahead stage, the accuracy of imbalance price/imbalance volume forecasts is generally low, as plant schedules are often only determined once the day-ahead price is known. However, basic probabilistic forecasts of system length can still be used to inform offer strategies on the day-ahead market. For instance, the probability of the system being long or short may be forecast with greater skill than relying on historic frequencies of occurrence, and the eventual difference between the day-ahead price and the imbalance price may be predicted more accurately than the imbalance price alone.

We illustrate this here with a simple model based on kernel density estimation and linear regression for the GB imbalance price. Separate density forecasts for long and short systems are combined according to the forecast probability of the system being long or short. Inputs to these models are based on public data available in real time. For comparison we also consider two benchmarks: (1) taking the day-ahead auction price as the imbalance price forecast and (2) climatology, where the forecast is the day-ahead price plus the average difference between day-ahead and imbalance price.

2.2. Intraday

As the point of delivery approaches, an increasing volume of data relevant to the future imbalance price becomes available. This typically includes generator and interconnector schedules; interventions by the system operator; intraday market order books and executed trades; weather forecasts/observations; and recent imbalance volumes and prices. The data may arrive continuously (as actions are taken by system operators and traders), at a fixed point before delivery, or not at all (due to a data outage). Each set of variables has all the potential to improve the imbalance forecast: knowing which generators are online provide an indication of the marginal cost of regulation energy; intraday market data reveals participants’ views on future imbalance prices; and plant or network outages impact the demand for regulation energy. This paper presents the performance of a proprietary forecasting tool developed by the authors in collaboration with market participants for the GB imbalance price from zero to five hours (ten 30-min settlement periods) ahead, leveraging this data. Figure 1 shows an example forecast as a fan plot.

2.3. Summary of Forecast Performance

To give an indication skill, the median from each half hourly forecast is evaluated using Mean Absolute Error (MAE). Data from January 2017 to October 2020 is used for training and validation with October 2020 to December 2020 used for out-of-sample testing. Results from the test period are reported here. In Figure 2, the skill of intraday forecasts is shown for lead-times from 0 to 5 h ahead. The intraday forecasts show an improvement of 5–40% over the climatological benchmark, with far superior performance for lead-times less than 3 h ahead. The day-ahead forecasts also shows a smaller by significant improvement of 3% over the benchmark.

3. Forecast Value

In this section we demonstrate the value of imbalance price forecasts in trading strategies for flexible assets exposed to the balancing market that are sufficiently small to be price-takers in this market. The forecasts described above are used to drive three trading strategies, all of which are compared in terms of revenue and risk. Two risk appetites are considered—risk neutral and risk averse—where in the latter revenue is maximised subject to a risk constraint. To simplify the analysis, non-overlapping intraday forecasts from the model described above are used to model a generic flexible asset (which could be the residual flexibility of a larger portfolio) as follows:

1.

The asset can deliver total energy $e_A$ between 0 and 1 MWh in each half hour period;

2.

A predetermined fraction of energy can be delivered to Power-to-X or $e_X$ with no losses. The remainder $e_A-e_X$ is delivered to the balancing market and remunerated at the imbalance price ${\pi }_S$;

3.

Power-to-X is remunerated at a fix price ${\pi }_X$, we consider values in the range 20 to 50 £/MWh;

4.

The asset is always available (implied by 1).

Since there is only one independent random variable, the imbalance price ${\pi }_S$, the uncertain revenue

$$R=(e_A-e_X){\pi }_S+e_X{\pi }_X$$

(1)

can be controlled by choosing the volume $e_X$ of energy to divert into Power-to-X. The assumption that the Power-to-X price ${\pi }_X$ is fixed is reasonable, as today’s options (heat, gas, self-consumption) have either a fixed or slowly varying price. In principle, this trading strategy could be implemented for any controllable power plant, provided that they are a price-taker in the balancing market.

We evaluate the outcome of each trading strategy in terms of mean revenue and Value-at-Risk (VaR), which is defined as the negative of a quantile of the revenue outcome series $\mathrm{VaR}=-F_R^{-1}\left(0.01\right)$ where $F_R(·)$ is the cumulative distribution function of revenue. The motivation for this metric is that a low value corresponds to low risk position, and vice versa. For example, a VaR of −£20 means that there is a 1% chance that the revenue will be below £20 in any period.

3.1. Case 1: Point Forecast, Zero/Max

Here deterministic forecasts of the imbalance forecast ${\widehat{\pi }}_S={\widehat{F}}_{{\pi }_S}^{-1}\left(0.5\right)$ are used, i.e., the median forecast where ${\widehat{F}}_{{\pi }_S}(·)$ is the predictive distribution function of the imbalance price, in a revenue maximization trading strategy. A simple min/max strategy is employed

$$e_X=\left\{\begin{array}{c}0\mathrm{if}{\widehat{\pi }}_S>{\pi }_X\hfill \\ 1\mathrm{if}{\widehat{\pi }}_S\le {\pi }_X\hfill \end{array}\right.$$

(2)

where all of the available energy is diverted either to the balancing market or to Power-to-X, depending on which is remunerated at the higher price.

3.2. Case 2: Probabilistic, Simple Hedge

In this case energy is hedged according to the probability of the imbalance price being lower than the Power-to-X price. The probability of the imbalance price being lower than the Power-to-X value may be calculated simply as ${\widehat{F}}_{{\pi }_S}\left({\pi }_X\right)$. The Power-to-X volume is then chosen to be $e_X={\widehat{F}}_{{\pi }_S}\left({\pi }_X\right)\times e_A$. For example, if there is a 50% probability of the imbalance price being lower than the Power-to-X price, half of the available volume is diverted to Power-to-X.

This basic heuristic results in power being diverted to the balancing market and Power-to-X in similar quantities when their respective prices are similar and/or the imbalance price is particularly uncertain, or in favour of the most profitable option when the prices are likely to be different. It therefore offers an effective hedge against large swings in cash flow and reduces risk.

3.3. Case 3: Probabilistic, Risk-Constrained

Finally, probabilistic forecasts are used to explicitly control risk in terms of VaR, by maximising revenue subject to a risk constraint. A probabilistic forecast of revenue is generated, substituting the predictive distribution of the imbalance price ${\widehat{F}}_{{\pi }_S}(·)$ into the expression for revenue, Equation (1). Since revenue in this framework is a linearly increasing function of imbalance price, a simple change of variables approach is used to convert the density forecasts of the imbalance price to revenue, for a chosen volume of generation $e_X$ diverted to Power-to-X. From this forecast, the revenue at risk can be constrained by simply selecting $e_X$ such that there is a 1% chance of observing revenue below a user-defined threshold value in any period. In the following example, we constrain the 1% VaR to $10-{\pi }_X$ £/MWh.

3.4. Results

The out-of-sample performance of these strategies has been evaluated in terms of revenue and risk (1% VaR) and are compared for the three cases and three imbalance price forecasts in Figure 3 using the same dataset as in Section 2.3. Across all cases, better qualify forecasts result in higher revenue and lower risk, and probabilistic information improves risk management, particularly in the risk-constrained case. In the first risk-neutral case, Case 1, which utilises deterministic forecasts, switching from the day-ahead to the more accurate intraday forecast increases revenue by up to 2%. Similar revenues are achieved with reduced risk exposure in Case 2 by utilising probabilistic forecasts and a simple hedging strategy. Finally, when trading is constrained by a risk threshold, switching from the day-ahead to intraday probabilistic forecasts improves revenue by up to 5% while maintaining desired risk exposure.

4. Discussion and Conclusions

Previous studies have implied or commented directly that imbalance price forecasting is impractical [3], however more recent works [7,11] and our own analysis have hopefully dispelled this myth: imbalance prices and volumes can be forecast skilfully and lend value to market participants, particularly intraday and through probabilistic forecasting. Here, we have reported the performance of simple day-ahead and advanced intraday imbalance price forecasts, and have shown that these add value to risk-neutral and risk-constrained trading strategies, increasing revenue by up to 2% and 5%, respectively.

Furthermore, the growing complexity of electricity markets, and rise of automatic trading and control of distributed and multi-vector resources, have increased the need for technical solutions to a task previously performed manually by traders [13]. Many organisations are forecasting the imbalance prices and volumes and would likely benefit from academic contributions adapting state-of-the-art statistical and machine learning methods for this task. Similarly, researchers publishing on trading strategies should be using state-of-the-art imbalance forecasts in their analysis to be representative of best practice.

Regional variations in market design and data availability impact imbalance forecasting and have not been explored here. Price formation and the availability of data in real-time can vary significantly between regions and therefore demand specific treatment. Regulation may also explicitly forbid speculation based on imbalance prices, as in Germany, and instil a reluctance to engage in sophisticated analytics in this space. As a result, it is possible that different approaches will favour different markets depending on the speed, volume, complexity, and quality of data available to forecasters, but there are also likely to be commonalities and knowledge worth sharing.

In summary, the volume and availability of data relevant to imbalance forecasting is growing and presents an opportunity for innovative forecasters to discover new sources of predictability, gain a competitive edge, and ultimately improve the efficiency of electricity markets. We hope that this topic will attract the attention and efforts of more researchers, and that future research will leverage the full range of diverse data that are available to inform imbalance price and volume forecasts.