Analyzing Electricity Markets with Increasing Penetration of Large-Scale Renewable Power Generation

Abstract

Global electricity markets are undergoing a rapid transformation in their energy mix to meet commitments towards sustainable electric grids. This change in energy mix engenders significant challenges, specifically concerning the management of non-dispatchable energy resources. System and market operators are required to meet power system security and reliability requirements whilst providing electricity at competitive prices. An overview of electricity markets is provided in this paper with a critical appraisal of each market’s ability to manage the large-scale energy mix transition. This paper provides a commentary on the distinct features of electricity market models implemented around the world and highlights the barriers within these market models that are hindering the energy mix transition. Various researchers and policymakers are proposing solutions and market reforms for the smooth transitioning of the energy mix. This paper presents a systematic review of the proposed solutions in the literature and critiques the effectiveness and ease of implementation of the reviewed solutions. Research gaps and future research directions are indicated to promote further exploration towards the effective integration of large-scale renewable energy technologies.

1. Introduction

Environmental concerns with regard to carbon emissions from conventional fossil fuel-based generating resources have been the chief driving force for the rapid transformation of electricity markets. Technological advancements, and government strategies and initiatives to tackle global warming have brought variable renewable energy (VRE) into the limelight. In the present day, nations across the globe are seeking ways to proliferate the integration of VRE into power grids [1]. Inertialess VRE is subverting traditional assumptions associated with power generation, transmission, and distribution system management. Reduction in the cost of VRE technologies implies a tangible paradigm shift in the economics of electricity generation options [2]. Electricity generation technology is transcending from conventional fossil fuel-based generation to stochastic renewable generation. Globally, an increase of 77% in VRE generation was recorded over 2019–2021, as illustrated in Figure 1 [3].

The power generated from VRE is cheaper than most fossil fuel-based generators due to its lower operating cost [4]. For power systems with a higher-than-average VRE penetration, the cost of electricity production should be anticipated to be lower than average [5]. However, the actual electricity prices must also reflect power delivery services, such as high flexibility and availability, which come at a cost. The wholesale electricity price fails to provide an explicit value and transparency for the flexibility and dispatchability that is required in an interconnected power system to maintain operating reserves [6].

System operators are responsible for operating an interconnected electricity network efficiently while ensuring stipulated standards of security and reliability are being met. Ancillary services are needed to maintain safety, security, and reliability within the network. Ancillary services represent the potential to deliver active and reactive powers and maintain key technical parameters of the network, such as frequency, voltage, network loading, and system restart processes [7]. As more VRE is integrated in the network, the system operators are expected to encounter emerging challenges to maintain system security primarily with issues such as inertia, frequency control, voltage management, and system strength. For instance, system operators are relying on increasing their procurement of reserves in order to compensate for the variability introduced by renewable power generation. As the share of VRE in the grid increases, the cost of procuring these reserves will rise, which may result in increased electricity costs [8].

Various new market models are being proposed by researchers and policymakers across the globe to outperform the existing models. The new market models are aimed toward harnessing the benefits of existing and advancing technologies in a fashion that supports the long-term interests of consumers while enabling system operators to operate a secure and reliable power system efficiently. The aim of this paper is to provide readers with a background of electricity markets and accentuating factors that led to the first transformation of the power industry and highlight distinct characteristics and challenges of various market models that are currently implemented predominantly in developed countries. In the past, various researchers have presented surveys on specific issues within electricity markets [9,10,11]. However, a holistic survey on impending challenges and proposed solutions as the markets transition towards increased VRE was missing.

The main contributions of this paper are as follows: (i) a holistic survey of existing market challenges that are hindering the transition to increased VRE; (ii) a critique of the proposed solutions to tackle the existing market challenges; (iii) recommendations for future research work to aid researchers and policymakers to accelerate the research in the area of electricity market transformation. The scope of this paper can be defined as a review of the existing academic and industry literature that highlights the issues present in centralized electricity markets and how the integration of VRE has exacerbated these issues, as well as relevant proposed solutions to mitigate such issues.

The paper is structured as follows: Section 2 provides an overview of electricity markets and Section 3 highlights the distinct characteristics of most advanced electricity markets around the world. Section 4 provides a commentary on various challenges and barriers in current electricity markets and provides a critique on the proposed solutions, such as the newly developed market models for the smooth transitioning of electric grids. Section 5 and Section 6 provide recommendations for future research and a conclusion, respectively.

2. Overview of Electricity Markets

In the past, large-scale utilities had command over all activities in the generation, transmission, and distribution of power, which monopolized the electric power industry [12]. State governments owned and operated the electricity sector, comprising generation, transmission, distribution, and retail within a vertically integrated market structure. Under this market structure, governments had no compulsions to keep the electricity prices competitive. Therefore, due to proliferation of energy demand, inefficient network management, and unreasonable tariff schemes, a monopolized market structure transitioned to a deregulated market structure [13].

In the 1990s, following the United Kingdom, most of the Commonwealth countries restructured the electricity industry by facilitating competition and allowing new entries of privatized firms to improve economic efficiency in the electricity sector [14]. In restructuring electricity markets, the initial step was to separate electricity generation and transmission activities. The subsequent step was to introduce competition in the electricity generation sector, either through gross-pools, bilateral contracts, or bidding in wholesale markets. However, the transmission networks were realized to be more efficient under a monopolistic scheme to avoid privatized firms from overcharging the transmission services [13]. Consequently, a system operator and a market operator were required to oversee the transactions occurring in the market while ensuring the integrity of the grid.

A competitive, deregulated electricity market environment compels the power producers to export electricity economically and enables the consumers to select their preferred tariff policies through different electricity retailers [9]. Around the world, there are different levels of deregulation formed by energy policies and market structures [15]. In general, the deregulated electricity market consists of three marketplaces according to different time resolutions, as illustrated in Figure 2 [16].

• Day-ahead market: for the settlement of an hour’s load demand 24 h in advance, based on forecasted load demand.
• Intra-day market: for the settlement of hour-ahead forecasted load demand.
• Real-time pricing: for the balance of the system during the operational hour.

In different marketplaces, there exist different pricing schemes. There are three major pricing schemes around the world [17]:

• Uniform Marginal Pricing (UMP): one price signal for the entire network under the system operator.
• Locational Marginal Pricing (LMP): optimal power flow-based pricing; different prices for different buses (nodes) in the transmission network.
• Zonal Marginal Pricing (ZMP): one single price for a specific region or zone.

Retailers buy electricity on the wholesale market. The wholesale market is managed by the market operator. The market operator provides dispatch instructions to the generators in order to meet the necessary demand while ensuring fair access to the market for the market participants. On the other hand, the system operator ensures that network reliability, stability, and security are maintained. The competitiveness amid generators and retailers in a deregulated market environment has enhanced social welfare and led to a reduction in the cost of electricity, after accounting for inflation [15].

There are mainly two types of market models in place: energy-only markets and capacity-energy markets. In an energy-only market, the generators obtain revenue exclusively from selling the generated electricity. In capacity-energy markets, both electricity spot price and capacity payments are provided to all types of generators. In an energy-only market model, the electricity spot market dictates the revenue stakeholders can generate in the physical electricity market. The spot prices in an energy-only market tend to be more volatile in nature due to the continuous fluctuations in the supply–demand balance [18]. This paper focuses on the existing and impending challenges of energy-only markets.

Ancillary services are crucial for the power system to operate securely and reliably. System imbalance amid generation and load results in frequency deviations that can impose a risk on the system security, and in extreme cases, can result in an overall system collapse [17]. There are various market products that are set up to achieve energy balance and frequency control for the power system to operate in a secure and reliable fashion. Some countries have a separate ancillary services market, which is also regulated by the system and market operator [19].

Financial markets are also an integral part of the deregulated market environment. The financial markets allow different contracts to be set up between generators, retailers, and investors, acting as insurance policies. The hedging contracts minimize the substantial risk of financial exposure imposed on market participants due to volatility in electricity prices by locking a fixed price for electricity that will be generated or consumed at a given time in the future [20].

3. Electricity Markets around the World

This section provides a commentary on the market models that are implemented in various developed countries. The distinct features of these market models are also highlighted. Figure 3 shows a representation of different levels of deregulation in various countries around the world [21].

According to the survey conducted in [17], North America has one of the most advanced wholesale electricity markets in the world with provision of precise and real-time price signals to its consumers. A real-time LMP scheme is implemented in the United States. Locational marginal prices are computed for a 5-min interval and price signals are published for the consumers [17]. The price settlement varies from ten minutes to an hour for different states. The PJM (Pennsylvania, Jersey, and Maryland) Power Pool interconnection is a regional transmission operator that regulates the electricity market of thirteen states of the United States of America [22]. This system has a day-ahead scheduling reserve market alongside an operating reserve market. The day-ahead scheduling reserve market aims to achieve 30-min reserves on a day-ahead basis, while the operating reserve market aims to settle the instantaneous imbalances via a short-term forward dispatch.

In the United Kingdom, a competitive wholesale electricity market structure is in place. The main stakeholders participating in the wholesale market are retailers, generators, traders, and customers. National Grid Electricity Transmission (NGET) is responsible for meeting the demand and supply for the interconnected power system. NGET performs a real-time demand and supply balancing mechanism. The implemented balancing mechanism imposes an imbalance penalty on a market participant when they generate or consume outside of their contracted limits [23].

The Nord Pool is one of the most prominent electricity market operators in Europe, covering the Northern European countries, such as Norway, Sweden, Denmark, Finland, Estonia, Germany, Latvia, and Lithuania [24]. The Nord pool has a day-ahead market that maximizes social welfare while adhering to the network constraints. Along with a day-ahead market, an intra-day market is operated within the Nord pool to obtain the required balance between supply and demand, allowing the participants to trade closer to the physical delivery and achieving overall security and reliability of the network. Most of the European electricity markets have a ZMP scheme; however, adopting an LMP scheme is suggested for European electricity markets according to the analysis conducted by the author of [25]. The author argues that an LMP includes both the electricity production and delivery cost to the end user, thus creating appropriate signals for network location and generation in a particular geographical zone, and thereby encouraging infrastructure investments in congested zones. Moreover, the LMP model will stimulate the development of short-term markets with greater liquidity as well as promoting the participation of distributed energy resources.

Australia has three different power systems based on three regions: the south-east coast, the south-west coast, and the Northern Territory. The Australian Energy Market Operator (AEMO) acts as both the system operator and market operator for the power systems in the south-east coast and the south-west coast [26]. The interconnected power system on south-east coast is the largest interconnected power system in the world in terms of geographical span. The south-east coast power system forms the National Electricity Market (NEM) of Australia, which has an energy-only market model [14]. The wholesale transaction of electricity takes place on a spot market in the NEM by instantaneously matching the supply and demand through a centrally coordinated dispatch process. Generators bid to provide the market with fixed amounts of electricity at specific rates. Bids of 5-min intervals are submitted each day by the generators. From the submitted bids, the AEMO appoints the generators needed to generate electricity on the basis of meeting forecasted demand in the most economical fashion [27].

As with all other system operators, the AEMO is accountable for ensuring that the power system is being operated safely, securely, and reliably. To meet this obligation, the AEMO manages key technical features of the power system, such as frequency and voltage, through ancillary services market. The AEMO controls eight markets to operate ancillary services. For each dispatch interval, a dispatch engine, called ‘Scheduling Pricing and Dispatch’, evaluates the offered contracts, and determines the clearing prices for each ancillary service market. The eight ancillary services can be grouped under one of the following three major categories [19].

• Frequency control ancillary services;
• Network control ancillary services;
• System restart ancillary services.

Apart from the physical electricity and ancillary services markets, different stakeholders participate in hedging contracts to manage the risk imposed on their assets by electricity price volatility. Financial derivatives, such as over-the-counter (OTC) market contracts, are mutual agreements amid generators and retailers, which are exposed to opposing risks in the wholesale electricity market [28]. However, the financial market is not regulated by the AEMO, thereby making these transactions less transparent to the governing bodies [28].

4. Challenges in Electricity Markets and Their Proposed Solutions

Fundamentally, the existing challenges in an energy-only market design can be grouped into two categories: traditional challenges and VRE-induced challenges. The traditional challenges have long existed in electricity markets; however, various new challenges have started to emerge with the proliferation of VRE. The VRE-induced challenges are associated with non-dispatchability due to the uncertainty and variability of VRE generation. Technical challenges exist due to the physical barriers of the power system, whereas economic challenges are related to inefficiencies of existing market models. Most of these challenges result in asymmetric investment signals that pose a threat to the long-term reliable operation of electricity markets if not addressed properly.

The major technical challenge within electricity markets is balancing the supply and demand instantaneously; an imbalance between supply and demand causes the system frequency to deviate from the standard frequency. Frequency deviation in the power system poses threats to the integrity of the grid, such as damaging the equipment or even a total blackout in extreme cases [29]. The system operators acquire frequency control services to ensure energy balance. There are different market products in the ancillary services market that are offered to secure reserves. These products are distinguished based on their technical aspects, such as response duration and mode of operation. The cost of acquiring these reserves is usually distributed among various market participants. However, as VRE penetration increases, the gross cost socialized among market participants will rise significantly [8].

Additionally, due to the shared nature of electric power systems, a unique barrier exists in the form of a failure to determine the desired reliability of the consumer [30]. In other words, the network cannot distinguish between consumers who are willing to pay for reliability and those who are not. According to the authors of [31], setting up contracts in which consumers reveal their desired reliability can be helpful to overcome this barrier, because in the current energy-only market model, the spot prices of electricity cannot transmit all information and signals that are required for reliable long-term operation and capacity investments. Typically, a significant percentage of electricity is traded months in advance through forward contracts and OTC markets, which allows generators and retailers to hedge their assets [32]. Variations in electricity demand, a lack of bulk storage, and unexpected unavailability of generation capacity induce a strong need for short-term trading. As a result, spot markets usually possess high liquidity.

Inflexible demand from the consumer side poses another problem for the existing wholesale electricity markets. Household and some commercial customers have a fixed tariff for buying electricity in combination with a flat-rate tariff [33]. Thereby, consumers are not exposed to the volatility in the wholesale electricity market and, as a result, fail to respond to even the extreme pricing events in the market [34]. The author of [35] claims that having a more flexible demand, where consumers actively respond to price signals, can result in a more efficient market as it will resolve a number of issues related to the security of supply, such as asymmetric investment incentives. This argument is further supported in [36], where the authors claim that the wholesale prices of electricity markets will be reflected in retail prices upon demand-side integration. However, in electricity markets around the world, system operators send power balancing services predominantly to the supply side to ensure network security and reliability in the short-term, and warrant capacity investments in the long-term.

Flexible demand-side resources, such as electric vehicles (EVs), may be used to alleviate supply–demand imbalances [37,38]. The authors of [39] propose a smart charging scheme that provides incentives for the EV owners to charge their batteries during periods of low electricity demand and supply stored energy to the grid during periods of high demand. The scheme is referred to as a time-of-use tariff and is effective for peak shaving of the varying electricity demand. However, uncoordinated charging of EVs can lead to significant operational challenges for system and market operators by further amplifying the electric load demand during the peak when users return from work, posing a threat to the integrity of the electric grid. As a result, distribution transformers and feeders will be overloaded [40] and power quality issues, such as voltage deviations and harmonic distortions, can arise [41]. Moreover, in already congested transmission systems, mismanaged EV charging will result in more severe congestions, consequently increasing wholesale electricity prices [38]. Therefore, if a strategic approach is not adopted in integrating EVs into electric grids, mismanaged EV charging can exacerbate power system and electricity market issues on both the distribution and transmission layers. The authors of [42] propose a methodology to model the electric load demand from EVs in a long-term capacity expansion framework. The ramifications of the amplified demand on the electric grid, operational costs, emissions, and investment signals are examined. The results indicate that optimized EV charging can reduce incremental costs in building new infrastructure to accommodate EVs. However, limited efforts have been made to study the role of market participants in providing aggregated EV services to the grid in a centralized market.

The current electricity market model is dispatched based on the forecasted load demand determined by the system operator. Each generator submits an offer for the amount of energy they are willing to sell at their marginal cost. The system operator then clears the market by stacking the offers from each generator in a merit order, dispatching the least-cost generator first. A diagrammatic representation of dispatch process on merit-order is illustrated in Figure 4 [43].

Solar PV and wind power have high capital costs, but the operational marginal costs of these generators are nearly zero [44]. The minimal generational cost of VRE has led to a drop in electricity prices, also referred to as the merit-order effect [45]. The priority dispatch of VRE performed in several European countries [46] has led to negative pricing events becoming prevalent [47]. In Australia, negative pricing events have become more frequent; interventions from the market operator are required during negative pricing events [6]. Frequent interventions from the market operator indicate a market failure. Furthermore, a decline in wholesale electricity prices has resulted in a loss of revenue for conventional generators, thus making it difficult for conventional generators to recover their capital costs. This is evident in Europe, where the increase in the share of VRE and the introduction of emission costs resulted in a drastic decline in electricity prices [48], thus making it harder for conventional generators to recover their fixed and operating costs.

One of the desirable characteristics of markets is to have greater economic efficiency. As a general rule, maximizing social welfare results in increased economic efficiency and, in a perfectly competitive market, social welfare is maximized when the offering price of a generator is its marginal operational cost [49]. However, one of the complicating factors of the electricity markets is that conventional generators have a considerable portion of fixed costs, such as no-load and start-up costs, which do not vary in line with generation output [50]. As a result, marginal and near-marginal generators are not able to recover their variable costs fully from the energy-only market model when the market price is based only on marginal cost [51]. Therefore, compensation mechanisms in electricity markets are being implemented. One of the most common practices is providing out-of-market subsidies to certain generators that fail to recover their gross operational costs at market prices. Although out-of-market subsidies are easy and intuitive to implement, they fail to send appropriate investment signals to new participants entering the market as well as deteriorating market transparency [52].

The authors of [53] conduct an analysis of different pricing mechanisms to reduce out-of-market subsidies. The pricing mechanisms reviewed are marginal pricing, convex hull pricing [52], extended locational marginal pricing [54], and Vickrey pricing [55]. In the Vickrey pricing mechanism, the market prices tend to rise relatively higher during low-demand periods as opposed to during high-demand periods. Hence, it is a good option to increase the profits of baseload generators selectively rather than peak generators if base generators are paid out-of-market payments.

Moreover, as the existing market dispatch process assumes that the participating generators have different—but non-zero—operational marginal costs, it is debated by policymakers as to whether the current dispatch process needs to be reformed when more VRE generators will be participating in the market with zero marginal costs [2]. Some researchers proposed that if the market prices are often low or negative, a capacity market model can be adopted to recover fixed costs, rather than altering the logic of the existing dispatch process [43]. Various European nations are starting to adopt the capacity market model to tackle this challenge.

There exists a debate over whether energy-only markets are appropriate for networks with high penetration of VRE. The authors of [18] present a case for a capacity-energy market by modelling the South Australian network. A multi-commodity inter-temporal Cournot game-theoretic model was proposed to simulate a capacity-energy market. The existing energy-only market and the proposed capacity-energy market designs were compared against a competitive benchmark for market power. Market power is defined as the ability of an individual or a group of generators to keep prices above the competitive levels for a significant amount of time, which results in market inefficiency. The results indicated that capacity-energy markets perform better than the current energy-only market model of the NEM. However, the proposed market design fails to eradicate market power completely.

The NEM of Australia is a classic example of an energy-only market that is facing challenges, such as inadequate system strength [56], negative pricing events [6], and sub-optimal capacity investments for generation mix [28] due to the expansion of VRE and inflexible demand. While the NEM has demonstrated volatility and prolonged phases off-equilibrium, it is also esteemed for its commendable technical and economic performance over the last 20 years. Nonetheless, since 2016, some of the regions in the NEM have exhibited instability. For instance, South Australia, a region comprising 40% VRE, lost its last baseload coal plant in April 2016 due to merit-order effects. Immediately after the decommissioning of the plant, wholesale futures prices tripled in South Australia. The regional security of supply declined, which was one of the contributing factors in the state-wide blackout on 28 September 2016 [57].

Furthermore, due to various subsidy schemes, the symbiosis between financial incentives and the physical needs of the power system has been distorted [58]. The existing subsidies provide fewer incentives to generate electricity at instances during peak demand times. The use of production subsidies has resulted in a higher adoption of renewable power purchase agreements (PPAs). “A PPA is a long-term agreement between a generator and a purchaser (a retailer or a consumer) for the sale and supply of energy” [59]. Wind and solar farms sell most of their generated electricity through PPAs. Usually, VRE generators sell renewable energy certificates to retailers at a fixed cost. PPAs provide revenue for VRE generators for producing as much electricity as possible at any time; in other words, VRE generators can keep generating as much electricity as they choose regardless of the needs of the power system. As a consequence of this, the liquidity ratio in the NEM has declined.

An analysis of the currently implemented production-based subsidy schemes in the NEM was conducted in [59]. The results show that, in the presence of an out-of-market production subsidy, the integration of new VRE will result in a decline in revenue for the existing generators. Hence, if market reform measures are not taken, the sustained use of production subsidies will result in non-desirable investments for capacity. In other words, generators will make revenue by maximizing production at any time, rather than investing in capacities for an optimal mix of generation. As a result, investments in building new conventional power plants will drop significantly, which could adversely affect the system strength [56]. Presently, due to the limitations in technology, VRE is incapable of completely replacing conventional generation [60]. Conventional generators with a fast ramp up rate are essential for the secure and stable functioning of power systems integrated with intermittent and inertialess VRE. Energy storage techniques can be deployed to mitigate uncertainty introduced by intermittent VRE. Smoothening the energy generated from VRE over different time periods reduces the uncertainty levels of certain VRE generators. However, the current capital costs associated with energy storage technologies inhibits the integration of bulk storage in power systems. Moreover, according to the economic analyses conducted in [61,62], the cost-recovery of energy storage technologies, under existing market models is difficult as the economic value of energy storage results from an opportunity for arbitrage, i.e., buying energy when electricity prices are low and selling it when the prices are high.

As the deployment of large-scale storage systems is still not deemed cost effective, grids with higher penetration of VRE are highly affected by the variability and intermittency of VRE. Efforts have been made by researchers to propose bidding strategies that can provide solutions to the uncertainty introduced within the grids by VRE. Various authors have proposed frameworks on associating a VRE generator with other energy sources to mitigate variability from renewable resources in a market with a two-stage settlement mechanism, with the consideration of a day-ahead market and a real-time balancing market. The authors of [63,64] proposed a combined wind–hydro bidding technique in the market that requires varying of the hydropower generation according to the expected energy deficit of wind. Large-scale storage systems are also coupled with plants in [65,66] to meet the energy deficit experienced by a VRE generator. However, these models mainly depend on the co-ownership of the two energy sources.

Efforts to increase the profits of VRE generators were made in [67,68]. A new bidding strategy for a wind, thermal, and PV solar system for profit maximization and emission minimization was proposed in [67], whereas the authors of [68] proposed a business model in which market participants with VRE could trade through both firm and risky power contracts. Risky power contracts allow wind power generators to trade uncertain future power for efficient wind aggregation. The results from both [67,68] resulted in increased profits for the renewable power generators.

Probabilistic market bidding frameworks are also presented by researchers for an electricity market with higher penetration from VRE. A simplistic approach of mapping deterministic bidding to probabilistic bidding in a market with a high VRE penetration is presented in [69]. Submitting probabilistic offers in a day-ahead market aids in formulating market mechanisms that can evaluate—and thus, incentivize or penalize—VRE generators regarding the precision and accuracy of the estimates of their generation. The proposed market design motivates VRE generators to disclose their actual estimates, while conventional generators are exposed to no additional penalties if there is no uncertainty in their offers. The disclosure of actual estimates from VRE generators allows the system operator to schedule sufficient resources to meet the demand.

The setting up of reliability contracts between conventional generators and VRE generators to avoid unexpected energy deficits due to variability in generation is proposed in [70,71]. In the proposed work, wind generators are penalized for not meeting their obligations for providing power as per their estimated offer in a day-ahead electricity market. The proposed market framework encourages wind generators to purchase options contracts from conventional generators to hedge their portfolio against penalties. The results from mathematical proofs and simulation studies indicate an increase in the revenue for VRE generators without adversely affecting the revenue generated by other generators. However, these studies do not comprehensively capture the strengths of options trading, as they only consider a single payment made to the conventional generators to hold the reserve. Therefore, a market model that ensures a capacity payment made to conventional generators to appropriately capture the costs associated with holding reserve capacity, as well as a payment for exercising the option when the actual energy deficit takes place, will be a better representation of options trading. A summary of the reviewed market models is provided in

Table 1. Summary of market models proposed under the two-stage settlement mechanism.

Peculiarities of the Studied Models	Main Highlights	Recommendations for Future Work
- A stochastic optimization technique that maximizes the joint profit of hydro and wind generators in a pool-based electricity market is proposed in [63,64]. - Hydro generation varies its output based on the realization of actual wind power generation. - The decisions to vary the output of hydro generation are made based on profit maximization.	- The proposed algorithm results in the reduction in penalties incurred by wind generators. - A decrease in the needs of conventional capacity reserves is also expected.	- Exploring the possibility of joint wind–hydro bids on the overall system reserves management. - Studying the effects of transmission network congestion on the profits for combined wind–hydro bidders. - Examining the proposed model without the assumption of a joint ownership and operation of the two generating plants.
- The coordinated operation of wind generation and battery energy storage is proposed in [65], in which the storage is operated by wind generators. - The use of battery storage as a palliative for the high variability of renewable production is proposed in [66], in which the utility company owns and operates the storage.	- Both models examine the ability of battery storage to reduce power imbalance due to variable wind generation based on storage capacity constraints. - Conducted research provides key insights into the trade-offs between energy storage capacity and maximum expected profit.	- Conducting extensive simulation studies for the proposed market framework on existing market models. - Formulating a problem, in which energy storage is owned by an independent rational market participant. - Formulating a problem, in which economic trade-offs that might emerge in using storage for the provision of ancillary services can be investigated.
- A bidding strategy for a wind–thermal–PV system participating in the energy and spinning reserve markets is proposed in [67]. - The bidding strategy is bi-objective, which maximizes the profits of the wind–thermal–PV system and minimizes emissions. - A hybrid weighted sum method and fuzzy satisfying approach are used.	- Uncertainty associated with the day-ahead energy, spinning reserve, imbalance prices, and VRE is modeled. - Increases the expected profit of generators.- Reduces the expected emission of generators.	- Considering a risk measuring index in the bi-objective bidding strategy to make decisions based on expected risk-adjusted profits. - Exploring the coordinated bidding from the wind–thermal–PV system as a price-maker in the market.
- An instrument for wind aggregation called a risky power contract is proposed in [68]. - Wind generators can trade uncertain future power generation with other VRE generators. - A competitive equilibrium is achieved in a non-cooperative game setting.	- Enables efficient uncertainty reduction. - Increases profits for wind generators.	- Studying the impact of risk power contracts on other market participants. - Exploring the proposed strategy for solar PV aggregation.
- A probabilistic market based on an affine transformation of strictly proper scoring rules, such as Brier Scores and Continuous Ranked Probability Scores is proposed in [69]. - The market accepts probabilistic offers from VRE generators. - Generators are rewarded or penalized based on the accuracy of submitted forecasts.	- The proposed market compels VRE generators to reveal their true forecasts. - The disclosure of actual estimates from VRE generators allows the system operator to schedule sufficient resources to meet the demand. - The VRE generators are held accountable for introducing the uncertainty in the system.	- Deriving mathematical proofs to evaluate whether a probabilistic market has the desirable attributes of electricity markets such as cost recovery, revenue adequacy, and incentive compatibility. - Supporting the proposed market framework with relevant simulation studies.
- A real options market-based approach is proposed in [70]. - Variable generators purchase options for reserve from flexible generators in an ex-ante options market. - Optimal strategies for generators in a coupled day-ahead and options markets are derived based on the Karush–Kuhn–Tucker (KKT) conditions.	- Increases renewable penetration. - Ensures delivery of reliable power. - No market participants are worse-off.	- Considering a cost model for renewable generators. - Considering both the strike price and the premium fee in the options contract to appropriately represent options trading. - Incorporating the ramifications of network congestion on the options market.
- A reliability contract between a renewable generator and a conventional generator is proposed in [71]. - A conventional generator fulfills the unmet commitments of a renewable generator for a reserve fee. - Optimal strategies for generators are derived by examining the first and second partial derivatives of the utility function.	- Increases the profits of both renewable and conventional generators. - Decreases the number of total unmet commitments.	- Studying the possibility of reducing fossil-fuel based generation under excess renewable generation. - Studying the impacts of varying penalty pricing to strategically vary the integration of renewable generation. - Modeling other flexible energy sources as providers of reserve such as energy storage.
- A Nash–Cournot energy-capacity market model is proposed in [18]. - A multi-commodity market that provides payments for generated electricity and investments in capacity is studied.	- Reduces spot price volatility. - Ensures cost recovery of generators. - Lower market power as compared to an energy-only market model.	- Considering risk management tools to manage the generated revenue based on spot price volatility. - Considering the impact of financial markets on market power.

.

The financial derivatives market also has a substantial impact on the activity of stakeholders in wholesale and retail markets. However, in most cases, the energy regulators do not regulate the electricity derivatives markets [72]. The contracted volume of energy and forward prices in financial markets affect generator bidding in electricity markets, resulting in price signals that fail to maximize social welfare. Moreover, the uncertainty of VRE generation inhibits these market participants from managing their portfolios with hedge contracts [59]. Hedge contracts have traditionally been used by generators and retailers as risk management tools against revenue volatility and were developed mainly for conventional generators. Since then, minimal changes have been made to these products. However, with the unprecedented change in the energy mix and proliferation of VRE, it is evident that the hedging needs of market participants are also changing. Therefore, the formulation of new hedge policies is vital to ensure that adequate risk management tools are present in the market for VRE generators. The lack of risk management tools available to market participants diminishes choice and affects liquidity, thereby hindering new entries into the market.

The Australian Renewable Energy Hub has introduced a number of purpose-built contracts for VRE generators that promise to minimize risk exposure due to weather intermittency while harnessing the benefits of VRE technology [73]. One of the proposed hedge contracts helps VRE generators to flatten the variability of PV solar output during the day through ‘Solar Shape and Inverse Solar Shape’ hedge contracts. The two inter-related products manage the intermittency of solar generation by tailoring the shape of contracts to specific periods of the day and provide an alternative to flat-rate contracts. The products work by reducing the merchant exposure of solar PV generators to prolonged periods of low market prices and are aimed towards relatively new generators to support the growth of renewable energy in the market.

Moreover, the authors of [74] propose an insurance market that operates along a day-ahead market in a two-stage market model. The first stage is prior to the actual realization of variable generation when insurance contracts are set up and the second stage is after the realization of variable generation when insurance contracts are executed. The insurance contract enables its buyer to exercise the right to claim a payment for an energy deficit experienced by the VRE generator by paying an upfront fee to a conventional generator, who would typically provide energy for this deficit. The results show that the revenue volatility of market participants is reduced through these hedge contacts. However, operating such insurance markets alongside the current electricity markets needs an appropriately designed legal and regulatory framework.

The highlighted challenges and proposed solutions of electricity markets that were presented in this survey are summarized in

Table 2. Summary of market challenges and proposed solutions in the literature.

Category	Challenge	Proposed Solution(s)
Traditional	Power balance	Flexible demand [35] Demand-side integration [36] Large-scale storage [61,62]
	Unknown reliability of consumers	Desired reliability contracts for consumers [31]
	Failure of cost-recovery	Convex hull pricing [52] Extended locational marginal pricing [54] Vickrey pricing [55]
VRE-induced	Variable generation	Risky power contracts [68] Probabilistic bidding [69] Energy storage [65,66]
	Merit-order effect and negative pricing events	Paying for flexibility services [6] Enabling capacity payments [18,43] Appropriately designed renewable subsidy schemes [59]
	Unmet day-ahead commitments by VRE generators	Real options market [67] Reliability contracts [68]
	Lack of risk management tools for VRE assets	Solar shape and inverse solar shape contracts [73] Two-stage insurance market [74]

.

5. Future Research Directions

The research findings indicate that although sustainable electric grids with higher penetration from VRE will result in a reduction in carbon emissions, further work is required from relevant stakeholders and policymakers toward harnessing the benefits of existing and advancing technologies in a manner that supports the long-term interests of consumers while also enabling system operators to efficiently operate a secure and reliable electric grid. For instance, further research is required in developing appropriate and distinct financial instruments and risk management tools that will provide equal opportunities for all market participants to maximize their own individual profits as per their distinct role in the market, whilst adhering to the needs of the power system.

Furthermore, an unprecedented paradigm shift is anticipated as traditionally inactive electricity consumers are turning into prosumers, i.e., consumers who are also producers of electricity. The prosumers are untapped sources; if deployed in a pertinent fashion, prosumers cannot only provide a significant share of renewable energy into the network, but also help in meeting the increasing energy demand, thus unlocking new revenue streams that may prove to be instrumental for all market participants. Technological advancements such as big data, machine learning, artificial intelligence, smart contracts, and blockchain technologies will play a vital role in enabling peer-to-peer energy trading and aggregated energy demand response in electricity markets. Hence, researchers and policymakers should channel their research efforts towards developing market models considering the future role of prosumers in electricity markets with large-scale aggregation for demand response and provision of ancillary services. In addition, aggregation strategies for EVs to provide frequency control and reactive power support services to the grid can be explored.

Moreover, in light of the extant literature, it can be conceded that although new market models are being proposed by various researchers and policymakers, limited efforts have been made to present a robust market evaluation framework to assess the performance of new market models in comparison to the existing ones. Therefore, research towards developing market evaluation frameworks to aid researchers in assessing the performance of the newly developed electricity market models is of crucial value.

6. Conclusions

Two decades ago, the restructuring of electricity markets was established in order to improve market efficiency and optimize resource allocation. Nevertheless, in this era, the chief driving force of electricity market transformation is the commitment to strengthen the sustainability of the electric power industry. VRE generation, such as wind and solar, has been utilized substantially to replace fossil fuel-based generation as world economies tackle global warming and climate change. While the energy transition is gaining momentum, with electric grids experiencing a higher penetration of VRE, numerous challenges within existing market models have surfaced.

In this paper, a holistic survey of the past, present, and future of electricity markets is conducted. The paper highlights existing barriers and challenges that are being experienced by market and system operators against the smooth transitioning to a sustainable electric grid. To overcome the impending challenges, researchers around the world have proposed various solutions. A thorough review of the proposed solutions and their effectiveness in tackling these challenges is presented in this paper. The conducted survey showcases a unique blend of solutions proposed in both academic articles and technical reports. In addition, recommendations for future work are provided to help researchers and policymakers determine research directions for unlocking the future of electric grid transformation.