Modeling of Collusion Behavior in the Electrical Market Based on Deep Deterministic Policy Gradient

Abstract

The evolution of the electricity market has brought the issues of market equilibrium and collusion to the forefront of attention. This paper introduces the Deep Deterministic Policy Gradient (DDPG) algorithm on the IEEE three-bus electrical market model. Specifically, it simulates the behavior of market participants through reinforcement learning (DDPG), and Nash equilibrium and the collusive equilibrium of the power market are simulated by setting different reward functions. The results show that, compared with the Nash equilibrium, collusion equilibrium can increase the price of nodal marginal electricity and reduce total social welfare.

1. Introduction

The analysis of power market equilibrium is pivotal in various domains, including market operation simulation, market mechanism design, trading decision-making by market entities, market power analysis, electricity price forecasting, and source-grid-load investment planning [1]. The bilevel optimization model is the predominant approach for modeling power market equilibrium. Solutions are primarily derived through two methods: the model transformation method, which leverages the Karush-Kuhn-Tucker (KKT) conditions or the strong duality theorem to convert the bilevel model into a single-level mathematical program with equilibrium constraints (MPEC) or an equilibrium problem with equilibrium constraints (EPEC) [2,3]; and the iterative solution method, which employs optimization algorithms based on agent-based models, such as reinforcement learning and deep reinforcement learning. The model transformation method presupposes that the lower-level model is a convex optimization problem, which restricts its applicability to unit commitment and clearing models in the power market and is more suited for single-period market equilibrium problems [4,5,6]. In contrast, the iterative solution method is adept at addressing multi-period market equilibrium problems. Notably, deep reinforcement learning circumvents the “curse of dimensionality” that can arise from the discretization of the action space in reinforcement learning, making it suitable for large-scale system models.

Currently, power market modeling predominantly employs Nash game equilibrium, with insufficient attention given to collusion equilibrium scenarios [7,8]. This oversight hinders a comprehensive understanding and effective regulation of the power market’s actual operations. Collusion among power producers is a real possibility in the electricity market, yet most existing research concentrates on Nash game equilibrium, neglecting the collusive equilibrium scenario. Due to the lack of research on collusion scenarios, market regulators cannot quickly identify collusion behaviors in the electricity market. Collusion can lead to market price distortions, inefficient resource allocation, consumer harm, and reduced market efficiency. Furthermore, the inadequate consideration of collusive equilibrium results in deficiencies in market mechanism design and policy formulation [9]. Existing market rules and regulatory policies may fail to identify and prevent collusive practices, thus jeopardizing fair competition and the stable operation of the electricity market. There is also a dearth of in-depth research on monitoring and punitive mechanisms for collusion, which hampers the creation of a strong deterrent against potential collusive activities. Additionally, overlooking the collusive equilibrium scenario restricts the assessment and management of power market risks. Collusion can induce market volatility and increased uncertainty, affecting the safe and stable operation of the power system. The absence of quantitative analysis and countermeasures for market risks under collusive equilibrium leaves the power market exposed to collusive risks.

Considering the problem mentioned above, this paper introduces an innovative application of the Deep Deterministic Policy Gradient (DDPG) algorithm to address the collusion equilibrium in the power market, validated through rigorous testing within the IEEE three-bus model. Using the DDPG model to simulate the power market, participants can converge to the market equilibrium point in different scenarios. The results show that the collusive equilibrium significantly increases the bidding price by market participants compared with Nash equilibrium. Concurrently, such collusive practices significantly elevate the node marginal price of electricity. The increased nodal marginal price not only indicates the distortion inflicted by collusion on the electricity market’s price formation mechanism but also underscores the detrimental effects of collusion on the market’s fairness and efficiency.

2. Problem Modeling

2.1. Electrical Market

Bidding curve and cost: Considering actual market operations, this paper employs slope bidding, assuming linearity between the unit’s power generation cost and its output. Given that unit costs are typically represented in quadratic form in existing literature, we adopt the secant method for approximation in this study [10].

Assume that the true cost of generator i is a quadratic function about output power. This quadratic model captures the relationship between the power output and the cost associated with it. The cost function is given by

$$\begin{array}{cc}\hfill {\mathrm{cost}}_i^{quar}& ={\displaystyle \frac{1}{2}}·b_i·p_i^2+a_i^{quar}·p_i\hfill \\ \hfill p_i^{\min }& \le p_i\le p_i^{\max }\hfill \end{array}$$

(1)

In this equation, ${\mathrm{cost}}_i^{quar}$ represents the quadratic cost of generator i, $b_i$ is the coefficient that determines the curvature of the cost function, $p_i$ is the output power of generator i at a given time interval, and $a_i^{quar}$ is the linear coefficient that affects the cost. The constraints $p_i^{\min }\le p_i\le p_i^{\max }$ ensure that the output power $p_i$ is within the operational limits of the generator, where $p_i^{\min }$ is the minimum power output and $p_i^{\max }$ is the maximum power output.

In this paper, we employ secant functions to approximate quadratic cost functions. The quadratic real cost is approximated using the secant approximation formula as follows:

$$\begin{array}{cc}\hfill {\mathrm{cost}}_i& =a_i·p_i\hfill \\ \hfill a_i& ={\displaystyle \frac{{\mathrm{cost}}_i^{quar}\left(p_i^{\max }\right)-{\mathrm{cost}}_i^{quar}\left(p_i^{\min }\right)}{p_i^{\max }-p_i^{\min }}}.\hfill \end{array}$$

(2)

Here, $a_i$ represents the marginal cost, which is the slope of the secant line between the maximum and minimum power outputs. This approximation allows us to estimate the cost at any output level within the range $p_i^{\min }\le p_i\le p_i^{\max }$. Furthermore, the bidding function is defined as

$$\begin{array}{cc}\hfill \mathrm{bidding}\mathrm{cost}& ={\alpha }_i\hfill \\ \hfill {\alpha }_i& \in \{0,3a_i\}.\hfill \end{array}$$

(3)

In this context, ${\alpha }_i$ denotes the bidding marginal cost, which is restricted to either zero or three times the marginal cost $a_i$. This binary choice reflects a strategic decision in bidding within the market.

Figure 1 illustrates the cost function for a unit, depicting both the quadratic curve and the secant line approximation, effectively capturing the relationship between power output and cost.

Market model:at various time intervals, the electrical consumer demand can be accurately modeled as a linear function, which is represented by the equation

$$\mathrm{consumer}\mathrm{price}=f·(D-q_d)$$

(4)

where, f is the slope of the demand function, D is the total demand, and $q_d$ is the quantity demanded. This linear model helps in understanding the relationship between the consumer price and the quantity demanded.

Subsequently, the consumer utility curve, which is a measure of the satisfaction derived from consuming a certain quantity of electricity, is expressed as follows:

$$\mathrm{consumer}\mathrm{utility}=f·D·q_d-{\displaystyle \frac{1}{2}}f·q_d^2$$

(5)

This quadratic utility function captures the diminishing marginal utility, where the second term $-{\displaystyle \frac{1}{2}}f·q_d^2$ indicates that as the quantity demanded increases, the additional satisfaction gained from consuming more electricity decreases.

In the pursuit of maximizing total social welfare, the Independent System Operator (ISO) clears the market under the constraints of nodal power balance, branch flow limitations, and generator output boundaries. This market clearing process at discrete time intervals can be effectively modeled using a DC power flow approach. The optimization problem is formulated as follows:

$$\begin{array}{cc}\hfill & \underset{(p_i,q_d)}{max}\sum _{d\in D}\left(f·D·q_d-{\displaystyle \frac{1}{2}}f·q_d^2\right)-\sum _{i\in G}{\mathrm{cost}}_i\hfill \\ \hfill & \mathrm{subject}\mathrm{to}:\hfill \\ \hfill & \sum _{d\in D}{q}_d=\sum _{i\in G}{p}_i\hfill \\ \hfill & -F\le \mathrm{PTDF}(q_t-p_t)\le F\hfill \\ \hfill & p_i^{\min }\le p_i\le p_i^{\max },\forall i\in G\hfill \end{array}$$

(6)

In this formula, the vector F represents the maximum allowable flow on each line within the network. The matrix PTDF, which is known as the Power Transfer Distribution Factor matrix, plays a crucial role in the system. Vectors $p_t$ and $q_t$ denote the power generation and demand across all buses, respectively, and these are linear combinations of $p_i$ and $q_d$.

At each time interval, the profit for each generator i is determined by maximizing the revenue less the cost. This can be mathematically represented as

$$max\left(r_i\right)={\lambda }_i·p_i-{\mathrm{cost}}_i,i\in \mathcal{G}$$

(7)

This formulation captures the economic cost of power generation, taking into account both the market price and the generation cost. The term ${\cos \mathrm{t}}_i$ represents the cost function for generator i, with $i\in \mathcal{G}$ indicating that the generator is located at bus i where ${\lambda }_i$ denotes the nodal price at bus i in Equation (7).

2.2. DDPG

In contrast to game theory methods that require full information, the agent-based simulation technique models the market as a partially observable Markov decision process, which employs a reinforcement learning approach. Within this framework, each generation agent g perceives an observable state $s_{gt}$ at time t, selects an action $a_{gt}$, and subsequently receives a reward $r_{gt}$. The objective for each agent is to maximize their cumulative reward over the entire time horizon T, although the specific formula for this cumulative reward is not included here for brevity.

The variables are interpreted in conjunction with market mechanisms as follows: The state $s_{\mathrm{gt}}$, which is a key component in defining the decision-making context for each generation agent g at time interval t, is constituted by the nodal prices from the previous time interval and the total load demand of the current time interval. Specifically, it is represented by the combination of these factors as

$$s_{gt}=({\lambda }_{1,t-1},{\lambda }_{2,t-1},\dots ,{\lambda }_{I,t-1},D_t)$$

where ${\lambda }_{i,t-1}$ denotes the nodal price at bus i during the previous time interval, and $D_t$ represents the total load demand at time t.

In this context, the strategic variable ${\alpha }_{\mathrm{gt}}$ corresponds to the action $a_{gt}$, which is generated by the GenCo agent g. However, their ranges may differ and require subsequent scaling. The payoff $r_{\mathrm{gt}}$, often termed as reward in the RL domain, is assumed to reflect the GenCos’ rational consideration of their own payoffs, thus equating payoffs to rewards in this study.

The Deep Deterministic Policy Gradient (DDPG) algorithm, an actor-critic and model-free approach, is based on the Deterministic Policy Gradient and operates in continuous state and action spaces. The DDPG algorithm is built around two main networks: the actor network, described by the policy function $\mu \left(s\right|{\theta }^{\mu })$ with parameters ${\theta }^{\mu }$, and the critic network, described by the action-value function $Q(s,a|{\theta }^Q)$ with parameters ${\theta }^Q$. To facilitate training, target networks are created: the actor target network ${\mu }^{\prime }$ with parameters ${\theta }^{{\mu }^{\prime }}$ and the critic target network $Q^{\prime }$ with parameters ${\theta }^{Q^{\prime }}$. The network architectures used in this study are depicted in Figure 2.

The Deep Deterministic Policy Gradient (DDPG) agent-based learning method leverages the Bellman equation to establish a recursive relationship, which is fundamental to the learning process. This relationship is expressed as $Q^{\mu }(s_t,a_t)=\mathbb{E}\left[r_t+\gamma Q^{\mu }\left(s_{t+1},\mu \left(s_{t+1}\right)\right)\right]$, where $Q^{\mu }(s_t,a_t)$ represents the action-value function, $r_t$ is the immediate reward, and $\gamma$ is the discount factor.

The loss function for the Q network, which is crucial for training the network, is given by: $L\left({\theta }^Q\right)={\mathbb{E}}_{{\mu }^{\prime }}\left[{\left(Q(s_t,a_t\mid {\theta }^Q)-y_t\right)}^2\right]$. Here, $y_t$ is the target value used for training, computed as $Q^{\mu }(s_t,a_t)=\mathbb{E}\left[r_t+\gamma Q^{\mu }\left(s_{t+1},\mu \left(s_{t+1}\right)\right)\right]$. This formulation allows the agent to learn the optimal policy by minimizing the difference between the predicted and target values of the Q function.

The network $\mu$ is updated by applying the chain rule to the previous equation with respect to its parameters:

$$\begin{array}{cc}\hfill {\nabla }_{{\theta }^{\mu }}J& \approx {\mathbb{E}}_{{\mu }^{\prime }}\left[{\nabla }_{{\theta }^{\mu }}Q(s,a\mid {\theta }^Q){|}_{s=s_t,a=\mu \left(s_t\mid {\theta }^{\mu }\right)}\right]\hfill \\ \hfill & ={\mathbb{E}}_{{\mu }^{\prime }}\left[{\nabla }_{a}Q(s,a\mid {\theta }^Q){|}_{s=s_t,a=\mu \left(s_t\right)}{\nabla }_{{\theta }^{\mu }}\mu (s\mid {\theta }^{\mu }){|}_{s=s_t}\right]\hfill \end{array}$$

In the DDPG algorithm, the sequence of operations is as follows: The policy network $\mu$ produces an action $u\left(s_t\right)$ given the current state $s_t$, and noise $n_t$ is introduced to this action, yielding $a_t=\mu \left(s_t\right)+n_t$. Upon taking action $a_t$, the environment transitions the agent to the next state $s_{t+1}$, along with a reward $r_t$. This experience, encapsulated as $(s_t,a_t,r_t,s_{t+1})$, is saved in the replay buffer. For training, a mini-batch consisting of N randomly selected experiences from the buffer, denoted as $\left\{(s_j,a_j,r_j,s_{j+1})\right\}$, is utilized.

Utilizing the mini-batch $\left(s_j,a_j,r_j,s_{j+1}\right)$, the policy network $\mu$ computes $\mu \left(s_j\right)$, which is then fed into the action-value network Q. Within this framework, the gradients are calculated via automatic differentiation. Specifically, the gradient of the action a is determined as

$${\nabla }_{a}Q\left(s,a\mid {\theta }^Q\right){|}_{s=s_j,a=\mu \left(s_j\right)}$$

(8)

and the gradient of the parameter ${\theta }^{\mu }$ is found by

$${\nabla }_{{\theta }^{\mu }}\mu \left(s\mid {\theta }^{\mu }\right){|}_{s=s_j}$$

(9)

These gradients facilitate the approximation of the policy gradient:

$${\nabla }_{{\theta }^{\mu }}J\approx {\displaystyle \frac{1}{N}}\sum _j\left[{\nabla }_{a}Q(s,a|{\theta }^Q){|}_{s=s_j,a=\mu \left(s_j\right)}{\nabla }_{{\theta }^{\mu }}\mu \left(s\right|{\theta }^{\mu }){|}_{s=s_j}\right]$$

(10)

Employing a small learning rate $l{r}_a$, updates to the network $\mu$’s parameters are made via gradient ascent to refine the policy’s performance.

$${\theta }^{\mu }\leftarrow {\theta }^{\mu }+l{r}_a{\nabla }_{{\theta }^{\mu }}J$$

(11)

Lastly, the target networks are softly updated with a small update rate $\tau$:

$$\begin{array}{cc}\hfill {\theta }^{Q^{\prime }}& \leftarrow \tau {\theta }^Q+(1-\tau ){\theta }^{Q^{\prime }}\hfill \\ \hfill {\theta }^{{\mu }^{\prime }}& \leftarrow \tau {\theta }^{\mu }+(1-\tau ){\theta }^{{\mu }^{\prime }}\hfill \end{array}$$

2.3. Infinitely Repeated Games

Electricity market auctions are conceptualized as an infinite series of identical static games, each incorporating a discount factor $\gamma$ that reflects the present value of future payoffs. This factor becomes crucial for generator companies (GenCo) g, as their present value of payoffs is determined by the discounted sum ${\sum }_{t=1}^{\infty }{\gamma }^{t-1}{r}_{gt}$. The closer $\gamma$ is to 1, the more significant the weight of future payoffs, signifying greater patience from the generator. When these static games are consistent over time, they constitute an infinitely repeated game $\Gamma (\infty ,\gamma )$. Gibbons’ [11] analysis of the infinitely repeated Prisoner’s Dilemma reveals that the grim-trigger strategy, characterized by initial cooperation and subsequent defection upon betrayal, constitutes a Nash Equilibrium when $\gamma \ge \underline{\gamma }$, with $\underline{\gamma }$ being the critical discount factor dependent on the payoffs from cooperation, betrayal, and mutual defection. The Folk Theorem [12] is that in such scenarios, highly patient players can secure higher payoffs than in single-stage games, with cooperative strategies emerging as Nash Equilibria as $\gamma$ approaches 1.

2.4. Nash Equilibrium and Collusion Equilibrium

We categorized observed episodes and agent strategies along a spectrum from competitive to collusiveby defining an episodic collusion measure. In the context of a Markov game, a collection of agent policies is identified as competitive, or a Nash equilibrium, if no agent i can unilaterally select a different policy to improve their expected total episode profit, given the fixed strategies of their opponents, as expressed by the maximization of individual expected rewards:

$$\max E\left[r_i\right].$$

(12)

Conversely, the same collection of policies is deemed collusive, or a monopolistic equilibrium, if it aims to maximize the expected total profit across all agents, which is the maximization of the sum of expected rewards for all agents:

$$\max E\left[\sum _{i=1}^n{r}_i\right].$$

(13)

3. Results

3.1. Setting

We employ multiple Deep Deterministic Policy Gradient (DDPG) agents to simulate an electricity market, where each agent independently learns its own policy, considering other agents as part of the environment. Centralized training, opponent modeling, and communication can improve convergence but require knowledge of other agents’ actions or policies, conflicting with market realities. We focus on the market strategy of each generator (gen) without prior knowledge of other gen’ behaviors, thus opting for an IQL-like architecture. Moreover, considerable empirical evidence indicates that such architectures are often effective in practical scenarios, a finding our experimental results will corroborate.

The process of employing multiple Deep Deterministic Policy Gradient (DDPG) agents to simulate an electricity market is delineated by DDPG. In this context, distinct agents are represented by the subscript g for their respective parameters. It is posited in this study that the strategic variable ${\alpha }_{gt}$ is allowed to vary without constraints within the interval of 0 to $3{\alpha }_g^m$, represented as ${\mathcal{A}}_g=[0,3{\alpha }_g^m]$. As given by the equation, the permissible bounds of the strategic variable require the output a of the actor network, which ranges from −1 to 1 as per the hyperbolic tangent function $tanh\left(x\right)$, to be scaled. This is shown in Figure 1.

$${\alpha }_{gt}=(a_{gt}+1)·{\displaystyle \frac{3}{2}}{\alpha }_g^m$$

(14)

The capacity of the replay buffer is established at $T_{\mathrm{cap}}=1000$. Training of the agents is initiated once the replay buffer reaches its full capacity. For the agent’s exploration mechanism, Gaussian noise is utilized. The default hyperparameters for the Deep Deterministic Policy Gradient (DDPG) algorithm are as follows: $\gamma =0$, $N=64$, $\tau ={10}^{-2}$, ${\mathrm{lr}}_a={10}^{-3}$, ${\mathrm{lr}}_c={10}^{-3}$, and T = 10,000 characterized by:

$$n_{gt}\sim \mathcal{N}\left(0,{\sigma }_t^2\right)$$

(15)

The ${\sigma }_t$ is initialized at a value of 1, and it is designed to gradually decrease to 0.02, with a minimum threshold established to prevent excessively small values from occurring once the training phase begins. The decay of ${\sigma }_t$ is governed by the following piecewise function:

$${\sigma }_t=\left\{\begin{array}{cc}1,\hfill & 0<t\le T_{\mathrm{cap}}\hfill \\ max(0.{999}^{t-T_{\mathrm{cap}}},0.02),\hfill & T_{\mathrm{cap}}<t\le T_{\mathrm{tra}}\hfill \end{array}\right.$$

(16)

In this context, the termination point for training, denoted by $T_{\mathrm{tra}}$, is set to $T_{\mathrm{tra}}=9000$. Once the value of t surpasses $T_{\mathrm{tra}}$, the training and the addition of noise are ceased to allow for the observation of the network’s unaided output, marking the transition into the testing phase. For a detailed account of the parameters of the three-node power market model, reference is made to [13]. The IEEE three-bus system model is depicted in Figure 3. Marginal cost prices for Gen are 17.5 USD/MWh and 20 USD/MWh, respectively. The max power for the line of Bus 1 to Bus 3 is 25 MWh.

3.2. Equilibrium Result

Within the electricity market model, we deployed both the Particle Swarm Optimization (PSO) and the Deep Deterministic Policy Gradient (DDPG) algorithms to simulate two distinct equilibrium scenarios: Nash equilibrium and collusive equilibrium. Under the Nash equilibrium, the PSO algorithm achieved an equilibrium point of (22.70, 24.65) as depicted in Figure 4, whereas the DDPG algorithm arrived at (22.59, 24.83) as shown in Figure 5. In the context of collusive equilibrium, the PSO algorithm determined the point (28.93, 29.58) presented in Figure 4, whereas the DDPG algorithm’s convergence was to (28.81, 29.14) illustrated in Figure 5. These results demonstrate the algorithms’ capability to converge to equilibrium points within their respective scenarios, with the DDPG algorithm exhibiting tighter convergence in the collusive equilibrium scenario. This performance is attributed to DDPG’s adaptability and robustness in managing continuous action spaces and high-dimensional state spaces. Consequently, the DDPG algorithm emerges as a potent tool for pinpointing equilibrium points within the electricity market model, thereby providing precise decision support for market participants. Additionally, Figure 6a,b indicate the presence of optimal collusion when blocking occurs.

3.3. Collusion Effect

Figure 7 demonstrates the substantial influence of collusive behavior on the profits of two power-generating units within the electricity market. Under collusive equilibrium conditions, these units enhance their profits by decreasing their generation levels and increasing the nodal marginal price. This conduct contravenes the tenets of market competition, potentially diminishing market efficiency and curtailing consumer surplus. Given that collusive behavior could lead to market power abuse and unfair competition, it is imperative for regulatory bodies to intensify their oversight to safeguard market fairness and transparency.

4. Conclusions

In order to solve the limitation of not fully considering the collusion equilibrium scenario in the current electricity market research, this study introduces the Deep Deterministic Policy Gradient (DDPG) algorithm to solve the collusion equilibrium in the electricity market. It is verified by the IEEE three-bus model. The results of the DDPG algorithm and PSO algorithm show that both algorithms can converge to market Nash equilibrium and collusion equilibrium. Further observation shows that when the units collude, the node marginal price and social welfare will be significantly increased, and the profits of the colluding units will be significantly increased. This study improves the understanding of collusive behavior in the electricity market and lays a foundation for regulatory intervention.