A Multi-Agent Reinforcement Learning Method for Cooperative Secondary Voltage Control of Microgrids

Abstract

This paper proposes a novel cooperative voltage control strategy for an isolated microgrid based on the multi-agent advantage actor-critic (MA2C) algorithm. The proposed method facilitates the collaborative operation of a distributed energy system (DES) by adopting an attention mechanism to adaptively boost information processing effectiveness through the assignment of importance scores. Additionally, the algorithm we propose, executed through a centralized training and decentralized execution framework, implements secondary control and effectively restores voltage deviation. The introduction of an attention mechanism alleviates the burden of information transmission. Finally, we illustrate the effectiveness of the proposed method through a DES consisting of six energy nodes.

1. Introduction

Recently the evolution of renewable energy sources has facilitated the transformation of the traditional power system. While distributed energy systems offer advantages such as low pollution and high energy efficiency, their large-scale integration into the traditional grid has posed new challenges for stable operation [1]. Wind and solar power exhibit fluctuations and intermittency, while the integration of power generation equipment with power electronics results in low immunity and weak support, adversely affecting the reliability of the power supply in the system. A microgrid is an effective method for utilizing distributed energy resources. It refers to a regional autonomous power system that comprises distributed energy sources (DES), energy storage devices, loads, and other components [2]. The control and coordination of microgrids are crucial for ensuring a high-quality power supply and enabling the integration of renewable energy sources. Microgrids are usually linked to the grid through public connection points and operate in grid connection mode. However, in cases where the main power grid experiences a failure, the microgrid will disconnect and operate autonomously, which is called islanding mode. In practice, the PMU (Phasor Measurement Unit) can collect real-time and accurate data from various distributed energy sources and other equipment in the microgrid, including voltage distribution, power flow, and other key parameters [3]. Deploying a synchronous micro-PMU on the feeder can achieve better voltage stability control because its precise synchronous measurement is conducive to the implementation of advanced control algorithms. In addition, real-time and accurate measurement of the voltage amplitude of the entire network can enhance control performance and improve the robustness and stability of the system in the face of interference [4]. Usually the microgrid adopts a hierarchical control architecture [5], enabling controllers at each level to work independently at different time scales [6]. The main function of the first-level control, also known as local control of the inverter, is to maintain system voltage and frequency stability while ensuring V-f stability and achieving automatic power sharing within a few seconds, which determines the stability of the microgrid [7,8]. The secondary control aims to formulate minute-scale control strategies based on the measured error values [9,10]. Finally, the optimization management layer, or the third-level control, is responsible for controlling the hourly economic operation and energy management hierarchy of the microgrid. Since the droop control commonly used in the first-level control is essentially proportional control, voltage values may deviate from the reference. Therefore, the implementation of secondary control becomes necessary for the purpose of maintaining the system voltage within a predetermined range.

Traditional centralized control relies on microgrid central control (MGCC) and requires a high communication bandwidth between agents. When the MGCC fails, the control process cannot continue, resulting in poor reliability. In addition, decentralized control is based on local information and local communication to obtain neighbor information and formulate collaborative control strategies. It enables a plug-and-play microgrid without the need for a complex communication network [11]. The control techniques for microgrids focused on distributed cooperative control are based on consensus theory. Bidram and Lewis [12] proposed using input–output feedback linearization to linearize the dynamic characteristics of inverters, further transforming the secondary control problem into a consensus tracking problem of a linear system, in which each agent interacts with its neighbors, formulating its own strategy based on the local state and observations of neighbors. In addition, model-based control methods such as nonlinear control [13], optimal control [14], and model predictive control (MPC) [15] have been extensively studied. These methods of voltage control are under the framework of a multi-agent system (MAS) [16], where the average measured voltage value is shared through a distributed communication protocol; the PI controller of each DG generates a control signal based on the error value to eliminate the relevant steady-state errors and reach consensus. However, an inverter-based large-signal dynamic model is nonlinear, and simplifying it may lead to the problem that the control strategy is not applicable after the disturbance produces a large deviation.

Data-driven algorithms have received widespread attention on account of their high scalability and flexibility such as reinforcement learning (RL) [17]. It goes through a process of trial and error and experiences accumulation [18], exploring optimal control strategies from a large number of samples, which are provided by simulated data or historical measurement data. Therefore, they have been widely applied in the power system [19,20]. The combination of deep neural networks with RL has enabled the approximation of the inputs and outputs of complex systems, which is known as deep reinforcement learning (DRL). It has demonstrated significant benefits in addressing large-scale challenges, and high-dimensional, and highly complex problems. Reference [21] proposes a self-controlled method based on soft actor-critic (SAC) to improve renewable energy consumption rates while performing real-time active power dispatch using observed states under stable operation. During system operation, RL can promote the integration of more renewable energy into the power system, ultimately achieving carbon neutrality goals.

Recently, multi-agent reinforcement learning (MARL) has been proposed for secondary control [22,23,24,25]. As agents interact with the environment and learn offline by simulating other agents’ strategies to cooperate, they can find the optimal strategy. After training, agents can adapt well to real-time decision making in unknown power grids or microgrids. Grid Mind [22] measures power flow data on smart grid transmission lines through DDPG and formulates a reward function to punish agents that deviate from voltage levels to learn optimal control strategies and change voltage set points. For the dynamic distribution network management system, Reference [23] uses the maximum entropy reinforcement learning framework for voltage control and improves resilience to controller failures by introducing a communication consensus strategy to reduce network losses. Reference [24] solves decentralized voltage control in an active distribution network through an attention-based MATD3 algorithm, improves communication efficiency, and can effectively handle drastic voltage fluctuations caused by rapid photovoltaic power generation while achieving fast decision making. Reference [25] uses the DDPG algorithm to jointly achieve reasonable distribution of reactive power and adjust bus voltage based on static var compensator (SVC), effectively reducing the number of transformer tap changes.

The deficiencies in the existing research make us consider the introduction of MARL to solve the problem of microgrid cooperative secondary voltage control, including policy scalability, cooperation and coordination, nonstationarity and uncertainty, and efficient communication problems [26,27]. The main achievements of this research are as follows. Firstly, we develop a multi-agent system (MAS) model of an inverter-based microgrid according to its dynamic characteristics, and the secondary voltage control problem is modeled as a networked partially observable Markov decision process (POMDP) observable by multiple agents. Then, we develop an efficient multi-agent actor-critic (MA2C) algorithm by introducing attention mechanisms to deliberately concentrate on particular information, guiding the update of action networks for better training results and improved communication efficiency. An action smoothing mechanism is employed to improve voltage stability. Finally, experiments are conducted on an IEEE 34-node system, demonstrating the proposed control strategy effectively restores terminal voltages of the microgrid to their reference values.

The rest of this paper is outlined as follows. In Section 2, we present the construction method of a multi-agent model for microgrids. In Section 3, we introduce an attention mechanism into an MA2C algorithm for voltage control, effectively improving the efficiency of communication. Moreover, we provide a specific solution process for an MA2C algorithm based on the attention mechanism. In Section 4, simulation based on the improved IEEE 34-node system is presented. Finally, we summarize the main research work of this article and introduce further research directions.

2. Problem Formulation

The DES of the microgrid might be considered an agent, and therefore, using MARL to maintain voltage stability can effectively improve the system’s dynamic performance. This section first introduces the basics of POMDP in MARL and then establishes a multi-agent reinforcement learning model for the secondary voltage control problem of an islanded microgrid.

2.1. Preliminaries for POMDP

The learning process of RL is essentially a trial-and-error process, where agents interact with the environment, exploring unknown states. By capturing immediate rewards from the environment, the agent adjusts its action policy, aiming to maximize the objective function and find the optimal strategy. This way of learning can be formalized as a Markov decision process [28]. A decentralized framework is used to build the architecture of the inverter-based microgrid. We establish a decentralized partially observable Markov decision process (Dec-POMDP) [29] for the agent because the agent can only obtain information by observing its own state and communicating with neighbors. In other words, each distributed generator (DG) realizes voltage stability under the condition of incomplete information.

The system model is typically characterized as a tuple $\left\{N,S,A,P,R,O,\gamma \right\}$. The topology can be abstracted into a multi-agent network $G=(V,ϵ)$, N agents can be regarded as a set V of vertices, and $ϵ$ represents the edges. S represents the global state space, $A=[A_1,A_2\dots A_N]$ is the joint action space, and the action value in this paper is the discrete secondary voltage set value, which constitutes the action space. The actual grid environment is abstracted into a transformation model, and the agent can realize the operation in the grid by interacting with the environment. It is described using a state transition function $P:S\times A\times S\to [0,1]$, and $O=[O_1,O_2\dots O_N]$ is the observation values of the agents. The global reward is $R=O\times A$. Moreover, $\gamma \in [0,1]$ is the discount factor used to calculate the cumulative reward to be maximized.

Figure 1 depicts the classic reinforcement learning training process. The agent obtains the observation value at each time step t and then calculates the action value as its strategy, which is then applied to the environment. Then the agent enters the next state according to the state transition probability and obtains immediate rewards. The agent needs to maximize the reward $R_t^{\pi }={\sum }_{\tau =t}^{\infty }{\gamma }^{\tau -t}{r}_{\tau }$, and under a given state and policy, the expected value of the return that an agent can obtain is $Q^{\pi }(s,a)=\mathrm{E}\left[R_t^{\pi }\right|s_t=s,a_t=a]$. The Q function denotes the expected cumulative return that an agent can obtain after taking a certain action in a given state. In the context of a specified strategy, the optimal policy function ${\pi }^{*}(a,s):a\in argma{x}_{a{}^{\prime }}{Q}^{*}(s,a{}^{\prime })$ could maximize the cumulative expected return. Each interaction is referred to as a step. An episode contains complete interaction steps.

Deep reinforcement learning (DRL) can learn control policies through interaction with the environment. In recent years, three general frameworks for the DRL algorithm have been used. The first is the value iteration method, such as Deep Q-Network (DQN) [30], which approximates the value function using deep neural networks (DNNs). The second is policy gradient algorithms, which directly learn parameterized policies that are suitable for continuous high-dimensional action spaces. Finally, the framework used in this paper is the advantage actor-critic (A2C) framework [31], which trains a value network with parameters $\theta$ to accurately estimate the advantage function $A^{\pi }(s,a)=Q^{\pi }(s,a)-V^{\pi }\left(s\right)$ to guide the training of a policy network with parameters $\omega$. Training the value network is equivalent to policy evaluation, while training the policy network is equivalent to policy improvement.

The loss function for policy network update is

$$L\left(\theta \right)=-\frac{1}{\left|B\right|}\sum _{t\in B}log{\pi }_{\theta }\left(a_t\right|s_t)A_t$$

(1)

Likewise, the loss function that guides the update of the value network is

$$L\left(\omega \right)=\frac{1}{2\left|B\right|}\sum _{t\in B}{(R_t-V_{\omega }\left(s_t\right))}^2$$

(2)

Each minibatch $B=(s_t,a_t,s_t{}^{\prime },r_t)$ contains an experience trajectory, and each return is estimated as $\widehat{R_t}={\sum }_{\tau =t}^{t_B-1}{\gamma }^{\tau -t}{r}_{\tau }$, where $t_B$ is the final step processed in the minibatch. Experience replay involves using a replay buffer B. At each time step, agents randomly select a minibatch from the experience playback buffer, which is used as the training data for this process. This trick allows the past experience to be reused and overcome problems with correlated data and nonstationary distribution in the experience data.

2.2. Microgrid Voltage Secondary Control

Figure 2 shows the large-signal model structure of the distributed generator. Distributed generators in microgrids typically require power electronic converters for grid integration, such as inverters that convert DC to AC based on the power invariant characteristic. In order to ensure voltage stability, voltage-controlled voltage source inverters (VCVSI) are commonly used, with a capacitor as an energy storage component on the DC side. Voltage source inverters usually adopt a double-loop control structure [32]. The outer loop is a functional loop, and the inner loop regulates the current, with a faster dynamic response than the outer loop.

According to the purpose of the outer loop control, common inverter control methods include PQ control, V-f control, and droop control. In this paper, we select the most widely used droop control [33], which does not require communication and automatically controls voltage and frequency changes by simulating the synchronous generator’s frequency characteristics. Its basic principle is that due to the low inertia of inverter-based DG, the operating curve satisfies the linearity property. Based on the inertia of synchronous generators, the voltage can be adjusted by changing the operating point on the droop curve.

In a microgrid, the transmission lines are typically relatively short, with low impedance and inductive circuits. The output of DES is primarily determined by the active and reactive power, and the voltage amplitude. Therefore, it is possible to achieve decoupled control of reactive power and voltage. Considering that the angle of attack and frequency are directly related, and frequency is the first derivative of the voltage phase difference over time, which is easy to detect, the angle frequency is used instead of the angle of attack, resulting in a more common droop control formula as follows.

$${\omega }_i={\omega }_{ni}-m_{P_i}{P}_i$$

(3)

$$v_{o,magi}^{*}=V_{ni}-n_{Q_i}{Q}_i$$

(4)

where $v_{o,magi}^{*}$ is the inverter terminal voltage amplitude for the internal loop, ${\omega }_i$ denotes the angular frequency, and ${\omega }_{ni}$, $V_{ni}$, respectively, refer to the primary control references. $m_{P_i}$, $n_{Q_i}$, respectively, signify the droop coefficients of active and reactive power. Subsequently, applying the Park transformation, the voltage and current in the original three-phase reference frame are converted into the $d-q$ reference frame. In order to facilitate the calculation in the next step, the amplitude of voltage typically aligns with the d-axis of these common frames, resulting in $v_{odi}^{*}=V_{ni}-n_{Q_i}{Q}_i$. The voltage compensation problem for a microgrid with N DGs can be transformed into a reinforcement learning problem for multi-agents.

It is important to acknowledge that droop control has the defect of error. Load changes can lead to a voltage deviation between them and the true value, and differences in inverters’ parameters and the connecting lines can cause unreasonable power distribution, even leading to issues such as circulating current and power reversal. In order to compensate this deviation and improve power quality, we study the microgrid distributed secondary control strategy. First, it is essential to model and analyze the microgrid. Figure 2 shows how we abstract the physical structure of the VCVSI-based DG through large-signal mathematical analysis [34], which consists of a DC power source, a three-phase inverter bridge, the power control loop, and the internal dual loop controllers, LC filter, and an output interface. Since the switching frequency of the inverter is quite fast, the dynamic characteristics of the three-phase inverter can be ignored, and we usually use an ideal DC voltage source in theoretical analysis. Reference [12] provides a detailed introduction on how to build impedance links in a large-signal model of DG, which is used to construct the improved IEEE 34 bus test system platform.

The secondary voltage control aims to synchronize $v_{o,magi}$ to its reference $v_{ref}$ [35]. Based on the above analysis, it can be inferred that the synchronization of the voltage amplitude of VCVSI $v_{o,magi}$ can be achieved by adjusting $v_{odi}$. Consequently, the next problem we need to consider is how to select an appropriate control input $V_{ni}$ to synchronize $v_{odi}\to v_{ref}$.

Model-based methods have many limitations in practical applications. Renewable energy sources have the characteristic of uncertainty, making it difficult to perform detailed modeling of each element in a microgrid. The equipment in the microgrid usually belongs to different equipment agents. Agents need to efficiently collect and process different sensor data, and deploying multiple synchronized sensors in the system that can monitor system data such as frequency or power flow is essential for a multi-agent environment [36].

In order to maintain the market environment, the information of private users cannot be shared, and only some public data can be obtained and used for regulation. With a large number of new energy sources running in the grid through adaptive changes in control methods and parameters, the dynamics of microgrids are becoming increasingly fast-paced.

To address this issue, The data-driven MARL method has received widespread attention in the field of the smart grid due to avoiding the problem of model inaccuracy. Microgrids containing DGs can be abstracted into a cyber–physical system, which includes a topology demonstrated by a directed graph $G=(V,E)$. $V=\{v_1,v_2,\dots ,v_N\}$ represents a nonempty set of N nodes or the set of DGs. In the isolated microgrid, each DG can be viewed as a node in the communication graph. Communication lines are represented by a set of edges $ϵ\in E$, with the adjacency matrix $A_{ij}$. $(v_j,v_i)$ represents an edge from node j to node i. We consider a time-invariant communication topology, so $A_{ij}$ is constant, and therefore Aij is constant. Aij is the weight of edge $(v_j,v_i)$, and $A_{ij}\ge 0$ if $(v_j,v_i)\in E$, otherwise $A_{ij}=0$.

Therefore, in this paper, based on local observations and limited communication, the DG secondary voltage control process is described as a multi-agent extended version of the Markov decision process (MDP), namely POMDP. Figure 3 depicts the schematic diagram of the secondary voltage control process of the microgrid system using the MARL framework, in which the microgrid includes two levels of information and physics. DES in the information layer is regarded as an agent. It not only needs to obtain its own state value and reward value, but also needs to exchange state values with its neighbors through the communication network. The obtained information is aggregated into the observation value of the agent through the attention mechanism. After processing, encoding, decoding, and other steps are sent as input to the respective action network and evaluation network, and then the strategy and the approximation of the value function are obtained. The final output action value realizes the secondary voltage control, and further interacts with the environment regarded as the physical layer to generate the next state. The relevant elements of the proposed MARL algorithm considered in this paper are as follows.

(1) Action Space: We select the voltage setting value in the droop control as the action value of the agent, and obtain 10 different action values on average from $1.00$ to 1.14 p.u. The set of all agent actions is expressed as $a=[v_{n1}\times v_{n2}\times \dots \times A_N]$.

(2) State Space: In order to reduce the communication pressure, we select as few parameters as possible to describe the operating state of the system as state variables while ensuring sufficient information, and obtain a state space containing necessary information as follows

$$s_{i,t}=({\delta }_i,P_i,Q_i,i_{odi},i_{oqi},i_{bdi},i_{bqi},v_{bdi},v_{bqi})$$

(5)

where ${\delta }_i$ is the phase angle difference between the reference coordinate system of each DES and the common reference coordinate system; $P_i$ and $Q_i$ represent the active and reactive power, respectively; $i_{odi}A$, $i_{oqi}A$ denote the d-axis component and q-axis quantity of the output current after passing through an LC filter. Meanwhile, $i_{bqi}A$, $i_{bqi}A$, $v_{bdi}kV$, $v_{bqi}kV$ represent the quadrature axis component of the voltage and current on the line connected to the main grid. The total state space is represented as the Cartesian product of individual agent’s states $S\left(t\right)=s_1\times s_2\times \dots \times s_N$.

(3) Observation Space: Assuming that the communication between agents is limited to neighbors, then the observations of each agent $o_{i,t}=s_{i,t}\cup m_{i,t}$ include observations from neighbors $m_{i,t}$ in addition to its own state values. We elaborate on how to generate $m_{i,t}$ using attention mechanisms in Section 3.

(4) Transition Probability: The transition probability $T(s{}^{\prime }\mid s,a)$ describes the dynamic behavior of the agent in the DES system. We follow the model proposed in [35] to build an environment for simulating microgrid operating conditions. The model parameters we use in this process are not involved in the process of designing the MARL algorithm in the next section because our method is data-driven.

(5) Reward function: A well-designed reward function provides optimal guidance for learning the best strategy. Reference [37] only uses $v_{odi}$ to calculate the reward function. Although the voltage of DG can be adjusted, the bus voltage may still exceed the normal range. Therefore, we consider the bus voltage value, $v_{bus}$, in the reward function and define an average relative error $error=\frac{1}{n_{bus}}{\displaystyle \sum _{i=0}^{n_{bus}}}\frac{\left|v_{nom}-v_{bus,i}\right|}{v_{nom}}$. We designed a reward function that includes error of DGs to help us quickly converge to the reference value (e.g., 1 p.u.) during the training process.

$$r_{i,t}=\left\{\begin{array}{cc}\left|error-0.05\right|\times n_{agent},& error\le 0.05\\ -\frac{(error-0.05)}{4}\times n_{agent},& 0.05<error\le 0.25\\ -10\times n_{agent},& otherwise\end{array}\right.$$

(6)

where $v_{nom}$ represents the nominal bus voltage value of the VCVSI, $r_{i,t}$ denotes the immediate reward obtained by agent i. Three working areas are set according to the level of voltage, including normal operating conditions ($error\le 0.05$), heavy load region ($0.05<error\ge 0.25$), and divergence region ($error\ge 0.25$). DGs that lead to voltage divergence or no power flow solution will receive larger penalties, while DGs whose voltage amplitude is closer to the set value $1 \mathrm{p}.\mathrm{u}.$ will receive higher rewards.

3. Methods

In the previous section, we describe the necessary elements required to formulate the voltage control problem into a MARL framework in detail. In this section, a multi-agent advantage actor-critic (MA2C) algorithm is proposed to solve this POMDP, which is an improvement on the IA2C algorithm in two ways. Firstly, each agent can obtain the policies of adjacent agents, including observable states and fingerprints. Secondly, a spatial discount factor is introduced to reduce rewards for distant agents, enabling agents to focus more on nearby situations. With these improvements, local rewards, value loss, and policy loss all consider the influence of neighboring agents. The method we proposed has the characteristics of data-driven, centralized training, decentralized execution, and rule integration, meeting the requirements for smart grid operation. Finally, an attention mechanism is introduced to improve communication efficiency.

3.1. MA2C Algorithm

In this section, we first introduce the independent advantage actor-critic (IA2C) algorithm. In IA2C, there is no consideration of information exchange between multiple agents, and each agent trains its own critic network $V_{\omega i}$ and actor network ${\pi }_{{\theta }_i}$, approximated by deep neural networks. We further supplement the definition of POMDP defined in Section 2. If the neighbor of agent i is $j\in N_i$, the local neighbor set can be written as $V_i=N_i\cup i$. Furthermore, we define the corresponding local reward, policy loss, and value loss for each agent.

The A2C framework is the mainstream architecture in RL. The independent A2C (IA2C) algorithm that emerges on this basis does not consider the information interaction between multiple agents. Under the assumption that the global reward and state can be shared between agents, each agent trains its own policy network and value network independently. The multi-agent A2C (MA2C) takes into account the information interaction between agents; that is, each agent can observe the action value, reward value, and state value of the neighbor agent, so the observability of each local agent improves and facilitates multi-agent cooperation and coordination. The local reward of centralized A2C, which is an extension of IA2C, can be written as

$$R_{t,i}=\widehat{R_t}+{\gamma }^{t_B-t}{V}_{{\omega }_i^{-}}(s_{t_B}\mid {\pi }_{\theta }{}_{-i}^{-})$$

(7)

The environment we define is partially observable, so each agent can only communicate with agents in its own neighborhood. In other words, the inputs of the actor network and critic network should be replaced with $s_{t,v_i}={s_{t,j}}_{j\in v_i}$ instead of $s_t$. Therefore, the loss function formula for the value network should be written as:

$$L{\left(\omega \right)}_i)=\frac{1}{2\left|B\right|}\sum _{t\in B}{(R_{t,i}-V_{{\omega }_i}(s_t,v_i))}^2$$

(8)

Similarly, the advantage function should be rewritten as $A_{t,i}=R_{t,i}-V_{{\omega }_i^{-}}(s_t,v_i)$. Therefore, the loss function formula for the policy network should be written as:

$$L\left({\theta }_i\right)=-\frac{1}{\left|B\right|}\sum _{t\in B}log{\pi }_{{\theta }_i}\left(a_{t,i}\right|s_t,v_i)A_{t,i}$$

(9)

For such a multi-agent environment, using traditional reinforcement learning algorithms separately for each agent is not suitable because from each agent’s perspective, the environment is nonstationary. Therefore, we follow the actor-critic framework and propose two improvements to stabilize the convergence of IA2C and enhance its fitting ability.

Firstly, we incorporated neighboring policy information to enrich the observation of agents. MA2C considers the information interaction among agents, allowing agents to acquire the actor network parameters of its adjacent agents, thus enhancing the observability of each agent and facilitating better coordination among them. As the policy information in the A2C framework is explicitly parameterized in the policy function ${\pi }_{{\theta }_{-i}^{-}}$, and considering that the power system state remains similar in the policy between the current and previous time steps, we only consider short-term policy sharing among neighbors. Thus, we use the current local neighborhood state $(s_t,v_i)$ and the latest policy sampled from neighboring agents ${\pi }_{t-1,N_i}={\left[{\pi }_{t-1,j}\right]}_{j\in N_i}$ as inputs to the DNNs [38], and the locally sampled policy is represented as

$${\pi }_{t,i}={\pi }_{{\theta }_i^{-}}(·\mid s_t,v_i,{\pi }_{t-1,N_i})$$

(10)

Secondly, to adjust the degree of cooperation among agents and reduce the impact of state and reward from neighbors, a spatial discount factor $\alpha$ is introduced into the calculation of the global reward value

$${\tilde{r}}_{t,i}=\sum _{d=0}^{D_i}\left(\sum _{j\in V\mid d(i,j=d)}{\alpha }^d{r}_{t,j}\right)$$

(11)

The discount factor $\alpha$ is different from the time discount factor $\gamma$, as it reduces signals according to spatial order rather than temporal order. Here, $D_i$ is the maximum Euclidean distance to agent i. After the discount, the global reward of each agent is no longer the same but adjusted according to the distance of neighbors, so that the control strategy achieves a more appropriate balance between exploration and utilization. Similarly, we discount neighboring states using $\alpha$

$${\tilde{s}}_{t,v_i}=\left[s_{t,i}\right]\cup \alpha {\left[s_{t,j}\right]}_{j\in N_i}$$

(12)

The discounted estimation for local reward should be represented as ${\widehat{R}}_{t,i}={\sum }_{\tau =t}^{t_B-1}{\gamma }^{\tau -t}{\tilde{r}}_{\tau ,i}$, therefore the discounted estimated local reward should be represented as

$${\tilde{R}}_{t,i}={\widehat{R}}_{t,i}+{\gamma }^{t_B-t}{V}_{{\omega }_i^{-}}({\tilde{s}}_{t_B},v_i,{\pi }_{t_B-1},N_i\mid {\pi }_{{\theta }_{-i}^{-}})$$

(13)

The loss function of the value network could be modified as

$$L{\left(\omega \right)}_i)=\frac{1}{2\left|B\right|}\sum _{t\in B}{({\tilde{R}}_{t,i}-V_{{\omega }_i}({\tilde{s}}_t,v_i,{\pi }_{t-1,N_i}))}^2$$

(14)

with the fingerprint information ${\pi }_{t-1,N_i}$ as input to the value network, it can capture the impact of ${\pi }_{{\theta }_{-i}^{-}}$, and the spatially discounted rewards ${\tilde{R}}_{t,i}$ are more relevant to the state, making the improved value network more stable. Considering the discounted advantage function ${\tilde{A}}_{t,i}={\tilde{R}}_{t,i}-V_{{\omega }_i^{-}}({\tilde{s}}_t,v_i,{\pi }_{t-1,N_i})$, the policy network’s loss function is written as follows

$$\begin{array}{cc}\hfill L\left({\theta }_i\right)=& \hfill -\frac{1}{\left|B\right|}\sum _{t\in B}(log{\pi }_{{\theta }_i}\left(a_{t,i}\right|{\tilde{s}}_t,v_i,{\pi }_{t-1,N_i}{A}_{t,i})\hfill \\ \hfill & \hfill -\beta \sum _{a_i\in A_i}{\pi }_{{\theta }_i}log{\pi }_{{\theta }_i}\left(a_i\right|{\tilde{s}}_t,v_i,{\pi }_{t-1,N_i})\hfill \end{array}$$

(15)

The second term is a regularization term we added; by introducing an entropy loss of this policy $\pi$, it can increase the exploration of early agents. It can emphasize the marginal effect of ${\pi }_{{\theta }_i}$ on the states of neighboring agents, thereby avoiding the agent from converging to a suboptimal policy.

3.2. Attention Mechanisms

MARL algorithms are challenging to train because of a large number of agents, action spaces, and dimensions, leading to long training times and difficulty in achieving convergence [39]. To overcome this difficulty of training complexity, A voltage control strategy for DG based on MARL and attention mechanism is proposed. The advantage of the soft attention mechanism is that higher rewards might be obtained by selectively focusing on the actions of other nodes. The proposed algorithm can effectively utilize neighbor’s parameter to train own network, and its convergence speed is faster than that of traditional RL.

The attention mechanism originates from researchers’ study of human visual observation and has been widely applied to computer vision tasks [40]. This mechanism can allocate a large attention coefficient to the parts that need to be focused on according to the criticality of the features, allowing the algorithm to adaptively calibrate the extracted feature information for effective training. The soft attention mechanism selectively ignores some information and aggregates the remaining information through weighted summation. This can separate important information and avoid interference from unimportant information, thus improving accuracy.

In this paper, an attention mechanism is introduced into the centralized MA2C. The computation process of the soft attention mechanism includes two steps. The first step is to calculate the weight coefficients based on the query and the key. The second step is to perform a weighted summation of the value based on the weight coefficients.

The input vector $(K,V)=[(k_1,v_1),(k_2,v_2),\dots ,(k_N,v_N)]$ appears in the form of key-value pairs. First, the similarity or correlation weight $a_i$ between the query vector and the key vector is calculated. A scoring function $f(k_i,q)$ is defined to determine the importance score, and the softmax function is used to normalize the original scores as:

$$W_i=softmax(f(k_i,q))=\frac{exp(f(k_i,q))}{{\sum }_{j=1}^{N}exp\left(f(k_j,q)\right)}$$

(16)

After obtaining the attention distribution based on the query vector and key-value pairs, the input information is weighted and summed up according to the attention weights as:

$$V={\sum }_{i=1}^N{W}_i·v_i$$

(17)

We embed the soft attention mechanism into the communication network, and the structure of the algorithm is shown in the Figure 4. h is the hidden layer, $m_i$ is the observation of each agent, and the global information $M_i$ is obtained after encoding and decoding by the neural network. Since the neural network is a fully connected layer structure, there is no distinction between the importance of information. In order to improve the efficiency of communication, we introduce the attention mechanism into the action network to selectively process observations, and finally input them into the policy network to participate in the update and obtain new action values. The structure that handles observation messages is called a message coordinator, which contains a predefined importance score. The superscript q constitutes the query feature space, and the superscript k constitutes the key-value feature space. The above formula shows that the importance score of agent j to agent i is obtained through query and key, and also needs to go through the softmax function. The sum of each agent’s importance scores relative to all neighbors should be 1.

$$W_i^j={{(}_i^q)}^T\left(m_{ij}^k\right),i\ne j$$

(18)

$$W_i=[W_i^1,\dots ,W_i^j,\dots ,W_i^N]=softmax(W_i^1,\dots ,W_i^j,\dots ,W_i^N)$$

(19)

$$\sum _{j=1}^N{W}_i^j=1,i\ne j$$

(20)

After calculating the importance scores, the global message of each agent can be obtained by taking the weighted sum of the importance scores of all its neighbors observations.

$$M_i=\sum _{j=1}^N{W}_i^j{m}_j$$

(21)

where $W_i^j$ is the weight of $m_j$ for $m_i$. The soft attention mechanism enables the agent to selectively process information during the decision-making process, and filter out information that does not affect the strategy by focusing on specific observations. This mechanism is fully differentiable, capable of end-to-end training of model parameters, suitable for gradient backpropagation of neural networks and combination with MARL. In addition, the measurement method of the soft attention mechanism is used to calculate the attention weight, which can be interpreted as a correlation measurement. This method has strong interpretability and provides a reference basis for modifying parameters and identifying data. All in all, the soft attention mechanism is suitable for MARL-based microgrid control due to its differentiability, interpretability, and robustness.

The complete MA2C algorithm embedded with attention mechanism follows the procedure outlined in Algorithm 1. The hyperparameters include the discount factor for distance decay $\alpha$, the learning rate for the actor network ${\eta }_{\omega }$, the learning rate for the critic network ${\eta }_{\theta }$, the time discount factor $\gamma$, the time step length for each episode L, and the total training episodes S. To clarify further, the agents participate in interactions with the environment for a substantial number of epochs (lines 2–25). In lines 4–7, agents shared their state information and neighbor information through the message coordinator network, where importance scores are computed using a set “query” and “key” feature space. Then each agent state and policy information are reassembled and encoded, processed with an attention mechanism, and sent as a message $m_{i,t}$ to its neighbors (line 11). Subsequently, the agents modify their states and the parameters of the policy network (lines 9–10). The state values are updated (lines 12–15). Afterwards, actions are applied to the environment (line 16), the agent transitions to a new state and immediately receives a reward (line 17), which will be included in the policy experience buffer (line 18). The policy gradient algorithm is employed to update these network parameters after each episode, by randomly selecting trajectories from the policy experience buffer (lines 20–25). If an episode is completed or there is no power flow solution, a DONE signal is flagged. Once the DONE signal is received, resetting all agents’ states to their initial states signifies the beginning of a new epoch.

The centralized training, decentralized execution approach is employed in our method [41]. In centralized training, the critic network evaluates and guides the update of the actor network by paying attention to valuable local observations in the environment, evaluating the data collected during decentralized execution, and using gradients to guide the actor network’s parameter updates, thus maximizing the overall expected return. During decentralized execution, the actor networks are controlled separately. Upon completion of the training process, the DNN parameters are set as constant, while the policy networks of agents are retained to facilitate real-time voltage regulation.

Algorithm 1: AM-MA2C algorithm

4. Simulation

We applied the proposed voltage control method to an improved IEEE 34-bus system (see Figure 5). The network consists of 6 DGs and a topology structure.

Table 1. Key configuration parameters of DG and load of the IEEE 34-node system.

		DG1, DG2, DG3, DG4		DG5, DG6
DG	$m_P$	$5.64\times {10}^{-5}$	$m_P$	$5.64\times {10}^{-5}$
	$n_Q$	$5.2\times {10}^{-4}$	$n_Q$	$6\times {10}^{-4}$
	$R_c$	$0.03\mathsf{\Omega }$	$R_c$	$0.03\mathsf{\Omega }$
	$L_c$	$0.35\mathrm{mH}$	$L_c$	$0.35\mathrm{mH}$
	$ome{g}_c$	31.41	$omeg{a}_c$	31.41
	$k_p$	4	$k_p$	4
	$k_i$	40	$k_i$	40
	Load1	Load2	Load3	Load4
Load	$1.5\mathsf{\Omega }$	$0.5\mathsf{\Omega }$	$1\mathsf{\Omega }$	$0.8\mathsf{\Omega }$
	$0.03\mathsf{\Omega }$	$0.017\mathsf{\Omega }$	$0.05\mathsf{\Omega }$	$0.02\mathsf{\Omega }$

presents the key characteristics of the DGs, lines, and loads. Utilizing pandpower, the method was executed in Python to enable power system modeling. The simulation platform was designed with the specifications of the lines and loads detailed in Reference [42]. To simulate real-world conditions of the power system, the microgrid was subjected to random load adjustments, resulting in deviations of $\pm 20\%$ from the expected values. Moreover, we introduced perturbations of $\pm 5\%$ to each load to simulate load disturbances. The monitoring sampling time for all DGs was set to $0.05$ s, and each DG was able to communicate with its adjacent DGs. The primary control at the lower level was implemented using the method proposed in Reference [43].

The discount factor for the state space is defined as $\alpha =0.9$, while the discount factor $\gamma$ is set to $0.99$. The participant learning rate is set to ${\eta }_{\theta }=5\times {10}^{-4}$, and the critic learning rate is set to ${\eta }_{\omega }=2.5\times {10}^{-4}$. Each model is trained for 2000 episodes, and the duration of each episode spans $T=20$ steps. To ensure a fair comparison, a different random seed is generated for each episode, and consistent training environments are ensured across different algorithms through the utilization of a shared random seed. We compare the proposed MA2C method with the MA2C method and the IA2C method to demonstrate its effectiveness.

Figure 6 shows the reward change curve obtained by training of the improved MARL algorithm applied to the IEEE 34-node system. It is evident that the addition of the communication network and appropriate spatial discount factor enhances the effectiveness of sample utilization and accelerates the training speed. The attention-based MA2C algorithm can converge to higher reward values. The obtained policies were evaluated for 20 episodes on various load disruptions after 2000 training episodes, with each agent using the same random seed in each training episode.

To exhibit the performance of our strategy, we simulated the transient process of voltage control in a microgrid system. Assuming an initial voltage of 1.0 p.u. and a frequency of 60 Hz, the simulation time was set to 1.5 s. At 0.4 s, the voltage secondary control was introduced using the MA2C algorithm with attention mechanism, which was trained in advance. The results of the simulation are depicted in Figure 7.

As shown in this figure, from 0 s to 0.4 s, the voltage dropped slightly below the set value due to only using droop control. After introducing the secondary control strategy, the end voltage amplitude of the microgrid was synchronized to 1.0 p.u. after a 0.15 s adjustment time.

We also compared the voltage control performance of different algorithms under high load changes ($\pm 25\%$). The voltages of the six DGs under different algorithms are shown in Figure 8. The horizontal axis from 0 to 5 represents the terminal bus voltage values of the six DGs, and the vertical axis represents the voltage in units. We aim to maintain the voltage of all DGs in the range around 1 p.u. In this context, “r” denotes the average reward over the past five control steps, and the blue dots indicates the magnitude of the voltage value. As shown in Figure 6, all algorithms have the ability to regulate the voltage within the acceptable range. However, these voltage amplitudes are closer to the reference using the proposed algorithm.

In order to assess the efficacy of the proposed methodology, we performed random testing on the model that had undergone training.

Table 2. Average reward value under random test.

Random Seeds	AM-Attention	MA2C	IA2C
Case 1	0.24	0.18	0.19
Case 2	0.23	0.21	0.18
Case 3	0.22	0.21	0.19
Case 4	0.24	0.22	0.19
Case 5	0.22	0.21	0.20
Average reward	0.23	0.21	0.19

exhibits the results of experiments with different random seeds. The average reward of the improved MA2C algorithm with the attention mechanism is 0.23, which is more than that of the MA2C and IA2C algorithms, respectively.

5. Discussion

In this paper, we formulate the secondary voltage control problem as a POMDP process that can be solved through reinforcement learning algorithms. We develop a new collaborative MA2C algorithm with an attention mechanism by introducing a communication network and appropriate spatial discount factor. Comprehensive experiments show that this algorithm is able to restore the DG bus voltage to the reference quickly, with voltage fluctuations that are stable and within the acceptable range. The algorithm gains better results in terms of convergence speed and voltage control performance. Moreover, the proposed control strategy only requires a communication network for exchanging state information between neighboring agents. Consequently, the proposed MARL algorithm can meet the demand for large-scale energy storage supporting the grid.

This indicates the huge potential of applying deep reinforcement learning in microgrid control, while there are also many challenges in scalability, generalization, security, and stability. We will work on building a more complex environment in the future based on real power system data and apply MARL algorithms to relatively complex frequency secondary recovery problems. Microgrid systems are highly cost-sensitive, and if decisions are made incorrectly, voltage violations can lead to serious consequences. Therefore, in the future, we will consider applying safety reinforcement learning to microgrid control.