A Deep Reinforcement Learning Method for Economic Power Dispatch of Microgrid in OPAL-RT Environment

Abstract

This paper focuses on the economic power dispatch (EPD) operation of a microgrid in an OPAL-RT environment. First, a long short-term memory (LSTM) network is proposed to forecast the load information of a microgrid to determine the output of a power generator and the charging/discharging control strategy of a battery energy storage system (BESS). Then, a deep reinforcement learning method, the deep deterministic policy gradient (DDPG), is utilized to develop the power dispatch of a microgrid to minimize the total energy expense while considering power constraints, load uncertainties and electricity price. Moreover, a microgrid built in Cimei Island of Penghu Archipelago, Taiwan, is investigated to examine the compliance with the requirements of equality and inequality constraints and the performance of the deep reinforcement learning method. Furthermore, a comparison of the proposed method with the experience-based energy management system (EMS), Newton particle swarm optimization (Newton-PSO) and the deep Q-learning network (DQN) is provided to evaluate the obtained solutions. In this study, the average deviation of the LSTM forecast accuracy is less than 5%. In addition, the daily operating cost of the proposed method obtains a 3.8% to 7.4% lower electricity cost compared to that of the other methods. Finally, a detailed emulation in the OPAL-RT environment is carried out to validate the effectiveness of the proposed method.

1. Introduction

The use of microgrids has drawn significant attention in recent years due to their potential to improve the stability and reliability of power distribution systems. A microgrid is a small-scale power grid that can operate independently or in connection with the main power grid [1,2]. Each microgrid can not only operate independently using its own control mechanism, but also can integrate the renewable energy resources, isolate the failed grid and save operation costs through peak shaving. A microgrid typically includes a combination of renewable energy resources (RERs), such as a photovoltaic (PV) plant, wind turbine generator (WTG) and battery energy storage system (BESS), as well as distributed energy sources (DESs), such as a gas turbine and diesel generators. Although the microgrid with RER integration provides flexibility and sustainability benefits, it also faces challenges in stable and power dispatch due to the uncertainties of intermittency that are unexpected and dynamic power variations resulting in real-time balancing operation. To solve these problems and ensure the cost effectiveness of microgrid systems, it is essential to minimize the operation costs via economic power dispatch (EPD). The EPD problem arises in microgrid systems and involves minimizing the total fuel cost of power generation, while ensuring that the power demand is met, and the generation limits are respected [3,4,5]. The objective is to optimize the generation schedule of multiple generators in the microgrid, such that the overall fuel cost is minimized without violating any operational constraints. The EPD problem can be formulated with different fuel cost functions, depending on the type of generators used in the microgrid. However, the complexity of the EPD problem increases with the integration of more RERs in the microgrid system. Therefore, it is necessary to develop an efficient and effective method to solve the EPD problem.

The EPD problem has been extensively studied in the literature, and various techniques have been proposed to solve the problem. Traditional methods such as linear programming (LP), quadratic programming (QP), and mixed-integer programming (MIP) have been widely used for solving the EPD problem in power systems [6,7,8]. However, these methods have some limitations, such as high computational complexity and the inability to handle the nonlinearities of the system [9,10,11]. Moreover, metaheuristic algorithms, such as the genetic algorithm (GA), particle swarm optimization (PSO), and differential evolution (DE), have shown good performance in solving the EPD problem in microgrid systems [12,13,14]. Nevertheless, it is worth noting that these methods may converge to suboptimal solutions and suffer from slow convergence rates [15,16,17].

Load forecasting is an important component of microgrid operation, as it can help predict the power demand of a microgrid in advance. Accurate load forecasting allows microgrid operators to plan and allocate resources effectively, which can improve the efficiency and reliability of microgrid operation [18,19]. In recent years, deep learning algorithms have shown promising results in solving various optimization problems in power systems. Long short-term memory (LSTM) networks are a type of recurrent neural network (RNN) that are well-suited for time series data, such as load forecasting. LSTM networks can capture long-term dependencies in time series data and are able to remember information from previous time steps. In this study, an LSTM network is utilized to predict the load demand of a microgrid. The LSTM network is trained on historical load data and can predict the load demand for the next time step based on the current and previous load values [20,21].

Deep reinforcement learning (DRL) has shown great potential for solving complex control problems in various domains. It combines the power of deep neural networks with the principles of reinforcement learning to enable agents to learn complex decision-making policies directly from high-dimensional inputs, without the need for hand-crafted features. Some popular DRL algorithms, such as the deep Q-network (DQN) and deep deterministic policy gradient (DDPG), have been applied to optimize the EPD strategy of microgrids [22,23]. The DQN is a model-free, off-policy algorithm that uses a deep neural network to approximate the Q-value function. The Q-value function estimates the expected total reward an agent will receive by taking a certain action in a certain state. Moreover, the DQN employs experience replay, and this helps to break the correlation between consecutive samples and stabilize the learning process [24,25]. On the other hand, the DQN aims to obtain the Q-value estimates in discrete action space. Therefore, in the continuous action domains, such as the EPD problem, the performance of the DQN is less than outstanding. In addition, the DDPG is an actor–critic algorithm that learns a deterministic policy function and a Q-value function simultaneously. The policy function maps states to actions, while the Q-value function estimates the expected total reward an agent will receive by taking a certain action in a certain state, following the policy function. DDPG uses a target network, a delayed copy of the Q-network and the policy network, to update the networks, which reduces the variance of the gradients and stabilizes the learning process [26,27,28]. Since DDPG can learn the policies in high-dimensional space and possess great potential to process continuous action problems, it will achieve better performance on the EPD problem than the DQN will.

DRL algorithms have several advantages in solving the EPD problem of power systems. Firstly, they can learn from experience and adapt to changes in a microgrid system, making them more flexible and adaptable than traditional methods. Secondly, DRL algorithms can handle stochastic and uncertain conditions, which are common in renewable energy sources such as solar and wind power. Thirdly, DRL algorithms can handle non-linear and non-convex optimization problems, which are often encountered in microgrid systems. Lastly, DRL algorithms can provide near-optimal solutions in a relatively short time, which is important for real-time applications. These advantages make DRL algorithms a promising solution for efficient and sustainable microgrid operation [29,30,31]. Therefore, this study proposes a novel approach that combines LSTM-based load forecasting with the DDPG algorithm to optimize the EPD strategy of microgrid systems. The proposed method uses historical load data to train the LSTM network to forecast the load demand for the next time step. The forecasted load demand is then used as input to the DDPG algorithm to determine the optimal dispatch strategy for the microgrid system. The objective of this research is to minimize the overall fuel cost of a microgrid while meeting the demand and supply constraints.

This study focuses on the EPD of a microgrid system on Cimei Island [32], which is connected to the main grid via submarine power cables. The microgrid system includes PV and WTG sources with a predetermined output power provided by a Taiwan power company (Taipower). To optimize the EPD strategy of the microgrid system, control strategies are developed for the efficient operation of the diesel generator, gas turbine generator, and the BESS to minimize electricity bills. Moreover, the proposed method is implemented using a hardware-in-the-loop (HIL) mechanism developed with OPAL-RT real-time simulator OP4510 and a floating-point digital signal processor (DSP) from Texas Instruments TMS320F28335 for BESS control. The rest of this study is organized as follows. The modeling, formulations and configurations of the microgrid are presented in Section 2, and the DDPG with LSTM method for EPD is proposed in Section 3. Some test cases with different EPD methods are investigated to verify the performance of the proposed methodology in Section 4. The emulation in an OPAL-RT real-time environment is presented in Section 5. Finally, some conclusions are presented in Section 6.

2. Microgrid Model

The microgrid model of Cimei Island [32] is utilized in this study. A single-line diagram of the island is shown in Figure 1, which includes a diesel generator, a gas turbine generator, a 355 kWp PV system, a 1000 kWh BESS, a wind farm with a 305 kW WTG system, loads, and some demand response resources. The BESS is capable of charging or discharging power from/to the microgrid, where the negative and positive signs indicate the charging and discharging power of the BESS, respectively. Moreover, power variations of PV and WTG systems for a day are provided by Taipower. Furthermore, the microgrid is connected to the main grid and actively participates in real-time electricity markets to take advantage of the real-time electricity prices. Thus, the power can flow between the main grid and the microgrid, where the positive sign indicates power consumption in the microgrid, and the negative sign indicates power flowing into the main grid.

Table 1. Three-stage electricity cost of Taipower.

Plan	USD/kWh		Time
On-Peak	USD 0.207	13:00~17:00	18:00~20:00
Half On-Peak	USD 0.207	7:00~13:00	17:00~18:00	20:00~22:00
Off-Peak	USD 0.06	0:00~7:00	22:00~24:00

lists the three-stage electricity rates of Taipower in USD. This study aims to minimize the costs of fuel generators and electricity bills to ensure profitability for microgrid operators while providing additional support to the main grid. The mathematical formulations used to solve this problem are presented below.

2.1. Conventional Power Generator

The microgrid system model, shown in Figure 1, involves the control of the total cumulative power consumption and the peak power to minimize the expenses incurred from fuel generators and electricity bills. The model encompasses a comprehensive representation of the DESs, the BESS, main grid, and power balance, along with their respective physical properties and technical constraints. Detailed descriptions of the model are presented below.

2.1.1. Diesel Generator

To take into account technical constraints, the power output of the diesel generator at each time slot, $t$, is represented as $P_t^{DG}$ and formulated as follows:

$$P_{min}^{DG}\le P_t^{DG}\le P_{max}^{DG}$$

(1)

The minimum and maximum power limits of the diesel generator are denoted as $P_{min}^{DG}$ and $P_{max}^{DG}$, respectively. To calculate the generation cost of the diesel generator, a conventional quadratic cost function is used as follows:

$$C_t^{DG}=[a_d\cdot (P_t^{DG}{)}^2+b_d\cdot P_t^{DG}+c_d]\mathsf{\Delta }t$$

(2)

where $\mathsf{\Delta }t$ is the duration of a time slot; $a_d$, $b_d$ and $c_d$ are positive coefficients.

2.1.2. Gas Turbine Generator

The power output of the gas turbine generator at each time slot, t, is represented as $P_t^{GT}$ and formulated as follows:

$$P_{min}^{GT}\le P_t^{GT}\le P_{max}^{GT}$$

(3)

where the minimum and maximum power limits of the gas turbine generator are denoted as $P_{min}^{GT}$ and $P_{max}^{GT}$, respectively. To calculate the generation cost of the gas turbine generator, a conventional quadratic cost function is used as follows:

$$C_t^{GT}=[a_g\cdot (P_t^{GT}{)}^2+b_g\cdot P_t^{GT}+c_g]\mathsf{\Delta }t$$

(4)

where $\mathsf{\Delta }t$ is the duration of a time slot; $a_g$, $b_g$ and $c_g$ are positive coefficients.

2.2. Renewable Power Generator and Loads

The power generation curves of PV and WTG systems are shown in Figure 2, and they are downsized proportionally to meet the capacity of the systems in this study. These curves are based on the daily average solar and wind power production curves provided by Taipower for the Taiwan area. Moreover, the hourly power output of PV and WTG systems varies according to the corresponding values shown in Figure 2. Let $P_t^{PV}$ and $P_t^{WTG}$ denote the power production of PV and WTG systems, respectively. The load connected to the microgrid varies according to the predicted value of the LSTM load forecasting model and is denoted by $P_t^d$. The net load of the microgrid at time slot $t$ is defined by Equation (5):

$$P_t^{net}=P_t^d-P_t^{PV}-P_t^{WTG}$$

(5)

2.3. BESS

To control the charging or discharging of the BESS in each time slot, $t$, the power output is denoted by $P_t^b$, with negative values indicate charging and positive values indicate discharging. The power output is constrained by Equation (6) to prevent over charging or discharging.

$${-P}_{max}^{b,ch}\le P_t^b\le P_{max}^{b,dis}$$

(6)

where $P_{max}^{b,dis}$ is the maximum discharging power and $P_{max}^{b,ch}$ represents the maximum charging power. Moreover, the BESS has a state of charge (SOC) denoted as $S_t^b$, which follows a certain dynamic behavior described by the following:

$$S_t^b=S_{t-1}^b-P_t^b\cdot \mathsf{\Delta }t/E_b$$

(7)

where $E_b$ is the capacity of BESS. Additionally, to ensure safe operation, it is important to maintain the SOC of the BESS within a secure range via the following:

$$S_{min}^b\le S_t^b \le S_{max}^b$$

(8)

where, to prevent over-discharging or over-charging, the lower and upper limits of SOC of the BESS are denoted as $S_{min}^b$ and, respectively, and they represent the acceptable minimum and maximum SOC values.

2.4. Electricity Trade and Power Balance

2.4.1. Purchase/Sell Electricity from/to Main Grid

The microgrid in this research has the option to purchase electricity from or sell electricity to the main grid through a contract with the utility. The real-time purchasing price at time slot $t$ is denoted as ${\rho }_t$ in the formulation, while the selling price is represented as $\beta$. As a result, the microgrid’s transaction cost can be expressed as Equation (9):

$$C_t^{Grid}=\left\{\begin{array}{c}{ \rho }_t{P}_t^{Grid}\cdot \mathsf{\Delta }t , P_t^{Grid}\ge 0\\ -\beta \left|P_t^{Grid}\right|\cdot \mathsf{\Delta }t , P_t^{Grid}<0\end{array}\right.$$

(9)

The power flow between the main grid and microgrid is denoted by $P_t^{Grid}$. The cost of purchasing electricity from the main grid is represented by ${C_t^{Grid}=\rho }_t{P}_t^{Grid}\cdot \mathsf{\Delta }t$, and the cost of selling electricity to the main grid is denoted by $C_t^{Grid}=-\beta \left|P_t^{Grid}\right|\cdot \mathsf{\Delta }t$. At each time slot,$t$, the microgrid can either purchase or sell electricity. Therefore, $P_t^{Grid}$ is subject to the following constraints:

$${-P}_{max}^{Grid}\le P_t^{Grid}\le P_{max}^{Grid}$$

(10)

The maximum power that the microgrid can handle at the point of common coupling (PCC) is denoted as $P_{max}^{Grid}$.

2.4.2. Ancillary Services

To maintain the operation of the main grid during power shortages, the microgrid can offer ancillary services such as a demand response, which can be purchased by the main grid at a price denoted by $\alpha$. Moreover, the aggregator can be found in Figure 1. The payment for the transaction can be expressed as follows:

$$C_t^{AS}=\alpha P_t^{DLs}\cdot \mathsf{\Delta }t$$

(11)

The aggregate power consumed by the dispatchable loads within the microgrid is denoted as $P_t^{DLs}$.

2.4.3. Power Balance

To ensure the reliable and secure operation of the microgrid system, it is crucial to schedule sufficient power generation to meet the demand. An energy management strategy is employed for the coordinated control of the generating units to maintain the power balance. In this study, as the main objective is to optimize the energy management of a grid-connected microgrid, the power generation dispatch at each time slot must satisfy the following power balance constraint:

$$P_t^{\mathrm{DG}}+P_t^{GT}+P_t^{Grid}+P_t^b=P_t^{net}-P_t^{DLs}$$

(12)

Equation (12) represents the power balance constraint, where the left-hand side represents the total power supplied by the generating units and the right-hand side represents the total power consumed by the loads. It is necessary to ensure that the power balance constraint is always satisfied.

In order to verify the performance of the proposed method for optimal EPD and power generation using the HIL mechanism, the active/reactive power control structure of the BESS proposed in [32] is adopted, and the controllers of the BESS are modeled in the DSP board control system. Additionally, since synchronous generators are typically used in diesel generator and gas turbine generator systems, the active/reactive power control structures of these systems are designed using virtual synchronous generators with inertia [33,34]. The controllers of the diesel generator and gas turbine generator systems are modeled in OP4510.

3. Proposed Methodology

3.1. Deep Reinforcement Learning

Deep reinforcement learning (DRL) is the combination of deep learning (DL) and reinforcement learning (RL) approaches. DL, which consists of a deep neural network divided into three parts, input, hidden and output layers, is able to perform learning in a complex and comprehensive data environment. RL includes the agent, environment, action and reward. The agent explores the environmental state, takes action under the dynamic environment, and looks forward to maximizing the reward.

The DQN, one of the DRL algorithms, can provide a stable solution to deep value-based RL. It has three main components, replay buffer, Q-network and target network as shown in Figure 3. The DQN employs experience replay, a technique that stores past experiences of the agent in a replay buffer and randomly samples batches of these experiences to update the Q-network. Moreover, the Q-network and target network comprise three main components: input, hidden and output layers. The input layer receives the state of the environment, which is usually represented as a set of features or raw sensory data. The hidden layer uses nonlinear functions as the activation function to transform the input into a high-level representation that captures the relevant information for decision-making. Finally, the output layer computes the Q-values for each action in the action space, which represents the expected future rewards if that action is taken.

3.2. MDP Model

In this section, the EPD operation of a microgrid is formulated as a Markov decision process (MDP). Firstly, the agent observes the state, $s_t$, of the environment, and then it selects and takes an action, $a_t$, which is based on the observed state. After that, the agent receives a reward, $r_t$, from environment and observes a new state, $s_{t+1}$, of the environment. The MDP model of EPD operation is defined by the tuple {S, A, T, R}, and the elements for the MDP model are defined in the following.

• System sate

The system state, $s_t\in S$, of the microgrid at time step t is denoted as $s_t$ = ($P_t^d$, $S_t^b, P_t^{PV}, P_t^{WTG}, {\rho }_t$) including the total load demand of the microgrid, $P_t^d$, the SOC of BESS, $S_t^b$, the power output of PV, $P_t^{PV}$, the power output of wind turbine, $P_t^{WTG}$, and the present electricity price, ${\rho }_t$.

• Action

$a_t\in A$ denotes the action at time step t and is taken by the agent under the current system state. It is described as $a_t$ = ($P_t^{GT}$, $P_t^{DG}, P_t^b$) including the output power of the gas turbine generator, $P_t^{GT}$, the output power of the diesel generator, $P_t^{DG}$ and the BESS charging or discharging power, $P_t^b$.

• Transition probability

The transition function, ${prob}_t$, denotes the transition probability from the system state, $s_{t-1}$, to state $s_t$ under the action $a_{t-1}$ as Equation (13):

$${prob}_t : s_{t-1}\times a_{t-1} \to s_t$$

(13)

As the above distribution is determined, the state transition is determined by system state $s_{t-1}$ and action $a_{t-1}$.

• Reward function

The reward function, $r_t$, is the negative equivalent of the microgrid operational cost at time step t. It consists of three parts: (1) $C_t^{DG}$ defined in (2), which is the operational cost of the diesel generator at time step t, (2) $C_t^{GT}$ defined in (4), which is the operational cost of gas turbine generator at time step t, and (3) the electricity cost from power grid $C_t^{Grid}$ defined in (9). Thus, the reward function can be defined as Equation (14):

$$r_t=-(C_t^{DG}+C_t^{GT}+C_t^{Grid})\times 0.01$$

(14)

In order to ensure the safety of the microgrid system, the power utility operator should guarantee enough power to supply the demand, so the power supplied from power grid, which is calculated as $C_t^{Grid},$ shall satisfy the power balance defined in (5).

3.3. Proposed DDPG with LSTM Method

A LSTM network is utilized to forecast the load demand information of the proposed microgrid model. It will be treated as one of the system state variables in the MDP model. Then, the DRL algorithm DDPG, which follows the MDP model to come out from the optimal power dispatch of power grid, is applied.

3.3.1. LSTM Network

The LSTM network architecture consists of one or several memory cells, with each cell comprising three main gates: the forget gate, the input gate, and the output gate. Each gate performs a specific function within the LSTM network [35]. In the LSTM network, the first step, forget gate $f_t$, is to decide whether the information coming from the previous timestamp should be kept or forgotten as the definition in Equation (15):

$$f_t=\mathsf{\sigma }(W_f·\left[ h_{t-1} , x_t\right]+b_f )$$

(15)

where $\mathsf{\sigma }$ is the activation function; $x_t$ is the input information at time step t; $h_{t-1}$ is the information coming from the previous time step, t − 1; $W_f$ and $b_f$ represent the weight and bias of the forget gate. A sigmoid function is adopted as the activation function that will make $f_t$ a number between 0 and 1, and the previous information, $C_{t-1}$, will be forgotten if $f_t$ = 0. In the next step, the input gate is used to decide the new information to be stored in the cell state. The equation of the input gate, $i_t$, at time step t is defined in Equation (16):

$$i_t=\mathsf{\sigma }(W_i·\left[ h_{t-1} , x_t\right]+b_i )$$

(16)

where $W_i$ and $b_i$ represent the weight and bias of the input gate. A sigmoid function is also adopted as the activation function of the input gate, which will let the value of $i_t$ be between 0 and 1. At the same time, an activation function, tanh, will be applied to new information as ${\widehat{C}}_t$ denotes in Equation (17), so that the new information will be a value between −1 and 1. However, the new information will not be memorized directly in the cell state, and it will be calculated and updated to the cell state as the equation defined in Equation (18).

$${\widehat{C}}_t=\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}(W_C·\left[ h_{t-1} , x_t\right]+b_C )$$

(17)

$$C_t=f_t·C_{t-1}+i_t·{\widehat{C}}_t$$

(18)

As the last step, the output gate is used to decide what would be the output. Also, the equation of the output gate, $o_t$, at time state t is defined in Equation (19):

$$o_t=\mathsf{\sigma }(W_o·\left[ h_{t-1} , x_t\right]+b_o )$$

(19)

where $W_o$ and $b_o$ represent the weight and bias of the output gate. Again, a sigmoid function as an activation function is adopted in the output gate. The cell state, $C_t$, using the tanh function and being multiplied by the output gate, $o_t$, is the output information, $h_t$, as defined in Equation (20):

$$h_t=o_t·\tanh \left(C_t\right)$$

(20)

Following the configuration in the LSTM network and using a single-layer memory cell with a mean squared error (MSE) loss function, the whole year hourly training data as the input information is denoted as $x_t$ = ($D_t$, $P_t^d$), where $P_t^d$ is the load demand of the microgrid in Cimei Island at timestamp $D_t$. The result of one particular day of forecast performance is shown in Figure 4, and the average deviation is less than 5%.

3.3.2. Procedure of DDPG with LSTM Method

The proposed DDPG with LSTM method for the power dispatch optimization of the microgrid will be described in detail as follows with the workflow shown in Figure 5. The DDPG is an actor–critic, model-free algorithm which operates over continuous action spaces to learn a Q-function and uses the Q-function to learn the policy. It includes four neural networks, the evaluated actor $\mu \left(s|{\theta }^{\mu }\right)$ with ${\theta }^{\mu }$ weights, the evaluated critic $Q\left(s,a|{\theta }^Q\right)$ with ${\theta }^Q$ weights, the target actor $\mu \left(s|{\theta }^{{\mu }^{\prime }}\right)$ with ${\theta }^{{\mu }^{\prime }}$ weights and the target critic $Q\left(s,a|{\theta }^{Q^{\prime }}\right)$ with ${\theta }^{Q^{\prime }}$ weights. The weights of the evaluated actor, ${\theta }^{\mu }$, and evaluated critic network, ${\theta }^Q,$ are randomly initialized and the weights of the target network are set as ${\theta }^{Q^{\prime }}$= ${\theta }^Q$ and ${\theta }^{{\mu }^{\prime }}$= ${\theta }^{\mu }$. Moreover, an additional experience replay buffer, $B$, which stores the past transition tuple $\left(s_t , a_t , r_t , s_{t+1}\right)$, is used and cleared at the beginning. At the first of each episode, $E=\{1, 2, 3\dots M\}$, and the normal distributed noise, $N_t\left(\mathrm{0,1}\right)$, is initialized and obtains the initial observation state $s_1.$ Following each time step, $t=\{1, 2, 3\dots T\}$, of the episode, the evaluated actor selects and executes an action, $a_t=\mu \left(s_t|{\theta }^{\mu }\right)$, with exploration noise, $N_t\left(\mathrm{0,1}\right).$ Then, it observes a reward, $r_t$, and new state, $s_{t+1}$, from the system environment of the microgrid with the load demand forecast from the LSTM network. This transition $(s_t , a_t , r_t , s_{t+1})$ will be stored in the experience replay buffer, $B$. In the next step, a minibatch of size N is sampled from the experience replay buffer, and the evaluated critic network calculates the estimation Q-value, $Q\left(s_i,a_i|{\theta }^Q\right)$, with the input information, $(s_i , a_i$), and the target actor network chooses a target action, $a_{i+1}=\mu \left(s_{i+1}|{\theta }^{{\mu }^{\prime }}\right)$. After that, target action $a_{i+1}$ and state $s_{i+1}$ are treated as the inputs of the target critic network. By adding the discounted target Q-value, which comes from the target critic network to the immediate reward, $r_i$, the target critic network will come out the moving target, $y_i$, that the evaluated critic tries to achieve. In addition, $y_i$ is denoted as Equation (21):

$$y_i=r_i+\gamma Q^{\prime }\left(s_{i+1},{\mu }^{\prime }\left(s_{i+1}|{\theta }^{{\mu }^{\prime }}\right)|{\theta }^{Q^{\prime }}\right)$$

(21)

Then, the loss function, L, based on temporal difference error computation is defined in Equation (22):

$$L=\frac{1}{N}{\sum }_i{\left(y_i-Q\left(s_i,a_i|{\theta }^Q\right)\right)}^2$$

(22)

It is used to minimize the loss and update the weights of the evaluated critic network. Also, the policy gradient for updating the evaluated actor network can be calculated using Equation (23):

$${\nabla }_{{\theta }^{\mu }} J\approx \frac{1}{N}\sum _i{\nabla }_{a}Q\left(s,a|{\theta }^Q\right){|}_{s=s_i,a=\mu \left(s_i\right)}{\nabla }_{{\theta }^{\mu }}\mu \left(s|{\theta }^{\mu }\right){|}_{s_i}$$

(23)

The weights of the evaluated critic and actor networks are updated sequentially as defined in Equation (24):

$$\left\{\begin{array}{c}{\theta }^Q\leftarrow {\theta }^Q+{\gamma }^Q{\nabla }_{{\theta }^Q}L\\ {\theta }^{\mu }\leftarrow {\theta }^{\mu }+{\gamma }^{\mu }{\nabla }_{{\theta }^{\mu }}J\end{array}\right.$$

(24)

where ${\gamma }^Q$ and ${\gamma }^{\mu }$ represent the learning rates for the gradient descent algorithm. In the last step, the target critic network and target actor network are also updated sequentially and softly at each episode defined in Equation (25):

$$\left\{\begin{array}{c}{\theta }^{Q^{\prime }}\leftarrow \tau {\theta }^Q+(1-\tau ){\theta }^{Q^{\prime }}\\ {\theta }^{{\mu }^{\prime }}\leftarrow \tau {\theta }^{\mu }+(1-\tau ){\theta }^{{\mu }^{\prime }}\end{array}\right.$$

(25)

where τ is a tiny gain value for the weight replacement between 0 and 1 that is based on the running average method, and the state machine will keep on going until the time step equal to $T$ and then return to the first step when starting the next episode. The following Algorithm 1 are the pseudo codes of the training process for the proposed DDPG with LSTM method.

Algorithm 1: Proposed DDPG with LSTM method

1. Initialize and train the LSTM network with $x_t$ = ($D_t$, $P_t^d$). 2. Randomly initialize evaluated actor network, $\mu \left(s|{\theta }^{\mu }\right)$, and critic network, $Q\left(s,a|{\theta }^Q\right)$, with weights ${\theta }^{\mu }$ and ${\theta }^Q$. 3. Initialize target actor and critic network ${\theta }^{{\mu }^{\prime }}$ and ${\theta }^{Q^{\prime }}$ with weights ${\theta }^{Q^{\prime }}\leftarrow {\theta }^Q$ and ${\theta }^{{\mu }^{\prime }}\leftarrow {\theta }^{\mu }$. 4. Initialize experience replay buffer, $B.$ 5. For episode $E$ = 1 to $M$, carry out the following. 6. Initialize the Gaussian exploration noise, $N_t\left(\mathrm{0,1}\right)$. 7. Receive the initial observation state $s_1$. 8. For time step t = 1 to $T$, carry out the following: 9.  Select action $a_t=\mu \left(s_t|{\theta }^{\mu }\right)$ with $N_t$ for the exploration noise. 10.   Execute action $a_t$ in the system environment with the load demand forecast    from the LSTM network and observe reward $r_t$ and new state $s_{t+1}$. 11.   Store transition $(s_t , a_t , r_t , s_{t+1})$ into experience replay buffer $B$. 12.   Sample a random minibatch of the size N transition, $(s_i , a_i , r_i , s_{i+1})$, from    experience replay buffer $B.$ 13.   Set $y_i=r_i+\gamma Q^{\prime }\left(s_{i+1},{\mu }^{\prime }\left(s_{i+1}|{\theta }^{{\mu }^{\prime }}\right)|{\theta }^{Q^{\prime }}\right)$ 14.   Update the evaluated critic network by minimizing the loss, L, as follows:        $L=\frac{1}{N}{\sum }_i{\left(y_i-Q\left(s_i,a_i|{\theta }^Q\right)\right)}^2$      ${\theta }^Q\leftarrow {\theta }^Q+{\gamma }^Q{\nabla }_{{\theta }^Q}L$ 15.   Update the evaluated actor policy using the sampled policy gradient:        ${\nabla }_{{\theta }^{\mu }} J\approx \frac{1}{N}\sum _i{\nabla }_{a}Q\left(s,a|{\theta }^Q\right){|}_{s=s_i,a=\mu \left(s_i\right)}{\nabla }_{{\theta }^{\mu }}\mu \left(s|{\theta }^{\mu }\right){|}_{s_i}$      ${\theta }^{\mu }\leftarrow {\theta }^{\mu }+{\gamma }^{\mu }{\nabla }_{{\theta }^{\mu }}J$ 16.   Softly update the target actor and critic network with the update rate, $\tau$, as follows:             $\left\{\begin{array}{c}{\theta }^{Q^{\prime }}\leftarrow \tau {\theta }^Q+(1-\tau ){\theta }^{Q^{\prime }}\\ {\theta }^{{\mu }^{\prime }}\leftarrow \tau {\theta }^{\mu }+(1-\tau ){\theta }^{{\mu }^{\prime }}\end{array}\right.$ 17. End for. 18. End for.

To improve the understanding of the proposed DDPG with LSTM method, the flowchart of the kernel of the DDPG algorithm is shown in Figure 6. Moreover, Figure 7 shows the training process of the DDPG with LSTM method for the EPD of the Cimei Island power system. The maximum value of episode $E$ is set to be 20,000 and one can see the trend of the average reward reaching convergence.

4. Benchmarking

In this study, the proposed method is compared with the experience-based EMS, Newton-PSO, and DQN methods in terms of optimization performance. Additionally, to use the experience-based EMS and Newton-PSO method, reasonable assumptions must be made about the stochastic and uncertain conditions in the microgrid. Therefore, the power output of the BESS needs to be scheduled and processed as shown in the flowchart in Figure 8 and is described as below.

• Set the initial values

Import the purchasing price of the main grid, and the SOC of the last hour with $S_{t=0}^b$ = 30% at hour 0; the average purchasing price is denoted as Equation (26)

$${\rho }_{avg}={\sum }_{t=0}^T\frac{{\rho }_t}{T}$$

(26)

where $T$ is the total duration of the economic dispatch operation. The objective of this study is to minimize the overall fuel cost within one day, so $T$ is set to 24.

• Check the purchasing price of the main grid.
  Check if the purchasing price of the main grid is cheaper than the average purchasing price.
• Make the decision.
  If the purchasing price of the main grid is cheaper than the average purchasing price, the BESS will charge at 100 kW; otherwise, the BESS will discharge at 100 kW.
  Check the SOC and decide the power output of the BESS.
• The SOC of the BESS is set to be 10% ≦ $S_t^b$ ≦ 100%. If the SOC is not within the range of the constraint conditions, the BESS does not charge or discharge; otherwise, the BESS follows the decision to charge or discharge. Moreover, the SOC of the BESS will carry on over to the next hour and proceed to the next optimization calculation.

4.1. Experience-Based EMS

The experience-based EMS is designed based on the power company expert who in charge of the power dispatch process. Its flowchart is shown in Figure 9 and described as follows.

• Set the initial values.
  Import the predictive load of the microgrid, the value of PV, the value of WTG, and the power output of the BESS.
• Check the purchasing price of main grid.
  Examine the purchasing price of main grid, and then calculate the fuel cost of the diesel generator and that of the gas turbine generator separately. Check if the buying price of the main grid is between the fuel cost of the diesel generator and that of the gas turbine generator.
• Make the decision.
  If the buying price of the main grid is between the fuel cost of the diesel generator and that of the gas turbine generator or the buying price of the main grid is the most expensive, then the output powers will be shared by the gas turbine and diesel generators; otherwise, the main grid will predominantly supply the load of the microgrid.
• Output the power of generator.
  The results of the EMS consist of the power commands of the diesel generator, gas turbine generator and the BESS and then are sent to the individual control blocks.

4.2. Newton-PSO

The Newton–Raphson method has been a standard solution algorithm to use for the power flow problem for decades. Newton’s method is a useful solution algorithm but has the disadvantage of slow convergence if it is not near the solution. Moreover, PSO is a population stochastic global optimizer, in which several particles form a swarm that evolve or fly throughout the problem hyperspace to search for an optimal or near-optimal solution. Many studies on Newton–Raphson-based power flow programs combined with PSO, i.e., Newton-PSO, have been proven to be effective in economic power dispatch [36,37]. Figure 10 explains the implementation of the Newton-PSO method. Initially, a swarm of random particles is created, and random positions of generator voltages are assigned to them; then, the best position is stored when they move through the search space. The best-fit particle of the minimum operation cost, $F$, shown in Equation (27) is tracked by the algorithm, and the position of each particle is affected by that of the best-fit particle in the entire swam. Information regarding the best solution is shared throughout the swarm by updating the position and velocity. Furthermore, the loop is ended when the predefined voltage error is met, or the maximum iterations are obtained. A series of experiments are conducted to properly tune the Newton-PSO parameters to suit the targeted optimal economic dispatch problem, which considers the fuel cost and electricity billing as an objective.

$$min: F=C_t^{DG}+C_t^{GT}+C_t^{Grid}-C_t^{AS}$$

(27)

4.3. DQN

The classic DQN method is a value based DRL algorithm that has shown remarkable performance in various domains. The DQN algorithm architecture is shown in Figure 3. It employs experience replay, target network, and epsilon-greedy exploration to enhance learning stability and efficiency. It is an extension of the Q-learning algorithm that uses a Q-network, which is a deep neural network with weight parameters, $\phi$, as an approximator for the action value function, $Q^{\pi }(s,a)$, as shown by Equation (28):

$$Q\left(s,a|\phi \right)\approx Q^{\pi }(s,a)$$

(28)

Based on the Bellman equation, the expression of the Q-network model can be represented as Equation (29):

$$Q\left(s,a|\phi \right)=Q\left(s,a|{\phi }^{-}\right)+\mathsf{\alpha }[r+\gamma \underset{a^{\ast }}{max}{Q}^{\ast }\left(s^{\ast },a^{\ast }|{\phi }^{\ast }\right)-Q\left(s,a|{\phi }^{-}\right)]$$

(29)

where $s^{\ast }$ and $a^{\ast }$ are the training state and the action to be performed at the next time step; $r$ is the reward received by the agent during the training process at each step; $\mathsf{\alpha }$ is the learning rate of the Q network; $\gamma$ is the discount factor. The target network is a separate neural network that is periodically updated to provide stable targets for the Q-value function as in Equation (30):

$$y=r+\gamma \underset{a^{\ast }}{max}{Q}^{\ast }\left(s^{\ast },a^{\ast }|{\phi }^{\ast }\right)$$

(30)

The Q-value targets are computed using the target network and subsequently define the objective function using the mean square error as the loss function of the neural network as in Equation (31):

$$L\left(\phi \right)={E\left[\right(y-Q\left(s,a|\phi \right))}^2]$$

(31)

which has its weights fixed for a certain number of iterations, and then is updated using the soft update technique as in Equation (32):

$$c\phi +\left(1-c\right){\phi }^{\ast }\to {\phi }^{\ast },c\in \left(\mathrm{0,1}\right)$$

(32)

where $c$ is the soft update rate between 0 and 1; ${\phi }^{\ast }$ is the weights of target network.

4.4. Performance Comparison

In this study, three different scenarios, Case A, Case B, and Case C, are designed to compare the impact of the experience-based EMS, Newton-PSO, DQN, and the proposed DDPG with LSTM methods on the optimization performance of a microgrid. The coefficients of the cost functions shown in Equations (2) and (4) and the minimum and maximum values of the generators are shown in

Table 2. Coefficients of fuel cost and power limits of generators in microgrid.

Generator	Pmin	Pmax	a	b	c
Gas turbine	60 kW	1250 kW	0.4969	0.0116	0.0001987
Diesel	50 kW	1250 kW	18.3333	0.10157	0.000000661

. The values of different limits and coefficients in Table 2 are selected based on the nominal specifications of gas turbines and diesel generators with a similar capacity to that of Taipower generators.

The load variation throughout the day, as depicted in Figure 4, is obtained from a well-trained LSTM load prediction model. The hourly solar and wind power generation curves shown in Figure 2 are utilized as the values for the PV and WTG system’s output powers. To quantify the outcomes, both the experience-based EMS and Newton-PSO methods employ the same SOC policy for charging and discharging the BESS as illustrated in Figure 8. For the quantification process, a particle count of 40 is set, and 5 independent runs are executed with a maximum of 100 iterations for each parameter variation. Moreover, the velocity settings shown in Figure 10, including the inertia weight and two acceleration constants, are set to −0.1618, 1.8903, and 2.1225, respectively. These parameters are determined through trial and error, considering the requirements of optimal performance and reasonable convergence speed within a maximum of 100 iterations. Furthermore, in the DQN method, the network consists of 2 fully connected hidden layers with 256 and 256 rectified linear unit (ReLU) neurons, respectively. The output layer is also a fully connected layer with 75 linear neurons. In addition, Case A and Case B represent the operation of the Cimei Island microgrid without selling electricity to the main grid and with the sale of electricity to the main grid, respectively. Case C is designed for a scenario in which the microgrid experiences sudden fuel shortage, requiring the gas turbine and diesel generator in the microgrid to operate at a minimum load of 60 kW and 50 kW, respectively, from 18:00 to 07:00. Additionally, the microgrid is obligated to participate in a load-shedding assistance program of 136.61 kW from 20:00 to 22:00 and transfer the load to the gas turbine and diesel generator units upon receiving the assistance instruction from the power company.

Figure 11, Figure 12 and Figure 13 represent the comparison of optimization results using the experience-based EMS, Newton-PSO, DQN, and DDPG with LSTM methods under three different scenarios: Case A, Case B, and Case C. Owing to the multiple particle optimization of the Newton-PSO method and the end-to-end learning capabilities of the DQN method, the total cost is well-reduced compare to that of the experience-based EMS method. Moreover, the degree of convergence of the DDPG with the LSTM method is more stable than that of the DQN method in continuous action space. Thus, the DDPG with LSTM method has the lowest average cost compared to that of the other methods. Furthermore, the results of optimization based on the DDPG with LSTM method are shown in

Table 3. Optimization results of DDPG with LSTM method in Case A.

Time (hr)	0–1	1–2	2–3	3–4	4–5	5–6	6–7	7–8	8–9	9–10	10–11	11–12	12–13	13–14	14–15	15–16	16–17	17–18	18–19	19–20	20–21	21–22	22–23	23–24
Cost (USD/hr)	70.88	75.06	76.42	74.79	74.98	74.98	74.55	74.85	66.05	54.37	49.26	50.1	49.62	50.13	54.48	63.03	74.6	88.52	95.23	100.85	106.67	106.75	75.63	70.98
GT Generation (kW)	60	61.6	60.45	113.2	107.1	81.2	62.76	202.75	224.98	256.88	247.12	244.03	252.89	171.9	195.83	206.64	215.53	205.09	199.69	229.16	232.76	307.54	64.64	115.36
DG Generation (kW)	50	50	50	50.02	50.02	50.01	50.01	446.66	339.49	191.24	151.79	163.31	149.06	230.35	253.22	327.33	432.39	578.3	648.66	675.7	728.84	639.13	50	50.02
PV Prediction (kW)	0	0	0	0	0	0	8.93	49.2	131.94	216.9	268.33	279.7	284	277.32	257.2	210.23	149.82	73.79	11.18	0	0	0	0	0
Grid (kW)	759.38	827.98	851.38	783.64	792.46	813.56	818.74	0	0	0	0	0	0	0	0	0	0	0	0	0	0	2.87	835.67	718.08
WTG Generation (kW)	149.12	141.27	133.42	146.04	145.19	149.12	156.96	164.81	168.74	172.66	168.74	164.81	166.78	164.81	168.74	160.89	156.96	149.12	153.04	141.27	133.42	137.34	139.31	141.27
BESS Generation (kW)	$-$99.9	$-$91.58	$-$98.18	$-$99.45	$-$99.69	$-$99.88	$-$99.97	98.61	82.83	77.8	74.42	65.42	47.42	46.76	35.43	53.43	59.62	78.9	99.7	68.31	$-$0.01	$-$0.04	0.04	$-$1.13
Electricity Price (USD/hr)	0.06	0.06	0.06	0.06	0.06	0.06	0.06	0.133	0.133	0.133	0.133	0.133	0.133	0.207	0.207	0.207	0.207	0.133	0.207	0.207	0.133	0.133	0.06	0.06
SOC(%)	39.99%	49.15%	58.97%	68.91%	78.88%	88.87%	98.87%	89%	80.72%	72.94%	65.5%	58.96%	54.22%	49.54%	46%	40.65%	34.69%	26.8%	16.83%	10%	10%	10.01%	10%	10.11%
Load Prediction (kW)	918.6	989.27	997.07	993.45	995.08	994.01	997.43	962.03	947.98	915.48	910.4	917.27	900.15	891.14	910.42	958.52	1014.32	1085.2	1112.27	1114.44	1095.01	1086.84	1089.66	1023.6

,

Table 4. Optimization results of DDPG with LSTM method in Case B.

Time (hr)	0–1	1–2	2–3	3–4	4–5	5–6	6–7	7–8	8–9	9–10	10–11	11–12	12–13	13–14	14–15	15–16	16–17	17–18	18–19	19–20	20–21	21–22	22–23	23–24
Cost (USD/hr)	70.88	75.38	75.9	75.53	75.66	75.38	74.57	83.83	64.36	51.97	46.89	46.85	44.67	20.63	24.27	34.81	48.36	96.53	105.28	107.88	106.75	105.44	76.16	72.22
GT Generation (kW)	60.01	68.98	97.3	61.47	62.24	61.37	61.71	258.4	241.33	212.63	187.19	186.46	175.19	225.89	232.71	229.32	239.94	242.81	231.65	241.26	209.01	221.13	67.41	99.58
DG Generation (kW)	50	50	50	50	50	50	50	478.33	306.15	213.29	186.14	186.3	174.18	623.12	651.77	758.08	879.22	619.48	716.4	731.91	752.62	728.37	50.07	50.1
PV Prediction (kW)	0	0	0	0	0	0	8.93	49.2	131.94	216.9	268.33	279.7	284	277.32	257.2	210.23	149.82	73.79	11.18	0	0	0	0	0
Grid (kW)	759.38	828.71	816.28	835.91	837.64	833.52	819.83	0	0	0	0	0	0	$-$500	$-$500	$-$500	$-$500	0	0	0	0	0	842.63	752.88
WTG Generation (kW)	149.12	141.27	133.42	146.04	145.19	149.12	156.96	164.81	168.74	172.66	168.74	164.81	166.78	164.81	168.74	160.89	156.96	149.12	153.04	141.27	133.42	137.34	139.31	141.27
BESS Generation (kW)	$-$99.91	$-$99.69	$-$99.93	$-$99.97	$-$99.99	$-$100	$-$100	11.29	99.82	100	100	100	100	100	100	100	88.38	0	0	0	$-$0.04	0	$-$9.76	$-$20.23
Electricity Price (USD/hr)	0.06	0.06	0.06	0.06	0.06	0.06	0.06	0.133	0.133	0.133	0.133	0.133	0.133	0.149 (selling)	0.149 (selling)	0.149 (selling)	0.149 (selling)	0.133	0.207	0.207	0.133	0.133	0.06	0.06
SOC(%)	39.99%	49.96%	59.95%	69.95%	79.95%	89.95%	99.95%	98.82%	88.84%	78.84%	68.84%	58.84%	48.84%	38.84%	28.84%	18.84%	10%	10%	10%	10%	10%	10%	10.98%	13%
Load Prediction (kW)	918.6	989.27	997.07	993.45	995.08	994.01	997.43	962.03	947.98	915.48	910.4	917.27	900.15	891.14	910.42	958.52	1014.32	1085.2	1112.27	1114.44	1095.01	1086.84	1089.66	1023.6

and

Table 5. Optimization results of DDPG with LSTM method in Case C.

Time (hr)	0–1	1–2	2–3	3–4	4–5	5–6	6–7	7–8	8–9	9–10	10–11	11–12	12–13	13–14	14–15	15–16	16–17	17–18	18–19	19–20	20–21	21–22	22–23	23–24
Cost (USD/hr)	70.87	75.43	76.54	75.37	75.6	75.37	74.61	79.66	78.71	62.43	53.94	52.98	53.76	46.44	50.49	59.85	71.16	86.77	177.85	183.04	83.54	85.8	75.75	72.53
GT Generation (kW)	60	60	60	60	60	60	60	221.83	295.26	242.52	216.31	216.45	220.59	193.71	217.33	249.77	280.61	273.26	60	60	196.6	196.59	60	60
DG Generation (kW)	50	50	50	50	50	50	50	475.9	384.33	285.98	229.08	219.55	223.24	176.15	194.29	252.92	328.41	490.08	50	50	50.01	50.02	50	50
PV Prediction (kW)	0	0	0	0	0	0	8.93	49.2	131.94	216.9	268.33	279.7	284	277.32	257.2	210.23	149.82	73.79	11.18	0	0	0	0	0
Grid (kW)	759.13	835.19	853.62	834.09	837.99	834.16	821.44	0	0	0	0	0	0	0	0	0	0	0	738.05	763.17	614.98	631.98	840.43	786.79
WTG Generation (kW)	149.12	141.27	133.42	146.04	145.19	149.12	156.96	164.81	168.74	172.66	168.74	164.81	166.78	164.81	168.74	160.89	156.96	149.12	153.04	141.27	133.42	137.34	139.31	141.27
BESS Generation (kW)	$-$99.65	$-$97.19	$-$99.97	$-$96.68	$-$98.1	$-$99.27	$-$99.9	50.29	$-$32.29	$-$2.58	27.94	36.76	5.54	79.15	72.86	84.71	98.52	98.95	100	100	100	70.91	$-$0.08	$-$14.46
Electricity Price (USD/hr)	0.06	0.06	0.06	0.06	0.06	0.06	0.06	0.133	0.133	0.133	0.133	0.133	0.133	0.207	0.207	0.207	0.207	0.133	0.207	0.207	0.237 (AS)	0.237 (AS)	0.06	0.06
SOC(%)	39.97%	49.68%	59.68%	69.35%	79.16%	89.09%	99.08%	94.05%	97.28%	97.53%	94.74%	91.06%	90.51%	82.6%	75.31%	66.84%	56.99%	47.09%	37.09%	27.09%	17.09%	10%	10.01%	11.45%
Load Prediction (kW)	918.6	989.27	997.07	993.45	995.08	994.01	997.43	962.03	947.98	915.48	910.4	917.27	900.15	891.14	910.42	958.52	1014.32	1085.2	1112.27	1114.44	1095.01	1086.84	1089.66	1023.6

. Table 3 presents the results for Case A, where the utility company implements a peak-shaving strategy with three different electricity prices. The main grid purchases electricity at a lower price than the average price from 22:00 to 07:00, while during the high or semi-high peak periods from 07:00 to 22:00, the main grid purchases electricity at a higher price than the average. In addition, Table 4 displays the outcomes for Case B, where the microgrid, based on the generation plan from Case A, fulfills its contractual obligation with the utility company by selling 500 kW of electricity to the main grid from 13:00 to 17:00 at an agreed price of 0.149 USD/kWh. In Case C, as illustrated in Table 5, a simulation is conducted to represent a sudden fuel shortage in the microgrid. During this period, the gas turbine and diesel generators operate at the minimum power output from 18:00 to 07:00 on the next day to conserve the fuel inventory. The microgrid purchases electricity from the main grid to supply the load. Additionally, the microgrid operator receives a notification to implement an auxiliary service plan, reducing the load by 136.61 kW from 20:00 to 22:00 at an agreed price of 0.237 USD/kWh. The load is then transferred to the gas turbine and diesel generators without any power outage.

Table 6. Comparison of fuel cost and electricity billings of various methods in Case A, B and C.

Method	Experience-Based EMS	Newton-PSO	DQN	DDPG
Cost of Case A (USD)	1830.78 (base)	1770.17 ($-$3.31%)	1769.75 ($-$3.33%)	1752.78 ($-$4.26%)
Cost of Case B (USD)	1791.9 (base)	1692.76 ($-$5.53%)	1678.69 ($-$6.32%)	1660.2 ($-$7.35%)
Cost of Case C (USD)	1973.79 (base)	1941.28 ($-$1.65%)	1964.38 ($-$0.48%)	1898.49 ($-$3.81%)

is the comparison of the daily total operating costs for the four methods using the experience-based EMS, Newton-PSO, DQN and DDPG with LSTM. As one can see, in Table 6, the performance judged according to the daily total cost of the proposed DDPG with LSTM method is better than that of the experience-based EMS, Newton-PSO and DQN methods. Though the Newton-PSO and DQN methods can possibly achieve overall optimization results, the proposed DDPG with LSTM method is still able to find the best results.

5. Experimentation

5.1. OPAL-RT Environment and Implementation

Figure 14 depicts the experimental setup of HIL utilized to optimize the energy management of a microgrid. Testing the performance of a real microgrid system is challenging due to the complexity of power components and consumer loads. To overcome this, a real-time simulator is employed as a crucial tool for exploring the power system’s characteristics. In this study, the OPAL-RT real-time high-speed simulator, manufactured by OPAL-RT Co., Ltd., in Montreal, QC, Canada, is chosen for its excellent integration capabilities with simulation software such as MATLAB and LabVIEW. The proposed method’s optimal solutions are verified through the HIL mechanism, making it practical for the Cimei Island microgrid. Moreover, the testing platform consists of various components, including an oscilloscope, a host computer equipped with an EMS containing different EPD methods, peripheral circuits, a CANalyst SN65HVD230 CAN bus transceiver communication module, and a DSP TMS320F28335 control board integrated with the HIL mechanism built with the OPAL-RT real-time simulator OP4510 and the RT-LAB environment. The OP4510 simulator features two 16-channel digital-to-analog converter modules (OP5330), one 16-channel analog-to-digital converter module (OP5340), and one 32-channel digital signal conditioning module (OP5353). The BESS controller is implemented using the C language on Texas Instruments TMS320F28335 DSP, and the observed signals are transferred to the oscilloscope via the serial peripheral interface (SPI). Furthermore, a dedicated CAN bus transceiver communication module is employed to receive active/reactive power commands of the BESS from the EMS.

Figure 15 provides an illustration of the OPAL-RT structure and its peripherals. The entire Cimei Island microgrid, with the exception of the BESS controller, is modeled in the host system and then transferred to OP4510 through Ethernet communication. In HIL, the inverter models are designed by the authors based on specifications [32,33,34]. The microgrid components, including the inverters of the BESS and the virtual synchronous generators of the diesel generator and gas turbine generator, are modeled in the field programmable gate array (FPGA) of OP4510. On the other hand, the PV, the BESS, WTG modules, and the diesel generator and gas turbine generator are modeled in the CPU of OP4510. Additionally, the active/reactive power commands for the diesel generator and gas turbine generator are transmitted from the EMS in the host system to the OP4510 CPU using Ethernet communication as well. The control blocks of the BESS operate with a sampling time of 1 ms, and the PWM switching frequency for the BESS inverter is set at 16 kHz.

5.2. Emulation Result

To minimize fuel costs based on changes in predictive load, PV, and WTG power, this study designed three cases and used the proposed DDPG with LSTM method to emulate the optimization results in real time using the HIL system depicted in Figure 14. In Case A, where no electricity is sold to the main grid, Figure 16a presents the emulated responses of the predictive load, $P^d$, PV $P^{PV}$, and WTG $P^{WTG}$ around 07:00. Moreover, the power outputs of the gas turbine generator, $P^{GT}$, and diesel generator, $P^{DG}$, are illustrated in Figure 16b, while the power flow in the main grid, $P^{Grid}$, and the power output of the BESS, $P^b$, are depicted in Figure 16c. In Case B, where 500 kW of power is supplied to the main grid from 13:00 to 17:00, Figure 17a presents the emulated responses of the predictive load, $P^d$, PV $P^{PV}$, and WTG $P^{WTG}$ around 13:00. Furthermore, the power outputs of the gas turbine generator, $P^{GT}$, and diesel generator, $P^{DG}$, are illustrated in Figure 17b, while the power flow in the main grid, $P^{Grid}$, and the power output of the BESS, $P^b$, are depicted in Figure 17c. In addition, Case C involves a load reduction agreement between the microgrid and power company, transferring 136.61 kW of the load to the gas turbine and diesel generator units from 20:00 to 22:00. Figure 18a presents the emulated responses of the predictive load, $P^d$, PV $P^{PV}$, and WTG $P^{WTG}$ around 22:00. The power outputs of the gas turbine generator, $P^{GT}$, and diesel generator, $P^{DG}$, are illustrated in Figure 18b, while the power flow in the main grid, $P^{Grid}$, and the power output of the BESS, $P^b$, are depicted in Figure 18c.

6. Conclusions

In this study, a novel DRL algorithm, which is the DDPG algorithm combined with LSTM, was proposed to obtain the optimal EPD solution for a microgrid. Compared to the experience-based EMS, Newton-PSO and DQN methods, the proposed DDPG with LSTM method has a much lower electricity cost and features superior optimization performance. Moreover, the microgrid built in Cimei Island was investigated to examine the compliance with the requirements of equality and inequality constraints and the performance of the proposed method. Furthermore, the HIL mechanism was conducted to verify the effectiveness of the proposed DDPG with LSTM method to prove it to be pragmatic. From the emulation results, it has been verified that the proposed DDPG with LSTM method could obtain the best optimal EPD solution owing to its capability to learn the policies in high-dimensional space, which is very helpful to reduce costs for the operation of the microgrid. The significant contributions of the proposed DDPG with LSTM method include (1) the easy adjustment of the mathematical models of a microgrid subjected to the power balance constraints according to parameter and load variations; (2) the successfully improved accuracy of load forecasting using LSTM, resulting in more efficient EPD; (3) the successful development of solving the optimal economic dispatch solution with the ability to handle the uncertainties and dynamic characteristics of microgrid systems effectively, making it a promising solution for EPD in real-world microgrid systems; (4) the successful implementation of the results of optimal economic dispatch and power generation using the proposed method in a microgrid system built with a HIL real-time simulator to prove to be pragmatic.