Deep Reinforcement Learning-Based Method for Joint Optimization of Mobile Energy Storage Systems and Power Grid with High Renewable Energy Sources

Abstract

The joint optimization of power systems, mobile energy storage systems (MESSs), and renewable energy involves complex constraints and numerous decision variables, and it is difficult to achieve optimization quickly through the use of commercial solvers, such as Gurobi and Cplex. To address this challenge, we present an effective joint optimization approach for MESSs and power grids that consider various renewable energy sources, including wind power (WP), photovoltaic (PV) power, and hydropower. The integration of MESSs could alleviate congestion, minimize renewable energy waste, fulfill unexpected energy demands, and lower the operational costs for power networks. To model the entire system, a mixed-integer programming (MIP) model was proposed that considered both the MESSs and the power grid, with the goal of minimizing costs. Furthermore, this research proposed a highly efficient deep reinforcement learning (DRL)-based method to optimize route selection and charging/discharging operations. The efficacy of the proposed method was demonstrated through many numerical simulations.

1. Introduction

Renewable energy sources such as wind, water, and solar energy have considerable capacity to lower energy costs and carbon emissions while playing a crucial role in creating low-carbon and sustainable energy systems, as highlighted in [1,2,3]. However, the intermittency and variability of renewable energy pose a great challenge to the safe and economic operation of power systems in regions with an abundance of renewable energy potential [4,5].To address this challenge, energy storage systems (ESSs) are a promising technology for the integration of renewable energy and the reduction in renewable curtailment.

Stationary energy storage systems (SESSs), the most conventional application mode of BES, have enabled energy conversion and capacity sharing during a limited time period [6,7]. Specifically, given the predicted renewable energy generation curve, SESSs, renewable energy, and thermal units were scheduled collaboratively to integrate renewable energy. However, SESS was inflexible, as it relies on large-capacity and long-distance transmission lines. Meanwhile, the utilization efficiency of SESSs also depended on its location, as it was difficult for the owners of a SESS to determine the optimal location because numerous constraints had to be considered, such as political, economic, social, and geographical factors [8,9,10]. Therefore, mobile energy storage systems (MESSs), as an ongoing development application mode of BES that couples energy and transportation systems, incorporate vehicles (e.g., electric vehicles (EVs), trains, ships, etc.), batteries, power converters, and transformers. As compared to SESSs, MESSs enable energy conversion and the capacity sharing of energy storage over longer time periods through their transportation network and considerably improve the utilization rate of battery assets and the emergency dispatching capability of the power system. MESSs had more flexible deployment capabilities and had a more remarkable capacity in energy arbitrage [11,12], renewable energy integration [13,14,15], peak-shaving [16,17], grid-congestion relief [18,19], and grid-investment deferral [20,21,22].

The authors of [11] proposed the concept of a utility-scale MESS, which incorporated electric trucks, energy storage, and energy conversion systems; constructed an optimization model involving charge scheduling and route planning of MESSs; and used the proposed model in the energy arbitrage of a power grid in California. The results revealed that, as compared to the SESS, the MESS had considerable benefits over a life-cycle time period. In [13], the researchers integrated MESS into a large-scale power distribution system in order to integrate renewable energy in remote areas and proposed a particle-swarm optimization to streamline a scheduling model of MESSs. The results showed that the proposed models and algorithm could considerably reduce the operating cost of the power system. The authors of [14] developed a decision framework that used MESS to improve system resilience for the emergency dispatching of the power distribution system, and they verified that the proposed decision framework could effectively improve system response-and-recovery speed in the emergency scenarios. In [18], the researchers integrated large-scale EVs into the electric energy distribution system and proposed a mixed-integer linear programming (MILP) model to optimize the charging schemes of EVs. The results showed that the proposed model could substantially alleviate grid congestion and improve the performance of EVs and grid benefits. The authors of [20] developed a collaborative planning model for SESSs and MESSs to reduce the capital cost of transmission lines, energy storage systems, and the operating costs of power systems. Meanwhile, ref. [20] used the proposed model to determine the optimal region for SESS and MESS implementation for China’s northwestern grid. The aging issue of mobile energy storage devices has also attracted attention. Reference [23] analyzed the influence mechanism of grid-connected operation on the life degradation of lithium-ion battery energy storage systems and established a cost accounting model for frequency regulation, considering the impact of battery life degradation. Reference [24] studied the frequency regulation problem of a power grid model that includes loads, traditional generators, and multiple electric vehicles, aiming to minimize the degradation of battery devices. Their proposed strategies demonstrated high effectiveness under actual operating conditions. In this paper, in order to emphasize the modeling of mobile energy storage systems and their joint optimization with the power grid, we have simplified the description of this part.

Recently, deep reinforcement learning (DRL) techniques have seen growing applications in battery management [25,26], grid management, and micro-grid management [27,28]. The authors of [25] presented a DRL-based method for a battery system in EVs, which consisted of a high-power battery pack in order to reduce energy loss and enhance thermal safety. The proposed strategy demonstrated advantages in terms of reducing calculation time and energy consumption. In [26], the researchers proposed a joint optimization framework of DRL and binary integer programming for the battery swapping–charging systems for EVs, which had favorable performance in a real-time demonstration, as well as improved computational efficiency and privacy preservation. A multi-agent system based on an RL method was applied to manage a stand-alone micro-grid including a power-production unit, a power-consumption unit, and a power-storage unit in [27]. The authors of [28] proposed an optimal control strategy based DRL, which had asynchronous advantages via its $actor-critic$ framework to manage and optimize an online energy system.A multi-agent DRL-based approach was developed by [29] for real-time route selection and dispatching of multiple coordinated MESSs, and it had the ability to handle a hybrid continuous-discrete action space and to strengthen resiliency after extreme events. In [30], the researchers proposed a DRL-based method that coordinated the scheduling of the MESSs and the resource allocation of micro-grids in order to minimize the overall cost of the system. In the future, a significant amount of data is expected to be generated in the fields of energy and transportation [19], which could provide significant data support for solving MESS’s challenges through the application of DRL-based methods.

We have summarized the commonly used forms of energy storage, including their optimization objectives, research methods, and integration with the power grid, as shown in

Table 1. Summary of energy management problem and power grid.

Authors		[31]	[9]	[10]	[11]	[32]	[30]	Ours
Year		2021	2020	2021	2021	2022	2020	2023
Modality	SESSs	✓
	MESSs		✓	✓	✓	✓	✓	✓
Objective function	Cost/benefit	✓	✓		✓	✓		✓
	Grid peak			✓
	Resilience						✓
Method	Math programming		✓		✓
	Heuristic					✓
	Control	✓		✓
	Deep reinforcement learning						✓	✓
Coordinated with	Power grid	✓	✓	✓		✓	✓	✓
	Renewable energy							✓

.

Therefore, based on the aforementioned research, most of the methods adopted to solve joint optimizations in power systems, MESSs, and other renewable energy components have been commercial solvers, heuristics, and meta-heuristics [12,20]. More recently, DRL-based methods have shown strong abilities in resolving combinatorial optimization challenges, and all have achieved remarkable results. To the best of our knowledge, however, few studies have focused on applying DRL to solve the cooperative scheduling challenges of batteries and power systems. Therefore, we proposed a joint mixed-integer programming (MIP) model for MESSs and the power grid. To address the nonlinearity and the elimination of integer variables, a hybrid strategy was proposed. A MESS model, which included 0–1 integer variables, was resolved using a DRL-based approach. The output from the DRL-based method was used as a portion of the input for the power-grid model, which was a simple linear programming (LP) model and easy to solve. The MESSs were formulated as constrained Markov decision processes (CMDPs), which served as the foundation for the DRL design. To enable the agent to make simultaneous decisions about the destination and power, a hybrid discrete-continuous action space was designed. The effectiveness of the proposed method was demonstrated through numerical experiments. Our contributions were summarized in three key areas:

(1)

We developed an MIP model for the joint system of the MESSs and power grids to minimize the total operating costs. The model included constraints on the output of thermal power and renewable energy sources, the energy transmission of the power grid and the MESSs, the locations of the MESSs, etc.;

(2)

We presented a formulation of the MESS challenge as a CMDP and introduce a DRL-based algorithm to make decisions in a hybrid action space that combined both discrete and continuous variables;

(3)

We proposed a new linear programming (LP) model that eliminated the constraints on the MESSs in the original model by using the DRL-based method, thereby eliminating the integer-variable constraints and significantly improving the solving time. Therefore, we proposed an algorithmic framework that combined DRL-based methods with the solutions of new LP models.

The remainder of this paper is structured as follows. In Section 2, the scheduling optimization challenge of the MESSs and the power grid is formulated as a MIP model. Section 3 describes the proposed DRL-based method. In Section 4, simulation studies are performed to validate the efficacy of the proposed approach. Section 5 presents the conclusions.

2. Mobile Energy Storage Systems and Power Grid Model

The joint optimization model for the MESSs and power grids aimed to minimize costs while considering constraints on energy transportation and operation. The objective function was comprised of two components: thermal power cost and MESS transportation cost, as follows:

$$min\sum _{g\in G}\sum _{u\in U_g}\sum _{t\in T}{C}_{u,t}^g{P}_{u,t}^g+c^{tra}\Delta t\sum _{g\in G}\sum _{f\in G}\sum _{t\in T}{\gamma }_{g,f,t},$$

(1)

where G denotes the electricity consumption node set and g denotes a single node; $U_g$ denotes the thermal power unit set in node g; T represents the decision cycle, and t represents time index, where time slot length $\delta t$ is set to 1 hour. The variables $C_{u,t}^g$ and $P_{u,t}^g$ indicate the generation cost of thermal power units and output, respectively. In addition, $c^{tra}$ denotes the transportation cost per unit of time, which was set to $20/h in this study. The expression ${\gamma }_{g,f,t}\in 0,1$ represents whether the MESSs were traveling between nodes g and f (1 indicated traveling and 0 indicated not traveling).

The constraints of the joint model were the following:

$$\begin{array}{}\mathrm{(2)}& P_{u,min}^U\le P_{u,t}^U\le P_{u,max}^U,\hfill & \hfill \forall u\in U_g,g\in G,t\in T,\mathrm{(3)}& P_{u,t}^U-P_{u,t-1}^U\le R{U}_{u,t},\hfill & \hfill \forall u\in U_g,g\in G,t\in T,\mathrm{(4)}& P_{u,t}^U-P_{u,t-1}^U\ge -R{D}_{u,t},\hfill & \hfill \forall u\in U_g,g\in G,t\in T,\mathrm{(5)}& 0\le P_{r,t}^R\le P_{r,t,max}^R,\hfill & \hfill \forall r\in R_g,g\in G,t\in T,\mathrm{(6)}& -P_{l,t,max}^{TL}\le P_{l,t}^{TL}\le P_{l,t,max}^{TL},\hfill & \hfill \forall l\in TL,g\in G,t\in T,\mathrm{(7)}& {\displaystyle \sum _{l\in TL}}{P}_{l,t}^{TL}=0,\hfill & \hfill \forall t\in T,\mathrm{(8)}& {\displaystyle \sum _{u\in U_g}}{P}_{u,t}^g+{\displaystyle \sum _{r\in R_g}}{P}_{r,t}^R+{\displaystyle \sum _{l\in TL}}{P}_{l,t}^{TL}=P_{g,t}^L+{\omega }_{g,t}{P}_{g,t}^{MESS},\hfill & \hfill \forall u\in U_g,r\in R_g,g\in G,t\in T,\mathrm{(9)}& 0\le P_{g,t}^{MESS}\le {\omega }_{g,t}{P}_{max},\hfill & \hfill \forall g\in G,t\in T,\mathrm{(10)}& {\displaystyle \sum _{g\in G}}{\omega }_{g,t}\le 1-{\displaystyle \sum _{f\in G}}{\gamma }_{g,f,t},\hfill & \hfill \forall g\in G,t\in T,\mathrm{(11)}& {\alpha }_{g,t}-{\beta }_{g,t}={\omega }_{g,t}-{\omega }_{t,(t-1)},\hfill & \hfill \forall g\in G,t\in T,\mathrm{(12)}& {\displaystyle \sum _{g\in G}}\left({\alpha }_{g,t}+{\beta }_{g,t}\right)\le 1,\hfill & \hfill \forall t\in T,\mathrm{(13)}& {\displaystyle \sum _{f\in G}}{\gamma }_{g,f,t}\ge {\beta }_{g,t},\hfill & \hfill \forall f\in G,t\in T,\mathrm{(14)}& {\alpha }_{f,t}-{\theta }_{f,t}={\displaystyle \sum _{g\in G}}\left({\gamma }_{g,f,t-1}-{\gamma }_{g,f,t}\right),\hfill & \hfill \forall f\in G,t\in T,\mathrm{(15)}& {\displaystyle \sum _{g\in G}}\left({\alpha }_{g,t}+{\theta }_{g,t}\right)\le 1,\hfill & \hfill \forall t\in T,\mathrm{(16)}& {\gamma }_{g,f,t}\ge {\gamma }_{g,f,t-1}-{\gamma }_{g,f\left(t-H_{g,f,t}\right)},\hfill & \hfill \forall g\in G,f\in G,t\in T.\end{array}$$

where $P_{u,min}^U$ and $P_{u,max}^U$ denote the minimum output and maximum output of thermal power units, respectively. The variables $R{U}_{u,t}$ and $R{D}_{u,t}$ denote the increased or decreased output of the thermal units, respectively, (with ramp constraints). In addition, $R_g$ is the set of renewable energy resources including WP, PV, and hydropower. The actual output of the renewable energy resources and the maximum output of the renewable energy resources are expressed as $P_{r,t}^R$ and $P_{r,t,max}^R$, respectively. Furthermore, $P_{l,t}^{TL}$ and $P_{l,t,max}^{TL}$ represent the actual transmission power and maximum transmission power of tie-line l, respectively. The expression $l=(g,f)\in TL$ denotes the tie-line between nodes g and f. The load in node g is represented as $P_{g,t}^L$.

The constraints were divided into two categories. The first category included the constraints on the power grid, including inequalities, as expressed in (2)–(8). Constraint (2) included the upper and lower limits of the thermal unit output. Constraints (3) and (4) represented the ramp constraints for thermal power units. The output of renewable energy resources was constrained by inequality, as shown in (5). The constraints on the transmission power of the tie-lines were expressed in (6) and (7). Constraint (8) represented the balance of power output for the whole joint system of the MESSs and power grids.

The second category included constraints (9)–(16) of the MESSs, with respect to [11]. Constraint (9) denoted the limit of the power output of the MESSs. The storage could not stop at a node if it was traveling between nodes and could only appear at one node at one time, as expressed in constraint (10). Equations (11)–(15) modeled the transportation status between nodes g and f, where ${\alpha }_{g,t}\in \{0,1\}$ and ${\beta }_{g,t}\in \{0,1\}$ were both binary variables that denoted whether the MESSs were moving to node g at time t or whether the MESSs were moving from node g at time t, respectively.In addition, ${\theta }_{g,t}\in \{0,1\}$ represented an auxiliary binary variable. Constraint (11) connected the change of the locator indicators ${\omega }_{g,t}$ with the arrival indicators ${\alpha }_{g,t}$ and depart indicators ${\beta }_{g,t}$. The arrival and departure could not occur simultaneously, as expressed in constraint (12). Constraint (13) managed the storage depart node. Constraints (14) and (15) ensure that the arrival indicators ${\alpha }_{g,t}$ are equal to 1 for the arrival times; otherwise, they were equal to 0. Constraint (16) calculated the transportation time $H_{g,f,t}$ from node g to node f. In this study, the traveling time was estimated as a square matrix for all nodes.

Model (1) contained a large number of integer variables, which are known to be computationally demanding. In this proposal, we considered a machine-learning approach to solve for the integer variables. This study formulated the storage movement process as a Markov decision process (MDP) and applied a DRL-based method to solve it.

3. Deep Reinforcement Learning-Based Method

This section begins with a concise overview of the evolution of DRL-based techniques. The MESSs challenge is then framed as an MDP. The agent in the DRL-based approach was represented by an electric truck and its operator. Next, we demonstrated the integration of the DRL-based technique into the MIP model. Finally, the formulation of the action space, state space, and reward function for the DRL-based method was established. The proximal gradient projection algorithm was employed to ensure the safe exploration by the agent.

3.1. Background of DRL

The basement of the reinforcement-learning phase was the Bellman equation [33]:

$$Q^{*}(s,a)={\mathbb{E}}_{n^{\prime }\sim \epsilon }\left[r+\gamma \underset{a^{\prime }}{max}{Q}^{*}\left(s^{\prime },a^{\prime }\right)\mid s,a\right],$$

(17)

where the optimal value of the state sequence s, represented by $Q(s,a)$, is determined by selecting the action a that maximizes the expected value of $Q(s,a)$.

Traditional reinforcement learning encounters the challenge of dimensionality. The authors of [34] addressed this issue by utilizing neural networks (NNs) to approximate the Q-value table. The Q-value was approximated using NNs, expressed as $Q(s,a;w)$, which was a close estimation of the true Q-value $Q(s,a)$. The weights of the NNs were represented by the variable w. The weight parameters were updated through the loss function [35]:

$$L_t\left(w\right)={\left\{Q\left(s_t,a_t;w\right)-\left[r_t+\gamma \underset{a^{\prime }\in \mathcal{A}}{max}Q\left(s_{t+1},a^{\prime };w_t\right)\right]\right\}}^2.$$

(18)

The maximization calculation in Equation (18) is computationally inefficient for continuous action spaces due to the non-convex nature of the function $Q(s,a;w)$ with respect to action a. This renders the maximization calculation NP-hard in the worst-case scenario [36]. To overcome this challenge, ref. [37] introduced the deterministic policy gradient (DPG) theorem which utilizes policy-based methods for continuous action spaces through the implementation of deterministic policies ${\mu }_{\theta }:S\to \mathcal{A}$. The objective of policy gradient methods is to discover a policy ${\pi }_{\theta }$ that optimizes the expected reward. The objective of the policy, $J\left({\pi }_{\theta }\right)$, is then updated through gradient descent [38]:

$${\nabla }_{\theta }J\left({\mu }_{\theta }\right)=\underset{s\sim {\rho }^{{\mu }_{\theta }}}{\mathbb{E}}\left[{\left.{\nabla }_{\theta }{\mu }_{\theta }\left(s\right){\nabla }_a{Q}^{{\mu }_{\theta }}(s,a)\right|}_{a={\mu }_{\theta }\left(s\right)}\right].$$

(19)

3.2. Algorithm Framework

The $actor-critic$ framework has proven to be effective in various domains, such as energy storage systems [26] and the game Go [39]. We adopted a similar structure in this proposal. The $Actor$ network was a strategic network designed to determine the power parameters of the MESSs, which had a continuous feasible range. The $Critic$ network assessed the actor–network’s parameter choices and decided on other discrete variables, such as destination selection, charging and discharging selection, etc.

The hybrid action space $\mathcal{A}$ was defined according to [36], as follows:

$$\mathcal{A}=\left\{\left(k,x_k\right)\mid x_k\in {\mathcal{X}}_k\mathrm{for}\mathrm{all}k\in K\right\},$$

(20)

which consists of discrete action $k\in \mathcal{K}$ and continuous action $x_k\in {\mathcal{X}}_k$. The Bellman equation transforms into [36], as follows:

$$Q\left(s_t,k_t,x_{k_t}\right)=\underset{r_t,s_{t+1}}{\mathbb{E}}\left[r_t+\gamma \underset{k\in K}{max}Q\left(s_{t+1},k,x_k^Q\left(s_{t+1}\right)\right)\mid s_t=s\right].$$

(21)

The value-based network’s weights are denoted as $\omega$, and the policy-based network’s weights are denoted as $\theta$. In each step t of the updating process, ${\omega }_t$ and ${\theta }_t$ were updated, respectively. First, the weight parameter, $\omega$, was estimated by employing the gradient descent to minimize the mean-squared Bellman error, which was similar to the DQN method. Then, $\omega$ and $\theta$ were fixed by maximizing $Q\left(s,k,x_k(s;\theta );\omega \right)$. The loss functions were formulated as follows [40]:

$$\begin{array}{c}\hfill {\ell }_t^Q\left(\omega \right)=\frac{1}{2}{\left[Q\left(s_t,k_t,x_{k_t};\omega \right)-y_t\right]}^2,\end{array}$$

(22)

$$\begin{array}{c}\hfill {\ell }_t^{\mathrm{\Theta }}\left(\theta \right)=-\sum _{k=1}^{K}Q\left(s_t,k,x_k\left(s_t;\theta \right);{\omega }_t\right),\end{array}$$

(23)

where $y_t=r_t+\gamma {max}_{k^{\prime }\in K}Q\left(s_{t^{\prime }},k,x_k\left(s_{t^{\prime }},{\theta }_t\right);{\omega }_t\right)$.

According to [41], there were three approaches for utilizing machine learning in combinatorial optimization challenges: using machine learning on its own, integrating machine learning with traditional optimization algorithms, and iteratively combining optimization and machine learning. For this proposal, we employed the DRL-based approach to efficiently eliminate integer variables in model (1). The objective function of model (1) served as the reward for reinforcement learning, and the DRL-based method was used to obtain the integer variables in constraints (9)–(16). The value of ${\omega }_{g,t}$ in constraint (8) was passed as an input to the new model.

The new model was described by the following:

$$\begin{array}{cc}\hfill & min\sum _{g\in G}\sum _{u\in U_g}\sum _{t\in T}{C}_{u,t}^g{P}_{u,t}^g+C^{tra}\hfill \\ \hfill & s.t.\mathrm{constraints}\left(2\right)-\left(8\right)\mathrm{in}\mathrm{model}\left(1\right),\hfill \end{array}$$

(24)

where $C^{tra}$ is the total transportation cost of the MESSs.

The algorithm framework is depicted in Figure 1. State $S_t$ (P1) consisted of the output of thermal units and renewable energy at grid nodes, while State $S_t$ (P2) included the remaining energy of the MESSs, the cost of thermal power units, and the location information. State $S_t$ was the combination of these five elements.

3.3. Constrained MDP Formulation

According to model (1), this study modeled the MESSs in this model as MDPs. However, due to the capacity constraint of the battery packs in the MESSs, the processes became CMDPs. To address this, we designed a hybrid action space and a state space for the CMDPs. A proximal gradient projection algorithm was proposed to ensure safe exploration by the agent.

3.3.1. Action Space $\mathcal{A}$, State Space $\mathcal{S}$, and Reward Function $\mathcal{R}$

Action space $\mathcal{A}$ was composed of two levels: The discrete action space that included a chosen charging/discharging/holding action and the chosen destination node; and the continuous action that included the output power parameters of the MESSs. At time t, the action $a_t$ was represented by $\left(\left(g_t,cd{h}_t\right)|P_{g,t}^{cdh}\right)$, where g is the chosen destination node, $cdh$ represents the charging/discharging/holding choice, and $P_{g,t}^{cdh}$ is the corresponding power at node g at time t. The range of actions and their values were defined by the following:

$$\mathcal{A}=\left\{\begin{array}{c}g\in \left\{g_1,g_2,...,g_N\right\}\hfill \\ cdh\in \{-1,0,1\}\hfill \\ P=\left\{\begin{array}{c}(0,P_{max}],cdh=-1\mathrm{or}chd=1\hfill \\ 0,cdh=0\hfill \end{array}\right.\hfill \end{array}\right.,$$

(25)

where N is the node’s number, and $cdh=$ −1, 0, and 1 means discharging, holding, and charging, respectively.

The state space $\mathcal{S}$ consisted of two parts, as illustrated in Figure 1. The variable ${\mathcal{S}}_t\left(P1\right)$ described the state of the power grid, and ${\mathcal{S}}_t\left(P2\right)$ described the state related to the MESS. At time t, $\mathcal{S}$ could be defined as ${\mathcal{S}}_t=\left(P_{u,t},R{E}_{g,t},E_{g,t},Loa{d}_{g,t},g\right)$, where $R{E}_{g,t}$ is the renewable energy resources at node g at time t, $E_{g,t}$ is the remaining energy in the MESSs at node g at time t, and $Loa{d}_{g,t}$ represents the load at node g. There was a capacity constraint for the MESSs, $\underline{E}\le E_{g,t}\le \overline{E}$, which indicated the battery pack had an upper and lower limit on its capacity, represented by $\underline{E}$ and $\overline{E}$, respectively.

The reward $\mathcal{R}$ was based on the objective value of the linear programming (LP) model (24). The reward in RL was a cumulative value, as depicted in Equation (21), where $Q(s,a)={\sum }_{t\in T}{\gamma }^t{r}_t$. In this study, the objective value of model (24) was used as the reward: $r_t=-Ob{j}_{m2}$, where $Ob{j}_{m2}$ represents the optimal objective value of model (24).

Theorem 1. 

In a finite MDP, if ${\sum }_{t\in T}-Ob{j}_{m2,t}$ was the maximum when training converged with a policy, then each $-Ob{j}_{m2,t}t\in T$ was the maximum in the policy.

Proof of Theorem 1. 

If $\varphi$ is a policy, then

$$q_{\pi }\left(s^{*},a^{*}\right)\ge v_{\pi }\left(s^{*}\right).$$

Suppose, then, that there is a policy ${\varphi }^{{}^{\prime }}$:

$$v_{{\pi }^{{}^{\prime }}}\left(s\right)\ge v_{\pi }\left(s\right)\mathrm{for}\mathrm{all}s,$$

$${\pi }^{{}^{\prime }}(a\mid s)=1\left(s^{*}=s\right)1\left(a=a^{*}\right)+1\left(s^{*}\ne s\right)\pi (a\mid s).$$

We considered $v^{*}\left(s\right)={max}_{\pi \in \mathrm{\Pi }}{v}_{\pi }\left(s\right)$ and $q^{*}(s,a)={max}_{\pi \in \mathrm{\Pi }}{q}_{\pi }(s,a)$. Now extrapolating backward, there existed $s^{*}$, resulting in the following:

$${max}_a{q}^{*}\left(s^{*},a\right)>v^{*}\left(s^{*}\right).$$

Suppose there existed a policy ${\pi }^{{}^{″}}$, as well as $a^{*}$:

$$q_{{\pi }^{{}^{″}}}\left(s^{*},a^{*}\right)={max}_a{q}^{*}\left(s^{*},a\right)$$

while $v^{*}$ was the maximum over all $\pi$, for $\pi ={\pi }^{{}^{″}}$, we obtained the following inequality:

$$q_{{\pi }^{{}^{″}}}\left(s^{*},a^{*}\right)>v_{{\pi }^{{}^{\prime }}}\left(s^{*}\right).$$

Therefore, we obtained the following:

$$v_{{\pi }^{{}^{\prime }}}\left(s^{*}\right)\ge q_{{\pi }^{{}^{″}}}\left(s^{*},a^{*}\right)>v_{{\pi }^{{}^{\prime }}}\left(s^{*}\right).$$

However, $q_{{\pi }^{{}^{″}}}\left(s^{*},a^{*}\right)$ was strictly larger than $v_{\pi }\left(s^{*}\right)$ for any policy $\pi$, so we obtained the following contradiction:

$$v_{{\pi }^{{}^{\prime }}}\left(s^{*}\right)>v_{{\pi }^{{}^{\prime }}}\left(s^{*}\right)$$

Therefore, if a policy was optimal, then it would be optimal for all states in all steps. □

3.3.2. State-Updating Process

Based on the design of $\mathcal{S}$ and $\mathcal{A}$, new states could be observed after taking actions via $\mathcal{A}$ in both the MESS environment and the power-grid environment. The state-updating equations for time-step t were described separately for the three cases of charging, discharging, and holding, as follows:

$$\mathrm{charge}:\left\{\begin{array}{c}{P}_{u,t^{{}^{\prime }}}=P_{u,t}^{LP}\hfill \\ R{E}_{g,t^{{}^{\prime }}}={\mathcal{M}}_{re}\left(R{E}_{g,t}\right)+E_{g,t}\hfill \\ E_{g,t^{{}^{\prime }}}=E_{g,t}+P_{g,t}\Delta h\hfill \\ Loa{d}_{g,t^{{}^{\prime }}}={\mathcal{M}}_{load}\left(Loa{d}_{g,t}\right)\hfill \\ g^{{}^{\prime }}=Action\left(g\right)\hfill \end{array}\right.,$$

$$\mathrm{discharge}:\left\{\begin{array}{c}{P}_{u,t^{{}^{\prime }}}=P_{u,t}^{LP}\hfill \\ R{E}_{g,t^{{}^{\prime }}}={\mathcal{M}}_{re}\left(R{E}_{g,t}\right)-E_{g,t}\hfill \\ E_{g,t^{{}^{\prime }}}=E_{g,t}-P_{g,t}\Delta h\hfill \\ Loa{d}_{g,t^{{}^{\prime }}}={\mathcal{M}}_{load}\left(Loa{d}_{g,t}\right)\hfill \\ g^{{}^{\prime }}=Action\left(g\right)\hfill \end{array}\right.,$$

$$\mathrm{hold}:\left\{\begin{array}{c}{P}_{u,t^{{}^{\prime }}}=P_{u,t}^{LP}\hfill \\ R{E}_{g,t^{{}^{\prime }}}={\mathcal{M}}_{re}\left(R{E}_{g,t}\right)\hfill \\ E_{g,t^{{}^{\prime }}}=E_{g,t}\hfill \\ Loa{d}_{g,t^{{}^{\prime }}}={\mathcal{M}}_{load}\left(Loa{d}_{g,t}\right)\hfill \\ g^{{}^{\prime }}=Action\left(g\right)\hfill \end{array}\right.,$$

where ${\mathcal{M}}_{re}$ and ${\mathcal{M}}_{load}$ are the matrices that hold the renewable energy sources and loads, respectively.

3.3.3. Proximal-Gradient Algorithm

In accordance with our previous discussion in Section 3.3.1, we studied the MESSs challenge under the CMDP framework. Several approaches were available for ensuring safe exploration by the agent, including the primal-dual algorithm, the adaptive-penalty method, and the gradient-projection algorithm. However, for the purposes of this study, we deemed the proximal-gradient algorithm to be both readily implementable and computationally efficient.

In the normal gradient-descent process, the iterative equation was the following:

$$x_{t+1}=x_t-{\alpha }_t·f_t\left(x_t\right),$$

(26)

where ${\alpha }_t$ is the step length in step t. When the solution ${\tilde{x}}_t$ exceeded its feasible domain, we needed to project it back into the feasible domain. Therefore, the iterative equation was transformed into the following:

$$x_t={\mathcal{P}}_X\left(\widetilde{x_t}\right):=argmin\left\{{∥x-\widetilde{x_t}∥}^2:x\in X\right\},$$

(27)

where ${\mathcal{P}}_X$ is the projection operator, which was chosen as the Euclidean distance function in our study, and $\widetilde{x_t}:=x_{t-1}-{\gamma }_n{\widehat{\nabla }}_{x}f\left(x_{t-1}\right)$ denoted the original infeasible solution.

3.4. Learning Process

As mentioned in Section 3.2, this proposal employed the "$actor-critic$" (A–C) framework to train the policy NN and value NN, respectively. To ensure the network was convergent and stable, we used target networks that were added to each NN of the A–C structure and added an experience replay pool, such as the double-DQN(DDQN) [42]. The learning process was also similar to the update process of the DDQN.

The required inputs for the algorithm included the initialized parameters such as exploration parameter $ϵ$, mini-batch size B, a probability distribution $\xi$, etc. The capacity N of the experience replay memory $\mathcal{D}$; the parameters of the $Actor$ network $\mathrm{\Theta }$ and the $Critic$ network Q; and their target networks $\widehat{\mathrm{\Theta }}$ and $\widehat{Q}$, respectively, also needed to be initialized.

At the beginning of each episode $i\in I$, where I is the maximum training time, the agent acquired an initial state by observation and computed continuous action parameters through network $\mathrm{\Theta }$: $x_k\leftarrow x_k\left(s_t,{\theta }_t\right)$. Then, it selected an action $a_t=\left(k_t,x_{k_t}\right)$, according to the $ϵ$-greedy policy:

$$a_t=\left\{\begin{array}{c}\mathrm{a}\mathrm{sample}\mathrm{from}\mathrm{distribution}\xi \mathrm{with}\mathrm{probability}ϵ\hfill \\ \left(k_t,x_{k_t}\right)\mathrm{such}\mathrm{that}{k}_t=arg\underset{k\in K}{max}Q\left(s_t,k,x_k;{\omega }_t\right)\mathrm{with}\mathrm{probability}1-ϵ\hfill \end{array}\right.$$

in each decision time-step t. Then, we performed the action in the MESSs and power grid environment to define the new state $s_{t+1}$ and the reward $r_t$, which was obtained by solving the model (24).

Next, the transition tuple $\left[s_t,a_t,r_t,s_{t+1}\right]$ was stored in $\mathcal{D}$. Through sampling a batch ${\left\{s_b,a_b,r_b,s_{b+1}\right\}}_{b\in \left[B\right]}$ of size B, the target $y_b$ could be calculated via equation

$$y_b=\left\{\begin{array}{cc}{r}_b\hfill & \hfill \mathrm{if}{s}_{b+1}\mathrm{is}\mathrm{the}\mathrm{terminal}\mathrm{state},\\ r_b+\underset{k\in K}{max}\gamma Q\left(s_{b+1},k,x_k\left(s_{b+1},{\theta }_t\right);{\omega }_t\right)\hfill & \hfill \mathrm{if}\mathrm{otherwise}.\end{array}\right.$$

Then, the stochastic gradients ${\nabla }_{\omega }{\ell }_t^Q\left(\omega \right)$ and ${\nabla }_{\theta }{\ell }_t^{\mathrm{\Theta }}\left(\theta \right)$ were calculated through ${\left\{y_b,s_b,a_b\right\}}_{b\in \left[B\right]}$, as shown in Equations (22) and (23). The weights were updated by ${\theta }_{t+1}\leftarrow {\theta }_t-{\beta }_t{\nabla }_{\theta }{\ell }_t^{\mathrm{\Theta }}\left({\theta }_t\right)$. In the above processes, the proximal gradient projection mentioned in Section 3.3.3 was applied. We used a soft update (Polyak averaging) method to update the target networks $\widehat{Q}=(1-\tau )Q+\tau \widehat{Q}$ and $\widehat{\mathrm{\Theta }}=(1-\tau )\mathrm{\Theta }+\tau \widehat{\mathrm{\Theta }}$, where $\tau \in [0,1]$ is a hyper-parameter, usually $\tau \ll 1$.

4. Case Studies

In order to verify the effectiveness of the proposed strategy, case studies were implemented for the integrated model, utilizing simulation data. Our comparison of the energy output after incorporating MESSs demonstrated that the proposed strategy was effective in reducing the proportion of thermal power output and enhancing the utilization of renewable energy sources.

4.1. Experiment Settings

The training process for the 10-node scenario was approximately 5 h in length, on a desktop computer with an NVIDIA GTX 3080 GPU and an Intel i7-13700KF CPU, and approximately 6 h on a server with an NVIDIA Tesla P100.The solver for the MIP and LP models was Gurobi Optimizer, version 9.11.

The MESSs comprised an electric truck, a battery pack, and an operator. The unit movement cost $c^{TRA}$ incorporated all expenses associated with the movement, including the charging costs of the electric truck and the operator’s wages. The experimental arithmetic was represented by a topological diagram comprising 10 nodes that were interconnected by electrical conduits. The hyper-parameters of the experiment were subject to multiple simulations to find a balance between convergence speed and stability. The main simulation parameters are shown in

Table 2. Experiment Parameters.

Parameter Names	Value
Maximum capacity $\overline{E}$ (MWh)	27
Maximum charge/discharge power $pw{r}_{MAX}$ (MW)	27
Maximum number of nodes n	10
Maximum number of thermal power units	3
Charging and discharging efficiency $\eta$	0.95
Transportation cost per unit of time $c^{TRA}$ ($/h)	20
Random exploration parameter $\epsilon$	0.9–0.1
Batch size B	128
Discount factor $\gamma$	0.95
Probability distribution in random exploration $\xi$	U(0,1)
Soft updating parameter $\tau$	0.1
Sizes of three layers of neural networks	[256,128,64]
Learning rate in policy network $l{r}_p$	$1\times {10}^{-6}$
Learning rate in value network $l{r}_v$	$1\times {10}^{-4}$

.

4.2. Results Analysis

Figure 2 depicts the convergence of the cumulative rewards achieved by the proposed DRL-based algorithm during the training phase. The x-axis represented the number of iteration rounds, with each increment on the axis representing 100 iterations. The y-axis displayed the accumulated cost, which was presented as a negative value. The graph demonstrated that the training reached convergence after approximately 1500 iterations. The training process was approximately 5 h for 7000 iterations, whereas the execution time of the trained network was virtually insignificant. By utilizing a trained network, we observed a significant improvement in computational efficiency.

Figure 3 shows the alteration in the distribution of renewable energy and thermal unit outputs within the system. As depicted in Figure 3a, certain nodes, such as nodes 3 and 8, exhibited a richer availability of renewable energy sources, whereas others, such as nodes 1, 6, and 7, exhibited virtually no presence of such resources. To cater to the electricity demand of these nodes with limited renewable energy, it was necessary to utilize thermal power units. Figure 3b presents a polar plot, demonstrating the distribution of renewable energy and thermal unit outputs in the absence of a connection to the MESS system. The figure clearly illustrates that some nodes, such as node 5, exhibited a predominant presence of renewable energy output, while others, such as node 8, were dominated by thermal power generation. Overall, the system exhibited a relatively substantial proportion of thermal power generation.

Figure 3c presents a polar plot that displays the optimal energy distribution among various nodes. The blue, red, and green segments in the plot depicted the local renewable energy sources, the energy generated by thermal units, and the energy transmitted through the MESSs, respectively. As shown in the plot, certain nodes, such as nodes 5 and 9, possessed abundant local renewable energy resources and, thus, exhibited a higher proportion of local energy output. These nodes were also the main charging nodes and were not typically involved in discharging the MESSs.Comparatively, nodes such as node 2 had both local renewable energy resources and energy transmitted by the MESSs contributing to their power supply. Furthermore, nodes such as node 8, which primarily relied on the energy transmitted by the MESSs, effectively reduced their share of the thermal power-unit output.

Figure 4 presents a chord diagram that displays the cumulative energy transfer within the decision cycle of the MESSs. The diagram provided a visual representation of the flow of energy transfer and its accumulation during the decision cycle. The direction of the arrows represents the flow of energy transfer. As depicted in the figure, some nodes were designated as primary charging nodes, such as nodes 3 and 5, while others served as primary discharging nodes, such as nodes 1 and 8. This was because these nodes, such as 3 and 5, possessed abundant renewable energy sources, which was a result of their geographical locations and other contributing factors. However, when these nodes were not connected to the MESSs, the surplus of renewable energy resources could lead to issues such as excess light or wind. By incorporating these nodes into the MESSs, the proportion of thermal power output was significantly reduced, thus improving the utilization of renewable energy sources. Some nodes, such as nodes 3 and 8, served as both supply and demand nodes, which was due to the intermittent nature of renewable energy and the mismatch between the supply of renewable energy and the peak demand for electricity. For example, the power generation capacity of PV was related to factors such as sunlight duration and weather and reached its peak output during sunny afternoons, but the local demand may not be able to consume all the generated electricity during this time period. Therefore, the excess electricity would then be transported to nearby nodes to meet the electricity demand of other nodes. In the evening, when the power output of PV decreased, a hydroelectric power supply from neighboring nodes could be required.

By conducting numerous experiments, we demonstrated that by integrating the MESSs into the power grid, it was feasible to transfer surplus renewable energy from nodes with an abundance of such resources, to nodes with limited resources. This resulted in a reduction in thermal power generation in the latter, thereby validating the proposed model and algorithm.

Furthermore, our proposed mobile energy storage system-grid joint optimization model has the potential to address network contingency events. First, our study on coupling mobile energy storage systems with the power grid contributes to achieving higher renewable energy penetration levels [43,44]. In the event of network contingencies (e.g., equipment failure, fluctuations in power demand, etc.), mobile energy storage systems can quickly adjust their output to balance supply and demand imbalances in the grid. Moreover, mobile energy storage systems can transfer electrical energy from one location to another within the grid in a short period of time, alleviating local transmission congestion issues and improving the stability of the entire grid. Second, by optimizing the dispatch strategy of mobile energy storage systems, we can allocate renewable energy resources reasonably across different time periods to meet grid demand. This will help mitigate power supply shortages or overloads caused by contingency events, further enhancing the stability and reliability of the grid [29]. Lastly, our research also indicates that mobile energy storage systems can help reduce the operational costs of the entire power system [45]. By utilizing renewable energy resources more efficiently, we can decrease our reliance on conventional fossil fuel-based generation, thus lowering the cost losses associated with network contingencies.

5. Conclusions

In this study, we developed a mixed-integer nonlinear programming (MINP) model that coupled MESSs with a power grid to balance a region with an uneven distribution of renewable energy. We modeled the MESSs as constrained Markov decision processes (CMDPs) and proposed a framework based on a deep reinforcement learning (DRL) algorithm that considered the discrete-continuous hybrid action space of the MESSs. We solved the constraint on battery capacity in the CMDP by applying a proximal-gradient-projection algorithm. Based on this, we reformulated the original MINP model as a linear programming (LP) model to arrive at our solution framework. The case study showed that our proposed algorithm and framework effectively improved the utilization of renewable energy and reduced the generation costs of thermal power units.

The DRL-based algorithm we proposed has good scalability and application prospects in energy challenges with discrete-continuous hybrid decision-action spaces. Furthermore, our approach of using a DRL-based method to eliminate integer decision variables and nonlinear constraints provided insight for solving the MINP challenges. However, to highlight the main innovation points of this paper, we simplified some constraints, such as overlooking power-flow constraints and the aging characteristics of batteries. In future research, we will consider these additional constraints and objective functions more carefully, including the initial investment cost for implementing MESSs, and attempt to optimize scheduling decisions over longer time periods.