High-Speed Three-Dimensional Aerial Vehicle Evasion Based on a Multi-Stage Dueling Deep Q-Network

Abstract

This paper proposes a multi-stage dueling deep Q-network (MS-DDQN) algorithm to address the high-speed aerial vehicle evasion problem. High-speed aerial vehicle pursuit and evasion are an ongoing game attracting significant research attention in the field of autonomous aerial vehicle decision making. However, traditional maneuvering methods are usually not applicable in high-speed scenarios. Independent of the aerial vehicle model, the implemented MS-DDQN-based method searches for an approximate optimal maneuvering policy by iteratively interacting with the environment. Furthermore, the multi-stage learning mechanism was introduced to improve the training data quality. Simulation experiments were conducted to compare the proposed method with several typical evasion maneuvering policies and to reveal the effectiveness and robustness of the proposed MS-DDQN algorithm.

1. Introduction

The pursuit–evasion game is a challenging problem involving non-cooperative confrontation, which is currently an active research topic in aerial vehicle guidance, navigation, and control. In a pursuit–evasion problem, non-partners aerial vehicles have conflicts of interest while the trackers in the game are committed to adjusting their strategy to address a smaller miss. On the contrary, an evasion problem aims to find an optimal strategy for the tracked aerial vehicle to maximize the miss between the two. Most traditional aerial vehicle evasion strategies adopt typical evasion strategies to avoid the attack of the pursuit aerial vehicle [1], including sinusoidal [2], step [3], square [4], and spiral [5] maneuvers. Conventional maneuvering policies have proven effective at aircraft evasion when pursuers motorize with a low velocity and weak maneuverability [6]. Nevertheless, it is challenging to achieve a successful evasion when pursuers have strong maneuverability.

Differential game theory is particularly effective for solving problems with conflicting interests. Considering the one-pursuer one-evader (P1N1) scenario, transforming it into a two-player zero-sum game problem [6], the differential game attains the optimal evasion strategy for the evading aerial vehicle through iteratively solving the Hamilton function based on the confrontation model. Several optimal evasion strategies have been designed to employ differential game theory. In [7], the authors investigated the evasion differential game problem of an infinite-evader infinite-pursuer in the Hilbert Space $l_2$ and provided conditions for a successful escape. Liang et al. [8] discussed the game problem between an attacker and the target with an active defense function and utilized differential game theory to produce the evader’s winning regions. A sufficient condition of an M-pursuer N-evader differential game problem was given by Ibragimov et al. [9]. Rilwan et al. [10] improved the dynamic function of the differential game problem and solved the evasion problem of the one-pursuer-one-evader in the Hilbert space. Based on nonlinear control and game theories, [11] proposed a state-dependent Riccati equation method in a two-player zero-sum differential game scenario. Asadi et al. [12] developed an analytical closed-form solution to deal with the vehicle-target assignment in a pursuit–evasion problem. Nevertheless, a differential game-based evading strategy has not been implemented in realistic scenarios due to low stability, model dependence, weak robustness, and poor adaptability.

With the fast development of computing science and deep learning (DL) theory, artificial intelligence (AI) technology has drawn increasing attention. As an essential branch of AI, deep reinforcement learning (DRL) pushes the field of AI to a higher level of autonomous systems understanding. Combining DL and reinforcement learning (RL), DRL establishes a direct connection between the perception and decision making of autonomous systems employing deep neural networks (DNNs). Unlike supervised learning, DRL iteratively learns the optimal policy by interacting with the environment instead of developing a regression model utilizing predefined labeled data [13]. The application of DRL technology is growing exponentially since the Alpha Go agent trained by Google DeepMind won the Go competition in 2016 [14]. DRL has several unique characteristics. First, DRL can generate an optimal policy for agents without recognizing the accurate mathematical model of both the environment and the agents. In particular, DRL shows significant performance when dealing with high model uncertainty and parameter fluctuation scenarios. Second, DRL shows adaptability in different environments since it actively interacts with the environment when generating a policy and affords a small exploration probability to guarantee sufficient environmental exploration even when the current policy is good enough.

Research on introducing DRL technology into pursuit–evasion problems has increased dramatically during the last five years. Sun et al. [15] introduced an adaptive dynamic programming (ADP) framework as an equation solver for differential games that successfully approximates the guidance law’s optimal solution. Gaudet et al. [16] developed an RL-based method for the interception problem that only depends on the measurement of line-of-sight angles and angular rates. Zhu et al. [17] proposed an aerial vehicle evasion strategy based on DQN, while Li et al. [18] introduced a deep deterministic policy gradient (DDPG) framework that considers small vehicle evasion strategies. Shalumov et al. [19] presented an optimal launch time and a guidance law via DRL in a target–missile–defender scenario. The work of Souza et al. [20] proposed a DRL method for decentralized multi-agent pursuit problems. In [21], Tipaldi et al. provided a detailed literature review of RL-based methods for aircraft pursuit and evasion.

Nevertheless, only a few methods for solving aircraft pursuit–evasion problems have been specifically designed for aircraft evasion, leading to aircraft evasion technology not meeting the current practical application requirements. Therefore, this paper proposes a multi-stage dueling DQN (MS-DDQN) algorithm to address the evasion problem of a vehicle equipped with ignition pulse engines. Furthermore, the integration of multi-stage learning technology significantly speeds up the training process. The simulation results indicate that the evasion strategy generated by the proposed MS-DDQN can maximize the miss and escape from the pursuer successfully.

The main contributions of this paper are the following:

• An MS-DDQN algorithm is proposed to improve the typical DQN algorithm to accelerate the convergence process and modify the quality of training data;
• An adaptive iterative learning framework is implemented for high-speed aerial vehicle evasion problems;
• Adequate comparative simulation experiments are implemented to verify the effectiveness and robustness of the proposed MS-DDQN algorithm.

The remainder of this paper is organized as follows: Section 2 presents the fundamental and problem formation. Section 3 introduces the MS-DDQN model and proposes the learning process to obtain the optimal evasion strategy. Section 4 presents the simulation experiments, and finally, Section 5 concludes this work.

2. Fundamental and Problem Formulation

2.1. Aerial Vehicle Pursuit–Evasion Model

This section proposes the pursuit–evasion model of two vehicles to provide an interactive simulation environment for the problem.

It is a one-pursuer one-evader scenario where two agents are moving toward one another, whose projection distance in the X-axis of the velocity–turn–climb coordinate system is $r_0=150 \mathrm{km}$. The pursuer uses a proportional navigation guidance (PNG) law to attack the evader with a maneuver period of $T_P=15 \mathrm{ms}$ and the maximum acceleration is $a_{maxP}=4.5 \mathrm{g}$, where $g=9.8 {\mathrm{m}/\mathrm{s}}^2$. The evader has four typical ignition pulse engines with a maneuver period of $T_E=50 \mathrm{ms}$ to avoid the pursuer’s attack with a maximum acceleration of $a_{maxE}=7.5 \mathrm{g}$. The orientation of the acceleration generated by the engines is perpendicular to the direction of the body. To make our testing scenario more challenging, we chose a larger velocity. The pursuer’s velocity was $V_P=1.875 \mathrm{km}/\mathrm{s}$, and the evader’s velocity was $V_E=4.5 \mathrm{km}/\mathrm{s}$. In this scenario, the evader is the agent that needs to be trained, while the pursuer is part of the simulation environment for the DRL problem described in this scenario. The agent learns the optimal evasion strategy by interacting with the pursuer to maximize the minimal miss, thoroughly discussed in Section 3.

The inertial reference frame, velocity–turn–climb coordinate frame, and line-of-sight (LOS) coordinate frame are denoted as $OXYZ\left(S_0\right)$, $O_2{X}_2{Y}_2{Z}_2\left(S_2\right)$, and $O_4{X}_4{Y}_4{Z}_4\left(S_4\right)$, respectively. For simplicity, the pursuer is denoted as P and the evader as E. Let ${{\theta }_i|}_{i=P,E}$ be the flight path angle and ${{\psi }_{vi}|}_{i=P,E}$ the heading angle. $q_{ϵ}$ and $q_{\beta }$ are the horizontal and vertical LOS angles, and the horizontal and vertical LOS rotational rates are represented by ${\dot{q}}_{ϵ}$ and ${\dot{q}}_{\beta }$, respectively. As presented in Equations (1) and  (2), ${\mathit{C}}_{\mathbf{2}-\mathbf{0}}$ and ${\mathit{C}}_{\mathbf{4}-\mathbf{0}}$ are the transformation matrices from $S_2$ to $S_0$ and from $S_4$ to $S_0$, respectively.

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\mathit{C}}_{\mathbf{2}-\mathbf{0}}={\left[\left.\begin{array}{ccc}cos{\theta }_{i}cos{\psi }_{vi}& -sin{\theta }_{i}sin{\psi }_{vi}& sin{\psi }_{vi}\\ sin{\theta }_i& cos{\theta }_i& 0\\ -cos{\theta }_{i}sin{\psi }_{vi}& sin{\theta }_{i}sin{\psi }_{vi}& cos{\psi }_{vi}\end{array}\right]\right|}_{i=P,E},\end{array}\end{array}$$

(1)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\mathit{C}}_{\mathbf{4}-\mathbf{0}}=\left[\begin{array}{ccc}cos{q}_{ϵ}cos{q}_{\beta }& -sin{q}_{ϵ}sin{q}_{\beta }& sin{q}_{\beta }\\ sin{q}_{ϵ}& cos{q}_{ϵ}& 0\\ -cos{q}_{ϵ}sin{q}_{\beta }& sin{q}_{ϵ}sin{q}_{\beta }& cos{q}_{\beta }\end{array}\right].\end{array}\end{array}$$

(2)

Figure 1 Left illustrates the conversion relationship between $S_0$ and $S_2$, where $\theta$ is the flight path angle and ${\psi }_v$ is the heading angle. Figure 1 Right depicts the relationship between $S_0$ and $S_4$ defined by the LOS angles $q_{ϵ}$ and $q_{\beta }$.

Let ${\mathit{X}}_{\mathit{i}}={\left[\begin{array}{ccc}{x}_i& y_i& z_i\end{array}\right]|}_{i=P,E}$ be the agent’s location in $S_0$ and let ${\mathit{V}}_{\mathbf{2}\mathit{i}}={\left[\begin{array}{ccc}{V}_i& 0& 0\end{array}\right]|}_{i=P,E}$ be the vehicle’s velocity in $S_2$. Then, we have:

$$\begin{array}{c}\hfill {\mathit{V}}_{\mathbf{0}\mathit{i}}={\mathit{C}}_{\mathbf{2}-\mathbf{0}}{\mathit{V}}_{\mathbf{2}\mathit{i}}={\left[\begin{array}{ccc}{v}_{0xi}& v_{0yi}& v_{0zi}\end{array}\right]|}_{i=P,E},\end{array}$$

(3)

where ${\mathit{V}}_{\mathbf{0}\mathit{i}},i=P,E$ is the vehicle’s velocity in $S_0$. The location’s derivative can be depicted as:

$$\begin{array}{c}\left\{\begin{array}{c}{\dot{x}}_i=V_{i}cos{\theta }_{i}cos{{\psi }_{vi}|}_{i=P,E}\hfill \\ {\dot{y}}_i=V_{i}cos{{\theta }_i|}_{i=P,E}\hfill \\ {\dot{z}}_i=-V_{i}cos{\theta }_{i}sin{{\psi }_{vi}|}_{i=P,E}\hfill \end{array}\right..\hfill \end{array}$$

(4)

Let ${\mathit{a}}_{\mathbf{4}\mathit{i}}={\left[\begin{array}{ccc}0& a_{iy}& a_{iz}\end{array}\right]|}_{i=P,E}$ be the aerial vehicle’s acceleration in $S_4$; then, we have:

$$\begin{array}{c}\hfill {\mathit{a}}_{\mathbf{0}\mathit{i}}={\mathit{C}}_{\mathbf{4}-\mathbf{0}}{{\mathit{a}}_{\mathbf{4}\mathit{i}}|}_{i=P,E},\end{array}$$

(5)

where ${{\mathit{a}}_{\mathbf{0}\mathit{i}}|}_{i=P,E}$ is the vehicle’s acceleration in $S_0$. ${\dot{\mathit{V}}}_{0i}$ is also reformulated as:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\dot{v}}_{0xi}=V_{i}cos{q}_{ϵ}cos{q_{\beta }|}_{i=P,E}\hfill \\ {\dot{v}}_{0yi}=V_{i}cos{q_{ϵ}|}_{i=P,E}\hfill \\ {\dot{v}}_{0zi}=-V_{i}cos{q}_{ϵ}sin{q_{\beta }|}_{i=P,E}\hfill \end{array}\right..\end{array}$$

(6)

Equations (7)–(10) are the formulas of $q_{ϵ}$, $q_{\beta }$, ${\dot{q}}_{ϵ}$, and ${\dot{q}}_{\beta }$ in $S_4$, respectively.

$$\begin{array}{c}\hfill \begin{array}{c}\hfill q_{ϵ}=arctan\frac{r_y}{\sqrt{r_x^2+r_z^2}},\end{array}\end{array}$$

(7)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill q_{\beta }=-arctan\frac{r_z}{r_x},\end{array}\end{array}$$

(8)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\dot{q}}_{ϵ}=\frac{\left(r_x^2+r_z^2\right){\dot{r}}_y-r_y\left(r_x{\dot{r}}_x+r_z{\dot{r}}_z\right)}{\left(r_x^2+r_y^2+r_z^2\right)\sqrt{r_x^2+r_z^2}},\end{array}\end{array}$$

(9)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\dot{q}}_{\beta }=\frac{r_z{\dot{r}}_x-r_x{\dot{r}}_z}{r_x^2+r_z^2},\end{array}\end{array}$$

(10)

where $r_x=x_E-x_P$, $r_y=y_E-y_P$, $r_z=z_E-z_P$, ${\dot{r}}_x={\dot{x}}_E-{\dot{x}}_P$, ${\dot{r}}_y={\dot{y}}_E-{\dot{y}}_P$, and ${\dot{r}}_z={\dot{z}}_E-{\dot{z}}_P$.

According to the basic principle of PNG, the pursuer’s acceleration of ${\mathit{a}}_{\mathbf{4}\mathit{P}}$ in $S_4$ is given by:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\mathit{a}}_{\mathbf{4}\mathit{P}}=\left[\begin{array}{ccc}0& N_{4y}v{\dot{q}}_{ϵ}& N_{4z}v{\dot{q}}_{\beta }\end{array}\right]\hfill \\ r=\sqrt{r_x^2+r_y^2+r_z^2}\hfill \\ vs.=\frac{r_x{\dot{r}}_x+r_y{\dot{r}}_y+r_z{\dot{r}}_z}{r}\hfill \end{array}\right.,\end{array}$$

(11)

where $N_{4y}=4$ and $N_{4z}=5$ are the PNG’s navigation ratios. Then, the aerial vehicle pursuit-evasion model can be defined by combining Equations (3)–(11)

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\mathit{X}}_{\mathit{i}}={\left[\begin{array}{ccc}{x}_i& y_i& z_i\end{array}\right]|}_{i=P,E}\hfill \\ {\dot{x}}_i=V_{i}cos{\theta }_{i}cos{{\psi }_{vi}|}_{i=P,E}\hfill \\ {\dot{y}}_i=V_{i}cos{{\theta }_i|}_{i=P,E}\hfill \\ {\dot{z}}_i=-V_{i}cos{\theta }_{i}sin{{\psi }_{vi}|}_{i=P,E}\hfill \\ {\mathit{V}}_{\mathbf{0}\mathit{i}}={\left[\begin{array}{ccc}{v}_{0xi}& v_{0yi}& v_{0zi}\end{array}\right]|}_{i=P,E}\hfill \\ {\mathit{a}}_{\mathbf{0}\mathit{i}}={\mathit{C}}_{\mathbf{4}-\mathbf{0}}{{\mathit{a}}_{\mathbf{4}\mathit{i}}|}_{i=P,E}\hfill \\ {\mathit{a}}_{\mathbf{4}\mathit{P}}=\left[\begin{array}{ccc}0& N_{4y}v{\dot{q}}_{ϵ}& N_{4z}v{\dot{q}}_{\beta }\end{array}\right]\hfill \\ {\mathit{a}}_{\mathbf{4}\mathit{E}}=\left[\begin{array}{ccc}0& a_{4Ey}& a_{4Ez}\end{array}\right]\hfill \end{array}\right.,\end{array}$$

(12)

where $a_{4ey}$ and $a_{4ez}$ are the evader’s accelerations in $S_4$.

2.2. Value-Based Reinforcement Learning

An RL problem can be uniquely defined by a Markov decision process (MDP) [22] tuple $\langle \mathcal{S},P,\mathcal{R},\gamma ,\mathcal{A}\rangle$, where $\mathcal{S}$ is the state space of the agent, P is the state transition probability matrix, $\mathcal{R}$ is the reward function, $\gamma$ is the discount factor of the cumulative reward, and $\mathcal{A}$ is the action space. MDP is the fundamental framework of RL, where each episode starts with the initial condition and ends with the termination condition. When the agent takes action, it gets an immediate reward $R_t$ and transfers to another state $s_{t+1}$. An RL problem intends to search for an optimal policy for a given MDP to maximize the cumulative reward $G_t$, introduced in Equation (13), and the optimal policy ${\pi }^{*}$ is a distribution of an action space $\mathcal{A}$ for a fixed state s, introduced in Equation (14).

$$\begin{array}{c}\hfill \begin{array}{c}\hfill G_t=\sum _{k=0}^T{\gamma }^k{r}_{t+k+1}=r_{t+1}+\gamma G_{t+1},\end{array}\end{array}$$

(13)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \begin{array}{c}\hfill {\pi }^{*}\left(a|s\right)=P\left[A_t=a|S_t=s\right]\end{array}.\end{array}\end{array}$$

(14)

In this aerial vehicle evasion problem, the state is the line-of-sight angles ($q_{\epsilon }$ and $q_{\beta }$) and the distance (r) of the two aerial vehicles, while the action is the operational status of the four pulse engines on or off. The designs of $\gamma$ and $\mathcal{R}$ are implemented in Section 3.3, and the termination conditions of this problem are as follows:

• The evader is successfully intercepted by the pursuer, which is called failure evasion;
• The evader escapes from the attack of the pursuer successfully, which is called a successful evasion.

The RL algorithms comprise value-based [23], policy-based [24], and actor-critic [25]. For a value-based discrete time MDP (DT-MDP), to maximize the cumulative reward $G_t$, the agent chooses an action according to the state-value function (Q-Function; Equation (15)) to iteratively update the policy until $\pi$ converges to the optimal solution ${\pi }^{*}$.

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill Q_{\pi }\left(s,a\right)& ={\mathbb{E}}_{\pi }\left[G_t|S_t=s_t,A_t=a\right]\hfill \\ \hfill & ={\mathbb{E}}_{\pi }\left[\sum _{k=0}^T{\gamma }^k{r}^{t+k+1}|S_t=s_t,A_t=a\right].\hfill \end{array}\end{array}$$

(15)

More specifically, a neural network is chosen as a value function approximator of Equation (15). In the optimal scenario, the agent chooses a greedy policy according to ${\pi }^{*}$:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\pi }^{*}\left(s\right)=arg\underset{a\in \mathcal{A}}{max}{Q}^{{\pi }^{*}}\left(s,a,\omega \right),\end{array}\end{array}$$

(16)

where $\omega$ is the parameter of the neural network.

3. MS-DDQN Algorithm Design

This section introduces the basic structure of the dueling DQN framework, proposes the MS-DDQN algorithm, and provides details on the learning process.

3.1. Dueling DQN Framework

The MS-DDQN framework is a multi-stage value-based DRL method, which improves the traditional DQN. The DQN establishes a direct connection between perception and the decision-making level of an autonomous system [26], as illustrated in Figure 2, where the replay memory collects data generated from the interaction between the agent (Evaluation Net) and the environment.

The sign of the convergence of the algorithm TD error converges to 0, and the TD error is calculated in the loss function module employing the formula ${\Delta }_{TD}=r+\underset{a^{\prime }\in \mathcal{A}}{max}Q\left(s^{\prime },a^{\prime };\theta \right)-Q\left(s,a;\theta \right)$. The $\theta$ parameters in the evaluation net are updated by the loss function utilizing a suitable gradient descent algorithm. Moreover, the ${\theta }^{-}$ parameters in the target net are copied from $\theta$ every C step, where C is a constant. The learning process ends when ${\Delta }_{TD}\to 0$.

However, DQN overfits the neural network approximator, overestimating the state’s value function. To overcome this drawback, Wang et al. [27] proposed a dueling network architecture that weakens the Q-network’s overestimation. Specifically, the dueling Q-network defines an advantage function to decouple the state-value function and the state-action-value function, as shown in Equation (17):

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Q\left(\mathcal{S},\mathcal{A},\omega ,\alpha ,\beta \right)=V\left(\mathcal{S},\omega ,\alpha \right)+A\left(\mathcal{S},\mathcal{A},\omega ,\beta \right),\end{array}\end{array}$$

(17)

where $\omega$, $\alpha$, and $\beta$ are the parameters that need to be optimized. Specifically, $\alpha$ is the parameter of the value function $V\left(\mathcal{S},\omega ,\alpha \right)$, $\beta$ is the parameter of the advantage function $A\left(\mathcal{S},\mathcal{A},\omega ,\beta \right)$, and $\omega$ is the parameter that two functions have in common. Equation (17) is often written in the form of Equation (18) to reflect the identifiability of simulated physical experiments.

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill Q\left(\mathcal{S},\mathcal{A},\omega ,\alpha ,\beta \right)& =V\left(\mathcal{S},\omega ,\alpha \right)\hfill \\ \hfill +& \left[A\left(\mathcal{S},\mathcal{A},\omega ,\beta \right)-\frac{1}{\mathcal{A}}\sum _{a^{\prime }\in \mathcal{A}}A\left(\mathcal{S},a^{\prime },\omega ,\beta \right)\right].\hfill \end{array}\end{array}$$

(18)

Figure 3 illustrates the difference between DQN and dueling DQN. Both networks involve m hidden layers and the same input–output dimension. The last hidden layer in the bottom figure of Figure 3 is divided into two parts: $V\left(\mathcal{S},\omega ,\alpha \right)$ and $A\left(\mathcal{S},\mathcal{A},\omega ,\beta \right)$, while the dueling DQN output is the sum of $V\left(·\right)$ and $A\left(·\right)$.

Figure 4 illustrates the complete learning process. In Figure 4, $ODE$ is the model state-solving module for state update, $CF$ is the coordinate transformation module, and $PG$ is the proportional guide law module that generates the acceleration command for the pursuer. The “policy learning” module in Figure 4 is the RL learning process shown in Figure 2. The “RL maneuver” is the training result of “policy learning”, which is used to generate acceleration commands for the evader. The entire framework is a recurrent process and terminates when the evader can get rid of the pursuer’s attack.

3.2. Multi-Stage Learning

DRL methods iteratively learn the optimal policy through interacting with the environment. Nonetheless, it is almost impossible for an agent to collect adequate high-quality sampling data with an initial stochastic policy when the state space is vast. Therefore, a multi-stage learning scheme is proposed to speed up the agent’s learning process by inserting several sub-mission nodes during the entire learning process to guide the agent step-by-step in reasonably learning the optimal policy. This idea is inspired by humans’ solutions to complex problems, as we typically decompose a considerable problem into several smaller ones that are serially connected. The solution of a sub-task is the initial condition of the next one.

The entire task is divided into two sub-tasks for the evasion problem examined in this paper. In the first training phase, the agent learns an applicable policy ${\pi }_{temp}$ with a certain stochastic exploration probability. ${\pi }_{temp}$ is a relatively ideal initial condition for second-stage training, although not the optimal policy with a high probability. The objective of second-stage training is to find an optimal policy ${\pi }^{\star }$ that maximizes the miss based on ${\pi }_{temp}$ by using the adequate high-quality data generated by the initial acceptable policy in the first stage of learning. Algorithm 1 presents the pseudo-code of this multi-stage training framework.

Algorithm 1 A multi-stage training framework with N sub-missions.

Input: N, Reward function for each phase $f_i,i=1,2,\cdots ,N$, an initial policy ${\pi }_0$. 1: Output: The optimal policy ${\pi }^{\star }$ 2: Environment initialization: $i=0$, empty the replay memory. 3: while$i<N$do 4:     Initialize the neural network. 5:     Fill the replay memory with sampling data generated by policy ${\pi }_i,{\epsilon }_i$. (${\epsilon }_i$ is a certain initial exploration probability for the $ith$ phase.) 6:     while ${\pi }_i$ is not converged do 7:         Network training in Dueling-DQN framework with reward function $f_i$. 8:         Update ${\pi }_i$ and replay memory. 9:     Save ${\pi }_i$, $i+=1$. 10: return  ${\pi }^{\star }$

3.3. Complete Learning Framework

The MS-DDQN method has four key elements: State space, action space, reward functions for each training phase, and the neural network structure. The choice of state space predominantly affects the quality of the final policy. In the vehicle evasion scenario, the distance, and LOS angles ($q_{ϵ}$ and $q_{\beta }$) are the three crucial factors, and Equation (19) denotes the agent’s state:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill s=\left[r,q_{ϵ},q_{\beta }\right],\end{array}\end{array}$$

(19)

where r is the distance between the two vehicles and $q_{ϵ}$ and $q_{\beta }$ are the LOS angles. The evader in this scenario maneuvers with four ignition pulse engines, hence the evader has nine maneuvering types (

Table 1. Action space of the evader.

Action	Acceleration	Action	Acceleration	Action	Acceleration
0	$\left[\begin{array}{ccc}0& 0& 0\end{array}\right]$	3	$\left[\begin{array}{ccc}0& 0& +max\end{array}\right]$	6	$\left[\begin{array}{ccc}0& +max& -max\end{array}\right]$
1	$\left[\begin{array}{ccc}0& +max& 0\end{array}\right]$	4	$\left[\begin{array}{ccc}0& 0& -max\end{array}\right]$	7	$\left[\begin{array}{ccc}0& -max& +max\end{array}\right]$
2	$\left[\begin{array}{ccc}0& -max& 0\end{array}\right]$	5	$\left[\begin{array}{ccc}0& +max& +max\end{array}\right]$	8	$\left[\begin{array}{ccc}0& -max& -max\end{array}\right]$

); the action’s dimension is one, with the agent selecting one of the nine actions according to the policy each time.

The three elements in each action in Table 1 are the evader’s acceleration in X, Y, and Z. ‘$+max$’ means the acceleration reaches the maximum value, and ‘$-max$’ is the minimum value (reverse maximum).

The choice of the reward function is another decisive factor in the training process. It should be a continuous function of the state, with the reward function of the first and second training phases given in Equations (20) and (21), respectively.

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{r}_1=k_{1}max{\left(\left|q_{ϵ}\right|,\left|q_{\beta }\right|\right)}^2\hfill \\ r_2=\left\{\begin{array}{c}10000\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} if evasion succeeded\hfill \\ 0\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} if evasion failed\hfill \end{array}\right.\hfill \\ r=r_1+r_2\hfill \\ k_1=\left\{\begin{array}{c}200tan0.1047t-1.5184 \hspace{1em}if action=0,1,2,3,4\hfill \\ 50tan0.1047t-1.5184\hspace{1em} if action=5,6,7,8\hfill \end{array}\right.\hfill \end{array}\right.,\end{array}$$

(20)

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{r}_1=k_{1}max{\left(\left|q_{ϵ}\right|,\left|q_{\beta }\right|\right)}^2\hfill \\ r_2=\left\{\begin{array}{c}1000r^2\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} if evasion succeeded\hfill \\ 0 \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} if evasion failed\hfill \end{array}\right.\hfill \\ r=r_1+r_2\hfill \\ k_1=\left\{\begin{array}{c}200tan0.1047t-1.5184\hspace{1em} if action=0,1,2,3,4\hfill \\ 50tan0.1047t-1.5184\hspace{1em} if action=5,6,7,8\hfill \end{array}\right..\hfill \end{array}\right.\end{array}$$

(21)

In Equations (20) and (21), the reward function is divided into two parts: The reward for the approaching process $r_1$ and the reward for the episode termination $r_2$. $k_1$ is the coefficient of $r_1$, which is originally designed to affect the sensitivity of $r_1$ to the LOS angles and increases with time. The design of $k_1$ formation guides the agent to consume the remaining fuel at the end of the escape to increase its line of sight with the pursuer. The introduction of the tangent function ensures that the reward is more sensitive to the agent’s action as time increases. The purpose of $r_1$ is to maximize one of the LOS angles, considering that the Y-direction is symmetrical to the Z-direction and $r_2$ is different in the two training phases. Therefore, the aim is to find, for the aerial vehicle, an acceptable policy in the first training phase and to maximize the miss in the second training phase.

Additionally, $r_1$ still exists in the second learning stage to avoid sparse rewards in RL. A sparse reward problem, i.e., the agent can get a nonzero reward if, and only if, it completes an episode of exploration, means that the agent does not have adequate data to modify its action [28]. It is hard for RL algorithms to converge when dealing with sparse reward problems. Therefore, both $r_1$ and $r_2$ are added to the reward function to ensure the adequacy of the training data.

The MS-DDQN estimates the policy’s value function through a Q-network.

Table 2. Structure of the neural network.

Layer	Input	Output	Activation Function
1	3	20	$ReLu\left(\right)$
2	20	20
3V	20	9
3A	20	9

presents the neural network structure, where $3V$ and $3A$ are the value and advantage parts of the Q function, respectively, and $ReLu\left(\right)$ is the activation function of the neural network.

Figure 5 illustrates the flow diagram of the training process. First, the flow chart presents the left channel to start first-phase training. Then, it continues to perform the second training phase after an acceptable policy is obtained, which is the initial policy of the second phase. The capacities of the two-training phase are hyper-parameters that need to be tuned manually. Finally, the algorithm stops when the target network converges. Furthermore, this paper adopted the network-retraining technique proposed in [29,30,31] to strengthen the network training quality.

The network training consists of procedures, initialization, and training. The data input into the replay memories during the initialization part are generated by the current temporary optimal policy with certain exploration probabilities ${ϵ}_1=0.2$ and ${ϵ}_2=0.2$. For the first stage of the training part, the optimal temporary policy is stochastic. In contrast, the second stage of the training part takes the policy generated by the first stage as the initial temporary optimal policy. Figure 6 depicts the relationship between the episode and exploration probability during the training process. The functional relationship between the exploration probability and episode is designed as an exponential function to make the agent’s exploration more adequate. The mathematical function of each segment in Figure 6 is presented at the right top of the figure. Episodes 0–1499 belong to the first training of the first phase, while episodes 1500–2099 are the second training of the first phase, and episodes 2100–2999 belong to the training in the second phase.

Table 3. Some hyper-parameters of MS-DDQN.

	Phase	Phase1-a	Phase1-b	Phase2
Parameter
$\gamma$		0.9	0.9	0.9
${ϵ}_0$		0.9	0.9	0.9
$\alpha$		0.01	0.01	0.01
M		5000	5000	2000
$Batch Size$		64	64	64

lists the hyper-parameters of MS-DDQN, where $\gamma$ denotes the discount factor of RL, ${ϵ}_0$ is the initial exploration probability, $\alpha$ is the learning rate of the Q-network, M is the capacity of the replay memory, and $Batch Size$ is the number of samples extracted each time during the Q-network learning.

4. Simulation Experiments

This section evaluates the evasion performance employing the strategy generated by MS-DDQN. For clarity, we rearranged the parameters presented in Section 2.1 into

Table 4. Parameters of the simulation scenario.

Parameter	Value	Parameter	Value
$r_0$	$150 \mathrm{km}$	g	$9.8 {\mathrm{m}/\mathrm{s}}^2$
$T_P$	$15 \mathrm{ms}$	$T_E$	$50 \mathrm{ms}$
$a_{maxP}$	$4.5 \mathrm{g}$	$a_{maxE}$	$7.5 \mathrm{g}$
$V_P$	$1.875 \mathrm{km}/\mathrm{s}$	$V_E$	$4.5 \mathrm{km}/\mathrm{s}$

.

Here, $r_0$ is the initial relative distance in the LOS coordinate frame ($S_4$) of the two vehicles, ${T_i|}_{i=P,E}$ is the maneuver cycle of the vehicle, ${a_{maxi}|}_{i=P,E}$ is the maximum acceleration of the vehicle in the LOS coordinate frame ($S_4$), and ${V_i|}_{i=P,E}$ is the velocity of the vehicle in LOS coordinate frame ($S_4$).

Figure 7 represents the relationship between the cumulative reward and episode in both multi- and single-phase training. Simultaneously, the curves verify the effectiveness of the multi-phase training. Figure 7 highlights that the average reward increased significantly as the episode increased in phase1 training. However, the average reward decreased slightly in episodes 1500–2100 due to the increase in $ϵ$ (see Figure 6), which means the agent explored the environment stochastically with a higher probability. Based on phase1, the average reward of phase2 was substantially higher than that of phase1, indicating a successful evasion and the effectiveness of the two-stage training technique implemented in the training process. Finally, the evader achieved a high-quality policy after 2700 training episodes. However, without multi-phase training, the single-phase training process maintained a low-level cumulative reward throughout the entire training process.

Comparative simulation experiments were designed on the miss distances of the evader using various maneuvers. Figure 8 and Figure 9 illustrate the simulation results when the evader implemented a random maneuvering policy and a square wave maneuver policy, respectively.

The misses were $0.20 \mathrm{m}$ and $0.26 \mathrm{m}$, respectively, and the simulation time is $24.057 \mathrm{s}$. The results indicate that the evader could not use traditional maneuvers to avoid the pursuer’s attack. Figure 10, Figure 11 and Figure 12 present the simulation results when the evader applied a neural network maneuver strategy generated by MS-DDQN. The scenarios involved nine relative locations with an initial flight path angle $\theta$ and a heading angle of 0. We conclude that the miss is much more significant than the traditional maneuver.

Figure 10 presents the evasion process when the evader implemented the policy generated by MS-DDQN with the initial relative positions $\left(-20 \mathrm{km},-20 \mathrm{km}\right)$, $\left(-20 \mathrm{km},0 \mathrm{km}\right)$, and $\left(-20 \mathrm{km},20 \mathrm{km}\right)$ in the initial inertial coordinate system. The results indicate that the slightest miss of the three scenarios was $18.4174 \mathrm{m}$, satisfying the problem requirement. Additionally, the figures for $time-a_y$ and $time-a_z$ illustrate that the evader tended to consume all of the remaining fuel at the end of the interception to increase its line of sight with the pursuer, which can be found in the figure for $time-a_y$. Allocating all fuel in the Y-direction, the output of the Z-direction remained zero throughout the process due to the limited fuel. Figure 11 and Figure 12 indicate the results from the same law as Figure 10.

To further demonstrate the feasibility of the MS-DDQN maneuver, more simulation experiments with different relative initial positions, flight path angles $\theta$, and heading angles ${\Psi }_v$ are presented.

Table 5. Miss distance (m) with initial flight path angle $\theta =10°$.

Initial Relative Distance (km)	Heading Angle ${\mathbf{\Psi }}_{\mathit{v}}\mathbf{(}\mathbf{°}\mathbf{)}$
	10	5	0	$\mathbf{-}\mathbf{5}$	$\mathbf{-}\mathbf{10}$
$\left(20,20\right)$	19.81	23.25	25.17	25.36	25.45
$\left(20,0\right)$	23.87	25.41	25.61	25.43	24.21
$\left(20,-20\right)$	25.69	25.53	24.90	23.12	18.80
$\left(0,20\right)$	24.11	28.97	29.90	30.50	30.69
$\left(0,0\right)$	30.29	30.71	30.75	30.48	29.71
$\left(0,-20\right)$	30.93	30.49	29.76	28.32	23.92
$\left(-20,20\right)$	13.71	9.79	13.15	13.79	14.19
$\left(-20,0\right)$	13.08	13.82	13.82	13.5	12.50
$\left(-20,-20\right)$	14.38	13.71	12.56	9.79	6.67

,

Table 6. Miss distance (m) with initial flight path angle ($\theta =5°$).

Initial Relative Distance (km)	Heading Angle ${\mathbf{\Psi }}_{\mathit{v}}\mathbf{(}\mathbf{°}\mathbf{)}$
	10	5	0	$\mathbf{-}\mathbf{5}$	$\mathbf{-}\mathbf{10}$
$\left(20,20\right)$	20.17	23.30	25.63	26.21	26.13
$\left(20,0\right)$	24.64	26.02	26.25	25.56	24.02
$\left(20,-20\right)$	25.83	25.98	25.26	23.60	19.21
$\left(0,20\right)$	28.93	31.25	31.93	32.21	32.42
$\left(0,0\right)$	32.44	32.56	32.44	32.22	31.72
$\left(0,-20\right)$	32.67	32.22	31.64	30.49	28.12
$\left(-20,20\right)$	14.14	17.39	16.93	17.36	17.90
$\left(-20,0\right)$	17.97	17.34	17.18	17.20	16.87
$\left(-20,-20\right)$	17.99	17.21	16.69	15.18	13.66

,

Table 7. Miss distance (m) with initial flight path angle $\theta =0°$.

Initial Relative Distance (km)	Heading Angle ${\mathbf{\Psi }}_{\mathit{v}}^{\mathbf{°}}$
	10	5	0	$\mathbf{-}\mathbf{5}$	$\mathbf{-}\mathbf{10}$
$\left(20,20\right)$	16.61	22.12	25.19	25.47	25.37
$\left(20,0\right)$	23.82	25.14	25.63	25.36	23.65
$\left(20,-20\right)$	25.59	25.72	24.68	22.12	17.41
$\left(0,20\right)$	30.45	32.10	32.80	33.01	33.24
$\left(0,0\right)$	33.33	33.39	33.23	33.07	32.54
$\left(0,-20\right)$	33.48	33.02	32.53	31.22	29.00
$\left(-20,20\right)$	16.69	19.12	18.57	19.06	19.64
$\left(-20,0\right)$	19.22	19.18	18.81	18.83	19.49
$\left(-20,-20\right)$	19.68	18.96	18.42	18.07	18.27

,

Table 8. Miss distance (m) with initial flight path angle $\theta =-5°$.

Initial Relative Distance (km)	Heading Angle ${\mathbf{\Psi }}_{\mathit{v}}^{\mathbf{°}}$
	10	5	0	$\mathbf{-}\mathbf{5}$	$\mathbf{-}\mathbf{10}$
$\left(20,20\right)$	11.14	19.92	23.16	24.21	24.17
$\left(20,0\right)$	21.31	23.43	23.98	23.69	21.08
$\left(20,-20\right)$	23.91	24.24	22.84	19.07	10.31
$\left(0,20\right)$	29.11	32.27	32.85	33.12	33.36
$\left(0,0\right)$	33.25	33.49	33.35	33.15	32.52
$\left(0,-20\right)$	33.55	33.12	32.59	31.56	29.19
$\left(-20,20\right)$	18.55	19.65	19.73	19.85	20.47
$\left(-20,0\right)$	20.68	19.68	19.52	19.70	19.96
$\left(-20,-20\right)$	20.45	19.77	19.47	18.48	18.74

and

Table 9. Miss distance (m) with initial flight path angle $\theta =-10°$.

Initial Relative Distance (km)	Heading Angle ${\mathbf{\Psi }}_{\mathit{v}}^{\mathbf{°}}$
	10	5	0	$\mathbf{-}\mathbf{5}$	$\mathbf{-}\mathbf{10}$
$\left(20,20\right)$	8.70	15.98	20.88	22.21	22.31
$\left(20,0\right)$	20.50	21.60	22.12	21.51	18.47
$\left(20,-20\right)$	22.51	22.22	20.22	15.94	6.58
$\left(0,20\right)$	26.72	30.83	31.92	32.34	32.54
$\left(0,0\right)$	32.15	32.53	32.59	32.34	31.37
$\left(0,-20\right)$	32.79	32.32	31.61	29.97	25.76
$\left(-20,20\right)$	19.89	19.79	19.68	20.15	20.72
$\left(-20,0\right)$	20.69	20.15	19.81	19.90	20.54
$\left(-20,-20\right)$	20.72	20.10	19.54	19.46	17.85

present the simulation results with an initial $\theta$ of 10°, 5°, 0°, $-5°$, and $-10°$, respectively.

Table 5, Table 6, Table 7, Table 8 and Table 9 illustrate that the slightest miss of these 225 scenarios was $6.58 \mathrm{m}$, satisfying the evasion problem requirement.

The simulation experiments indicate that the evading policy generated by MS-DDQN can avoid the pursuer’s attack in a large domain. For clarity, we also conducted a generic simulation experiment by uniformly sampling 10,000 points in a range of $20\times 20 \mathrm{km}$ with initial $\theta =5°$ and ${\Psi }_v=5°$ (Figure 13).

This figure illustrates a successful evasion of the evader where the missing range is $\left[26.12 \mathrm{m},33.34 \mathrm{m}\right]$.

5. Conclusions

This paper proposed an MS-DDQN algorithm to address the problem of high-speed aircraft evading. The evader was treated as the agent for DRL, and the optimal policy was generated through iteratively interacting with the simulation environment by the MS-DDQN framework. Moreover, the proposed two-stage learning method significantly improved the data quality and sped up the training process. As a model-free RL algorithm, MS-DDQN is an adaptive value-based DRL framework that does not require the vehicle’s mathematical model.

The effectiveness and robustness of this method were verified through various simulation experiments, illustrating a successful evasion in a large domain when utilizing the policy generated by MS-DDQN. Future research on autonomous decisions for evasion will combine policy-based DRL methods and multi-vehicle autonomous decision-making systems.