Globally Guided Deep V-Network-Based Motion Planning Algorithm for Fixed-Wing Unmanned Aerial Vehicles

Abstract

Fixed-wing UAVs have shown great potential in both military and civilian applications. However, achieving safe and collision-free flight in complex obstacle environments is still a challenging problem. This paper proposed a hierarchical two-layer fixed-wing UAV motion planning algorithm based on a global planner and a local reinforcement learning (RL) planner in the presence of static obstacles and other UAVs. Considering the kinematic constraints, a global planner is designed to provide reference guidance for ego-UAV with respect to static obstacles. On this basis, a local RL planner is designed to accomplish kino-dynamic feasible and collision-free motion planning that incorporates dynamic obstacles within the sensing range. Finally, in the simulation training phase, a multi-stage, multi-scenario training strategy is adopted, and the simulation experimental results show that the performance of the proposed algorithm is significantly better than that of the baseline method.

1. Introduction

With the development and wide application of UAV technology, fixed-wing UAVs have gradually become important tools in many fields due to their high maneuverability and survivability [1,2,3]. Unlike quadcopters, fixed-wing UAVs have huge advantages in endurance, flight speed, payload capacity, etc. [4,5], resulting in great potential in both civil and military applications [6]. Motion planning, as one of the key problems in flight control, is crucial for realizing safe, efficient, and autonomous flight missions. Collision avoidance is a key point in motion planning [7,8]. However, the flight characteristics of fixed-wing UAVs are very complex due to kinematic constraints [9]. Therefore, the goal of this paper is to learn a collision-free, safe, and fast motion planning method for fixed-wing UAVs that satisfies the kinematic constraints.

Unlike the path planning problem, kinodynamic motion planning for fixed-wing UAVs requires consideration of kinematic equations, kinematic constraints (e.g., obstacle avoidance), and kinodynamic constraints (e.g., velocity, acceleration, and turning radius constraints, etc.) in order to successfully reach the target location [10]. Although the traditional motion planning algorithm becomes less computationally practical for extremely dense scenarios, by simplifying the model, solvers enable real-time motion planning in many situations of interest.

A key challenge is to consider the second-order kinematics model and corresponding constraints of fixed-wing UAVs while achieving safe and fast motion planning in multi-obstacle (dynamic and static obstacles) environments. This is straightforward in a static environment, using first-order kinematics equations or simplifying the dynamics module, but the mixture of these factors makes it difficult to solve. Although existing studies have achieved some results in UAV path planning, it is still a worthwhile research problem [11,12].

In this work, we propose a hybrid A* deep V-network (HADVN) algorithm to realize collision-free motion planning for fixed-wing UAVs. In the proposed algorithm, a hierarchical two-layer motion planning framework is designed. First, considering the kinematic constraints of fixed-wing UAVs, such as the turning radius and the speed range, a global planner is designed based on to a modified hybrid A* search, providing a reference guidance with respect to static obstacles. With the help of global guidance, a local RL planner is designed to accomplish kino-dynamic, feasible, and collision-free motion planning that incorporates dynamic obstacles within the sensing range. Specifically, to improve the network generalization, a deep V-network with an attention mechanism is designed in local RL planner and a multi-stage, multi-scenario training strategy is adopted in the training process. The contributions of this work are as follows:

• Unlike the literature [13,14,15,16,17,18,19], the method proposed in this paper can realize safe and fast path planning while satisfying the second-order kinematics model and constraints of fixed-wing UAVs;
• For the motion planning problem studied in this paper, a deep V-network based on the attention mechanism is adopted with a multi-stage, multi-scenario training strategy to improve training efficiency and network generalization. The effectiveness of the algorithm is verified by comparison simulation experiments.

2. Related Works

(1)

Traditional motion planning algorithms for fixed-wing UAVs: Existing work on the navigation problem in cluttered environments for fixed-wing UAVs often focuses on traditional algorithms. The A* algorithm is employed to generate planning path in [13,14]. The authors in [15,16] proposed an improved potential field method to help UAVs avoid obstacles in local path planning. However, the above papers only consider the UAV as a mass point and neglect its kinematic model. The authors in [17] modeled a fixed-wing UAV as a first-order equation and defined a safe flight corridor based on its maneuvering characteristics and Dubins curve to achieve obstacle avoidance. A multi-objective optimization model was established in [20] and proposed an improved particle swarm optimization to solve the three-dimensional path planning problem in complex environments. The work proposed in [21] combined the ant colony algorithm, the self-organizing mapping algorithm, and the optimal Dubins trajectory to achieve the task assignment and path planning for multiple UAVs. Considering the kinematic model of fixed-wing UAVs as second-order systems, Ref. [22] proposed an improved RRT* algorithm to realize 3D path planning.

(2)

Deep neural network-based learning methods in the field of robotics: In recent years, with the increase in computational power, deep neural network-based learning methods have shown great potential in dealing with high-dimensional and complex environmental states [23,24], especially in the field of robotics, such as robotic arms [25], wheeled and multi-legged robots [26,27,28], etc., which have been applied to some experimental real-world scenarios. For example, in [29], the authors proposed an imitation learning approach based on perceptual information to realize motion planning for UAVs, and Ref. [30] used deep learning to achieve effective 3D exploration of a vehicle in an indoor environment.

(3)

Deep reinforcement learning (DRL)-based motion planning for fixed-wing UAVs: Unlike the supervised learning methods mentioned above, the RL method is applicable to sequential decision control problems, based on which agents can learn to explore strategies that maximize reward by interacting with the environment. Under the assumption that the UAV flies at a fixed altitude, Li et al. [18] designed the action space to satisfy the first-order fixed-wing UAV kinematic constraints, and implemented a deep Q-network to generate a safe path for a fixed-wing UAV. Based on the same model, Wu et al. [19] proposed a multicritical delay depth deterministic policy gradient method. Moreover, Li et al. [31] considered the second-order kinematic constraints, and designed a deep deterministic policy gradient algorithm to enable UAVs’ complete path planning in a 3D environment with only static obstacles; Yan et al. [32] established a posture assessment model to evaluate the collision risk between followers for the case with only neighboring aircraft, and designed a DRL algorithm to implement a collision-free strategy for fixed-wing UAVs.

3. Preliminaries

3.1. Problem Formulation

Consider a scenario where a fixed-wing UAV must navigate from an initial position $p_0$ to a goal position $p_g$ on the plane ${\mathbb{R}}^3$, surrounded by a number of non-communicating UAVs and static obstacles. The control state of a fixed-wing UAV at time t can be represented by ${\varsigma }_t$ (defined in Section 3.2), $u_t$ is the control input (defined in Section 4.3.1). The next state ${\varsigma }_{t+1}$ can be obtained through $f\left({\varsigma }_t,u_t\right)$, where f is the kinematic model of the fixed-wing UAV (defined in Section 3.2). $O_t$ denotes the area occupied by the fixed-wing UAV, $O^s$ by static obstacles and $O_t^j$ by each surrounding UAV. We aim to drive the ego-UAV with the kinematic model f, to the goal position $p_n$ under the collision avoidance constraints and boundary constraints of control state and control input, as shown in (1a), (1c) and (1d).

We then illustrate this problem in a RL framework. At each time-step t, the ego-UAV first obtains its state $s_t$ (defined in Section 4.3.1), the set of the neighboring UAVs’ states $S_t={\cup }_{i\in \{1,\dots ,N\}}{s}_t^i$ within the local FOV and a local segment of global guidance ${\mathcal{G}}^{*}$, expressed as ${\mathcal{G}}_t^{*}$, then takes action $a_t$, leading to the immediate reward $R(s_t,a_t)$ (defined in Section 3.2) and next state $s_{t+1}=h(s_t,a_t)$, under the transition model h. We use the superscript $j\in 1,\dots ,N$ to denote the j-th nearby UAV within the local FOV and omit the superscript when referring to the UAV.

With the above definition, the objective is transmitted to learn a policy $\pi$ for the UAV that minimizes time to goal while avoiding conflicts with neighbors and static obstacles, defined as

$$\begin{array}{cc}\hfill & {\pi }^{*}=\underset{\pi }{argmax}\mathbb{E}\left[\sum _{t=0}^T{\gamma }^{t·v_{max}}R\left(s_t,\pi \left(s_t,S_t\right)\right)\right]\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill & s.t.{\varsigma }_{t+1}=f\left({\varsigma }_t,u_t\right)\hfill \end{array}$$

(1a)

$$\begin{array}{cc}\hfill & s_T=goal\hfill \end{array}$$

(1b)

$$\begin{array}{cc}\hfill & O_t\left({\varsigma }_t\right)\cap O_t^j=\varnothing ;O_t\left({\varsigma }_t\right)\cap O^s=\varnothing \hfill \\ \hfill & u_t\in U,s_t\in S,{\varsigma }_t\in \Theta \hfill \end{array}$$

(1c)

$$\begin{array}{cc}\hfill & \forall t\in \left[0,T\right],\forall j\in \left\{1,\dots ,n\right\}\hfill \end{array}$$

(1d)

where (1a) is the transition dynamic constraints considering the kinematic model of the fixed-wing UAV; (1b) is the terminal constraints. Note that the $goal$ in (1b) is not exactly the goal position $p_n$, since $s_T$ is defined as the terminal state of $s_t$, which will be given in Section 4.3.1. (1c) is the collision avoidance constraints and S, U, and $\Theta$ are the set of admissible states, inputs, and the set of admissible control states, respectively. It is assumed that the fixed-wing UAV can obtain its current position and velocity (e.g., with on-board sensing) and the information of all the static obstacles, so it can calculate the global guidance at the start of each run. However, only the relative position and velocity of other UAVs can be observed when they are within their local FOV. Moreover, we assumed that other UAVs follow the policy that is generated by the optimal reciprocal collision avoidance (ORCA) algorithm [33].

3.2. Dynamics of Fixed-Wing UAV

At time-step t, the control state of a fixed-wing UAV can be represented by the tuple ${\varsigma }_t=(x_t,y_t,z_t,v_t,{\theta }_t,v_{zt})$, where $(x_t,y_t,z_t)$ denotes the position in 3D space; $v_t$ and $v_{zt}$ are the horizontal and vertical velocities relative to a inertial frame; ${\theta }_t$ denotes the heading angle.

To simplify the planning process, we decouple the horizontal and vertical motion directions of the fixed-wing UAV; then, the kinematic model of the fixed-wing UAV can be expressed as (see (1a))

$$\frac{\mathrm{d}}{dt}\left\{\begin{array}{c}{x}_t\hfill \\ y_t\hfill \\ z_t\hfill \\ v_t\hfill \\ {\theta }_t\hfill \end{array}\right\}=\left\{\begin{array}{c}{v}_{t}cos{\theta }_t\hfill \\ v_{t}sin{\theta }_t\hfill \\ v_{zt}\hfill \\ f_v\left(v_t,u_t\right)\hfill \\ f_{\theta }\left({\theta }_t,{\omega }_t\right)\hfill \end{array}\right\}$$

(2)

$$\begin{array}{c}{f}_v\left(v_t,u_t\right)=clip\left(v_t+u_t·dt,v_{min},v_{max}\right)\hfill \\ f_{\theta }\left({\theta }_t,{\omega }_t\right)={\theta }_t+{\omega }_t·dt\hfill \\ u_t\in [u_{min},u_{max}],{\omega }_t\in [{\omega }_{min},{\omega }_{max}],v_{zt}\in \left[v_{zmin},v_{zmax}\right]\hfill \end{array}$$

(3)

where $u_t$ and ${\omega }_t$ are the control commands.

For fixed-wing UAVs, a certain relative velocity with respect to air is necessary to generate enough lift to balance gravity. Moreover, the horizontal minimum turning radius is another limitation in fixed-wing UAV flight, resulting in constraints of heading angular velocity. To that end, three vehicle kinematic constraints are considered in this planning stage: horizontal and vertical velocity, horizontal acceleration, and heading angular velocity (which are Equation (3)).

Similarly, For each agent $j\in [1,n]$, $p_t^j={\left[x_t^j,y_t^j,z_t^j\right]}^T\in {\mathbb{R}}^3$ denotes its position in 3D space, $v_t^j\in {\mathbb{R}}^2$ and $v_{zt}^j\in \mathbb{R}$ its horizontal and vertical velocities at step t relative to a inertial frame, and $r_j$ the minimum radius of the ball that can completely encompass the j-th UAV.

4. Approach

In this section, we first describe the overall system structure, and then present details of our approach.

4.1. Systems Description

Our goal is to learn a kino-dynamic feasible and collision-free navigation policy via reinforcement learning, while relying on a local segment of global path and information within the sensing range as input, as shown in Figure 1. First, in the environment awareness layer, UAV obtains its self state and other UAVs’ information through sensors. Secondly, in the global planning layer, the global planner, which knows the information of all the static obstacles, provides a global guidance for ego-UAV. Then, in the local planning layer, a local RL planner is designed to accomplish kino-dynamic feasible and collision-free motion planning with the help of global guidance and other dynamic information within the sensing range. Finally, in the execution layer, ego-UAV executes control inputs and interacts with the environment.

It is assumed that the UAV knows the information of all the static obstacles and calculates the global guidance at the start when the scene changes. However, the trajectories of other UAVs are unknown and the ego-UAV can only obtain the current information of them when they are within its local field of view.

To that end, we adapt a hierarchical two-layer motion planning framework to tackle the navigation problem. The global planner is designed based on to a modified hybrid A* search, providing a reference guidance for ego-UAV with respect to static obstacles (Section 4.2). With the help of global guidance, a local RL planner is designed to accomplish kino-dynamic feasible and collision-free motion planning that incorporates dynamic obstacles within the sensing range (Section 4.3).

4.2. Global Guidance Planner

The main role of global guidance is to provide long-term global information and a safety reference path corresponding to static obstacles. With the help of global guidance, the proposed local RL planner is able to avoid potential collisions and unnecessary detours only by exploiting sensing information within a local area (e.g., a field of view).

Here, we employ a modified hybrid A* search to generate a safe and kino-dynamics feasible global path. Instead of straight-line segments, motion primitives with respect to the fixed-wing UAV dynamics described in Equations (2) and (3) are used as edges connecting two nodes. In contrast to standard hybrid A* search, the modifications are in three aspects.

4.2.1. Primitive Generation

In contrast to [13,14,15,16,17,18,19], a fixed-wing UAV dynamics respecting to second-order system Equation (2) and kinematic constraints Equation (3) is considered. To achieve this, the current heading angle and horizontal velocity are added to the node information, that is, the current node is defined as $P_k=(x_k,y_k,{\theta }_k,v_k,P_{k-1})$, where $P_{k-1}$ is the parent node. To expand the neighboring node, 11 equally spaced points within $[u_{min},u_{max}]$ and 7 equally spaced points within $[{\omega }_{min},{\omega }_{max}]$ are selected to obtain $n_i$ different combinations of action values. Then, the neighboring node $P_{k+1}^i=(x_{k+1}^i,y_{k+1}^i,{\theta }_{k+1}^i,v_{k+1}^i,P_k)$ of $P_k$ is calculated as

$$\begin{array}{c}{x}_{k+1}^i=x_k+v_{k+1}^{i}cos{\theta }_{k+1}^i\Delta t\hfill \\ y_{k+1}^i=y_k+v_{k+1}^{i}sin{\theta }_{k+1}^i\Delta t\hfill \\ v_{k+1}^i=clip(v_k+u_{k+1}^i\Delta t)\hfill \\ {\theta }_{k+1}^i={\theta }_k+{\omega }_{k+1}^i\Delta t\hfill \end{array}$$

(4)

where $i=1,\dots ,n_i$,$n_i$ is the maximun number of neighboring nodes.

4.2.2. Analytic Expansion

Inspired by reference [34], an analytic expansion scheme is induced to deal with the difficulty of having a primitive end exactly at the goal and to speed up the searching. To achieve this, Dubins curves [35,36], which are widely used in path planning for vehicles subject to turn radius constraints such as fixed-wing UAVs and underwater robots, are incorporated into the hybrid A* algorithm. Specifically, for the current node $P_k$, the trajectory from $P_k$ to $P_g$, which is kino-dynamics-feasible and minimal with respect to the distance is computed under Dubins curves assumption. If it satisfies collision-free condition, the searching is terminated in advance. This strategy can improve efficiency of generating global guidance path greatly, since it reduces the complexity of the search space and terminates the searching earlier.

4.2.3. Cost Function

Similar to the classical A* algorithm, the notation g is used to represent the actual cost from the initial node $P_0$ to the current node $P_k$. As we aim to find a path that has the shortest distance and is smooth enough, g is calculated as

$$\begin{array}{c}g=g\left(P_k\right)+g_{k+1,u,\omega }^i,k=0,1,\dots \\ g_{k+1,u,\omega }^i=v_{k+1}^i·\Delta t+{\alpha }_u\left|u_{k+1}-u_k\right|+{\alpha }_{\omega }\left|{\omega }_{k+1}-{\omega }_k\right|\end{array}$$

(5)

In addition, the heuristic cost from $Pk$ to $Pg$, h is calculated based on the length of the Dubins curve $l_{dk}^g$ as designed in the previous section, i.e., $h_{k,u,\omega }^i=l_{dk}^g$.

In summary, the flow of the hybrid A* algorithm is shown in Algorithm 1, where $\mathcal{P}$ and $\mathcal{C}$ refer to the open and closed list.

4.3. Local RL Planner

In this part, we aim to develop a decision-making algorithm to provide an estimate of the cost to go in dynamic environments with moving UAVs, which can inform the action, leading to higher rewards.

4.3.1. RL Formulation

For an ego-UAV, the observation vector is composed of three parts: states of itself, the surrounding UAVs’ states, and a local segment of global guidance, defined as

$$o_t=\left[s_t,S_t,{\mathcal{G}}_t^{*}\right]$$

(6)

where $s_t$ is the ego-UAV state, $S_t$ is the aggregation of neighbouring UAVs’ states, N is the total number of surrounding UAVs within sensing range, ${\mathcal{G}}_t^{*}$ is the global guidance given by Section 3.2. Specifically, each UAV’s state is parametrized as

$$\begin{array}{c}{s}_t=[d_{gt},h_t,v_t,{\theta }_t,v_{zt},r]\left(\mathrm{ego}-\mathrm{UAV}\right)\hfill \\ s_t^j=[{\overline{p}}_{jt},{\overline{v}}_{jt},r_j,{\overline{d}}_{jt},r_j+r](\mathrm{Other}\mathrm{UAVs},\mathrm{labelled}\mathrm{by}j,j=1,\dots ,N)\hfill \end{array}$$

(7)

where $d_{gt}$ and $h_t$ denote the horizontal distance and vertical distance to the goal, respectively; ${\overline{p}}_{jt}$, ${\overline{v}}_{jt}$, and ${\overline{d}}_{jt}$ represent the relative position, velocity and distance of the jth UAV with respect to ego-UAV; r denote the minimum radius (size) of the ball that can encompass ego-UAV, respectively.

Here, we seek to learn the optimal policy for the ego-agent $\pi :\left(o_t\right)\to a_t$ mapping the ego-agent’s observation of the environment to a probability distribution of actions. According to (1), we consider a continuous action space $a_t\in {\mathbb{R}}^3$ and define an action vector as

$$a_t=\left[u_t,{\omega }_t,v_{zt}\right]$$

(8)

where $u_t\in [u_{min},u_{max}]$, ${\omega }_t\in [{\omega }_{min},{\omega }_{max}]$, $v_{zt}\in \left[v_{zmin},v_{zmax}\right]$.

For simplicity, it is assumed that the UAV is at maximum vertical speed when it is far from the goal position. In other words, $v_{zt}$ is updated as below:

$$v_{zt}=\left\{\begin{array}{c}{v}_{zmax},h_t>v_{zmax}·\Delta t\hfill \\ \frac{h_t}{\Delta t},v_{zmin}·\Delta t\le h_t\le v_{zmax}·\Delta t\hfill \\ v_{zmin},h_t<v_{zmin}·\Delta t\hfill \end{array}\right.$$

(9)

For this task, a compound reward function is designed to feed back signals considering multiple subobjectives and generate dense rewards to promote convergence. More specifically, at each step, the reward function is given by

$$R_t=R_{1t}+R_{2t}+R_{3t}$$

(10)

where $R_{1t}$, $R_{2t}$, and $R_{3t}$ are designed for the purpose of collision avoidance, goal arrival, and global information awareness, respectively.

Firstly, the collision avoidance reward $R_{1t}$ is obtained by

$$R_{1t}=\left\{\begin{array}{c}{r}_{collision},{\overline{d}}_t\le 0\hfill \\ r_{collision}+{\alpha }_1·{\overline{d}}_t,0<{\overline{d}}_t\le d_{safe}\hfill \\ 0,{\overline{d}}_t>d_{safe}\hfill \end{array}\right.$$

(11)

where $r_{collision}$ is a large negative penalty for collision, ${\overline{d}}_t$ is the minimum sensing distance to the nearest obstacle in real time, and ${\alpha }_1$ is a small positive number. The second line makes the UAV alert to the nearest obstacle, assigned a penalty value if ${\overline{d}}_t$ is less than a predefined threshold $d_{safe}$.

Secondly, the goal arrival reward $R_{2t}$ is computed by

$$R_{2t}=\left\{\begin{array}{c}{r}_{success},d_{gt}\le d_{min}\hfill \\ {\alpha }_2\left(d_{gt-1}-d_{gt}\right),d_{gt}>d_{min}\hfill \\ r_{fail},t\ge t_{max}\hfill \end{array}\right.$$

(12)

where $r_{success}$ is a large positive reward for arrival, while ${\alpha }_2$ is a small positive number used to motivate the UAV to move towards the goal when navigating, and $d_{min}$ is the Euclidean distance from the UAV to its goal UAV when it is identified as reaching the goal, since the fixed-wing UAV may not be able to reach the goal accurately.

Thirdly, to encourage the UAV to take the global information into account, while simultaneously not following the global guidance strictly, $R_3\left(t\right)$ is computed by

$$R_{3t}=\left\{\begin{array}{c}0,d_{at}\le d_{amax}\hfill \\ -{\alpha }_3·\left(d_{amax}-d_{at}\right),d_{at}>d_{amax}\hfill \end{array}\right.$$

(13)

where $d_{at}$ denotes the distance between the current UAV and the global guidance path at the time of t; ${\alpha }_3$ is a small positive number; $d_{amax}$ is the predefined allowable deviation distance.

4.3.2. Policy Network Architecture

Figure 2 depicts the network structure of HADRL. Here is the description in detail. Considering that $o_t$ consists of three groups, i.e., ($s_t$, $S_t$, ${\mathcal{G}}_t^{*}$), we use two embedding functions to encode them separately. Through three fully connected (FC) layers, the embeddings of local segment of global guidance ${\mathcal{G}}_t^{*}$ can be easily represented as

$$e_l=F{C}_l\left({\mathcal{G}}_t^{*}\right)$$

(14)

As the numbers of surrounding UAVs in field of view are uncertain, we customize an attention embedding block to aggregate the states of the neighbors $S_t$. As shown in Figure 3, the aggregated embedding of $e_N$ is computed by

$$e_N=Attention\left(S_t\right)=\sum _{j\in N}{\xi }_N^{j}F{C}_{N1}\left(s_t^j\right)$$

(15)

The attention weight ${\xi }_N^j$ is calculated as

$$\begin{array}{c}{\xi }_N^j=\frac{exp\left(q_N^j\right)}{{\sum }_{j}exp\left(q_N^j\right)}\hfill \\ q_N^j=F{C}_{N2}(\frac{{\sum }_{j}F{C}_{N3}\left(s_j\right)}{N}\Vert s_j)\hfill \end{array}$$

(16)

As shown in Figure 2, we concatenate the fixed-length representation of the other agent’s states with the ego-agent’s state and the embedding of global guidance to create a joint state representation. This representation vector is fed to four fully connected layers (FCL). The output estimates the state-value function $V^{\pi }\left(S_t\right)={\mathbb{E}}_{S_{t+1:\infty }}\left[{\sum }_{l=0}^{\infty }{r}_{t+l}\right]$, with a linear activation function.

4.4. Hybrid A* Deep V Network (HADVN) Algorithm

An agent is unlikely to reach a goal position with an initial RL policy. Hence, during the pretraining phase, we use the ORCA as an expert and perform supervised training to train the policy and value function parameters for $n_{ORCA}$ steps. By setting the current state and using the ORCA, we obtain a sequence of state, action, next state, and reward in a buffer $\mathcal{B}\leftarrow \{o_k,a_k,o_{k+1},r_k\}$. Then, we compute advantage estimates and perform a supervised training step:

$${\theta }_{k+1}^V=arg\underset{{\theta }^V}{min}{\mathbb{E}}_{(a_k,o_k,r_k)\sim {\mathcal{D}}_{ORCA}}\left[‖V_{\theta }\left(o_k\right)-V_k^{trag}‖\right]$$

(17)

where ${\theta }^V$ is the value function and policy parameters.

In this work, we train the policy using the deep V-learning algorithm proposed by Chen C et al. [37]. The reasons for choosing a deep V-network instead of a Q-network in this paper are as follows: first, compared with deep Q-networks, the structure of V-networks are simpler, since their output is no longer different combinations of the action space, but can be simplified to a single node; second, in the supervised learning, the optimal policy can be computed faster, and the data-processing operations can be simplified; third, since the upper and lower bounds of the action quantities (acceleration and angular velocity) are different in this paper, the optimal policy can be computed quickly. We propose to jointly train the guidance policy $V^{\pi }$ with the ORCA algorithm. Algorithm 1 describes the proposed training strategy and has two main phases: supervised and RL training. First, we randomly initialize the value function parameters ${\theta }^V$. Then, a multi-scene multi-stage training framework is used for the training of the network. More details about the different training scenarios considered are given in Section 5.

Algorithm 1 Hybrid A* deep V-network (HADVN) algorithm

Input: target value network, number of supervised and RL training episodes $\{n_{ORCA},n_{episodes}\}$, size of mini-batch $n_{mini-batch}$ Output: Optimal value network $V^{\pi }$  1: Initialize the optimal value network V, the target value network $\widehat{V}$  2: Initialize the open list $\mathcal{P}$ and close list $\mathcal{C}$, add $P_0$ to $\mathcal{P}$  3: while $\mathcal{P}$ not empty do  4:     From $\mathcal{P}$ take the node with the smallest total cost f and record it as the current node $P_k$, add $P_k$ to $\mathcal{C}$;  5:     if $‖\left(x_k,y_k\right)-\left(x_g,y_g\right)‖\le d_{min}$ then  6:         return $\left\{P_0,\cdots ,P_k\right\}$;  7:     else  8:         generate Dubins curves $l_{dk}^g$ from $P_k$ to $P_g$;  9:     end if 10:     if $l_{dk}^g$ satisfies the no-collision condition, then 11:         generate path points $\left\{P_{k+1},\dots P_g\right\}$ on $l_{dk}^g$, return $\left\{P_0,\cdots ,P_g\right\}$; 12:     end if 13:     Based on discretized action combinations $\left(u_{k+1}^i,{\omega }_{k+1}^i\right)$ and Equation (4), generate the set of neighboring nodes $\left\{P_{k+1}^i\right\}$ of $P_k$, where $i=1,\cdots ,n_i$; 14:     for $P_{k+1}^i$in $\left\{P_{k+1}^i\right\}$ do 15:         if $P_{k+1}^i$ is beyond the map boundary, close to an obstacle or already in $\mathcal{C}$ in, then 16:            continue; 17:         end if 18:         Using Equation (5), calculate $g=g\left(P_k\right)+g_{k+1,u,\omega }^i$; 19:         if $P_{k+1}^i$not in $\mathcal{O}$, then 20:            add $P_{k+1}^i$ to $\mathcal{O}$; 21:         else if $g\ge g\left(P_{k+1}^i\right)$, then 22:            continue; 23:         end if 24:     end for 25:     $g\left(P_{k+1}^i\right)\leftarrow g$, change the parent node of $P_{k+1}^i$ to $P_k$; 26:     Calculate the total cost f of the node $P_{k+1}^i$; 27: end while 28: Generate ${\mathcal{G}}^{*}$ from $\left\{P_0,\cdots ,P_k\right\}$; 29: Back to step 2 if training scenario is changed; 30: while  $episode<n_{episodes}$ do 31:     Initialize $\mathcal{B}$ and states; 32:     for $k=0,\dots ,n_{mini-batch}$ do 33:         if $episode\le n_{ORCA}$ then 34:            using ORCA to get $a_k$; 35:         else 36:            $a_k=arg{max}_{a_k\in A}R(o_k,a_k)+{\gamma }^{\Delta t·v_{max}}V\left(o_{k+1}\right)$; 37:         end if 38:         $\{o_k,a_k,o_{k+1},r_k,done\}=step(o_k,a_k)$; 39:         Store $\mathcal{B}\leftarrow \{o_k,a_k,o_{k+1},r_k\}$; 40:         if $done$ then 41:            $episode+=1$, updating the target network $\widehat{V}\leftarrow V$; 42:         end if 43:     end for 44:     if $episode<n_{episodes}$ then 45:         Supervised training 46:     else 47:         Randomly take a $\{o_k,a_k,o_{k+1},r_k\}$ from $\mathcal{B}$; 48:         Calculate of the target value $y_k=r_k+{\gamma }^{\Delta t·v_{max}}\widehat{V}\left(o_{k+1}\right)$; 49:         Update the optimal value network V by gradient descent; 50:     end if 51: end while

5. Results

This section quantifies the performance throughout the training procedure, and compares the proposed method against the baseline approach, socially attentive reinforcement learning (SARL) algorithm [37], which adopts LSTM to deal with an uncertain number of dynamic obstacles in the environment. Although SARL only considers the first-order kinematic model of the mobile robot, in the comparative experiments, we improve both its pretraining and subsequent incremental training sessions to make it applicable to the generation of motion trajectories of fixed-wing UAVs.

5.1. Experimental Setup

The proposed training algorithm builds upon deep V-learning. We used a laptop with a 12th Intel Core i9 and 32 GB of RAM for training. Hyperparameter values are summarized in

Table 1. Hyperparameters.

Parameters	Value
Discount factor	0.9
Learning rate	0.001
Number of training rounds	300
$n_{ORCA}$	5000
$n_{episodes}$	30,000
$n_{mini-batch}$	100
$r_{collision}$	−0.25
$r_{success}$	1
$r_{fail}$	−0.1
${\alpha }_1$	0.05
${\alpha }_2$	0.1
${\alpha }_3$	0.01
${\alpha }_u$	0.01
${\alpha }_{\omega }$	0.01
$d_{safe}$	5 m
$d_{min}$	1 m
$d_{amax}$	10 m
$\Delta t$	0.25 s
$t_{max}$	25 s

.

5.2. Training Procedure

To train and evaluate our method, we selected a multi-scene, multi-stage training framework used for the training of the network, which can effectively increase the convergence speed of the network and further improve the generalization ability and effectiveness of the strategy, compared to training only in a single complex environment [38,39,40]. Here, we consider three different scenarios. In the first scenario, there are no static obstacles, and the number of static obstacles is increased in the second and third scenarios. The obstacles are defined as cylinders of equal height and radius with randomly generated locations. To increase the difficulty of obstacle and collision avoidance, the UAV to be planned is randomly distributed with other adjacent UAVs on a horizontal circle, and the goal position of the UAV to be planned is randomly placed on the other side of the circle.

Prior to reinforcement learning training, the network was first pretrained using the ORCA algorithm. In the pretraining phase, all UAV strategies were given by ORCA, 3000 rounds of experience were collected in the experience pool, and 50 rounds of training were performed on the optimal strategy network with a learning rate of 0.001.

Training is completed in a total of three phases, corresponding to three different scenarios, with 9000 training rounds for each scenario. Every 300 rounds, the experience pool is emptied and the success rate of the algorithm in the current environment is tested. The experiments show that staged training can significantly improve the speed of convergence, as shown in Figure 4.

5.3. Qualitative Analysis

This section analyzes the trained method. As shown in Figure 4 and Figure 5, the average reward is about 0.6 and the success rate is close to 0.98 at 2100 training rounds, indicating that the algorithm has learned the ability to avoid obstacles in the absence of static obstacles. When the environment is changed in 9000 rounds, the algorithm’s success rate and average reward show a significant drop, indicating that the ability to avoid static obstacles has not been learned. After training to the 13,800th round, the average reward for the second scenario is about 0.4 and the success rate is around 0.9. In the third scenario, after training to round 21,000, the average reward for the second scenario basically is about 0.3, with a success rate of around 0.8.

As shown in Figure 5, the average reward becomes lower as the static obstacles in the environment increase. This is because UAVs have to take a longer detour to avoid obstacles and collisions.

Figure 6 gives one planning episode from top view: in yellow is depicted the trajectory of the robot; in other colors are the other UAVs; in black are the obstacles; and the time interval between each UAV is the same. Figure 7 plots the x, y, z responses of the ego-UAV. The solid line represents the curves of the ego-UAV, while the dashed line gives the goal position, from which we can see that the UAV successfully avoids all static obstacles and other UAVs while reaching the goal safely, as marked by the red dot. When using our approach, the robot initiates a collision avoidance maneuver early enough to lead to a smooth trajectory and faster arrival at the goal.

The paths planned in a random obstacle environment are tested at the end of training using the HADVN algorithm, as shown in Figure 8. It can be observed that the network has learned the experience of avoiding randomly generated static obstacles and other UAVs.

To simulate the complicated unknown disturbances, random Gaussian noises that affected the fixed-wing UAV dynamics are considered. We set random Gaussian disturbance as in [41], and a mean of 0 as well as a covariance of 0.5, for the x or y position, respectively. We run the trained HADRL in three different scenarios with the obstacle number as 10, 15, and 20, respectively, and record the results of 100 episodes, given in

Table 2. Comparison of success rate with random Gaussian noises.

Methods	HADRL (without Noises)	HADRL (with Noises in x Position)	HADRL (with Noises in y Position)
Scenario 1	0.86	0.72	0.71
Scenario 2	0.79	0.65	0.63
Scenario 3	0.76	0.60	0.61

. The success rate is used to evaluate the performance, which is the proportion that ego-UAV reaches its goal without a collision. We can conclude that our approach demonstrates a good performance in the presence of random Gaussian disturbances without extra training.

5.4. Performance Results

This section compares it with the baseline method, SARL. The simulation results are shown in Figure 9 and Figure 10, it can be seen that during the training process of the network, in the first stage, the difference between the average reward and success rate of the SARL algorithm and the HADRL algorithm is not very large, and the SARL algorithm already has a high success rate after pretraining the initial network, indicating that the SARL algorithm can plan a collision-free and safe trajectory in the absence of static obstacles. However, in the second and third phases, the average reward and success rate of the SARL algorithm decrease significantly, while our proposed HADRL algorithm is able to learn and adapt to more complex environments very quickly with the help of global guidance.

Moreover, to illustrate the advantages of our proposed HADRL algorithm, comparative simulations are performed, with the ORCA and SARL algorithms. The scenarios—which are the same as in Section 5.3—and the results of 100 episodes are recorded. Two metrics—success rate and execution time—are used to evaluate the performance, where execution time is the time algorithm used to reach the goal. The results are shown in

Table 3. Comparison of success rate with other planning strategies.

Methods	ORCA	SARL	HADRL
Scenario 1	0.63	0.78	0.86
Scenario 2	0.55	0.62	0.79
Scenario 3	0.43	0.57	0.76

and

Table 4. Comparison of execution time with other planning strategies.

Methods	ORCA	SARL	HADRL
Scenario 1	0.32	0.12	0.13
Scenario 2	0.35	0.13	0.14
Scenario 3	0.36	0.13	0.16

. It is observed that HADRL has a higher success rate in three scenarios. As for the execution time, HADRL and SARL show the same performance, which is better than ORCA. Hence, we can conclude that our algorithm performs better in terms of computational efficiency and scalability.

6. Conclusions

For complex environments with static obstacles and other flying UAVs, this paper fully considers the second-order kinematic model and constraints, and proposes a HADVN motion planning algorithm based on global path guidance and a local RL planner so that fixed-wing UAVs can reach the target point safely within a preset time. Unlike the widely used deep Q-networks, our approach employs a deep V-network to deal with the continuous motion space. In addition, considering the variable number of neighboring UAVs in the sensing range of the UAV, an attention mechanism is added to the front end of the network to improve the network generalization. Finally, a multi-stage, multi-scene training strategy is used in training process and the effectiveness of the algorithm is verified by simulation experiments.