A Dual Aircraft Maneuver Formation Controller for MAV/UAV Based on the Hybrid Intelligent Agent

Abstract

This paper proposes a hybrid intelligent agent controller (HIAC) for manned aerial vehicles (MAV)/unmanned aerial vehicles (UAV) formation under the leader–follower control strategy. Based on the high-fidelity three-degrees-of-freedom (DOF) dynamic model of UAV, this method decoupled multiple-input-multiple-output (MIMO) systems into multiple single-input-single-output (SISO) systems. Then, it innovatively combined the deep deterministic policy gradient (DDPG) and the double deep Q network (DDQN) to construct a hybrid reinforcement learning-agent model, which was used to generate onboard desired state commands. Finally, we adopted the dynamic inversion control law and the first-order lag filter to improve the actual flight-control process. Under the working conditions of a continuous S-shaped large overload maneuver for the MAV, the simulations verified that the UAV can achieve accurate tracking for the complex trajectory of the MAV. Compared with the traditional linear quadratic regulator (LQR) and DDPG, the HIAC has better control efficiency and precision.

1. Introduction

Aiming at increasingly fast-paced and high-intensity air combat, the use of MAVs as combat operations leaders with a certain number of UAVs as wingers to form a hybrid formation of UAV/MAV has become the development trend for future air confrontations. Among them, the two-aircraft formation consisting of an MAV and a UAV is one of the most typical combat styles. In MAV/UAV formations, the unmanned system must be able to share information and carry out cooperative operations with the manned systems across systematic boundaries [1]. The Fast Lightweight Autonomy (FLA) Program by the Defense Advanced Research Projects Agency (DARPA) has developed an advanced algorithm that enables an MAV or a UAV to operate autonomously without a human operator, the Global Positioning System (GPS), or any data resources. DARPA’s Lifelong Learning Machines (L2M) Project also aims to develop new machine learning methods that enable unmanned systems to continuously adapt to new environments and remember what they have learned [2]. Meanwhile, the U.S. Air Force’s Loyal Wingman Program aims to enhance the autonomy of UAVs and improve their combat capabilities in complex war environments [3]. Moreover, the recently proposed Skyborg program is working on the combination of manned and unmanned combat aerial vehicles. Therefore, improving the capability of autonomous flight control has become an important direction for the development of future UAV technology.

One of the research hotspots of UAV autonomous control capability is the formation flight-control problem [4]. In terms of the traditional design of the formation controller, Ref. [5] proposed a sliding mode controller for MAV/UAV formation flights based on a layered architecture. However, it makes extensive simplifications on the strong nonlinear dynamic model of MAV/UAVs, and was only validated by simulations for flat trajectories. Ref. [6] considered sensor noise and developed a leader–follower formation PID controller for multi-robots, which can achieve better performance in limiting position deviations. Furthermore, Ref. [7] proposed a parallel approach control law for fixed-wing UAV formations under the leader–follower strategy. Ref. [8] referred to an idea of multi-channel decoupling that split the MIMO system into multiple SISO systems and used sliding mode control to track the reference trajectory, which can be further applied to formation-control problems. Refs. [9,10] proposed a consensus-based multiple aircraft cooperative formation control method, but the consensus theory analysis was highly dependent on the linearized dynamic model, which limited its further application in a complex nonlinear dynamic system. Refs. [11,12] developed a formation controller where the commands were generated independently of the dynamic model, decreasing the control precision in extreme working conditions. Refs. [13,14] considered the confrontation situation and adopted pre-defined maneuver strategy collections, taking typical maneuvers as the basic units and building a collection of maneuver strategies with free combinations of various basic units. However, due to the model uncertainty and non-cooperative environment, this method hardly dealt with complex working conditions. Therefore, the intelligent agent method has become a novel research trend because of its weak model dependence and strong ability in terms of strategy exploration. Refs. [15,16] adopted deep neural networks to learn aircraft-maneuvering strategies and made progress in enhancing the autonomous maneuvering capability of UAVs. However, UAV formation control is a high-dimensional dynamical control problem with tightly coupled variables. When traditional neural networks learn such complex behaviors, they cause problems such as low training efficiency and difficulty in stable convergence [17]. Among the novel neural networks, the double deep Q network (DDQN) algorithm has shown good performance in control problems with discrete action sets by fitting the value functions of state actions through neural networks [18,19,20], but it cannot be applied to control problems with continuous variables. Based on the deterministic policy gradient (DPG) algorithm, DeepMind proposed the deep deterministic policy gradient (DDPG) algorithm which is proven to perform well on many kinds of continuous control problems [21,22,23]. However, in the field of aircraft control, the large variation in the angle of attack commands will increase the load on the attitude control loop [24]. Meanwhile, when it comes to complex tasks with multiple continuous control variables problems, DDPG has problems with unstable networks and low exploration efficiency [25,26,27,28]. For the above dilemma, some scholars have turned to hybrid reinforcement learning methods in recent years. By adding discrete “meta-actions” to continuous control problems, Ref. [29] partially solved the reinforcement learning traps and improved exploration efficiency. The experiments verified its superiority to the traditional continuous strategy algorithm in some cases. [30] proposed the parametrized deep Q-network for the hybrid action space without approximation or relaxation, which provides a reference for solving the hybrid control problem.

Based on the above analysis, it is obvious that the formation controller must be able to better adapt to complex flight conditions in future confrontation situations, e.g., continuous large overload maneuvers for the MAV, etc. Therefore, inspired by [29], we propose a hybrid reinforcement intelligent agent controller based on the decoupling of multi-channels, which can effectively solve the problem of formation-tracking under continuous maneuvering conditions. It should be emphasized that when designing controllers based on artificial intelligent methods, especially when the reinforcement learning controller is directly applied to the generation of underlying flight-control commands, the lack of flight dynamic constraints can easily bring about problems. Due to the lack of dynamic constraints, the attitude control system cannot quickly track the commands, leading to flight instability. Therefore, this paper introduced the dynamic inversion controller and the first-order lag filter to the hybrid reinforcement learning agent to enhance the smoothness and executability of control commands.

In summary, the main contributions of this paper are as follows:

(1) A hybrid intelligent agent was designed based on the novel concept of “meta-action” to further enhance formation control performance. The hybrid intelligent agent combined DDPG and DDQN according to the specific formation control targets;

(2) The framework of the HIAC was developed that combined the dynamic inversion controller and the first-order lag filter with the hybrid intelligent agent to effectively overcome the common drawbacks of reinforcement learning;

(3) The superiority of the HIAC method was validated with experiments of nominal conditions. Monte Carlo simulations with different initial conditions were then conducted to verify the adaptability of the HIAC.

The organization of this paper is as follows: Section 2 establishes the UAV dynamic model and formation-control targets. Section 3 designs the novel formation controller HIAC based on the DDPG/DDQN hybrid intelligent agent. The dynamic inversion controller and first-order lag filter are introduced to the framework of the HIAC as well. Section 4 conducts the experiments of nominal conditions and 100 Monte Carlo simulations with varying initial conditions. Finally, we summarize the research conclusion of this paper in Section 5.

2. Mathematical Modeling

2.1. UAV Dynamic Model

The main concern in dual aircraft formation flights is the real-time position, velocity, and attitude of the two aircraft, so it is necessary to establish a dynamic model of the UAV according to the forces on the mass as shown in Figure 1. To simplify the problem, the constraints flight envelope is ignored.

In the ground inertial coordinate system $o-xyz$, $V$ is the UAV flight velocity. $\gamma$ and $\psi$ are the flight path angle and flight azimuth angle, respectively. The flight adopts the Bank-To-Turn (BTT), which is considered to have no sideslip. $\alpha$ is the attack angle, and $\sigma$ is the bank angle. The engine thrust and drag of the aircraft are denoted by $T$ and $D$, respectively. $n$ is the normal overload of the UAV in the velocity coordinate system $o-x_V{y}_V{z}_V$. Ignoring the wind disturbance in the flight, the three-degrees-of-freedom of the dynamic model for the UAV is established as follows [31,32,33]:

$$H=\left\{\begin{array}{c}\dot{x}=V\cos \gamma \sin \psi \hfill \\ \dot{y}=V\cos \gamma \cos \psi \hfill \\ \dot{z}=V\sin \gamma \hfill \\ \dot{V}=\left(T-D\right)/m-g\sin \gamma \hfill \\ \dot{\gamma }=g\left(n\cos \sigma -\cos \gamma \right)/V\hfill \\ \dot{\psi }=-gn\sin \sigma /\left(V\cos \gamma \right)\hfill \end{array},\right.$$

(1)

where $m$ is the weight of the aircraft, which is considered constant in this paper, and $g$ is the local gravity.

The engine thrust $T$ can be denoted by

$$T=\eta T_{\max },$$

(2)

where $\eta$ is the throttle manipulator, and its range is defined as $\left[0, 1\right]$. $T_{\max }$ is the maximum thrust that the engine can achieve.

The air drag $D$ consists of the parasite drag and the induced drag, which can be expressed as follows [31]:

$$D=C_{D_P}\rho V^{2}S/2+2C_{D_I}{n}^2{m}^2{g}^2/\left(\rho V^{2}S\right),$$

(3)

where $S$ is the reference area of the UAV. $C_{D_P}$ is the parasite drag coefficient. $C_{D_I}$ is the induced drag coefficient. $\rho$ is the atmospheric density, which varies with the altitude of the aircraft in the stratosphere. It is calculated by [34]

$$\rho ={\rho }_0\cdot e^{-z/z_0},$$

(4)

where ${\rho }_0=1.225$ kg/m³ and $z_0=6700$ m.

2.2. Formation Control Targets

In this paper, the formation control target of the UAV was determined based on the leader–follower formation strategy. Taking a typical dual aircraft formation flight as an example, the formation configuration of the MAV/UAV was designed as shown in Figure 2. Since the reference trajectory of the MAV as the leader aircraft is known, the flight velocity, attitude, and position can be obtained from the sensors mounted within the MAV. The winger aircraft can receive real-time flight data from the MAV through the onboard data chain and complete the trajectory tracking and formation control autonomously. During the flight, it is required that the UAV and MAV keep a specific formation throughout the whole flight, as shown in Figure 2.

2.2.1. Flight Velocity Control Targets

The MAV and UAV keep the same formation flight velocity. The reference velocity of the MAV is $V_L$, and the UAV velocity is $V_W$, then the velocity deviation $\Delta V$ is

$$\Delta V=\left|V_L-V_W\right|.$$

(5)

The MAV and UAV keep the same flight path angle in formation flight. The MAV flight path angle is ${\gamma }_L$, and the UAV flight path angle is ${\gamma }_W$, then the flight path angle deviation $\Delta \gamma$ is

$$\Delta \gamma =\left|{\gamma }_L-{\gamma }_W\right|.$$

(6)

The MAV and UAV keep the same flight azimuth angle in formation flight. The flight azimuth angle of the MAV is ${\psi }_L$, and the flight azimuth angle of the UAV is ${\psi }_W$, then the deviation of the flight azimuth angle $\Delta \psi$ is

$$\Delta \psi =\left|{\psi }_L-{\psi }_W\right|.$$

(7)

The flight velocity and attitudes of the UAV should be consistent with the MAV within an allowable error

$$\Delta V\le V_{\Delta \max },\hspace{1em}\Delta \gamma \le {\gamma }_{\Delta \max },\hspace{1em}\Delta \psi \le {\psi }_{\Delta \max },$$

(8)

where $V_{\Delta \max }$, ${\gamma }_{\Delta \max }$, ${\psi }_{\Delta \max }$ represent the error thresholds of the velocity, flight path angle, and flight azimuth angle of the UAV, respectively.

2.2.2. Flight Distance Control Targets

The UAV is located around the MAV and maintains the specified formation distance. $\Delta D$ denote the distance between the MAV and the UAV in the ground inertial coordinate system. $\Delta D_x$, $\Delta D_y$ and $\Delta D_z$ denote the spatial distance of $\Delta D$ as follows:

$$\Delta D=\sqrt{\Delta D_x{}^2+\Delta D_y{}^2+\Delta D_z{}^2}.$$

(9)

Summarily, the UAV should keep a distance larger than the safe flight distance from the MAV, which is as follows:

$$D_{\Delta \min }\le \Delta D\le D_{\Delta \max },$$

(10)

where $D_{\Delta \min }$ and $D_{\Delta \max }$ represent the thresholds of the safe distance.

3. Design of the HIAC

The HIAC first adopted a DDPG/DDQN hybrid reinforcement learning method to train the agent model to generate the tracking commands. Then, we further designed a dynamic inversion controller and a first-order lag filter to construct an improved formation flight controller. Overall, the HIAC consists of three parts, i.e., desired state command solver, dynamic inversion controller, and first-order lag filter. The framework of the HIAC is shown in Figure 3.

In order to track the MAV, the HIAC adopted the current deviation between the states of the UAV and the states of the MAV, i.e., $\Delta V$, $\Delta \gamma$ and $\Delta \psi$ as inputs, and outputs the control commands of the thrust, normal overload, and bank angle, i.e., $\eta$, $n$ and $\sigma$. The main difference of the HIAC from other traditional controllers is that to further enhance the control accuracy, the HIAC adopted a hybrid intelligent agent as the desired command solver to generate the desired commands, $V_c$, ${\gamma }_c$ and ${\psi }_c$. Then, these commands were sent to the dynamic inversion controller to generate the control commands, ${\eta }_c$, $n_c$ and ${\sigma }_c$. Finally, the first-order lag filter further smoothed ${\eta }_c$, $n_c$ and ${\sigma }_c$ to improve the executability of these commands. The three parts will be introduced in detail as the order of the information flow.

3.1. Desired Command Solver

Learning from the idea of “meta-action”, we partially discretized the control variables in the continuous control problems and developed a continuous–discrete mixed action space according to the characteristics of these control variables. Based on this process, we constructed a hybrid intelligent agent based on DDPG and DDQN to control $V$, $\gamma$, $\psi$ and $D$ of the UAV.

3.1.1. Framework of Hybrid Intelligent Agent Based on DDPG/DDQN

Based on the traditional Q-Learning algorithm, DDQN uses the neural network to fit the value function. It adopts discrete action sets to define the strategy and evaluates the Q value of the generated strategy through the Critic network. Compared with the traditional DQN algorithm [18,19,20], DDQN decouples the action selection strategy of the Q value and the calculation of the Q value and solves the problem of overestimation of the Q value compared with the traditional methods.

DDPG adopts the Actor–Critic network based on DQN and uses continuous action sets to define the control strategy. The model consists of the Actor–Critic network, where the Critic evaluates the actions generated by the Actor, and the Actor feeds back the evaluation results to the Critic for policy optimization [23]. More proofs and conclusions of the DDQN and DDPG can be found in [18,23], respectively.

However, the DDQN and DDPG suffer from different drawbacks when applied in practical engineering. Although the DDQN is easier to converge when compared with DDPG, it can only deal with discrete and low-dimensional action spaces. However, most of the practical targets, especially physical control targets, have continuous and high-dimensional action spaces. Moreover, even though the continuous space can be transferred into the discrete space, DDQN will generate high high-dimensional action space in this process and finally cause quite low computational efficiency. Meanwhile, although DDPG can solve the problem of continuous and high-dimensional action spaces, it is more likely to diverge than DDQN. Therefore, learning from “meta-action”, we proposed a hybrid intelligent agent combining the DDQN and DDPG according to their complementary characteristics. Considering the value range and the control precision of $V$, $\gamma$ and $\psi$, we adopted the idea of multi-channel decoupling to perform partial discretization of the action space. For the velocity control agent $V_c$, the DDPG was used to generate the set of continuous state commands. Because the value range of $V_c$ is larger than ${\gamma }_c$ and ${\psi }_c$, discretizing the continuous action space with high precision will lead to dimension explosion. Meanwhile, for the angle control agents ${\gamma }_c$ and ${\psi }_c$, the DDQN is used to generate the set of discretized state commands. Combining the DDQN and DDPG can improve the capability of convergence when these two agents are trained together.

The framework of the desired commands solver was designed as shown in Figure 4. It includes three agents which process the variation of the state commands $V_c$, ${\gamma }_c$ and ${\psi }_c$, respectively. Based on the decoupling principles between different agents, each agent calculates the action $A_V$, $A_{\gamma }$ and $A_{\psi }$, and updates the desired state commands respectively. The outputs are executed by the flight-control system of the UAV and fed back the rewards of each agent. The total reward function $R_{\sum }$ is expressed by

$$R_{\sum }=R_{\sum }^{\left(D,V\right)}+R_{\sum }^{\left(\gamma \right)}+R_{\sum }^{\left(\psi \right)},$$

(11)

where $R_{\sum }^{\left(D,V\right)}$, $R_{\sum }^{\left(\gamma \right)}$ and $R_{\sum }^{\left(\psi \right)}$ are components of $R_{\sum }$ in each agent.

To construct an intelligent agent based on the DDPG/DDQN hybrid reinforcement learning network, it was necessary to transform the trajectory tracking problem into a Markov decision process, which mainly includes three parts, i.e., the state space, the action space, and the reward function.

3.1.2. State Space $S$

According to the targets of formation flight control, the state space $S$ is designed as follows:

$$S=[\Delta V,\Delta D,\Delta \dot{V},\Delta \gamma ,{\displaystyle \int \Delta \gamma \mathrm{d}t}, \Delta \dot{\gamma },\Delta \psi ,{\displaystyle \int \Delta \psi \mathrm{d}t} ,\Delta \dot{\psi }],$$

(12)

where $\Delta D$ equals

$$\Delta D=\Delta D_0+{\displaystyle \int \Delta V\mathrm{d}t},$$

(13)

where $\Delta D_0$ is the flight distance deviation between the MAV and the UAV at the initial epoch. The integral items $\Delta V$, $\Delta \gamma$ and $\Delta \psi$ are the cumulative deviation from the initial epoch till the current epoch. $\Delta \dot{V}$, $\Delta \dot{\gamma }$ and $\Delta \dot{\psi }$ is the deviation rate of the velocity, flight azimuth angle, and flight path angle.

3.1.3. Action Space $A$

The action space $A$ is defined as follows:

$$A=\left[A_V,A_{\gamma },A_{\psi }\right],$$

(14)

where the action $A_V$ denotes the correction value of the UAV velocity commands $\Delta V_c$, the action $A_{\gamma }$ denotes the correction value of the UAV flight path angle commands $\Delta {\gamma }_c$, and the action $A_{\psi }$ denotes the correction value of the UAV flight azimuth angle commands $\Delta {\psi }_c$, i.e.,

$$\left\{\begin{array}{l}\Delta V_c=A_V,\Delta {\gamma }_c=A_{\gamma },\Delta {\psi }_c=A_{\psi }\hfill \\ \left|\Delta V_c\right|\le {\lambda }_{V_c\max }{}_c,\left|\Delta {\gamma }_c\right|\le {\lambda }_{{\gamma }_c\max },\left|\Delta {\psi }_c\right|\le {\lambda }_{{\psi }_c\max },\hfill \end{array}\right.$$

(15)

where ${\lambda }_{V_c\max }$, ${\lambda }_{{\gamma }_c\max }$ and ${\lambda }_{{\psi }_c\max }$ are the maximum of corrections, respectively. $A_V$ is used to generate the set of continuous velocity commands. $A_{\gamma }$, $A_{\psi }$ is used to generate the set of discretized angle commands. Specifically, the discretization can be further expressed as follows:

$$\begin{array}{l}\left|{\Omega }_{{\gamma }_c}\right|=2⌈{\lambda }_{{\gamma }_c\max }/\partial {\gamma }_c⌉+1\\ \left|{\Omega }_{{\psi }_c}\right|=2⌈{\lambda }_{{\psi }_c\max }/\partial {\psi }_c⌉+1\end{array}$$

(16)

$$\begin{array}{l}{\Omega }_{{\gamma }_c}=\left\{\left.A_{\gamma }\right|0,\pm \partial {\gamma }_c,\pm 2\partial {\gamma }_c,\cdots ,\pm \left(\left|{\Omega }_{{\gamma }_c}\right|-1\right)\partial {\gamma }_c/2,\pm {\lambda }_{{\gamma }_c\max }\right\}\\ {\Omega }_{{\psi }_c}=\left\{\left.A_{\psi }\right|0,\pm \partial {\psi }_c,\pm 2\partial {\psi }_c,\cdots ,\pm \left(\left|{\Omega }_{{\psi }_c}\right|-1\right)\partial {\psi }_c/2,\pm {\lambda }_{{\psi }_c\max }\right\}.\end{array}$$

(17)

Then, the update of desired state commands is

$$\left\{\begin{array}{l}{V}_c\leftarrow V_c+\Delta V_c\\ {\gamma }_c\leftarrow {\gamma }_c+\Delta {\gamma }_c\\ {\psi }_c\leftarrow {\psi }_c+\Delta {\psi }_c.\end{array}\right.$$

(18)

3.1.4. Reward Function $R_{\sum }$

According to the formation control targets of UAVs, the reward function $R_{\sum }$ was designed as follows:

$$R_{\sum }=R_P+R_N+R_C$$

(19)

where $R_P$ is the reward sub-function, which gives a positive response when the flight state of the UAV meets the control targets. $R_N$ is the penalty sub-function, which gives a negative response when the flight states exceed the allowable error of the control target. $R_C$ is the command limiting function, which can limit the values of the control commands ${\eta }_c$, $n_c$, ${\sigma }_c$. More specifically, $R_C$ can smooth the variation of the control commands to finally reduce energy consumption.

$R_P$ is calculated by

$$R_P=10\times \left({\epsilon }_D^2+{\epsilon }_V^2+{\epsilon }_{\gamma }^2+{\epsilon }_{\psi }^2\right)$$

(20)

where ${\epsilon }_D$, ${\epsilon }_V$, ${\epsilon }_{\gamma }$, and ${\epsilon }_{\psi }$ are reward coefficients, which are defined as follows

$$\begin{array}{l}{\epsilon }_D=\left\{\begin{array}{l}1,D_{\Delta \min }\le \Delta D\le D_{\Delta }{}_{\max }\\ 0,\Delta D<D_{\Delta \min } \mathrm{or} \Delta D>D_{\Delta \max }\end{array}\right.,\\ {\epsilon }_V=\left\{\begin{array}{l}1-\Delta V/V_{\Delta \max },\Delta V\le V_{\Delta \max }\\ 0,\Delta V>V_{\Delta \max }\end{array}\right.,\\ {\epsilon }_{\gamma }=\left\{\begin{array}{l}1-\Delta \gamma /{\gamma }_{\Delta \max },\Delta \gamma <{\gamma }_{\Delta \max }\\ 0,\Delta \gamma \ge {\gamma }_{\Delta \max }\end{array}\right.,\\ {\epsilon }_{\psi }=\left\{\begin{array}{l}1-\Delta \psi /{\psi }_{\Delta \max },\Delta \psi <{\psi }_{\Delta \max }\\ 0,\Delta \psi \ge {\psi }_{\Delta \max }\end{array}\right..\end{array}$$

(21)

$R_N$ is calculated by

$$R_N=-100\times \left(e_D^2+e_V^2+e_{\gamma }^2+e_{\psi }^2\right)$$

(22)

where $e_D$, $e_V$, $e_{\gamma }$, and $e_{\psi }$ are penalty coefficients, which are defined as follows:

$$\begin{array}{l}{e}_D=\left\{\begin{array}{l}1,\Delta D<D_{\Delta \min } \mathrm{or} \Delta D>D_{\Delta \max }\\ 0,D_{\Delta \min }\le \Delta D\le D_{\Delta \max }\end{array}\right.,\\ e_V=\left\{\begin{array}{l}1,\Delta V>2V_{\Delta \max }\\ \Delta V/V_{\Delta \max }-1,V_{\Delta \max }\le \Delta V\le 2V_{\Delta \max }\\ 0,\Delta V<V_{\Delta \max }\end{array}\right.,\\ e_{\gamma }=\left\{\begin{array}{l}1,\Delta \gamma >2{\gamma }_{\Delta \max }\\ \Delta \gamma /{\gamma }_{\Delta \max }-1,{\gamma }_{\Delta \max }\le \Delta \gamma \le 2{\gamma }_{\Delta \max }\\ 0,\Delta \gamma <{\gamma }_{\Delta \max }\end{array}\right.,\\ e_{\psi }=\left\{\begin{array}{l}1,\Delta \psi >2{\psi }_{\Delta \max }\\ \Delta \psi /{\psi }_{\Delta \max }-1,{\psi }_{\Delta \max }\le \Delta \psi \le 2{\psi }_{\Delta \max }\\ 0,\Delta \psi <{\psi }_{\Delta \max }\end{array}\right..\end{array}$$

(23)

$R_C$ is calculated by

$$R_C=-0.2\left(\left|{\eta }_c\right|/{\eta }_{\max }+\left|n_c\right|/n_{\max }+\left|{\sigma }_c\right|/{\sigma }_{\max }\right).$$

(24)

3.2. Dynamic Inversion Controller

To realize tracking of the commands of a given flight trajectory, the dynamic inversion control law was designed as follows [35]

$$\begin{array}{l}{\dot{V}}_c={\varpi }_V\left(V_c-V\right)\\ {\dot{\gamma }}_c={\varpi }_{\gamma }\left({\gamma }_c-\gamma \right)\\ {\dot{\psi }}_c={\varpi }_{\psi }\left({\psi }_c-\psi \right)\end{array}$$

(25)

where ${\varpi }_V$, ${\varpi }_{\gamma }$, and ${\varpi }_{\psi }$ denote the bandwidth of the controller, respectively. $V_c$, ${\gamma }_c$, ${\psi }_c$ denote the desired state commands of the flight velocity, the flight path angle, and the flight azimuth angle, respectively.

Since the UAV commands follow the dynamic constraints by Equation (1), considering Equations (1), (2), and (25) yields

$$\begin{array}{l}{T}_c={\eta }_c{T}_{\max }=\left[D+m{\varpi }_V\left(V_c-V\right)+mg\sin \gamma \right],\\ N_{\gamma }={\varpi }_{\gamma }V\left({\gamma }_c-\gamma \right)/g+\cos \gamma ,\\ N_{\psi }={\varpi }_{\psi }V\left({\psi }_c-\psi \right)\cos \gamma /g,\end{array}$$

(26)

where $N_{\gamma }$ and $N_{\psi }$ denote the normal overload and lateral overload, respectively. The throttle ${\delta }_c$, normal overload $n_c$ and bank angle ${\sigma }_c$ are selected as the control commands. Then, the UAV control command was designed as follows:

$$F=\left\{\begin{array}{l}{\eta }_c=\left[D+m{\varpi }_V\left(V_c-V\right)+mg\sin \gamma \right]/T_{\max }\\ n_c=\sqrt{N_{\gamma }{}^2+N_{\psi }{}^2}\\ {\sigma }_c=\arctan \left(N_{\psi }/N_{\gamma }\right)\end{array}\right..$$

(27)

Moreover, the control command must satisfy the constraints:

$${\eta }_{\min }\le {\eta }_c\le {\eta }_{\max },0\le n_c\le n_{\max },\left|{\sigma }_c\right|\le {\sigma }_{\max }$$

(28)

where ${\eta }_{\min }$ and ${\eta }_{\max }$ is the minimum and maximum values of the throttle commands, respectively. $n_{\max }$ is the maximum value of the normal overload, and ${\sigma }_{\max }$ is the maximum value of the bank angle.

3.3. First-Order Lag Filter

Considering the fact that the UAV cannot instantly complete the change of the engine thrust, normal overload, and bank angle, a first-order lag filter model was constructed to simulate the delayed variation processes of these three variables:

$$G=\left\{\begin{array}{l}\dot{\eta }=\left({\eta }_c-\eta \right)/{\tau }_{\delta }\\ \dot{n}=\left(n_c-n\right)/{\tau }_n\\ \dot{\sigma }=\left({\sigma }_c-\sigma \right)/{\tau }_{\sigma }\end{array}\right.,$$

(29)

where ${\eta }_c$, $n_c$, ${\sigma }_c$ represent the control commands of the throttle, normal overload, and bank angle, respectively. ${\tau }_{\delta }$, ${\tau }_n$, and ${\tau }_{\sigma }$ represent the response time of the UAV control system accordingly.

Summarily, considering Equations (1), (27), and (29), the UAV flight process can be presented by the control equations as follow:

$$\left\{\begin{array}{l}F{\left(V_c,{\gamma }_c,{\psi }_c\right)}^{\mathrm{T}}={\left[{\eta }_c,n_c,{\sigma }_c\right]}^{\mathrm{T}}\\ G{\left({\eta }_c,n_c,{\sigma }_c\right)}^{\mathrm{T}}={\left[\dot{\eta },\dot{n},\dot{\sigma }\right]}^{\mathrm{T}}\\ H{\left(\eta ,n,\sigma \right)}^{\mathrm{T}}={\left[V,\gamma ,\psi \right]}^{\mathrm{T}}\end{array}\right..$$

(30)

Equation (30) reveals the calculation process from the desired control commands to the actual control commands. It is clear that the premise to realize the formation flight is to acquire the desired control commands of the UAV $V_c$, ${\gamma }_c$, ${\psi }_c$ under the specific formation strategy. Then, the ultimate flight trajectory can be obtained by the Runge–Kutta method.

4. Simulation Validation

4.1. Simulation Design

Based on the 3-DOF dynamic model in this paper, the MAV was designed to make a complex maneuver and provide the reference trajectory and control commands, accordingly. Under the leader–follower formation strategy, the UAV adopts the HIAC, DDPG, and LQR to track the MAV and keep the dual aircraft formation, respectively. LQR is a commonly used guidance method for tracking multi-state trajectories in aerospace engineering and it has been validated by extensive flight tests [36,37]. Therefore, we compared the proposed method with LQR and DDPG to verify its superiority in the following Section 4.2 and Section 4.3. The design of DDPG is described in Section 3.1.

First, the experiment of nominal conditions was conducted to analyze the superiority of the proposed method in detail. Meanwhile, the initial values greatly affect the performance of the reinforcement learning models. Therefore, the generalization ability of the model was required to be fully verified. Then, 100 Monte Carlo experiments were conducted to verify the adaptability of this method to different initial conditions.

The simulations were conducted by Matlab2021a and the 3-DOF dynamic model was built by Simulink. The total simulation time was $T$, the simulation interval was $\Delta T$, and the specific experimental parameters are shown in

Table 1. The experimental parameter settings.

Parameters	Settings	Parameters	Settings
$T$ (s)	50	${\eta }_{\min }$	0
$\Delta T$ (s)	0.1	${\eta }_{\max }$	1
$T_{\max }$ (lb)	25,600	$n_{\max }$	6
$m$ (kg)	14,470	${\sigma }_{\max }$ (rad)	$\pi /2$
$g$ (m/s2)	9.81	$D_{\Delta \max }$ (m)	600
$S$ (ft2)	400	$D_{\Delta \min }$ (m)	100
$C_{D_P}$	0.02	$V_{\Delta \max }$ (m/s)	50
$C_{D_I}$	0.1	${\psi }_{\Delta \max }$ (rad)	0.2
${\tau }_{\delta }$ (s)	0.6	${\gamma }_{\Delta \max }$ (rad)	0.2
${\tau }_n$ (s)	0.5	${\lambda }_{V_c\max }$ (m/s)	50
${\tau }_{\sigma }$ (s)	0.5	${\lambda }_{{\gamma }_c\max }$ (rad)	$\pi /2$
${\varpi }_V$ (s)	0.3	${\lambda }_{{\psi }_c\max }$ (rad)	$\pi /2$
${\varpi }_{\gamma }$ (s)	0.2	$\partial {\gamma }_c$ (rad)	$\pi /180$
${\varpi }_{\psi }$ (s)	0.2	$\partial {\psi }_c$ (rad)	$\pi /180$

.

The training methods of DDPG and DDQN refer to [18,23], respectively. Learning rate, max episode, discount factor, and experience buffer length were set as the same for both DDPG and DDQN. In addition, the batch size of DDPG was set to 256, and the batch size of DDQN was set to 64. The specific parameters are shown in

Table 2. The training parameters of DDPG/DDQN.

Parameters	Settings
Learning Rate	0.0001
Max Episode	25,000
Batch Size (DDPG)	256
Batch Size (DDQN)	64
Discount Factor	0.99
Experience Buffer Length	1 × 106

.

4.2. Basic Principles of LQR

The implementation of LQR mainly includes three parts: linearization of the motion model, design of the tracking controller for the reference trajectory, and solution of the feedback gain matrix.

By linearizing the dynamic model of the UAV in Equation (1) with small deviations, the linear system can be obtained as follows:

$$\dot{X}=AX+Bu.$$

(31)

Equation (31) can be expressed by

$$\left[\begin{array}{l}\delta \dot{x}\\ \delta \dot{y}\\ \delta \dot{z}\\ \delta \dot{V}\\ \delta \dot{\gamma }\\ \delta \dot{\psi }\end{array}\right]=\left[\begin{array}{cccccc}{A}_{11}& A_{12}& A_{13}& A_{14}& A_{15}& A_{16}\\ A_{21}& A_{22}& A_{23}& A_{24}& A_{25}& A_{26}\\ A_{31}& A_{32}& A_{33}& A_{34}& A_{35}& A_{36}\\ A_{41}& A_{42}& A_{43}& A_{44}& A_{45}& A_{46}\\ A_{51}& A_{52}& A_{53}& A_{54}& A_{55}& A_{56}\\ A_{61}& A_{62}& A_{63}& A_{64}& A_{65}& A_{66}\end{array}\right]\left[\begin{array}{l}\delta x\\ \delta y\\ \delta z\\ \delta V\\ \delta \gamma \\ \delta \psi \end{array}\right]+\left[\begin{array}{ccc}{B}_{11}& B_{12}& B_{13}\\ B_{21}& B_{22}& B_{23}\\ B_{31}& B_{32}& B_{33}\\ B_{41}& B_{42}& B_{43}\\ B_{51}& B_{52}& B_{53}\\ B_{61}& B_{62}& B_{63}\end{array}\right]\left[\begin{array}{l}\delta \eta \\ \delta n\\ \delta \sigma \end{array}\right].$$

(32)

Set the given MAV trajectory as the reference, the state space is defined as follows:

$$\begin{array}{l}\delta x=x_W-x_L,\delta y=y_W-y_L,\delta z=z_W-z_L,\\ \delta V=V_W-V_L,\delta \gamma ={\gamma }_W-{\gamma }_L,\delta \psi ={\psi }_W-{\psi }_L.\end{array}$$

(33)

The control commands are defined as follows:

$$\delta \eta =\eta -{\eta }_L,\delta n=n-n_L,\delta \sigma =\sigma -{\sigma }_L,$$

(34)

where $A$ and $B$ are the partial derivative coefficient matrix calculated according to the motion differential equation and the feature points of the reference trajectory. The calculation results are as follows:

$$\begin{array}{l}{A}_{11}=A_{12}=A_{13}=0,\\ A_{14}=\cos \gamma \sin \psi ,A_{15}=-V\sin \gamma \sin \psi ,A_{16}=V\cos \gamma \cos \psi ,\\ A_{21}=A_{22}=A_{23}=0,\\ A_{24}=\cos \gamma \cos \psi ,A_{25}=-V\sin \gamma \cos \psi ,A_{26}=-V\cos \gamma \sin \psi ,\\ A_{31}=A_{32}=A_{33}=A_{36}=0,\\ A_{34}=\sin \gamma ,A_{35}=V\cos \gamma ,\\ A_{41}=A_{42}=A_{46}=0,A_{43}=D_z/m,\\ A_{44}=D_V/m,A_{45}=-g\cos \gamma ,\\ A_{51}=A_{52}=A_{53}=A_{56}=0,\\ A_{54}=-g\left(n\cos \sigma -\cos \gamma \right)/V^2,A_{55}=g\sin \gamma /V,\\ A_{61}=A_{62}=A_{63}=A_{66}=0,\\ A_{64}=g\sin \sigma n/\left(V^2\cos \gamma \right),A_{65}=-gn\sin \sigma \sin \gamma /\left(V{\cos }^2\gamma \right),\\ B_{11}=B_{12}=B_{13}=B_{21}=B_{22}=B_{23}=B_{31}=B_{32}=B_{33}=0,\\ B_{41}=T_{\max }/m,B_{42}=B_{43}=0,\\ B_{51}=-D_n/m,B_{52}=g\cos \sigma /V,B_{53}=-gn\sin \sigma ,\\ B_{61}=0,B_{62}=-g\sin \sigma /\left(V\cos \gamma \right),B_{63}=-gn\cos \sigma /\left(V\cos \gamma \right).\end{array}$$

(35)

where $D_z$, $D_V$ and $D_n$ are the partial derivatives of the drag $D$ on the feature point of the reference trajectory to the flight height $z$, velocity $V$ and normal overload $n$ respectively. Define the optimal control performance index from $t_0$ to $t_f$ as follows:

$$J=0.5{\displaystyle {\int }_{t_0}^{t_f}\left[X^{\mathrm{T}}\left(t\right)QX\left(t\right)+u^{\mathrm{T}}\left(t\right)Ru\left(t\right)\right]} \mathrm{dt},$$

(36)

where $Q$ and $R$ are the weight matrices of state and control respectively. $Q$ is positive semi-definite and $R$ is positive-definite. Then, there exists an optimal control law $u^{*}=-K^{*}X$ to minimize the above performance index, and the feedback gain matrix $K^{*}$ is

$$K^{*}=\left[\begin{array}{cccccc}{K}_{\eta 1}& K_{\eta 2}& K_{\eta 3}& K_{\eta 4}& K_{\eta 5}& K_{\eta 6}\\ K_{n1}& K_{n2}& K_{n3}& K_{n4}& K_{n5}& K_{n6}\\ K_{\sigma 1}& K_{\sigma 2}& K_{\sigma 3}& K_{\sigma 4}& K_{\sigma 5}& K_{\sigma 6}\end{array}\right],$$

(37)

$$K^{*}=-R^{-1}{B}^{\mathrm{T}}P$$

(38)

where $P$ is the solution of the Riccati equation. It is calculated by

$$-PA-A^{\mathrm{T}}P+PB{R}^{-1}{B}^{\mathrm{T}}P-Q=0.$$

(39)

Define $Q$ and $R$ as follows:

$$\begin{array}{l}Q=\mathrm{diag}\left[Q_1,Q_2,Q_3,Q_4,Q_5,Q_6\right],\\ R=\mathrm{diag}\left[R_1,R_2,R_3\right].\end{array}$$

(40)

To reflect the impact of the flight relative distance in the dual aircraft formation flight. Set $Q_1=Q_2=Q_3$ and define $\delta D^2=\Delta D^2=\delta x^2+\delta y^2+\delta z^2$, then

$$\begin{array}{ll}\hfill J=& 0.5{\displaystyle {\int }_{t_0}^{t_f}\left[\left(Q_1\delta D^2+Q_4\delta V^2+Q_5\delta {\gamma }^2+Q_6\delta {\psi }^2\right)\right.}\hfill \\ & \left.+\left(R_1\delta {\eta }^2+R_2\delta n^2+R_3\delta {\sigma }^2\right)\right]\mathrm{dt}.\hfill \end{array}$$

(41)

According to Bryson Law [38], $Q$ and $R$ are set as follows:

$$\begin{array}{l}{Q}_1{D}^2{}_{\Delta \max }=Q_4{V}^2{}_{\Delta \max }=Q_5{\gamma }^2{}_{\Delta \max }=Q_6{\psi }^2{}_{\Delta \max }\\ =R_1{\eta }^2{}_{\max }=R_2{n}^2{}_{\max }=R_3{\sigma }^2{}_{\max }.\end{array}$$

(42)

Set $Q_1=1$, then other parameters can be obtained. According to $u^{*}$, the control commands can be obtained:

$$\begin{array}{l}\eta ={\eta }_L-\left(K_{\eta 1}\delta x+K_{\eta 2}\delta y+K_{\eta 3}\delta z+K_{\eta 4}\delta V+K_{\eta 5}\delta \gamma +K_{\eta 6}\delta \psi \right),\\ n=n_L-\left(K_{n1}\delta x+K_{n2}\delta y+K_{n3}\delta z+K_{n4}\delta V+K_{n5}\delta \gamma +K_{n6}\delta \psi \right),\\ \sigma ={\sigma }_L-\left(K_{\sigma 1}\delta x+K_{\sigma 2}\delta y+K_{\sigma 3}\delta z+K_{\sigma 4}\delta V+K_{\sigma 5}\delta \gamma +K_{\sigma 6}\delta \psi \right).\end{array}$$

(43)

Since the feedback gains obtained at different feature points of the reference trajectory are different, the monotonic flights can be selected as an independent variable, and the feedback gain coefficient of the offline design can be interpolated to obtain the corresponding control commands.

4.3. Experiment of Nominal Conditions

In the experiment of nominal conditions, the initial position of the MAV was $x_{L0}=0$ m, $y_{L0}=0$ m, $z_{L0}=\mathrm{10,000}$ m, $V_{L0}=400$ m/s, ${\gamma }_{L0}=\pi /6$, ${\psi }_{L0}=0$. The initial position of the UAV was $x_{W0}=100$ m, $y_{W0}=100$ m, $z_{W0}=\mathrm{10,000}$ m, $V_{W0}=400$ m/s, ${\gamma }_{W0}=\pi /6$, ${\psi }_{W0}=0$.

The formation flight trajectories of MAV and UAV of the three methods are shown in Figure 5. The MAV is designed to make continuous S-shaped large maneuver with a maximum overload of about 4 g at 1 s, 11 s, 29 s and 41 s, respectively. Figure 5 indicates that the LQR, DDPG, and HIAC can realize the stable tracking of the given trajectory of the MAV under large, overloaded maneuvers and reach the target of the designed formation.

Figure 6a–c shows the control commands of the UAV, i.e., the thrust, the normal overload, and the bank angle, generated by the LQR, DDPG and HIAC, respectively, with reference commands of the MAV. Figure 7a–c shows the errors between the control commands of the LQR, DDPG, and HIAC and the reference commands of the MAV. Figure 6 illustrates that there are four peaks in the curves of the control commands due to the four large, overloaded maneuvers. Moreover, compared with the LQR and DDPG, the trend of the control commands of the HIAC can be better consistent with the MAV in thrust, normal overload, and bank angle. Especially in the control of the normal overload, the HIAC has mitigated the sharp change of the commands generated by the reinforcement learning controller to a certain extent. It can provide more smooth and executable control commands under large maneuvers. However, during the large maneuver of the MAV, in order to track the reference commands, it inevitably generates a certain amount of extra adjustment for the thrust, overload, and bank angle for the three methods.

Figure 8 shows the change of the three controlled states of the UAV, i.e., the velocity, the flight path angle, and the flight azimuth angle, generated by LQR, DDPG and HIAC. Figure 9 shows the deviation of the three controlled states and the relative distance. It can be seen from Figure 8 that the change trend of the controlled state of the HIAC is basically the same as that of the MAV, and the formation maintenance performance is obviously better than that of the LQR and DDPG. Especially, the HIAC can keep up with most of the fluctuations of the MAV in the flight velocity and the flight azimuth angle. Moreover, Figure 9 shows that compared with the LQR and DDPG, the control precision of the HIAC has been significantly improved, and the control deviation can rapidly decrease to nearly 0 under the large maneuver. Figure 9d indicates that the HIAC successfully limits the formation distance within the safe distance between 100 m and 600 m while LQR and DDPG fail. The LQR continuously accumulates distance deviation due to the velocity deviation during the flight, and ultimately, the formation distance reveals a divergent trend. Meanwhile, although the relative distance of the DDPG gradually converges, it still extends beyond the safe distance at the end of the flight.

Table 3. RMS and maximum errors of the four states of the LQR, DDPG, and HIAC in nominal conditions.

Controller		Velocity (m/s)	Flight Path Angle (rad)	Flight Azimuth Angle (rad)	Relative Distance (m) Safe Distance [100, 600]
LQR	RMS	5.6957	0.4737	0.6202	516.7072
	Max.	15.3307	0.8027	0.7833	710.2799
DDPG	RMS	8.3953	0.2183	0.4493	444.1190
	Max.	13.3379	0.5241	0.5315	610.6078
HIAC	RMS	5.3647	0.1401	0.2174	460.0709
	Max.	6.0780	0.3586	0.2391	552.1845

presents the root mean square (RMS) errors and maximum errors of the four controlled states of the LQR, DDPG and HIAC. It is clear that both the RMS error and maximum error of the HIAC are smaller than those of the LQR and DDPG. Moreover, the HIAC has a reduction of 5.81%, 70.44%, and 64.95%, respectively, in the RMS error of the velocity, flight path angle and flight azimuth angle compared with the LQR, and has a reduction of 60.35%, 55.32% and 69.47% in the maximum error of velocity, flight path angle and flight azimuth angle, respectively, compared with the LQR. The HIAC has a reduction of 36.10%, 35.85% and 51.61%, respectively, in the RMS error of velocity, flight path angle and flight azimuth angle compared with the DDPG, and has a reduction of 54.43%, 31.57% and 55.01% in the maximum error of velocity, flight path angle and flight azimuth angle, respectively, compared with the DDPG.

In summary, the proposed HIAC significantly improves the state control performance and guarantees that the flight distance stays within a safe distance as well.

4.4. Monte Carlo Experiments

In order to further test how the HIAC adapts to various initial conditions, 100 Monte Carlo simulations were carried out by adding random deviations to the nominal conditions.

The initial position of the MAV is $x_{L0}=0$ m, $y_{L0}=0$ m, $z_{L0}=\mathrm{10,000}$ m, $V_{L0}=400$ m/s, ${\gamma }_{L0}=\pi /6$, ${\psi }_{L0}=0$. The baseline of initial values of the UAV is $x_{W0}=100$ m, $y_{W0}=100$ m, $z_{W0}=\mathrm{10,000}$ m, $V_{W0}=400$ m/s, ${\gamma }_{W0}=\pi /6$, ${\psi }_{W0}=0$. Then, random deviations which follow the uniform distributions were added to these six baselines, respectively. The specific values of the deviations are presented in

Table 4. Uniform distribution of deviations for the six initial values.

Numbers of Monte Carlo Simulations	X (m)	Y (m)	Z (m)	Velocity (m/s)	Flight Path Angle (rad)	Flight Azimuth Angle (rad)
100	[−50, 550]	[−50, 550]	[−1000, 1000]	[−100, 100]	[−$\pi$/18, $\pi$/18]	[−$\pi$/18, $\pi$/18]

.

Figure 10 is the scatterplot of the Monte Carlo simulation results of the velocity errors, flight path angle error, flight azimuth angle, and relative distance for the LQR, DDPG and HIAC. For each evaluation index, the horizontal axis is the RMS error, and the vertical axis is the maximum error. It can be seen that the HIAC can fulfill the control target in the magnitude of velocity. Meanwhile, because the training threshold is set quite strictly in order to achieve better control performance, the maximum error and RMS error of the flight path angle and flight azimuth angle may extend out of the threshold when the extreme deviations are added to the initial values. However, the HIAC can still present a satisfactory control accuracy of the angle compared with the DDPG and LQR. Moreover, in terms of the safety distance, the HIAC can stay within a safe distance of 100 m to 600 m from the MAV, which reaches the distance control target. However, the DDPG and LQR gradually extend out of the safe distance as the initial values vary. Statistically, compared with LQR and DDPG, the HIAC has smaller values in both the RMS error and maximum error of these four evaluation indices. In summary, the performance of the HIAC in formation control is better than that of the other two methods, which is consistent with the simulation results under nominal conditions. It is believed that the HIAC has significant adaptability to the varying initial conditions.

5. Conclusions

In this study, a novel HIAC method was proposed, which is able to enhance the smoothness and executability of control commands and improve the control performance of the MAV/UAV flight formation. First, based on the idea of “meta-action” in hybrid reinforcement learning, the formation control was modeled as a continuous–discrete space control problem. Then, we proposed the framework of the HIAC, and the hybrid intelligent agent model based on the DDPG/DDQN was designed through multi-channel decoupling. Finally, we carried out simulations of nominal conditions and 100 Monte Carlo simulations in varying initial conditions. The simulation results showed that, compared with the traditional LQR and DDPG, the HIAC has better performance of high control precision and rapid convergence. Meanwhile, the adaptability of HIAC to the varying initial conditions was verified as well.

For further practical applications, HIAC can gradually support practical scenarios such as formation military operations and terrain surveys. In particular, two aspects should be considered when applying HIAC. The first is the reliability of the method. HIAC should be preliminarily trained with a large number of ground tests before the real flights, to ensure that intelligent control gradually takes authority over traditional flight-control methods. The second is the portability of the method. At present, the method supports the deployment of reinforcement learning on hardware such as DSP, and FPGA, and can realize airborne portability and the online training of agent models.