PPO-Exp: Keeping Fixed-Wing UAV Formation with Deep Reinforcement Learning

Abstract

Flocking for fixed-Wing Unmanned Aerial Vehicles (UAVs) is an extremely complex challenge due to fixed-wing UAV’s control problem and the system’s coordinate difficulty. Recently, flocking approaches based on reinforcement learning have attracted attention. However, current methods also require that each UAV makes the decision decentralized, which increases the cost and computation of the whole UAV system. This paper researches a low-cost UAV formation system consisting of one leader (equipped with the intelligence chip) with five followers (without the intelligence chip), and proposes a centralized collision-free formation-keeping method. The communication in the whole process is considered and the protocol is designed by minimizing the communication cost. In addition, an analysis of the Proximal Policy Optimization (PPO) algorithm is provided; the paper derives the estimation error bound, and reveals the relationship between the bound and exploration. To encourage the agent to balance their exploration and estimation error bound, a version of PPO named PPO-Exploration (PPO-Exp) is proposed. It can adjust the clip constraint parameter and make the exploration mechanism more flexible. The results of the experiments show that PPO-Exp performs better than the current algorithms in these tasks.

1. Introduction

In recent years, unmanned aerial vehicles (UAVs) have been widely used in military and civil fields, such as in tracking [1], surveillance [2], delivery [3], and communication [4]. Due to the inherent defects, such as fewer platform functions and a light payload, it is difficult for a single UAV to perform diversified tasks in complex environments [5]. The cooperative formation composed of multiple UAVs can effectively compensate for the lack of performance and has many advantages in performing combat tasks. Thus, the formation control of UAVs has become a hot topic and attracted much attention [6,7].

Traditional solutions are usually based on accurate models of the platform and disturbance, such as model predictive control [8] and consistency theory [9]. This paper [10] proposed a group-based hierarchical flocking control approach, which did not need the global information of the UAV swarms. The study in [11] researched the mission-oriented miniature fixed-wing UAV flocking problem and proposed an architecture that decomposes the complex problem; it was the first work that successfully integrated the formation flight, target recognition, and tracking missions into simply an architecture. However, due to the influence of environmental disruption, these methods are difficult to accurately model [12]. This seriously limits the application scope of traditional analysis methods. Therefore, with the emergence of machine learning (ML), the reinforcement learning (RL) [13,14] method to solve the above problem has received increasing attention [15]. RL applies to decision-making control problems in unknown environments and has achieved successful applications in the robotics field [16,17,18].

At present, some works have integrated RL into the formation coordination control problem solution and preliminarily verified the feasibility and effectiveness in the simulation environment. Most existing schemes use the particle agent model for the rotary-wing UAV. The researchers [19] first researched RL in coordinated control, and applied the Q-learning algorithm and potential field force method to learn the aggregation strategy. After that, ref. [20] proposed a multi-agent self-organizing system based on a Q-learning algorithm. Ref. [21] investigates second-order multi-agent flocking systems and proposed a single critic reinforcement learning approach. The study in [22] proposes a UAV formation coordination control method based on the Deep Deterministic Policy Gradient algorithm, which enabled UAVs to perform navigation tasks in a completely decentralized manner in a large-scale complex environment.

Different from rotary-wing UAVs, the formation coordination control of fixed-wing UAVs is more complex and more vulnerable to environmental disturbance; therefore, different control strategies are required [23]. The Dyna-Q($\lambda$) and Q-flocking algorithm are proposed [24,25] for solving the discrete state & action space fixed-wing UAV flocking problem under complex noise environments with deep reinforcement learning. To deal with the continuous space, ref. [26,27] proposed a fixed-wing UAV flocking method in continuous spaces based on deep RL with the actor–critic model. The learned policy can be directly transferred to the semi-physical simulation. Ref. [28] focused on the nonlinear attitude control problem and devised a proof-of-concept controller using proximal policy optimization.

However, the above methods also assume that UAVs fly with different attitudes, so the interaction (collision) between the followers can be ignored, and the followers in the above methods are seen as independent. Under the independent condition, these single-agent reinforcement learning algorithms can be effective due to the stationary environment [29]. However, in real tasks, even when the attitude is different, the collision still may happen when the attitude difference is not significant, and the UAVs adjust their roll angles.

In real tasks, the followers can interact with each other, and it is also common for them to collide in some scenarios, such as the identical attitude flocking task. However, this scenario is rarely studied. Ref. [30] proposed a collision-free multi-agent flocking approach MA2D3QN by using the local situation map to generate the collision risk map. The experimental results demonstrate that it can reduce the collision rate. The followers’ reward function in MA2D3QN is only related to the leader and itself; however, other followers can also provide some information. This indicates that the method did not fully consider the interaction between the followers.

However, MA2D3QN did not demonstrate the ability to manage the non-stationary multi-agent environment [29], and the experiments also show collision judgments with high computation. With the number of UAVs rising, the computation time also increases. Furthermore, some problems in the above methods on fixed-wing UAVs have not been adequately solved, such as the generalization aspect and communication protocol; the most concerning problem is the minimum cost of the formation.

To consider the communication protocol of the formation, this paper takes the maximum communication distance between the UAVs into consideration, with a minimum cost communication protocol to guide the UAVs to send the message in the formation-keeping process. Under this protocol, the centralized training method for the UAVs is designed; only the leader needs to equip the intelligence chip. The main contributions of this work are as follows:

• Research the formation keeping task with continuous space through reinforcement learning, and building the RL formation-keeping environment with OpenAI gym, and constructing the reward function for the task.
• Design the communication protocol for the UAVs’ formation with one leader who can make decisions intelligently and five followers who receive the decisions from the leader. The protocol is feasible even when the UAVs are far away from each other. Under this protocol, the followers and leader can communicate at a low cost.
• Analyze the PPO-Clip algorithm, give the estimation error bound of its surrogate, and elaborate on the relationship between the bound and hyperparameter $\epsilon$: the higher $\epsilon$, the more exploration, the larger the bound.
• Propose a variation of PPO-clip: PPO-Exp. The PPO-Exp separates the exploration reward and regular reward in the task of formation keeping, and estimates the advantage function from them, respectively. The adaptive mechanism is used to adjust $\epsilon$ to balance the estimation error bound and exploration. The experiments demonstrate this mechanism with effectiveness for improving performance.

This paper is organized as follows. The first section introduced the current research on UAV flocking. Section 3 describes the background of the formation-keeping task and introduces reinforcement learning briefly. In Section 4, the formation-keeping environment is constructed, and the reward of the formation process is designed. Section 5 discusses the dilemma between the estimation error bound and exploration ability of PPO-Clip, and proposes PPO-Exp to balance the dilemma. Section 6 shows the experimental setup and results. Section 7 provides the conclusions of the paper.

2. Related Work

This section reviews current research about fixed-wing UAV flocking and formation-keeping approaches with deep reinforcement learning. According to the training architecture, this paper divides the current methods into the following two categories: centralized and decentralized. The difference between the two categories is as follows:

The centralized methods utilize the leader and all the followers’ states in the training model, and the obtained optimal policy can control all of the followers so that they flock to the leader. The decentralized methods only use one follower and the leader’s state to train the policy, and the obtained optimal policy could only control one follower. If there are several followers in the task, the policy and intelligence chip should be deployed on all of the followers.

2.1. Decentralized Approach

The paper [24] proposed a reinforcement learning flocking approach Dyna-Q($\lambda$) to flock the fixed-wing UAV under the stochastic environment. To learn a model in the complex environment, the authors used Q($\lambda$) [31] and Dyna architecture to train each fixed-wing follower to follow the leader, and combined internal models to deal with the influence of the stochastic environment. In [25], the authors further proposed Q-Flocking, which is a model-free and variable learning parameter algorithm based on Q-learning. Compared to Dyna-Q($\lambda$), Q-Flocking removed the internal models and proved it could also converge to the solutions. For simplification, Q-Flocking and Dyna-Q($\lambda$) also require that the state and action spaces are discrete, which is inappropriate. In [26], the authors first developed a DRL-based approach for the continuous state and action spaces fixed-wing UAV flocking. The proposed method is based on the Continuous Actor-Critic Learning Automation(CACLA) algorithm [32], with the experience replay technique embedded to improve the training efficiency. Ref. [33] considered a more complex flocking scenario, where the enemy threat is considered in the dynamic environment. To learn the optimal control policies, the authors use the situation assessment module to transfer the state of UAVs to the situation map stack. Then, the stack is input into the proposed Dueling Double Deep Q-network(D3QN) algorithm to update the policies until convergence. Ref. [34] proposed the Multi-Agent PPO algorithm to decentralize learning in the two–group fixed-wing UAV swarms dog fight control. To accelerate the learning speed, the classical rewarding scheme is added to the resource baseline, which could reduce the state and action spaces.

The advantage of decentralized methods is that these methods could be deployed on the distribution UAV systems, which could extend to the large-scale UAV formation. The disadvantage of the centralized methods is as follows:

• These methods also require all of the followers to be equipped with intelligence chips, which increase the costs.
• These methods do not consider the collision and communication problem, due to the use of only local information.

The decentralized approaches also assume that UAVs fly at different heights, and then the collision problem could be ignored. However, in real-world applications, the collision problem must be considered [30].

2.2. Centralized Approach

Ref. [35] studied the collision avoidance fixed-wing UAV flocking problem. To manage collision among the UAVs, the authors proposed the PS-CACER algorithm, which receives the global information of UAV swarms through the plug-n-play embedding module. Ref. [30] proposed a collision-free approach by transferring the global state information to the local situation map and constructing the collision risk function for training. To improve the training efficiency, the reference-point-based action selection technique is proposed to assist the UAVs’ decisions.

The advantages of the centralized methods are as follows:

• These methods could reduce the cost of the formation. Under the centralized architecture, the formation system only requires the leader to equip the intelligence chip. The followers only need to send their state information to the leader and receive the feedback commands.
• These methods could consider collision avoidance and communication in the formation due to their use of global information.

The disadvantage of the centralized method is the dependence on the leader. Ref. [36] pointed out that the defect or jamming of the leader causes failure in the whole formation system.

When the number of UAVs increases or the tasks are complex, the centralized methods face the dimension curse and lack of learning ability problems. A popular approach is learning the complex tasks with a hierarchical method [37,38], which divides the complex tasks into several sub-tasks and uses the centralized method to optimize the hierarchies. The hierarchical reinforcement learning approaches are applied in the quadrotors swarm system [37,38], but are rarely used in fixed-wing UAV systems.

Even when using global information in training, the current centralized approaches fail to consider communication in the formation. Compared to current centralized approaches, the approach proposed in this paper considers the communication in formation, and provides the communication protocol. Through the communication protocol, the formation system could be considered as one leader with an intelligence chip and five followers without intelligence chips; the leader collects the followers’ information, with a centralized train on the intelligence chip. The followers receive the command from the leader through this protocol and execute.

3. Background

This section will introduce the kinematic model of the fixed-wing UAV, restate the formation keeping problem, and briefly introduce reinforcement learning.

3.1. Problem Description

The formation task can be described as follows: At the beginning, the formation is orderly (shown in Figure 1), which is a common formation designed in [39]). The goal of the task is to reach the target area (the green circle area) with the formation in as orderly a way as possible; when the leader enters the target area, the mission is complete.

During the task, assume the UAVs are flying at a fixed attitude; then, each UAV in the formation can also be described as a six-degree of freedom (6DoF) dynamic model. However, analyzing the six-degree model directly is very complex; it will increase the space scale and make control more difficult. The 6DoF model can be simplified to the 4DoF model; to compensate for the loss incurred during this simplification, random noise is introduced into the model [27], and the dynamic equations of ith UAV in the formation can be written as follows:

$${\dot{\xi }}_i=\frac{\mathrm{d}}{\mathrm{d}t}\left[\begin{array}{c}{x}_i\hfill \\ y_i\hfill \\ {\psi }_i\hfill \\ {\phi }_i\hfill \end{array}\right]=\left[\begin{array}{c}{v}_{i}cos{\psi }_i+{\eta }_{x_i}\\ v_{i}sin{\psi }_i+{\eta }_{y_i}\\ -\left({\alpha }_{\mathrm{g}}/v_i\right)tan{\phi }_i+{\eta }_{{\psi }_i}\\ f({\phi }_i,{\phi }_{i,d})\end{array}\right]$$

(1)

where $(x_i,y_i)\in {\mathbb{R}}^2$ is the planar position, and ${\psi }_i\in {\mathbb{R}}^1,{\phi }_i\in {\mathbb{R}}^1$ represent the heading and roll angle, respectively, (see Figure 1). The $v_i$ is the velocity, and ${\alpha }_g$ is the gravity acceleration. The random noise values ${\eta }_{x_i},{\eta }_{y_i},{\eta }_{{\phi }_i},{\eta }_{{\psi }_i}$ are the normal distributions, its means are ${\mu }_{x_i},{\mu }_{y_i},$${\mu }_{{\phi }_i},$${\mu }_{{\psi }_i}$, and its variances are ${\sigma }_{x_i}^2,{\sigma }_{y_i}^2,{\sigma }_{{\phi }_i}^2,{\sigma }_{{\psi }_i}^2$, respectively, (the gray dotted circles in Figure 1 show the area of influence, of random factors); they represent the random factors introduced by simplification and environment noise.

A simple control strategy can make the formation satisfactory when the environment’s noise is low. However, under a strong inference environment, such as one with strong turbulence, the random factors will be apparent, leading the formation to maintain the complexity of the task. If no effective control is provided, the formation will break up quickly, (this is demonstrated in Figure 2), and a crash may happen.

Furthermore, even though there is an effective control policy for the formation, the coupling between the control and communication protocol can also be an unsolved challenge. Because the communication range of UAVs is limited, if the UAV wants to know others’ states, it has to wait for other UAVs out of range to send state information to UAVs it can communicate with, which in turn send state information to it. If no harmonic protocol is applied in the formation control, the asynchronous and nonstationary elements will be introduced into the formation control, making the control strategy more complex.

3.2. Reinforcement Learning

In the last part, the solution of differential Equation (1) can be represented as the current dynamic parameters adding the integral items by difference equation methods such as the Runge–Kutta method. So, the UAV formation control can be modeled as a Markov Decision Process(MDP), which refers to the decision process that satisfies the Markov property.

The MDP also can be described as the tuple $(\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma )$. $\mathcal{S}$ represents the state space, $\mathcal{A}$ represents the action space, and $\mathcal{P}:\mathcal{S}\times \mathcal{A}\times \mathcal{S}\to \mathcal{R}$ is the transition probability. The reward function is $r:\mathcal{S}\times \mathcal{A}\to \mathcal{R}$, and $\gamma \in (0,1)$ is the discount factor, which leads the agent to pay more attention to the current reward.

Reinforcement learning can solve the MDP well to maximize the discounted return, as follows: $R_t={\sum }_{t=0}^{\infty }{\gamma }^{t}r\left(s_t\right)$. The main approaches of RL are divided into the following three categories: value-based, model-based, and policy-based. The policy-based methods have been developed and widely used in various tasks in recent years. These methods directly optimize the value function by the policy gradient:

$$\begin{array}{c}\hfill \nabla {\mathcal{J}}_{\pi }\left(\theta \right)={\mathbb{E}}_{{\pi }_{\theta }}\left[{\nabla }_{\theta }\sum _{t=0}^{T}log{\pi }_{\theta }\left(s_t,a_t\right)A_{\pi }\right]\end{array}$$

(2)

where $A_{\pi }$ is the advantage function that is equal to the state-action value function, and the the state value function is subtracted, as follows:

$$\begin{array}{c}\hfill A_{\pi }\left(S_t,a_t\right)=E_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k}\left|S_t=s,a_t=a\right.\right]-E_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k}\left|S_t=s\right.\right]\end{array}$$

(3)

PPO (Proximal Policy Optimization) is one of the most famous policy gradient methods in continuous state and action space [40]. In policy gradient descent, PPO updates the following equation at each update epoch:

$$\begin{array}{c}\hfill {\mathcal{L}}^{\mathrm{Clip},\theta }={\mathbb{E}}_{{\pi }_{{\theta }_{old}}}\left[min\left(r_t\left(\theta \right)A_{{\pi }_{{\theta }_{old}}},clip\left(r_t\left(\theta \right),1-\epsilon ,1+\epsilon \right)A_{{\pi }_{{\theta }_{old}}}\right)\right]\end{array}$$

(4)

However, using the constant clip coefficient $\epsilon$, the PPO also proved its lack of exploration ability and difficulty in convergence. Therefore, designing an efficient dynamic mechanism to adjust $\epsilon$ and ensure greater exploration and faster convergence is also challenging.

4. Formation Environment

This section constructs the fixed-wing UAV formation-keeping environment, the formation topology, communication and control protocols, and collision. Communication loss is also considered in the environment through the reward design.

4.1. State and Action Spaces

In the course of the formation task, based on the 4DoF Equation (1), it is modified to a more realistic control environment. For the ith UAV, assume the thrust of the UAV is controllable, and it will generate a linear acceleration ${\alpha }_{v_i}=\dot{v_i}$. Moreover, assume the torque of the roll angle is controllable too, and add the roll angle acceleration ${\alpha }_{{\phi }_i}=\dot{w_i}=\ddot{{\phi }_i}$ into the dynamic equations. Finally, the dynamic equations of ith UAV can be modified as follows:

$${\dot{\xi }}_i=\frac{\mathrm{d}}{\mathrm{d}t}\left[\begin{array}{c}{x}_i\hfill \\ y_i\hfill \\ {\psi }_i\hfill \\ {\phi }_i\hfill \\ v_i\hfill \\ w_i\hfill \end{array}\right]=\left[\begin{array}{c}{v}_{i}cos{\psi }_i+{\alpha }_{v_i}cos\left({\psi }_i\right)t+{\eta }_{x_i}\\ v_{i}sin{\psi }_i+{\alpha }_{v_i}sin\left({\psi }_i\right)t+{\eta }_{y_i}\\ -\left({\alpha }_{\mathrm{g}}/v_i\right)tan{\phi }_i+{\eta }_{{\psi }_i}\\ {\omega }_i+{\eta }_{{\omega }_i}\\ {\alpha }_{v_i}\\ {\alpha }_{{\phi }_i}\end{array}\right]$$

(5)

To control the UAVs, linear acceleration and roll angle acceleration are input. For control, we have the dynamic model of ith UAV:

$${\ddot{\xi }}_i=\frac{\mathrm{d}}{\mathrm{d}t}\left[\begin{array}{c}\dot{x_i}\hfill \\ \dot{y_i}\hfill \\ \dot{{\psi }_i}\hfill \\ \dot{{\phi }_i}\hfill \end{array}\right]=\left[\begin{array}{c}{\alpha }_{v_i}cos{\psi }_i\\ {\alpha }_{v_i}sin{\psi }_i\\ \frac{-{\alpha }_{g}f({\phi }_i,{\phi }_{i,d})}{v_i{cos}^2{\phi }_i}+\frac{{\alpha }_{v_i}{\alpha }_{g}tan{\phi }_i}{v_i^2}\\ {\alpha }_{{\phi }_i}\end{array}\right]$$

(6)

The state and action spaces for existing methods in UAVs controlled by reinforcement learning are often discrete, but in the real world, the state space is continuous and changes continuously as time goes on. Therefore, combining the analysis of the previous dynamics, we define the state tuple of the ith UAV as ${\xi }_i:=(x_i,y_i,{\psi }_i,{\phi }_i,v_i,w_i)$. The planar position $(x_i,y_i)\in {\mathbb{R}}^2$, heading ${\psi }_i\in S^1$, roll angle ${\phi }_i\in S^1$, line and angle velocity $v,w\in \mathbb{R}$ are determined by solving the differential Equation (5).

In the action space, although the engine can produce fixed thrust, the real thrust acting on the UAVs in the nonuniform atmospheric environment is not of the same value as the engine product. So, we define the action space by $a_i:=({\alpha }_{v_i},{\alpha }_{{\phi }_i})$. Assume the UAVs can also produce the same acceleration in positive and negative directions, where we have ${\alpha }_{v_i}\in [-{\alpha }_{v_{i}max},{\alpha }_{v_{i}max}]$, and ${\alpha }_{{\phi }_i}\in [-{\alpha }_{{\phi }_{i}max},{\alpha }_{{\phi }_{i}max}]$. The action will influence $\dot{{\xi }_i}$ through Equation (6), and then influence the $\dot{{\xi }_i}$ indirectly.

After defining the individual state and action of the UAV, we define the formation system state and action by sticking to the individual state (action) as a vector. Define the state of system $\xi :=[{\xi }_1,\cdots ,{\xi }_6]$, and the action of system $\mathbf{a}:=[a_1,\cdots ,a_6]$.

4.2. Communication and Control Protocol

To ensure the UAV formation consumes less energy in the information send and receive process, and ensure the reinforcement learning method can be helpful in the task, the communication and control protocol for the UAV formation will be provided in this part.

As is shown in Figure 1, the formation is of a Leader–Follower structure; in terms of hardware, all the UAVs are equipped with gyroscopes and accelerometers to monitor their action and state parameters. Only the leader has the “brain” chip that can make decisions intelligently; the followers only have the chips that can receive the control command signals, take the command action and send the state signals.

To describe this relationship, the graph model is introduced. Use the communication graph ${\mathcal{G}}_t$ to describe the communication ability of the formation at time t [39]:

$$\begin{array}{c}\hfill {\mathcal{G}}_t=(6,{\mathcal{V}}_t,{\mathcal{E}}_t)\end{array}$$

(7)

where ${\mathcal{V}}_t=\{v_1,\cdots ,v_6\}$ is the set of nodes that represent UAVs, the ${\mathcal{E}}_t$ represent the arc set at time t, e.g., $e_{i,j}\in {\mathcal{E}}_t$ denote an arc from node i to node j, which means the UAV i can communicate with UAV j directly at time t. The adjacent matrix ${\mathcal{A}}_t=\left\{a_{i,j}\right\}$ of graph ${\mathcal{G}}_t$ is used to describe the communicated situation of formation in real-time, e.g., at the initial time, the adjacent matrix is as follows:

$$\begin{array}{c}\hfill {\mathcal{A}}_0=\left[\begin{array}{cccccc}0& 1& 0& 1& 0& 1\\ 1& 0& 1& 0& 0& 0\\ 0& 1& 0& 0& 0& 0\\ 1& 0& 0& 0& 1& 0\\ 0& 0& 0& 1& 0& 0\\ 1& 0& 0& 0& 0& 0\end{array}\right]\end{array}$$

(8)

The adjacent matrix is symmetric, and its element $a_{ij}$ indicates the communication situation of UAV i and UAV j. If $a_{ij}=1$, then $a_{ji}=1$, and the ith and jth UAVs can share their state, and the control command can be sent from i to j or j to i. The adjacent matrix is updated in real-time. If the distance between two UAVs is greater than the communication limited distance $d_{com}$, the corresponding elements of the adjacent matrix will be 0.

Additionally, at the initial time, the formation is connected, and the connected component $\mathcal{W}$ is 1. l If the UAVs want to keep in communication with all the others, the graph $\mathcal{G}$ should only have one connected component. In the graph model, this condition could be transferred to $\mathcal{G}$. The methods that judge whether an undirected graph is connected include union-find disjoint sets, DFS, and BFS [41]. So, after DFS or BFS, the task fails when the connected component number $\mathcal{W}$ of graph $\mathcal{G}$ is more than 1. When the formation works, $\mathcal{W}$ should be 1.

When the formation works, the protocol should be active to support the UAVs communicating with each other. The communication protocol’s primary purpose is to send all the UAVs’ states to the leader for the decision; the control protocol sends the action command to all the UAVs. When the formation is as orderly as it was at first, the information only needs to obey the transfer route (shown in Figure 1), so the whole formation can be controlled well. However, when the noise disturbs the position of UAVs, it makes the connection between the UAVs that are not connected at the initial time. It breaks the connection between the UAVs that are connected at the initial time. To handle the chaos brought about by the noise, a communication and control protocol is shown in Figure 3.

In Figure 3a the communication protocol is shown, where the block in ith row represents the communication priority of the corresponding UAV. For the priority, the bigger the number, the higher the priority. Priority 1, 2 determines the order of communication. If the priority is 0, both parties have no communication probability. i.e., when the leader0 and follower3 are within the communication range of follower5, the follower5 will send the information to leader0 instead of follower3.

The protocol is designed based on the communication object: to send all the followers’ state information to the leader to support the decision. So, the principle of the protocol is to give the followers closer to the leader higher priority, such as followers 1, 3 and 5.

Figure 3b has a similar meaning to the control protocol. The target of the control protocol sends the control information to all the UAVs. The control protocol motivates the leader to send the control information to the followers that connects as much as the followers. Therefore, leader0, and follower1, 2, and 5 have priority 2 because they can connect with up to 2 other followers.

4.3. Reward Scheme

The goal of the formation-keeping task is to reach the target area and ensure the formation is as orderly as possible. At first, the orderliness of the formation is of primary concern. So, some geometric parameters are defined to describe the formation. The followers in the formation can be divided into two categories, one is on an oblique line with the leader, like followers 3 and 4, and another is on a straight line with the leader. Only follower 5 belongs to this category. The linear between the leader and the position where the follower should be located is called the baseline (see the back lines in Figure 4). Then, it is easy to know the first category followers have a baseline with a slope, and the second follower’s baseline does not. For the follower i, the length of the initial baseline is $l_i$, and the initial slope is $k_i=tan\theta$ (the first category).

To make sure the UAV agent can return to the position that makes formation more orderly, for ith UAV, the formation reward is designed as follows:

$$\begin{array}{c}\hfill R_{f,i}=-max\{di{s}_{a,i},|di{s}_{b,i}-l_i\left|\right\}\end{array}$$

(9)

where $di{s}_{a,i}$ represents the distance between the follower i and the baseline along the vertical line of the baseline, and $di{s}_{b,i}$ represents the distance between the leader and follower i along the baseline. The formation reward is $R_{f,i}$.

When the UAVs belong to the first category follower (e.g., follower 3), the distance $di{s}_{a,3}$ can be calculated by the following formula:

$$\begin{array}{c}\hfill di{s}_{a,3}=\frac{|x_{3}tan\theta -y_3+(y_0-x_{0}tan\theta )|}{\sqrt{1+tan\theta }}\end{array}$$

(10)

Followers 2 and 4 have the same $di{s}_a$ as in the above equation. The distance $di{s}_b$ also can be obtained with the following formula:

$$\begin{array}{c}\hfill di{s}_{b,3}=(x_3-x_0)cos\theta +(y_3-y_0)sin\theta +l_3\end{array}$$

(11)

For the second category follower (follower 5), it is easy to know that the reward can be represented as the following simple formation:

$$\begin{array}{c}\hfill R_{f,5}=-max\left\{\right|x_0-x_5|,|y_0-y_5-l_5\left|\right\}\end{array}$$

(12)

Furthermore, the main target of UAVs formation is to reach the target area, which is a circle with center coordinates $(x_{tar},y_{tar})$ and radius $r_{tar}$. To encourage the formation to reach the target area, a sparse reward is designed as the destination reward:

$$\begin{array}{c}\hfill R_d=\left\{\begin{array}{c}0,\sqrt{{(x_0-x_{tar})}^2+{(y_0-y_{tar})}^2}\le r_{tar}\\ 10,000,otherwise\end{array}\right.\end{array}$$

(13)

We only calculate the distance of the leader. Only when the formation reaches the target area do the UAVs receive this sparse reward, and the learning process will halt. It leads to the UAVs not only needing to take minor actions to ensure that the orderly formation is not disorganized by the disturbance, but also needing to adjust direction to reach the target area. From the reward design view, UAV agents need to try different actions to discover and obtain a sparse signal. To accelerate the learning, the exploration rewards, as described in the literature [42], are designed as the incentive reward:

$$\begin{array}{c}\hfill R_{e,i}=-max\left\{\right|x_i-x_{tar}|,|y_i-y_{tar}\left|\right\}\end{array}$$

(14)

When the formation is closer to the target area, it will receive a higher exploration reward, leading the UAV agent to learn to reach the target area.

Meanwhile, some UAVs are too close and crash together, or they are too far and cease communicating with each other. In that case, the formation will suffer permanent destruction, and the task will halt.

Setting the minimum distance for crashes makes it easy to obtain the halt condition of UAV crashes. Then, the penalty should be added to avoid the above situation. This penalty is designed as a formal sparse reward as follows:

$$\begin{array}{cc}\hfill R_p& =\left\{\begin{array}{c}-10,000,d_{i,j}\le d_{cra},\forall i,j=0,1,\cdots 5\\ -10,000,\mathcal{W}>1\\ 0,\mathrm{others}\end{array}\right.\hfill \end{array}$$

(15)

where the $d_{·,j}$ represents the minimum distance between the jth UAV and another five UAVs: $d_{·,j}={min}_i\left\{d_{i,j}\right\},\forall i=1,\cdots ,6,i\ne j$. The lowest communication distance is $d_{com}$, once the minimum distance $d_{·,j}$ less than $d_{com}$, the jth UAV will lose the communication ability with other UAVs. In addition, $d_{cra}$ is the crash distance; as long as the distance between two UAVs is less than this, the two UAVs might crash.

Finally, the reward of the formation system at time T can be represented as the sum of the following reward function:

$$\begin{array}{c}\hfill R\left(T\right)=\sum _{i=1}^6\left[R_{f,i}\left(T\right)+R_{e,i}\left(T\right)\right]+R_d\left(T\right)+R_p\left(T\right)\end{array}$$

(16)

5. PPO-Exp

PPO is one of the most popular deep reinforcement learning algorithms in continuous tasks that achieved outstanding performance. The PPO embedded the Actor–Critic algorithm, which uses a deep neural network as an Actor for policy generation, and another deep neural network as a Critic for policy estimating. The structure of PPO can be seen in Figure 5; the Actor interacts with the environment, collects the trajectories: $\{s_t,a_t,r_t,s_{t+1}\}$ and stores them in the buffer, then it uses the buffer and the value function estimated by the Critic to optimize the Actor network’s hyperparameter according to following surrogate:

$$\begin{array}{c}\hfill {\mathcal{L}}_t^{\mathrm{Clip},\theta }=\left\{\begin{array}{c}(1+\epsilon )A_{{\pi }_{{\theta }_{t-1}}};A_{{\pi }_{{\theta }_{t-1}}}>0,r_t>1+\epsilon \\ (1-\epsilon )A_{{\pi }_{{\theta }_{t-1}}};A_{{\pi }_{{\theta }_{t-1}}}<0,r_t<1-\epsilon \\ r_t·A_{{\pi }_{{\theta }_{t-1}}};otherwise\end{array}\right.\end{array}$$

(17)

where the $A_{{\pi }_{{\theta }_{t-1}}}$ is the advantage function defined in Equation (3). The Critic network’s hyperparameter $\varphi$ is updated by minimizing the following MSE error:

$$\begin{array}{c}\hfill {\mathcal{L}}_t^{\mathrm{Clip},\varphi }=\sum _t{\left(y_t-Q_{\varphi }(s_t,a_t)\right)}^2\end{array}$$

(18)

$$\begin{array}{c}\hfill y_t=r_t+\gamma ·Q_{\varphi }(s_{t+1},{\pi }_{{\theta }_{old}}\left(s_{t+1}\right))\end{array}$$

(19)

The gradient of Equations (17) and (18) is computed and used to update the hyperparameters $\theta$ and $\varphi$ until they converge or reach maximum steps. In surrogate (17), the PPO restricted the difference between new and old policy by using the clip trick to restrain the ratio $r_t=\frac{{\pi }_{\theta }(s_t,a_t)}{{\pi }_{{\theta }_{old}}(s_t,a_t)}$. It could be considered a constraint on updated policy; under it, the ratio should satisfy the following constraint: $1-\epsilon \le r_t\le 1+\epsilon$. Then, the updated policy is restricted as follows:

$$\begin{array}{c}\hfill \frac{|{\pi }_{\theta }(s_t,a_t)-{\pi }_{{\theta }_{old}}(s_t,a_t)|}{{\pi }_{{\theta }_{old}}(s_t,a_t)}\le \epsilon \end{array}$$

(20)

The coefficient $\epsilon$ is also a constant in the range $(0,1)$ in PPO-Clip; from the inequality (20), it can be seen that the relative deviation is bound between ${\pi }_{{\theta }_{old}}$ and ${\pi }_{\theta }$. When this deviation is under $\epsilon$, as the increase in $r_t$ is observed, the ${\mathcal{L}}_t^{\mathrm{Clip},\theta }$ increase as well, but when the deviation exceeds $\epsilon$, even if the $r_t$ is increases, the ${\mathcal{L}}_t^{\mathrm{Clip},\theta }$ maintains its value. It shows the exploration within the constraint $\epsilon$; however, when the relative difference is beyond $\epsilon$, the exploration is not encouraged by clipping the result to $(1+\epsilon )A_{{\pi }_{{\theta }_{old}}}$. Figure 6 shows the surrogate of PPO-Clip in different $\epsilon$. The large $\epsilon$ could encourage the agent to explore more and accept more policies. However, enlarging $\epsilon$ will lead to the estimated error of the surrogate. The PPO-Clip is the off-policy algorithm. The data generated by the old policy will be used as new policy updates. When $\parallel r_t-1\parallel \le \epsilon$, the estimated error bound of ${\mathcal{L}}^{\mathrm{Clip},\theta }$ will increase as $\epsilon$ increases. For convenience, denote the following assumption:

Assumption 1. 

In the previous t timestep of policy update, the ratio $r_k$ satisfies $\parallel r_k-1\parallel \le \epsilon ,$$\forall k=1,\cdots ,t$.

Under Assumption 1, the following Lemma is given for auxiliary proof of the error bound:

Lemma 1. 

Under Assumption 1, the difference of state distribution resulting from the policy satisfies the following inequality:

$$\begin{array}{c}\hfill \parallel {\rho }^{{\pi }_{{\theta }_t}}-{\rho }^{{\pi }_{{\theta }_{t-1}}}\parallel \le \frac{\epsilon ·\gamma }{1-\gamma }\end{array}$$

(21)

Proof. 

The distribution ${\rho }^{{\pi }_{\theta }}$ can be rewritten as [43]:

$$\begin{array}{c}\hfill {\rho }^{{\pi }_{\theta }}=(1-\gamma )\sum _{k=0}^{\infty }{\gamma }^k·d_{{\pi }_{\theta }}^k\end{array}$$

(22)

where $d_{{\pi }_{\theta }}^k$ is the distribution resulting from ${\pi }_{\theta }$ at k timestep. Using the Markov property, $\forall s^{\prime }\in \mathcal{S}$, the $d_{{\pi }_{\theta }}^k\left(s^{\prime }\right)$ could be decompose as follows:

$$\begin{array}{c}\hfill d_{{\pi }_{\theta }}^k\left(s^{\prime }\right)=\sum _{s,a}{d}_{{\pi }_{\theta }}^{k-1}\left(s\right)·{\pi }_{\theta }\left(a\right|s)·P\left(s^{\prime }\right|s,a)\end{array}$$

(23)

Using the decomposition, the following equation holds:

$$\begin{array}{cc}\hfill & d_{{\pi }_{{\theta }_t}}^k\left(s^{\prime }\right)-d_{{\pi }_{{\theta }_{t-1}}}^k\left(s^{\prime }\right)=\sum _{s,a}\left[d_{{\pi }_{{\theta }_t}}^{k-1}\left(s\right)·{\pi }_{{\theta }_t}\left(a\right|s)-d_{{\pi }_{{\theta }_{t-1}}}^{k-1}\left(s\right){\pi }_{{\theta }_{t-1}}\left(a\right|s)\right]P\left(s^{\prime }\right|s,a)\hfill \\ \hfill & =\sum _{s,a}\left[d_{{\pi }_{{\theta }_t}}^{k-1}\left(s\right)·{\pi }_{{\theta }_t}\left(a\right|s)-d_{{\pi }_{{\theta }_{t-1}}}^{k-1}\left(s\right){\pi }_{{\theta }_{t-1}}\left(a\right|s)+d_{{\pi }_{{\theta }_{t-1}}}^{k-1}\left(s\right){\pi }_{{\theta }_{t-1}}\left(a\right|s)-d_{{\pi }_{{\theta }_{t-1}}}^{k-1}\left(s\right){\pi }_{{\theta }_{t-1}}\left(a\right|s)\right]\hfill \\ \hfill & ·P\left(s^{\prime }\right|s,a)\hfill \\ \hfill & =\sum _{s,a}\left[{\pi }_{{\theta }_t}\left(a\right|s)-{\pi }_{{\theta }_{t-1}}\left(a\right|s)\right]·d_{{\pi }_{{\theta }_{t-1}}}^{k-1}\left(s\right)·P\left(s^{\prime }\right|s,a)\hfill \\ \hfill & +\sum _{s,a}\left[d_{{\pi }_{{\theta }_{t-1}}}^{k-1}\left(s\right)-d_{{\pi }_{{\theta }_{t-1}}}^{k-1}\left(s\right)\right]·{\pi }_{{\theta }_{t-1}}\left(a\right|s)·P\left(s^{\prime }\right|s,a)\hfill \end{array}$$

(24)

Using the triangle inequality, the following equation hold:

$$\begin{array}{cc}\hfill & \sum _{s,a}\parallel {\pi }_{{\theta }_t}\left(a\right|s)-{\pi }_{{\theta }_{t-1}}\left(a\right|s)\parallel ·d_{{\pi }_{{\theta }_t}}^{k-1}\left(s\right)P\left(s^{\prime }\right|s,a)+\sum _{s,a}\parallel d_{{\pi }_{{\theta }_t}}^{k-1}\left(s\right)-d_{{\pi }_{{\theta }_{t-1}}}^{k-1}\left(s\right)\parallel ·{\pi }_{{\theta }_{t-1}}\left(a\right|s)·P\left(s^{\prime }\right|s,a)\hfill \\ \hfill & \ge \parallel \sum _{s,a}\left[{\pi }_{{\theta }_t}\left(a\right|s)-{\pi }_{{\theta }_{t-1}}\left(a\right|s)\right]·d_{{\pi }_{{\theta }_{t-1}}}^{k-1}\left(s\right)·P\left(s^{\prime }\right|s,a)\hfill \\ \hfill & +\sum _{s,a}\left[d_{{\pi }_{{\theta }_{t-1}}}^{k-1}\left(s\right)-d_{{\pi }_{{\theta }_{t-1}}}^{k-1}\left(s\right)\right]·{\pi }_{{\theta }_{t-1}}\left(a\right|s)·P\left(s^{\prime }\right|s,a)\parallel =\parallel d_{{\pi }_{{\theta }_t}}^k\left(s^{\prime }\right)-d_{{\pi }_{{\theta }_{t-1}}}^k\left(s^{\prime }\right)\parallel \hfill \end{array}$$

(25)

Sum up the inequality (25) to calculate the expectation on $s^{\prime }$:

$$\begin{array}{cc}\hfill & \parallel d_{{\pi }_{{\theta }_t}}^k-d_{{\pi }_{{\theta }_{t-1}}}^k\parallel \hfill \\ \hfill & =\sum _{s^{\prime }}\parallel d_{{\pi }_{{\theta }_t}}^k\left(s^{\prime }\right)-d_{{\pi }_{{\theta }_{t-1}}}^k\left(s^{\prime }\right)\parallel \le \sum _{s,a}\parallel {\pi }_{{\theta }_t}\left(a\right|s)-{\pi }_{{\theta }_{t-1}}\left(a\right|s)\parallel ·d_{{\pi }_{{\theta }_t}}^{k-1}\left(s\right)\sum _{s^{\prime }}P\left(s^{\prime }\right|s,a)\hfill \\ \hfill & +\sum _{s,a}\parallel d_{{\pi }_{{\theta }_t}}^{k-1}\left(s\right)-d_{{\pi }_{{\theta }_{t-1}}}^{k-1}\left(s\right)\parallel ·{\pi }_{{\theta }_{t-1}}\left(a\right|s)·\sum _{s^{\prime }}P\left(s^{\prime }\right|s,a)\hfill \\ \hfill & =\parallel {\pi }_{{\theta }_t}\left(a\right|s)-{\pi }_{{\theta }_{t-1}}\left(a\right|s)\parallel +\parallel d_{{\pi }_{{\theta }_t}}^{k-1}-d_{{\pi }_{{\theta }_{t-1}}}^{k-1}\parallel \hfill \\ \hfill & \le \parallel \frac{{\pi }_{{\theta }_t}\left(a\right|s)-{\pi }_{{\theta }_{t-1}}\left(a\right|s)}{{\pi }_{{\theta }_{t-1}}\left(a\right|s)}\parallel ·\parallel {\pi }_{{\theta }_{t-1}}\left(a\right|s)\parallel +\parallel d_{{\pi }_{{\theta }_t}}^{k-1}-d_{{\pi }_{{\theta }_{t-1}}}^{k-1}\parallel \hfill \\ \hfill & \le \parallel \frac{{\pi }_{{\theta }_t}\left(a\right|s)-{\pi }_{{\theta }_{t-1}}\left(a\right|s)}{{\pi }_{{\theta }_{t-1}}\left(a\right|s)}\parallel +\parallel d_{{\pi }_{{\theta }_t}}^{k-1}-d_{{\pi }_{{\theta }_{t-1}}}^{k-1}\parallel \le \epsilon +\parallel d_{{\pi }_{{\theta }_t}}^{k-1}-d_{{\pi }_{{\theta }_{t-1}}}^{k-1}\parallel \hfill \\ \hfill & \le 2\epsilon +\parallel d_{{\pi }_{{\theta }_t}}^{k-2}-d_{{\pi }_{{\theta }_{t-1}}}^{k-2}\parallel \le k\epsilon \hfill \end{array}$$

(26)

Using Equation (22), the following equation holds:

$$\begin{array}{cc}\hfill & \parallel {\rho }^{{\pi }_{{\theta }_t}}-{\rho }^{{\pi }_{{\theta }_{t-1}}}\parallel \le \frac{1}{\gamma -1}\sum _{k=0}^{\infty }{\gamma }^k\parallel d_{{\pi }_{{\theta }_t}}^k-d_{{\pi }_{{\theta }_{t-1}}}^k\parallel \hfill \\ \hfill & \le \frac{1}{\gamma -1}\sum _{k=0}^{\infty }{\gamma }^k·k·\epsilon =\frac{\epsilon ·\gamma }{1-\gamma }\hfill \end{array}$$

(27)

   □

Using this Lemma, the estimation error of the PPO-Clip could be obtained:

Theorem 1. 

Under the Assumption 1, the estimation error of PPO-Clip is satisfied:

$$\begin{array}{c}\hfill \mathrm{Err}\left[{\mathcal{L}}^{\mathrm{Clip},\theta }\right]=\mathrm{Err}\left[{\mathbb{E}}_{{\pi }_{{\theta }_{old}}}\left[\frac{{\pi }_{\theta }}{{\pi }_{{\theta }_{old}}}{A}_{{\pi }_{{\theta }_{old}}}\right]\right]\le \frac{\epsilon ·\gamma }{1-\gamma }{\mathbb{E}}_{s\sim {\mathrm{Unif}}_{\mathcal{S}},a\sim {\pi }_{\theta }}\left[A_{\pi }(s,a)\right]\end{array}$$

(28)

Proof. 

When $\parallel r_t-1\parallel \le \epsilon$, the surrogate of the PPO-Clip will be degraded [40]:

$$\begin{array}{c}\hfill {\mathcal{L}}^{\mathrm{Clip},\theta }={\mathbb{E}}_{{\pi }_{{\theta }_{old}}}\left[\frac{{\pi }_{\theta }}{{\pi }_{{\theta }_{old}}}{A}_{{\pi }_{{\theta }_{old}}}\right],\parallel \frac{{\pi }_{\theta }-{\pi }_{{\theta }_{old}}}{{\pi }_{{\theta }_{old}}}\parallel \le \epsilon \end{array}$$

(29)

The above surrogate is the importance sampling estimator of the objective of the new policy [44]:

$$\begin{array}{c}\hfill {\mathbb{E}}_{{\pi }_{{\theta }_{old}}}\left[\frac{{\pi }_{\theta }}{{\pi }_{{\theta }_{old}}}{A}_{\pi }(s,{\pi }_{{\theta }_{old}}\left(s\right))\right]\approx {\mathbb{E}}_{{\pi }_{\theta }}\left[A_{\pi }(s,{\pi }_{\theta }\left(s\right))\right]\end{array}$$

(30)

However, the estimator uses the data generated by ${\pi }_{{\theta }_{old}}$, and the state distribution of ${\mathcal{L}}^{\mathrm{Clip},\theta }$ is derived from ${\rho }^{{\pi }_{{\theta }_{old}}}$. Therefore, the estimation error is satisfied:

$$\begin{array}{cc}\hfill & \mathrm{Err}\left[{\mathbb{E}}_{{\pi }_{{\theta }_{old}}}\left[\frac{{\pi }_{\theta }}{{\pi }_{{\theta }_{old}}}{A}_{\pi }(s,{\pi }_{{\theta }_{old}}\left(s\right))\right]\right]=∥{\mathbb{E}}_{{\pi }_{{\theta }_{old}}}\left[\frac{{\pi }_{\theta }}{{\pi }_{{\theta }_{old}}}{A}_{\pi }(s,{\pi }_{{\theta }_{old}}\left(s\right))\right]-{\mathbb{E}}_{{\pi }_{\theta }}\left[A_{\pi }(s,{\pi }_{\theta }\left(s\right))\right]∥\hfill \\ \hfill & =∥{\int }_s{\rho }^{{\pi }_{{\theta }_{old}}}\left(s\right){\int }_{a\sim {\pi }_{{\theta }_{old}}}\frac{{\pi }_{\theta }\left(a\right|s)}{{\pi }_{{\theta }_{old}}\left(a\right|s)}{A}_{\pi }(s,a)dads-{\int }_s{\rho }^{{\pi }_{\theta }}\left(s\right){\int }_{a\sim {\pi }_{\theta }}{A}_{\pi }(s,a)dads∥\hfill \\ \hfill & \le {\int }_s∥{\rho }^{{\pi }_{{\theta }_{old}}}\left(s\right)-{\rho }^{{\pi }_{\theta }}\left(s\right)∥{\int }_{a\sim {\pi }_{\theta }}∥A_{\pi }(s,a)∥dads\hfill \end{array}$$

(31)

Consider the positive advantage situation and expand the integral of a; the following equation will hold:

$$\begin{array}{c}\hfill \mathrm{Err}\left[{\mathbb{E}}_{{\pi }_{{\theta }_{old}}}\left[\frac{{\pi }_{\theta }}{{\pi }_{{\theta }_{old}}}{A}_{\pi }(s,{\pi }_{{\theta }_{old}}\left(s\right))\right]\right]\le {\int }_s∥{\rho }^{{\pi }_{{\theta }_{old}}}\left(s\right)-{\rho }^{{\pi }_{\theta }}\left(s\right)∥{\int }_a{\pi }_{\theta }\left(a\right|s)A_{\pi }(s,a)dads\end{array}$$

(32)

Using the conclusion of Lemma 1, the following error bound could be obtained:

$$\begin{array}{cc}\hfill & \mathrm{Err}\left[{\mathbb{E}}_{{\pi }_{{\theta }_{old}}}\left[\frac{{\pi }_{\theta }}{{\pi }_{{\theta }_{old}}}{A}_{\pi }(s,{\pi }_{{\theta }_{old}}\left(s\right))\right]\right]\le {\int }_s\frac{\epsilon ·\gamma }{1-\gamma }{\int }_a{\pi }_{\theta }\left(a\right|s)A_{\pi }(s,a)dads\hfill \\ \hfill & ={\int }_s\frac{\epsilon ·\gamma }{1-\gamma }·\left|\mathcal{S}\right|·\frac{1}{\left|\mathcal{S}\right|}{\int }_a{\pi }_{\theta }\left(a\right|s)A_{\pi }(s,a)dads\hfill \\ \hfill & =\frac{\epsilon ·\gamma }{1-\gamma }·\left|\mathcal{S}\right|·{\int }_s\frac{1}{\left|\mathcal{S}\right|}{\int }_a{\pi }_{\theta }\left(a\right|s)A_{\pi }(s,a)dads\hfill \\ \hfill & =\frac{\epsilon ·\gamma }{1-\gamma }·\left|\mathcal{S}\right|·{\mathbb{E}}_{s\sim {\mathrm{Unif}}_{\mathcal{S}},a\sim {\pi }_{\theta }}\left[A_{\pi }(s,a)\right]\hfill \end{array}$$

(33)

where the ${\mathrm{Unif}}_{\mathcal{S}}$ represents the uniform distribution of the state.    □

Theorem 1 confirms the positive relationship between the estimation error and $\epsilon$. By using it, a more clear conclusion could be obtained:

Remark 1. 

In PPO-Clip, the high ε could enhance the exploration but will result in a high estimation error bound of the surrogate; the low ε could decrease the error bound but will restrict the exploration.

Therefore, to deal with the exploration and estimation error problems mentioned in Remark 1, this paper considers making the $\epsilon$ adaptive in different situations. The last part designed the sparse reward $R_d$, and the exploration reward $R_e$ is designed as the incentive reward. The agent should explore more in the task to receive a high-level $R_d$ and $R_e$. So, when these rewards are too low, the agent should release the restriction on $r_t$ to encourage the exploration. When these rewards are high and stable, the restriction on $r_t$ increases to ensure the estimation of the surrogate is accurate.

So, the exploration advantage function $A_{\pi }^{exp}(s_t,a_t)$ can be used to represent the advantage function that is estimated by $R_d$ and $R_e$, which can reflect the exploration ability of the agent:

$$\begin{array}{cc}\hfill & A_{\pi }^{exp}\left(S_t,a_t\right)=E_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k(R_d(t+k)+\sum _{i=1}^6{R}_{e,i}(t+k))\left|S_t=s,a_t=a\right.\right]-\hfill \\ \hfill & E_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k(R_d(t+k)+\sum _{i=1}^6{R}_{e,i}(t+k))\left|S_t=s\right.\right]\hfill \end{array}$$

(34)

According to the exploration function, an exploration PPO algorithm is proposed with an adaptive clip parameter $\epsilon$. When the exploration advantage function is lower than last time, to improve the exploration ability, $\epsilon$ will be enlarged. Otherwise, the $\epsilon$ will be reduced, restraining the updated policy in a trust region. To sum up, the adaptive mechanism is designed as follows:

$$\begin{array}{c}\hfill \epsilon \left(t\right)=\left\{\begin{array}{c}\epsilon (t-1)-clip(\frac{A_{{\pi }_{{\theta }_t}}^{exp}-A_{{\pi }_{{\theta }_{t-1}}}^{exp}}{A_{{\pi }_{{\theta }_{t-1}}}^{exp}},0,\frac{\epsilon (t-1)}{2});A_{{\pi }_{{\theta }_t}}^{exp}-A_{{\pi }_{{\theta }_{t-1}}}^{exp}>0\\ \epsilon (t-1)+clip(\frac{A_{{\pi }_{{\theta }_t}}^{exp}-A_{{\pi }_{{\theta }_{t-1}}}^{exp}}{A_{{\pi }_{{\theta }_{t-1}}}^{exp}},0,\frac{\epsilon (t-1)}{2});A_{{\pi }_{{\theta }_t}}^{exp}-A_{{\pi }_{{\theta }_{t-1}}}^{exp}<0\\ \epsilon (t-1);otherwise\end{array}\right.\end{array}$$

(35)

The clip function in the above equations is to restrict the adaptive mechanism and avoid the $\epsilon$ being abnormal. Through the variation of the exploration advantage function, the exploration-based adaptive $\epsilon$ mechanism is proposed. When simply replacing the constant $\epsilon$ with the adaptive $\epsilon$, the PPO will be PPO-Exploration $\epsilon$(PPO-Exp). With the restriction of old policies, new policies will be adjusted automatically. The surrogate of the PPO-Exp is as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}^{Exp,\theta }={\mathbb{E}}_{{\pi }_{{\theta }_{old}}}\left[min\left(r_t\left(\theta \right)A_{{\pi }_{{\theta }_{old}}},clip\left(r_t\left(\theta \right),1-\epsilon \left(t\right),1+\epsilon \left(t\right)\right)A_{{\pi }_{{\theta }_{old}}}\right)\right]\end{array}$$

(36)

The Algorithm of PPO-Exp in the formation environment could be seen in Algorithm 1. The exploration and estimation error problem in PPO-Exp could be adapted without delay, and the following Proposition will give the exploration range and the estimation error decrease rate in different situations:

Algorithm 1 PPO-Exploration $\epsilon$ with formation keeping task.

Initialize ${\pi }_0$,${\varphi }_0$. for$i=0,1,2,\cdots$, N do     for $t=1,\cdots ,T$ do         The leader0 collects state information $\left\{s_{t,i}\right|i=1,\cdots ,5\}$ through the communication protocol (Figure 3a)         Run policy ${\pi }_{\theta }$, obtain the action $\left\{a_{t,i}\right|i=0,1,\cdots ,5\}$, and send them using the control protocol (Figure 3b).         The leader and followers execute the action commands and receive a reward as follows: $(R_f\left(t\right),R_e\left(t\right),R_d\left(t\right),R_p\left(t\right))$         Store $(s_t,a_t,s_{t+1},R_t)$ at the buffer.     end for Transitions data from buffer, and estimate ${\widehat{A}}_{{\pi }_{{\theta }_t}}$, ${\widehat{A}}_{{\pi }_{{\theta }_t}}^{exp}$, respectively.     if ${\widehat{A}}_{{\pi }_{{\theta }_t}}^{exp}-{\widehat{A}}_{{\pi }_{{\theta }_{t-1}}}^{exp}>0$ then         $\epsilon \left(t\right)=\epsilon (t-1)-clip(\frac{{\widehat{A}}_{{\pi }_{{\theta }_t}}^{exp}-{\widehat{A}}_{{\pi }_{{\theta }_{t-1}}}^{exp}}{{\widehat{A}}_{{\pi }_{{\theta }_{t-1}}}^{exp}},0,\frac{\epsilon (t-1)}{2})$     end if     if ${\widehat{A}}_{{\pi }_{{\theta }_t}}^{exp}-{\widehat{A}}_{{\pi }_{{\theta }_{t-1}}}^{exp}<0$ then         $\epsilon \left(t\right)=\epsilon (t-1)+clip(\frac{{\widehat{A}}_{{\pi }_{{\theta }_t}}^{exp}-{\widehat{A}}_{{\pi }_{{\theta }_{t-1}}}^{exp}}{{\widehat{A}}_{{\pi }_{{\theta }_{t-1}}}^{exp}},0,\frac{\epsilon (t-1)}{2})$     end if     for $j=1,\cdots ,M$ do         $\widehat{{\mathcal{L}}_{\theta }}={\sum }_{t=1}^{T}min(r_t·{\widehat{A}}_{{\pi }_{{\theta }_t}},clip(1-\epsilon ,1+\epsilon ,r){\widehat{A}}_{{\pi }_{{\theta }_t}})$         Update $\theta$ by SGD or Adam.     end for     Update critic network parameter ${\varphi }_t$ by minimizing:     ${\sum }_{k=1}^T{({\sum }_{t^{\prime }>k}{\gamma }^{t^{\prime }-t}{R}_t-V_{\varphi }\left(s_t\right))}^2$ end for

Proposition 1. 

In PPO-Exp, when ${\widehat{A}}_{{\pi }_{{\theta }_t}}^{exp}-{\widehat{A}}_{{\pi }_{{\theta }_{t-1}}}^{exp}<0$, the exploration range of next policy will be expanded to $\frac{\parallel {\pi }_{{\theta }_t}-{\pi }_{{\theta }_{t-1}}\parallel }{\parallel {\pi }_{{\theta }_{t-1}}\parallel }\le \epsilon +\frac{{\widehat{A}}_t^{exp}-{\widehat{A}}_{t-1}^{exp}}{{\widehat{A}}_{t-1}^{exp}}\le \frac{3\epsilon (t-1)}{2}$; when ${\widehat{A}}_{{\pi }_{{\theta }_t}}^{exp}-{\widehat{A}}_{{\pi }_{{\theta }_{t-1}}}^{exp}>0$, in next update, the error bound of the surrogate will decrease to $O(\frac{\epsilon (t-1)}{2})$.

Proof. 

When ${\widehat{A}}_{{\pi }_{{\theta }_t}}^{exp}-{\widehat{A}}_{{\pi }_{{\theta }_{t-1}}}^{exp}<0$, according to Equation (35), it is easy to see the next policy will be expanded to $\frac{\parallel {\pi }_{{\theta }_t}-{\pi }_{{\theta }_{t-1}}\parallel }{\parallel {\pi }_{{\theta }_{t-1}}\parallel }\le \epsilon +clip(\frac{{\widehat{A}}_t^{exp}-{\widehat{A}}_{t-1}^{exp}}{{\widehat{A}}_{t-1}^{exp}},0,\frac{\epsilon (t-1)}{2})$. Then, the following inequality will hold:

$$\begin{array}{c}\hfill 0\le clip(\frac{{\widehat{A}}_t^{exp}-{\widehat{A}}_{t-1}^{exp}}{{\widehat{A}}_{t-1}^{exp}},0,\frac{\epsilon (t-1)}{2})\le \frac{\epsilon (t-1)}{2}\end{array}$$

(37)

So, the following inequality is held:

$$\begin{array}{c}\hfill \frac{\parallel {\pi }_{{\theta }_t}-{\pi }_{{\theta }_{t-1}}\parallel }{\parallel {\pi }_{{\theta }_{t-1}}\parallel }\le \epsilon +\frac{\epsilon (t-1)}{2}=\frac{3\epsilon (t-1)}{2}\end{array}$$

(38)

When ${\widehat{A}}_{{\pi }_{{\theta }_t}}^{exp}-{\widehat{A}}_{{\pi }_{{\theta }_{t-1}}}^{exp}>0$, and Assumption 1 is satisfied, it is obvious that the conclusion of Theorem 1 could be used in PPO-Exp. So, using Equation (35) and Theorem 1, the PPO-Exp’s decrease rate of the bound is as follows:

$$\begin{array}{cc}\hfill & \Delta \mathrm{Err}\left[{\mathcal{L}}_{\theta }^{\mathrm{Exp}}\right]=\mathrm{Err}\left[{\mathbb{E}}_{{\pi }_{{\theta }_{t-1}}}\left[\frac{{\pi }_{{\theta }_t}}{{\pi }_{{\theta }_{t-1}}}{A}_{{\pi }_{{\theta }_{t-1}}}\right]\right]-\mathrm{Err}\left[{\mathbb{E}}_{{\pi }_{{\theta }_{t-2}}}\left[\frac{{\pi }_{{\theta }_{t-1}}}{{\pi }_{{\theta }_{t-2}}}{A}_{{\pi }_{{\theta }_{t-2}}}\right]\right]\hfill \\ \hfill & \le \gamma \frac{\epsilon \left(t\right)-\epsilon (t-1)}{1-\gamma }·\left|\mathcal{S}\right|·∥{\mathbb{E}}_{s\sim {\mathrm{Unif}}_{\mathcal{S}},a\sim {\pi }_{{\theta }_t}}\left[A_{\pi }(s,a)\right]-{\mathbb{E}}_{s\sim {\mathrm{Unif}}_{\mathcal{S}},a\sim {\pi }_{{\theta }_{t-1}}}\left[A_{\pi }(s,a)\right]∥\hfill \\ \hfill & \le \gamma \frac{\epsilon \left(t\right)-\epsilon (t-1)}{1-\gamma }·\left|\mathcal{S}\right|·\Gamma \hfill \\ \hfill & \le \gamma \frac{(\epsilon (t-1)+clip(\frac{{\widehat{A}}_t^{exp}-{\widehat{A}}_{t-1}^{exp}}{{\widehat{A}}_{t-1}^{exp}},0,\frac{\epsilon (t-1)}{2}))-\epsilon (t-1)}{1-\gamma }·\left|\mathcal{S}\right|·\Gamma \hfill \\ \hfill & \le \gamma \frac{\frac{3\epsilon (t-1)}{2}}{1-\gamma }·\left|\mathcal{S}\right|·\Gamma =O(\frac{\epsilon (t-1)}{2})\hfill \end{array}$$

(39)

where the $\Gamma$ is the upper bound of advantage:

$$\begin{array}{c}\hfill \Gamma =\underset{\forall t}{max}∥{\mathbb{E}}_{s\sim {\mathrm{Unif}}_{\mathcal{S}},a\sim {\pi }_{{\theta }_t}}\left[A_{\pi }(s,a)\right]-{\mathbb{E}}_{s\sim {\mathrm{Unif}}_{\mathcal{S}},a\sim {\pi }_{{\theta }_{t-1}}}\left[A_{\pi }(s,a)\right]∥\end{array}$$

(40)

□

Proposition 1 indicates that the PPO-Exp could encourage the agents to adjust the exploration in different situations. The next section will validate it through numerical experiments.

6. Numerical Experiments

This section compares the PPO-Exp with four common reinforcement learning algorithms (PPO-Clip, PPO-KL, TD3, DDPG) in the formation-keeping task, and compared the performace of PPO-Exp and PPO-Clip in the formation changing task and obstacle avoidance task.

6.1. Experimental Setup

In terms of hardware, all the experiments are completed on the Windows 10 (64-bit) operating system, Intel(R) Core i7 processor, 16 GB memory, and 4 GB video memory. As for software, OpenAI-gym [45] is used to design the reinforcement learning environment and the physics rulers of the UAVs’ formation.

The formation task is modeled on the OpenAI gym environment. See Figure 1; the position of the leader and followers can be seen in

Table 1. The initial position of UAVs’ formation.

	Leader0	Follower1	Follower2	Follower3	Follower4	Follower5
Position X	160	190	220	130	100	160
Position Y	190	160	100	160	100	130

. The formation is updated by the dynamic equations solved by the difference method per 0.5 s per time mesh grid. The environment noises are set as $N(0,1)$ default. The target area is designed as a circle at $(200,400)$ with a radius of 40.

6.2. Experiments on PPO-Exploration $\epsilon$

The following famous continuous space RL algorithms are explored in this section: TD3, DDPG, PPO-KL, and PPO-clip; they are compared to the proposed method under the formation-keeping task.

• PPO-Clip [40]: Proximal Policy Optimization with Clip(PPO-Clip) function.
• PPO-KL [40]: Proximal Policy Optimization with KL-divergence(PPO-KL) constrain.
• DDPG [46]: Deep Deterministic Policy Gradient(DDPG) algorithm, which is a continuous action deep reinforcement learning algorithm that uses Actor–Critic architecture. In DDPG, the deterministic policy gradient is used to update the Actor parameter.
• TD3 [47]: Twin Delayed Deep Deterministic (TD3) policy gradient algorithm, which is a variant of DDPG. The TD3 introduced the delaying policy updates mechanism and the double network architecture to manage the per-update error and overestimation bias in DDPG.

The main hyperparameters of the contrast experiment are shown in

Table 2. The main hyperparameters of the algorithm used in the experiment.

Parameter Name	TD3	DDPG	PPO-KL	PPO-Clip	PPO-Exp
$\gamma$	0.9	0.9	0.9	0.9	0.9
$A_{LR}$	0.00005	0.00005	0.00005	0.00005	0.00005
$C_{LR}$	0.0002	0.0002	0.0002	0.0002	0.0002
Batch	32	32	32	32	32
$A_{US}$			10	10	10
$C_{US}$			10	10	10
EPS			${10}^{-8}$	${10}^{-8}$	${10}^{-8}$
$D_{KL}\left(target\right)$			0.01
$\lambda$			0.5
${\epsilon }_{clip}$			0.1	0.1	0.1
${\tau }_{DDPG}$		0.01
$VA{R}_{DDPG}$		3
Explore Step	500
$di{m}_{HIDDEN}$	32

. The blank area in the above table means the algorithm does not include this parameter.

Set the episode length be 200; the results of PPO-Exploration $\epsilon$ and other comparing algorithms are shown in Figure 7a. As the learning curves indicated, the PPO series methods achieved better performance; in all variations of PPO, the PPO-Exp has the best performance. It is validated that the adaptive mechanism based on exploration makes sense during policy updating. Figure 7b shows the change of $\epsilon$; the series $\epsilon \left(t\right)$ is stationary, and varies around 0.05, although the initial value is 0.1, which means 0.05 is the balance point between exploration and exploitation found by PPO-Exp. Meanwhile, the episode reward curve of PPO-Exp is higher than PPO-Clip’s, validating the idea that exploration from PPO-Exp is efficient.

6.3. Experiments on Formation Keeping

Only the learning curve was unable to declare whether the algorithm works well, so the trained PPO-Exp is used to perform 200s; the formation track can be seen in Figure 8. In this way, there is only a slight distortion in the formation, indicating that PPO-Exp can perform better in real tasks than PPO-Clip.

Furthermore, to evaluate the results, we plotted the heading $\psi$ and the velocity v during 200 s in Figure 9. Figure 9a shows that followers 1, 4, and 5 are approaching gradually as time goes on. Followers 2, 3 and the leader, have no such trend to converge gradually; however, all the heading deviations are no more than ${10}^{\circ }$. In Figure 9b, the velocity of each UAV is shown. The velocities of followers 1, 3, 4, and 5 diverge a little and then converge. Corresponding to Figure 9a, followers 1, 4, and 5 are closer in terms of the value of velocity and heading; the leader and follower 2 are far away from these followers, but the velocity difference is not more than 1.5 m/s as well. This inspired us to design the reward based on the velocity and heading.

To illustrate the influence of environmental noise on formation keeping, the results show the formation track with no control in Figure 2a. To verify that the proposed centralized method saves time, this section further compares the decentralized version of PPO-Exp: PPO-Exp-Dec, which, similar to MAPPO, needs all six UAV agents to learn the control policy at the same time.

To validate that the protocol can reduce the communication cost and avoid placing the UAVs out of the communication range, this section also compares the protocol-free version: PPO-exp-pro. The results can be seen in

Table 3. The experimental results in different algorithms.

Algorithm	$\mathsf{\Gamma }$	T	${\mathit{r}}_{\mathit{coll}}\mathbf{(}\mathbf{\%}\mathbf{)}$	${\mathit{r}}_{\mathit{fail}}\mathbf{(}\mathbf{\%}\mathbf{)}$
PPO-exp	−19,197.2 $\pm \mathbf{1307}.\mathbf{4}$	$2.19\pm 0.04$	0.93 ± 0.01	$\mathbf{0}.\mathbf{32}\pm \mathbf{0}.\mathbf{02}$
PPO-exp-dec	−20,374.7 $\pm 1926.4$	$10.06\pm 0.08$	$1.01\pm 0.02$	$0.35\pm 0.01$
PPO-exp-pro	−23,001.3 $\pm 2507.2$	$2.43\pm 0.03$	$0.98\pm 0.03$	$12.48\pm 1.76$
PPO-clip	−20,305.7 $\pm 1588.6$	$2.14\pm 0.06$	$0.97\pm 0.02$	$0.94\pm 0.03$
Greedy	−39,074.5 $\pm 3806.5$	$\mathbf{1}.\mathbf{15}\pm \mathbf{0}.\mathbf{04}$	$12.32\pm 1.32$	$10.56\pm 0.65$

. $\Gamma$ represents the episode reward, T represents time per episode, $r_{col}$ and $r_{fai}$ represents the collision rate and failure to communicate rate, respectively.

To further verify the effectiveness of the proposed method, ablation experiments are performed (see Figure 2a,b and Figure 8b). Figure 8b shows the trained PPO-clip without the exploration mechanism. Although there is no UAV crash, the leader and follower3 are very close, and the formation is not as orderly as the PPO-Exp. Figure 2a shows the result of no action taken, where the UAVs will crash, and the formation will break up. Figure 2b shows the trained PPO-clip with $\epsilon =0.05$, which is the balance point in the PPO-Exp. However, the experimental result shows it performs worse; there is one follower that loses communication with leader, and one follower almost crashes with the leader. The result illustrates that the PPO-Exp with adaptive $\epsilon$ is better than the PPO-Clip with a good $\epsilon$. In summary, the ablation experiments also indicated that PPO-Exp performs better than other algorithms in terms of learning curves and the real-task.

6.4. Experiment on More Complex Tasks

To further show the efficiency of PPO-Exp in fixed-wing UAV formation keeping, this part design two more complex scenarios: formation changing and obstacle avoidance task, the UAV formation perform 120 s on each task. This part mainly compared the performance of PPO-Exp and PPO-Clip on these tasks.

The goal for the formation changing task is changing the formation shown in Figure 1 to the vertical formation. The vertical formation also expects the differences between leader and followers are as small as possible in coordinates on the x-axis. For guiding the followers to change the formation, this paper utilizes the absolute difference value of x coordinates to modify the flocking reward. The modified flocking reward (9) and (12) could be represented as follows:

$$\begin{array}{c}\hfill R_{f,i}=\parallel x_0-x_i\parallel ,\forall i=1,\cdots ,5\end{array}$$

(41)

Then the total reward (16) can be rewritten as follows:

$$\begin{array}{c}\hfill R\left(T\right)=\sum _{i=0}^5\left[\parallel x_0\left(T\right)-x_i\left(T\right)\parallel +R_{e,i}\left(T\right)\right]+R_d\left(T\right)+R_p\left(T\right)\end{array}$$

(42)

where the $x_0\left(T\right),x_i\left(T\right)$ represent the x coordinates of leader and ith follower at time T, respectively. To encourage the UAV system to take more exploration on forming new formation, the flocking reward is added to the exploration advantage function:

$$\begin{array}{cc}\hfill & A_{\pi }^{exp}\left(S_t,a_t\right)=E_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k(R_d(t+k)+\sum _{i=0}^5\left[\parallel x_0\left(T\right)-x_i\left(T\right)\parallel +R_{e,i}(t+k))\right]\left|S_t=s,a_t=a\right.\right]-\hfill \\ \hfill & E_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k(R_d(t+k)+\sum _{i=0}^5[\parallel x_0\left(T\right)-x_i\left(T\right)\parallel +R_{e,i}(t+k)\left)\right]\left|S_t=s\right.\right]\hfill \end{array}$$

(43)

Training the task with PPO-Exp and PPO-Clip, the training parameters are kept as same as in the previous part except episode length. After training, the test result of PPO-Exp is shown in Figure 10a, and the PPO-Clip is shown in Figure 10b. To evaluate the performance, this paper draws the plots of the x coordinates and timesteps of the leader and followers in Figure 10c,d. The closer the x coordinates of followers to that of the leader, the better the performance will be. The x coordinates of followers in (c) converge to the leader faster than (d), representing that PPO-Exp can change vertical formation faster than PPO-Clip.

To further evaluate the formed vertical formation. Denote the terminal time as $t_{ter}$, calculate the average difference between the followers and leader in x coordinates in the last ten timesteps, and denote the result as ${\delta }_x$, which can be represented as follows:

$$\begin{array}{c}\hfill {\delta }_x=\frac{1}{5}\sum _{i=1}^5\sum _{t>t_{ter}-10}\parallel x_0\left(t\right)-x_i\left(t\right)\parallel \end{array}$$

(44)

The low ${\delta }_x$ indicates the follower is close to the leader in x coordinates. In PPO-Clip, the calculated ${\delta }_x\approx 95.383$, but in PPO-Exp, the calculated ${\delta }_x\approx 43.816$, which is nearly half of the PPO-Clip.

Compared to the control strategy in formation keeping, the followers in formation changing tasks perform good cooperation. All followers maneuver orderly to the position where the leader’s x-coordinate is located. To avoid the UAVs collide each other, the followers decided to move to different positions on the y-axis. The followers take different maneuvers depending on their initial position to reach the position. e.g., follower 4, in the initial time, is far away from the leader in x-coordinates. For follower 4, a collision avoidance path is moving to the tail of the newly formed formation. Therefore, the follower4 achieves a large angle arc maneuver and moves to the tail of the formed vertical formation.

The target of the obstacle avoidance task is to reach the target area and avoid crashing into the obstacle. This paper considers a circle area on the plane as an obstacle. Denote the coordinates of the obstacle center is $(x_{obs},y_{obs})$, and the radius is $r_{obs}$. A simple approach to consider this situation is to add a penalty on the formation system reward when the UAVs crash on the obstacle, the penalty effect. The penalty for crashing into the obstacle is denoted as follows:

$$\begin{array}{c}\hfill R_{o,i}=\left\{\begin{array}{c}0,\sqrt{{(x_i-x_{obs})}^2+{(y_i-y_{obs})}^2}\le r_{obs}\\ -10,000,otherwise\end{array}\right.\end{array}$$

(45)

Similar to the exploration reward $R_{e,i}$, to

$$\begin{array}{c}\hfill R_{e,i}^{obs}=min\left\{\right|x_i-x_{obs}|,|y_i-y_{obs}\left|\right\}\end{array}$$

(46)

Then the total reward (16) can be rewritten as follows:

$$\begin{array}{c}\hfill R\left(T\right)=\sum _{i=0}^5\left[R_{f,i}\left(T\right)+R_{e,i}\left(T\right)+R_{e,i}^{obs}\left(T\right)+R_{o,i}\left(T\right)\right]+R_d\left(T\right)+R_p\left(T\right)\end{array}$$

(47)

To encourage the UAV system to take more exploration on avoid obstacle, the exploration reward in avoid obstacle $R_{r,i}^{obs}$ is added to the exploration advantage function:

$$\begin{array}{cc}\hfill & A_{\pi }^{exp}\left(S_t,a_t\right)=E_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k(R_d(t+k)+\sum _{i=0}^5\left[R_{f,i}\left(T\right)+R_{e,i}(t+k)+R_{e,i}^{obs}\left(T\right))\right]\left|S_t=s,a_t=a\right.\right]-\hfill \\ \hfill & E_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k(R_d(t+k)+\sum _{i=0}^5[R_{f,i}\left(T\right)+R_{e,i}(t+k)+R_{e,i}^{obs}\left(T\right))]\left|S_t=s\right.\right]\hfill \end{array}$$

(48)

Training the obstacle to avoid task with PPO-Exp and PPO-Clip, the training parameters are kept as same as the previous part except episode length. After training with PPO-Exp and PPO-Clip, the test results of obstacle avoid task are shown in Figure 11a, and the results of PPO-Clip can be seen in Figure 11b. A follower in the formation trained by PPO-Clip crashed on the obstacle at 94 timesteps. The formation trained by PPO-Exp performed the arc maneuvers and avoided the obstacle. PPO-Exp performs better than PPO-Clip because it can explore more policies to reach the target area and discover a good path to avoid obstacles. However, the PPO-Clip still tries to reach the target area straight.

Compared to the formation keeping task without obstacles, the obstacle scenario requires the formation system to explore more to avoid the obstacle. Therefore, in this scenario, compared to the fixed $\epsilon$ PPO-Clip, the PPO-Exp shows better performance because it could adjust their $\epsilon$ to balance exploration and estimation error. Then the PPO-Exp explored the large-angle arc maneuvers and performed them to avoid the obstacle.

7. Conclusions

This paper studies a flocking scenario consistent with one leader (with an intelligence chip) and several followers(without an intelligence chip). The reinforcement learning environment is constructed (continuous action and state space) with an OpenAI gym, and the reward is designed as a regular part and an exploration part. A low-communication cost protocol is provided to ensure the UAVs can communicate the state and action information between leader and followers. In addition, a variation of Proximal Policy Optimization is proposed to balance the dilemma between the estimation error bound and the exploration ability of PPO. The proposed method can help UAVs adjust the explore strategy, and the experiments demonstrate it has better performance than the current algorithms such as PPO-KL, PPO-clip, and DDPG.