Safe Reinforcement Learning for Transition Control of Ducted-Fan UAVs

Abstract

Ducted-fan tail-sitter unmanned aerial vehicles (UAVs) provide versatility and unique benefits, attracting significant attention in various applications. This study focuses on developing a safe reinforcement learning method for back-transition control between level flight mode and hover mode for ducted-fan tail-sitter UAVs. Our method enables transition control with a minimal altitude change and transition time while adhering to the velocity constraint. We employ the Trust Region Policy Optimization, Proximal Policy Optimization with Lagrangian, and Constrained Policy Optimization (CPO) algorithms for controller training, showcasing the superiority of the CPO algorithm and the necessity of the velocity constraint. The transition trajectory achieved using the CPO algorithm closely resembles the optimal trajectory obtained via the well-known GPOPS-II software with the SNOPT solver. Meanwhile, the CPO algorithm also exhibits strong robustness under unknown perturbations of UAV model parameters and wind disturbance.

1. Introduction

In recent years, vertical take-off and landing (VTOL) unmanned aerial vehicles (UAVs) have gained considerable attention due to their unique advantages. Distinct from traditional fixed-wing UAVs, VTOL UAVs are capable of taking off and landing vertically, eliminating the need for a runway [1]. Moreover, when compared to multi-rotor UAVs, they offer several benefits, such as a larger payload capacity, higher cruise speed, and longer flight ranges [2]. This is primarily attributed to their reliance on wings for lift production, as opposed to the multiple rotors utilized by multi-rotor UAVs. Among the various VTOL UAV configurations, such as tilt rotor, tail sitter, and vectored thrust, the ducted-fan tail-sitter fixed-wing UAV stands out as a unique design. Notably, they eliminate the need for additional moving parts to achieve VTOL capabilities, leading to a simplified mechanical design that is both easier to maintain and less prone to mechanical failures [3]. Moreover, the incorporation of a ducted fan enhances the propulsion efficiency, allowing the UAVs to cover longer distances. The ducted fan also contributes to noise reduction, making these UAVs particularly suitable for noise-sensitive environments [4].

Ducted-fan tail-sitter fixed-wing UAVs integrate the characteristics of both traditional fixed-wing and multi-rotor UAVs, enabling them to perform level flight and hover operations. However, during the transition process, the UAV experiences aerodynamic instability due to a stall occurring at the fixed wing [5]. As a result, the control strategies for transition maneuvers between these two flight modes are of paramount importance, necessitating extensive investigation and research. Generally, two prevalent methods to address this issue can be found in the current literature [3].

The first approach conceptualizes the transition process as a trajectory optimization problem. In [6], the direct collocation method was employed to obtain the optimal transition trajectory by Li et al. An optimization approach using the interior point method was proposed that focused on altitude changes during transition in [7]. The optimal feed-forward control input for transition was computed via sequential quadratic programming by Kubo and Suzuki [8]. Banazadeh devised a gradient-based algorithm based on the classical Cauchy method to generate optimal transition trajectories [9]. Naldi and Marconi [10] utilized mixed-integer nonlinear programming to tackle the minimum-time and minimum-energy optimal transition problems.

The second approach is structured in two stages: the first stage entails devising the desired trajectory during the transition, and the second stage involves designing a controller to track the trajectory established in the first stage [3]. Jeong et al. proposed a continuous ascent transition trajectory consisting of the angle of attack and flight path angle and invoked dynamic inversion control for tracking [11]. Flores provided a desired velocity trajectory and implemented a recurrent neural network-based controller for feedback linearization [12]. Cheng and Pei established a transition corridor based on the constraint of maintaining a fixed altitude [13]. They planned the desired velocity in the corridor and utilized an adaptive controller [14].

Alternatively, reinforcement learning (RL) has emerged as a promising approach to address various challenges in the domain of UAVs, owing to its ability to learn and adapt to dynamic environments [15]. As a result, there has been growing interest in leveraging RL algorithms to tackle the transition problem of VTOL UAVs. In [16], an RL-based controller for hybrid UAVs was designed that can not only automatically complete the transition but can also be adapted to different configurations. Xu [17] proposed a soft landing control algorithm based on the RL method. Yuksek employed the deep deterministic policy gradient algorithm to address the transition flight problem for tilt-rotor UAVs [18].

In this paper, we solve the back-transition task of ducted-fan tail-sitter fixed-wing UAVs (i.e., from level flight mode to hover mode) using safe RL algorithms. While prior work [16] mainly emphasizes the successful execution of transition maneuvers, our focus is on minimizing altitude changes and the transition time, while adhering to velocity constraints. Our method, compared to [18], integrates trajectory optimization and control problems into an RL-based learning process, thereby reducing the complexity and computational effort. To the best of our knowledge, this is one of the first works in which the RL methodology is utilized to solve the back-transition control problem of ducted-fan tail-sitter UAVs.

The main contributions of our work are as follows.

1.

We develop a mathematical model of ducted-fan fixed-wing UAV dynamics. Based on this model, we create a training environment for ducted-fan UAVs in OpenAI GYM [19] using the fourth-order Runge–Kutta method.

2.

Taking into account the velocity constraint during the transition process, we train controllers using Trust Region Policy Optimization (TRPO) with fixed penalty, Proximal Policy Optimization with Lagrangian (PPOLag), and Constrained Policy Optimization (CPO). We assess the performance of these algorithms and demonstrate the superiority of the CPO algorithm.

3.

Comparing the CPO algorithm with the optimal trajectory obtained via GPOPS-II [20], we find that the performance of CPO closely approximates the optimal trajectory. In addition, the CPO algorithm has robustness under unknown perturbations of UAV model parameters and wind disturbance, which is lacking in the GPOPS-II software.

This paper is organized as follows. In Section 2, a mathematical model of a ducted-fan fixed-wing UAV is described. In Section 3, the general structure of the RL transition controller is introduced, and the reward function, action, and observation are explained. In Section 4, comparisons between CPO and other RL algorithms are reported. We also compare the transition trajectory of CPO with GPOPS-II and verify the robustness of the CPO algorithm. In Section 5, concluding remarks and future works are reported.

2. Mathematical Modeling

In this section, we describe a three-degree-of-freedom (DOF) longitudinal model for ducted-fan fixed-wing UAVs. The model is derived from the Newton and Euler theorems and is simplified from the full 6-DOF dynamic model. This 3-DOF model is able to speed up the process of assessing the impact of different reward functions and hyper-parameter settings compared to the 6-DOF dynamic model [21]. It is equipped with four groups of control vanes as the main control surfaces, each consisting of three movable vanes. In addition, four groups of fixed vanes are situated above the main control surfaces, intended to balance the anti-torque generated by the rotor, with each fixed group consisting of two fixed vanes (see Figure 1). The four groups of control vanes are numbered 1, 2, 3, and 4 and are employed to change the attitude of the UAVs (see Figure 2). Each group of three movable vanes, controlled by a single servo, deflects by the same angle. We use ${\delta }_1$, ${\delta }_2$, ${\delta }_3$, and ${\delta }_4$ to represent the deflection of group numbers 1, 2, 3, and 4.

2.1. Three-DOF Dynamics

For the ducted-fan VTOL UAV, two right-handed coordinate systems are applied to describe the states of the aircraft (see Figure 3). The inertial frame axes are denoted as {${\Gamma }^I$:$X^I$,$Y^I$,$Z^I$} and the body frame axes are denoted as {${\Gamma }^B$:$X^B$,$Y^B$,$Z^B$} with the origin located at the center of mass. Throughout this section, the superscripts (${.}^I$) and (${.}^B$) are utilized to specify whether a variable is formulated in the inertial or body frame. The position of the aircraft in ${\Gamma }^I$ is described by $\xi ={\left[x,y,z\right]}^T$ and the velocity of the aircraft in ${\Gamma }^B$ is defined as $V^B={\left[u,v,w\right]}^T$. The Euler angle vector (i.e., roll, pitch, and yaw) is described by $\Theta ={\left[\varphi ,\theta ,\psi \right]}^T$ and the angular velocity vector with respect to the body frame is denoted by $\Omega ={\left[p,q,r\right]}^T$. It should be noted that based on the above-mentioned definition, the pitch angle $\theta =0^{\circ }$ at the level flight condition, and the pitch angle $\theta ={90}^{\circ }$ at the landing and hover conditions (see Figure 4). Thus, the longitudinal dynamics of the aircraft are derived as follows:

$$\left\{\begin{array}{cc}\hfill \dot{u}& =gsin\theta +\left(Dcos\alpha -T-Lsin\alpha \right)/m-qw\hfill \\ \hfill \dot{w}& =\left(F_{cw}+Lcos\alpha +Dsin\alpha +F_{cs}\right)/m+qu-gcos\theta \hfill \\ \hfill \dot{x}& =wcos\theta -usin\theta \hfill \\ \hfill \dot{z}& =ucos\theta +wsin\theta \hfill \\ \hfill \dot{\theta }& =q\hfill \\ \hfill \dot{q}& =\left(M_{pitch}+M_{cw}+M_{cs}\right)/I_{yy}\hfill \end{array}\right.$$

(1)

where g is gravity acceleration; $I_{yy}$ is the moment of inertia on the $Y^B$ axis; $\alpha$ represents the angle of attack. All the forces and moments are discussed below.

2.2. Rotor

In this subsection, the rotor model is discussed based on basic momentum theory and blade element theory [22]. Considering the airspeed along the $X^B$ axis, the configuration of the blades, the airflow through the rotor, and the thrust of the rotor can be expressed as [23]

$$\begin{array}{cc}\hfill v_b& =u+\frac{2}{3}{\omega }_{r}r\left(\frac{3}{4}{K}_{twist}\right)\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill T& =\frac{1}{4}\left(v_b-v_i\right){\omega }_r{r}^2{\rho }_{\infty }{a}_{0}b{c}_r\hfill \end{array}$$

(3)

where $V^B={\left[u,v,w\right]}^T$ is the velocity in the body frame, ${\rho }_{\infty }$ is the air density, ${\omega }_r$ represents the angular velocity of the rotor, r is the radius of the rotor and $K_{twist}$ is the twist of the blades, $a_0$ is the rotor lift curve slope, b is the number of blades, $c_r$ is the chord of the rotor blade, and the induced velocity $v_i$ and the far-field velocity $v_f$ can be expressed as

$$\begin{array}{cc}\hfill v_f& =\sqrt{{\left(u-v_i\right)}^2+w^2+v^2}\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill v_i& =T/2{\rho }_{\infty }\pi r^2{v}_f\hfill \end{array}$$

(5)

The expressions for T and $v_i$ can be solved iteratively through Equations (2)–(5) using the Newton–Raphson method.

2.3. Aerodynamics Model

The aerodynamic forces and moments include the lift L, the drag D, and the pitching moment $M_{pitch}$. They are primarily dependent on the fuselage, duct, and mostly on the wings. The angle of attack has a significant impact on the aerodynamic model, which can be expressed as follows:

$$\alpha =\left\{\begin{array}{c}\hfill 0\mathrm{if}w=0\\ \hfill -sgn\left(w\right)\frac{\pi }{2}\mathrm{if}u=0\\ \hfill arctan\left(\frac{w}{u}\right)\mathrm{if}u<0\\ \hfill arctan\left(\frac{w}{u}\right)+\pi \mathrm{if}u>0\end{array}\right.$$

(6)

The ducted-fan UAV can be regarded as an entire lifting body, including the fuselage, wings, and duct [15]. This simplification introduces some inaccuracies in the aerodynamic data. To address these discrepancies, it is essential to incorporate compensation for the disturbances within the aerodynamic data. In order to account for the modeling errors, the lift and drag coefficients are multiplied by a perturbation factor, sampled from a uniform distribution ranging between 0.8 and 1.2.

During the transition mode, high angle of attack conditions can induce wing stall phenomena, leading to a significant reduction in lift. Traditional linear aerodynamic coefficient models are insufficient in accurately capturing the complexities of the aerodynamic behavior. As demonstrated in [24], an advanced aerodynamic model that integrates both the linear lift model and the effects of wing stall can be formulated as follows:

$$\begin{array}{cc}\hfill C_L\left(\alpha \right)=& (1-\sigma \left(\alpha \right))\left[C_{L_0}+C_{L_{\alpha }}\alpha \right]\hfill \end{array}$$

$$\begin{array}{cc}\hfill & +\sigma \left(\alpha \right)\left[C_{L_{max}}sin\alpha cos\alpha \right]\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill C_D\left(\alpha \right)=& C_{D_p}+\frac{{\left(C_{L_0}+C_{L_{\alpha }}\alpha \right)}^2}{\pi eAR}\hfill \end{array}$$

(8)

where

$$\begin{array}{cc}\hfill \sigma \left(\alpha \right)=& \frac{1+e^{-M\left(\alpha -{\alpha }_0\right)}+e^{M\left(\alpha +{\alpha }_0\right)}}{\left(1+e^{-M\left(\alpha -{\alpha }_0\right)}\right)\left(1+e^{M\left(\alpha +{\alpha }_0\right)}\right)}\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill C_{L_{\alpha }}=& \frac{\pi AR}{1+\sqrt{1+{(AR/2)}^2}}\hfill \end{array}$$

(10)

The lift and drag coefficients are shown in Figure 5. The pitching moment coefficients will be expressed by the linear model as

$$\begin{array}{c}\hfill C_m\left(\alpha \right)=C_{m_0}+C_{m_{\alpha }}\alpha \end{array}$$

(11)

Thus, the lift L, drag D, and pitching moment $M_{pitch}$ can be written as

$$\begin{array}{cc}\hfill L& =\frac{1}{2}{\rho }_{\infty }{V}^{2}S{C}_L\left(\alpha \right)\hfill \\ \hfill D& =\frac{1}{2}{\rho }_{\infty }{V}^{2}S{C}_D\left(\alpha \right)\hfill \\ \hfill M_{pitch}& =\frac{1}{2}{\rho }_{\infty }{V}^{2}S{C}_m\left(\alpha \right)\hfill \end{array}$$

(12)

V is the air speed of the UAV and S is the wing area.

2.4. Momentum Drag

Due to the existence of crosswinds, the duct must generate a force to align the incoming airflow with its orientation. This results in a reaction force known as momentum drag. Moreover, crosswinds lead to the formation of a region with higher velocity over the near edge of the duct, as the air surrounding it is pulled into the duct by the rotor. This increased lift on the edge produces a moment that causes the vehicle to turn away from the crosswind, referred to as the momentum moment [22]. The formulas for the momentum drag and moment can be represented as follows:

$$\begin{array}{cc}\hfill F_{cw}& =-v_i{\rho }_{\infty }\pi r^{2}w\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill M_{cw}& =C_{duct}{\rho }_{\infty }r\left|r\right|w\hfill \end{array}$$

(14)

where w is the velocity of the Z-axis in the body frame, and $v_i$ is determined by Equation (5).

2.5. Control Vanes

Each control vane can be modeled as an airfoil. In [3,14,15], the lift slope coefficient of vane is assumed to be constant. However, our Computational Fluid Dynamics simulations have shown that the lift coefficient of the vane remains virtually unchanged when subjected to a large angle of attack. Thus, based on the simple model in [25], the lift slope coefficient $C_{lcs}$ of the vane can be expressed as

$$\begin{array}{c}\hfill C_{lcs}=\left\{\begin{array}{cc}0.0625\hfill & \mathrm{if}\left|\alpha \right|\le {16}^{\circ }\hfill \\ 0\hfill & \mathrm{if}\left|\alpha \right|>{16}^{\circ }\hfill \end{array}\right.\end{array}$$

(15)

The dynamic pressure $q_{cs}$ on each vane can be expressed as

$$\begin{array}{c}\hfill q_{\mathrm{cs}}=\frac{1}{2}{\rho }_{\infty }{\left(u-v_i\right)}^2\end{array}$$

(16)

where u is the velocity of the X-axis in the body frame. Based on the control allocation method [15], the equivalent vane deflection of pitch (Y-axis) is given as follows:

$$\begin{array}{c}\hfill {\delta }_y={\delta }_2-{\delta }_4\end{array}$$

(17)

The drag forces of vanes can be neglected [15], and the vane’s angle of attack only depends on the vane’s deflection [3]. Thus, the force and moment generated by the control vanes are as follows:

$$\begin{array}{cc}\hfill F_{cs}& =q_{cs}{S}_{cs}{C}_{lcs}{\delta }_y\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill M_{cs}& =q_{cs}{S}_{cs}{C}_{lcs}{l}_e{\delta }_y\hfill \end{array}$$

(19)

where $S_{cs}$ represents the vane area and $l_e$ is the arm of pitch moment.

3. Approach

3.1. Problem Formulation

In this section, we provide a mathematical expression for the UAV back-transition problem. In the back-transition process, the most straightforward strategy is that the UAV climbs at a large pitch angle while converting the kinetic energy into potential energy [26] (see Figure 6a). Although this approach offers a short transition time, it results in substantial altitude loss. In this paper, our desired trajectory, referred to as a “neat transition”, aims to complete the transition as quickly as possible with minimal altitude loss [13] (see Figure 6b). To achieve this transition, we introduce an additional constraint on the Z-axis velocity, expressed as

$$∥v_z∥<1$$

Thus, we pay attention to the altitude loss and transition time after the transition has been completed. The RL problem is defined as follows.

RL UAV Back-Transition Problem:

Minimize:

1.

Terminal Velocity Error: $\parallel v\parallel \mathrm{at}t=t_f$

2.

Terminal Attitude Error: $∥\theta -\pi /2∥\mathrm{at}t=t_f$

3.

Terminal Angular Velocity Error: $\parallel \omega \parallel \mathrm{at}t=t_f$

4.

Terminal Time: $t_f$

5.

Height loss:$∥z_{t_f}-z_{t_0}∥$

Subject to:

1.

Angular Velocity Constraint: pitch angle velocity rate $q<{180}^{\circ }$

2.

Control Constraints: $\left|{\delta }_y\right|<{30}^{\circ }$ and ${\omega }_{rmin}<{\omega }_r<{\omega }_{rmax}$

3.

Neat Transition Constraint: $∥v_z∥<1$ m/s

4.

Equations of Motion (set by the environment)

To primarily optimize the terminal altitude loss and transition time, we opt to provide error margins for the terminal velocity, terminal angle, and terminal angular velocity. When the error falls within these margins, the transition is deemed successful. The terminal error range can be expressed as follows:

$$\begin{array}{cc}\hfill \parallel v_{t_f}\parallel & \le 1\mathrm{m}/\mathrm{s}\hfill \\ \hfill ∥{\theta }_{t_f}-\pi /2∥& \le 0.08\mathrm{rad}\hfill \\ \hfill \parallel {\omega }_{t_f}\parallel & \le 0.08\mathrm{rad}/\mathrm{s}\hfill \end{array}$$

3.2. Algorithm

To address the neat transition constraint, traditional RL methods typically apply a fixed penalty through the reward function. However, if the penalty is too small, the agent may learn unsafe behavior, while an excessively severe penalty might result in the agent’s inability to learn anything. In contrast, safe reinforcement learning algorithms employ the Lagrangian approach to tackle constraints, automatically balancing the weights between rewards and penalties. This ensures that the entire transition process adheres to the constraints while exploring the optimal transition performance. Among safe RL algorithms, we introduce the Constrained Policy Optimization (CPO) algorithm to solve the ducted-fan UAV back-transition problem [27].

3.2.1. Constrained Markov Decision Process (CMDP)

A Markov decision process (MDP) is a tuple, $(S,A,P,\gamma ,R)$, where S is the set of states, A is the set of actions, and $R:S\times A\times S\to \mathbb{R}$ is the reward function. $P:S\times A\times S\to [0,1]$ is the transition probability function (where $P\left(s^{\prime }\mid s,a\right)$ is the probability of transitioning to state $s^{\prime }$ under the previous state s and the action a). $\gamma \in [0,1]$ is the discount factor for future rewards, which represents a trade-off between immediate and future rewards. $\pi$ refers to a policy, which is a function that maps a state to an action. $\pi \left(a\right|s)$ denotes the probability of selecting action a in state s. A trajectory $\tau$ is a set of MDP tuples in one episode during which an agent interacts with the environment under policy $\pi$.

A constrained Markov decision process (CMDP) is an MDP subjected to some constraints. A set $C_1,\dots ,C_m$ (with each one as $C_i:S\times A\times S\to \mathbb{R}$) is referred to as cost functions, similar to the reward function [28]. The limits $d_1,\dots ,d_m$ represent the thresholds of the cost functions. For the policy gradient algorithm, value functions, action-value functions, and advantage functions are defined as $V^{\pi }$, $Q^{\pi }$, and $A^{\pi }$.

$$\begin{array}{c}\hfill Q^{\pi }\left(s_t,a_t\right)={\mathbb{E}}_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+k}\right]\\ \hfill V^{\pi }\left(s_t\right)={\mathbb{E}}_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+k}\right]\\ \hfill A^{\pi }(s,a)=Q^{\pi }(s,a)-V^{\pi }\left(s\right)\end{array}$$

(20)

Similarly, we can replace reward R in the above equation with cost C to acquire $V_C^{\pi }$, $Q_C^{\pi }$, and $A_C^{\pi }$.

In a CMDP, the actor–critic framework consists of three parts: the actor, the critic, and the cost critic. The actor is responsible for selecting actions based on the current state of the agent, while the critic and cost critic estimate the expected discounted reward and cost based on the current state. Typically, these three parts are usually approximated by neural networks (NN). In the case of CMDP, let $J\left(\pi \right)$ denote the expected discounted reward:

$$\begin{array}{c}\hfill J\left(\pi \right)=\underset{\tau \sim \pi }{\mathrm{E}}\left[\sum _{t=0}^{\infty }{\gamma }^t{R}_t\right]\end{array}$$

(21)

Similarly, the cumulative cost can be described as

$$\begin{array}{c}\hfill J_{C_i}\left(\pi \right)=\underset{\tau \sim \pi }{\mathrm{E}}\left[\sum _{t=0}^{\infty }{\gamma }^t{C}_i\left(s_t,a_t,s_{t+1}\right)\right]\end{array}$$

(22)

Thus, the set of feasible policies is

$$\begin{array}{c}\hfill {\Pi }_C\doteq \left\{\pi \in \Pi :\forall i,J_{C_i}\left(\pi \right)\le d_i\right\}\end{array}$$

(23)

The objective in a CMDP is to find a policy that maximizes $J\left(\pi \right)$ in ${\Pi }_C$:

$$\begin{array}{cc}\hfill {\pi }^{*}& =arg\underset{\pi \in {\Pi }_C}{max}J\left(\pi \right)\hfill \end{array}$$

(24)

3.2.2. Constrained Policy Optimization

CPO adheres to the monotonic improvement theory proposed by Trust Region Policy Optimization (TRPO) [29]. By constraining the difference between the old and new strategies to a small step size, it becomes possible to approximate the lower bound of the objective function and the upper bound of the cost objective function between the old and new policies, resulting in the following expression [27]:

$$\begin{array}{c}\hfill J\left({\pi }^{\prime }\right)-J\left(\pi \right)\approx {\mathbb{E}}_{\tau \sim \pi }\left[\sum _{t=0}^{\infty }{A}^{\pi }\left(s_t,a_t\right)\frac{{\pi }^{\prime }\left(a_t|s_t\right)}{\pi \left(a_t|s_t\right)}\right]\end{array}$$

(25)

$$\begin{array}{c}\hfill J_{C_i}\left({\pi }^{\prime }\right)-J_{C_i}\left(\pi \right)\approx {\mathbb{E}}_{\tau \sim \pi }\left[\sum _{t=0}^{\infty }{A}_{C_i}^{\pi }\left(s_t,a_t\right)\frac{{\pi }^{\prime }\left(a_t|s_t\right)}{\pi \left(a_t|s_t\right)}\right]\end{array}$$

(26)

Due to the approximation of the difference and the requirement in importance sampling that the two strategy distributions must not be too far apart, a constraint is added on the average Kullback–Leibler (KL) divergence between the old and new policies:

$$\begin{array}{c}\hfill {\overline{D}}_{KL}\left(\pi \parallel {\pi }^{\prime }\right)\le \delta \end{array}$$

(27)

where $\delta$ is a step size. By employing the minorization-maximization (MM) algorithm, policy boosting can be guaranteed through maximizing an approximate lower bound on the difference between the old and new policies (often called the alternative objective function). Similarly, if the sum of the upper bound of the difference between the cost function of the old and the new strategy and the cost function of the old strategy satisfies the constraint, then the cost objective function of the new policy can also satisfy the constraint. As a result, this optimization problem can be expressed as

$$\underset{\theta }{max}{\mathbb{E}}_{\left(s_t,a_t\right)\sim \pi }\left[\frac{{\pi }_{\theta }\left(a_t\mid s_t\right)}{\pi \left(a_t\mid s_t\right)}{\widehat{A}}^{\pi }\left(s_t,a_t\right)\right]$$

$$\begin{array}{c}\hfill \mathrm{s}.\mathrm{t}.J_{C_i}\left(\pi \right)+{\mathbb{E}}_{\left(s_t,a_t\right)\sim \pi }\left[\frac{{\pi }_{\theta }\left(a_t\mid s_t\right)}{\pi \left(a_t\mid s_t\right)}{\widehat{A}}_{C_i}^{\pi }\left(s_t,a_t\right)\right]\le d_i\forall i\end{array}$$

(28)

$${\overline{D}}_{KL}\left(\pi \parallel {\pi }^{\theta }\right)\le \delta$$

where $\theta$ is the parameter of the policy network and $\widehat{A}$ is the advantage value function estimated by generalized advantage estimation [30]. Since the KL divergence limits the difference in the policy distribution to a small step size, the objective and constraint functions in the above equation can be expanded in the first order, while the KL divergence constraint is expanded in the second order. Let the gradient of the objective function be g, the gradient of constraint i be $b_i$, the Hessian of KL divergence be H, and c be defined as $J_{C_i}\left(\pi \right)-d_i$. Thus, the approximate problem is

$$\begin{array}{cc}\hfill {\theta }_{k+1}=arg\underset{\theta }{max}& g^T\left(\theta -{\theta }_k\right)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.c_i+b_i^T\left(\theta -{\theta }_k\right)& \le 0i=1,\dots ,m\hfill \\ \hfill \frac{1}{2}{\left(\theta -{\theta }_k\right)}^{T}H& \left(\theta -{\theta }_k\right)\le \delta \hfill \end{array}$$

(29)

Due to the computational complexity of the Hessian matrix, CPO uses a Fisher information matrix to approximate H, which makes the problem above a convex optimization problem [27]. To solve this convex optimization problem, we denote the Lagrangian multipliers $\lambda$ and $\nu$. Then, when the original problem has a feasible solution, a dual to (29) can be expressed as

$$\begin{array}{c}\hfill \underset{\begin{array}{c}\lambda \ge 0\\ \nu \ge 0\end{array}}{max}\frac{-1}{2\lambda }\left(g^T{H}^{-1}g-2{\nu }^T{b}^T{H}^{-1}g+{\nu }^T{b}^T{H}^{-1}b\nu \right)+{\nu }^{T}c-\lambda \delta \end{array}$$

(30)

If ${\lambda }^{*}$ and ${\nu }^{*}$ are the optimal solution to the dual problem, the policy can be updated as follows:

$$\begin{array}{c}\hfill {\theta }^{k+1}={\theta }^k+\frac{1}{{\lambda }^{*}}{H}^{-1}\left(g-b{\nu }^{*}\right)\end{array}$$

(31)

When there is no feasible solution to the original problem, CPO will execute the recovery method, which will change the search direction. The recovery method is as follows:

$$\begin{array}{c}\hfill {\theta }^{k+1}={\theta }^k-\sqrt{\frac{2\delta }{b^T{H}^{-1}b}}{H}^{-1}b\end{array}$$

(32)

Inspired by TRPO, the line search method is used to adjust the step size of the update to avoid any error caused by the approximation.

3.2.3. Implementation Details

In this section, we give the corresponding design of the state space, action space, and reward function in RL. The state and action spaces are described as

$$\begin{array}{cc}\hfill s& ={\left(sin\theta ,cos\theta ,v_x,v_z,q,t_{left},I_{v_x},I_{v_z},I_{\theta },z\right)}^T\in R^{10},\hfill \\ \hfill a& ={\left({\delta }_y,{\omega }_r\right)}^T\in R^2.\hfill \end{array}$$

(33)

where ${\delta }_y$ represents the vane deflection of pitch, ${\omega }_r$ is the angular velocity of the rotor, $\theta$ is the pitch angle, $v_x$ and $v_z$ are the velocity of the X-axis and Z-axis in the inertial frame, q is the angular velocity of the pitch angle, $t_{left}$ is the remaining time during one episode (the maximum time for an episode is 10 s), and z is the height. $I_{v_x},I_{v_z},I_{\theta }$ is the integral error of $v_x$, $v_z$, $\theta$, which can be expressed as

$$\begin{array}{cc}\hfill e_{\theta }& =\theta -\pi /2,e_{v_x}=v_x,e_{v_z}=v_z\hfill \end{array}$$

(34)

$$\begin{array}{cc}\hfill I_n& =\eta I_{n-1}+e_n\hfill \end{array}$$

(35)

where $\eta$ is the integral coefficient, and the integration of the error is approximated by such an incremental approach as Equation (35). The design of the reward function is a key factor affecting the performance of the reinforcement learning algorithm. The reward function R and the cost function C are given by the following:

$$\begin{array}{c}\Phi \left(s_t\right)=k_1\sqrt{v_x^2+v_z^2}+k_2\left|\theta -\pi /2\right|\end{array}$$

(36)

$$\begin{array}{c}R=\kappa +(\Phi \left(s_{t+1}\right)-\Phi \left(s_t\right))\times {\left(0.5\right)}^{t/10}-\chi I\left(s\right)-\beta (q>\pi )-\xi (\left|z\right|>2)-\tau \end{array}$$

(37)

$$\begin{array}{c}C=1\mathrm{if}∥v_z∥\ge 1\mathrm{else}0\end{array}$$

(38)

where the various terms are described as follows.

1.

$\kappa$ is a bonus reward for successful transition, where the terminal velocity, pitch angle, and angular velocity are all within specified limits. In addition, different amounts of bonuses are also given based on the loss of height when a successful transition state is reached. We use $Z_T$ to represent the terminal height loss.

$$\begin{array}{cc}\hfill \kappa & =\left\{\begin{array}{cc}1000& \mathrm{if}{Z}_T\le 0.125\mathrm{m}\\ 100-100\times {log}_2{Z}_T& \mathrm{if}{Z}_T\le 1\mathrm{m}\\ 100-90\times {log}_2{Z}_T& \mathrm{if}{Z}_T\le 2\mathrm{m}\\ 10& \mathrm{else}\end{array}\right.\hfill \end{array}$$

(39)

2.

$\Phi \left(s_{t+1}\right)-\Phi \left(s_t\right)$ is in the form of reward shaping. The aim of reward shaping is to accelerate the learning process, where k1 and k2 denote the weights of the speed and angle rewards. The term ${\left(0.5\right)}^{t/10}$, which is gradually decaying over time, is designed to reduce the transition time.

3.

$\chi$ is constant and $I\left(s\right)=∥0.02\times I_{v_x}^2+I_{\theta }^2+I_{v_z}^2∥$, $\beta$ is a penalty for an angle exceeding the limit, and $\xi$ is a penalty when the UAV loses more than two meters in altitude. $\tau$ is a fixed time penalty.

The cost limit d is set at 5. The terminal condition of an episode is either meeting the termination condition specified in Section 3.1 or reaching the maximum time of 10 s.

4. Simulation and Results

In this section, we present the experimental results. In our experiments, the controller update and sensor data download frequency was set as 50 HZ. For our experiments, the initial flight mode was set to level flight, with the initial angle of attack sampled from a uniform distribution ranging between $6^{\circ }$ and ${15}^{\circ }$. The initial pitch angle was set equal to the angle of attack, the initial height was set at 50 m, and the corresponding horizontal velocity could be determined using the following equation:

$$\begin{array}{c}\hfill v_{level}=-\sqrt{\frac{mg}{\frac{1}{2}{C}_L\left(\alpha \right){\rho }_{\infty }S}}\end{array}$$

(40)

4.1. Comparing CPO and TRPO with Fixed Penalty

To demonstrate the superiority of safe reinforcement learning compared to other conventional reinforcement learning algorithms with a fixed penalty, we conducted three different sets of experiments, which were named CPO, TRPO with penalty 1, and TRPO with penalty 5. The experimental results are shown in Figure 7.

During training, we took into account the randomization of the initial state and the perturbation of the aerodynamic coefficients. However, in evaluating the transition performance of CPO and TRPO with penalty, we have not considered these factors. Three sets of experiments with the same initial state (angle of attack at 6${}^{\circ }$) for the sake of comparison are shown in Figure 8, where the arrows represent the terminal state.

In Figure 7 and Figure 8, we can observe that the trajectory corresponding to TRPO with penalty 1 converges to a locally optimal continuous ascending path, resulting in a fast transition time but significant altitude loss. In contrast, the trajectory corresponding to TRPO with penalty 5 consistently satisfies the constraints, but its terminal angle and speed exceed the predefined range, ultimately failing to complete the task. The CPO algorithm, however, effectively meets the constraints while accomplishing the transition with a minimal altitude loss of 0.1 m, which complies with the requirements of a neat transition.

4.2. Comparing CPO and PPOLag

In the domain of safe reinforcement learning, numerous algorithms can address constraint-related issues. For the sake of code development convenience, we have chosen to compare the CPO algorithm with Proximal Policy Optimization with Lagrangian, also called PPOLag (see Figure 9).

PPOLag is a variant of the Proximal Policy Optimization (PPO) algorithm, which employs a Lagrangian relaxation approach to handle constraints. Constraints are integrated into the objective function using a penalty term that is computed with the help of the Lagrange multiplier. The Lagrange multiplier is updated during the training process based on the discrepancy between the current constraint value and its cost limit, which serves as a weighting factor to balance the trade-off between the reward function and the constraints.

PPOLag is a simpler algorithm in principle and easier to implement. However, as shown in Figure 9b, this method may cause oscillations near the cost limit of the constraint, leading to poor performance of the agent (as shown in Figure 9a). Consequently, although PPOLag can complete the transition, its performance is inferior to that of CPO. The best-trained model with PPOLag (at an angle of attack of 6°) has a terminal height loss of 1.2 m and a transition time of 7.84 s, but CPO has a terminal height loss of 0.1 m and a transition time of 6.94 s.

4.3. Comparison with GPOPS-II

According to the problem formulation in Section 3.1, this problem can also be considered as an optimal control problem. As a result, we choose to employ GPOPS-II when computing the optimal trajectory. In this case, the control variables for GPOPS-II are the same as in our approach—namely, the angular velocity of the rotor and elevator deflection. The experimental results without perturbations at an initial angle of attack ${10}^{\circ }$ are compared in Figure 10 and Figure 11. The cost function of GPOPS-II is the following:

$$\begin{array}{c}\hfill J=0.5t_f+0.2\left|Z_{t_f}\right|\end{array}$$

(41)

where $t_f$ is the transition time and $Z_{t_f}$ is the terminal height loss.

The height loss of GPOPS-II is 0.01 m with a transition time of 5.21 s, while the height loss of CPO is 0.103 m with a transition time of 5.62 s. As can be seen from the figures, the performance of the CPO algorithm closely approximates that of GPOPS-II. However, GPOPS-II has two main drawbacks. Firstly, as a model-based approach, it provides only optimal feed-forward control input, which is an ideal solution. When the dynamics model of the UAV changes, it requires re-solving based on the altered model. In contrast, the CPO algorithm is highly robust and can still perform the transition task with high performance despite certain modeling errors and wind disturbance, as discussed later in Section 4.4 and Section 4.5. Secondly, as micro-controllers onboard aerial robots generally only have limited computational power, the optimization of GPOPS-II can be only executed offline. However, RL algorithms such as CPO can solve the transition problem online after a policy model has been trained, which saves a great deal of computational resources.

4.4. Robustness Validation

During the training process, the randomness of the UAV is only determined by the randomness of the initial state and aerodynamic parameters. Therefore, in order to assess the robustness of the system, we chose to randomize various factors, including the mass, moment of inertia, and aerodynamic parameters of lift and drag (with a larger range), by employing a uniform distribution. Furthermore, we took sensor noise into account because measurement errors in the height can potentially lead to a significant decline in the system’s performance, especially for terminal height loss. To validate the robustness of sensor noise, Gaussian noise was introduced into the height measurements. It is important to note that we did not consider these perturbations during training, meaning that the UAV had not been exposed to them before.

To enable a fair comparison, we only randomized the initial state and tested each perturbation individually. Specifically, we conducted 500 experiments for each perturbation, while maintaining the same transition success condition as in Section 3.1. We report the average performance of the UAV across all experiments (see

Table 1. Experimental results under different perturbations.

Perturbation Type	Parameter Range	Success Rate	Transition Time	Terminal Height Loss	Terminal Velocity	Terminal Pitch Angle
Mass	21%	94%	6.30 s	0.27 m	1.01 m/s	95.09°
Lift and Drag	40%	97.0%	6.09 s	0.109 m	0.966 m/s	93.76°
Inertia	40%	100%	6.02 s	0.072 m	0.80 m/s	93.42°
Sensor Noise	$\left|\mu \right|\le 0.2$, $0.005\le \sigma$$\le 0.05$	100%	6.54 s	0.133 m	0.95 m/s	93.05°

). At the same time, the trajectories under each disturbance group (50 in total) are also provided. We chose to use the pitch angle and velocity of the UAV during the transition process to draw 2D plane curves (see Figure 12, Figure 13, Figure 14 and Figure 15), where the different color simply indicates that these trajectories are different. Through the figures, we can find that the UAV can still complete the transition task with excellent performance despite the change in UAV model parameters. At the same time, we can observe that even when the UAV does not transition successfully, its terminal velocity and pitch angle mostly remain close to our desired range.

4.5. Transition under Wind Disturbance

In this section, we aim to verify the wind disturbance rejection ability of our method. In RL-based robotic control design, the sim-to-real gap is a challenging problem because there are inevitable mismatches between the simulator setting and real-world setting in the UAV control problem. This is not only due to the unknown perturbations of the UAV parameters but also the presence of adversarial disturbances such as wind in the real world. To account for wind disturbance, we introduce two different wind scenarios: constant-magnitude wind (ranging from −5 m/s to 5 m/s) and sinusoidal wind (with an amplitude of 5 m/s and time periods of 2 s, 3 s, 4 s, and 5 s), both along the horizontal direction, which is the $X^I$ axis. Furthermore, the dynamics of ducted-fan fixed-wing UAVs in the presence of wind are described in detail in [3,14]. Therefore, we recreate the dynamic equations of the UAV using their modeling of wind.

To address the wind disturbance, we employ domain randomization [31] and retrain the RL agent under three different conditions: the two wind scenarios mentioned above and a no-wind scenario. Domain randomization is an approach used to overcome the sim-to-real gap by randomizing the simulator environment during the training of the RL control policy. By adapting the domain randomization approach, we aim to retrain a UAV controller that is robust against wind disturbance.

The trajectory comparison among the constant-magnitude wind (5 m/s), sinusoidal wind (time period 2 s), and no-wind conditions is shown in Figure 16, where the arrows represent the terminal state and all initialized from the angle of attack $6^{\circ }$.

From Figure 8 and Figure 16, we can see that compared to the controller trained without domain randomization, the terminal height loss of the drone retrained with domain randomization remains nearly the same, while the time taken increases by 1.52 s. This phenomenon can be intuitively understood as follows: the RL algorithm selects actions with the highest expected returns under various wind disturbance scenarios, rather than actions that can achieve high returns in windless environments, but may fail to complete the task in windy conditions. Thus, a certain degree of performance loss is reasonable.

In order to better evaluate the resistance of our method to wind disturbances, we applied the UAV in 100 experiments under three conditions and from different initial states, in addition to considering the perturbation of the aerodynamic coefficients during the transition (a uniform distribution ranging between 0.8 and 1.2). The experimental results are shown in Figure 17 (success rate 90%), where the different color simply indicates that these trajectories are different.

In Figure 17, we observe that in most cases, the UAV is able to withstand wind disturbances and complete the transition task. However, for those trajectories that do not satisfy the terminal constraint in Section 3.1, we find that the trajectories generally terminate with an attitude close to vertical hover (${90}^{\circ }$) and a small velocity. This suggests that the UAV is less resistant to interference in the hovering state. In conventional approaches, a common method is to switch to the hover controller when the UAV transitions from level flight to near-hover. Wind disturbance can be resisted by switching between the two controllers and enhancing the hover controller’s ability to handle wind disturbance. In RL, resistance to wind disturbance in hovering should be treated as a separate task, and the problem should be addressed using multi-task RL or meta-learning, which will be the focus of our future work.

5. Conclusions

In this study, we have successfully developed a safe reinforcement learning-based approach for neat transition control during the back-transition process of ducted-fan fixed-wing UAVs. By constructing a three-degree-of-freedom longitudinal model and implementing the CPO algorithm, our method effectively addresses the challenges of integrating trajectory optimization and control methods. By comparison, we found that the introduction of a velocity constraint leads to better performance compared to adding a penalty to the reward. Furthermore, we also found that our method closely resembles GPOPS-II’s performance without the need for prior knowledge. Additionally, we confirmed the robustness of the CPO algorithm and found that even when the transition was not successful, the terminal conditions remained close to our desired terminal range. Future research directions include enabling the UAV to complete multi-tasks (i.e., from hover to level flight, hover under wind disturbance, and level flight), ensuring robustness against wind disturbance, and validating the approach in the real world.