Transient Controller Design Based on Reinforcement Learning for a Turbofan Engine with Actuator Dynamics

Abstract

To solve the problem of transient control design with uncertainties and degradation in the life cycle, a design method for a turbofan engine’s transient controller based on reinforcement learning is proposed. The method adopts an actor–critic framework and deep deterministic policy gradient (DDPG) algorithm with the ability to train an agent with continuous action policy for the continuous and violent turbofan engine state change. Combined with a symmetrical acceleration and deceleration transient control plan, a reward function with the aim of servo tracking is proposed. Simulations under different conditions were carried out with a controller designed via the proposed method. The simulation results show that during the acceleration process of the engine from idle to an intermediate state, the controlled variables have no overshoot, and the settling time does not exceed 3.8 s. During the deceleration process of the engine from an intermediate state to idle, the corrected speed of high-pressure rotor has no overshoot, the corrected-speed overshoot of the low-pressure rotor does not exceed 1.5%, and the settling time does not exceed 3.3 s. A system with the designed transient controller can maintain the performance when uncertainties and degradation are considered.

1. Introduction

A turbo engine, which is a classical type of aero engine, is a sophisticated piece of thermal equipment with symmetrical geometry. In recent years, with the rapid development of the aerospace industry, the capacity for supersonic and hypersonic flight over a wider flight envelope has been demanded for aero engines [1]. To achieve these goals, more and more complex structures—such as variable cycle, adaptive cycle and turbine-based combined cycle (TBCC) systems—are applied to aero engines, making their modeling and control design more difficult [2]. The control design plays an essential role in the integral aero engine system for a controller, which is responsible for keeping the system asymptotically stable, minimizing the transient process time, and maintaining enough margin to keep the engine working in the event of a surge, extreme temperatures, or excess revolutions.

However, control design is becoming more challenging with modern control theory, making engines so sophisticated that they cannot be modeled accurately. One of the challenges is the existence of disturbances and uncertainties in the aero engine system, which can affect the performance and stability of the system. Therefore, rejection of disturbances and uncertainties has been a critical design objective, which is traditionally achieved by observer and robust control design methods. Observer control is widely used to reject the disturbance [3,4], while robust control can be applied with model uncertainties [5,6,7]. Both observer and robust control design depend on an accurate linear model, where the uncertainties are introduced into the system via linearization. When an accurate linear model is unable to be obtained, these methods design controllers with the idea of sacrificing some performance for robustness. Another challenge is that control parameters should be changed when the components degrade after operating under tough working conditions for a long period of time throughout the whole asymmetrical life cycle [8,9]. When performance degrades, a system with traditional controllers will be vulnerable [10].

Therefore, reinforcement learning is taken into consideration, because it combines the advantages of optimal control and adaptive control [11], meaning that the desired performance can be achieved and the parameters can be adjusted to obtain the required robustness. This is critical because the optimal controller can be obtained without knowing the full system dynamics. As a branch of artificial intelligence, the key to reinforcement learning in feedback control is training the agent with a policy deciding the action of the system. Many algorithms—such as policy iteration, value iteration, Q-learning, deep Q-networks (DQNs), and deep deterministic policy gradient (DDPG)—have been proposed to learn the parameters under different conditions. Policy iteration and value iteration are basic methods of finding optimal values and optimal policy by solving the Bellman equation where policy iteration converges to the optimal value in fewer steps, making value iteration easier to implement [12]. Q-learning methods are based on the Q-function, which is also called the action-value function [13]. The Q-function designs an adaptive control algorithm that converges online to the optimal control solution for complete unknown systems. The DQN is capable of solving problems of high-dimensional observation space, but it cannot be straightforwardly applied to continuous domains, since it relies on finding the action that maximizes the action-value function [14]. The development of DDPG, which is a method of learning continuous action policy, originates from the policy gradient (PG) method developed in 2000 [15]. In 2014, Silver presented deterministic policy gradient (DPG) [16]. More details about the development of DDPG from DPG were given by Lillicrap [13]. Studies using the reinforcement learning method described above have been carried out in the aeronautics and aerospace control industry for learning policies for autonomous planetary landing [17] and unmanned aerial vehicle control [18]. In [19], a deep reinforcement learning technique is applied to a conventional controller for spacecraft. The author of [20] demonstrated that deep reinforcement learning has a possibility to exceed the conventional model-based feedback control in the field of flow control. The DPG algorithm has been adopted to design coupled multivariable controllers for variable cycle engines at set points [21]. The DDPG algorithm has been used to adjust the engine pressure ratio control law online in order to decrease fuel consumption for an adaptive cycle engine [22]. Reinforcement learning is also applied to prediction of aero engines’ gas path health state [23] and life-cycle maintenance [24]. However, few applications of reinforcement learning have been directly used for aero engine control design—especially the transient control design. Turbofan engines take action continuously, with states changing quickly in a wide range of working conditions with uncertainties and degradation. Reinforcement learning is likely to be an ideal way to design the controller for a turbofan engine, since its optimal design procedure is independent of the knowledge of full system dynamics. Therefore, a reinforcement-learning-based controller design method with the agent trained with the DDPG algorithm is proposed in this paper. This is achieved with the turbofan engine nonlinear model, which reduces the uncertainties introduced into the system via linearization. Moreover, this approach has the advantage of designing the set point controller and transient controller together with the same policy, which will restrain the jump from one to the other. A series of improvements are proposed to improve the stability of the closed-loop system, making the training process achievable with a nonlinear model by solving the problem of divergence. Symmetrical performance is achieved in the acceleration and deceleration process of the engine with the designed controller.

The rest of this paper is organized as follows: In Section 2, a nonlinear model of a dual-spool turbofan engine is built and linearized. In Section 3, a brief background of reinforcement learning and DDPG is given. In Section 4, the method of designing the controller for a turbofan engine with reinforcement learning is presented. In Section 5, a series of simulations are applied in different conditions. The simulation results are compared with a traditional gain-scheduled controller, which is designed with linear matrix inequality (LMI) based on an LPV model. Finally, conclusions are given in Section 6.

2. System Uncertainties Analysis

Consider the follow nonlinear system:

$$\{\begin{array}{l}\dot{\mathit{x}}(t)=f(\mathit{x}(t),\mathit{u}(t),\mathit{d}(t))\\ \mathit{y}(t)=g(\mathit{x}(t),\mathit{u}(t),\mathit{d}(t))\end{array}$$

(1)

where $\mathit{x}(t)\in R^x$ is the state vector of the system, $\mathit{u}(t)\in R^u$ is the input vector of the system, $\mathit{y}(t)\in R^y$ is the output vector of the system, and $\mathit{d}(t)\in R^d$ is the disturbance vector.

Moreover, a linear system is proposed to approximate the dynamic of the nonlinear system for decreasing the nonlinear complexity and making it easier to design the controller k(t), which regulates the y(t) to the desired outputs r(t), based on classic control theory or modern control theory. This can be described as follows:

$$\{\begin{array}{l}\delta \dot{\mathit{x}}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\mathit{A}(\mathit{x}-{\mathit{x}}_e)+\mathit{B}(\mathit{u}-{\mathit{u}}_e)+\mathit{G}(\mathit{d}-{\mathit{d}}_e)\\ \delta \mathit{y}=\mathit{C}(\mathit{x}-{\mathit{x}}_e)+\mathit{D}(\mathit{u}-{\mathit{u}}_e)+\mathit{H}(\mathit{d}-{\mathit{d}}_e)\end{array}$$

(2)

where x_e is the steady-state vector of the system, u_e is the input vector that keeps the system working at x_e, y_e is the steady output vector of the system at state x_e with input u_e, and d_e is the disturbance vector at x_e. For a stable system, (x_e, u_e, d_e), which keeps the system working at steady state, always exists. The nonlinear system at steady state can be described as follows:

$$\{\begin{array}{l}0 =f({\mathit{x}}_e,{\mathit{u}}_e,{\mathit{d}}_e)\\ {\mathit{y}}_e=g({\mathit{x}}_e,{\mathit{u}}_e,{\mathit{d}}_e)\end{array}$$

(3)

Matrices of the linear system are usually obtained by linearizing at steady points. A is the state matrix, B is the input matrix, C is the output matrix, D is the feedforward matrix, G is a disturbance matrix, and H is a disturbance matrix. They are obtained as follows:

$\mathit{A}={\frac{\partial f}{\partial \mathit{x}}|}_{({\mathit{x}}_e,{\mathit{u}}_e,{\mathit{d}}_e)}$, $\mathit{B}={\frac{\partial f}{\partial \mathit{u}}|}_{({\mathit{x}}_e,{\mathit{u}}_e,{\mathit{d}}_e)}$, $\mathit{G}={\frac{\partial f}{\partial \mathit{d}}|}_{({\mathit{x}}_e,{\mathit{u}}_e,{\mathit{d}}_e)}$, $\mathit{C}={\frac{\partial g}{\partial \mathit{x}}|}_{({\mathit{x}}_e,{\mathit{u}}_e,{\mathit{d}}_e)}$, $\mathit{D}={\frac{\partial g}{\partial \mathit{u}}|}_{({\mathit{x}}_e,{\mathit{u}}_e,{\mathit{d}}_e)}$, $\mathit{H}={\frac{\partial g}{\partial \mathit{d}}|}_{({\mathit{x}}_e,{\mathit{u}}_e,{\mathit{d}}_e)}$.

As a result, uncertainties are introduced into the system. This is described as follows:

$$\{\begin{array}{l}\omega (\dot{\mathit{x}})=f(\mathit{x},\mathit{u},\mathit{d})-f({\mathit{x}}_e,{\mathit{u}}_e,{\mathit{d}}_e)-\mathit{A}(\mathit{x}-{\mathit{x}}_e)-\mathit{B}(\mathit{u}-{\mathit{u}}_e)-\mathit{G}(\mathit{d}-{\mathit{d}}_e)\\ \omega (\mathit{y})=g(\mathit{x},\mathit{u},\mathit{d})-g({\mathit{x}}_e,{\mathit{u}}_e,{\mathit{d}}_e)-\mathit{C}(\mathit{x}-{\mathit{x}}_e)-\mathit{D}(\mathit{u}-{\mathit{u}}_e)-\mathit{H}(\mathit{d}-{\mathit{d}}_e)\end{array}$$

(4)

The uncertainties depend on the difference between the current state x and the linearized steady state x_e. In order to reduce the uncertainties of the linear system when it is used to approximate the nonlinear system over the whole working condition, the linear parameter-varying (LPV) model, which is widely used in modern control theory, is introduced.

For the nonlinear system shown in Equation (1), the LPV model can be obtained by linearizing at n different equilibrium points shown in Equation (3) and scheduling with a parameter. Then, the nonlinear system can be approximated with the LPV model as follows:

$$\{\begin{array}{l}\dot{\mathit{x}}=\mathit{A}(\mathit{p})\mathit{x}+\mathit{B}(\mathit{p})\mathit{u}\\ \mathit{y}=\mathit{C}(\mathit{p})\mathit{x}+\mathit{D}(\mathit{p})\mathit{u}\end{array}$$

(5)

where $\mathit{p}=\left[\begin{array}{ccccc}{p}_1& p_2& p_3& \cdots & p_n\end{array}\right]$ are the scheduled parameters and ${\displaystyle \sum _{j=1}^n{p}_j=1,}\hspace{1em}{p}_j\ge 0$. System matrices are defined as follows:

$$\{\begin{array}{cc}\mathit{A}(\mathit{p})={\displaystyle \sum _{j=1}^n{p}_j\mathit{A}(j)}& \mathit{B}(\mathit{p})={\displaystyle \sum _{j=1}^n{p}_j\mathit{B}(j)}\\ \mathit{C}(\mathit{p})={\displaystyle \sum _{j=1}^n{p}_j\mathit{C}(j)}& \mathit{D}(\mathit{p})={\displaystyle \sum _{j=1}^n{p}_j\mathit{D}(j)}\end{array}$$

(6)

where A(j), B(j), C(j), and D(j) are the jth matrices obtained by linearizing the nonlinear system at equilibrium points $({\mathit{x}}_e(j),{\mathit{u}}_e(j),{\mathit{d}}_e(j))$.

Although the LPV model is proposed with the idea of reducing the uncertainties by reducing the difference between x and x_e, the uncertainties introduced into the system in the linearization still exist, and are very hard to calculate.

Moreover, the dynamics and uncertainty of the actuators cannot be ignored in the transient process of an aero engine, and this is usually simplified as a first-order function, which is defined as follows:

$$\mathit{u}={\tau }_a\mathit{v}\approx {\tilde{\tau }}_a\mathit{v}=\left[\begin{array}{cccc}\frac{k_1}{{\tilde{\tau }}_{a1}s+1}& 0& \cdots & 0\\ 0& \frac{k_2}{{\tilde{\tau }}_{a2}s+1}& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & \frac{k_{\mathit{u}}}{{\tilde{\tau }}_{au}s+1}\end{array}\right]\mathit{v}$$

(7)

where u represents the inputs of the engine and v represents the control signals given by the controller. As a result, uncertainties will be introduced into the system again.

The augmented nonlinear system can be described as follows:

$$\{\begin{array}{c}\dot{\mathit{x}}(t)=f(\mathit{x},\mathit{v},{\mathit{\tau }}_a,\mathit{d})\\ \mathit{y}(t)=g(\mathit{x},\mathit{v},{\mathit{\tau }}_a,\mathit{d})\end{array}$$

(8)

The augmented LPV linear model is described as follows:

$$\{\begin{array}{cc}\mathit{A}(\mathit{p})={\displaystyle \sum _{j=1}^n{p}_j\mathit{A}(j)}& \mathit{B}(\mathit{p})=({\displaystyle \sum _{j=1}^n{p}_j\mathit{B}(j)){\tilde{\mathit{\tau }}}_{\mathit{a}}}\\ \mathit{C}(\mathit{p})={\displaystyle \sum _{j=1}^n{p}_j\mathit{C}(j)}& \mathit{D}(\mathit{p})=({\displaystyle \sum _{j=1}^n{p}_j\mathit{D}(j)){\tilde{\mathit{\tau }}}_{\mathit{a}}}\end{array}$$

(9)

Controller design with reinforcement learning is based on the augmented nonlinear model denoted in Equation (8), while controller design with modern control theory relies on the augmented LPV model with uncertainties denoted in Equation (9).

3. Reinforcement Learning Algorithm

3.1. Preliminaries

Definition 1.

The Markov decision process (MDP), which is one of the bases of reinforcement learning, is a memoryless stochastic process denoted with a tuple <S, A, P, R, γ>, where S is a finite set of state, A is the set of action, P is the state transition probability matrix, R is the reward function, and γ is the discounted factor.

Definition 2.

Cumulative reward represents the sum of discounted future reward:

$$r_t^{\gamma }={\displaystyle {\sum }_{i=t}^{\infty }{\gamma }^{(i-t)}r(s_i,a_i)}$$

(10)

where discounted factor$\gamma \in [0,1]$, state$s_i\in S$, action$a_i\in A$, and reward function$r:S\times A\to ℝ$.

Theorem 1.

Supposing that the gradient of deterministic policy${\nabla }_{\theta }{\mu }_{\theta }(s)$and the gradient of action-value function${\nabla }_a{Q}^{\mu }(s,a)$exist, the parameter$\theta$of the policy is adjusted in the direction of the performance gradient defined as follows:

$$\begin{array}{l}{\nabla }_{\theta }J({\mu }_{\theta })={\displaystyle {\int }_S{\rho }^{\mu }(s)}{\nabla }_{\theta }{\mu }_{\theta }(s){\nabla }_a{Q}^{\mu }(s,a){|}_{a={\mu }_{\theta }(s)}\mathrm{d}s\\ \hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={\mathbb{E}}_{s~{\rho }^{\mu }}[{\nabla }_{\theta }{\mu }_{\theta }(s){\nabla }_a{Q}^{\mu }(s,a){|}_{a={\mu }_{\theta }(s)}]\end{array}$$

(11)

where${\mu }_{\theta }$is a deterministic policy with parameter$\theta$, ${\rho }^{\mu }$is the discounted state distribution with policy$\mu$, $Q^{\mu }$is the action-value function with policy$\mu$, and$J$is the performance objective$s\in S$, $a\in A$ [15].

3.2. Framework of Reinforcement Learning

The structure of reinforcement learning consists of an agent and an environment. At time t, the agent observes the state s_t and reward r_t of the environment, and executes action a_t following the internal policy π_θ. Then, the environment outputs the next state s_t+1 and reward r_t+1. This is shown in Figure 1.

The purpose of reinforcement learning is to obtain the agent that contains a policy π_θ maximizing the cumulative reward defined in Equation (10) by interacting with the environment. The procedure of training the agent with DDPG is shown in Figure 2.

In this procedure, the DDPG algorithm trains the agent with the actor–critic framework shown in Figure 3. Both critic and actor combine two neural networks: the estimation network, and the target network. The four neural networks are denoted as $\mu (s|{\theta }^{\mu })$, $Q(s,a|{\theta }^Q)$, ${\mu }^{\prime }(s|{\theta }^{{\mu }^{\prime }})$, and $Q^{\prime }(s,{\mu }^{\prime }(s|{\theta }^{{\mu }^{\prime }})|{\theta }^{Q^{\prime }})$, with weights ${\theta }^{\mu }$, ${\theta }^Q$, ${\theta }^{{\mu }^{\prime }}$, and ${\theta }^{Q^{\prime }}$, respectively.

(1)

The actor network $\mu (s|{\theta }^{\mu })$ represents the optimal action policy. It is responsible for iteratively updating the network weights ${\theta }^{\mu }$, choosing the current action a_i based on the current state s_i, and obtaining the next state s_i+1 and the reward r_i;

(2)

The critic network $Q(s,a|{\theta }^Q)$ represents the Q-value obtained after taking action following the policy defined with the actor network at every state s. It is used to update the network weights ${\theta }^Q$ and calculate the current $Q(s_i,a_i|{\theta }^Q)$;

(3)

The target actor network ${\mu }^{\prime }(s|{\theta }^{{\mu }^{\prime }})$ is a copy of actor network $\mu (s|{\theta }^{\mu })$. The weights of the target actor network are updated with the following soft update algorithm:

$${\theta }^{{\mu }^{\prime }}\leftarrow \tau {\theta }^{\mu }+(1-\tau ){\theta }^{{\mu }^{\prime }}$$

(12)

where $\tau$ is the updating factor;

(4)

The target critic network $Q^{\prime }(s,{\mu }^{\prime }(s|{\theta }^{{\mu }^{\prime }})|{\theta }^{Q^{\prime }})$ is a copy of the critic network, and is used to calculate y_i. Similarly, the weights are updated with the following soft update algorithm:

$${\theta }^{Q^{\prime }}\leftarrow \tau {\theta }^Q+(1-\tau ){\theta }^{Q^{\prime }}$$

(13)

where $\tau$ is the updating factor.

With a relatively small $\tau$, the weight of the target network will change slowly, increasing the stability of the system and making the training process easier to converge. The relationships of the four networks are shown in Figure 4. The purpose of the training process is to find the optimal weights of the networks.

Finally, the optimal weights of the actor network and critic network are obtained. Therefore, the actor and critic networks will work together to achieve the user-prescribed goal.

4. Reinforcement Learning Controller Design Procedure for Turbofan Engines

In this section, the procedure of how to apply reinforcement learning to turbofan engines’ transient control design is provided.

4.1. Framework Definition

As an example of the nonlinear system described in Equation (1), the dynamic of a two-spool turbofan engine can be modeled as follows [24]:

$$\{\begin{array}{c}{\dot{n}}_1=f_1(n_1,n_2,\mathit{v},\mathit{d})\\ {\dot{n}}_2=f_2(n_1,n_2,\mathit{v},\mathit{d})\end{array}$$

(14)

where n₁ is the low-pressure rotor speed, n₂ is the high-pressure rotor speed, v is the control variable vector, and d is the disturbance vector.

Because the corrected rotor represents the characteristic of the turbofan better, the outputs of the nonlinear system can be denoted as $\mathit{y}={\left[\begin{array}{cc}{n}_{1cor}& n_{2cor}\end{array}\right]}^T$, and the states of the nonlinear system can be denoted as $\mathit{x}={\left[\begin{array}{cc}{n}_{1cor}& n_{2cor}\end{array}\right]}^T$.

$$\{\begin{array}{c}{n}_{1cor}=n_1\sqrt{\frac{288.15}{T_1}}\\ n_{2cor}=n_2\sqrt{\frac{288.15}{T_2}}\end{array}$$

(15)

where T₁ is the temperature before the fan, and T₂ is the temperature before the compressor.

Moreover, the control value could be defined as $\mathit{v}(t)={\left[\begin{array}{cc}{A}_8(t)& W_f(t)\end{array}\right]}^T$ in general, where A₈ is the throat area of the nozzle, and W_f is the fuel flow in the burner.

Ways to implement the desired objectives during the transient process could include open-loop fuel–air ratio control or closed-loop $\dot{n}$ control. For the fuel–air ratio command to be transformed into rotor speed, the trajectory of n₁ and n₂ was selected as the command in this paper. This is defined as $\mathit{r}(t)={\left[\begin{array}{cc}{n}_{1cor}(t)& n_{2cor}(t)\end{array}\right]}^T$.

The simplest structure of a closed-loop output feedback control system is shown in Figure 5. In order to implement transient control with a reinforcement learning method with the set command and control values above, the controller module is replaced with observations, reward, and agent modules.

The inputs of observation are system output y and the error between reference command r and system outputs e = r − y. The output of observation is the observed signal $\mathit{o}={[\begin{array}{ccc}{\displaystyle \int e\mathrm{dt}}& e& \mathit{y}\end{array}]}^T$. The inputs of reward are e and u, which are related to the accuracy and energy of the control. The output of reward is the cumulative reward r_t. The input of the agent is o and the output of the agent is v.

4.2. DDPG Agent Creation

The agent is trained with the aforementioned DDPG algorithm and actor–critic structure. Value functions are approximated with neural networks. This process is called representation. The number of layers, the number of neurons, and the connection of each layer should be defined based on the complexity of the problem. Moreover, both critic and actor representation options consist of learning rate and gradient threshold. The DDPG agent options include sample time, target smooth factor, discount factor, mini-batch size, experience buffer length, noise options variance, and noise options’ variance decay rate. Then, the DDPG agent can be created with the specified critic representation, actor representation, and agent options.

4.3. Reward Function

In order to track the trajectory that contains the prescribed performance of the transient process of the engine, the reward function should reflect the error between the command and the output of the engine. Moreover, the error in the past should be taken into consideration, because the transient process is a continuous process where the state changes rapidly. Finally, a positive constant value should be given to keep the training process working from start to finish, because the agent will have the tendency to stop early with the purpose of not counting the cumulative error. Therefore, the reward function is set as follows:

$$r_t=1-\left|e_t\right|-0.1\left|e_{t-1}\right|$$

(16)

4.4. Problems and Solutions

After applying the aforementioned settings, the reinforcement learning method can be preliminarily introduced into the controller design process for a turbofan engine. However, some problems still need to be solved. One of the most important problems is how to keep the environment convergent when the agent is trained. The turbofan engine model used in this paper in the training process was solved with the Newton–Raphson method, which means that the initial conditions cannot be far away from reasonable values. Otherwise, the engine model will be divergent and the reward will be uncontrolled and unbelievable. Moreover, the training results will be invalid, and a lot of time will be wasted calculating the meaningless results. Therefore, some improvements must be added to the design process.

First of all, the initial parameters of the engine must be scheduled with the state n_2cor. When the reinforcement learning explores the performance of the environment, n_2cor represents the state of the engine. n_1cor should match n_2cor, as well as other coupled parameters, such as temperature and internal pressure. The initial condition of the nonlinear model is defined with the following parameters: mass flow of air, bypass ratio, pressure ratio of the compressor, pressure of the fan, and pressure ratio of the turbine.

Secondly, the control structure should be added into the closed-loop system between the agent and the engine. If the controller is designed with the traditional reinforcement learning method, meaning that the outputs of the agent are control actions, it will be very hard for the turbofan engine to be convergent. Therefore, a more efficient control structure should be introduced. With the new structure shown in Figure 6, the outputs become the parameters of the controller rather than the control value. For example, if the controller adopts the PI control law, the outputs of the agent are parameters K_p and K_i, which are scheduled with neural networks within the agent.

Thirdly, the stop simulation module should be adopted. The convergent condition is defined as $\left|\epsilon \right|<{10}^{-6}$ in the engine model. The termination signal is configured as follows:

$$isdone=\{\begin{array}{cc}0& \max (\left|\epsilon \right|)\ge 1\\ 1& else\end{array}$$

(17)

where $\epsilon$ is the iteration error vector of the engine model, which implies the astringency of the model.

With the termination signal, the training during an episode will stop in a timely manner when the model becomes divergent and the outputs become invalid.

4.5. Training Options

Training options specify the parameters in the process of agent training. These include the maximum number of episodes, the maximum steps per episode, the score-averaging window length, and the stop training value.

In conclusion, the process of designing a controller for a turbofan engine can be listed as follows:

Step 1: Set up the training framework by setting the inputs and outputs of the observation, reward, and agent;

Step 2: Create the DDPG agent by specifying critic representation, actor representation, and agent options;

Step 3: Set the reference signal that represents the desired performance for the turbofan engine, and define the reward function according to the purpose of the training process;

Step 4: Modify the structure with the aim of improving the convergence of the system;

Step 5: Train the agent with the DDPG algorithm.

5. Simulation and Verification

In this section, an example of designing a controller for a dual-spool turbofan engine using reinforcement learning is given, and it is compared with the gain-scheduled proportion integral (GSPI) controller designed with LMI in reference [25]. In order to validate the effectiveness of the reinforcement learning design method, the chosen control structure is also PI, which is widely used in turbofan engine control. The structure of the reinforcement learning proportion integral (RLPI) controller is shown in Figure 6, where a = [K_p K_i].

5.1. Options Specification

Training options are set with the parameters in

Table 1. Training options.

Function	Description	Value
Critic Representation Options	Learn Rate	0.001
	Gradient Threshold	1
Actor Representation Option	Learn Rate	0.0001
	Gradient Threshold	1
DDPG Agent Options	Sample time	0.01
	Target Smooth Factor	0.003
	Discount Factor	1
	Mini-Batch Size	64
	Experience Buffer Length	1,000,000
	Noise Options Variance	0.3
	Noise Options’ Variance Decay Rate	0.00001
Training Options	Sample time	0.01
	Maximum Episodes	20,000
	Maximum Steps per Episode	1000
	Score-Averaging Window Length	2
	Stop Training Value	996

. Training scope includes conditions from the idle to intermediate states, where n_1cor ranges from 7733 r/min to 10,065 r/min, and n_2cor ranges from 9904 r/min to 10,588 r/min.

5.2. Simulation Results in Ideal Conditions and with Uncertainties

The performance of the system with the RLPI controller is validated with the step response of acceleration from idle to intermediate and deceleration from intermediate to idle. Meanwhile, the reinforcement learning design method is only applied to the fuel flow control loop, with the purpose of minimizing the disturbance. Parameters of the nozzle area are shown in Figure 7. As mentioned in Section 2, the uncertainties of the actuators result from the order uncertainty or the dynamic uncertainty. In this section, the time constant of the actuator is changed from 0.1 to 0.2. Simulation results are shown in Figure 8 and

Table 2. Performance of the system with an RLPI controller in different conditions.

τa	Speed	Ts/s	σ%	State
0.1	n1cor	1.43	0	Acceleration
		2.23	1.50	Deceleration
	n2cor	3.84	0	Acceleration
		3.33	0	Deceleration
0.2	n1cor	1.18	0	Acceleration
		2.90	2.23	Deceleration
	n2cor	3.71	0	Acceleration
		3.15	0	Deceleration

. It is illustrated in Figure 8 that the controllers have the ability to track the command with a settling time of no more than 3.8 s and overshooting by no more than 1.5% when ${\tau }_a=0.1$, which are defined as ideal conditions. The performances are very close to one another when the time constant of the actuator changes. This means that when the uncertainties of the actuator exist, the RLPI controller still maintains the performance of the closed-loop system. It is illustrated in

Table 3. Performance of the systems with GSPI and RLPI controllers when degradation takes place.

Controller	Speed	Ts/s	σ%	State
GSPI	n1cor	2.67	0	Acceleration
		3.78	0.39	Deceleration
	n2cor	5.31	0	Acceleration
		3.17	0	Deceleration
RLPI	n1cor	2.53	0	Acceleration
		4.40	0.75	Deceleration
	n2cor	5.27	0	Acceleration
		4.50	0	Deceleration

that the settling time changes by no more than 0.7 s and the overshooting increases by 0.73% when the time constant of the actuator increases.

5.3. Simulation Results with Degradation

Degradation occurs inevitably with the life cycle of the turbofan engine, meaning the system cannot work as effectively as before. Traditionally, more fuel will be consumed to obtain the desired performance.

In this section, the degradation is simulated by reducing the compressor efficiency by 5%. Simulation results are shown in Figure 9, Figure 10 and Figure 11 and Table 3. It can be concluded that the transient performance of the system decreases when the efficiency of the compressor decreases for both deceleration and acceleration processes compared with ideal conditions. It is shown in Figure 10 that the maximum difference in n_1cor is about 300 r/min and the maximum difference in n_2cor is about 100 r/min. The n_1cor change with GSPI is smaller than that with RLPI, and the n_2or change with RLPI is smaller than that with GSPI. Moreover, it is shown in Figure 9 that the response of the corrected rotor speed is smoother with RLPI, due to the nonlinearity of RLPI. The transient process with RLPI also has faster transient response and smaller transient error.

Cumulative error is used as a criterion of servo tracking, which represents the performance of transient control. The result is shown in Figure 12, where cumulative error (CE) is defined as follows:

$$CE=(n-n_{cm\mathit{d}})/n$$

(18)

where n_cmd is the command. The results show that the cumulative error of both GSPI and RLPI increases, but the cumulative error of RLPI is much better. For n_1cor, the performance of the turbofan controlled with the RLPI controller after degradation is almost equal to the performance of the turbofan controlled with the GSPI controller before degradation.

The results in Figure 13 show that when the components’ efficiency decreases, more fuel is needed to keep the engine working at the desired speed. It is also illustrated in Figure 13 that the system with RLPI reduces fuel flow faster during deceleration, and adds fuel faster during acceleration. This explains the results, where RLPI tracked the command better in all conditions. It should also be noted that the surge margin (SM) [26] reduces from 17.17 to 12.19 when the degradation takes place.

The comparison of settling time under different conditions is shown in Figure 14, where the solid lines represent the performance with GSPI and the dotted lines represent the performance with RLPI. In the case of settling time, the RLPI has similar performance to GSPI. However, as noted above, the GSPI has better ability in terms of command tracking.

6. Conclusions

This paper presents a method of designing a reinforcement learning controller for turbofan engines. A nonlinear model of the engine was developed as the environment for the reinforcement training process. A DDPG algorithm with actor–critic architecture and a traditional control structure—which was PI in this paper—is presented. The performance of the designed RLPI controller was verified under the chosen conditions and compared with GSPI controllers. The simulation results show that the closed-loop system with the RLPI controller has the desired performance in the transient process. Additionally, the RLPI controller’s ability to deal with large uncertainties and degradation is proven by comparing the simulation results of actuators’ uncertainties, and by compressor efficiency decreasing with the ideal condition.

Studies that should be carried out in the future are as follows: Firstly, the fuel flow and nozzle area need to be trained together in order to achieve better performance, but the training process will be harder to converge. Secondly, different working conditions in the flight envelope should be considered with all-round simulation. Thirdly, control structures other than PI should also be validated in the future. Finally, it is expected that experiments could be carried out with real engines in the future.