Robust Trajectory Tracking Control for Continuous-Time Nonlinear Systems with State Constraints and Uncertain Disturbances

Abstract

In this paper, a robust trajectory tracking control method with state constraints and uncertain disturbances on the ground of adaptive dynamic programming (ADP) is proposed for nonlinear systems. Firstly, the augmented system consists of the tracking error and the reference trajectory, and the tracking control problems with uncertain disturbances is described as the problem of robust control adjustment. In addition, considering the nominal system of the augmented system, the guaranteed cost tracking control problem is transformed into the optimal control problem by using the discount coefficient in the nominal system. A new safe Hamilton–Jacobi–Bellman (HJB) equation is proposed by combining the cost function with the control barrier function (CBF), so that the behavior of violating the safety regulations for the system states will be punished. In order to solve the new safe HJB equation, a critic neural network (NN) is used to approximate the solution of the safe HJB equation. According to the Lyapunov stability theory, in the case of state constraints and uncertain disturbances, the system states and the parameters of the critic neural network are guaranteed to be uniformly ultimately bounded (UUB). At the end of this paper, the feasibility of the proposed method is verified by a simulation example.

1. Introduction

With the continuous application of automatic driving technology [1,2] and intelligent robot technology [3,4], the role of the safety-critical system has attracted extensive attention. In the process of designing a controller, safety is the primary consideration compared to other performances. For the control system with strict safety requirements, the CBF was applied to the control system as a tool to achieve the purpose of state constraints.

Reinforcement learning (RL) can be regarded as the technology of strategy learning and evaluation learning. In the actual engineering application, although the phenomenon of the dimension curse exists in dynamic programming, RL can deal with it well, and we also call it adaptive dynamic programming (ADP) [5,6,7]. Adaptive dynamic programming is an intelligent control method, and it is also an approximate tool to deal with optimal control problems. However, the analytical solution of the Hamilton–Jacobi–Bellman (HJB) equation is generally difficult to obtain; therefore, the adaptive dynamic programming (ADP) can learn the solution of the HJB equation online by the neural network (NN) approximation method [8,9,10]. At present, a variety of control methods based on ADP have been proposed by researchers to deal with the problem of trajectory tracking and optimal control [11,12,13,14,15,16].

Adaptive dynamic programming enables the complex nonlinear system to achieve the desired tracking control goal [17,18,19,20]. In reference [17], the tracking performance of continuous-time nonlinear systems was analyzed by considering the influence of input constraints. Due to the influence of the actual situation, a series of uncertain disturbance factors are often considered. Therefore, robust optimal tracking control has become a research hot spot. In reference [18], solving tracking problems for complex nonlinear systems with uncertainty can be more difficult, and the adaptive criticism technique was used to solve the robust tracking control problems of nonlinear systems with random disturbances. Considering the nonlinear system with a continuous-time matching uncertainty, an effective robust tracking control method was adopted, and the discounted coefficient was selected for the nominal augmented error system in references [19,20]. Considering the system with disturbances, $H_{\infty }$ tracking control was used in control systems with disturbances [21]. In order to reduce the design cost and waste of resources and adjust the accuracy of the control system, a tracking control method based on the event triggering was proposed [22]. Considering the optimal regulation problem, a new non-quadratic discount performance function was proposed in reference [23]. In reference [24], an improved adaptive robust tracking method was proposed for the uncertainty of nonlinear systems and successfully extended to the mass-spring-damper system. The tracking control method proposed above enabled the feasibility of the control strategy and enabled the system to achieve the predetermined control target. However, none of the tracking control methods proposed above consider the state constraints problem.

In references [25,26,27,28], different ADP-based methods were proposed to solve various engineering problems. In some specific environments, the control system is often required to have reliable security. The purpose for which the safety system was designed is to find its control strategy by conforming to the safety specifications specified by the physical constraints of the system [29]. The use of the CBF method to solve the safety constraints of systems with strict requirements has attracted extensive attention [30,31,32,33]. Let the states displayed by the system converge to the desired equilibrium point; an approximate adjustment method for solving the optimization problem of safety boundary control was proposed, and the cost of violating the safety constraint was directly embedded into the value function [34]. In reference [35], the application of the CBF was introduced, and the verification method and the characteristics of implementation safety in the context of a safety-critical control system are summarized. The discrete-time state constraints problem was described in reference [36], and the HJB equation with the CBF was solved by using the approximate properties of the neural network.

In this paper, a new guaranteed cost robust tracking method with state constraints and uncertain disturbances is proposed. This method can guarantee the convergence of the system error under conditions of uncertain disturbances and state constraints. The discounted coefficient is selected for the nominal augmented system with tracking errors. In addition, the CBF is added to the system to solve the constraint problem of system states. Finally, the approximation property of the critic NN is adopted to deal with the HJB equation. The contributions of this paper are described below:

• For robust tracking control problems, the CBF is applied to the tracking control system with uncertain disturbances so that the system can still have good tracking performance in the case of state constraints;
• Combining the traditional adaptive control method with the CBF, the CBF is directly extended to the original system, and the CBF is used as a penalty function to punish unsafe behavior;
• A new guaranteed cost robust adaptive tracking method with state constraints and uncertain disturbances is proposed to solve the safety HJB equation through the critic NN learning framework, and the critic NN parameters are guaranteed to be uniformly ultimately bounded (UUB) under the influence of state constraints and uncertain disturbances.

The arrangement of other parts of this article is described below: Section 2 states the preliminary knowledge and introduces the relevant contents of the control barrier function. Section 3 describes the selection of discount value functions for the nominal augmented system and introduces the form of the new cost function after adding the barrier function. Section 4 introduces the learning method of a critic neural network with state constraints and uncertain disturbances. In Section 5, the effectiveness of the proposed method is verified by a simulation example. Finally, some conclusions are summarized in Section 6.

2. Preliminaries

2.1. Problem Statement

Consider the following uncertain nonlinear safety system

$$\dot{x}(t)=f(x(t))+g(x(t))u(t)+\mathsf{\Delta }f(x(t)),$$

(1)

where $x(t)$∈$\mathsf{\Phi }$⊂${\mathbb{R}}^n$ is the state variable, $u(t)$∈$\mathcal{U}$⊂${\mathbb{R}}^m$ is the control vector, $\mathsf{\Phi }$ represents the safe feasible states set, $\mathcal{U}$ represents all admissible input sets, $f(x(t))$∈${\mathbb{R}}^n$ and $g(x(t))$∈${\mathbb{R}}^m$ are known functions with $f(0)=0$, and $\mathsf{\Delta }f(x(t))$∈${\mathbb{R}}^n$ is the unknown perturbation term with $\mathsf{\Delta }f(0)=0$. Here, let the initial state $x(0)=x_0$; we assume that there exists a constant $g_M$ and it satisfies $0<$$\parallel g(x)\parallel$$\le g_M$ for $\forall x\in {\mathbb{R}}^n$, and $\mathsf{\Delta }f(x)=g(x)d(x)$, where $d(x)\in {\mathbb{R}}^m$ is an unknown perturbation, and the known perturbation function $d_M(x)>0$ is the boundary of $d(x)$, here, $\parallel d(x)\parallel \le d_M(x)$. In addition, $d(0)=0$ and $d_M(0)=0$.

Assumption 1.

Let the reference trajectory of the system (1) be $x_d(t)$, and $x_d(t)$ is a bounded function, which is limited and generated by the command generator ${\dot{x}}_d(t)=r(x_d(t))$. Meanwhile, the reference trajectory $x_d\in {\mathbb{R}}^n$ and the command function $r\in {\mathbb{R}}^n$ are all Lipschitz continuous.

2.2. Control Barrier Function

The application of the CBF further solves the constraint problem of the system [36]. In a predefined security set, the CBF candidate is always positive and tends to infinity at the defined set boundary. The CBF has a negative derivative at infinity, so the CBF will not reach infinity. If the state of the system is close to the safety boundary, then the condition that the derivative is negative will return the state to the safety set, so that the state displayed by the system will be maintained within the predetermined set. The safe feasible set $\mathsf{\Phi }$ consists of operational constraints and safety specifications [34],

$$\mathsf{\Phi }=\{x\in {\mathbb{R}}^n|h(x)\ge 0\},$$

(2)

$$\partial \mathsf{\Phi }=\{x\in {\mathbb{R}}^n|h(x)=0\},$$

(3)

$$Int\mathsf{\Phi }=\{x\in {\mathbb{R}}^n|h(x)>0\},$$

(4)

where $\partial \mathsf{\Phi }$ represents the boundary of the safe feasible set $\mathsf{\Phi }$, $Int\mathsf{\Phi }$ represents the interior of the set $\mathsf{\Phi }$, and h is a continuously differentiable function of x, which is composed of a one-dimensional system constraint range.

The CBF candidate $B(x)$ satisfies all of the following properties,

$$\frac{1}{{\alpha }_1(h(x))}\le B(x)\le \frac{1}{{\alpha }_2(h(x))},\forall x\in int\mathsf{\Phi }$$

(5)

$$\dot{B}(x)\le {\alpha }_3(h(x)),\forall x\in int\mathsf{\Phi }$$

(6)

where ${\alpha }_1(·)$, ${\alpha }_2(·)$, and ${\alpha }_3(·)$ are Lipschitz class $\mathcal{K}$ functions, and $B(x)$ is a control barrier function.

Assumption 2.

Under the condition of uncertain disturbances, to ensure that the states of the system are constrained. We use a logarithmic control barrier function $B_r(x)$, which satisfies the following properties,

$$\left\{\begin{array}{c}{B}_r(x)>0,\forall x\in \mathsf{\Phi },\hfill \\ B_r(x)\to \infty ,\forall x\in \partial \mathsf{\Phi }.\hfill \end{array}\right.$$

(7)

Besides, $B_r(x)$ is monotonically decreasing for ∀x∈Φ.

Under the condition of satisfying Assumption 2, the expression of the specific logarithmic barrier function can be defined as

$$B_r(x)=-log(\frac{\gamma h(x)}{\gamma h(x)+1}).$$

(8)

In (8), the parameter $\gamma$ is a constant, and $\gamma$ also determines the speed at which $B_r(x)$ is limited as it approaches the safety barrier.

Before describing the modified robust tracking method with constraints, we first make the following definitions and assumptions.

Definition 1.

The safety control input set of the nonlinear system (1) is given below

$${\mathcal{U}}_c=\{u\in {\mathbb{R}}^m|x^u\in int\mathsf{\Phi }\},$$

(9)

where $x^u$ is the system state associated with the control strategy u, and $int$Φ is the interior of the set defined in (4).

Assumption 3.

The initial condition of the nonlinear system $(1)$ is strictly within Φ; in other words, $x_0\in int\mathsf{\Phi }$. Assume that the initial set of allowed inputs is not empty and satisfies ${\mathcal{U}}_a=\mathcal{U}\cap {\mathcal{U}}_c$. In addition, the control strategy $u(x_0)\in {\mathcal{U}}_a$ exists.

3. Guaranteed Cost Robust Tracking Design with State Constraints and Uncertain Disturbances

3.1. Modified Robust Adaptive Tracking Control

The augmented system is constructed by combining the tracking error and the reference trajectory. Before describing the modified robust adaptive tracking control, the tracking error is written as $e_x(t)=x(t)-x_d(t)$. According to (1), the tracking error system is derived as

$$\begin{array}{c}\hfill {\dot{e}}_x(t)=f(x_d(t)+e_x(t))+g(x_d(t)+e_x(t))u(t)+\mathsf{\Delta }f(x_d(t)+e_x(t))-r(x_d(t)),\end{array}$$

(10)

where $r(x_d(t))$ is a Lipschitz continuous function.

By considering the tracking error dynamics (10), the infinite horizon cost function is given below [37]

$$\begin{array}{c}\hfill \overline{V}(e_x(t),u)={\int }_t^{\infty }{e}^{-\alpha (\tau -t)}U(e_x(\tau ),u(\tau ))d(\tau ),\end{array}$$

(11)

where $\alpha >0$ is a discount factor, and $U(e_x,u)$$=e_x^{T}Q{e}_x+u^{T}Ru$, both $Q\in {\mathbb{R}}^{n\times n}$ and $R\in {\mathbb{R}}^{m\times m}$ are symmetric positive definite matrices.

Under the condition of state constraints and uncertain disturbances, the purpose of dealing with the guaranteed cost tracking problem is to find the control input $u=u(e_x(t),x_d(t))$ and a positive real number ${\overline{V}}^{*}$; then the tracking error $e_x(t)$ converges to zero. Meanwhile, the cost function described in (11) satisfies $\overline{V}<{\overline{V}}^{*}$. It should be pointed out that ${\overline{V}}^{*}$ is called a guaranteed cost function, and the control u is called a guaranteed cost control input.

Remark 1.

The discount term $e^{-\alpha (\tau -t)}$ given in (11) is mainly used to ensure that the cost function is $\overline{V}<\infty$ since the control $u(e_x(t),x_d(t))$ contains a part depending on the reference trajectory $x_d(t)$. In the absence of the discount term, $u(e_x(t),x_d(t))$ may make (11) to be unbounded. If the reference trajectory $x_d(t))$ does not converge to zero, the cost function (11) is unbounded without considering the discount term $e^{-\alpha (\tau -t)}$.

Let $s(t)={[e_x^T(t),x_d^T(t)]}^T$∈${\mathbb{R}}^{2n}$, and the augmented system of error dynamics can be given

$$\dot{s}(t)=F(s(t))+G(s(t))u(t)+\mathsf{\Delta }F(s(t)),$$

(12)

the specific forms of $F(s(t))$ and $G(s(t))$ can be expressed as

$$F(s(t))=\left[\begin{array}{c}f(x_d(t)+e_x(t))-r(x_d(t))\\ r(x_d(t))\end{array}\right]$$

and

$$G(s(t))=\left[\begin{array}{c}g(x_d(t)+e_x(t))\\ 0\end{array}\right],$$

and $\mathsf{\Delta }F(s(t))=G(s(t))d(s(t))$, and because we know that $\parallel d(x)\parallel \le d_M(x)$, it is very easy to know that the uncertain disturbance term $\parallel d(s(t))\parallel \le d_M(s(t))$ holds, and $d_M(s(t))$ is the boundary of the uncertain disturbance term $d(s(t))$.

Remark 2.

There is a random disturbance term $d(s(t))$ in the augmented system (12) described above, which makes the process of designing the controller difficult. In the following introduction, the augmented error system (12) is equivalent to the optimal control of its nominal system, and the tracking problem with the random disturbance is transformed into an optimization adjustment problem with the discounted value function.

Considering the existence of the uncertain term $d(s(t))$, the nominal system description of the system (12) is

$$\dot{s}(t)=F(s(t))+G(s(t))u(t).$$

(13)

Inspired by references [29,36], $B_r(x)$ is combined with the nominal augmented system (13), and the modified value function is

$$\begin{array}{c}\hfill V(s(t))={\int }_t^{\infty }{e}^{-\alpha (\tau -t)}[\rho d_M^2(s(\tau ))+s^T(\tau )Q_{T}s(\tau )+u^T(\tau )Ru(\tau )+B_r(s(\tau ))]d(\tau ),\end{array}$$

(14)

where $Q_T$=$diag\{Q,0_{n\times n}\}$, $\rho ={\lambda }_{max}(R)$, the maximum eigenvalue of R can be expressed by ${\lambda }_{max}(R)$, both $Q\in {\mathbb{R}}^{n\times n}$ and $R\in {\mathbb{R}}^{m\times m}$ are weighted symmetric positive definite matrices of augmented systems, and $\alpha >0$ is a discount coefficient.

According to Bellman’s principle of the optimal control theory [38], the minimum value of the Hamiltonian of the modified value function (14) of the nominal system (13) is given

$$\begin{array}{c}\hfill H_{min}(s,u,V_s)=\rho d_M^2(s)+s^T{Q}_{T}s+u^{T}Ru-\alpha V(s)+B_r(s)+V_s^T(F(s)+G(s)u),\end{array}$$

(15)

where $V_s=\partial V/\partial s$, and the cost function $V^{*}(s(t))$ can be considered as

$$V^{*}(s(t))=\underset{u}{min}V(s(t)).$$

(16)

For the system (13) with the control barrier value function (14), since the equation $\partial H(s,u^{*},V_s^{*})/\partial u^{*}=0$ holds, we can obtain the optimal control input $u^{*}$ from (15)

$$u^{*}=-\frac{1}{2}{R}^{-1}{G}^T(s)V_s^{*},$$

(17)

where $V_s^{*}=\partial V^{*}(s)/\partial s$, and $V^{*}(s)$ denotes the optimal value $V(s)$.

3.2. State Constraints Analysis

In the process of designing a robust tracking controller, the CBF as a constraint tool makes the states of the system evolve within the specified constraints, and the system can maintain good performance within the set safety constraints. The CBF provides a constraint tool for safety-critical systems to optimize the performance of other control objectives and clearly explains the priority of security compared to other performance indexes. In order to further describe that the CBF is bounded, it is described below that the boundedness of the CBF is demonstrated by changing the order of the controller.

Lemma 1.

Consider an admissible feedback control strategy $u_1\in {\mathcal{U}}_a$; there is the following time-invariant positive definite function Z, which satisfies $Z\in N^1$

$$\begin{array}{c}\hfill \frac{\partial V^T}{\partial x}(F(s)+G(s)u_1)+\rho d_M^2(s)+s^T{Q}_{T}s+u_1^{T}R{u}_1-\alpha V(s)+B_r(s)=0,\end{array}$$

(18)

$$V(s_0,u_1)=Z(s_0,u_1),$$

(19)

where V is the value function of the system for all $t\in [0,\infty )$, and the following formula holds

$$V(s,u)=Z(s,u).$$

(20)

Proof. 

Assume $V(s,u_1)>0$ exists and is continuously differentiable; then, we have

$$\begin{array}{cc}\hfill V(s(t),u_1)-V(s_0,u_1)=& {\int }_0^t\dot{V}(s(\tau ),u_1)d(\tau )={\int }_0^t\frac{\partial V}{\partial s}(F+G{u}_1)d(\tau ).\hfill \end{array}$$

(21)

Considering (21), there are also

$$\begin{array}{cc}\hfill Z(s(t),u_1)-Z(s_0,u_1)=& -{\int }_0^{t}P(s(\tau ),u_1)d(\tau ),\hfill \end{array}$$

(22)

where $P(s,u)=\rho d_M^2(s)+s^T{Q}_{T}s+u^{T}Ru-\alpha V(s)+B_r(s)$.

We can derive from (21) and (22)

$$\begin{array}{cc}\hfill Z(s(t),u_1)-V(s(t),u_1)=& {\int }_0^t(-\frac{\partial V}{\partial s}(F+G{u}_1)-P(s(\tau ),u_1))d(\tau )+Z(s_0,u_1)-V(s_0,u_1).\hfill \end{array}$$

(23)

Combining (18), (21), and (23), we can obtain

$$\begin{array}{cc}\hfill Z(s(t),u_1)-V(s(t),u_1)=& {\int }_0^t(P(s(\tau ),u_1)-P(s(\tau ),u_1))d(\tau )=0.\hfill \end{array}$$

(24)

Therefore, we can obtain

$$Z(s(t),u_1)=V(s(t),u_1).$$

(25)

This completes the proof. □

Lemma 2.

We consider a series of positive definite value functions $V(s,t,u_1)$, $V(s,t,u_2)$, …, and $V(s,t,u_i)$, and the corresponding abbreviations are $V_1$, $V_2$, …, and $V_k$, which are concerned with the allowable control inputs $u_1(s,t)$, $u_2(s,t)$, …, and $u_k(s,t)\in {\mathcal{U}}_a$. Then, the Hamiltonian value defined in (15) satisfies the following conditions

$$H_{mi{n}_1}\le H_{mi{n}_2}\le \dots H_{mi{n}_i},$$

(26)

and the $CBF$ candidate $B_r^k$ is bounded in the range of $1<k<i$.

Proof. 

Assume that $0\le k\le j\le i$ is satisfied for any j and k, and the condition $H_{mi{n}_k}\le H_{mi{n}_j}$ holds; therefore, one has

$$V_j=V_k+V_o,$$

(27)

where $V_o=V_o(s(t),u_k)$. According to (17), $u^{*}$ can be rewritten as

$$u^{*}=-\frac{1}{2}{R}^{-1}{G}^T\nabla V_j,$$

$$\begin{array}{cc}\hfill H_{mi{n}_j}=& \nabla V_j^T(F+G(-\frac{1}{2}{R}^{-1}{G}^T\nabla V_j))+T(s)+\frac{1}{4}\nabla V_j^{T}G{R}^{-1}{G}^T\nabla V_j.\hfill \end{array}$$

Considering $T(s)=\rho d_M^2(s)+s^T{Q}_{T}s+B_r(s)-\alpha V(s)$. According to (27), one may obtain

$$\begin{array}{c}\hfill H_{mi{n}_j}=H_{mi{n}_k}+\nabla V_o^T(F+G{u}_k^{*})-(u_o^{*T}R{u}_o^{*}).\end{array}$$

According to the above description, since $H_{mi{n}_j}-H_{mi{n}_k}+(u_o^{*T}R{u}_o^{*})\ge 0$, we can obtain

$$\frac{\partial V_o(s,u_k)}{\partial t}\ge 0.$$

Because lim${}_{t\to \infty }{V}_o(s(t))=0$, the following results are obtained

$$V_o\le 0,$$

$$V(s,t,u_1)>V(s,t,u_2)>\dots >V(s,t,u_i).$$

From the above Lemmas 1 and 2, we can obtain

$$Z(s(t),u_k)<Z(s(t),u_1),1<k<i.$$

(28)

In the above derivation, not only $Z(s(t),u_k)$ is bounded, but also $P(s,u)$ is positive definite, and then $B_r^k$ is also bounded. In other words, in the case of state constraints, the system states will not reach the safety boundary in the process of tracking the reference trajectory. This proves that the CBF is bounded within each moment. □

Theorem 1.

For the performance optimization problem described in (16), let both Assumption 2 and Assumption 3 hold. Through the improvement of control input (17), the security of the tracking state is guaranteed within a certain range for all $t>0$.

Proof. 

Through the introduction to Lemmas 1 and 2 above, the performance functions $Z(s,u_k)$ and candidate function $B_r^k$ are bounded at each moment after the control input (17) is changed. From Assumptions 1 and 2, at the boundary of the constraint range, the value of the barrier function $B_r^k$ will reach infinity; in other words, the CBF remains is bounded at any moment, which ensures that the states of the system never reach the safe boundary. □

In the above introduction, the CBF is directly added to the cost function, which makes the states of the system constrained. This method is applicable to the guaranteed cost robust trajectory tracking control without initial admissible control. The traditional tracking controller usually needs the initial admissible control law. Although the appropriate initial admissible control law is found, the appropriate initial admissible control law may not satisfy the condition of state constraints.

Due to the existence of the discount term $e^{-\alpha (\tau -t)}$ in Equation (11), to guarantee the stability of the closed-loop system in the process of the tracking reference trajectory process, a guaranteed cost adaptive critic NN learning framework is designed. Before proceeding to the next step, we make the following assumption.

Assumption 4.

Let $J_1(s)$ be a candidate of Lyapunov function and satisfy the condition of ${\dot{J}}_1(s)=\nabla J_1^T(s)(F(s)+G(s)u^{*})<0$, and $J_1(s)$ is continuously differentiable, where $\nabla J_1(s)=\partial J_1(s)/\partial s$. Assume there exists a symmetric positive definite matrix $\Lambda (s)$, and the condition of expression ${(\nabla J_1(s))}^T(F(s)+G(s)u^{*})=-{(\nabla J_1(s))}^T\Lambda (s)\nabla J_1(s)$ holds.

4. Design of Guaranteed Cost Adaptive Critic NN Learning Framework

In this section, the approximation property of the critic NN is used to approximate the solution of the safety HJB Equation (15), a guaranteed cost adaptive critic NN learning framework is proposed, the weight of the critic NN is updated through online the learning scheme, and all the vectors of the critic NN finally are guaranteed to be UUB. Considering the cost function described in (16), we design a critic NN to approximate the cost function $V^{*}(s(t))$ and its partial derivative

$$V^{*}(s)=W^T\varphi (s)+{ϵ}_v(s),$$

(29)

$$\nabla V^{*}(s)={(\nabla \varphi (s))}^{T}W+\nabla {ϵ}_v(s),$$

(30)

where W∈${\mathbb{R}}^l$ is the ideal vector of the critic neural network, the activation function of the critic NN can be expressed as $\varphi (s)={[{\phi }_1{\phi }_2{\phi }_3···{\phi }_l]}^T\in {\mathbb{R}}^l$, l is the number of hidden-layer neurons, $\nabla \varphi (s)$ is denoted as the derivative of $\varphi (s)$, the approximation error of the critic NN is denoted by ${ϵ}_v(s)$, and $\nabla {ϵ}_v(s)$ is the derivative of ${ϵ}_v(s)$.

Assumption 5.

The vector W of the critic NN is bounded by a positive constant, i.e., $\parallel W\parallel <W_M$, the activation function $\varphi (s)$ and its derivative $\nabla \varphi (s)$, the critic NN error ${ϵ}_v(s)$ and its derivative $\nabla {ϵ}_v(s)$, are bounded, and satisfy $\parallel \varphi (s)\parallel <{\varphi }_{\epsilon }$, $\parallel \nabla \varphi (s)\parallel <{\varphi }_{d\epsilon }$, $\parallel {ϵ}_v(s)\parallel <{ϵ}_{\sigma }$, and $\parallel \nabla {ϵ}_v(s)\parallel$$<{ϵ}_{d\sigma }$, where ${\varphi }_{\epsilon }$, ${\varphi }_{d\epsilon }$, ${ϵ}_{\sigma }$, and ${ϵ}_{d\sigma }$ are positive constants.

From Equations (15), (16), and (30), the approximate error of the safety HJB form is

$$\begin{array}{c}\hfill \rho d_M^2(s)+s^T{Q}_{T}s+u^{*T}R{u}^{*}-\alpha V^{*}(s)+B_r(s)+W^T\nabla \varphi (s)(F(s)+G(s)u^{*})+{ϵ}_{v1}(s)=0,\end{array}$$

(31)

where ${ϵ}_{v1}(s)=\nabla {ϵ}_v^T(s)(F(s)+G(s)u^{*})$.

Considering Equations (17) and (30), we can draw the following conclusion,

$$u^{*}=-\frac{1}{2}{R}^{-1}{G}^T(s){(\nabla \varphi (s))}^{T}W+{ϵ}_{v2},$$

(32)

where ${ϵ}_{v2}=-(1/2)R^{-1}{G}^T(s)\nabla {ϵ}_v$. At the same time, we substitute (32) into (31) and can obtain

$$\begin{array}{c}\hfill \rho d_M^2(s)+s^T{Q}_{T}s-\alpha W^T\varphi +B_r(s)+W^T\nabla \varphi F-\frac{1}{4}{W}^T\nabla \varphi \varrho {(\nabla \varphi )}^{T}W-\omega =0,\end{array}$$

(33)

where $\varrho =G(s)R^{-1}G(s)$, and $\omega =\alpha {ϵ}_v-{(\nabla {ϵ}_v)}^{T}F+(1/4){(\nabla {ϵ}_v)}^T\varrho \nabla {ϵ}_v+(1/2)W^T\nabla \varphi \varrho \nabla {ϵ}_v$ is the approximation error.

We do not know the value of the ideal weight W; therefore, by using the critic NN to approximate the cost function $V^{*}(s)$ as

$$\widehat{V}(s)={\widehat{W}}^T\varphi (s),$$

(34)

where $\widehat{W}$ denotes the estimated value of the ideal vector W, and $\widehat{V}$ is the estimated value of the ideal cost function $V^{*}$. We can obtain the approximate HJB equation form from Equations (15) and (34)

$$\begin{array}{c}\hfill \widehat{H}(s,u,{\widehat{V}}_s)=\rho d_M^2(s)+s^T{Q}_{T}s+u^{T}Ru-\alpha \widehat{V}(s)+B_r(s)+{\widehat{V}}_s^T(F(s)+G(s)u).\end{array}$$

(35)

Based on Equation (34), the control input $\widehat{u}(s)$ can be approximated by

$$\widehat{u}(s)=-\frac{1}{2}{R}^{-1}{G}^T(s){(\nabla \varphi (s))}^T\widehat{W}.$$

(36)

Through Equations (31) and (35), we define the HJB equation error caused by the critic NN in the approximation process as

$$\begin{array}{c}\hfill \epsilon =\rho d_M^2(s)+s^T{Q}_{T}s-\alpha {\widehat{W}}^T\varphi +B_r(s)+{\widehat{W}}^T\nabla \varphi F-\frac{1}{4}{\widehat{W}}^T\nabla \varphi \varrho {(\nabla \varphi )}^T\widehat{W}.\end{array}$$

(37)

The estimation error of the weights of the critic NN is defined as $\tilde{W}$, and we can obtain

$$\tilde{W}=W-\widehat{W}.$$

(38)

The HJB approximation error can be defined as

$$\epsilon =-{\tilde{W}}^T\xi +\frac{1}{4}{\tilde{W}}^T\nabla \varphi \varrho {(\nabla \varphi )}^T\tilde{W}+\omega ,$$

(39)

where $\xi =\nabla \varphi (F(s)+G(s)\widehat{u})-\alpha \varphi (s)$. The Lyapunov function candidate $J_1(s)$ is shown in Assumption 4, and we take $\mathsf{\Pi }(s,\widehat{u})$ as an indicator function and define it as

$$\mathsf{\Pi }(s,\widehat{u})=\left\{\begin{array}{c}0,\mathrm{if}\hspace{1em}{J}_1^T(s)(F(s)+G(s)\widehat{u})<0,\hfill \\ 1,\mathrm{else}.\hfill \end{array}\right.$$

(40)

We choose $\widehat{W}$ to minimize the square residual $E=(1/2){\epsilon }^T\epsilon$, and then we obtain the minimum value of the HJB approximation error $\epsilon$. We use the gradient descent method as the critic vector adjustment optimization law

$$\begin{array}{c}\hfill \dot{\widehat{W}}=-\beta \overline{\xi }(L(s)+Y(s)+B_r(s)+\rho d_M^2(s))+\frac{\beta }{2}\mathsf{\Pi }(s,\widehat{u})\nabla \varphi \varrho J_1(s)+\beta ((K_1{\theta }^T-K_2)+A(s))\widehat{W},\end{array}$$

(41)

where $\overline{\xi }=\xi /{(1+{\xi }^T\xi )}^2$, $\theta =\xi /(1+{\xi }^T\xi )$, $L(s)={\widehat{W}}^T\nabla \varphi F-\alpha {\widehat{W}}^T\varphi +s^T{Q}_{T}s$, $Y(s)=-(1/4){\widehat{W}}^T\nabla \varphi \varrho {(\nabla \varphi )}^T\widehat{W}$, and $A(s)=(1/4)\nabla \varphi \varrho {(\nabla \varphi )}^T\widehat{W}{(\theta /(1+{\xi }^T\xi ))}^T$, and $\beta >0$ is a learning rate that determines the convergence speed of the critic NN. $K_1$ and $K_2$ are two tuning parameters.

From the above description, it is deduced that the weight estimation error is

$$\begin{array}{c}\hfill \dot{\tilde{W}}=-\beta \overline{\xi }({\tilde{W}}^T\xi +\tilde{Y}(s)-B_r(s))-\frac{\beta }{2}\mathsf{\Pi }(s,\widehat{u})\nabla \varphi \varrho J_1-\beta ((K_1{\theta }^T-K_2)-\tilde{A}(s))(W-\tilde{W}),\end{array}$$

(42)

where $\tilde{Y}(s)=-(1/4){\tilde{W}}^T\nabla \varphi \varrho {(\nabla \varphi )}^T\tilde{W}$, $\tilde{A}(s)=(1/4)\nabla \varphi \varrho {(\nabla \varphi )}^T(W-\tilde{W}){(\theta /(1+{\xi }^T\xi ))}^T$.

Theorem 2.

Consider the nominal system (13), the modified value function (15), and the tuning laws (41). Only if all the above Assumptions 1–5 hold, then the critic NN error $\tilde{W}$, the system state x, and the control input $u^{*}$ are guaranteed to be UUB.

Proof. 

Analyze the Lyapunov candidate function described below

$$L(t)=V_s(s(t))+\frac{1}{2}{\tilde{W}}^T{\beta }^{-1}\tilde{W}.$$

(43)

The result of deriving Equation (43) is shown as

$$\begin{array}{cc}\hfill \dot{L}(t)=& {\dot{V}}_s(s(t))+{\tilde{W}}^T{\beta }^{-1}\dot{\tilde{W}}\hfill \\ \hfill =& {\dot{L}}_V+{\dot{L}}_W.\hfill \end{array}$$

(44)

The first term ${\dot{L}}_V$ is

$$\begin{array}{cc}\hfill {\dot{L}}_V=& {\tilde{W}}^T\nabla \varphi (s)(F(s)+G(s)u)+\nabla {ϵ}_v^T(s)(F(s)+G(s)u)\hfill \\ \hfill =& W^T(\nabla \varphi (s)F(s)-\frac{1}{2}{D}_1\widehat{W})+{ϵ}_{v1}(s)\hfill \\ \hfill =& W^T\nabla \varphi (s)F(s)+\frac{1}{2}{W}^T{D}_1(W-\widehat{W})-\frac{1}{2}{W}^T{D}_{1}W+{ϵ}_{v1}(s)\hfill \\ \hfill =& W^T\nabla \varphi (s)F(s)+\frac{1}{2}{W}^T{D}_1\tilde{W}-\frac{1}{2}{W}^T{D}_1(s)W+{ϵ}_{v1}(s)\hfill \\ \hfill =& W^T\sigma +\frac{1}{2}{W}^T{D}_1\tilde{W}+{ϵ}_{v1}(s),\hfill \end{array}$$

(45)

where ${ϵ}_{v1}(s)=\nabla {ϵ}_{v1}^T(s)(F(s)-\frac{1}{2}G(s)R^{-1}G{(s)}^T\nabla \varphi (s)\tilde{W})$, $\sigma =\nabla \varphi (s)(F(s)+G(s)u)$, and $D_1=\nabla \varphi \varrho {(\nabla \varphi )}^T$.

The second term ${\dot{L}}_W$ can be obtained by (41)

$$\begin{array}{cc}\hfill {\dot{L}}_W=& {\tilde{W}}^T{\beta }^{-1}\dot{\tilde{W}}\hfill \\ \hfill =& {\tilde{W}}^T{\beta }^{-1}[-\beta \overline{\xi }({\tilde{W}}^T\xi +\tilde{A}(s)-B_r(s))-\frac{\beta }{2}\mathsf{\Pi }(s,\widehat{u})\nabla \varphi \varrho J_1-\beta ((K_1{\theta }^T-K_2)\hfill \\ & -\tilde{A}(s))(W-\tilde{W})]\hfill \\ \hfill =& {\tilde{W}}^T[-\overline{\xi }({\tilde{W}}^T\xi +\tilde{A}(s)-B_r(s))-\frac{1}{2}\mathsf{\Pi }(s,\widehat{u})\nabla \varphi \varrho J_1-((K_1{\theta }^T-K_2)-\tilde{A}(s))(W-\tilde{W})]\hfill \\ \hfill =& {\tilde{W}}^T[-\overline{\xi }(({\tilde{W}}^T\xi -\omega )-\frac{1}{4}{\tilde{W}}^T{D}_1\tilde{W})-((K_1{\theta }^T-K_2)-\frac{D_1}{4}(W-\tilde{W})\frac{{\theta }^T}{m})(W-\tilde{W})]\hfill \\ & -\frac{1}{2}\mathsf{\Pi }(s,\widehat{u}){\tilde{W}}^T\nabla \varphi \varrho J_1\hfill \\ \hfill =& {\tilde{W}}^T[-\theta {\tilde{W}}^T\theta +\frac{\theta \omega }{m}+\frac{\theta {\tilde{W}}^T{D}_1\tilde{W}}{4}]-{\tilde{W}}^T[(K_1{\theta }^T-K_2)(W-\tilde{W})\hfill \\ & -\frac{D_1}{4}(W-\tilde{W}){(\theta /m)}^T(W-\tilde{W})]-c,\hfill \end{array}$$

(46)

where $m=1+{\xi }^T\xi$, $c=-\frac{\mathsf{\Pi }(s,\widehat{u}){\tilde{W}}^T\nabla \varphi \varrho J_1}{2}$. Further, we can obtain

$$\begin{array}{cc}\hfill {\dot{L}}_W=& -{\tilde{W}}^T\theta {\tilde{W}}^T\theta +\frac{{\tilde{W}}^T\theta \omega }{m}+\frac{{\tilde{W}}^T\theta {\tilde{W}}^T{D}_1\tilde{W}}{4}-{\tilde{W}}^T(K_1{\theta }^T-K_2)(W-\tilde{W})+\frac{{\tilde{W}}^T{D}_1{\theta }^T}{4m}(W\hfill \\ & -\tilde{W})(W-\tilde{W})-c\hfill \\ \hfill =& -{\tilde{W}}^T\theta {\tilde{W}}^T\theta +\frac{{\tilde{W}}^T\theta \omega }{m}+\frac{{\tilde{W}}^T\theta {\tilde{W}}^T{D}_1\tilde{W}}{4}-{\tilde{W}}^T(K_1{\theta }^T-K_2)W+{\tilde{W}}^T(K_1{\theta }^T-K_2)\tilde{W}\hfill \\ & +\frac{{\tilde{W}}^T{D}_{1}W{\theta }^T}{4m}W-\frac{{\tilde{W}}^T{D}_1\tilde{W}{\theta }^T}{4m}W-\frac{{\tilde{W}}^T{D}_{1}W{\theta }^T}{4m}\tilde{W}+\frac{{\tilde{W}}^T{D}_1\tilde{W}{\theta }^T}{4m}\tilde{W}-c.\hfill \end{array}$$

(47)

Taking the sum of the terms ${\dot{L}}_V$ and ${\dot{L}}_W$, we obtain

$$\begin{array}{cc}\hfill \dot{L}(t)=& W^T\sigma +\frac{1}{2}{W}^T{D}_1\tilde{W}-{\tilde{W}}^T\theta {\tilde{W}}^T\theta +\frac{{\tilde{W}}^T\theta \omega }{m}+\frac{{\tilde{W}}^T\theta {\tilde{W}}^T{D}_1\tilde{W}}{4}-{\tilde{W}}^T(K_1{\theta }^T-K_2)W\hfill \\ & +{\tilde{W}}^T(K_1{\theta }^T-K_2)\tilde{W}+\frac{{\tilde{W}}^T{D}_{1}W{\theta }^T}{4m}W-\frac{{\tilde{W}}^T{D}_1\tilde{W}{\theta }^T}{4m}W-\frac{{\tilde{W}}^T{D}_{1}W{\theta }^T}{4m}\tilde{W}\hfill \\ & +\frac{{\tilde{W}}^T{D}_1\tilde{W}{\theta }^T}{4m}\tilde{W}-c+{ϵ}_{v1}(s).\hfill \end{array}$$

(48)

Assume that $Z={[{\tilde{W}}^T\theta ,{\tilde{W}}^T]}^T$, then we can obtain

$$\dot{L}(t)=-Z^T\left[\begin{array}{cc}I& -\frac{W^T{D}_1}{8m}-\frac{K_1^T}{2}\\ -\frac{D_{1}W}{8m}-\frac{K_1}{2}& K_2-\frac{{\theta }^{T}W{D}_1}{4m}\end{array}\right]Z$$

$$+Z^T\left[\begin{array}{cc}\frac{\omega }{m}& \\ \frac{D_{1}W{\theta }^{T}W}{4m}+K_{2}W-K_1{\theta }^{T}W\end{array}\right]+b+d,$$

(49)

where

$$b=c+W^T\sigma +\parallel {ϵ}_{v1}(s)\parallel ,$$

(50)

$$d=\frac{W^T{D}_1(\tilde{W}{\theta }^T\tilde{W}-\tilde{W}{\theta }^{T}W-W{\theta }^T\tilde{W})}{4m}.$$

(51)

Define

$$M=\left[\begin{array}{cc}I& -\frac{W^T{D}_1}{8m}-\frac{K_1^T}{2}\\ -\frac{D_{1}W}{8m}-\frac{K_1}{2}& K_2-\frac{{\theta }^{T}W{D}_1}{4m}\end{array}\right],$$

(52)

$$a=\left[\begin{array}{cc}\frac{\omega }{m}& \\ \frac{D_{1}W{\theta }^{T}W}{4m}+K_{2}W-K_1{\theta }^{T}W\end{array}\right].$$

(53)

Let the tuning parameters $K_1$, $K_2$, and $\gamma$ be chosen so that $M>0$, and we obtain

$$\dot{L}(t)<-{\parallel \mathrm{Z}\parallel }^2{\sigma }_{min}(M)+\parallel a\parallel \parallel \mathrm{Z}\parallel +\mu ,$$

(54)

where $\mu =b+d$. In summary, the Lyapunov derivative $\dot{L}(t)$ is negative if

$$\parallel \mathrm{Z}\parallel >\frac{\parallel a\parallel }{2{\sigma }_{min}(M)}+\sqrt{{\displaystyle \frac{\mu }{{\sigma }_{min}(M)}}+{\displaystyle \frac{a^2}{4{\sigma }_{min}(M)}}}.$$

(55)

Based on the Lyapunov theorem [39], as long as the selected appropriately tuning parameters $K_1$, $K_2$, and $\gamma$ make the formula (32) hold, in the case of state constraints and uncertain disturbances, the critic NN weight error $\tilde{W}$, the system state x, and the control input $u^{*}$ are guaranteed to be UUB, and the nonlinear system (1) is guaranteed to be closed-loop stable in the presence of state constraints and uncertain disturbances. The proof is completed. □

5. Simulation

We consider a spring-mass-damping system with nonlinear properties [22], and the system dynamics of the spring-mass-damper are as follows [24]

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\dot{x}}_1=x_2,\hfill \\ {\dot{x}}_2=-\frac{K(x_2)}{M}-\frac{C}{M}{x}_1+\frac{1}{M}u+p{x}_{1}sin(x_2),\hfill \end{array}\right.\end{array}$$

(56)

where $x={[x_1,x_2]}^T\in {\mathbb{R}}^2$ and the nonlinear condition $K(x)=x^3$, $x_1$, and $x_2$ are the position and velocity, respectively, and u is the force applied to the object. M is the mass of the object. K is the stiffness constant of the spring, and C is the damping. The above system dynamics parameters are $M=$1 kg and $C=0.5$ N· s/m. A mismatched disturbance may lead to system instability. Considering that the system still has stable performance under disturbances, the uncertain disturbance matching the system is selected, the uncertain disturbance term $d(x)=p{x}_{1}sin(x_2)$, and we assume that $p\in [-1,1]$ and $d_M(x)=\parallel x\parallel$.

In the simulation process, considering that the initial allowable control law is not required, to make the tracking errors of the system converge to zero, the reference trajectory gradually tending to zero is selected, and the following reference trajectory $x_d(t)$ is given

$${\dot{x}}_d=\left[\begin{array}{c}-0.5x_{d1}-x_{d2}cos(x_{d1})\\ sin(x_{d1})-x_{d2}\end{array}\right],$$

(57)

the initial condition is given as $x_d(0)={[0.15,0.25]}^T$, and we set the augmented state vector as $s={[e_x^T,x_d^T]}^T$, and then combine (56) with (57), the dynamics of the augmented system can be derived

$$\dot{s}=\left[\begin{array}{c}{s}_2+s_4+0.5s_3+s_{4}cos(s_3)\\ -{(s_2+s_4)}^3-0.5(s_1+s_3)-sin(s_3)+s_4\\ -0.5s_3-s_{4}cos(s_3)\\ sin(s_3)-s_4\end{array}\right]+\left[\begin{array}{c}0\\ 1\\ 0\\ 0\end{array}\right](u+d(s)),$$

(58)

where $s={[s_1,s_2,s_3,s_4]}^T={[e_{x1},e_{x2},x_{d1},x_{d2}]}^T$ with $e_{xi}=x_i-x_{di}$.

To constrain the states of the system in augmented system dynamics (58), the control barrier function used is as follows

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{B}_{r1}(s_1+s_3)=-log(\frac{\gamma h(s_1+s_3)}{\gamma h(s_1+s_3)+1})\hfill \\ B_{r2}(s_2+s_4)=-log(\frac{\gamma h(s_2+s_4)}{\gamma h(s_2+s_4)+1}).\hfill \end{array}\right.\end{array}$$

(59)

The state constraints of the system are given as $-0.2\le x_1\le 0.35$ and $-0.15\le x_2\le 0.4$, and the parameter $\gamma =0.02$.

To complete the design of robust trajectory tracking control, the modified value function (14) can be specified as

$$\begin{array}{c}\hfill V(s(t))={\int }_t^{\infty }{e}^{-\alpha (\tau -t)}[\rho d_M^2(s(\tau ))+s^T(\tau )Q_{T}s(\tau )+u^T(\tau )Ru(\tau )+B_{r1}(s(\tau ))+B_{r2}(s(\tau ))]d(\tau ).\end{array}$$

(60)

Besides, we select the learning rate as $\beta =1.5$ and the discount factor $\alpha$ = $0.15$, respectively. In order to deal with the approximate optimal control for the nominal augmented part of (58), we choose $Q_T$ = $diag\{5I_2,0_{2\times 2}\}$ and $R=I$, and I denotes an identity matrix of appropriate dimensions. In this example, the activation function for the critic NN is chosen as $\varphi (s)$ = ${[s_1^2,s_1{s}_2,s_1{s}_3,s_1{s}_4,s_2^2,s_2{s}_3,s_2{s}_4,s_3^2,s_3{s}_4,s_4^2]}^T$. In addition, the weights of the critic NN are denoted as ${\widehat{W}}_c$ = ${[W_{c1},W_{c2},\dots ,W_{c10}]}^T$. The initial value of the state is given as $x(0)={[-0.2,0.4]}^T$, and it is easy to calculate the initial error vector according to $s(0)=x(0)-x_d(0)$, so the initial state of the augmented system is $s(0)={[-0.35,0.15,0.15,0.25]}^T$. In order to satisfy the condition of persistency of excitation, an exploration noise $exp(-0.25t)si{n}^2(t)cos(t)$ is added during the training of the neural network.

The convergence of critic parameters is shown in Figure 1, and the critic parameters after 30 s converge to $\widehat{W}={[3.3767,0.9606,0.8867,0.7752,1.9266,1.0686,1.105,1.067,1.0992,1.0898]}^T$. Figure 2 shows the control inputs of the system. Figure 3 shows the trajectory of the tracking errors $e_{x1}$ and $e_{x2}$ of the system without state constraints. Figure 4 shows the tracking error of the system under state constraints. Figure 5 and Figure 6 show that the system tracks the reference trajectory without state constraints, and we can see that the system states violate the constraints. Figure 7 and Figure 8 show that the system tracks the desired trajectory with state constraints, and that under the condition of state constraints and uncertain disturbances, the system still maintains good performance. The method described in this paper can ensure the stability of the closed-loop system. In summary, the simulation results display the effectiveness of the proposed method.

6. Conclusions

This paper presented a robust trajectory tracking method for nonlinear systems with state constraints and uncertain disturbances based on adaptive dynamic programming. Firstly, the system error was combined with the reference trajectory to construct the augmented system, and at the same time, the nominal system of the augmented system was considered. In order to overcome the uncertain disturbances of the augmented system, the discount coefficient was introduced into the nominal system, and the CBF was added into the nominal system with the discount coefficient to constrain the states of the system. In addition, cost functions and control strategies were learned by designing a guaranteed cost adaptive critic NN learning framework. Finally, the simulation results demonstrated that the described method can converge the system error within the state constraints. In the next work, we will try to extend the state constraints method to discrete-time tracking control systems and multi-agent systems.